steven

Using KGDB tracepoint module(KGTP)

steven | 05 September, 2010 22:06

Using KGDB tracepoint module(KGTP)
                         To trace your real-time program

Steven Yang, DSLab
Aug 2010


Asa powerful debug tool, GDB tracepoint could observe the behavior your real-time program whose orrectness depends on its real-time behavior, delays introduced by a debugger might cause the program to change its behavior drastically, or perhaps fail, even when the code itself is correct.

http://sourceware.org/gdb/current/onlinedocs/gdb/Tracepoints.html#Tracepoints

http://code.google.com/p/kgtp/

I tested the GTP on
    Intel Pentium 3 CPU
    GDB 7.2
    -O0 kernel
    GTP20100830 or later

The steps are as following:

1 Compile kernel with -O0 optimization
===================
  1) Disable CONFIG_CC_OPTIMIZE_FOR_SIZE
     General setup
    -> Optimize for size
  2) And then modify the Makefile of kernel:
     replace
      KBUILD_CFLAGS   += -O2
     with
        KBUILD_CFLAGS   += -O0
 
  Also even -O0 optimizes out values - you could mark those values volatile to prevent that from happening though.

2 Install the latest GDB(7.2)
===================
  #wget ftp://ftp.gnu.org/gnu/gdb/gdb-7.2.tar.gz
  #tar -xzf gdb-7.2.tar.gz
  #cd gdb
  #./configure
  #make
  #make install
  Note: I found gdb fail to 'collect' some global variables and local variables using gdb 6.8 on Intel-Pentium 3.

3 Get the latest GTP source code
===================
  #svn checkout http://kgtp.googlecode.com/svn/ kgtp
  Further info please refer to http://code.google.com/p/kgtp/

4 Compile and load KGTP
===================
  #cd kgtp/trunk/
  #make
  #insmod gtp.ko

5 Trace kernel functions
===================
    ------------------------------------screen snapshot begin
    debian:/home/linux-2.6.33.6# whereis gdb
    gdb: /usr/local/bin/gdb
    debian:/home/linux-2.6.33.6# /usr/local/bin/gdb vmlinux
    GNU gdb (GDB) 7.2
    Copyright (C) 2010 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
    and "show warranty" for details.
    This GDB was configured as "i686-pc-linux-gnu".
    For bug reporting instructions, please see:
    <http://www.gnu.org/software/gdb/bugs/>...
    Reading symbols from /home/linux-2.6.33.6/vmlinux...done.
    (gdb) target remote /proc/gtp
    Remote debugging using /proc/gtp
    0x00000000 in ?? ()
    (gdb) trace vfs_readdir
    Tracepoint 1 at 0xc11756a0: file fs/readdir.c, line 24.
    (gdb) trace fs/readdir.c:29
    Tracepoint 2 at 0xc11756ca: file fs/readdir.c, line 29.
    (gdb) actions 1
    Enter actions for tracepoint 1, one per line.
    End with a line saying just "end".
    >collect *file
    >end
    (gdb) actions 2
    Enter actions for tracepoint 2, one per line.
    End with a line saying just "end".
    >collect res
    >end
    (gdb) tstart
    (gdb) shell ls
    COPYING        Module.symvers  crypto    ipc              net       usr
    CREDITS        README          drivers   kernel           samples   virt
    Documentation  REPORTING-BUGS  firmware  lib              scripts   vmlinux
    Kbuild         System.map      fs        mm               security  vmlinux.o
    MAINTAINERS    arch            include   modules.builtin  sound     vmlinux_O2
    Makefile       block           init      modules.order    tools
    (gdb) tstop
    (gdb) tfind
    Found trace frame 0, tracepoint 1
    #0  0xc11756a1 in vfs_readdir (file=0xc3082800, filler=0, buf=0x0)
        at fs/readdir.c:24
    24              struct inode *inode = file->f_path.dentry->d_inode;
    (gdb) p *file
    $1 = {f_u = {fu_list = {next = 0xc6d8b780, prev = 0xc7ade480}, fu_rcuhead = {
          next = 0xc6d8b780, func = 0xc7ade480}}, f_path = {mnt = 0xc6c05300,
        dentry = 0xc761d440}, f_op = 0xc1709a20, f_lock = {{rlock = {raw_lock = {
          slock = 0}}}}, f_count = {counter = 2}, f_flags = 624640,
      f_mode = 29, f_pos = 0, f_owner = {lock = {raw_lock = {lock = 16777216}},
        pid = 0x0, pid_type = PIDTYPE_PID, uid = 0, euid = 0, signum = 0},
      f_cred = 0xc3082780, f_ra = {start = 0, size = 0, async_size = 0,_pos = -1}, fte_data = 0x0, f_ep_links = {
        next = 0xc3082874, prev = 0xc3082874}, f_mapping = 0xc761c74c}
    (gdb) tfind
    Found trace frame 1, tracepoint 2
    0xc11756cb      29              res = security_file_permission(file, MAY_READ);
    (gdb) p res
    $2 = -20
    (gdb) list
    24              struct inode *inode = file->f_path.dentry->d_inode;
    25              int res = -ENOTDIR;
    26              if (!file->f_op || !file->f_op->readdir)
    27                      goto out;
    28
    29              res = security_file_permission(file, MAY_READ);
    30              if (res)
    31                      goto out;
    32
    33              res = mutex_lock_killable(&inode->i_mutex);
    (gdb)
    ------------------------------------screen snapshot end
      
6 Trace kernel module
===================
debian:~# cat /proc/modules | grep 3c59x
3c59x 42521 0 - Live 0xc86dd000
debian:~# modinfo 3c59x
filename:       /lib/modules/2.6.33.6-tp/kernel/drivers/net/3c59x.ko
author:         Donald Becker <becker@scyld.com>
description:    3Com 3c59x/3c9xx ethernet driver

    -------------------------------------------SCREEN SNAPSHOT BEGIN
    debian:/home/linux-2.6.33.6# /usr/local/bin/gdb vmlinux
    GNU gdb (GDB) 7.2
    Copyright (C) 2010 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
    and "show warranty" for details.
    This GDB was configured as "i686-pc-linux-gnu".
    For bug reporting instructions, please see:
    <http://www.gnu.org/software/gdb/bugs/>...
    Reading symbols from /home/linux-2.6.33.6/vmlinux...done.
    (gdb) target remote /proc/gtp
    Remote debugging using /proc/gtp
    0x00000000 in ?? ()
    (gdb) add-symbol-file /lib/modules/2.6.33.6-tp/kernel/drivers/net/3c59x.ko 0xc86dd000
    add symbol table from file "/lib/modules/2.6.33.6-tp/kernel/drivers/net/3c59x.ko" at
        .text_addr = 0xc86dd000
    (y or n) y
    Reading symbols from /lib/modules/2.6.33.6-tp/kernel/drivers/net/3c59x.ko...done.
    (gdb) trace boomerang_start_xmit
    Tracepoint 1 at 0xc86df4bc: file drivers/net/3c59x.c, line 2094.
    (gdb) actions
    Enter actions for tracepoint 1, one per line.
    End with a line saying just "end".
    >collect *dev
    >end
    (gdb) trace drivers/net/3c59x.c:2107
    Tracepoint 2 at 0xc86df509: file drivers/net/3c59x.c, line 2107.
    (gdb) actions
    Enter actions for tracepoint 2, one per line.
    End with a line saying just "end".
    >collect ioaddr
    >collect vp->cur_tx
    >end
    (gdb) tstart
        //...wait until packet received
    (gdb) tstop
    (gdb) tfind
    Found trace frame 0, tracepoint 1
    #0  0xc86df4bd in boomerang_start_xmit (skb=0x0, dev=0xc7a39800)
        at drivers/net/3c59x.c:2094
    2094            struct vortex_private *vp = netdev_priv(dev);
    (gdb) p *dev
    $1 = {name = "eth0", '00' <repeats 11 times>, name_hlist = {next = 0x0,
        pprev = 0xc785aec0}, ifalias = 0x0, mem_end = 0, mem_start = 0,
      base_addr = 3364381696, irq = 5, if_port = 8 '\b', dma = 0 '00',
      state = 3, dev_list = {next = 0xc1a91e18, prev = 0xc7a0d834}, napi_list = {
        next = 0xc7a3983c, prev = 0xc7a3983c}, unreg_list = {next = 0xc7a39844,
        prev = 0xc7a39844}, features = 2051, ifindex = 4, iflink = 4, stats = {
        rx_packets = 13880, tx_packets = 6880, rx_bytes = 1359797,
        tx_bytes = 1356809, rx_errors = 0, tx_errors = 0, rx_dropped = 0,
        tx_dropprx_length_errors 0, rx_frame_errors = 0,
        //...omit some content
    (gdb) tfind
    Found trace frame 1, tracepoint 2
    0xc86df50a      2107            if (vp->cur_tx - vp->dirty_tx >= TX_RING_SIZE) {
    (gdb) p ioaddr
    $2 = (void *) 0xc8886400
    (gdb) p vp->cur_tx
    $3 = 7049
    (gdb) list
    2102                    pr_debug("boomerang_start_xmit()\n");
    2103                    pr_debug("%s: Trying to send a packet, Tx index %d.\n",
    2104                               dev->name, vp->cur_tx);
    2105            }
    2106
    2107            if (vp->cur_tx - vp->dirty_tx >= TX_RING_SIZE) {
    2108                    if (vortex_debug > 0)
    2109                            pr_warning("%s: BUG! Tx Ring full, refusing to send buffer.\n",
    2110                                       dev->name);
    2111
        (gdb)
    -------------------------------------------SCREEN SNAPSHOT END




7 Trace my hello module
===================

#insmod hello.ko
#cat /proc/modules | grep hello
hello 1092 0 - Live 0xc95f7000
#cat /sys/module/hello/sections/.bss
0xc95f7308

    -------------------------------------------SCREEN SNAPSHOT BEGIN
    #gdb vmlinux
    GNU gdb (GDB) 7.2
    Copyright (C) 2010 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
    and "show warranty" for details.
    This GDB was configured as "i686-pc-linux-gnu".
    For bug reporting instructions, please see:
    <http://www.gnu.org/software/gdb/bugs/>...
    Reading symbols from /home/linux-2.6.33.6/vmlinux...done.
    (gdb) target remote /proc/gtp
    Remote debugging using /proc/gtp
    0x00000000 in ?? ()
    (gdb) add-symbol-file /home/steven/hello/hello.ko 0xc95f7000 -s .bss 0xc95f7308
    add symbol table from file "/home/steven/hello/hello.ko" at
        .text_addr = 0xc95f7000
        .bss_addr = 0xc95f7308
    (y or n) y
    Reading symbols from /home/steven/hello/hello.ko...done.
    (gdb) trace /home/steven/hello/hello.c:22
    Tracepoint 1 at 0xc95f7094: file /home/steven/hello/hello.c, line 22.
    (gdb) actions
    Enter actions for tracepoint 1, one per line.
    End with a line saying just "end".
    >collect tv,year,cnt
    >end
    (gdb) tstart
              //wait until your tracepoint has been trigger...
[90603.898148] tv_sec: 1283802951 - 6
[90603.903361] tv_sec: 1283802951 - 2010
   (gdb) tstop
    (gdb) tfind
    Found trace frame 0, tracepoint 1
    #0  0xc95f7095 in test_thread (data=0x0) at /home/steven/hello/hello.c:22
    22            cnt++;
    (gdb) p tv
    $1 = {tv_sec = 1283802951, tv_usec = 313616}
    (gdb) p cnt
    $2 = 6
    (gdb) p year
    $3 = 2010
    (gdb)

    -------------------------------------------SCREEN SNAPSHOT END

hello.c


  1 #include <linux/kthread.h>
  2 #include <linux/module.h>
  3 #include <linux/init.h>
  4
  5 static struct task_struct *test_task;
  6 volatile struct timeval tv;
  7 int year;
  8
  9 int test_thread(void *data)
 10 {
 11         volatile int cnt = 0;
 12         set_current_state(TASK_INTERRUPTIBLE);
 13         schedule_timeout(1 * HZ); /* wait for the terminal ready */
 14
 15         while(1){
 16                 set_current_state(TASK_INTERRUPTIBLE); /* this is required, or thread never sleep*/
 17                 if(kthread_should_stop()) break;
 18
 19                 year = 2010;
 20                 do_gettimeofday (&tv);
 21                 printk("\ntv_sec: %d - %d\n", (int)tv.tv_sec, cnt);
 22                 cnt++;
 23
 24                 do_gettimeofday (&tv);
 25                 printk("tv_sec: %d - %d\n", (int)tv.tv_sec, year);
 26                 cnt++;
 27
 28                 //reschedule after 5 seconds(approximately)
 29                 schedule_timeout(60 * HZ);
 30         }
 31
 32         return 0;
 33 }
 34
 35 static int __init mt_init(void)
 36 {
 37         int err;
 38         test_task = kthread_create(test_thread, NULL, "test_task");
 39         if(IS_ERR(test_task)){
 40                 printk("Unable to start kernel thread.\n");
 41                 err = PTR_ERR(test_task);
 42                 test_task = NULL;
 43                 return err;
 44         }
 45         wake_up_process(test_task);
 46         return 0;
 47 }
 48
 49
 50 static void __exit mt_cleanup(void)
 51 {
 52         if(test_task){
 53                 kthread_stop(test_task);
 54                 test_task = NULL;
 55         }
 56 }
 57
 58 module_init(mt_init);
 59 module_exit(mt_cleanup);
 60
 61 MODULE_LICENSE("GPL");

 

Notice:
1) Do not change the path of your module after you compile it and before you debug it.
2) If I use add-symbol-file /home/steven/hello/hello.ko 0xc95f7000, gdb can not print the variables correctly(only get zero).
3) Please use gdb 7.2 or later version.

 

Comments

1 2 3 4 5 6 7  Next»

So fun article is! I agree the idea!

Coach Outlet Stores | 02/08/2011, 00:01

So fun article is! I agree the idea!

Re: Using KGDB tracepoint module(KGTP)

baby boy bedding | 10/10/2011, 04:24

Great post!

This is really interesting, You are a very skilled blogger. I have joined your feed and look forward to seeking more of your fantastic post. Also, I've shared your site in my social networks!

North Face UK | 11/10/2011, 17:36

This is really interesting, You are a very skilled blogger. I have joined your feed and look forward to seeking more of your fantastic post. Also, I've shared your site in my social networks!

An additional little important Coach Outlet Online information we gamble Coach Factory Outlet Online you might not get recognized is Coach Outlet Online would be the the majority of searched Coach Diaper Bag Outlet along with popular bags. The smart and proper response Coach Factory Outlet would be to research somewhere else for that distinct Coach Factory Outlet Online bag and many more until you have Coach Outlet Online and don't quit your own ambitions. If your aspiration is to individual Coach Factory Outlet Online purses, and then suggest your current desires become a reality on the internet and also finding that ladies Coach Factory Outlet Online handbag that you want.

Coach Outlet Online | 21/11/2011, 17:22

An additional little important Coach Outlet Online information we gamble Coach Factory Outlet Online you might not get recognized is Coach Outlet Online would be the the majority of searched Coach Diaper Bag Outlet along with popular bags. The smart and proper response Coach Factory Outlet would be to research somewhere else for that distinct Coach Factory Outlet Online bag and many more until you have Coach Outlet Online and don't quit your own ambitions. If your aspiration is to individual Coach Factory Outlet Online purses, and then suggest your current desires become a reality on the internet and also finding that ladies Coach Factory Outlet Online handbag that you want.

North Face

North Face | 21/11/2011, 17:52

European capitals are more deficient in exhibitions or entertainments of this nature. There is

coach outlet coupon

coach outlet coupon | 17/12/2011, 00:24

At coach outlet on the internet internet websites, you could come across different authentic coach handbags, coach outlet coupon purses and coach wallets at discounted costs and, you also get 80% off retail coach goods. coach factory outlet solutions, although offered at minimal price tag, are guaranteed to be genuine and not counterfeited and greatest high quality. All coach coupons goods are along with unique coach tags.
juicy couture outlet can be the world's most irreverent, glamorous, and pleasurable existence style producer for ladies and girls of all ages. Founded in Los Angeles by two fashion-obsessed wonderful friends, juicy couture outlet is acknowledged for producing the observe suit an icon of casual luxury.
uggs boots with its magnificence and grace grow to be so well-liked nowadays. Practically everybody would like to wear this skeepskin boots to go by way of winter, even Timberland boots as the ancestor of uggs outlet boots from the globe, persons of comprehensive ages still select UGGs not having hesitate.
The north face outlet Denali jacket is by far the best selling fleece jacket from the The north face outlet collection. The Denali jacket functions perfectly as an insulating layer under a waterproof shell for outdoor activities like hiking and skiing. You can also wear a north face outlet Denali fleece as an outer layer to block light rain and snow.

Hope you work better and better.

cheap uggs | 19/12/2011, 17:08

Hope you work better and better.

Coach

Coach | 21/12/2011, 05:33

url,[url=http://www.coachs-factorystores.org/]Coach Factory Store[/url],a title h1,Chanel Purses.a title,Coach Factory Store,a h1.Coach Outlet.a, the one,
Coach Factory Store.

Discount North Face

Discount North Face | 22/12/2011, 00:43

motion was made, at the last term, to set aside the nonsuit on the following grounds: 1. That by the

ugg アグ ブーツ

ugg アグ ブーツ | 23/12/2011, 17:34

adiPUREとadiZERO最ugg アグ ブーツ
初に考えたのを持っナイキ ジョーダンて来るために、過去に確かにadidasのサッカーシューズの2つです、そして2011年から2つの用語はもはや排他的なサッカーのシリーズはありサッカースパイクません。 adiPUREとadiZEROのバスケットボールシューズの出現をされているとして、アディダスのテニスの製品ラインもadiPURE PROとadiZEROフェザーは、2つの新しいテニスシューズをオープンしました。カテゴリーは違いますが、家族の特性が接続されていますが、adiPURE PROは、外観の古典的なシンプルさと性能の全体的なバランスをもたらす、とadiZEROフェザー好調ファミリは、スピード感を持って最も鮮やかに反映さadiZEROなります。靴のファンの可用性の認識の考え方のジョーダンシューズー概念を作成し、唯一の私たちが待って見ることができるようにフルラインの製品ラインを持つこのサッカーシューズなど

Coach products on Coach Outlet Store1941.net are from authentic coach factory,sale without tax,FREE SHIPPING on orders over $80. In addition, Coach has built a strong presence in the U.S. through Coach outlet boutiques located within select department stores and specialty retailer locations. In 1999, Coach outlet launched its on-line store at www.coach.com(coach outlet online)- First coach outlet online store.In a Manhattan loft, six artisans handcrafted a collection of leather goods using skills handed down from generation to generation. Discerning consumers soon began to seek out the quality and unique nature ofCoach outlet online craftsmanship. As of July 2, 2011, there are nearly 500 Coach outlet stores in the United States and Canada, with more expected to open this calendar year.In 2011,Coach launched its on-line store at coach outlet online store

yan80376577yan80376577 | 23/12/2011, 18:04

Coach products on Coach Outlet Store1941.net are from authentic coach factory,sale without tax,FREE SHIPPING on orders over $80.

In addition, Coach has built a strong presence in the U.S. through Coach outlet boutiques located within select department stores and specialty retailer locations. In 1999, Coach outlet launched its on-line store at www.coach.com(coach outlet online)- First coach outlet online store.In a Manhattan loft, six artisans handcrafted a collection of leather goods using skills handed down from generation to generation. Discerning consumers soon began to seek out the quality and unique nature ofCoach outlet online craftsmanship.

As of July 2, 2011, there are nearly 500 Coach outlet stores in the United States and Canada, with more expected to open this calendar year.In 2011,Coach launched its on-line store at coach outlet online store

Louis Vuitton Bags

Gucci Outlet | 27/12/2011, 16:52

Gucci Outlet You will start looking at truly a few sort of gucci bags with captivating pattern and specific pattern while you click our gucci outlet store online. Collectively with to area the classification of possibilities ordered. layout and layout and layout and layout wild colours and crisp lines, filled with youthful elegance. Ultra texture inside the merchandise or support parts, highlighting the excellent level of level of excellent level of exceptional and luxury.

Coach Outlet by which each and every and just about every bag from our mentor store is not merely durable but in inclusion is you advantageous companion wether in working or life. just about each and every and just about every and just about every one and just about just about every one and just about just about every one only just one solitary bag from our mentor store is not merely durable but in inclusion is you excellent companion wether in working or life. Welcome you can nicely be found out to our mentor Outlet Online. Our mentor Outlet store offers a tremendous amount of truly a few marketing and marketing and marketing and marketing and marketing and marketing and marketing and marketing actions to our customers.

Louis Vuitton Bags store, we usually provide our purchasers producing utilization to the tremendous amount of truly a few the tremendous amount of great-quality louis vuitton bags and louis vuitton shoulderbags. Moreover, you also can uncover pretty a few other possibilities inside the louis vuitton here. Loved producing utilization using the tremendous amount of of our customers, our louis vuitton on-line is concentrated in delivering a area for grownup grownup males and ladies these instances who adore the Louis Vuitton products.Stylish appearances, remarkable exceptional materials, well-known producer brand name, all of people make Louis Vuitton Bags produce to turn into selected in world wide.

Alvina Charlotte

Alvinacharlotte | 06/01/2012, 07:04

Took a lot of time to read but I really found this very interesting and informative, thank you buddy for sharing.
SEO Company

SEO Services

SEM Services
I think we need to bring more ideas for this purpose. Involvement of young people can be handy in this regard. I am happy to find a good post here. Thank you
Social Media Marketing Services

Pay Per Click Services

Link Building Services

Content Writing Services
Very informative post. Keep up the good work. I would really look forward to your other posts
Logo Design

Logo Design

Cheap Logo Design

Affordable Logo Design
Thanks for your marvelous posting! I quite enjoyed reading it, you are a great author.I will be sure to bookmark your blog and definitely will come back from now on. I want to encourage that you continue your great job, have a nice day!
Logo Design Contests

Cheap Logo Design

Essay Help

Buy Essay
Nice and very helpful information i have got from your post. Even your whole blog is full of interesting information which is the great sign of a great blogger.
Cheap Custom Essays

Help With Essay

Write My Essay

Dissertation Help
Good post. Thanks for sharing this useful information with us.
BA Dissertation

BSC Dissertation

Business Dissertation

Doctoral Dissertation

Economics Dissertation

Engineering Dissertation

Finance Dissertation

Graduate Dissertation

International Finance Dissertation

Journalism Dissertation

MBA Dissertation

MSC Dissertation

Music Dissertation

PhD Dissertation

Sports Dissertation

Statistics Dissertation

Undergraduate Dissertation

Hogan

Hogan | 10/01/2012, 23:30

Hogan
Il nostro premier negozio di scarpe on the web haya una vasta gamma di hogan sito ufficiale.Sony ericsson express cercando every l . a . più efficace nei panni di artista italiano si possono trovare nella direzione di uno dei luoghi più ideale! Styledrops il negozio di rete fornisce le hogan scarpe genuini any prezzi più convenienti. prendere inside considerazione il vostro pace any guardare my partner and i prezzi, elizabeth guardare le scarpe Hogan Recensioni di vitale dalle sezioni scarpe scarpe hogan del nostro sito. Si possono trovare inside tutto uno dei prezzo di commercializzazione più efficace sulle scarpe Hogan mai, elizabeth spendere meno ancora di più minus le nostre hogan outlet promozioni enorme elizabeth occasioni!
Appear nel net più noti Scarpe Hogan negozio specializzato inside scarpe da artista, abiti male outlet hogan impotence accessori, Styledrops offre not assortimento completo di scarpe Hogan every uomo, sul mercato inside una gamma estremamente ampia di tipi di ease and comfort hogan interactive appear all'aperto, sports activity, everyday, città. si possono trovare elizabeth acquistare le scarpe inside tutto il vostro desiderio Hogan uomini proprio qui any Styledrops: il nostro compito sarà quello di possono trovare inside tutto uno dei prodotti più efficaci any sito ufficiale hogan disposizione every voi inside uno dei costi più efficace every il mercato.

High quality and inexpensive you deserve

Coach Outlet Store Online | 16/01/2012, 17:46

High quality and inexpensive you deserve

Tiffany and Co Outlet

Tiffany and Co Outlet | 30/01/2012, 16:39

Exclamation diversity, write this blog to the life; Let people deeply touched. Very happy to be able to browse the search to such a good blog too thank you have your support me more confident.

Mulberry Factory Shop

Mulberry Factory Shop | 31/01/2012, 17:59

U notice, simply because it is at the line, does not imply you may have a trade relationship. So, as long as you work hard, not what is not do. Come on we see your results.

Add comment
  The code is case-insensitive  If the code is not clear enough, please click the image to refresh | Maximum attempts: 10
1 2 3 4 5 6 7  Next»
 
Accessible and Valid XHTML 1.0 Strict and CSS
© DSLab - Powered by LifeType - Design by BalearWeb