* [9fans] system call trace version of 9vx available. @ 2010-05-19 0:22 ron minnich 2010-05-19 1:54 ` David Leimbach 0 siblings, 1 reply; 36+ messages in thread From: ron minnich @ 2010-05-19 0:22 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs http://bitbucket.org/rminnich/vx32 this is my hack of vx32. It includes a ram block device, but more importantly, it supports the system call trace code. In the 'root' of the repo is the syscalltrace directory; cd in there to make the syscalltrace binary. Comments and improvements welcome. Noah wrote the first version of syscalltrace.c. Thanks to Russ for suggestions that led to substantial improvements in the code; it's still not good enough (Jim McKie made a really great improvement which you can see in the 9k kernel) and I don't think it yet belongs in the main vx32 tree. That said, you can get a *lot* of work done with this version, and we've found syscalltrace very helpful on the bg/p effort, so I wanted to make sure people could see it. I've seen questions go by on this list lately that use of syscalltrace might have answered. I use syscalltrace all the time now when I hit a problem I don't understand. It would be helpful for those debugging cpu and auth problems, I think. Thanks to Russ, Jim, and Noah for their help and improvements. Any errors are mine. ron ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [9fans] system call trace version of 9vx available. 2010-05-19 0:22 [9fans] system call trace version of 9vx available ron minnich @ 2010-05-19 1:54 ` David Leimbach 2010-05-19 1:53 ` erik quanstrom ` (2 more replies) 0 siblings, 3 replies; 36+ messages in thread From: David Leimbach @ 2010-05-19 1:54 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs [-- Attachment #1: Type: text/plain, Size: 1365 bytes --] On Tue, May 18, 2010 at 5:22 PM, ron minnich <rminnich@gmail.com> wrote: > http://bitbucket.org/rminnich/vx32 > > this is my hack of vx32. > Were all of the binaries within recompiled against this code? Running 9vx on my iMac is pretty smooth! Dave > > It includes a ram block device, but more importantly, it supports the > system call trace code. > > In the 'root' of the repo is the syscalltrace directory; cd in there > to make the syscalltrace binary. > > Comments and improvements welcome. Noah wrote the first version of > syscalltrace.c. Thanks to Russ for suggestions that led to substantial > improvements in the code; it's still not good enough (Jim McKie made a > really great improvement which you can see in the 9k kernel) and I > don't think it yet belongs in the main vx32 tree. That said, you can > get a *lot* of work done with this version, and we've found > syscalltrace very helpful on the bg/p effort, so I wanted to make sure > people could see it. I've seen questions go by on this list lately > that use of syscalltrace might have answered. I use syscalltrace all > the time now when I hit a problem I don't understand. It would be > helpful for those debugging cpu and auth problems, I think. > > Thanks to Russ, Jim, and Noah for their help and improvements. Any > errors are mine. > > ron > > [-- Attachment #2: Type: text/html, Size: 1943 bytes --] ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [9fans] system call trace version of 9vx available. 2010-05-19 1:54 ` David Leimbach @ 2010-05-19 1:53 ` erik quanstrom 2010-05-19 2:13 ` ron minnich 2010-05-19 5:40 ` Bakul Shah 2 siblings, 0 replies; 36+ messages in thread From: erik quanstrom @ 2010-05-19 1:53 UTC (permalink / raw) To: 9fans > Were all of the binaries within recompiled against this code? Running 9vx > on my iMac is pretty smooth! trace device doesn't require user land support. - erik ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [9fans] system call trace version of 9vx available. 2010-05-19 1:54 ` David Leimbach 2010-05-19 1:53 ` erik quanstrom @ 2010-05-19 2:13 ` ron minnich 2010-05-19 4:37 ` David Leimbach 2010-05-19 5:40 ` Bakul Shah 2 siblings, 1 reply; 36+ messages in thread From: ron minnich @ 2010-05-19 2:13 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs On Wed, May 19, 2010 at 1:54 AM, David Leimbach <leimy2k@gmail.com> wrote: > Were all of the binaries within recompiled against this code? Running 9vx > on my iMac is pretty smooth! Hm, not sure what you mean. ron ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [9fans] system call trace version of 9vx available. 2010-05-19 2:13 ` ron minnich @ 2010-05-19 4:37 ` David Leimbach 2010-05-19 4:52 ` ron minnich 2010-05-19 11:06 ` Ethan Grammatikidis 0 siblings, 2 replies; 36+ messages in thread From: David Leimbach @ 2010-05-19 4:37 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs [-- Attachment #1: Type: text/plain, Size: 659 bytes --] On Tue, May 18, 2010 at 7:13 PM, ron minnich <rminnich@gmail.com> wrote: > On Wed, May 19, 2010 at 1:54 AM, David Leimbach <leimy2k@gmail.com> wrote: > > > Were all of the binaries within recompiled against this code? Running > 9vx > > on my iMac is pretty smooth! > > Hm, not sure what you mean. > > ron > > There were pre-built binaries in this cloned repo, I'm totally unable to rebuild the Mac OS X binary, due to some "impossible constraint" in some assembly or something like that per my recollection. I was merely wondering if you expected people to ignore those binaries and build their own or if those were already built. Dave [-- Attachment #2: Type: text/html, Size: 1038 bytes --] ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [9fans] system call trace version of 9vx available. 2010-05-19 4:37 ` David Leimbach @ 2010-05-19 4:52 ` ron minnich 2010-05-19 5:26 ` David Leimbach 2010-05-19 11:06 ` Ethan Grammatikidis 1 sibling, 1 reply; 36+ messages in thread From: ron minnich @ 2010-05-19 4:52 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs On Wed, May 19, 2010 at 4:37 AM, David Leimbach <leimy2k@gmail.com> wrote: > There were pre-built binaries in this cloned repo, I'm totally unable to > rebuild the Mac OS X binary, due to some "impossible constraint" in some > assembly or something like that per my recollection. I was merely wondering > if you expected people to ignore those binaries and build their own or if > those were already built. > Dave I just cloned rsc/vx32. That said, I wonder if when I pushed my clone to rminnich/vx32 it pushed some binaries? It should not have. The only changes I have made are to the 9vx/9vx part of vx32. ron ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [9fans] system call trace version of 9vx available. 2010-05-19 4:52 ` ron minnich @ 2010-05-19 5:26 ` David Leimbach 0 siblings, 0 replies; 36+ messages in thread From: David Leimbach @ 2010-05-19 5:26 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs [-- Attachment #1: Type: text/plain, Size: 936 bytes --] On Tue, May 18, 2010 at 9:52 PM, ron minnich <rminnich@gmail.com> wrote: > On Wed, May 19, 2010 at 4:37 AM, David Leimbach <leimy2k@gmail.com> wrote: > > > There were pre-built binaries in this cloned repo, I'm totally unable to > > rebuild the Mac OS X binary, due to some "impossible constraint" in some > > assembly or something like that per my recollection. I was merely > wondering > > if you expected people to ignore those binaries and build their own or if > > those were already built. > > Dave > > I just cloned rsc/vx32. That said, I wonder if when I pushed my clone > to rminnich/vx32 it pushed some binaries? It should not have. > > The only changes I have made are to the 9vx/9vx part of vx32. > > ron > > I see, well it could just be that I've not run 9vx in a while too :-) Things certainly seemed a lot easier to deal with this time around in certain areas around networking and such. Dave [-- Attachment #2: Type: text/html, Size: 1348 bytes --] ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [9fans] system call trace version of 9vx available. 2010-05-19 4:37 ` David Leimbach 2010-05-19 4:52 ` ron minnich @ 2010-05-19 11:06 ` Ethan Grammatikidis 1 sibling, 0 replies; 36+ messages in thread From: Ethan Grammatikidis @ 2010-05-19 11:06 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs [-- Attachment #1: Type: text/plain, Size: 994 bytes --] On 19 May 2010, at 05:37, David Leimbach wrote: > > > On Tue, May 18, 2010 at 7:13 PM, ron minnich <rminnich@gmail.com> > wrote: > On Wed, May 19, 2010 at 1:54 AM, David Leimbach <leimy2k@gmail.com> > wrote: > > > Were all of the binaries within recompiled against this code? > Running 9vx > > on my iMac is pretty smooth! > > Hm, not sure what you mean. > > ron > > There were pre-built binaries in this cloned repo, I'm totally > unable to rebuild the Mac OS X binary, due to some "impossible > constraint" in some assembly or something like that per my > recollection. This error? libvxc/syscall.h:8: error: can't find a register in class ‘BREG’ while reloading ‘asm’ What works for me on 10.5 is this: cd vx32/src make 9vx/9vx It runs; not extensively tested but appears to run fine. On 10.6 you may have to define _XOPEN_SOURCE as Bakul Shah described. -- Simplicity does not precede complexity, but follows it. -- Alan Perlis [-- Attachment #2: Type: text/html, Size: 4278 bytes --] ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [9fans] system call trace version of 9vx available. 2010-05-19 1:54 ` David Leimbach 2010-05-19 1:53 ` erik quanstrom 2010-05-19 2:13 ` ron minnich @ 2010-05-19 5:40 ` Bakul Shah 2010-05-19 5:45 ` erik quanstrom 2010-05-19 15:09 ` ron minnich 2 siblings, 2 replies; 36+ messages in thread From: Bakul Shah @ 2010-05-19 5:40 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs On Tue, 18 May 2010 18:54:21 PDT David Leimbach <leimy2k@gmail.com> wrote: > > Were all of the binaries within recompiled against this code? Running 9vx > on my iMac is pretty smooth! vx32/src/9vx.*.gz are the same as before (in case you are running those). Compiling 9vx on a MAC OS 10.6.3 required changing HOST_CC line in src/Makefrag to HOST_CC := $(CC) -fno-inline -m32 -D_XOPEN_SOURCE Undoubtedly there is a cleaner way. I copied vx32/src/9vx/syscalltrace/ to $home in 9vx and compiled syscalltrace there. term% 8.out -c /bin/echo Boo! 511 echo Brk 0x3233 0000b450 00000000 00000000 = 0 "" 0x11af077310cde470 0x11af077310d0da40 511 echo Pwrite 0x31d6 1 0000a458/"Boo!." 5 -0x1Boo! = 5 "" 0x11af07731e11e758 0x11af077326aed448 511 echo Open 0x32c0 00009ec0/"#c/pid" 00000000 = 3 "" 0x11af0773315f14c0 0x11af0773316790a0 511 echo Pread 0x3287 3 0fffff00/"........511." 20 -0x1 = 12 "" 0x11af07733f047958 0x11af07733f09cca0 511 echo Close 0x32ee 3 = 0 "" 0x11af07734a6dad78 0x11af07734a707c38 511 echo Exits 0x1bd5 0/""cwrite: /proc/511/syscall: failed 12 bytes: process exited tterm% Very nice! Is the the syscall trace data format easy to parse? If so it can be massaged in various ways. ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [9fans] system call trace version of 9vx available. 2010-05-19 5:40 ` Bakul Shah @ 2010-05-19 5:45 ` erik quanstrom 2010-05-19 15:13 ` ron minnich 2010-05-19 15:09 ` ron minnich 1 sibling, 1 reply; 36+ messages in thread From: erik quanstrom @ 2010-05-19 5:45 UTC (permalink / raw) To: 9fans > term% 8.out -c /bin/echo Boo! > 511 echo Brk 0x3233 0000b450 00000000 00000000 = 0 "" 0x11af077310cde470 0x11af077310d0da40 > 511 echo Pwrite 0x31d6 1 0000a458/"Boo!." 5 -0x1Boo! > = 5 "" 0x11af07731e11e758 0x11af077326aed448 > 511 echo Open 0x32c0 00009ec0/"#c/pid" 00000000 = 3 "" 0x11af0773315f14c0 0x11af0773316790a0 > 511 echo Pread 0x3287 3 0fffff00/"........511." 20 -0x1 = 12 "" 0x11af07733f047958 0x11af07733f09cca0 > 511 echo Close 0x32ee 3 = 0 "" 0x11af07734a6dad78 0x11af07734a707c38 > 511 echo Exits 0x1bd5 0/""cwrite: /proc/511/syscall: failed 12 bytes: process exited > tterm% ron, please enlighten the ignorant. could you constrast this with truss? or maybe there's a man page? - erik ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [9fans] system call trace version of 9vx available. 2010-05-19 5:45 ` erik quanstrom @ 2010-05-19 15:13 ` ron minnich 0 siblings, 0 replies; 36+ messages in thread From: ron minnich @ 2010-05-19 15:13 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs On Tue, May 18, 2010 at 10:45 PM, erik quanstrom <quanstro@quanstro.net> wrote: > ron, please enlighten the ignorant. could you constrast this with > truss? or maybe there's a man page? I need to write one. The biggest diff from truss is that the program itself is dead simple, since most of the work is done by the kernel and the data relayed to the program as text. The program is really a fancier, fork-following version of this script: while (echo startsyscall > /proc/$1/ctl) cat /proc/$1/syscalltrace All the formatting etc. is in the kernel. ron ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [9fans] system call trace version of 9vx available. 2010-05-19 5:40 ` Bakul Shah 2010-05-19 5:45 ` erik quanstrom @ 2010-05-19 15:09 ` ron minnich 2010-05-19 17:18 ` Bakul Shah 1 sibling, 1 reply; 36+ messages in thread From: ron minnich @ 2010-05-19 15:09 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs On Tue, May 18, 2010 at 10:40 PM, Bakul Shah <bakul+plan9@bitblocks.com> wrote: > term% 8.out -c /bin/echo Boo! > 511 echo Brk 0x3233 0000b450 00000000 00000000 = 0 "" 0x11af077310cde470 0x11af077310d0da40 > 511 echo Pwrite 0x31d6 1 0000a458/"Boo!." 5 -0x1Boo! > = 5 "" 0x11af07731e11e758 0x11af077326aed448 > 511 echo Open 0x32c0 00009ec0/"#c/pid" 00000000 = 3 "" 0x11af0773315f14c0 0x11af0773316790a0 > 511 echo Pread 0x3287 3 0fffff00/"........511." 20 -0x1 = 12 "" 0x11af07733f047958 0x11af07733f09cca0 > 511 echo Close 0x32ee 3 = 0 "" 0x11af07734a6dad78 0x11af07734a707c38 > 511 echo Exits 0x1bd5 0/""cwrite: /proc/511/syscall: failed 12 bytes: process exited > tterm% The format arose out of discussions with nemo and others. It is a straight text layout of system call params and return. The = separates the params and return. The format is: pid textname syscall-name pc [params] = retval errstr start-nanoseconds end-nanoseconds Anything that is a pointer to memory gets printed this way: pointervalue/"string" The string has a '.' printed if isgraph(char) is 0. So, example: 511 echo Open 0x32c0 00009ec0/"#c/pid" 00000000 = 3 "" 0x11af0773315f14c0 0x11af0773316790a0 pid 511, echo, did an Open, at pc 0x32c0, file name was at 9ec0 and the value was "#c/pid", mode 0, and the result was 3, no error as the errstr was empty: "", and it took 0x11af0773316790a0 - 0x11af0773315f14c0 nanoseconds or 556000 nanoseconds. I need to fix syscalltrace to get rid of this annoying non-error: cwrite: /proc/511/syscall: failed 12 bytes: process exited Just have not done it yet, I'll take a patch. ron ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [9fans] system call trace version of 9vx available. 2010-05-19 15:09 ` ron minnich @ 2010-05-19 17:18 ` Bakul Shah 2010-05-19 17:41 ` ron minnich ` (2 more replies) 0 siblings, 3 replies; 36+ messages in thread From: Bakul Shah @ 2010-05-19 17:18 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs On Wed, 19 May 2010 08:09:50 PDT ron minnich <rminnich@gmail.com> wrote: > > The format arose out of discussions with nemo and others. > > It is a straight text layout of system call params and return. The = > separates the params and return. The format is: > pid textname syscall-name pc [params] = retval errstr > start-nanoseconds end-nanoseconds > > Anything that is a pointer to memory gets printed this way: > pointervalue/"string" > > The string has a '.' printed if isgraph(char) is 0. One can grok most of this by staring at the syscalltrace output. The small hex value after syscall name -- is it the return pc? Very nice but now some gripes! 0. Name syscalltrace is too long :-) 1. Curiously, an actual errstr is not enclosed in "..". 2. Seems long long values are printed as two numbers (but I haven't checked carefully enough to be sure). 3. Printing . for isgraph(char) loses information. 4. Perhaps buffersize should be settable to deal with args pointing to large areas. 5. Occasional hangs. Not sure what the cause is. What happens if the passed in ptr points to invalid memory? Given 3., in the kernel I would've just copied a binary structure and let the userland worry about any formatting. I "fixed" a few things in syscalltrace (diff below): - strace -c echo boo # extra look up in /bin if needed - strace -c rc -c 'ls /' # allow - args to command - strace -c strace -c echo foo # allow recursive use strace -c rc </dev/null |[2] wc # send trace output to stderr, not stdout. There is still some bug that makes syscalltrace hang, particularly if its output is piped to another process. Trivia: rc makes 192 syscalls, on freebsd $PLAN9/bin/rc makes 157, /bin/sh makes 87, and zsh makes 11259 calls! I can imagine showing syscalls of each traced process on a separate scrolling time line. This will allow one to see timing relationships and call patterns. Zoom to see appropriate scale, click on a call (or select a range) to see call details. Can also show who forks whom, with a connecting line to the new timeline that just popped up for the new process. Color can be used for procs or syscalls. etc. Not sure when/if I will get around to implementing something like this though. -- bakul diff ../syscalltrace/syscalltrace.c ./syscalltrace.c 4a5 > #include <stdio.h> 121c122 < print("%s", s->buf); --- > fprint(2, "%s", s->buf); 151c152 < break; --- > goto done; 155c156 < --- > done: 164a166,170 > if (cmd[0] != '/') { > char* pcmd = malloc(strlen(cmd) + 5); > sprintf(pcmd, "/bin/%s", cmd); > exec(pcmd, args); > } ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [9fans] system call trace version of 9vx available. 2010-05-19 17:18 ` Bakul Shah @ 2010-05-19 17:41 ` ron minnich 2010-05-19 18:23 ` Bakul Shah 2010-05-19 17:51 ` erik quanstrom 2010-05-19 17:53 ` ron minnich 2 siblings, 1 reply; 36+ messages in thread From: ron minnich @ 2010-05-19 17:41 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs On Wed, May 19, 2010 at 10:18 AM, Bakul Shah <bakul+plan9@bitblocks.com> wrote: > 0. Name syscalltrace is too long :-) pick a name and I'll change it. > 1. Curiously, an actual errstr is not enclosed in "..". that goes on the bug list. If you want, use the bugtracker at bitbucket.org, on my rminnich/vx32 project, and add these bugs. Also add your changes below as an enhancement and I'll apply it. > 2. Seems long long values are printed as two numbers > (but I haven't checked carefully enough to be sure). that's an error, but I have not seen it. > 3. Printing . for isgraph(char) loses information. that's life. I'm happy to consider a \hex as an alternate print scheme, but that complicates parsing? Not sure. Advice welcome. nemo? > 4. Perhaps buffersize should be settable to deal with > args pointing to large areas. yes. It's on the list of "things I want" > 5. Occasional hangs. Not sure what the cause is. weird. Have not seen this. If you can reproduce it let me know. > > What happens if the passed in ptr points to invalid memory? I used uvalidaddr and you'll get the same error as you would in the system call: suicide. > > Given 3., in the kernel I would've just copied a binary > structure and let the userland worry about any formatting. no. No, no, no, no, no. no. If there is any consistent thing in Plan 9, it's that ctl and other files use text. It's great. This binary idea is a cancer. 1. with text, I can mount /proc from an ARM, and do system call tracing on my 386 laptop: text is architecture-independent 2. with kernel formatting, I can use awk and rc and perl and whatever I want to implement tracing 3. textual formats don't need versioning of binary structures. So, no. It stays as text :-) > I can imagine showing syscalls of each traced process on a > separate scrolling time line. This will allow one to see > timing relationships and call patterns. Zoom to see > appropriate scale, click on a call (or select a range) to see > call details. Can also show who forks whom, with a > connecting line to the new timeline that just popped up for > the new process. Color can be used for procs or syscalls. > etc. Not sure when/if I will get around to implementing > something like this though. That's a very fine idea. Will our draw libraries support an interface with this degree of capability? Note that it's very easy, on Plan 9, with a shell script, to put every traced proc into a different window. Thanks for the corrections! ron ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [9fans] system call trace version of 9vx available. 2010-05-19 17:41 ` ron minnich @ 2010-05-19 18:23 ` Bakul Shah 2010-05-19 18:31 ` erik quanstrom ` (2 more replies) 0 siblings, 3 replies; 36+ messages in thread From: Bakul Shah @ 2010-05-19 18:23 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs On Wed, 19 May 2010 10:41:26 PDT ron minnich <rminnich@gmail.com> wrote: > On Wed, May 19, 2010 at 10:18 AM, Bakul Shah <bakul+plan9@bitblocks.com> wrote > > > 0. Name syscalltrace is too long :-) > > pick a name and I'll change it. I used strace but don't really care what you call it as long as it is short! How about ratrace (Ron's ascii trace)? > > 3. Printing . for isgraph(char) loses information. > > that's life. I'm happy to consider a \hex as an alternate print > scheme, but that complicates parsing? Not sure. Advice welcome. nemo? A middle ground is to use counted bytestrings. For example 120:<120 bytes of random goop> > This binary idea is a cancer. > 1. with text, I can mount /proc from an ARM, and do system call > tracing on my 386 laptop: text is architecture-independent > 2. with kernel formatting, I can use awk and rc and perl and whatever > I want to implement tracing > 3. textual formats don't need versioning of binary structures. > > So, no. It stays as text :-) Ok! I don't feel strongly either way. But I hope you do consider counted bytestrings to represent random memory. It is cheap to parse and produce and doesn't lose info. ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [9fans] system call trace version of 9vx available. 2010-05-19 18:23 ` Bakul Shah @ 2010-05-19 18:31 ` erik quanstrom 2010-05-19 18:33 ` ron minnich 2010-05-19 18:46 ` John Floren 2 siblings, 0 replies; 36+ messages in thread From: erik quanstrom @ 2010-05-19 18:31 UTC (permalink / raw) To: bakul+plan9, 9fans > Ok! I don't feel strongly either way. But I hope you do > consider counted bytestrings to represent random memory. > It is cheap to parse and produce and doesn't lose info. no keep it text. plan 9 wins this way. "bytestrings" (it seems to me that byte strings are neither) would require knowledge of the producing kernel. that's just problematic with plan 9. - erik ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [9fans] system call trace version of 9vx available. 2010-05-19 18:23 ` Bakul Shah 2010-05-19 18:31 ` erik quanstrom @ 2010-05-19 18:33 ` ron minnich 2010-05-19 20:30 ` Bakul Shah 2010-05-19 18:46 ` John Floren 2 siblings, 1 reply; 36+ messages in thread From: ron minnich @ 2010-05-19 18:33 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs On Wed, May 19, 2010 at 11:23 AM, Bakul Shah <bakul+plan9@bitblocks.com> wrote: > Ok! I don't feel strongly either way. But I hope you do > consider counted bytestrings to represent random memory. > It is cheap to parse and produce and doesn't lose info. bear in mind that 99.9999999% of the time (well, that's an exaggeration!) people do a quick one-off run of this type of tool to see something. Thus, it should be biased to human readability. Counted bytestrings doesn't quite do that. But I can escape the characters that are not printable if you want. \@through \whatever ron ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [9fans] system call trace version of 9vx available. 2010-05-19 18:33 ` ron minnich @ 2010-05-19 20:30 ` Bakul Shah 2010-05-19 20:38 ` ron minnich 0 siblings, 1 reply; 36+ messages in thread From: Bakul Shah @ 2010-05-19 20:30 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs On Wed, 19 May 2010 11:33:24 PDT ron minnich <rminnich@gmail.com> wrote: > On Wed, May 19, 2010 at 11:23 AM, Bakul Shah <bakul+plan9@bitblocks.com> wrote: > > > Ok! I don't feel strongly either way. =A0But I hope you do > > consider counted bytestrings to represent random memory. > > It is cheap to parse and produce and doesn't lose info. > > bear in mind that 99.9999999% of the time (well, that's an > exaggeration!) people do a quick one-off run of this type of tool to > see something. Thus, it should be biased to human readability. Counted > bytestrings doesn't quite do that. But I can escape the characters > that are not printable if you want. > \@through \whatever Thanks. For a quick one-off it may not matter but can be valuable when you are looking for a needle in the haystack. kdump for example shows all the data (and uses a sidebyside addr,hex,ascii format when there are unprintable chars). Note that most people are likely to use a frontend tool than directly cat /proc/<pid>/syscall. Which reminds me... Is there a reason why just doing cat /proc/<pid>/syscall shouldn't start tracing? Seems to me, opening the device should be enough to start tracing and closing it enough to stop tracing. You write "startsyscall" to <pid>/syscall for every trace buffer read -- don't quite understand why that is needed. -- bakul truss excerpt: read(3,"libpthread.so.1 libthr.so.1\nlib"...,4096) = 81 (0x51) read(3,0x800535000,4096) = 0 (0x0) close(3) = 0 (0x0) open("/var/run/ld-elf.so.hints",O_RDONLY,0160) = 3 (0x3) read(3,"Ehnt\^A\0\0\0\M^@\0\0\0\M-,\^B\0"...,128) = 128 (0x80) Corresponding ktrace/kdump excerpt: 61555 cat CALL read(0x3,0x800535000,0x1000) 61555 cat GIO fd 3 read 81 bytes "libpthread.so.1 libthr.so.1 libpthread.so.2 libthr.so.2 libkse.so.3 libthr.so.3 " 61555 cat RET read 81/0x51 61555 cat CALL read(0x3,0x800535000,0x1000) 61555 cat GIO fd 3 read 0 bytes "" 61555 cat RET read 0 61555 cat CALL close(0x3) 61555 cat RET close 0 61555 cat CALL open(0x80052d74b,O_RDONLY,<unused>0x70) 61555 cat NAMI "/var/run/ld-elf.so.hints" 61555 cat RET open 3 61555 cat CALL read(0x3,0x7fffffffdc00,0x80) 61555 cat GIO fd 3 read 128 bytes 0x0000 4568 6e74 0100 0000 8000 0000 ac02 0000 |Ehnt............| 0x0010 0000 0000 ab02 0000 0000 0000 0000 0000 |................| 0x0020 0000 0000 0000 0000 0000 0000 0000 0000 |................| 0x0030 0000 0000 0000 0000 0000 0000 0000 0000 |................| 0x0040 0000 0000 0000 0000 0000 0000 0000 0000 |................| 0x0050 0000 0000 0000 0000 0000 0000 0000 0000 |................| 0x0060 0000 0000 0000 0000 0000 0000 0000 0000 |................| 0x0070 0000 0000 0000 0000 0000 0000 0000 0000 |................| 61555 cat RET read 128/0x80 More verbose but more informative and more useful. ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [9fans] system call trace version of 9vx available. 2010-05-19 20:30 ` Bakul Shah @ 2010-05-19 20:38 ` ron minnich 2010-05-19 21:48 ` ron minnich 2010-05-19 22:02 ` Bakul Shah 0 siblings, 2 replies; 36+ messages in thread From: ron minnich @ 2010-05-19 20:38 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs On Wed, May 19, 2010 at 1:30 PM, Bakul Shah <bakul+plan9@bitblocks.com> wrote: > You write "startsyscall" to <pid>/syscall for every trace > buffer read -- don't quite understand why that is needed. It gives you the option of not restarting the system call until later. There could be more complex usage scenarios. But it's a very good point. We've talked about dumping this data via the trace facility instead. Trace facility is well designed for this type of output -- I've got a proposal in which Sape says is ok for changing the format of the trace records. So, things can still change. One question though -- once you get to the point of really complex examination of system calls, would acid be better? Or do you see a use for ratrace (that name is kind of funny and has the virtue of being typable on my keyboard with one hand) just to dump more data and make all data visible? Your point about lost information is a good one. ron ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [9fans] system call trace version of 9vx available. 2010-05-19 20:38 ` ron minnich @ 2010-05-19 21:48 ` ron minnich 2010-05-19 22:41 ` ron minnich 2010-05-19 22:02 ` Bakul Shah 1 sibling, 1 reply; 36+ messages in thread From: ron minnich @ 2010-05-19 21:48 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs 1.name changed to ratrace 2. output now to stderr 3. phooey. For some reason when a syscall fails the errstr prints an empty string. 9vx/trap.c: any suggestions? ron ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [9fans] system call trace version of 9vx available. 2010-05-19 21:48 ` ron minnich @ 2010-05-19 22:41 ` ron minnich 2010-05-19 22:52 ` Nick LaForge 0 siblings, 1 reply; 36+ messages in thread From: ron minnich @ 2010-05-19 22:41 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs On Wed, May 19, 2010 at 2:48 PM, ron minnich <rminnich@gmail.com> wrote: > 3. phooey. For some reason when a syscall fails the errstr prints an > empty string. 9vx/trap.c: any suggestions? Fixed. Ron ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [9fans] system call trace version of 9vx available. 2010-05-19 22:41 ` ron minnich @ 2010-05-19 22:52 ` Nick LaForge 0 siblings, 0 replies; 36+ messages in thread From: Nick LaForge @ 2010-05-19 22:52 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs You THINK you have lots of control when you're dropping 'into' acid. ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [9fans] system call trace version of 9vx available. 2010-05-19 20:38 ` ron minnich 2010-05-19 21:48 ` ron minnich @ 2010-05-19 22:02 ` Bakul Shah 2010-05-19 22:25 ` ron minnich 1 sibling, 1 reply; 36+ messages in thread From: Bakul Shah @ 2010-05-19 22:02 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs On Wed, 19 May 2010 13:38:36 PDT ron minnich <rminnich@gmail.com> wrote: > On Wed, May 19, 2010 at 1:30 PM, Bakul Shah <bakul+plan9@bitblocks.com> wro= > te: > > > You write "startsyscall" to <pid>/syscall for every trace > > buffer read -- don't quite understand why that is needed. > > It gives you the option of not restarting the system call until later. > There could be more complex usage scenarios. I don't understand this. > But it's a very good point. We've talked about dumping this data via > the trace facility instead. Trace facility is well designed for this > type of output -- I've got a proposal in which Sape says is ok for > changing the format of the trace records. You will go over to the dark side of binary formats? Horrors! > So, things can still change. One question though -- once you get to > the point of really complex examination of system calls, would acid be > better? Or do you see a use for ratrace (that name is kind of funny > and has the virtue of being typable on my keyboard with one hand) just > to dump more data and make all data visible? Your point about lost > information is a good one. There is value in being able to trace syscalls outside of a debugger. For instance I have done things like strace -f make -j14 |& grep 'open("' ktrace -di make -j14 && kdump | grep 'open("' to look at parallel build problems due to messed up dependencies. I have used tracing to find all the annoying hidden config files, dlopened shared libs, pam (pluggable authentication module) and what not that programs seem to use these days. I like the simplicity of cat <pid>/syscall! ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [9fans] system call trace version of 9vx available. 2010-05-19 22:02 ` Bakul Shah @ 2010-05-19 22:25 ` ron minnich 2010-05-19 22:46 ` Devon H. O'Dell 2010-05-20 0:26 ` Bakul Shah 0 siblings, 2 replies; 36+ messages in thread From: ron minnich @ 2010-05-19 22:25 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs On Wed, May 19, 2010 at 3:02 PM, Bakul Shah <bakul+plan9@bitblocks.com> wrote: >> It gives you the option of not restarting the system call until later. >> There could be more complex usage scenarios. > > I don't understand this. You read the "start of the system call" message. The process is stopped. It has not run the system call. You have a lot of options at that point. You could, programatically, drop the person into acid (the debugger, not the liquid) and, when they exit, resume the process. You really do have lots of control. > You will go over to the dark side of binary formats? Horrors! no, the change might include not being binary. Don't know. thanks ron ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [9fans] system call trace version of 9vx available. 2010-05-19 22:25 ` ron minnich @ 2010-05-19 22:46 ` Devon H. O'Dell 2010-05-20 0:26 ` Bakul Shah 1 sibling, 0 replies; 36+ messages in thread From: Devon H. O'Dell @ 2010-05-19 22:46 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs 2010/5/19 ron minnich <rminnich@gmail.com>: > On Wed, May 19, 2010 at 3:02 PM, Bakul Shah <bakul+plan9@bitblocks.com> wrote: > >>> It gives you the option of not restarting the system call until later. >>> There could be more complex usage scenarios. >> >> I don't understand this. > > You read the "start of the system call" message. The process is > stopped. It has not run the system call. > > You have a lot of options at that point. You could, programatically, > drop the person into acid (the debugger, not the liquid) and, when > they exit, resume the process. You really do have lots of control. But there's nothing more fun than watching someone scream in a HCl bath of high molarity, especially after they've made a dumb programming error! (Sorry for the noise, I had to.) --dho ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [9fans] system call trace version of 9vx available. 2010-05-19 22:25 ` ron minnich 2010-05-19 22:46 ` Devon H. O'Dell @ 2010-05-20 0:26 ` Bakul Shah 2010-05-20 0:32 ` ron minnich 1 sibling, 1 reply; 36+ messages in thread From: Bakul Shah @ 2010-05-20 0:26 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs On Wed, 19 May 2010 15:25:52 PDT ron minnich <rminnich@gmail.com> wrote: > On Wed, May 19, 2010 at 3:02 PM, Bakul Shah <bakul+plan9@bitblocks.com> wrote > : > > >> It gives you the option of not restarting the system call until later. > >> There could be more complex usage scenarios. > > > > I don't understand this. > > You read the "start of the system call" message. The process is > stopped. It has not run the system call. > > You have a lot of options at that point. You could, programatically, > drop the person into acid (the debugger, not the liquid) and, when > they exit, resume the process. You really do have lots of control. What I don't understand is why do I need these options. I just want tracing. I don't want to do acid (debugger, low pH liquid or LSD)! But never mind. I probably won't get it. Why is the performance so poor? cd ratrace mk clean time ratrace -o /dev/null -c mk # about 19.67 seconds mk clean time mk # about 0.88 seconds And here I thought naming it ratrace would make it go faster. [yes, I added the -o option for precisely this sort of thing) ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [9fans] system call trace version of 9vx available. 2010-05-20 0:26 ` Bakul Shah @ 2010-05-20 0:32 ` ron minnich 2010-05-20 5:46 ` ron minnich 2010-05-21 7:07 ` Bakul Shah 0 siblings, 2 replies; 36+ messages in thread From: ron minnich @ 2010-05-20 0:32 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs On Wed, May 19, 2010 at 5:26 PM, Bakul Shah <bakul+plan9@bitblocks.com> wrote: > time ratrace -o /dev/null -c mk # about 19.67 seconds did you want [2]>/dev/null? > mk clean > time mk # about 0.88 seconds > > And here I thought naming it ratrace would make it go faster. Speed is left as an exercise for the reader. But, if you want, you can use ratrace to trace ratrace to see how long things take. I assume it is the cost of doing the output, but who knows? ron ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [9fans] system call trace version of 9vx available. 2010-05-20 0:32 ` ron minnich @ 2010-05-20 5:46 ` ron minnich 2010-05-21 7:07 ` Bakul Shah 1 sibling, 0 replies; 36+ messages in thread From: ron minnich @ 2010-05-20 5:46 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs I had a question from someone about the syscalltrace. It has not relationship to the earlier devtrace work that Floren and I did. Thanks ron ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [9fans] system call trace version of 9vx available. 2010-05-20 0:32 ` ron minnich 2010-05-20 5:46 ` ron minnich @ 2010-05-21 7:07 ` Bakul Shah 1 sibling, 0 replies; 36+ messages in thread From: Bakul Shah @ 2010-05-21 7:07 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs On Wed, 19 May 2010 17:32:19 PDT ron minnich <rminnich@gmail.com> wrote: > On Wed, May 19, 2010 at 5:26 PM, Bakul Shah <bakul+plan9@bitblocks.com> wro= > te: > > > time ratrace -o /dev/null -c mk # about 19.67 seconds > > did you want [2]>/dev/null? No, because that eats time output as well. My change uses a new fd if -o is specified, 2 otherwise. > > mk clean > > time mk # about 0.88 seconds > > > > And here I thought naming it ratrace would make it go faster. > > Speed is left as an exercise for the reader. On a 5 year old T42 thinkpad running freebsd: mk clean time ratrace -o /dev/null -c mk # 2.57 seconds mk clean time mk # 0.18 seconds So may be it is 9vx (and freebsd on i386 vs macos on x86_64) But there are other issues: term% cat x.c #include <u.h> #include <libc.h> void main(int c, char**v) { fork()? print("parent\n") : print("child\n"); exits(nil); } term% 8c x.c && 8l x.8 && ratrace -c 8.out 178 8.out Rfork 0x1259 00002014child = 180 "" 0x11af831799634530 0x11af83179b3a4368 178 8.out Pwrite 0x297d 1 0ffffe0c/"parent." 7 -0x1parent = 6 "" 0x11af83179f9d7a60 0x11af8317a93a9d28 = 7 "" 0x11af8317b1743468 0x11af8317bb73ed78 180 8.out Open 0x120b 0000601c/"#c/pid" 00000000 = 3 "" 0x11af8317c915bc18 0x11af8317c91b5998 178 8.out Open 0x120b 0000601c/"#c/pid" 00000000 = 3 "" 0x11af8317d3cd6700 0x11af8317d3d2fcb0 180 8.out Pread 0x29b0 3 0fffff2c/"........180." 20 -0x1 = 12 "" 0x11af8317dea5ad90 0x11af8317deab08a8 180 8.out Close 0x1239 3 = 0 "" 0x11af8317ebe03a98 0x11af8317ebe39210 180 8.out Exits 0x118b 0/""cwrite: /proc/180/syscall: failed 12 bytes: process exited 178 8.out Pread 0x29b0 3 0fffff2c/"........178." 20 -0x1 = 12 "" 0x11af8317fac10d30 0x11af8317fac65c90 178 8.out Close 0x1239 3 = 0 "" 0x11af83180d0e04c0 0x11af83180d113910 178 8.out Exits 0x118b 0/""cwrite: /proc/178/syscall: failed 12 bytes: process exited term% Notice the child's Pwrite is missing but its result shows up. After the 2nd line of ratrace output there should be = 0 "" 0x11af831799634530 0x11af83179b3XXXXX to indicate rfork's return in the child but that is missing too. I need to find time to hack on this! ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [9fans] system call trace version of 9vx available. 2010-05-19 18:23 ` Bakul Shah 2010-05-19 18:31 ` erik quanstrom 2010-05-19 18:33 ` ron minnich @ 2010-05-19 18:46 ` John Floren 2 siblings, 0 replies; 36+ messages in thread From: John Floren @ 2010-05-19 18:46 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs On Wed, May 19, 2010 at 2:23 PM, Bakul Shah <bakul+plan9@bitblocks.com> wrote: > On Wed, 19 May 2010 10:41:26 PDT ron minnich <rminnich@gmail.com> wrote: >> On Wed, May 19, 2010 at 10:18 AM, Bakul Shah <bakul+plan9@bitblocks.com> wrote >> >> > 0. Name syscalltrace is too long :-) >> >> pick a name and I'll change it. > > I used strace but don't really care what you call it as long > as it is short! How about ratrace (Ron's ascii trace)? > I think "ratrace" is a damn fine name, given that it's both "Ron's ASCII trace" and "rat race" John -- "With MPI, familiarity breeds contempt. Contempt and nausea. Contempt, nausea, and fear. Contempt, nausea, fear, and .." -- Ron Minnich ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [9fans] system call trace version of 9vx available. 2010-05-19 17:18 ` Bakul Shah 2010-05-19 17:41 ` ron minnich @ 2010-05-19 17:51 ` erik quanstrom 2010-05-19 17:59 ` ron minnich 2010-05-19 17:53 ` ron minnich 2 siblings, 1 reply; 36+ messages in thread From: erik quanstrom @ 2010-05-19 17:51 UTC (permalink / raw) To: 9fans > > if (cmd[0] != '/') { > > char* pcmd = malloc(strlen(cmd) + 5); > > sprintf(pcmd, "/bin/%s", cmd); > > exec(pcmd, args); > > } shouldn't that be if (cmd[0] != '/') exec(smprint("/bin/%s", cmd), args); with no silly stdio? - erik ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [9fans] system call trace version of 9vx available. 2010-05-19 17:51 ` erik quanstrom @ 2010-05-19 17:59 ` ron minnich 0 siblings, 0 replies; 36+ messages in thread From: ron minnich @ 2010-05-19 17:59 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs On Wed, May 19, 2010 at 10:51 AM, erik quanstrom <quanstro@labs.coraid.com> wrote: >> > if (cmd[0] != '/') { >> > char* pcmd = malloc(strlen(cmd) + 5); >> > sprintf(pcmd, "/bin/%s", cmd); >> > exec(pcmd, args); >> > } > > shouldn't that be > > if (cmd[0] != '/') > exec(smprint("/bin/%s", cmd), args); > indeed. ron ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [9fans] system call trace version of 9vx available. 2010-05-19 17:18 ` Bakul Shah 2010-05-19 17:41 ` ron minnich 2010-05-19 17:51 ` erik quanstrom @ 2010-05-19 17:53 ` ron minnich 2010-05-19 18:43 ` Bakul Shah 2 siblings, 1 reply; 36+ messages in thread From: ron minnich @ 2010-05-19 17:53 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs I'll only take that patch if it does NOT include stdio.h. As for output ... I'm conflicted on output on 1 vs. 2. But it is nice that you can see normal output of the traced process. But, hmm, if traced process prints on 2, well ... you'll lose it. So, my feeling is, if you are *really* concerned about output and interaction with the process, run rc in another window, and syscalltrace <pid-of-rc> ron ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [9fans] system call trace version of 9vx available. 2010-05-19 17:53 ` ron minnich @ 2010-05-19 18:43 ` Bakul Shah 2010-05-19 18:51 ` ron minnich 0 siblings, 1 reply; 36+ messages in thread From: Bakul Shah @ 2010-05-19 18:43 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs On Wed, 19 May 2010 10:53:59 PDT ron minnich <rminnich@gmail.com> wrote: > I'll only take that patch if it does NOT include stdio.h. Well, you have the trivial diff so do what you want. > As for output ... I'm conflicted on output on 1 vs. 2. But it is nice > that you can see normal output of the traced process. But, hmm, if > traced process prints on 2, well ... you'll lose it. 1 is the working channel -- where most useful thing show up. The issue is that the output of the traced process is intermingled with syscalltrace and that just doesn't make sense. They are different things. With output to 2 I can redirect it where I want. BTW, truss does the same thing (output to stderr). ktrace on FreeBSD finesses by just dumping trace output to a file and then kdump is used to show it. > So, my feeling is, if you are *really* concerned about output and > interaction with the process, run rc in another window, and > syscalltrace <pid-of-rc> Too much work. ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [9fans] system call trace version of 9vx available. 2010-05-19 18:43 ` Bakul Shah @ 2010-05-19 18:51 ` ron minnich 2010-05-19 18:57 ` erik quanstrom 0 siblings, 1 reply; 36+ messages in thread From: ron minnich @ 2010-05-19 18:51 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs On Wed, May 19, 2010 at 11:43 AM, Bakul Shah <bakul+plan9@bitblocks.com> wrote: > BTW, truss does the same thing (output to stderr). ktrace on > FreeBSD finesses by just dumping trace output to a file and > then kdump is used to show it. > strace on linux sends to stderr as well. OK, you win, I'll change that one. It bothers me to lose error output however. I should probably open /dev/cons on fd 3 and let people redirect or something. That way, default behavior is simple and we still have child process owning 1` and 2. ron ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [9fans] system call trace version of 9vx available. 2010-05-19 18:51 ` ron minnich @ 2010-05-19 18:57 ` erik quanstrom 0 siblings, 0 replies; 36+ messages in thread From: erik quanstrom @ 2010-05-19 18:57 UTC (permalink / raw) To: 9fans > strace on linux sends to stderr as well. > > OK, you win, I'll change that one. It bothers me to lose error output however. if you really need bug-for-bug compatability, you can always ratrace >[1=2] | ... - erik ^ permalink raw reply [flat|nested] 36+ messages in thread
end of thread, other threads:[~2010-05-21 7:07 UTC | newest] Thread overview: 36+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2010-05-19 0:22 [9fans] system call trace version of 9vx available ron minnich 2010-05-19 1:54 ` David Leimbach 2010-05-19 1:53 ` erik quanstrom 2010-05-19 2:13 ` ron minnich 2010-05-19 4:37 ` David Leimbach 2010-05-19 4:52 ` ron minnich 2010-05-19 5:26 ` David Leimbach 2010-05-19 11:06 ` Ethan Grammatikidis 2010-05-19 5:40 ` Bakul Shah 2010-05-19 5:45 ` erik quanstrom 2010-05-19 15:13 ` ron minnich 2010-05-19 15:09 ` ron minnich 2010-05-19 17:18 ` Bakul Shah 2010-05-19 17:41 ` ron minnich 2010-05-19 18:23 ` Bakul Shah 2010-05-19 18:31 ` erik quanstrom 2010-05-19 18:33 ` ron minnich 2010-05-19 20:30 ` Bakul Shah 2010-05-19 20:38 ` ron minnich 2010-05-19 21:48 ` ron minnich 2010-05-19 22:41 ` ron minnich 2010-05-19 22:52 ` Nick LaForge 2010-05-19 22:02 ` Bakul Shah 2010-05-19 22:25 ` ron minnich 2010-05-19 22:46 ` Devon H. O'Dell 2010-05-20 0:26 ` Bakul Shah 2010-05-20 0:32 ` ron minnich 2010-05-20 5:46 ` ron minnich 2010-05-21 7:07 ` Bakul Shah 2010-05-19 18:46 ` John Floren 2010-05-19 17:51 ` erik quanstrom 2010-05-19 17:59 ` ron minnich 2010-05-19 17:53 ` ron minnich 2010-05-19 18:43 ` Bakul Shah 2010-05-19 18:51 ` ron minnich 2010-05-19 18:57 ` erik quanstrom
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).