[9fans] crashing 9vx

[9fans] crashing 9vx

[ron minnich]

I'm trying to create a reproducer. This little gem does crash it
frequently. Note that tar and mk install are doing lots of creates too
-- you're welcome to try this.

term% cat lotsafiles.c
#include <u.h>
#include <libc.h>

void
main(int, char**)
{
	int i = 0;
	while (1) {
		int fd;
		char *name = smprint("/tmp/%d", i);
		fd = create(name, OWRITE, 0666);
		if (fd < 0) exits(nil);
		close(fd);
		i++;
	}
}

Re: [9fans] crashing 9vx

[Philippe Anel]

Could not crash with your program, but it crashed quite fast with this one:


#include <u.h>
#include <libc.h>
#include <thread.h>

enum {
    NPROC=10,
};

void
crproc(void * p)
{
    int i = (int)p;

    while (1) {
        int fd;
        char *name = smprint("/tmp/%d", i);
        fd = create(name, OWRITE, 0666);
        if (fd < 0) exits(nil);
        // free(name);
        close(fd);
        i += NPROC;
    }
}

void
threadmain(int, char**)
{
    for (int n = 0; n < NPROC-1; n++)
        proccreate(crproc, (void *)n, 4096);
    crproc((void *) (NPROC-1));
}


ron minnich wrote:
> I'm trying to create a reproducer. This little gem does crash it
> frequently. Note that tar and mk install are doing lots of creates too
> -- you're welcome to try this.
>
> term% cat lotsafiles.c
> #include <u.h>
> #include <libc.h>
>
> void
> main(int, char**)
> {
> 	int i = 0;
> 	while (1) {
> 		int fd;
> 		char *name = smprint("/tmp/%d", i);
> 		fd = create(name, OWRITE, 0666);
> 		if (fd < 0) exits(nil);
> 		close(fd);
> 		i++;
> 	}
> }
>
>
>

Re: [9fans] crashing 9vx

[Philippe Anel]

I rewrote a simple version with fork(). And got a crash until I move :

#ifndef __APPLE__
    if(!usetty && !nofork && fork() > 0)
        _exit(0);
#endif

before mach0init();

Now it no longer crashes.

Phil;

Re: [9fans] crashing 9vx

[Philippe Anel]

Sorry, I was talking about 9vx/main.c:^main of course.
But forget it, it still crashes ... it just takes more time before crashing.
I suspected the fork in main() because I do not have (for a very very
long time) crash with '-F' flag (ie nofork).

Phil;

Philippe Anel wrote:
> I rewrote a simple version with fork(). And got a crash until I move :
>
> #ifndef __APPLE__
>    if(!usetty && !nofork && fork() > 0)
>        _exit(0);
> #endif
>
> before mach0init();
>
> Now it no longer crashes.
>
> Phil;
>
>

Re: [9fans] crashing 9vx

[ron minnich]

On Fri, May 28, 2010 at 6:40 AM, Philippe Anel <xigh@bouyapop.org> wrote:
> Could not crash with your program, but it crashed quite fast with this one:


and mine did not crash at all with that one. What system were you on?

ron

Re: [9fans] crashing 9vx

[Philippe Anel]

centos 5.4 x64 with your 9vx.

ron minnich wrote:
> On Fri, May 28, 2010 at 6:40 AM, Philippe Anel <xigh@bouyapop.org> wrote:
>
>> Could not crash with your program, but it crashed quite fast with this one:
>>
>
>
> and mine did not crash at all with that one. What system were you on?
>
> ron
>
>
>

Re: [9fans] crashing 9vx

[ron minnich]

you folks who are crashing 9vx in startup.

If you run it -g, what happens?

ron

Re: [9fans] crashing 9vx

[Charles Forsyth]

>If you run it -g, what happens?

init: starting /bin/rc
1416: signal: sys: segmentation violation

Re: [9fans] crashing 9vx

[Philippe Anel]

FYI, I found that in src/9vx/main.c, in main(), setmach(&mach0) is
called before mach0init() and thus machkeyinit() ... if you move
machkeyinit() from mach0init() to main(), before setmach(&mach0) ... do
you still have a crash ?

BTW, it does not fix the lotsafiles bug ...

Phil;

Charles Forsyth wrote:
>> If you run it -g, what happens?
>>
>
> init: starting /bin/rc
> 1416: signal: sys: segmentation violation
>
>
>

Re: [9fans] crashing 9vx

[ron minnich]

On Fri, May 28, 2010 at 11:36 AM, Charles Forsyth <forsyth@terzarima.net> wrote:
>>If you run it -g, what happens?
>
> init: starting /bin/rc
> 1416: signal: sys: segmentation violation
>
>

Wow. That's a level of quality and fulfillment I did not believe
possible. 32-bit or 64-bit?

gcc -v -v says what?

gcc version 4.4.3 (Ubuntu 4.4.3-4ubuntu5)

for me.

Could it be because your laptop is 220V?

ron

Re: [9fans] crashing 9vx

[erik quanstrom]

> Wow. That's a level of quality and fulfillment I did not believe
> possible. 32-bit or 64-bit?
>
> gcc -v -v says what?
>
> gcc version 4.4.3 (Ubuntu 4.4.3-4ubuntu5)
>
> for me.
>
> Could it be because your laptop is 220V?

never attribute to funny hardware that
which can be adequately explained by
broken locking.

- erik

Re: [9fans] crashing 9vx

[ron minnich]

OK, somebody sent a hint that it might make sense to take the -O3 out
of the make flags. Done.

Result: I can now get through this command:
hget -v http://plan9.bell-labs.com/plan9/download/plan9.iso.bz2>/tmp/iso.bz2
|[2]aux/statusbar plan9.iso

without an explosion.

We'll see. I'm always ready to suspect gcc since it has been a cause
of so much trouble in the past :-)

ron

Re: [9fans] crashing 9vx

[EBo]

On Sat, 29 May 2010 03:39:46 +0000, ron minnich <rminnich@gmail.com>
wrote:
> OK, somebody sent a hint that it might make sense to take the -O3 out
> of the make flags. Done.
>
> Result: I can now get through this command:
> hget -v
> http://plan9.bell-labs.com/plan9/download/plan9.iso.bz2>/tmp/iso.bz2
> |[2]aux/statusbar plan9.iso
>
> without an explosion.

Interesting.  I had not noticed that 9vx was building on my machines with:

  gcc -fno-inline -c -g -O3 -MD -std=gnu99 -O2 -march=i486 -pipe

and

  gcc -m32 -c -nostdinc --g -O3 -MD -std=gnu99 -O2 -march=i486 -pipe
-fno-stack-protector -m80387 -mfp-ret-in-387

in other places.  I would have removed -O3 if I had noticed it before.
I'm curious why the "-fno-stack-protector -m80387 -mfp-ret-in-387" was
required?

Ron, did you use any optimization or taken it out completely.  Also, are
you going to add this patch to your repository, or should we handle it on
our end?

  EBo --

Re: [9fans] crashing 9vx

[ron minnich]

On Sat, May 29, 2010 at 4:05 AM, EBo <ebo@sandien.com> wrote:

>  gcc -fno-inline -c -g -O3 -MD -std=gnu99 -O2 -march=i486 -pipe
>
> and
>
>  gcc -m32 -c -nostdinc --g -O3 -MD -std=gnu99 -O2 -march=i486 -pipe
> -fno-stack-protector -m80387 -mfp-ret-in-387

The stack protector code came in to some distros (e.g. ubuntu) and it
causes major trouble (for coreboot among other things) if you do
things slightly out of the standard. You need  -fno-stack-protector I
would assume with vx32.

> Ron, did you use any optimization or taken it out completely.  Also, are
> you going to add this patch to your repository, or should we handle it on
> our end?

it's now commited to my vx32 tree. Fast is good, but lack of blowups is better.

ron

Re: [9fans] crashing 9vx

[EBo]

> The stack protector code came in to some distros (e.g. ubuntu) and it
> causes major trouble (for coreboot among other things) if you do
> things slightly out of the standard. You need  -fno-stack-protector I
> would assume with vx32.

yes, it was for building vx32 as I recall.  Thanks for the pointer.

> it's now commited to my vx32 tree.

OK.  Over the weekend I will update Tvx and post to TCL.  I want to
resolve a couple of other minor issues first, but this can/should be
included too.

> Fast is good, but lack of blowups is better.

you are preaching to the choir ;-)

  EBo --

Re: [9fans] crashing 9vx

[erik quanstrom]

On Fri May 28 23:41:33 EDT 2010, rminnich@gmail.com wrote:
> OK, somebody sent a hint that it might make sense to take the -O3 out
> of the make flags. Done.

why were the flags set to -O3?

- erik

Re: [9fans] crashing 9vx

[ron minnich]

On Sat, May 29, 2010 at 4:09 AM, erik quanstrom <quanstro@quanstro.net> wrote:

> why were the flags set to -O3?

I have no idea, I did not set this up. Sometimes, for some of the
weirder tricks the GNU/Linux guys play, you have to have at least -O2
... inb/outb macros being one example.

ron

Re: [9fans] crashing 9vx

[Brian L. Stuart]

> OK, somebody sent a hint that it
> might make sense to take the -O3 out
> of the make flags. Done.
>
> Result: I can now get through this command:
> hget -v http://plan9.bell-labs.com/plan9/download/plan9.iso.bz2>/tmp/iso.bz2
> |[2]aux/statusbar plan9.iso
>
> without an explosion.

This is weird.  I just built 9vx on FreeBSD without
the -O3.  But instead of being more stable, that
one crashed on startup, like Charles reported.
Namely:

9vx panic: user fault: signo=11 addr=3850cb67 [useraddr=cb67] read=1 eip=80b973c esp=493ffac0
aborting, to dump core.

With the -O3, the crashes are rare, and seem to be
associated with heavy I/O.

BLS

Re: [9fans] crashing 9vx

[ron minnich]

Here are your debug options:
        case '1':
                singlethread = 1;
                break;
        case 'A':
                doabort++;
                break;
        case 'B':
                abortonfault++;
                break;
        case 'K':
                tracekdev++;
                break;
        case 'F':
                nofork = 1;
                break;
        case 'M':
                tracemmu++;
                break;
        case 'P':
                traceprocs++;
                break;
        case 'S':
                tracesyscalls++;
                break;
        case 'U':
                nuspace = atoi(EARGF(usage()));
                break;
        case 'X':
                vx32_debugxlate++;
                break;

On Sun, May 30, 2010 at 2:46 PM, Brian L. Stuart <blstuart@bellsouth.net> wrote:
> This is weird.  I just built 9vx on FreeBSD without
> the -O3.  But instead of being more stable, that
> one crashed on startup, like Charles reported.
> Namely:
>
> 9vx panic: user fault: signo=11 addr=3850cb67 [useraddr=cb67] read=1 eip=80b973c esp=493ffac0
> aborting, to dump core.

-X is super-slick. But you can probably see what can be done here.

I wonder if you could run -X with your immediate failure and put it on
pastebin.com or similar.

ron

Re: [9fans] crashing 9vx

[Philippe Anel]

You also have to recompile vx library.

Phil

Brian L. Stuart wrote:
>> OK, somebody sent a hint that it
>> might make sense to take the -O3 out
>> of the make flags. Done.
>>
>> Result: I can now get through this command:
>> hget -v http://plan9.bell-labs.com/plan9/download/plan9.iso.bz2>/tmp/iso.bz2
>> |[2]aux/statusbar plan9.iso
>>
>> without an explosion.
>>
>
> This is weird.  I just built 9vx on FreeBSD without
> the -O3.  But instead of being more stable, that
> one crashed on startup, like Charles reported.
> Namely:
>
> 9vx panic: user fault: signo=11 addr=3850cb67 [useraddr=cb67] read=1 eip=80b973c esp=493ffac0
> aborting, to dump core.
>
> With the -O3, the crashes are rare, and seem to be
> associated with heavy I/O.
>
> BLS
>
>
>
>

Re: [9fans] crashing 9vx

[erik quanstrom]

On Sun May 30 12:00:25 EDT 2010, xigh@bouyapop.org wrote:
> You also have to recompile vx library.
>
> Phil
>
> Brian L. Stuart wrote:
> >> OK, somebody sent a hint that it
> >> might make sense to take the -O3 out
> >> of the make flags. Done.
> >>
> >> Result: I can now get through this command:
> >> hget -v http://plan9.bell-labs.com/plan9/download/plan9.iso.bz2>/tmp/iso.bz2
> >> |[2]aux/statusbar plan9.iso
> >>
> >> without an explosion.
> >>
> >
> > This is weird.  I just built 9vx on FreeBSD without
> > the -O3.  But instead of being more stable, that
> > one crashed on startup, like Charles reported.
> > Namely:
> >
> > 9vx panic: user fault: signo=11 addr=3850cb67 [useraddr=cb67] read=1 eip=80b973c esp=493ffac0
> > aborting, to dump core.
> >
> > With the -O3, the crashes are rare, and seem to be
> > associated with heavy I/O.

you may be right, but it seems too easy to blame gcc.
a better fit for the facts so far would seem to me that
9vx' locking is broken.  the optimization may just put
more pressure on broken locking.

putting a couple of prints in the startup will also change
timing.  if you can eliminate or cause a crash by adding
or removing prints, then you can be sure that there is a
locking/timing problem in 9vx.  although that doesn't prove
that gcc is blameless, it would be a reasonable assumption.

- erik

Re: [9fans] crashing 9vx

[Brian L. Stuart]

> you may be right, but it seems too easy to blame gcc.
> a better fit for the facts so far would seem to me that
> 9vx' locking is broken.  the optimization may just
> put
> more pressure on broken locking.

I would certainly agree that the variability of the
crashes feels like a mutual exclusion problem.  The
wide variety of effects of changing optimization
seems to by trying really hard to tell us something.
Of course, after two days of house-hunting I could
probably convince myself that the phase of the moon
is involved.

BLS

Re: [9fans] crashing 9vx

[Bakul Shah]

On Sun, 30 May 2010 17:59:49 +0200 Philippe Anel <xigh@bouyapop.org>  wrote:
> You also have to recompile vx library.

Doesn't help.

> Brian L. Stuart wrote:
> > With the -O3, the crashes are rare, and seem to be
> > associated with heavy I/O.

When I run s9fes (Scheme 9 from Empty Space) tests, some of
them fail or 9vx crashes and AFAIK they don't do much I/O.
They all pass on Plan9.

Just for kicks I compiled 9vx with clang-2.7.  With -O3 it
comes up fine. The initial acme window doesn't disappear
(like it does with gcc -O3) but I couldn't compile anything.
Probably I made a mistake. Will try this later.

Without -O it comes up but the initial acme window
disappears. But compiles do work now. s9fes test "GC lists"
that used to fail in random ways with gcc compiled 9vx
finishes now without complaints but "Hyper Operations"
failed, and things went downhill from there.  Rerunning
yields the same result so at least this is consistent!

So it seems there are multiple problems.

Re: [9fans] crashing 9vx

[Brian L. Stuart]

> You also have to recompile vx
> library.

I'm pretty sure I did.  I did a gmake clean
followed by gmake 9vx/9vx in vx32/src.  I'm
pretty sure I saw the libraries being compiled
as the compile commands flew by on the screen.

BLS

Re: [9fans] crashing 9vx

[ron minnich]

On Mon, May 31, 2010 at 1:14 AM, Brian L. Stuart <blstuart@bellsouth.net> wrote:
>> You also have to recompile vx
>> library.
>
> I'm pretty sure I did.  I did a gmake clean
> followed by gmake 9vx/9vx in vx32/src.  I'm
> pretty sure I saw the libraries being compiled
> as the compile commands flew by on the screen.

then it rebuilt the library.

Something is wrong, for sure, and the hard part is getting a solid
reproducer(2), which is a necessary first step.

And, I agree, it sure seems it's more than one thing.

ron

Re: [9fans] crashing 9vx

[ron minnich]

So can we sum up the 9vx state?

Here's my summary:
tinycore, 2.6.32, gcc 4.4.3
1. pretty easy to blow up with an hget of the plan 9 iso on SMP, -O3
2. lotsafiles *sometimes* fails, other times runs with no trouble on SMP, -O3
3. rebuild without -O3, and it is quite solid

If we can get a reasonable list, we can try to get to a reproducer.
losafiles is not it I think. Also, if this discussion is annoying
anyone who is only concerned with Plan 9 :-) we might want to move to
a different list.

thanks

ron

Re: [9fans] crashing 9vx

[jake]

> So can we sum up the 9vx state?
>
> Here's my summary:
> tinycore, 2.6.32, gcc 4.4.3
> 1. pretty easy to blow up with an hget of the plan 9 iso on SMP, -O3
> 2. lotsafiles *sometimes* fails, other times runs with no trouble on SMP, -O3
> 3. rebuild without -O3, and it is quite solid
May I ask why you're using -O3? -O3 turns on a scary amount of optimizations,
could that be the problem?

Re: [9fans] crashing 9vx

[EBo]

> May I ask why you're using -O3? -O3 turns on a scary amount of
> optimizations,
> could that be the problem?

-O3 was what was in the original makefile.  we have been discussing
removing it and if we should use any optimization.

  EBo --

Re: [9fans] crashing 9vx

[Jorden M]

On Mon, May 31, 2010 at 12:32 PM, ron minnich <rminnich@gmail.com> wrote:
> So can we sum up the 9vx state?
>
> Here's my summary:
> tinycore, 2.6.32, gcc 4.4.3
> 1. pretty easy to blow up with an hget of the plan 9 iso on SMP, -O3
> 2. lotsafiles *sometimes* fails, other times runs with no trouble on SMP, -O3
> 3. rebuild without -O3, and it is quite solid

Maybe newer GCC versions are causing trouble? Would it be worth it to
try compiling with a GCC 3.x and seeing if the problem goes away?

>
> If we can get a reasonable list, we can try to get to a reproducer.
> losafiles is not it I think. Also, if this discussion is annoying
> anyone who is only concerned with Plan 9 :-) we might want to move to
> a different list.
>
> thanks
>
> ron
>
>

Re: [9fans] crashing 9vx

[Charles Forsyth]

>Maybe newer GCC versions are causing trouble?

the 9vx that doesn't work for me is one that was compiled
over a year ago and has run happily on several Ubuntu and other
Linux versions until Ubuntu 10.04LTS. although i realise that
thanks to shared libraries, compiler variations could still play
a part, i'm still more inclined to suspect either a system-level
change, a difference of system configuration (eg, for shared libraries),
or a change to one of the supporting libraries, and there
are a few to choose from:
	linux-gate.so.1 =>  (0xb788d000)
	libX11.so.6 => /usr/lib/libX11.so.6 (0xb7785000)
	libpthread.so.0 => /lib/tls/i686/cmov/libpthread.so.0 (0xb776c000)
	libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb7608000)
	libxcb.so.1 => /usr/lib/libxcb.so.1 (0xb75ee000)
	libdl.so.2 => /lib/tls/i686/cmov/libdl.so.2 (0xb75ea000)
	/lib/ld-linux.so.2 (0xb788e000)
	libXau.so.6 => /usr/lib/libXau.so.6 (0xb75e6000)
	libXdmcp.so.6 => /usr/lib/libXdmcp.so.6 (0xb75e1000)

9vx also makes inventive use of mmap and sigsegv.

Re: [9fans] crashing 9vx

[ron minnich]

I built 9vx without a GUI.
  549  make PLAN9GUI=nogui 9vx/9vx


% hg update
process 49 sleeps with 1 locks held, last lock 0x735400 locked at pc
0x7f8ee4ef8e98, sleep called from 0x40b6b6
9vx panic: sigsegv on cpu3
aborting, to dump core.
Aborted

this is a bit more info than I was getting.

Just FYI.

ron

Re: [9fans] crashing 9vx

[EBo]

> never attribute to funny hardware that
> which can be adequately explained by
> broken locking.
>
> - erik

That's a nice quotable quote ;-)

[9fans] crashing 9vx

[yy]

I keep hearing how 9vx is very unstable and have a lot of problems,
but after a few months working on it I have not received many bug
reports. Probably I won't know how to fix these problems, but at least
I'd like to have them documented. Also, I'm updating the autogenerated
kernel files and would like to check if this makes any difference.

So, if you are using the 9vx version at
http://bitbucket.org/yiyus/vx32/ (or ron's version, which is almost
the same) and you have a reproducible way to crash it, could you
please fill an issue in bitbucket or send me an email? Thanks.

--
- yiyus || JGL . 4l77.com

Re: [9fans] crashing 9vx

[Lucio De Re]

On Thu, Oct 28, 2010 at 04:00:32PM +0200, yy wrote:
>
> So, if you are using the 9vx version at
> http://bitbucket.org/yiyus/vx32/ (or ron's version, which is almost
> the same) and you have a reproducible way to crash it, could you
> please fill an issue in bitbucket or send me an email? Thanks.
>
I only use 9vx as proof of concept and I have only two reportable issues:
I can't get "cpu" to work where "drawterm" seems to succeed and on one
occasion I used "DEL" to terminate a task and the whole of 9vx shut down.
The latter is a little scary, but I have been very reluctant to test
it :-)

The "cpu" doesn't succeed seems to mean that I can't persuade it to use
the target host as auth server.  I could just be showing my ignorance
and/or stupidity.

++L