Re: [9fans] 9pcfs mballoc bug info

Re: [9fans] 9pcfs mballoc bug info

[jmk]

For a long time we've had trouble rebooting our fileserver when the network is
busy; 'busy' seems to mean lots of machines try to attach at once. There is a
panic, sometimes it's the mbfree panic, but mostly just a page fault due to a
nil pointer somewhere.

When it happened again last week I looked at some of the faults and through a
tortuous path including a number of leaps of faith started looking at the
locking in getchan and il.c in general, it just looks bogus but I haven't gotten
back to it to convince myself or not.

--jim

Re: [9fans] 9pcfs mballoc bug info

[nemo]

[-- Attachment #1: Type: text/plain, Size: 427 bytes --]

When I changed the code a bit to say in the 905 txstart:

if (!FREE)
	mbfree();

that was exactly what happen, the second caller of mbfree()
(the second path through the 905 driver) got a nil pointer as
the message buffer in the ring, and paniced, a page fault
followed. Not a surprise, though.

The ring descriptors were different, but the message buffer
was the same. Only need to find out who does that...




[-- Attachment #2: Type: message/rfc822, Size: 1865 bytes --]

From: jmk@plan9.bell-labs.com
To: 9fans@cse.psu.edu
Subject: Re: [9fans] 9pcfs mballoc bug info
Date: Mon, 5 Feb 2001 13:43:02 -0500
Message-ID: <20010205184310.14DD1199F4@mail.cse.psu.edu>

For a long time we've had trouble rebooting our fileserver when the network is
busy; 'busy' seems to mean lots of machines try to attach at once. There is a
panic, sometimes it's the mbfree panic, but mostly just a page fault due to a
nil pointer somewhere.

When it happened again last week I looked at some of the faults and through a
tortuous path including a number of leaps of faith started looking at the
locking in getchan and il.c in general, it just looks bogus but I haven't gotten
back to it to convince myself or not.

--jim

Re: [9fans] 9pcfs mballoc bug info

[nemo]

[-- Attachment #1: Type: text/plain, Size: 864 bytes --]

The bug happens only when the net is *sloooow* to send the
first packet (seems to happen w/ the arp requests being
sent). I can reproduce the bug quite easily by starting
a fs kernel and attaching to it from a different building
in our campus (yes, our network is slow :-( ).

I think that what happens is that a message buffer gets
linked twice in the transmit ring, perhaps due to a timeout
and retransmission.

Nevertheless, I had to get a fs up quickly and stopped
debugging by now to install one (within the same building,
hence working ;-) ).

My current problem is that although the box has two 4G IDE
disks, the fs kernel thinks the file system is full even
before unpacking the plan9.9gz package.

In case somebody has a hint about what I am doing wrong,
my config string for the device was ch0fh2.

Perhaps I'm just too sleepy...


[-- Attachment #2: Type: message/rfc822, Size: 2358 bytes --]

From: "Boyd Roberts" <boyd@planete.net>
To: <9fans@cse.psu.edu>
Subject: Re: [9fans] 9pcfs mballoc bug info
Date: Mon, 5 Feb 2001 18:56:54 +0100
Message-ID: <037001c08f9d$068b3e00$0ab9c6d4@cybercable.fr>

> Didn't fix it yet, but in my case, txstart905() (plan9pc/etherlnk3.c)
> calls mbfree()  twice with the same buffer. Only transmit() and interrupt()
> (same file) call txstart905; and both routines hold the ilock on ctlr->wlock.

these dup frees are rarely easy to track down.  i'd add some code
before the free to detect it and print some debug, before the panic
leaves you with just a hex stack backtrace.  unfortunately this
may change the timing and the problem may go away.

Re: [9fans] 9pcfs mballoc bug info

[Francisco J Ballesteros]

jmk@plan9.bell-labs.com wrote:
>
> are we looking at the same driver? in the one i have there is
> one call to mbfree in the 905 section:
>
>                 if(pd->mb != nil){
>                         mbfree(pd->mb);
>                         pd->mb = nil;
>                 }

Yes, and that call to mbfree() was the one made twice.
My first silly try was to put the mbfree inside an if,
to do it only if the mb was not already FREE.
The second caller of mbfree for the same pd->mb would then
file mb to be nil, and panic.

There are two different pd's with
the same pd->mb, and both are calling mbfree. My silly try
was trying to ensure that was the case.

Re: [9fans] 9pcfs mballoc bug info

[jmk]

nemo@gsyc.escet.urjc.es:
> When I changed the code a bit to say in the 905 txstart:
>
> if (!FREE)
> 	mbfree();
>
> that was exactly what happen, the second caller of mbfree()
> (the second path through the 905 driver) got a nil pointer as
> the message buffer in the ring, and paniced, a page fault
> followed. Not a surprise, though.
>
> The ring descriptors were different, but the message buffer
> was the same. Only need to find out who does that...


are we looking at the same driver? in the one i have there is
one call to mbfree in the 905 section:

		if(pd->mb != nil){
			mbfree(pd->mb);
			pd->mb = nil;
		}

[9fans] 9pcfs mballoc bug info

[nemo]

Hi,

	Didn't fix it yet, but in my case, txstart905() (plan9pc/etherlnk3.c)
calls mbfree()  twice with the same buffer. Only transmit() and interrupt()
(same file) call txstart905; and both routines hold the ilock on ctlr->wlock.

Looking at the code that should not happen, but it happens.

If I find out what's going on, I'll let you know.

Re: [9fans] 9pcfs mballoc bug info

[Boyd Roberts]

> Didn't fix it yet, but in my case, txstart905() (plan9pc/etherlnk3.c)
> calls mbfree()  twice with the same buffer. Only transmit() and interrupt()
> (same file) call txstart905; and both routines hold the ilock on ctlr->wlock.

these dup frees are rarely easy to track down.  i'd add some code
before the free to detect it and print some debug, before the panic
leaves you with just a hex stack backtrace.  unfortunately this
may change the timing and the problem may go away.