[9fans] fossil 'sync' lockup?

[9fans] fossil 'sync' lockup?

[Venkatesh Srinivas]

Hi,

I executed the fossil 'sync' command when my terminal was under
moderate I/O load earlier today. Fossil seems to have locked up. Has
anyone seen anything like this?

I have a dump of the first 256mb of kernel memory and all physical
memory. Are there any structures that are kept at reasonably constant
places in memory, so that I can start trawling them to see what fossil
was doing when it died?

Thanks,
-- vs

Re: [9fans] fossil 'sync' lockup?

[erik quanstrom]

> I executed the fossil 'sync' command when my terminal was under
> moderate I/O load earlier today. Fossil seems to have locked up. Has
> anyone seen anything like this?
>
> I have a dump of the first 256mb of kernel memory and all physical
> memory. Are there any structures that are kept at reasonably constant
> places in memory, so that I can start trawling them to see what fossil
> was doing when it died?

how did you generate this dump?  i've never done that.

the short answer is that any non-automatic variables have
fixed address.  for example, the process table is here
port/proc.c:/^procalloc in .arena

but given such an image, i can't think of a convienent way of
doing this with existing tools.

supposing that i haven't missed an easy way to do this,
one strategy for getting where you want to be is to write
a small program that can convert the raw dump into
snap(6) format.  then snapfs can be used to debug as
usual.

unfortunately one would first need to teach acid where
things are in your dump so you can find the process table,
and figure out where where the fossil procs are and
find the segments which belong to them.  the acid kernel
library should be of some help.

i hope i'm missing something.

- erik

Re: [9fans] fossil 'sync' lockup?

[Venkatesh Srinivas]

On Sun, Sep 6, 2009 at 11:54 AM, erik quanstrom<quanstro@quanstro.net> wrote:
>> I executed the fossil 'sync' command when my terminal was under
>> moderate I/O load earlier today. Fossil seems to have locked up. Has
>> anyone seen anything like this?

The locking I can see along the path of 'sync' is vtlock (exclusive
lock) of the Epoch lock in fsSync, the per-file locks in
fileMetaFlush, and the cache's dirty lock and main lock in cacheFlush.
Luckily VtLocks know their holder...

Why are VtLocks in liboventi? And why does liboventi have an
implementation of a subset of the thread library (rwlocks and
rendezvous?)

>> I have a dump of the first 256mb of kernel memory and all physical
>> memory. Are there any structures that are kept at reasonably constant
>> places in memory, so that I can start trawling them to see what fossil
>> was doing when it died?
>
> how did you generate this dump?  i've never done that.

I was running my terminal in qemu. Switch to the monitor, pmemsave and memsave.

> the short answer is that any non-automatic variables have
> fixed address.  for example, the process table is here
> port/proc.c:/^procalloc in .arena

Okay, thanks! Luckily nm worked with 9pcf and I found procalloc; I was
going to start at the mach structure (always at KZERO + 0x15000) ,
find the current process structure's address, and then trawl memory
for processes around it till I found fossil.

> i hope i'm missing something.

I hope so too. grep and seeking to the right offsets in a dump was not
my least favorite thing ever, but its close.

Thanks,
-- vs

Re: [9fans] fossil 'sync' lockup?

[erik quanstrom]

> Why are VtLocks in liboventi? And why does liboventi have an
> implementation of a subset of the thread library (rwlocks and
> rendezvous?)

fossil and venti do seem to break the rules a bit.  they
don't pass style(6) muster. i  can only guess this is for
portability.  but as p9p already provides a full thread
library, so at the least, i don't see a continued need
for them.

perhaps russ knows more?

> Okay, thanks! Luckily nm worked with 9pcf and I found procalloc; I was
> going to start at the mach structure (always at KZERO + 0x15000) ,
> find the current process structure's address, and then trawl memory
> for processes around it till I found fossil.

it might pay off to teach acid about this.  i think by
setting the map or creatively fabricating a /proc/n
entry.  other than mem and segment, i believe you
can fake the other files with /dev/zero or /dev/null.

- erik