[9fans] kfs locks

[9fans] kfs locks

[David Tolpin]

Hi,

cmd_sync sets 

rlock(&mainlock); 

...
syncall();
...
runlock(&mainlock);

...
syncproc, the background process forked by kfs, runs 
syncblock()
without setting rlock(&mainlock)

Can it the cause that I am getting wrenwrite errors after
'check f' on startup, if cmd_sync is not called after cmd_check
and before forking 'sync' and 'serve'?

I am still struggling to understand the problem. I've changed
the disk to be sure and still can reproduce the error if cmd_sync
is not called after the list of free blocks is rebuilt after blackdown.

David Tolpin

Re: [9fans] kfs locks

[Charles Forsyth]

[-- Attachment #1: Type: text/plain, Size: 497 bytes --]

the intent is that the sync command ensures that all current IO has gone
to disc without allowing file system activity
meanwhile--that's what the rlock of mainlock does--
but syncproc is allowed to run
concurrently with file system activity
to encourage the disc to be reasonably up to date
but without ensuring it.
there are interlocks on the buffers themselves (for instance)
to prevent confusion between several processes active
at once, whether for file system activity or syncproc.

[-- Attachment #2: Type: message/rfc822, Size: 2490 bytes --]

From: David Tolpin <dvd@davidashen.net>
To: 9fans@cse.psu.edu
Subject: [9fans] kfs locks
Date: Sun, 7 Mar 2004 21:55:37 +0400 (AMT)
Message-ID: <200403071755.i27HtbcR080605@adat.davidashen.net>

Hi,

cmd_sync sets 

rlock(&mainlock); 

...
syncall();
...
runlock(&mainlock);

...
syncproc, the background process forked by kfs, runs 
syncblock()
without setting rlock(&mainlock)

Can it the cause that I am getting wrenwrite errors after
'check f' on startup, if cmd_sync is not called after cmd_check
and before forking 'sync' and 'serve'?

I am still struggling to understand the problem. I've changed
the disk to be sure and still can reproduce the error if cmd_sync
is not called after the list of free blocks is rebuilt after blackdown.

David Tolpin

Re: [9fans] kfs locks

[David Tolpin]

> there are interlocks on the buffers themselves (for instance)
> to prevent confusion between several processes active
> at once, whether for file system activity or syncproc.
>

But still,

If I sync immediately after rebuilding the freelist, the
kfs works fine. If I let syncproc do that, wrenwrite is passed
offsets not present in the file system eventually.

After a lot of investigation, I can only image a possible
cause in insufficient interlocking, but fail to understand
how this happens. 

This is not a hardware problem, I am able to reproduce it
with a different fast enough computer.

David

Re: [9fans] kfs locks

[Charles Forsyth]

in a previous message, with various debugging prints turned on
and inserted, there seemed to be driver requests to abort a command.
you don't seem to see the problem unless you power cycle,
and although i don't use it as much as i did, i still use kfs a
fair bit (and i do sometimes just switch off that machine),
and i don't see similar problems.
now, it's quite likely that i'm running an older version of kfs,
and there were changes in chk in november (for instance), although
they look benign.  i thought i'd like to know more about the
actual cause of wrenwrite errors before trying to guess what and
where the problem might be!

Re: [9fans] kfs locks

[David Tolpin]

> in a previous message, with various debugging prints turned on
> and inserted, there seemed to be driver requests to abort a command.

write is being aborted, because it is asked to write at offset 
not present on the disk. It looks like the in-memory structure 
is corrupt.

> you don't seem to see the problem unless you power cycle,
> and although i don't use it as much as i did, i still use kfs a
> fair bit (and i do sometimes just switch off that machine),
> and i don't see similar problems.
> now, it's quite likely that i'm running an older version of kfs,

It is quite likely that your machine is slower. If I slow down
the kfs or kernel code by inserting more debugging, the problem
disappears because the structure is flushed before being overwritten.

> actual cause of wrenwrite errors before trying to guess what and
> where the problem might be!

offsets are negative vlongs. 

Re: [9fans] kfs locks

[jmk]

On Sun Mar  7 15:22:37 EST 2004, dvd@davidashen.net wrote:
> > in a previous message, with various debugging prints turned on
> > and inserted, there seemed to be driver requests to abort a command.
> 
> write is being aborted, because it is asked to write at offset 
> not present on the disk. It looks like the in-memory structure 
> is corrupt.
> 
> > you don't seem to see the problem unless you power cycle,
> > and although i don't use it as much as i did, i still use kfs a
> > fair bit (and i do sometimes just switch off that machine),
> > and i don't see similar problems.
> > now, it's quite likely that i'm running an older version of kfs,
> 
> It is quite likely that your machine is slower. If I slow down
> the kfs or kernel code by inserting more debugging, the problem
> disappears because the structure is flushed before being overwritten.
> 
> > actual cause of wrenwrite errors before trying to guess what and
> > where the problem might be!
> 
> offsets are negative vlongs. 

The offset is supposed to be checked way before getting to the
ATA driver:
1) the write and seek system calls check the offset is not negative;
2) the higher level of the disc driver checks the address requested
   is within the partition.

What does your partition table look like?

Re: [9fans] kfs locks

[David Tolpin]

> The offset is supposed to be checked way before getting to the
> ATA driver:

Yes, I see. 
Not all offsets are negative, some. I've added debugging print (("%lld",...)) 
to wrenwrite, not to the ATA driver. I am getting offsets beyond the range of the
filesystem, sometimes negative.

> What does your partition table look like?

geometry 29336832 512 16383 16 63
part data 0 29336832
part plan9 63 8225280
part 9fat 63 102463
part fs 102463 7442495
part nvram 7442495 7442496
part swap 7442496 8225280
part plan9.1 8225280 16450560
part swap.cpu 8225343 10322495