[9fans] disabling swap

[9fans] disabling swap

[Steve Simon]

I have just hit a fossil deadlock. the symptom is simple enough,
fossil wedged, stats continued to be updated and I could run whatis
in a rio window but any attempt to access the local fossil caused
the command to hang.

I was trying to mirror sources when it happened so I had a lot of
processes all writing to fossil at the same time - well, perhaps a dozen.

If anyone is interested I have put my 9pccpuf and a phone camera
image of ^t^t^p of the hang in http://www.quintile.net/doorstep/deadlock.jpg
and http://www.quintile.net/doorstep/9pccpuf.

If anyone extracts info from this I would like to learn how they do it.

I made a feeble attempt to diagnose the problem myself (below) and it looks
as though it was trying to swap, however, though I do have a /dev/sdXX/swap
I believe I have disabled the one place where swap is enabled (/cfg/$sysname/cpurc).

having said this I think /dev/mem is indicating that I _do_ have swap enabled. either
way I should not have been swapping as I have masses of free RAM.

-Steve (confused)


cpu% acid /386/9pccpuf
/386/9pccpuf:386 plan 9 boot image
/sys/lib/acid/port
/sys/lib/acid/386
acid: src(0xf01d3968)
/sys/src/9/port/fault.c:231
 226		k = kmap(new);
 227		kaddr = (char*)VA(k);
 228
 229		if(loadrec == 0) {			/* This is demand load */
 230			c = s->image->c;
>231			while(waserror()) {
 232				if(strcmp(up->errstr, Eintr) == 0)
 233					continue;
 234				kunmap(k);
 235				putpage(new);
 236				faulterror("sys: demand load I/O error", c, 0);
acid: src(0xf01d730f)
/sys/src/9/port/proc.c:576
 571		/* Only reliable way to see if we are Running */
 572		if(p->mach == 0) {
 573			p->newtlb = 1;
 574			ok = 1;
 575		}
>576		unlock(runq);
 577		spllo();
 578
 579		return ok;
 580	}
 581
acid:
cpu%

cpu% cat /dev/swap
2134908928 memory
4096 pagesize
27037 kernel
112466/494181 user
0/160000 swap
4588640/71348716 kernel malloc
0/16777216 kernel draw

cpu% grep swap /cfg/$sysname/cpurc
# swap is broken
# swap `{ls /dev/fs/swap /dev/sd*/swap >[2] /dev/null | sed 1q}

Re: [9fans] disabling swap

[erik quanstrom]

On Mon Sep 14 20:09:25 EDT 2009, steve@quintile.net wrote:
> I have just hit a fossil deadlock. the symptom is simple enough,
> fossil wedged, stats continued to be updated and I could run whatis
> in a rio window but any attempt to access the local fossil caused
> the command to hang.
[..]
> If anyone is interested I have put my 9pccpuf and a phone camera
> image of ^t^t^p of the hang in http://www.quintile.net/doorstep/deadlock.jpg
> and http://www.quintile.net/doorstep/9pccpuf.

wasn't there a recent report of fossil hanging when sync was
done on the console with lots of io on the fs?

i could be missing something, but i don't see evidence that
swap (more accurately, demand paging) has anything to do
with the problem.  qpc and lpc just point to the pc of the
last blocking qlock or splhi.  that could be a very long way
(and time) from the current pc.

the fossil procs are in rendez, which could be any number
of vtLocks.

have you applied cinap's patch to periodic.c?
i would be very interested if that solves the problem.
(http://9fans.net/narchive/2009/03/487, /n/sources/patch/fossil-sleep-stress)

he wasn't seen a lockup, but i could see how a number
of spinning periodic threads could give the appearance
of making no progress since they would spin in a lock
acquiring a lock that a thread with something to do
may need.

- erik