[9fans] fun with rfork

[9fans] fun with rfork

[C H Forsyth]

here is part of an application of rfork(RFMEM|RFPROC ...).
most irrelevant code has been removed.
libinit was called early on before the process switched to
a stack on the malloc heap, and that's where it is running when it
calls oscmd.

what happens next?

	--------

void
libinit(char *imod)
{
	...
	int fd, n, pid;

	...
	/*
	 * guess at a safe stack for vstack
	 */
	ustack = (ulong)&fd;
	...
	/* switch to stack on the heap ... */
	tramp(sp+KSTACK, up->func, up->arg);
}

typedef struct Targ Targ;
struct Targ
{
	int	fd;
	int*	spin;
	char*	cmd;
};

void
exectramp(Targ *targ)
{
	*targ->spin = 0;
	...

	exec(argv[0], argv);
	exits("");
}

int
oscmd(char *cmd, int *rfd, int *sfd)
{
	Targ targ;
	int spin, *spinptr, fd[2];

	if(pipe(fd) < 0)
		return -1;

	spinptr = &spin;
	spin = 1;

	targ.fd = fd[0];
	targ.cmd = cmd;
	targ.spin = spinptr;

	switch(rfork(RFMEM|RFPROC|RFFDG|RFENVG|RFREND)) {
	case -1:
		return -1;
	case 0:
		vstack(&targ);			/* Never returns */
	default:
		while(*spinptr)
			;
		break;
	}
	close(fd[0]);

	*rfd = fd[1];
	*sfd = fd[1];
	return 0;
}

TEXT	vstack(SB),$0
	MOVL	arg+0(FP), AX
	MOVL	ustack(SB), SP
	PUSHL	AX
	CALL	exectramp(SB)
	POPL	AX			/* dammit ken! */
	RET

Re: [9fans] fun with rfork

[Bruce Ellis]

the spinptr code was replaced quite a while ago because it
just doesn't work.  a rendezvous replaced it.

Re: [9fans] fun with rfork

[Charles Forsyth]

[-- Attachment #1: Type: text/plain, Size: 141 bytes --]

even more than that!
i ought to have said that it's perhaps most interesting on
an x86.  (Hint: how and where could you call rendezvous?)

[-- Attachment #2: Type: message/rfc822, Size: 2383 bytes --]

From: "Bruce Ellis" <brucee@chunder.com>
To: <9fans@cse.psu.edu>
Subject: Re: [9fans] fun with rfork
Date: Sat, 1 Nov 2003 10:35:53 +1100
Message-ID: <012b01c3a007$e44099c0$8201a8c0@cc77109e>

the spinptr code was replaced quite a while ago because it
just doesn't work.  a rendezvous replaced it.

Re: [9fans] fun with rfork

[Bruce Ellis]

looks a bit like this ...

 targ.fd = fd[0];
 targ.cmd = cmd;
 targ.tag = &tag;

 switch(rfork(RFFDG|RFPROC)) {
 case -1:
  return -1;
 case 0:
  vstack(&targ);   /* Never returns */
 default:
  rendezvous(&tag, 0);
  break;
 }

----- Original Message -----
From: "Charles Forsyth" <forsyth@caldo.demon.co.uk>
To: <9fans@cse.psu.edu>
Sent: Saturday, November 01, 2003 10:03 AM
Subject: Re: [9fans] fun with rfork


> even more than that!
> i ought to have said that it's perhaps most interesting on
> an x86.  (Hint: how and where could you call rendezvous?)
>

Re: [9fans] fun with rfork

[Charles Forsyth]

[-- Attachment #1: Type: text/plain, Size: 3928 bytes --]

yes i see now why you said the spinptr code didn't work, but
the code i posted was a little different.

the spinptr code really did work originally: the code was a little different
from what i posted earlier--it had
	switch(fork()){
which became
	switch(rfork(RFFDG|RFPROC)) {
in your version

BUT in Plan 9 when the code was originally written, an rfork with RFMEM
caused RFMEM to be implied(!) in all subsequent rforks (and thus forks),
so the original spinptr worked because although it said `fork', it was still
sharing the address space.  (i first noticed the feature when i
had a little trouble with the 8-1/2 event library.)

when Plan 9's implementation (fortunately) later changed
to eliminate that unpleasant effect, the fork made the memory unshared,
spinptr stopped working and a rendezvous would indeed
be needed.  on the other hand, the vstack probably isn't needed
then because the two processes are not sharing the stack.


my rfork call in the example was
	rfork(RFMEM|RFPROC|RFFDG|RFENVG|RFREND)
which kept the RFMEM (explicitly), and that does keep the spinptr working.
not that it's an efficient method, but there's a reason it's hard to
use rendezvous straightforwardly in that particular context.

it was really some of the peculiar effects of RFMEM and consequent shared (malloc'd)
stack in this particular case that interested me, as an example of how plausible
code and helpful comments can be completely misleading.

the idea behind spinptr/vstack is that the two processes are sharing
the malloc'd stack after the rfork so the child calls assembly-language vstack to shift
its stack to its private, unshared memory (rfork even with RFMEM doesn't
share the stack segment).  until the child does that, the parent had
better not budge because that would change the stack on which the child is
still executing, corrupting its data.  once the child has switched to its unshared stack,
by setting SP to the value of ustack, which points into the unshared stack segment, it can
let the parent continue.

it works, but for odd reasons, and it's critical how the code is arranged.

one key point is that the comment next to vstack, is wrong (on an x86).
the set of possible execution traces is rather bizarre, and depends who
runs first after the rfork.   the child needs to call something in assembly
language to change the stack pointer.  they both return from rfork.

if the parent returns first, it blocks on the
	while(*spinptr)
		;
the child process runs, calls vstack, changes its stack pointer, calls exectramp
on that new stack, writes zero to the spin value, and allows the parent to proceed.

if the child returns from rfork first, it calls vstack, changes its stack pointer, etc.
the parent returns from rfork, finds *spinptr zero and simply continues on its way.

that's the way the code is written but it overlooks the stack sharing: the child must call
vstack (it can't change the stack pointer otherwise), so its return address will
be at the default: following the call to vstack.  but its stack is the parent's too,
so its vstack return address replaces the parent's rfork return address, and
when it runs it doesn't execute the switch but
ends up on the other side of the vstack call, and falls through into the default: case.
of course on a multiprocessor the two paths can run
concurrently,  or if a time slice hits on a uniprocessor,
they can be interleaved.

	switch(rfork(RFPROC|RFMEM|RFREND|RFNOTEG|RFFDG|RFNAMEG|RFENVG)) {
	case -1:
		goto Error;
	case 0:
		/* if child returns first from rfork, its call to vstack replaces ... */
		vstack(t);
		/* ... parent's return address from rfork and parent returns here */
	default:
		/* if parent returns first from rfork, it comes here */
		/* can't call anything: on shared stack until child releases spin in exectramp */
		while(*spinptr)
			;
		break;
	}

[-- Attachment #2: Type: message/rfc822, Size: 2867 bytes --]

From: "Bruce Ellis" <brucee@chunder.com>
To: <9fans@cse.psu.edu>
Subject: Re: [9fans] fun with rfork
Date: Sat, 1 Nov 2003 22:40:44 +1100
Message-ID: <007701c3a06d$158bd1b0$8201a8c0@cc77109e>

looks a bit like this ...

 targ.fd = fd[0];
 targ.cmd = cmd;
 targ.tag = &tag;

 switch(rfork(RFFDG|RFPROC)) {
 case -1:
  return -1;
 case 0:
  vstack(&targ);   /* Never returns */
 default:
  rendezvous(&tag, 0);
  break;
 }

----- Original Message -----
From: "Charles Forsyth" <forsyth@caldo.demon.co.uk>
To: <9fans@cse.psu.edu>
Sent: Saturday, November 01, 2003 10:03 AM
Subject: Re: [9fans] fun with rfork


> even more than that!
> i ought to have said that it's perhaps most interesting on
> an x86.  (Hint: how and where could you call rendezvous?)
>