[9fans] trying to understand how fork/pipe a filtering program

[9fans] trying to understand how fork/pipe a filtering program

[Axel Belinfante]

after the suggestion of resample(1) some days ago
I've tried run it as a filter from another program.
this other program is non-threaded; for testing all
it does is call initdraw and read an image file
after which the resample routine is called.

in my resample routine it seems I need a call to sleep
to avoid the resample program to hang (in Pread).
(replacing the sleep by a fprint(2, "something\n") works too).

I'm just trying to understand what goes on -
is this the way to do this? if not, what am I missing?
I'm sorry if it is something obvious; I'm just trying to learn.

Axel.

Image*
resample(Image *from)
{
	int p[2];
	Image *img;

	if(pipe(p) < 0){
		fprint(2, "pipe fails: %r\n");
		exits("Epipe");
	}
	switch(fork()){
	case -1:
		fprint(2, "fork fails: %r\n");
		exits("Efork");
	default:
		if (writeimage(p[0], from, 0) < 0)
			fprint(2, "error writeimage\n");
		close(p[0]);
		/* seems we need sleep to give up cpu so resample can run */
		sleep(1);
		img = readimage(display, p[1], 0);
		close(p[1]);
		waitpid();
		return img;
	case 0:
		dup(p[1], 0);
		dup(p[0], 1);
		execl("/bin/resample", "resample", "-x50%", nil);
		fprint(2, "execl fails: %r\n");
		exits("Eexec");
	}
	return nil;
}

Re: [9fans] trying to understand how fork/pipe a filtering program

[geoff]

I think you want something more like this:

	int p[2], kid, pid;
	Image *img;

	if(pipe(p) < 0)
		sysfatal("can't make a pipe: %r");
	kid = fork();
	switch(kid){
	case -1:
		sysfatal("can't fork: %r");
	default:
		if (writeimage(p[0], from, 0) < 0)
			fprint(2, "error writeimage\n");
		close(p[0]);
		img = readimage(display, p[1], 0);
		close(p[1]);
		/*
		 * we may have unknown children forked by a previous
		 * program (e.g., rc) that then execed us.
		 */
		while ((pid = waitpid()) != kid && pid != -1)
			continue;
		return img;
	case 0:
		dup(p[1], 0);
		dup(p[0], 1);
		close(p[0]);
		close(p[1]);
		execl("/bin/resample", "resample", "-x50%", (char *)nil);
		sysfatal("can't exec resample: %r");
	}

Re: [9fans] trying to understand how fork/pipe a filtering program

["Nils O. Selåsdal"]

Axel Belinfante wrote:
> after the suggestion of resample(1) some days ago
> I've tried run it as a filter from another program.
> this other program is non-threaded; for testing all
> it does is call initdraw and read an image file
> after which the resample routine is called.
> 
> in my resample routine it seems I need a call to sleep
> to avoid the resample program to hang (in Pread).
> (replacing the sleep by a fprint(2, "something\n") works too).
> 
> I'm just trying to understand what goes on -
> is this the way to do this? if not, what am I missing?
> I'm sorry if it is something obvious; I'm just trying to learn.

Remember to close the pipe ends you don't need in the respective
parent/child, as well as the original pipes after dup'ing them.

Re: [9fans] trying to understand how fork/pipe a filtering program

[Axel Belinfante]

suggestions by geoff and noselasd helped to
improve my code but did not solve the problem.

the problem seems to be (related to?) that I'm
trying to use one pipe in a bidirectional way.
using two pipes unidirectionally does make it work.

actually, the whole thing gives me some feeling of
deja-vu of the thing I brought up here about a year
ago in thread 'thread confusion', which was then
explained by russ in http://9fans.net/archive/2005/09/270

Axel.

Re: [9fans] trying to understand how fork/pipe a filtering program

[rog]

i'd have thought that something like the following should work:

	int p[2], q[2], kid, pid;
	Image *img;

	if(pipe(p) < 0 || pipe(q) < 0)
		sysfatal("can't make a pipe: %r");
	kid = fork();
	switch(kid){
	case -1:
		sysfatal("can't fork: %r");
	default:
		close(p[0]);
		close(q[1]);
		if (writeimage(p[1], from, 0) < 0)
			fprint(2, "error writeimage\n");
		close(p[1]);
		img = readimage(display, q[0], 0);
		close(q[0]);
		/*
		 * we may have unknown children forked by a previous
		 * program (e.g., rc) that then execed us.
		 */
		while ((pid = waitpid()) != kid && pid != -1)
			continue;
		return img;
	case 0:
		dup(p[0], 0);
		dup(q[1], 1);
		close(p[1]);
		close(q[0]);
		execl("/bin/resample", "resample", "-x50%", (char *)nil);
		sysfatal("can't exec resample: %r");
	}

Re: [9fans] trying to understand how fork/pipe a filtering program

[Axel Belinfante]

> i'd have thought that something like the following should work:

correct, it does work.

I tried to say that in my last message.
I'm sorry if it only added to the confusion.
the main thing is that what you suggest indeed does work.

Axel.

Re: [9fans] trying to understand how fork/pipe a filtering program

[rog]

oops skim reading again! i'd thought i'd read "using two pipes unidirectionally *doesn't* make it work."
so why can't you just use two pipes unidirectionally? you can't do anything
else in this example, because by using the same pipe for input and output,
the command itself keeps its own input open, and therefore never receives EOF.

Re: [9fans] trying to understand how fork/pipe a filtering program

[Axel Belinfante]

> oops skim reading again!
> i'd thought i'd read "using two pipes unidirectionally *doesn't* make it work."
> so why can't you just use two pipes unidirectionally? you can't do anything
> else in this example, because by using the same pipe for input and output,
> the command itself keeps its own input open, and therefore never receives EOF.

seems it took me a while to realize that.


the funny thing is that, somehow, using the same pipe
for input and output does work (at least for me)
if in the parent I have an fprint or a sleep between
the writing+closing part and the reading part,
as in the message with which I started this thread.
that said, I'm much happier with the 'two unidirectional pipes' solution.

Axel.

Re: [9fans] trying to understand how fork/pipe a filtering program

[geoff]

Maybe it's been too long a week, but I could have sworn that a Plan 9
pipe was a pair of unidirectional channels (queues in Streams
terminology), so closing one end would cause a zero-byte read at the
other, independently in each direction.  pipe(3) seems to support
this.

Re: [9fans] trying to understand how fork/pipe a filtering program

[rog]

> Maybe it's been too long a week, but I could have sworn that a Plan 9
> pipe was a pair of unidirectional channels (queues in Streams
> terminology), so closing one end would cause a zero-byte read at the
> other, independently in each direction.  pipe(3) seems to support
> this.

the problem is that neither end was actually being closed,
as both ends of the pipe were dup'd onto stdin/stdout of the
child process. (the underlying close only happens when all references
have gone).

there is an alternative, i realised last night - you can do a zero-length
write after doing the writeimage, which will cause the child to read EOF,
which is usually fine (unless the child re-reads EOF, expecting it to remain EOF,
which can be the case with some buffered IO libraries and one-char lookahead;
depends whether it implements ungetc(EOF) - bio has this issue, for example)

Re: [9fans] trying to understand how fork/pipe a filtering program

[rog]

> I'm just trying to understand what goes on -
> is this the way to do this? if not, what am I missing?

ahh, having thought about the original question, i finally understand what was going on.
readimage doesn't need EOF, since it knows exactly how many bytes it needs to read.
hence axel's original program could work without a zero-length write.
however the reason he needed the sleep was that without it,
both processes were reading the original image's bytes from the pipe at the same time,
which (depending on scheduling) would result in too little data (and a hangup) or
a corrupt image message, or both.

if you were guaranteed that the image data would fit into the pipe without
blocking, then you could wait for the process to exit before reading,
but the two pipe solution is preferable.

Re: [9fans] trying to understand how fork/pipe a filtering program

[Skip Tavakkolian]

> if you were guaranteed that the image data would fit into the pipe without
> blocking, then you could wait for the process to exit before reading,
> but the two pipe solution is preferable.

or rendezvous

Re: [9fans] trying to understand how fork/pipe a filtering program

[rog]

> or rendezvous

i'm not sure rendezvous is an option here (you've no control over the child
process after execing it), and even if you did (e.g. by doing the wait in another
process), it still doesn't get around the fact that
when the data's large you need to be reading it at the same time that
the child is writing it - there's no appropriate time to rendezvous.

Re: [9fans] trying to understand how fork/pipe a filtering program

[Skip Tavakkolian]

>> or rendezvous
> 
> i'm not sure rendezvous is an option here (you've no control over the child
> process after execing it), and even if you did (e.g. by doing the wait in another
> process), it still doesn't get around the fact that
> when the data's large you need to be reading it at the same time that
> the child is writing it - there's no appropriate time to rendezvous.

right. Axel's question was about running resample in parallel.
as to why it works with a sleep, it's all stated in pipe(3). if you give
enough time to resample, it gets the first read on the pipe,
then parent gets the second read - which will come from a
write from resample.

i was thinking 
	switch (fork()) {
	child:	while (1) { rendezvous(0,0); if (read()<0)exit(); rendezvous(1,1); write(); }
	parent:	while (more) { rendezvous(0,1); write(); rendezvous(1,0); read(); }
	}

which isn't possible with an uncooperative program, and is a little dangerous.

wouldn't ioproc(2) be a better solution for this class of problems?

Re: [9fans] trying to understand how fork/pipe a filtering program

[Axel Belinfante]

> > I'm just trying to understand what goes on -
> > is this the way to do this? if not, what am I missing?
> 
> ahh, having thought about the original question, i finally understand
> what was going on.  readimage doesn't need EOF, since it knows exactly
> how many bytes it needs to read.
> hence axel's original program could work without a zero-length write.

all true. I did try the zero-length write (should have mentioned that).
it did not work with resample (you already suggested this possible
in your previous post) I also tried it with cat instead of resample,
and then (I think - tried a bit too many variants) it did work.

I am using the two pipe solution.

regarding Skip's comments: I'm not sure I completely understand.
the parallelism is not vital. I just want to use an existing
program (resample in this particular case) as filter to read
some of my data, process it, and give me the result, after which
I continue with my own thing.  I could have included the code of
the existing program directly in mine (as page does - I actually
first lifted that code from page), but then I'd have to track
possible changes to that code (I don't suppose there will be many
but that's not the point).

Axel.

Re: [9fans] trying to understand how fork/pipe a filtering program

[Gorka guardiola]

There is a third solution. You are using it, page is using it. It is for images
(so a lot of data will probably be moving around). Convert it to a library.
If it is not a lot of data and closeness to the app is not inmportant then
your approach is probably correct (that or convertint it into a service).

On 9/3/06, Axel Belinfante <Axel.Belinfante@cs.utwente.nl> wrote:
> I continue with my own thing.  I could have included the code of
> the existing program directly in mine (as page does - I actually
> first lifted that code from page), but then I'd have to track
> possible changes to that code (I don't suppose there will be many
> but that's not the point).
>
> Axel.
>


-- 
- curiosity sKilled the cat

Re: [9fans] trying to understand how fork/pipe a filtering program

[Gorka guardiola]

On 9/4/06, Gorka guardiola <paurea@gmail.com> wrote:
> There is a third solution. You are using it, page is using it. It is for images
When I said you are using it, I was talking about resample.

-- 
- curiosity sKilled the cat

Re: [9fans] trying to understand how fork/pipe a filtering program

[Skip Tavakkolian]

[-- Attachment #1: Type: text/plain, Size: 718 bytes --]

> regarding Skip's comments: I'm not sure I completely understand.
> the parallelism is not vital. I just want to use an existing
> program (resample in this particular case) as filter to read
> some of my data, process it, and give me the result, after which
> I continue with my own thing. 

just this: in your example, if parent needs to write more than some
pipe-buf-limit (i think 32k) to child and/or if child produces output
larger than this limit and blocks on the write, it would be a problem.

i attached my example of using only one pipe which requires two
rendezvous points and customization of the slave proc; this could
have a deadlock when a child requires multiple reads to cause a write.

[-- Attachment #2: pingpong.c --]
[-- Type: text/plain, Size: 1329 bytes --]

#include <u.h>
#include <libc.h>

void
ping(int *fd)
{
	int n, i = 0;
	char buf[1024];

	while (1) {
		i++;
		memset(buf, 'a', 1024);
		rendezvous(0, 0);

		if (i > 2) break;
		if ((n = write(fd[1], buf, sizeof(buf))) != sizeof(buf))
			break;

		print("ping: wrote %d bytes\n", n);

		rendezvous(1,0);
		if ((n = read(fd[0], buf, sizeof(buf))) <= 0)
			break;

		print("ping: read %d bytes\n", n);
		write(1, buf, n);
		write(1, "\n\n", 2);
	}
	close(fd[1]);
	close(fd[0]);
}

void
pong(int *fd)
{
	int n;
	char buf[1024];

	while(1) {
		rendezvous(0, 0);
		if ((n = read(fd[0], buf, sizeof(buf))) <= 0) break;

		print("pong: read %d bytes\n", n);
		write(1, buf, n);
		write(1, "\n\n", 2);

		memset(buf, 'b', sizeof(buf));
		rendezvous(1, 0);
		if ((n = write(fd[1], buf, sizeof(buf))) != sizeof(buf)) break;

		print("pong: wrote %d bytes\n", n);
	}
	close(fd[0]);
	close(fd[1]);
}

void
pingpong(void)
{
	int fd[2], pid, kid;
	
	if (pipe(fd) < 0)
		sysfatal("pipe failed: %r");

	switch (kid = rfork(RFPROC|RFFDG)) {
	case -1:
		sysfatal("fork failed: %r");
	case 0:
		pong(fd);
		break;
	default:
		ping(fd);
		while ((pid = waitpid()) != kid && pid != -1)
			;
		break;
	}
}

void
main(int argc, char **argv)
{
	pingpong();
	exits(nil);
}

Re: [9fans] trying to understand how fork/pipe a filtering program

[Francisco J Ballesteros]

I think there is an example in section 5.4 (or 5.5?) in the draft at
http://lsub.org/who/nemo/9.intro.pdf

hth

On 9/4/06, Skip Tavakkolian <9nut@9netics.com> wrote:
just this: in your example, if parent needs to write more than some
pipe-buf-limit (i think 32k) to child and/or if child produces output
larger than this limit and blocks on the write, it would be a problem.

i attached my example of using only one pipe which requires two
rendezvous points and customization of the slave proc; this could
have a deadlock when a child requires multiple reads to cause a write.