* [9fans] trying to understand how fork/pipe a filtering program @ 2006-08-30 22:46 Axel Belinfante 2006-08-30 23:05 ` geoff ` (2 more replies) 0 siblings, 3 replies; 19+ messages in thread From: Axel Belinfante @ 2006-08-30 22:46 UTC (permalink / raw) To: 9fans after the suggestion of resample(1) some days ago I've tried run it as a filter from another program. this other program is non-threaded; for testing all it does is call initdraw and read an image file after which the resample routine is called. in my resample routine it seems I need a call to sleep to avoid the resample program to hang (in Pread). (replacing the sleep by a fprint(2, "something\n") works too). I'm just trying to understand what goes on - is this the way to do this? if not, what am I missing? I'm sorry if it is something obvious; I'm just trying to learn. Axel. Image* resample(Image *from) { int p[2]; Image *img; if(pipe(p) < 0){ fprint(2, "pipe fails: %r\n"); exits("Epipe"); } switch(fork()){ case -1: fprint(2, "fork fails: %r\n"); exits("Efork"); default: if (writeimage(p[0], from, 0) < 0) fprint(2, "error writeimage\n"); close(p[0]); /* seems we need sleep to give up cpu so resample can run */ sleep(1); img = readimage(display, p[1], 0); close(p[1]); waitpid(); return img; case 0: dup(p[1], 0); dup(p[0], 1); execl("/bin/resample", "resample", "-x50%", nil); fprint(2, "execl fails: %r\n"); exits("Eexec"); } return nil; } ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [9fans] trying to understand how fork/pipe a filtering program 2006-08-30 22:46 [9fans] trying to understand how fork/pipe a filtering program Axel Belinfante @ 2006-08-30 23:05 ` geoff 2006-08-31 16:26 ` "Nils O. Selåsdal" 2006-09-01 14:35 ` rog 2 siblings, 0 replies; 19+ messages in thread From: geoff @ 2006-08-30 23:05 UTC (permalink / raw) To: 9fans I think you want something more like this: int p[2], kid, pid; Image *img; if(pipe(p) < 0) sysfatal("can't make a pipe: %r"); kid = fork(); switch(kid){ case -1: sysfatal("can't fork: %r"); default: if (writeimage(p[0], from, 0) < 0) fprint(2, "error writeimage\n"); close(p[0]); img = readimage(display, p[1], 0); close(p[1]); /* * we may have unknown children forked by a previous * program (e.g., rc) that then execed us. */ while ((pid = waitpid()) != kid && pid != -1) continue; return img; case 0: dup(p[1], 0); dup(p[0], 1); close(p[0]); close(p[1]); execl("/bin/resample", "resample", "-x50%", (char *)nil); sysfatal("can't exec resample: %r"); } ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [9fans] trying to understand how fork/pipe a filtering program 2006-08-30 22:46 [9fans] trying to understand how fork/pipe a filtering program Axel Belinfante 2006-08-30 23:05 ` geoff @ 2006-08-31 16:26 ` "Nils O. Selåsdal" 2006-08-31 19:55 ` Axel Belinfante 2006-09-01 14:35 ` rog 2 siblings, 1 reply; 19+ messages in thread From: "Nils O. Selåsdal" @ 2006-08-31 16:26 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs Axel Belinfante wrote: > after the suggestion of resample(1) some days ago > I've tried run it as a filter from another program. > this other program is non-threaded; for testing all > it does is call initdraw and read an image file > after which the resample routine is called. > > in my resample routine it seems I need a call to sleep > to avoid the resample program to hang (in Pread). > (replacing the sleep by a fprint(2, "something\n") works too). > > I'm just trying to understand what goes on - > is this the way to do this? if not, what am I missing? > I'm sorry if it is something obvious; I'm just trying to learn. Remember to close the pipe ends you don't need in the respective parent/child, as well as the original pipes after dup'ing them. ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [9fans] trying to understand how fork/pipe a filtering program 2006-08-31 16:26 ` "Nils O. Selåsdal" @ 2006-08-31 19:55 ` Axel Belinfante 2006-08-31 20:15 ` rog ` (2 more replies) 0 siblings, 3 replies; 19+ messages in thread From: Axel Belinfante @ 2006-08-31 19:55 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs suggestions by geoff and noselasd helped to improve my code but did not solve the problem. the problem seems to be (related to?) that I'm trying to use one pipe in a bidirectional way. using two pipes unidirectionally does make it work. actually, the whole thing gives me some feeling of deja-vu of the thing I brought up here about a year ago in thread 'thread confusion', which was then explained by russ in http://9fans.net/archive/2005/09/270 Axel. ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [9fans] trying to understand how fork/pipe a filtering program 2006-08-31 19:55 ` Axel Belinfante @ 2006-08-31 20:15 ` rog 2006-08-31 20:24 ` Axel Belinfante 2006-08-31 20:24 ` rog 2006-09-01 3:43 ` geoff 2 siblings, 1 reply; 19+ messages in thread From: rog @ 2006-08-31 20:15 UTC (permalink / raw) To: 9fans i'd have thought that something like the following should work: int p[2], q[2], kid, pid; Image *img; if(pipe(p) < 0 || pipe(q) < 0) sysfatal("can't make a pipe: %r"); kid = fork(); switch(kid){ case -1: sysfatal("can't fork: %r"); default: close(p[0]); close(q[1]); if (writeimage(p[1], from, 0) < 0) fprint(2, "error writeimage\n"); close(p[1]); img = readimage(display, q[0], 0); close(q[0]); /* * we may have unknown children forked by a previous * program (e.g., rc) that then execed us. */ while ((pid = waitpid()) != kid && pid != -1) continue; return img; case 0: dup(p[0], 0); dup(q[1], 1); close(p[1]); close(q[0]); execl("/bin/resample", "resample", "-x50%", (char *)nil); sysfatal("can't exec resample: %r"); } ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [9fans] trying to understand how fork/pipe a filtering program 2006-08-31 20:15 ` rog @ 2006-08-31 20:24 ` Axel Belinfante 0 siblings, 0 replies; 19+ messages in thread From: Axel Belinfante @ 2006-08-31 20:24 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs > i'd have thought that something like the following should work: correct, it does work. I tried to say that in my last message. I'm sorry if it only added to the confusion. the main thing is that what you suggest indeed does work. Axel. ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [9fans] trying to understand how fork/pipe a filtering program 2006-08-31 19:55 ` Axel Belinfante 2006-08-31 20:15 ` rog @ 2006-08-31 20:24 ` rog 2006-08-31 20:45 ` Axel Belinfante 2006-09-01 3:43 ` geoff 2 siblings, 1 reply; 19+ messages in thread From: rog @ 2006-08-31 20:24 UTC (permalink / raw) To: 9fans oops skim reading again! i'd thought i'd read "using two pipes unidirectionally *doesn't* make it work." so why can't you just use two pipes unidirectionally? you can't do anything else in this example, because by using the same pipe for input and output, the command itself keeps its own input open, and therefore never receives EOF. ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [9fans] trying to understand how fork/pipe a filtering program 2006-08-31 20:24 ` rog @ 2006-08-31 20:45 ` Axel Belinfante 0 siblings, 0 replies; 19+ messages in thread From: Axel Belinfante @ 2006-08-31 20:45 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs > oops skim reading again! > i'd thought i'd read "using two pipes unidirectionally *doesn't* make it work." > so why can't you just use two pipes unidirectionally? you can't do anything > else in this example, because by using the same pipe for input and output, > the command itself keeps its own input open, and therefore never receives EOF. seems it took me a while to realize that. the funny thing is that, somehow, using the same pipe for input and output does work (at least for me) if in the parent I have an fprint or a sleep between the writing+closing part and the reading part, as in the message with which I started this thread. that said, I'm much happier with the 'two unidirectional pipes' solution. Axel. ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [9fans] trying to understand how fork/pipe a filtering program 2006-08-31 19:55 ` Axel Belinfante 2006-08-31 20:15 ` rog 2006-08-31 20:24 ` rog @ 2006-09-01 3:43 ` geoff 2006-09-01 10:57 ` rog 2 siblings, 1 reply; 19+ messages in thread From: geoff @ 2006-09-01 3:43 UTC (permalink / raw) To: 9fans Maybe it's been too long a week, but I could have sworn that a Plan 9 pipe was a pair of unidirectional channels (queues in Streams terminology), so closing one end would cause a zero-byte read at the other, independently in each direction. pipe(3) seems to support this. ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [9fans] trying to understand how fork/pipe a filtering program 2006-09-01 3:43 ` geoff @ 2006-09-01 10:57 ` rog 0 siblings, 0 replies; 19+ messages in thread From: rog @ 2006-09-01 10:57 UTC (permalink / raw) To: 9fans > Maybe it's been too long a week, but I could have sworn that a Plan 9 > pipe was a pair of unidirectional channels (queues in Streams > terminology), so closing one end would cause a zero-byte read at the > other, independently in each direction. pipe(3) seems to support > this. the problem is that neither end was actually being closed, as both ends of the pipe were dup'd onto stdin/stdout of the child process. (the underlying close only happens when all references have gone). there is an alternative, i realised last night - you can do a zero-length write after doing the writeimage, which will cause the child to read EOF, which is usually fine (unless the child re-reads EOF, expecting it to remain EOF, which can be the case with some buffered IO libraries and one-char lookahead; depends whether it implements ungetc(EOF) - bio has this issue, for example) ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [9fans] trying to understand how fork/pipe a filtering program 2006-08-30 22:46 [9fans] trying to understand how fork/pipe a filtering program Axel Belinfante 2006-08-30 23:05 ` geoff 2006-08-31 16:26 ` "Nils O. Selåsdal" @ 2006-09-01 14:35 ` rog 2006-09-01 15:33 ` Skip Tavakkolian 2006-09-03 19:09 ` Axel Belinfante 2 siblings, 2 replies; 19+ messages in thread From: rog @ 2006-09-01 14:35 UTC (permalink / raw) To: 9fans > I'm just trying to understand what goes on - > is this the way to do this? if not, what am I missing? ahh, having thought about the original question, i finally understand what was going on. readimage doesn't need EOF, since it knows exactly how many bytes it needs to read. hence axel's original program could work without a zero-length write. however the reason he needed the sleep was that without it, both processes were reading the original image's bytes from the pipe at the same time, which (depending on scheduling) would result in too little data (and a hangup) or a corrupt image message, or both. if you were guaranteed that the image data would fit into the pipe without blocking, then you could wait for the process to exit before reading, but the two pipe solution is preferable. ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [9fans] trying to understand how fork/pipe a filtering program 2006-09-01 14:35 ` rog @ 2006-09-01 15:33 ` Skip Tavakkolian 2006-09-01 15:59 ` rog 2006-09-03 19:09 ` Axel Belinfante 1 sibling, 1 reply; 19+ messages in thread From: Skip Tavakkolian @ 2006-09-01 15:33 UTC (permalink / raw) To: 9fans > if you were guaranteed that the image data would fit into the pipe without > blocking, then you could wait for the process to exit before reading, > but the two pipe solution is preferable. or rendezvous ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [9fans] trying to understand how fork/pipe a filtering program 2006-09-01 15:33 ` Skip Tavakkolian @ 2006-09-01 15:59 ` rog 2006-09-01 18:42 ` Skip Tavakkolian 0 siblings, 1 reply; 19+ messages in thread From: rog @ 2006-09-01 15:59 UTC (permalink / raw) To: 9fans > or rendezvous i'm not sure rendezvous is an option here (you've no control over the child process after execing it), and even if you did (e.g. by doing the wait in another process), it still doesn't get around the fact that when the data's large you need to be reading it at the same time that the child is writing it - there's no appropriate time to rendezvous. ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [9fans] trying to understand how fork/pipe a filtering program 2006-09-01 15:59 ` rog @ 2006-09-01 18:42 ` Skip Tavakkolian 0 siblings, 0 replies; 19+ messages in thread From: Skip Tavakkolian @ 2006-09-01 18:42 UTC (permalink / raw) To: 9fans >> or rendezvous > > i'm not sure rendezvous is an option here (you've no control over the child > process after execing it), and even if you did (e.g. by doing the wait in another > process), it still doesn't get around the fact that > when the data's large you need to be reading it at the same time that > the child is writing it - there's no appropriate time to rendezvous. right. Axel's question was about running resample in parallel. as to why it works with a sleep, it's all stated in pipe(3). if you give enough time to resample, it gets the first read on the pipe, then parent gets the second read - which will come from a write from resample. i was thinking switch (fork()) { child: while (1) { rendezvous(0,0); if (read()<0)exit(); rendezvous(1,1); write(); } parent: while (more) { rendezvous(0,1); write(); rendezvous(1,0); read(); } } which isn't possible with an uncooperative program, and is a little dangerous. wouldn't ioproc(2) be a better solution for this class of problems? ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [9fans] trying to understand how fork/pipe a filtering program 2006-09-01 14:35 ` rog 2006-09-01 15:33 ` Skip Tavakkolian @ 2006-09-03 19:09 ` Axel Belinfante 2006-09-04 14:52 ` Gorka guardiola 2006-09-04 19:26 ` Skip Tavakkolian 1 sibling, 2 replies; 19+ messages in thread From: Axel Belinfante @ 2006-09-03 19:09 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs > > I'm just trying to understand what goes on - > > is this the way to do this? if not, what am I missing? > > ahh, having thought about the original question, i finally understand > what was going on. readimage doesn't need EOF, since it knows exactly > how many bytes it needs to read. > hence axel's original program could work without a zero-length write. all true. I did try the zero-length write (should have mentioned that). it did not work with resample (you already suggested this possible in your previous post) I also tried it with cat instead of resample, and then (I think - tried a bit too many variants) it did work. I am using the two pipe solution. regarding Skip's comments: I'm not sure I completely understand. the parallelism is not vital. I just want to use an existing program (resample in this particular case) as filter to read some of my data, process it, and give me the result, after which I continue with my own thing. I could have included the code of the existing program directly in mine (as page does - I actually first lifted that code from page), but then I'd have to track possible changes to that code (I don't suppose there will be many but that's not the point). Axel. ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [9fans] trying to understand how fork/pipe a filtering program 2006-09-03 19:09 ` Axel Belinfante @ 2006-09-04 14:52 ` Gorka guardiola 2006-09-04 20:50 ` Gorka guardiola 2006-09-04 19:26 ` Skip Tavakkolian 1 sibling, 1 reply; 19+ messages in thread From: Gorka guardiola @ 2006-09-04 14:52 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs There is a third solution. You are using it, page is using it. It is for images (so a lot of data will probably be moving around). Convert it to a library. If it is not a lot of data and closeness to the app is not inmportant then your approach is probably correct (that or convertint it into a service). On 9/3/06, Axel Belinfante <Axel.Belinfante@cs.utwente.nl> wrote: > I continue with my own thing. I could have included the code of > the existing program directly in mine (as page does - I actually > first lifted that code from page), but then I'd have to track > possible changes to that code (I don't suppose there will be many > but that's not the point). > > Axel. > -- - curiosity sKilled the cat ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [9fans] trying to understand how fork/pipe a filtering program 2006-09-04 14:52 ` Gorka guardiola @ 2006-09-04 20:50 ` Gorka guardiola 0 siblings, 0 replies; 19+ messages in thread From: Gorka guardiola @ 2006-09-04 20:50 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs On 9/4/06, Gorka guardiola <paurea@gmail.com> wrote: > There is a third solution. You are using it, page is using it. It is for images When I said you are using it, I was talking about resample. -- - curiosity sKilled the cat ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [9fans] trying to understand how fork/pipe a filtering program 2006-09-03 19:09 ` Axel Belinfante 2006-09-04 14:52 ` Gorka guardiola @ 2006-09-04 19:26 ` Skip Tavakkolian 2006-09-04 20:25 ` Francisco J Ballesteros 1 sibling, 1 reply; 19+ messages in thread From: Skip Tavakkolian @ 2006-09-04 19:26 UTC (permalink / raw) To: 9fans [-- Attachment #1: Type: text/plain, Size: 718 bytes --] > regarding Skip's comments: I'm not sure I completely understand. > the parallelism is not vital. I just want to use an existing > program (resample in this particular case) as filter to read > some of my data, process it, and give me the result, after which > I continue with my own thing. just this: in your example, if parent needs to write more than some pipe-buf-limit (i think 32k) to child and/or if child produces output larger than this limit and blocks on the write, it would be a problem. i attached my example of using only one pipe which requires two rendezvous points and customization of the slave proc; this could have a deadlock when a child requires multiple reads to cause a write. [-- Attachment #2: pingpong.c --] [-- Type: text/plain, Size: 1329 bytes --] #include <u.h> #include <libc.h> void ping(int *fd) { int n, i = 0; char buf[1024]; while (1) { i++; memset(buf, 'a', 1024); rendezvous(0, 0); if (i > 2) break; if ((n = write(fd[1], buf, sizeof(buf))) != sizeof(buf)) break; print("ping: wrote %d bytes\n", n); rendezvous(1,0); if ((n = read(fd[0], buf, sizeof(buf))) <= 0) break; print("ping: read %d bytes\n", n); write(1, buf, n); write(1, "\n\n", 2); } close(fd[1]); close(fd[0]); } void pong(int *fd) { int n; char buf[1024]; while(1) { rendezvous(0, 0); if ((n = read(fd[0], buf, sizeof(buf))) <= 0) break; print("pong: read %d bytes\n", n); write(1, buf, n); write(1, "\n\n", 2); memset(buf, 'b', sizeof(buf)); rendezvous(1, 0); if ((n = write(fd[1], buf, sizeof(buf))) != sizeof(buf)) break; print("pong: wrote %d bytes\n", n); } close(fd[0]); close(fd[1]); } void pingpong(void) { int fd[2], pid, kid; if (pipe(fd) < 0) sysfatal("pipe failed: %r"); switch (kid = rfork(RFPROC|RFFDG)) { case -1: sysfatal("fork failed: %r"); case 0: pong(fd); break; default: ping(fd); while ((pid = waitpid()) != kid && pid != -1) ; break; } } void main(int argc, char **argv) { pingpong(); exits(nil); } ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [9fans] trying to understand how fork/pipe a filtering program 2006-09-04 19:26 ` Skip Tavakkolian @ 2006-09-04 20:25 ` Francisco J Ballesteros 0 siblings, 0 replies; 19+ messages in thread From: Francisco J Ballesteros @ 2006-09-04 20:25 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs I think there is an example in section 5.4 (or 5.5?) in the draft at http://lsub.org/who/nemo/9.intro.pdf hth On 9/4/06, Skip Tavakkolian <9nut@9netics.com> wrote: just this: in your example, if parent needs to write more than some pipe-buf-limit (i think 32k) to child and/or if child produces output larger than this limit and blocks on the write, it would be a problem. i attached my example of using only one pipe which requires two rendezvous points and customization of the slave proc; this could have a deadlock when a child requires multiple reads to cause a write. ^ permalink raw reply [flat|nested] 19+ messages in thread
end of thread, other threads:[~2006-09-04 20:50 UTC | newest] Thread overview: 19+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2006-08-30 22:46 [9fans] trying to understand how fork/pipe a filtering program Axel Belinfante 2006-08-30 23:05 ` geoff 2006-08-31 16:26 ` "Nils O. Selåsdal" 2006-08-31 19:55 ` Axel Belinfante 2006-08-31 20:15 ` rog 2006-08-31 20:24 ` Axel Belinfante 2006-08-31 20:24 ` rog 2006-08-31 20:45 ` Axel Belinfante 2006-09-01 3:43 ` geoff 2006-09-01 10:57 ` rog 2006-09-01 14:35 ` rog 2006-09-01 15:33 ` Skip Tavakkolian 2006-09-01 15:59 ` rog 2006-09-01 18:42 ` Skip Tavakkolian 2006-09-03 19:09 ` Axel Belinfante 2006-09-04 14:52 ` Gorka guardiola 2006-09-04 20:50 ` Gorka guardiola 2006-09-04 19:26 ` Skip Tavakkolian 2006-09-04 20:25 ` Francisco J Ballesteros
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).