* [9fans] Pipes on UNIX @ 2007-09-06 13:04 Lluís Batlle 2007-09-06 13:57 ` sqweek 2007-09-06 14:04 ` Enrico Weigelt 0 siblings, 2 replies; 14+ messages in thread From: Lluís Batlle @ 2007-09-06 13:04 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs Hello, I understand from [1] that named pipes are suposed to use the filesystem as an "unlimited" buffer for the pipe. This would be an advantage against a pipe deadlock I'm experiencing using unnamed pipes (a CGI communicating wit Apache), but I cannot reproduce the "unlimited buffer" in Linux. Maybe I should fcntl the named pipe or something similar setting a buffer limit, and the filesystem will be used? Or maybe the named pipes were never supposed to use the disk as the pipe buffer? I don't know any other place where I may ask this question and get answered. My google search didn't bring me useful results since now. Otherwise I will probably write a simple program which acts as an unlimited pipe, using a disk backend as data queue. [1] http://www.cs.cmu.edu/~UniCon/reference-manual/Reference_Manual_41.html (Sorry for a non-plan9 question...) ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [9fans] Pipes on UNIX 2007-09-06 13:04 [9fans] Pipes on UNIX Lluís Batlle @ 2007-09-06 13:57 ` sqweek 2007-09-06 14:02 ` Lluís Batlle 2007-09-06 14:04 ` Enrico Weigelt 1 sibling, 1 reply; 14+ messages in thread From: sqweek @ 2007-09-06 13:57 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs On 9/6/07, Lluís Batlle <viriketo@gmail.com> wrote: > Hello, > > I understand from [1] that named pipes are suposed to use the > filesystem as an "unlimited" buffer for the pipe. This would be an > advantage against a pipe deadlock I'm experiencing using unnamed pipes > (a CGI communicating wit Apache), but I cannot reproduce the > "unlimited buffer" in Linux. On linux you want fifo(7), which says "When processes are exchanging data via the FIFO, the kernel passes all data internally without writing it to the file system.". Should answer the rest of your questions. -sqweek ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [9fans] Pipes on UNIX 2007-09-06 13:57 ` sqweek @ 2007-09-06 14:02 ` Lluís Batlle 2007-09-06 14:12 ` sqweek 0 siblings, 1 reply; 14+ messages in thread From: Lluís Batlle @ 2007-09-06 14:02 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs 2007/9/6, sqweek <sqweek@gmail.com>: > On linux you want fifo(7), which says "When processes are exchanging > data via the FIFO, the kernel passes all data internally without > writing it to the file system.". Should answer the rest of your > questions. Oh, thank you. I expected that mkfifo(1) would point to that page if it existed, and in my installation it doesn't point. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [9fans] Pipes on UNIX 2007-09-06 14:02 ` Lluís Batlle @ 2007-09-06 14:12 ` sqweek 2007-09-06 14:42 ` Lluís Batlle 0 siblings, 1 reply; 14+ messages in thread From: sqweek @ 2007-09-06 14:12 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs On 9/6/07, Lluís Batlle <viriketo@gmail.com> wrote: > 2007/9/6, sqweek <sqweek@gmail.com>: > > On linux you want fifo(7), > Oh, thank you. I expected that mkfifo(1) would point to that page if > it existed, and in my installation it doesn't point. You expected a GNU man page to be useful? Are you kidding me?? -sqweek ;) ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [9fans] Pipes on UNIX 2007-09-06 14:12 ` sqweek @ 2007-09-06 14:42 ` Lluís Batlle 2007-09-06 14:47 ` Gorka Guardiola 0 siblings, 1 reply; 14+ messages in thread From: Lluís Batlle @ 2007-09-06 14:42 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs I was more used to blame GNU's code than man pages :) To Enrico, in fact I want to use the disk-buffered pipe program in a shell. Something like: generate_data | diskpipe | data_eater I don't think a 9p server would do anything here. Thank you, Lluís. 2007/9/6, sqweek <sqweek@gmail.com>: > On 9/6/07, Lluís Batlle <viriketo@gmail.com> wrote: > > 2007/9/6, sqweek <sqweek@gmail.com>: > > > On linux you want fifo(7), > > Oh, thank you. I expected that mkfifo(1) would point to that page if > > it existed, and in my installation it doesn't point. > > You expected a GNU man page to be useful? Are you kidding me?? > -sqweek ;) > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [9fans] Pipes on UNIX 2007-09-06 14:42 ` Lluís Batlle @ 2007-09-06 14:47 ` Gorka Guardiola 2007-09-06 14:57 ` Lluís Batlle 2007-09-06 15:52 ` ron minnich 0 siblings, 2 replies; 14+ messages in thread From: Gorka Guardiola @ 2007-09-06 14:47 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs On 9/6/07, Lluís Batlle <viriketo@gmail.com> wrote: > I was more used to blame GNU's code than man pages :) > > To Enrico, in fact I want to use the disk-buffered pipe program in a shell. > Something like: > generate_data | diskpipe | data_eater > I don't think a 9p server would do anything here. > generate_data > input cat input > output data_eater < output with output and input being files from a file server. -- - curiosity sKilled the cat ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [9fans] Pipes on UNIX 2007-09-06 14:47 ` Gorka Guardiola @ 2007-09-06 14:57 ` Lluís Batlle 2007-09-06 15:52 ` ron minnich 1 sibling, 0 replies; 14+ messages in thread From: Lluís Batlle @ 2007-09-06 14:57 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs I was thinking in a general solution for having pipes with very big buffers... but your much simpler proposal fixes perfectly my problem ;) I'm having too much problems on over-engineering these last days... Thank you, Lluís. 2007/9/6, Gorka Guardiola <paurea@gmail.com>: > On 9/6/07, Lluís Batlle <viriketo@gmail.com> wrote: > > I was more used to blame GNU's code than man pages :) > > > > To Enrico, in fact I want to use the disk-buffered pipe program in a shell. > > Something like: > > generate_data | diskpipe | data_eater > > I don't think a 9p server would do anything here. > > > > generate_data > input > cat input > output > data_eater < output > > with output and input being files from a file server. > > -- > - curiosity sKilled the cat > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [9fans] Pipes on UNIX 2007-09-06 14:47 ` Gorka Guardiola 2007-09-06 14:57 ` Lluís Batlle @ 2007-09-06 15:52 ` ron minnich 2007-09-06 16:06 ` Gorka Guardiola 1 sibling, 1 reply; 14+ messages in thread From: ron minnich @ 2007-09-06 15:52 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs On 9/6/07, Gorka Guardiola <paurea@gmail.com> wrote: > generate_data > input > cat input > output > data_eater < output This in the general case is a bad idea, from many angles :-) ron ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [9fans] Pipes on UNIX 2007-09-06 15:52 ` ron minnich @ 2007-09-06 16:06 ` Gorka Guardiola 2007-09-06 20:17 ` Lluís Batlle 0 siblings, 1 reply; 14+ messages in thread From: Gorka Guardiola @ 2007-09-06 16:06 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs I wrote it without coffee... would be more like: generate_data > input & data_eater < output :-). And input and output can come from a stupid fileserver like pipefs... or pipebackedbyaharddiskfs :-). -- - curiosity sKilled the cat ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [9fans] Pipes on UNIX 2007-09-06 16:06 ` Gorka Guardiola @ 2007-09-06 20:17 ` Lluís Batlle 2007-09-06 20:33 ` erik quanstrom 2007-09-06 21:51 ` Charles Forsyth 0 siblings, 2 replies; 14+ messages in thread From: Lluís Batlle @ 2007-09-06 20:17 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs I better explain my problem. :) I have a CGI for Apache. I wrote it in C, and it simply parses text from a html textarea, giving another html in output. I made it as a state machine which outputs and outputs according to the input characters. Apache communicates with the CGI process with two pipes linked to the process' stdin and stdout. But I think that apache code is like this: while() { read(socket,buffer) write(pipe_process1, buffer) } while() { read(pipe_process2, buffer) write(socket,buffer) } Therefore, if I had a CGI similar to 'cat' (and I have), it would deadlock due to the pipe_process2 not being read by apache if it gets filled, the process blocks, and then pipe_process1 gets filled. As I want the CGI to accept input from almost any length, I need a "pipe buffer" big enough. Then, based on Gorka suggestion (which I didn't understand as a 9pserver ;), I finally agreed on the easiest solution wrapping the CGI with a sh script: #!/bin/sh head -n 10000000 > inputfile # limiting the disk ussage to 10MB, raw cutting. mycgi < inputfile #EOF That's all. I will not write any "superpipe" program, or any 9pserver, because at the end I need the disk. The sh wrapper works well enough. Thank you for your help in this non-plan9-related problem! Simply the unicon documentation pointed to something strange in named pipes which I couldn't check anywhere. :) Regards, Lluís 2007/9/6, Gorka Guardiola <paurea@gmail.com>: > I wrote it without coffee... would be more like: > > generate_data > input & > data_eater < output > > :-). > > And input and output can come from a stupid fileserver like pipefs... > or pipebackedbyaharddiskfs :-). > -- > - curiosity sKilled the cat > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [9fans] Pipes on UNIX 2007-09-06 20:17 ` Lluís Batlle @ 2007-09-06 20:33 ` erik quanstrom 2007-09-06 20:40 ` Lluís Batlle 2007-09-06 21:51 ` Charles Forsyth 1 sibling, 1 reply; 14+ messages in thread From: erik quanstrom @ 2007-09-06 20:33 UTC (permalink / raw) To: 9fans > I better explain my problem. :) > I have a CGI for Apache. I wrote it in C, and it simply parses text > from a html textarea, giving another html in output. I made it as a > state machine which outputs and outputs according to the input > characters. > > Apache communicates with the CGI process with two pipes linked to the > process' stdin and stdout. > But I think that apache code is like this: > while() > { > read(socket,buffer) > write(pipe_process1, buffer) > } > while() > { > read(pipe_process2, buffer) > write(socket,buffer) > } > > Therefore, if I had a CGI similar to 'cat' (and I have), it would > deadlock due to the pipe_process2 not being read by apache if it gets > filled, the process blocks, and then pipe_process1 gets filled. no, you wouldn't. i believe the problem is the cgi and apache disagree on the termination condition. make sure you close the fd 0 when you are done reading. - erik ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [9fans] Pipes on UNIX 2007-09-06 20:33 ` erik quanstrom @ 2007-09-06 20:40 ` Lluís Batlle 0 siblings, 0 replies; 14+ messages in thread From: Lluís Batlle @ 2007-09-06 20:40 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs 2007/9/6, erik quanstrom <quanstro@quanstro.net>: > > Therefore, if I had a CGI similar to 'cat' (and I have), it would > > deadlock due to the pipe_process2 not being read by apache if it gets > > filled, the process blocks, and then pipe_process1 gets filled. > > no, you wouldn't. > > i believe the problem is the cgi and apache disagree on the termination > condition. make sure you close the fd 0 when you are done reading. I used gdb, and my process doesn't reach at all the end of reading. It gets blocked writting to stdout. The system input TCP queue to apache is still filled with 70KB of data in the test I did. And the browser didn't receive any byte. And I know my program processed an amount of data (it makes the hard disk pain a bit, so I can 'hear' when it does something). ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [9fans] Pipes on UNIX 2007-09-06 20:17 ` Lluís Batlle 2007-09-06 20:33 ` erik quanstrom @ 2007-09-06 21:51 ` Charles Forsyth 1 sibling, 0 replies; 14+ messages in thread From: Charles Forsyth @ 2007-09-06 21:51 UTC (permalink / raw) To: 9fans > Thank you for your help in this non-plan9-related problem! Simply the > unicon documentation pointed to something strange in named pipes which > I couldn't check anywhere. :) i couldn't work out where they got that impression or information. i'm sure i've never seen that in any of the unix-y implementations i've read. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [9fans] Pipes on UNIX 2007-09-06 13:04 [9fans] Pipes on UNIX Lluís Batlle 2007-09-06 13:57 ` sqweek @ 2007-09-06 14:04 ` Enrico Weigelt 1 sibling, 0 replies; 14+ messages in thread From: Enrico Weigelt @ 2007-09-06 14:04 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs * Lluís Batlle <viriketo@gmail.com> wrote: Hi, although, it's quite OT ... > I understand from [1] that named pipes are suposed to use the > filesystem as an "unlimited" buffer for the pipe. Actually, not. (at least on Unix) Pipes are FIFOs with an limited buffer size (ie on linux-2.4 of one page=4k, on linux-2.6 IMHO 64k). And pipes are always meant for an sequential and continous data flow between reader and writer. Yes, there are some deadlock flaws (experienced them by myself ie. w/ sun-jdk), but they don't come from a limited buffer, but unclean handling of tricky situations (ie too large or asymetric chunks on reader vs. writer). You probably can work around them by reading byte per byte (ugly!). Maybe you would like to have a look at Unix Domain Sockets. They support bi-directional transfers, fd passing (at least @ Linux) and packet transfer. You can control many things using setsockopt(1). > Or maybe the named pipes were never supposed to use the disk as > the pipe buffer? Exactly. For those things you could simply write an little 9p server ;-) cu -- --------------------------------------------------------------------- Enrico Weigelt == metux IT service - http://www.metux.de/ --------------------------------------------------------------------- Please visit the OpenSource QM Taskforce: http://wiki.metux.de/public/OpenSource_QM_Taskforce Patches / Fixes for a lot dozens of packages in dozens of versions: http://patches.metux.de/ --------------------------------------------------------------------- ^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2007-09-06 21:51 UTC | newest] Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2007-09-06 13:04 [9fans] Pipes on UNIX Lluís Batlle 2007-09-06 13:57 ` sqweek 2007-09-06 14:02 ` Lluís Batlle 2007-09-06 14:12 ` sqweek 2007-09-06 14:42 ` Lluís Batlle 2007-09-06 14:47 ` Gorka Guardiola 2007-09-06 14:57 ` Lluís Batlle 2007-09-06 15:52 ` ron minnich 2007-09-06 16:06 ` Gorka Guardiola 2007-09-06 20:17 ` Lluís Batlle 2007-09-06 20:33 ` erik quanstrom 2007-09-06 20:40 ` Lluís Batlle 2007-09-06 21:51 ` Charles Forsyth 2007-09-06 14:04 ` Enrico Weigelt
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).