[9fans] Pipes on UNIX

[9fans] Pipes on UNIX

[Lluís Batlle]

Hello,

I understand from [1] that named pipes are suposed to use the
filesystem as an "unlimited" buffer for the pipe. This would be an
advantage against a pipe deadlock I'm experiencing using unnamed pipes
(a CGI communicating wit Apache), but I cannot reproduce the
"unlimited buffer" in Linux. Maybe I should fcntl the named pipe or
something similar setting a buffer limit, and the filesystem will be
used? Or maybe the named pipes were never supposed to use the disk as
the pipe buffer?

I don't know any other place where I may ask this question and get
answered. My google search didn't bring me useful results since now.

Otherwise I will probably write a simple program which acts as an
unlimited pipe, using a disk backend as data queue.

[1] http://www.cs.cmu.edu/~UniCon/reference-manual/Reference_Manual_41.html

(Sorry for a non-plan9 question...)

Re: [9fans] Pipes on UNIX

[sqweek]

On 9/6/07, Lluís Batlle <viriketo@gmail.com> wrote:
> Hello,
>
> I understand from [1] that named pipes are suposed to use the
> filesystem as an "unlimited" buffer for the pipe. This would be an
> advantage against a pipe deadlock I'm experiencing using unnamed pipes
> (a CGI communicating wit Apache), but I cannot reproduce the
> "unlimited buffer" in Linux.

 On linux you want fifo(7), which says "When processes are exchanging
data via the FIFO, the kernel passes  all  data  internally  without
writing it to the file system.". Should answer the rest of your
questions.
-sqweek

Re: [9fans] Pipes on UNIX

[Lluís Batlle]

2007/9/6, sqweek <sqweek@gmail.com>:
>  On linux you want fifo(7), which says "When processes are exchanging
> data via the FIFO, the kernel passes  all  data  internally  without
> writing it to the file system.". Should answer the rest of your
> questions.
Oh, thank you. I expected that mkfifo(1) would point to that page if
it existed, and in my installation it doesn't point.

Re: [9fans] Pipes on UNIX

[Enrico Weigelt]

* Lluís Batlle <viriketo@gmail.com> wrote:

Hi,

although, it's quite OT ...

> I understand from [1] that named pipes are suposed to use the
> filesystem as an "unlimited" buffer for the pipe. 

Actually, not. (at least on Unix)

Pipes are FIFOs with an limited buffer size (ie on linux-2.4 of one 
page=4k, on linux-2.6 IMHO 64k). And pipes are always meant for an 
sequential and continous data flow between reader and writer. Yes, 
there are some deadlock flaws (experienced them by myself ie. w/ 
sun-jdk), but they don't come from a limited buffer, but unclean 
handling of tricky situations (ie too large or asymetric chunks on 
reader vs. writer). You probably can work around them by reading
byte per byte (ugly!).

Maybe you would like to have a look at Unix Domain Sockets.
They support bi-directional transfers, fd passing (at least @ Linux)
and packet transfer. You can control many things using setsockopt(1).

> Or maybe the named pipes were never supposed to use the disk as
> the pipe buffer?

Exactly.

For those things you could simply write an little 9p server ;-)
 

cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------

Re: [9fans] Pipes on UNIX

[sqweek]

On 9/6/07, Lluís Batlle <viriketo@gmail.com> wrote:
> 2007/9/6, sqweek <sqweek@gmail.com>:
> >  On linux you want fifo(7),
> Oh, thank you. I expected that mkfifo(1) would point to that page if
> it existed, and in my installation it doesn't point.

 You expected a GNU man page to be useful? Are you kidding me??
-sqweek ;)

Re: [9fans] Pipes on UNIX

[Lluís Batlle]

I was more used to blame GNU's code than man pages :)

To Enrico, in fact I want to use the disk-buffered pipe program in a shell.
Something like:
generate_data | diskpipe | data_eater
I don't think a 9p server would do anything here.

Thank you,
Lluís.

2007/9/6, sqweek <sqweek@gmail.com>:
> On 9/6/07, Lluís Batlle <viriketo@gmail.com> wrote:
> > 2007/9/6, sqweek <sqweek@gmail.com>:
> > >  On linux you want fifo(7),
> > Oh, thank you. I expected that mkfifo(1) would point to that page if
> > it existed, and in my installation it doesn't point.
>
>  You expected a GNU man page to be useful? Are you kidding me??
> -sqweek ;)
>

Re: [9fans] Pipes on UNIX

[Gorka Guardiola]

On 9/6/07, Lluís Batlle <viriketo@gmail.com> wrote:
> I was more used to blame GNU's code than man pages :)
>
> To Enrico, in fact I want to use the disk-buffered pipe program in a shell.
> Something like:
> generate_data | diskpipe | data_eater
> I don't think a 9p server would do anything here.
>

generate_data > input
cat input > output
data_eater < output

with output and input being files from a file server.

-- 
- curiosity sKilled the cat

Re: [9fans] Pipes on UNIX

[Lluís Batlle]

I was thinking in a general solution for having pipes with very big buffers...
but your much simpler proposal fixes perfectly my problem ;)

I'm having too much problems on over-engineering these last days...

Thank you,
Lluís.

2007/9/6, Gorka Guardiola <paurea@gmail.com>:
> On 9/6/07, Lluís Batlle <viriketo@gmail.com> wrote:
> > I was more used to blame GNU's code than man pages :)
> >
> > To Enrico, in fact I want to use the disk-buffered pipe program in a shell.
> > Something like:
> > generate_data | diskpipe | data_eater
> > I don't think a 9p server would do anything here.
> >
>
> generate_data > input
> cat input > output
> data_eater < output
>
> with output and input being files from a file server.
>
> --
> - curiosity sKilled the cat
>

Re: [9fans] Pipes on UNIX

[ron minnich]

On 9/6/07, Gorka Guardiola <paurea@gmail.com> wrote:

> generate_data > input
> cat input > output
> data_eater < output

This in the general case is a bad idea, from many angles :-)

ron

Re: [9fans] Pipes on UNIX

[Gorka Guardiola]

I wrote it without coffee... would be more like:

generate_data > input &
data_eater < output

:-).

And input and output can come from a stupid fileserver like pipefs...
or pipebackedbyaharddiskfs :-).
--
- curiosity sKilled the cat

Re: [9fans] Pipes on UNIX

[Lluís Batlle]

I better explain my problem. :)
I have a CGI for Apache. I wrote it in C, and it simply parses text
from a html textarea, giving another html in output. I made it as a
state machine which outputs and outputs according to the input
characters.

Apache communicates with the CGI process with two pipes linked to the
process' stdin and stdout.
But I think that apache code is like this:
while()
{
  read(socket,buffer)
  write(pipe_process1, buffer)
}
while()
{
  read(pipe_process2, buffer)
  write(socket,buffer)
}

Therefore, if I had a CGI similar to 'cat' (and I have), it would
deadlock due to the pipe_process2 not being read by apache if it gets
filled, the process blocks, and then pipe_process1 gets filled.

As I want the CGI to accept input from almost any length, I need a
"pipe buffer" big enough.

Then, based on Gorka suggestion (which I didn't understand as a
9pserver ;), I finally agreed on the easiest solution wrapping the CGI
with a sh script:
#!/bin/sh
head -n 10000000 > inputfile   # limiting the disk ussage to 10MB, raw cutting.
mycgi < inputfile
#EOF

That's all. I will not write any "superpipe" program, or any 9pserver,
because at the end I need the disk. The sh wrapper works well enough.

Thank you for your help in this non-plan9-related problem! Simply the
unicon documentation pointed to something strange in named pipes which
I couldn't check anywhere. :)

Regards,
Lluís

2007/9/6, Gorka Guardiola <paurea@gmail.com>:
> I wrote it without coffee... would be more like:
>
> generate_data > input &
> data_eater < output
>
> :-).
>
> And input and output can come from a stupid fileserver like pipefs...
> or pipebackedbyaharddiskfs :-).
> --
> - curiosity sKilled the cat
>

Re: [9fans] Pipes on UNIX

[erik quanstrom]

> I better explain my problem. :)
> I have a CGI for Apache. I wrote it in C, and it simply parses text
> from a html textarea, giving another html in output. I made it as a
> state machine which outputs and outputs according to the input
> characters.
>
> Apache communicates with the CGI process with two pipes linked to the
> process' stdin and stdout.
> But I think that apache code is like this:
> while()
> {
>   read(socket,buffer)
>   write(pipe_process1, buffer)
> }
> while()
> {
>   read(pipe_process2, buffer)
>   write(socket,buffer)
> }
>
> Therefore, if I had a CGI similar to 'cat' (and I have), it would
> deadlock due to the pipe_process2 not being read by apache if it gets
> filled, the process blocks, and then pipe_process1 gets filled.

no, you wouldn't.

i believe the problem is the cgi and apache disagree on the termination
condition.  make sure you close the fd 0 when you are done reading.

- erik

Re: [9fans] Pipes on UNIX

[Lluís Batlle]

2007/9/6, erik quanstrom <quanstro@quanstro.net>:
> > Therefore, if I had a CGI similar to 'cat' (and I have), it would
> > deadlock due to the pipe_process2 not being read by apache if it gets
> > filled, the process blocks, and then pipe_process1 gets filled.
>
> no, you wouldn't.
>
> i believe the problem is the cgi and apache disagree on the termination
> condition.  make sure you close the fd 0 when you are done reading.
I used gdb, and my process doesn't reach at all the end of reading. It
gets blocked writting to stdout. The system input TCP queue to apache
is still filled with 70KB of data in the test I did. And the browser
didn't receive any byte. And I know my program processed an amount of
data (it makes the hard disk pain a bit, so I can 'hear' when it does
something).

Re: [9fans] Pipes on UNIX

[Charles Forsyth]

> Thank you for your help in this non-plan9-related problem! Simply the
> unicon documentation pointed to something strange in named pipes which
> I couldn't check anywhere. :)

i couldn't work out where they got that impression or information.
i'm sure i've never seen that in any of the unix-y implementations i've read.