Re: [TUHS] First appearance of named pipes

Re: [TUHS] First appearance of named pipes

[Noel Chiappa]

    > From: Paul Ruizendaal

    > The paper is from late 1981. ... When did FIFO's become a
    > standard Unix feature?

Err, V4? :-) At least, that's when pipes arrived (I think - we don't have V4
sources, but there are indications that's when they appeared), and a pipe is a
FIFO. RAND ports just allowed (effectively) a pipe to have a name in the file
system.

The implementation of both is pretty straight-forward. A pipe is just a file
which has a maximum length, after which the writer is blocked. A port is
just a pipe (it uses the pipe code) whose inode appears in the file system.

    > From: Clem Cole

    > I think the code is on one of the 'USENIX' tapes in Warren's archives.

Doc is here:

  https://minnie.tuhs.org//cgi-bin/utree.pl?file=BBN-V6/doc/ipc

and sources for all that are here:

 https://minnie.tuhs.org//cgi-bin/utree.pl?file=BBN-V6/dmr
 https://minnie.tuhs.org//cgi-bin/utree.pl?file=BBN-V6/ken

(port.c is in 'dmr', not 'ken'where it should be).

	Noel

Re: [TUHS] First appearance of named pipes

[Paul Ruizendaal]

Thanks for the feedback, all.

Rand ports were done in 1977 by Sunshine/Zucker. I’ve only come across Rand Ports in the context of V6 and the Arpa crowd 1977-1981. I’ve never seen a reference to Rand Ports on V7 or later. This of course does not mean that it did not exist.

I’ve dug further, and it would seem that named pipes under the name ‘fifo’ appeared first in SysIII (1980). That matches with Luderer’s remark. It does not seem to exist in the Research editions. It only appears in BSD in the Reno release, 1990. All in all, it would seem that ‘fifo’s were a SysV thing for most of the 80’s, with the BSD lineage using domain sockets instead (as Clem mentioned).

Interestingly, Luderer also refers to a 1978 paper by Steve Holmgren (one of the Arpa Unix authors), suggesting ’sockets’ (in today’s parlance) for interproces communication.

Paul

PS really nobody on the list recalls Luderer's (et al.) distributed Unix and how it related to other work ??


> On 6 Mar 2020, at 23:44, Noel Chiappa <jnc@mercury.lcs.mit.edu> wrote:
> 
> 
>> From: Paul Ruizendaal
> 
>> The paper is from late 1981. ... When did FIFO's become a
>> standard Unix feature?
> 
> Err, V4? :-) At least, that's when pipes arrived (I think - we don't have V4
> sources, but there are indications that's when they appeared), and a pipe is a
> FIFO. RAND ports just allowed (effectively) a pipe to have a name in the file
> system.
> 
> The implementation of both is pretty straight-forward. A pipe is just a file
> which has a maximum length, after which the writer is blocked. A port is
> just a pipe (it uses the pipe code) whose inode appears in the file system.
> 
>> From: Clem Cole
> 
>> I think the code is on one of the 'USENIX' tapes in Warren's archives.
> 
> Doc is here:
> 
>  https://minnie.tuhs.org//cgi-bin/utree.pl?file=BBN-V6/doc/ipc
> 
> and sources for all that are here:
> 
> https://minnie.tuhs.org//cgi-bin/utree.pl?file=BBN-V6/dmr
> https://minnie.tuhs.org//cgi-bin/utree.pl?file=BBN-V6/ken
> 
> (port.c is in 'dmr', not 'ken'where it should be).
> 
> 	Noel

Re: [TUHS] First appearance of named pipes

[Clem Cole]

[-- Attachment #1: Type: text/plain, Size: 2843 bytes --]

below.. in-line

On Sat, Mar 7, 2020 at 7:18 AM Paul Ruizendaal <pnr@planet.nl> wrote:

> Thanks for the feedback, all.
>
> Rand ports were done in 1977 by Sunshine/Zucker. I’ve only come across
> Rand Ports in the context of V6 and the Arpa crowd 1977-1981. I’ve never
> seen a reference to Rand Ports on V7 or later. This of course does not mean
> that it did not exist.
>
Steve Glaser hacked was playing with them at Tektronix in 1979, as he had
put them into their V6 system before I got there IIRC.   I switched the
user code to use Chesson's MPX in V7, which is why I think they never were
used much in V7.

I've forgotten what Bruce used for UNET - I'm CC'ing him here, hoping to
jog his memory.,




>
> I’ve dug further, and it would seem that named pipes under the name ‘fifo’
> appeared first in SysIII (1980). That matches with Luderer’s remark. It
> does not seem to exist in the Research editions. It only appears in BSD in
> the Reno release, 1990. All in all, it would seem that ‘fifo’s were a SysV
> thing for most of the 80’s, with the BSD lineage using domain sockets
> instead (as Clem mentioned).
>

Yes, that's right.  And if you were someone like Masscomp or Pyramid trying
to thread both systems, we had both in our kernels.



>
> Interestingly, Luderer also refers to a 1978 paper by Steve Holmgren (one
> of the Arpa Unix authors), suggesting ’sockets’ (in today’s parlance) for
> interproces communication.
>
> Paul
>
> PS really nobody on the list recalls Luderer's (et al.) distributed Unix
> and how it related to other work ??
>
>
> > On 6 Mar 2020, at 23:44, Noel Chiappa <jnc@mercury.lcs.mit.edu> wrote:
> >
> >
> >> From: Paul Ruizendaal
> >
> >> The paper is from late 1981. ... When did FIFO's become a
> >> standard Unix feature?
> >
> > Err, V4? :-) At least, that's when pipes arrived (I think - we don't
> have V4
> > sources, but there are indications that's when they appeared), and a
> pipe is a
> > FIFO. RAND ports just allowed (effectively) a pipe to have a name in the
> file
> > system.
> >
> > The implementation of both is pretty straight-forward. A pipe is just a
> file
> > which has a maximum length, after which the writer is blocked. A port is
> > just a pipe (it uses the pipe code) whose inode appears in the file
> system.
> >
> >> From: Clem Cole
> >
> >> I think the code is on one of the 'USENIX' tapes in Warren's archives.
> >
> > Doc is here:
> >
> >  https://minnie.tuhs.org//cgi-bin/utree.pl?file=BBN-V6/doc/ipc
> >
> > and sources for all that are here:
> >
> > https://minnie.tuhs.org//cgi-bin/utree.pl?file=BBN-V6/dmr
> > https://minnie.tuhs.org//cgi-bin/utree.pl?file=BBN-V6/ken
> >
> > (port.c is in 'dmr', not 'ken'where it should be).
> >
> >       Noel
>
>

[-- Attachment #2: Type: text/html, Size: 4566 bytes --]

Re: [TUHS] First appearance of named pipes

[Derek Fawcus]

On Sat, Mar 07, 2020 at 01:17:09PM +0100, Paul Ruizendaal wrote:
> 
> Interestingly, Luderer also refers to a 1978 paper by Steve Holmgren (one of the Arpa Unix authors), suggesting ’sockets’ (in today’s parlance) for interproces communication.

Could that simply be bleed over of terminology from the ARPAnet / Internet
usage, in that "socket" is used to refer to protocol end points?

i.e. see these from 1970:

  https://tools.ietf.org/html/rfc54
  https://tools.ietf.org/html/rfc55
  https://tools.ietf.org/html/rfc60

DF

Re: [TUHS] First appearance of named pipes

[Rob Pike]

Always bemused me that to get a named local I/O connection one ended
up with "Unix domain (what does that even mean?) sockets" rather than
named pipes, especially since sockets are about as natural a Unix
concept as lawn mowers. I've been told, but haven't confirmed, that
early sockets didn't even support read and write. They still don't
support open and close, and never will.

Networks are not intrinsically more special than any other I/O
peripheral, but they have become gilded unicorns mounted on rotating
hovercrafts compared to the I/O devices Unix supported before them.

-rob

On Sun, Mar 8, 2020 at 3:48 AM Derek Fawcus
<dfawcus+lists-tuhs@employees.org> wrote:
>
> On Sat, Mar 07, 2020 at 01:17:09PM +0100, Paul Ruizendaal wrote:
> >
> > Interestingly, Luderer also refers to a 1978 paper by Steve Holmgren (one of the Arpa Unix authors), suggesting ’sockets’ (in today’s parlance) for interproces communication.
>
> Could that simply be bleed over of terminology from the ARPAnet / Internet
> usage, in that "socket" is used to refer to protocol end points?
>
> i.e. see these from 1970:
>
>   https://tools.ietf.org/html/rfc54
>   https://tools.ietf.org/html/rfc55
>   https://tools.ietf.org/html/rfc60
>
> DF

Re: [TUHS] First appearance of named pipes

[Larry McVoy]

I hear you but sockets are here to stay.  Sun tried to get rid of them
by going to a STREAMS networking stack (not saying that was in any way
a better answer, just different).  Didn't work, they had to put sockets
back, there was just way too much software written around the socket API.

I tried to make a more sane interface and never got to something that
handled all the edge cases.

Did Plan 9 make it sane?  If so, care to say how or point me at something
like Masscomp's introduction to network programming?

On Sun, Mar 08, 2020 at 01:36:14PM +1100, Rob Pike wrote:
> Always bemused me that to get a named local I/O connection one ended
> up with "Unix domain (what does that even mean?) sockets" rather than
> named pipes, especially since sockets are about as natural a Unix
> concept as lawn mowers. I've been told, but haven't confirmed, that
> early sockets didn't even support read and write. They still don't
> support open and close, and never will.
> 
> Networks are not intrinsically more special than any other I/O
> peripheral, but they have become gilded unicorns mounted on rotating
> hovercrafts compared to the I/O devices Unix supported before them.
> 
> -rob
> 
> On Sun, Mar 8, 2020 at 3:48 AM Derek Fawcus
> <dfawcus+lists-tuhs@employees.org> wrote:
> >
> > On Sat, Mar 07, 2020 at 01:17:09PM +0100, Paul Ruizendaal wrote:
> > >
> > > Interestingly, Luderer also refers to a 1978 paper by Steve Holmgren (one of the Arpa Unix authors), suggesting ???sockets??? (in today???s parlance) for interproces communication.
> >
> > Could that simply be bleed over of terminology from the ARPAnet / Internet
> > usage, in that "socket" is used to refer to protocol end points?
> >
> > i.e. see these from 1970:
> >
> >   https://tools.ietf.org/html/rfc54
> >   https://tools.ietf.org/html/rfc55
> >   https://tools.ietf.org/html/rfc60
> >
> > DF

-- 
---
Larry McVoy            	     lm at mcvoy.com             http://www.mcvoy.com/lm 

Re: [TUHS] First appearance of named pipes

[Dave Horsfall]

On Sun, 8 Mar 2020, Rob Pike wrote:

> Always bemused me that to get a named local I/O connection one ended up 
> with "Unix domain (what does that even mean?) sockets" rather than named 
> pipes, especially since sockets are about as natural a Unix concept as 
> lawn mowers.

Indeed...

> I've been told, but haven't confirmed, that early sockets didn't even 
> support read and write.

They had their own I/O calls such as send()/recv() (and still do).

> They still don't support open and close, and never will.

Huh; imagine my surprise when I named a function "shutdown()" because
it was called at SIGTERM to clean up...

> Networks are not intrinsically more special than any other I/O 
> peripheral, but they have become gilded unicorns mounted on rotating 
> hovercrafts compared to the I/O devices Unix supported before them.

And that's being polite...  They are the worst interface that I have ever 
seen (including OS/360).  At a previous $ORKPLACE there was a library that 
said simply "I am server on port N" and "I want to contact a service on 
a.b.c.d on port N"; I wish I'd stole^Wborrowed it when we eventually 
parted company.

-- Dave

Re: [TUHS] First appearance of named pipes

[arnold]

Rob Pike <robpike@gmail.com> wrote:

> Networks are not intrinsically more special than any other I/O
> peripheral, but they have become gilded unicorns mounted on rotating
> hovercrafts compared to the I/O devices Unix supported before them.

And another one for the fortune file ... :-)

Arnold

Re: [TUHS] First appearance of named pipes

[Ralph Corderoy]

Hi Larry,

> Did Plan 9 make it sane?  If so, care to say how or point me at something
> like Masscomp's introduction to network programming?

There's Plan 9's dial(2) and friends.
http://man.cat-v.org/plan_9/2/dial
The ctl file of a connection takes commands like ‘hangup’.
http://man.cat-v.org/plan_9/3/ip

The idea of dial() just doing the right thing lives on with Go's
https://golang.org/pkg/net/#pkg-overview

-- 
Cheers, Ralph.

Re: [TUHS] First appearance of named pipes

[arnold]

Ralph Corderoy <ralph@inputplus.co.uk> wrote:

> Hi Larry,
>
> > Did Plan 9 make it sane?  If so, care to say how or point me at something
> > like Masscomp's introduction to network programming?
>
> There's Plan 9's dial(2) and friends.
> http://man.cat-v.org/plan_9/2/dial
> The ctl file of a connection takes commands like ‘hangup’.
> http://man.cat-v.org/plan_9/3/ip
>
> The idea of dial() just doing the right thing lives on with Go's
> https://golang.org/pkg/net/#pkg-overview
>
> -- 
> Cheers, Ralph.

But more than that is that all files are served via the 9P protocol,
even local ones.

This is a great intro to Plan 9: https://www.youtube.com/watch?v=6m3GuoaxRNM

Just over an hour and a quarter, but well worth the time.

Arnold

[TUHS] sockets (was Re: First appearance of named pipes)

[Derek Fawcus]

On Sun, Mar 08, 2020 at 01:36:14PM +1100, Rob Pike wrote:
> Always bemused me that to get a named local I/O connection one ended
> up with "Unix domain (what does that even mean?) sockets" rather than
> named pipes,

Yeah - I always found that a bit weird, having to use socketpair()
to get a bidirectional "pipe".

> I've been told, but haven't confirmed, that
> early sockets didn't even support read and write. They still don't
> support open and close, and never will.

Err - granted on the open; but if my memory serves, close() has been
supported on them ever since I started using them ('87).  Otherwise
the normal fork/dup/close/exec pattern for child processes would not
work.

Now what would have been useful is a way to have distinct fd's for
the local read and write end of (e.g.) a TCP socket - such that one
direction could be closed w/o closing the other.  Or maybe some
fcntl() to dup the bidirectional fd in to a pair of unidirectional
fds.  That way one could dispense with shutdown for closing one
direction, making children and fd passed programs socket agnostic.

DF

Re: [TUHS] First appearance of named pipes

[Grant Taylor via TUHS]

[-- Attachment #1: Type: text/plain, Size: 1584 bytes --]

On 3/6/20 3:44 PM, Noel Chiappa wrote:
> a pipe is a FIFO.

Hum....

Does that mean that I should be able to replace pipes in my shell 
commands with a collections of FIFOs?  I'm not saying that I would want 
to.  Well, at least not beyond an academic exercise.

> RAND ports just allowed (effectively) a pipe to have a name in the 
> file system.

I've got to admit that this statement, particularly the first two words, 
caused me a lot of trouble.  The first few times I read it I kept 
thinking that RAND ported something from another platform / 
architecture.  I now think it effectively means "(the) port (solution) 
from RAND".  Suffice it to say that many of the emails in this thread 
didn't make sense with my first, seemingly incorrect, understanding, and 
made a LOT more sense with my second, seemingly correct, understanding.

> The implementation of both is pretty straight-forward. A pipe is 
> just a file which has a maximum length, after which the writer is 
> blocked. A port is just a pipe (it uses the pipe code) whose inode 
> appears in the file system.

Intriguing.

It is interesting learning the history of things that I've taken for 
granted for 25 years.

This thread aligns quite well with some external reading that I've done 
trying to learn more about file descriptors and redirection.  TL;DR: 
The redirection symbols (syntax) weren't doing what I thought they were 
doing.  I now believe that they change values of variables used for I/O. 
  Much like DD statements in JCL.



-- 
Grant. . . .
unix || die


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4013 bytes --]

Re: [TUHS] sockets (was Re: First appearance of named pipes)

[Grant Taylor via TUHS]

[-- Attachment #1: Type: text/plain, Size: 528 bytes --]

On 3/8/20 9:13 AM, Derek Fawcus wrote:
> Now what would have been useful is a way to have distinct fd's for 
> the local read and write end of (e.g.) a TCP socket - such that 
> one direction could be closed w/o closing the other.

I believe that this can be done, now.  At least I've read that it's 
possible for one end to close (FIN) a TCP connection without the other 
end also closing.  Thus you end up with the one-way data flow that is 
still ACKed the way that TCP does.



-- 
Grant. . . .
unix || die


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4013 bytes --]

Re: [TUHS] sockets (was Re: First appearance of named pipes)

[Larry McVoy]

On Mon, Mar 09, 2020 at 05:22:57PM -0600, Grant Taylor via TUHS wrote:
> On 3/8/20 9:13 AM, Derek Fawcus wrote:
> >Now what would have been useful is a way to have distinct fd's for the
> >local read and write end of (e.g.) a TCP socket - such that one direction
> >could be closed w/o closing the other.
> 
> I believe that this can be done, now.  At least I've read that it's possible
> for one end to close (FIN) a TCP connection without the other end also
> closing.  Thus you end up with the one-way data flow that is still ACKed the
> way that TCP does.

Yes, it can be tricky to do it portably but I have.

Re: [TUHS] First appearance of named pipes

[arnold]

Grant Taylor via TUHS <tuhs@minnie.tuhs.org> wrote:

> On 3/6/20 3:44 PM, Noel Chiappa wrote:
> > a pipe is a FIFO.
>
> Hum....
>
> Does that mean that I should be able to replace pipes in my shell 
> commands with a collections of FIFOs?

Absolutely:

	$ mkfifo the_fifo
	$ ls -l  the_fifo
	prw-rw-r-- 1 arnold arnold 0 Mar 10 09:28 the_fifo
	$ echo foo > the_fifo & sleep 1 ; cat the_fifo 
	[1] 3721
	foo
	[1]+  Done                    echo foo > the_fifo

As you stated, not that you'd want to do that, but you can.

Arnold

Re: [TUHS] First appearance of named pipes

[Chet Ramey]

On 3/9/20 7:22 PM, Grant Taylor via TUHS wrote:
> On 3/6/20 3:44 PM, Noel Chiappa wrote:
>> a pipe is a FIFO.
> 
> Hum....
> 
> Does that mean that I should be able to replace pipes in my shell commands
> with a collections of FIFOs?  

Yes, absolutely. Just like a pipe is a FIFO, a FIFO is just a pipe with
a name that appears in the file system.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
		 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/

Re: [TUHS] First appearance of named pipes

[Dave Horsfall]

On Tue, 10 Mar 2020, Chet Ramey wrote:

> Yes, absolutely. Just like a pipe is a FIFO, a FIFO is just a pipe with 
> a name that appears in the file system.

So they would still be subject to the same 8-block limit?  I haven't 
delved into the finer points of named pipes as I rarely use them.

Related question: are they the same as Unix-domain sockets?  Oddly enough 
I haven't used those much either.

-- Dave

Re: [TUHS] First appearance of named pipes

[Chet Ramey]

On 3/10/20 4:26 PM, Dave Horsfall wrote:
> On Tue, 10 Mar 2020, Chet Ramey wrote:
> 
>> Yes, absolutely. Just like a pipe is a FIFO, a FIFO is just a pipe with a
>> name that appears in the file system.
> 
> So they would still be subject to the same 8-block limit?  I haven't delved
> into the finer points of named pipes as I rarely use them.

I believe that on Linux, at least, they have the same capacity limits
(64K), but you can modify that. I don't know about BSD.

> Related question: are they the same as Unix-domain sockets?  Oddly enough I
> haven't used those much either.

They're similar, I guess ("I am but a shadowy reflection of you."). But I
haven't used unix domain sockets, either.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
		 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/

Re: [TUHS] First appearance of named pipes

[Grant Taylor via TUHS]

[-- Attachment #1: Type: text/plain, Size: 2361 bytes --]

On 3/10/20 1:29 AM, arnold@skeeve.com wrote:
> Absolutely:
> 
> 	$ mkfifo the_fifo
> 	$ ls -l  the_fifo
> 	prw-rw-r-- 1 arnold arnold 0 Mar 10 09:28 the_fifo
> 	$ echo foo > the_fifo & sleep 1 ; cat the_fifo
> 	[1] 3721
> 	foo
> 	[1]+  Done                    echo foo > the_fifo
> 
> As you stated, not that you'd want to do that, but you can.

Thank you for your reply Arnold.

As I was reading your reply, I realized that I did not fully convey the 
question that I was still mulling over in my head.  (More in a moment.)

This thread is one of about three things happening in my life that have 
to do with pipes, FIFOs, and file descriptors.  I managed to articulate 
the simpler of the questions while reading Noel's email.

The larger more onerous question is could I leverage exec to alter where 
file descriptors 0 (STDIN), 1 (STDOUT), and 2 (STDERR) are set to, 
including changing 1 to the value of a FIFO, and 0 of a subsequent 
command to also be the value of the FIFO, thus have pipe like behavior 
between two commands without using a pipe or redirection as in ">".

This has also gotten me to wonder about the possibility of having 
multiple commands output to a file descriptor; 1 / 2 / other, that is 
input to a separate command.  Sort of the opposite of tee, in a manner 
of speaking.  I'll try to articulate:

$ mkfifo test.fifo
$ exec 3>&1
$ exec 1> test.fifo
$ for l in {a..z}; do echo $l; sleep 1; done &
$ for L in {A..Z}; do echo $L; sleep 1; done &
$ for n in {1..100}; do echo $n; sleep 1; done &
$ exec 1>&3
$ cat test.fifo

This seems special to me in that I have three processes (for loops) 
writing into what is effectively the same pipe.

After having mulled this over for a few days and typing this out, I 
realize that the "pipe" is really just a fifo and that in this case the 
fifo is a named pipe on the file system.  I could do the same thing with 
a file.  Historically I would have done the same thing with a file.  But 
now I realize that the file is not required and that I can use a fifo 
which is in memory and never hits the disk.  (Save for creating the name 
interface to the pipe / fifo.)

At least, I think that's all accurate.

I would be very eager to learn from anyone who is willing to teach me 
pointers.  :-)



-- 
Grant. . . .
unix || die


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4013 bytes --]

Re: [TUHS] First appearance of named pipes

[Grant Taylor via TUHS]

[-- Attachment #1: Type: text/plain, Size: 445 bytes --]

On 3/10/20 2:26 PM, Dave Horsfall wrote:
> Related question: are they the same as Unix-domain sockets?

Prior to this thread, I would have said that named pipes / FIFOs are 
unidirectional and that sockets are bi-directional.  But there have been 
a number of things in this thread that call that understanding into 
question.  Now I need to mull things over and get a new working 
understanding.



-- 
Grant. . . .
unix || die


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4013 bytes --]

Re: [TUHS] First appearance of named pipes

[arnold]

Hi Grant.

Grant Taylor via TUHS <tuhs@minnie.tuhs.org> wrote:

> The larger more onerous question is could I leverage exec to alter where 
> file descriptors 0 (STDIN), 1 (STDOUT), and 2 (STDERR) are set to, 
> including changing 1 to the value of a FIFO, and 0 of a subsequent 
> command to also be the value of the FIFO, thus have pipe like behavior 
> between two commands without using a pipe or redirection as in ">".

There's nothing preventing you from doing that. After the fork() and
before the exec(), just close() and dup() the relevant fds in the right
order and you're set.

> This has also gotten me to wonder about the possibility of having 
> multiple commands output to a file descriptor; 1 / 2 / other, that is 
> input to a separate command.  Sort of the opposite of tee, in a manner 
> of speaking.  I'll try to articulate:
>
> $ mkfifo test.fifo
> $ exec 3>&1
> $ exec 1> test.fifo
> $ for l in {a..z}; do echo $l; sleep 1; done &
> $ for L in {A..Z}; do echo $L; sleep 1; done &
> $ for n in {1..100}; do echo $n; sleep 1; done &
> $ exec 1>&3
> $ cat test.fifo

I don't think that this is any different from:

	(for l in {a..z}; do echo $l; sleep 1; done &
	 for L in {A..Z}; do echo $L; sleep 1; done &
	 for n in {1..100}; do echo $n; sleep 1; done &) | cat

which reduces to:

	(for l in {a..z}; do echo $l; sleep 1; done &
	 for L in {A..Z}; do echo $L; sleep 1; done &
	 for n in {1..100}; do echo $n; sleep 1; done &) > /some/file

(You might want to background that whole mess given that the final
pipeline will sleep for 100 seconds.)

> This seems special to me in that I have three processes (for loops) 
> writing into what is effectively the same pipe.

It's not any different than calling `stty -tostop' and then simply
backgrounding the three loops at the terminal.  Try it!

(This is the beauty of the Unix model, an fd is just a data sink,
we don't care where it goes.)

> I would be very eager to learn from anyone who is willing to teach me 
> pointers.  :-)

At the cost of tooting my own horn, I recommend my book "Linux Programming
by Example: The Fundamentals", which, despite the "Linux" in the title,
covers basic Unix programming, including file descriptor manipulation
of the sort under discussion here.

HTH,

Arnold

Re: [TUHS] sockets (was Re: First appearance of named pipes)

[Peter Pentchev]

[-- Attachment #1: Type: text/plain, Size: 1016 bytes --]

On Mon, Mar 09, 2020 at 05:22:57PM -0600, Grant Taylor via TUHS wrote:
> On 3/8/20 9:13 AM, Derek Fawcus wrote:
> > Now what would have been useful is a way to have distinct fd's for the
> > local read and write end of (e.g.) a TCP socket - such that one
> > direction could be closed w/o closing the other.
> 
> I believe that this can be done, now.  At least I've read that it's possible
> for one end to close (FIN) a TCP connection without the other end also
> closing.  Thus you end up with the one-way data flow that is still ACKed the
> way that TCP does.

Yep, in the next sentence Derek mentioned "dispense with shutdown",
meaning the shutdown(2) syscall that does exactly that. What he meant
was, wouldn't it be nice to be able to do that with close(2) instead?

G'luck,
Peter

-- 
Peter Pentchev  roam@{ringlet.net,debian.org,FreeBSD.org} pp@storpool.com
PGP key:        http://people.FreeBSD.org/~roam/roam.key.asc
Key fingerprint 2EE7 A7A5 17FC 124C F115  C354 651E EFB0 2527 DF13

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [TUHS] sockets (was Re: First appearance of named pipes)

[Derek Fawcus]

On Mon, Mar 23, 2020 at 10:49:43AM +0200, Peter Pentchev wrote:
> On Mon, Mar 09, 2020 at 05:22:57PM -0600, Grant Taylor via TUHS wrote:
> > On 3/8/20 9:13 AM, Derek Fawcus wrote:
> > > Now what would have been useful is a way to have distinct fd's for the
> > > local read and write end of (e.g.) a TCP socket - such that one
> > > direction could be closed w/o closing the other.
> > 
> > I believe that this can be done, now.  At least I've read that it's possible
> > for one end to close (FIN) a TCP connection without the other end also
> > closing.  Thus you end up with the one-way data flow that is still ACKed the
> > way that TCP does.
> 
> Yep, in the next sentence Derek mentioned "dispense with shutdown",
> meaning the shutdown(2) syscall that does exactly that. What he meant
> was, wouldn't it be nice to be able to do that with close(2) instead?

Quite.

One point being that one could fork/exec a program with those fd's attached
to stdin/stdout and it could operate as a normal filter, w/o having to
understand it was using a socket.

(i.e. closing stdout [hence triggering a FIN], while still reading from stdin)

Plus various other games achievable by replumbing fd's.

DF

Re: [TUHS] sockets (was Re: First appearance of named pipes)

[Grant Taylor via TUHS]

[-- Attachment #1: Type: text/plain, Size: 646 bytes --]

On 3/24/20 3:47 AM, Derek Fawcus wrote:> One point being that one could 
fork/exec a program with those fd's
> attached to stdin/stdout and it could operate as a normal filter, w/o
> having to understand it was using a socket.

Hum.  This is quite intriguing.

Now I'm wondering what sort of mistchif I could get up to doing things 
with Bash (et al.) supporting /dev/tcp pseudo files.

> (i.e. closing stdout [hence triggering a FIN], while still reading
> from stdin)

I like it.

> Plus various other games achievable by replumbing fd's.

Ya.  There's quite a bit to think about here.



-- 
Grant. . . .
unix || die


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4013 bytes --]

Re: [TUHS] sockets (was Re: First appearance of named pipes)

[Richard Salz]

[-- Attachment #1: Type: text/plain, Size: 200 bytes --]

> One point being that one could fork/exec a program with those fd's
> attached to stdin/stdout and it could operate as a normal filter, w/

Hasn't that pretty much worked ever since BSD wrote inetd?

[-- Attachment #2: Type: text/html, Size: 288 bytes --]

Re: [TUHS] sockets (was Re: First appearance of named pipes)

[Grant Taylor via TUHS]

[-- Attachment #1: Type: text/plain, Size: 323 bytes --]

On 3/25/20 5:47 PM, Richard Salz wrote:
> Hasn't that pretty much worked ever since BSD wrote inetd?

(x)inetd does allow for the incoming connections (and outgoing replies). 
  But I'm not aware of (x)inetd altering anything for programs 
initiating new outbound connections.



-- 
Grant. . . .
unix || die


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4013 bytes --]

Re: [TUHS] sockets (was Re: First appearance of named pipes)

[Richard Salz]

[-- Attachment #1: Type: text/plain, Size: 191 bytes --]

> (x)inetd does allow for the incoming connections (and outgoing replies).
>   But I'm not aware of (x)inetd altering anything for programs
> initiating new outbound connections.
>

netcat ?

[-- Attachment #2: Type: text/html, Size: 437 bytes --]

Re: [TUHS] sockets (was Re: First appearance of named pipes)

[Grant Taylor via TUHS]

[-- Attachment #1: Type: text/plain, Size: 598 bytes --]

On 3/25/20 6:18 PM, Richard Salz wrote:
> netcat ?

Sure.  netcat, et al., do make it possible to initiate outbound connections.

Maybe it's the fog in my brain at the moment, but I don't see how to 
connect netcat's STDIO to another process cleanly.  Maybe I need to 
think more about the following hypothetical example:

  · netcat's STDOUT to grep's STDIN
  · grep's STDOUT to netcat's STDIN

Where netcat listens on a port and grep does some useful work over the 
stream coming through netcat.

Thank you for prodding my brain Richard.



-- 
Grant. . . .
unix || die


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4013 bytes --]

Re: [TUHS] sockets (was Re: First appearance of named pipes)

[Tony Finch]

So I fooled around making an HTTP request from bash...

	(printf 'HEAD / HTTP/1.0\r\nHost: dotat.at\r\n\r\n'; cat 1>&2) \
		0<>/dev/tcp/dotat.at/http 1>&0

And I wondered if it would work with ksh too. It does, but there is weirdness.

Bash's <> operator follows POSIX, so in my one-liner the 0 is redundant.

https://pubs.opengroup.org/onlinepubs/007908799/xcu/chap2.html#tag_001_007_007

However when I look at a ksh man page I find it says

https://github.com/att/ast/blob/master/src/cmd/ksh93/sh.1#L3458

    <>word    Open file word for reading and writing as standard output.

I thought POSIX got features like this from ksh so I'm curious that a
weird little incompatibilty like this has crept in.

(The copy of ksh.1 I have from the CSRG archives lacks <> so I guess that
version was ksh88?)

Tony.
-- 
f.anthony.n.finch  <dot@dotat.at>  http://dotat.at/
Sole: Northerly 6 or 7 at first in west, otherwise easterly or northeasterly 5
or 6. Moderate or rough, occasionally very rough in west. Showers in west.
Good.

Re: [TUHS] sockets (was Re: First appearance of named pipes)

[Dave Horsfall]

[-- Attachment #1: Type: text/plain, Size: 294 bytes --]

On Wed, 25 Mar 2020, Grant Taylor via TUHS wrote:

> · netcat's STDOUT to grep's STDIN
> · grep's STDOUT to netcat's STDIN

Are you trying to set up a loop of processes or something?  I'm not sure 
if that is even possible, although you can't rule out creative uses of 
dup2() etc...

-- Dave

Re: [TUHS] sockets (was Re: First appearance of named pipes)

[Rob Pike]

I can't find it now, but there's a very short rc script that does a
modestly realistic telnet client in Plan 9.

But you know, that's not Unix.

-rob

On Thu, Mar 26, 2020 at 1:39 PM Dave Horsfall <dave@horsfall.org> wrote:
>
> On Wed, 25 Mar 2020, Grant Taylor via TUHS wrote:
>
> > · netcat's STDOUT to grep's STDIN
> > · grep's STDOUT to netcat's STDIN
>
> Are you trying to set up a loop of processes or something?  I'm not sure
> if that is even possible, although you can't rule out creative uses of
> dup2() etc...
>
> -- Dave

Re: [TUHS] sockets (was Re: First appearance of named pipes)

[George Michaelson]

Options negotiation and the URG/PUSH always freaked me out. PAD (the
X.25 equivalent) was a bugger to work with. From memory, the yorkbox
was a forked pair of processes one to read, one to write. it didn't
work very well to be honest. I tended to convert to half-duplex mode
and construct valid lines of input before sending them (which is not
very editor friendly unelss you like Teco, which I didn't since it was
complicated. I stuck to SOS and ed)

Telnet is pretty much just "read and write for networks" except for
the options. Back in the days of the BBN Butterfly, the gethostaddr()
table for de interwebz was a linear list, and UCL was at the back of
the hosts.txt sort and the time it took the daemon to work out who we
were, for a login: prompt, was 1-2 sec close to the 30 second
drop-link-he's-dead-jim timer in getty or whatever it was then. Sad. I
think we made a lot of drama about read and write for networks.
Really, asynchronous communicating processes is a lot of fun. I went
to Milners lectures on the calculus of communicating systems, it was
also too hard, I lost it.

That was also when the real underlying routing (pre BGP) was a
push-down list, which dropped our routes because we were the boring
british side of things and LRU cache said no.  Rebooting the right
fuzzball or BBN box generall brought it back.

X25 was a good fit for PAD. small packets. Enter the ATM cell size
discussion <----here.

On Thu, Mar 26, 2020 at 1:09 PM Rob Pike <robpike@gmail.com> wrote:
>
> I can't find it now, but there's a very short rc script that does a
> modestly realistic telnet client in Plan 9.
>
> But you know, that's not Unix.
>
> -rob
>
> On Thu, Mar 26, 2020 at 1:39 PM Dave Horsfall <dave@horsfall.org> wrote:
> >
> > On Wed, 25 Mar 2020, Grant Taylor via TUHS wrote:
> >
> > > · netcat's STDOUT to grep's STDIN
> > > · grep's STDOUT to netcat's STDIN
> >
> > Are you trying to set up a loop of processes or something?  I'm not sure
> > if that is even possible, although you can't rule out creative uses of
> > dup2() etc...
> >
> > -- Dave

Re: [TUHS] sockets (was Re: First appearance of named pipes)

[Grant Taylor via TUHS]

[-- Attachment #1: Type: text/plain, Size: 812 bytes --]

On 3/25/20 8:38 PM, Dave Horsfall wrote:
> Are you trying to set up a loop of processes or something?

Nope.  I'll try another way

           +------------------+   +-------------+
--(TCP)-->+ socket    stdout +---+ stdin       |
           |        nc        |   |        grep |
<--(TCP)--| socket     stdin +---+ stdout      |
           +------------------+   +-------------+

This example is back to the functionality that (x)inetd would provide.

The idea being that grep* would act on the data that came in the TCP 
connection and send the matching lines out the same TCP connection.

*grep is just a hypothetical example here.

> I'm not sure if that is even possible, although you can't rule out 
> creative uses of dup2() etc...

~chuckle~



-- 
Grant. . . .
unix || die


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4013 bytes --]

Re: [TUHS] sockets (was Re: First appearance of named pipes)

[Derek Fawcus]

On Wed, Mar 25, 2020 at 07:47:48PM -0400, Richard Salz wrote:
> > One point being that one could fork/exec a program with those fd's
> > attached to stdin/stdout and it could operate as a normal filter, w/
> 
> Hasn't that pretty much worked ever since BSD wrote inetd?

The socket fd is dup'ed and then applied to stdin & stdout for the child
process.  So if say the child closes stdout, stdin still has the underlying
'file' open in both read and write mode, hence the reader at the far end
will not receive any form of 'close' notification (e.g. EOF on read).

If a socket fd could be split (or partially dup'ed) in to new read-only
and write-only fds, and the original fd then closed, that 'issue' would
go away.  Obviously it hasn't been much of an issue...

DF

Re: [TUHS] sockets (was Re: First appearance of named pipes)

[Peter Pentchev]

[-- Attachment #1: Type: text/plain, Size: 1261 bytes --]

On Thu, Mar 26, 2020 at 01:38:17PM +1100, Dave Horsfall wrote:
> On Wed, 25 Mar 2020, Grant Taylor via TUHS wrote:
> 
> > · netcat's STDOUT to grep's STDIN
> > · grep's STDOUT to netcat's STDIN
> 
> Are you trying to set up a loop of processes or something?  I'm not sure if
> that is even possible, although you can't rule out creative uses of dup2()
> etc...

This can't really be done with netcat, but it's quite easy to do with
socat; here's an example with a trivial program that reads lines from
its standard input and writes a single line to its standard output:

    [roam@straylight ~]$ socat -v tcp4:nimbus.fccf.net:25 exec:./heysmtp.py
    > 2020/03/28 13:09:04.005497  length=48 from=0 to=47
    220 nimbus.fccf.net ESMTP Postfix (Debian/GNU)\r
    < 2020/03/28 13:09:04.018931  length=6 from=0 to=5
    QUIT\r
    > 2020/03/28 13:09:04.035387  length=15 from=48 to=62
    221 2.0.0 Bye\r
    [roam@straylight ~]$

All the output was actually from socat because of the "-v" option
specified.

G'luck,
Peter

-- 
Peter Pentchev  roam@{ringlet.net,debian.org,FreeBSD.org} pp@storpool.com
PGP key:        http://people.FreeBSD.org/~roam/roam.key.asc
Key fingerprint 2EE7 A7A5 17FC 124C F115  C354 651E EFB0 2527 DF13

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [TUHS] sockets (was Re: First appearance of named pipes)

[Peter Pentchev]

[-- Attachment #1: Type: text/plain, Size: 1606 bytes --]

On Sat, Mar 28, 2020 at 01:12:35PM +0200, Peter Pentchev wrote:
> On Thu, Mar 26, 2020 at 01:38:17PM +1100, Dave Horsfall wrote:
> > On Wed, 25 Mar 2020, Grant Taylor via TUHS wrote:
> > 
> > > · netcat's STDOUT to grep's STDIN
> > > · grep's STDOUT to netcat's STDIN
> > 
> > Are you trying to set up a loop of processes or something?  I'm not sure if
> > that is even possible, although you can't rule out creative uses of dup2()
> > etc...
> 
> This can't really be done with netcat, but it's quite easy to do with
> socat; here's an example with a trivial program that reads lines from
> its standard input and writes a single line to its standard output:
> 
>     [roam@straylight ~]$ socat -v tcp4:nimbus.fccf.net:25 exec:./heysmtp.py
>     > 2020/03/28 13:09:04.005497  length=48 from=0 to=47
>     220 nimbus.fccf.net ESMTP Postfix (Debian/GNU)\r
>     < 2020/03/28 13:09:04.018931  length=6 from=0 to=5
>     QUIT\r
>     > 2020/03/28 13:09:04.035387  length=15 from=48 to=62
>     221 2.0.0 Bye\r
>     [roam@straylight ~]$
> 
> All the output was actually from socat because of the "-v" option
> specified.

...but, of course, this is still not what Derek was talking about
earlier - there is no separation of the file descriptors connected to
the socket: closing the stdout one would not result in a FIN being sent
along the line.

G'luck,
Peter

-- 
Peter Pentchev  roam@{ringlet.net,debian.org,FreeBSD.org} pp@storpool.com
PGP key:        http://people.FreeBSD.org/~roam/roam.key.asc
Key fingerprint 2EE7 A7A5 17FC 124C F115  C354 651E EFB0 2527 DF13

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [TUHS] sockets (was Re: First appearance of named pipes)

[Steffen Nurpmeso]

Peter Pentchev wrote in
<20200328111428.GA1431416@straylight.m.ringlet.net>:
 |On Sat, Mar 28, 2020 at 01:12:35PM +0200, Peter Pentchev wrote:
 |> On Thu, Mar 26, 2020 at 01:38:17PM +1100, Dave Horsfall wrote:
 |>> On Wed, 25 Mar 2020, Grant Taylor via TUHS wrote:
 |>> 
 |>>> · netcat's STDOUT to grep's STDIN
 |>>> · grep's STDOUT to netcat's STDIN
 |>> 
 |>> Are you trying to set up a loop of processes or something?  I'm \
 |>> not sure if
 |>> that is even possible, although you can't rule out creative uses \
 |>> of dup2()
 |>> etc...
 |> 
 |> This can't really be done with netcat, but it's quite easy to do with
 |> socat; here's an example with a trivial program that reads lines from
 |> its standard input and writes a single line to its standard output:
 |> 
 |>     [roam@straylight ~]$ socat -v tcp4:nimbus.fccf.net:25 exec:./heysmtp.\
 |>     py
  ...

perl(1) has IPC::Open2 for that:

  use IPC::Open2;
  # We use `csop' for hashing
  my $MAILX = 'LC_ALL=C s-nail -#:/';

  sub hash_em{
     die "hash_em: open: $^E"
        unless my $pid = open2 *RFD, *WFD, $MAILX;
     foreach my $e (@ENTS){
        print WFD "csop hash32 $e->{name}\n";
        my $h = <RFD>;
        chomp $h;
        $e->{hash} = $h
     }
     print WFD "x\n";
     waitpid $pid, 0;
  }

  hash_em()

 |...but, of course, this is still not what Derek was talking about
 |earlier - there is no separation of the file descriptors connected to
 |the socket: closing the stdout one would not result in a FIN being sent
 |along the line.

Just wanted to add earlier in the thread that on some systems
shutdown(2) equals close(2).  At least it was like that on Mac OS
X (by then) Snow Leopard.

--steffen
|
|Der Kragenbaer,                The moon bear,
|der holt sich munter           he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

Re: [TUHS] sockets (was Re: First appearance of named pipes)

[Tony Finch]

Norman Wilson <norman@oclsc.org> wrote:
>
> I wish POSIX and Linux and the BSDs would catch up; that
> was only 30 years ago.

After sockets were added to BSD, pipe() was implemented using them. But it
was a slightly different implementation than socketpair().

e.g.
https://minnie.tuhs.org/cgi-bin/utree.pl?file=4.2BSD/usr/src/sys/sys/uipc_syscalls.c

and the amusing comment at the top of
https://minnie.tuhs.org/cgi-bin/utree.pl?file=4.2BSD/usr/src/sys/sys/uipc_pipe.c

But the socket-backed pipes weren't bidirectional even though they could
have been - see this fix in FreeBSD
https://svnweb.freebsd.org/base/head/sys/kern/uipc_syscalls.c?r1=12843&r2=13146

Which did not last long because socket-backed pipe were soon replaced
with a new (bidirectional) implementation
https://svnweb.freebsd.org/base/head/sys/kern/sys_pipe.c?view=log&log_pagestart=200#rev13675

Tony.
-- 
f.anthony.n.finch  <dot@dotat.at>  http://dotat.at/
Ardnamurchan Point to Cape Wrath: South or southwest 4 or 5, increasing 6 to
gale 8, then veering west later. Rough or very rough, occasionally high in
west. Rain or drizzle, then showers. Moderate or good, occasionally poor at
first.

[TUHS] sockets (was Re: First appearance of named pipes)

[Norman Wilson]

Derek Fawcus:

  Yeah - I always found that a bit weird, having to use socketpair()
  to get a bidirectional "pipe".

In the Research system, pipes became bidirectional when
they became streams.  That happened slightly before my
time, but so far as I know it broke absolutely nothing.

Some time in the late 1980s, the System V people wanted
to allow pipes to be streams, but were worried about the
bidirectionality.  They proposed to have a new system call
to make a bidirectional pipe.

I attended a meeting with the relevant programmers and
program manager to find out why they thought pipes couldn't
just be bi-directional, as they had been without fuss in
the Research system for some years.  They agreed with me
that that was how it ought to be; the trouble was that
System V releases all had to pass an official System V
Verification Suite (reasonable enough), and that suite
checked not only that you could read the one pipe file
descriptor and write the other, but that you couldn't
do it the wrong way.

Wait a minute, I said.  I'm pretty sure that's not how
the official System V Interface Description reads.  Anyone
got a current copy handy?

We found one, and we looked, and sure enough, the official
verification suite was wrong.  The specification said
that data written to fd[1] must be readable from fd[0],
but nothing about the other direction: full-duplex pipes
were not required but neither were they outlawed!

The programming group was delighted: I'd given them the
ammo they needed to do it right (make pipes streams, and
make them full-duplex by default).  I believe that is
how it came out, though the only reference I have is
Solaris 10, where the manual page specifically says
that what pipe(2) makes is full-duplex (and a stream).

I wish POSIX and Linux and the BSDs would catch up; that
was only 30 years ago.

Norman Wilson
Toronto ON