> From: Paul Ruizendaal > The paper is from late 1981. ... When did FIFO's become a > standard Unix feature? Err, V4? :-) At least, that's when pipes arrived (I think - we don't have V4 sources, but there are indications that's when they appeared), and a pipe is a FIFO. RAND ports just allowed (effectively) a pipe to have a name in the file system. The implementation of both is pretty straight-forward. A pipe is just a file which has a maximum length, after which the writer is blocked. A port is just a pipe (it uses the pipe code) whose inode appears in the file system. > From: Clem Cole > I think the code is on one of the 'USENIX' tapes in Warren's archives. Doc is here: https://minnie.tuhs.org//cgi-bin/utree.pl?file=BBN-V6/doc/ipc and sources for all that are here: https://minnie.tuhs.org//cgi-bin/utree.pl?file=BBN-V6/dmr https://minnie.tuhs.org//cgi-bin/utree.pl?file=BBN-V6/ken (port.c is in 'dmr', not 'ken'where it should be). Noel
Thanks for the feedback, all.
Rand ports were done in 1977 by Sunshine/Zucker. I’ve only come across Rand Ports in the context of V6 and the Arpa crowd 1977-1981. I’ve never seen a reference to Rand Ports on V7 or later. This of course does not mean that it did not exist.
I’ve dug further, and it would seem that named pipes under the name ‘fifo’ appeared first in SysIII (1980). That matches with Luderer’s remark. It does not seem to exist in the Research editions. It only appears in BSD in the Reno release, 1990. All in all, it would seem that ‘fifo’s were a SysV thing for most of the 80’s, with the BSD lineage using domain sockets instead (as Clem mentioned).
Interestingly, Luderer also refers to a 1978 paper by Steve Holmgren (one of the Arpa Unix authors), suggesting ’sockets’ (in today’s parlance) for interproces communication.
Paul
PS really nobody on the list recalls Luderer's (et al.) distributed Unix and how it related to other work ??
> On 6 Mar 2020, at 23:44, Noel Chiappa <jnc@mercury.lcs.mit.edu> wrote:
>
>
>> From: Paul Ruizendaal
>
>> The paper is from late 1981. ... When did FIFO's become a
>> standard Unix feature?
>
> Err, V4? :-) At least, that's when pipes arrived (I think - we don't have V4
> sources, but there are indications that's when they appeared), and a pipe is a
> FIFO. RAND ports just allowed (effectively) a pipe to have a name in the file
> system.
>
> The implementation of both is pretty straight-forward. A pipe is just a file
> which has a maximum length, after which the writer is blocked. A port is
> just a pipe (it uses the pipe code) whose inode appears in the file system.
>
>> From: Clem Cole
>
>> I think the code is on one of the 'USENIX' tapes in Warren's archives.
>
> Doc is here:
>
> https://minnie.tuhs.org//cgi-bin/utree.pl?file=BBN-V6/doc/ipc
>
> and sources for all that are here:
>
> https://minnie.tuhs.org//cgi-bin/utree.pl?file=BBN-V6/dmr
> https://minnie.tuhs.org//cgi-bin/utree.pl?file=BBN-V6/ken
>
> (port.c is in 'dmr', not 'ken'where it should be).
>
> Noel
[-- Attachment #1: Type: text/plain, Size: 2843 bytes --] below.. in-line On Sat, Mar 7, 2020 at 7:18 AM Paul Ruizendaal <pnr@planet.nl> wrote: > Thanks for the feedback, all. > > Rand ports were done in 1977 by Sunshine/Zucker. I’ve only come across > Rand Ports in the context of V6 and the Arpa crowd 1977-1981. I’ve never > seen a reference to Rand Ports on V7 or later. This of course does not mean > that it did not exist. > Steve Glaser hacked was playing with them at Tektronix in 1979, as he had put them into their V6 system before I got there IIRC. I switched the user code to use Chesson's MPX in V7, which is why I think they never were used much in V7. I've forgotten what Bruce used for UNET - I'm CC'ing him here, hoping to jog his memory., > > I’ve dug further, and it would seem that named pipes under the name ‘fifo’ > appeared first in SysIII (1980). That matches with Luderer’s remark. It > does not seem to exist in the Research editions. It only appears in BSD in > the Reno release, 1990. All in all, it would seem that ‘fifo’s were a SysV > thing for most of the 80’s, with the BSD lineage using domain sockets > instead (as Clem mentioned). > Yes, that's right. And if you were someone like Masscomp or Pyramid trying to thread both systems, we had both in our kernels. > > Interestingly, Luderer also refers to a 1978 paper by Steve Holmgren (one > of the Arpa Unix authors), suggesting ’sockets’ (in today’s parlance) for > interproces communication. > > Paul > > PS really nobody on the list recalls Luderer's (et al.) distributed Unix > and how it related to other work ?? > > > > On 6 Mar 2020, at 23:44, Noel Chiappa <jnc@mercury.lcs.mit.edu> wrote: > > > > > >> From: Paul Ruizendaal > > > >> The paper is from late 1981. ... When did FIFO's become a > >> standard Unix feature? > > > > Err, V4? :-) At least, that's when pipes arrived (I think - we don't > have V4 > > sources, but there are indications that's when they appeared), and a > pipe is a > > FIFO. RAND ports just allowed (effectively) a pipe to have a name in the > file > > system. > > > > The implementation of both is pretty straight-forward. A pipe is just a > file > > which has a maximum length, after which the writer is blocked. A port is > > just a pipe (it uses the pipe code) whose inode appears in the file > system. > > > >> From: Clem Cole > > > >> I think the code is on one of the 'USENIX' tapes in Warren's archives. > > > > Doc is here: > > > > https://minnie.tuhs.org//cgi-bin/utree.pl?file=BBN-V6/doc/ipc > > > > and sources for all that are here: > > > > https://minnie.tuhs.org//cgi-bin/utree.pl?file=BBN-V6/dmr > > https://minnie.tuhs.org//cgi-bin/utree.pl?file=BBN-V6/ken > > > > (port.c is in 'dmr', not 'ken'where it should be). > > > > Noel > > [-- Attachment #2: Type: text/html, Size: 4566 bytes --]
On Sat, Mar 07, 2020 at 01:17:09PM +0100, Paul Ruizendaal wrote: > > Interestingly, Luderer also refers to a 1978 paper by Steve Holmgren (one of the Arpa Unix authors), suggesting ’sockets’ (in today’s parlance) for interproces communication. Could that simply be bleed over of terminology from the ARPAnet / Internet usage, in that "socket" is used to refer to protocol end points? i.e. see these from 1970: https://tools.ietf.org/html/rfc54 https://tools.ietf.org/html/rfc55 https://tools.ietf.org/html/rfc60 DF
Always bemused me that to get a named local I/O connection one ended
up with "Unix domain (what does that even mean?) sockets" rather than
named pipes, especially since sockets are about as natural a Unix
concept as lawn mowers. I've been told, but haven't confirmed, that
early sockets didn't even support read and write. They still don't
support open and close, and never will.
Networks are not intrinsically more special than any other I/O
peripheral, but they have become gilded unicorns mounted on rotating
hovercrafts compared to the I/O devices Unix supported before them.
-rob
On Sun, Mar 8, 2020 at 3:48 AM Derek Fawcus
<dfawcus+lists-tuhs@employees.org> wrote:
>
> On Sat, Mar 07, 2020 at 01:17:09PM +0100, Paul Ruizendaal wrote:
> >
> > Interestingly, Luderer also refers to a 1978 paper by Steve Holmgren (one of the Arpa Unix authors), suggesting ’sockets’ (in today’s parlance) for interproces communication.
>
> Could that simply be bleed over of terminology from the ARPAnet / Internet
> usage, in that "socket" is used to refer to protocol end points?
>
> i.e. see these from 1970:
>
> https://tools.ietf.org/html/rfc54
> https://tools.ietf.org/html/rfc55
> https://tools.ietf.org/html/rfc60
>
> DF
I hear you but sockets are here to stay. Sun tried to get rid of them by going to a STREAMS networking stack (not saying that was in any way a better answer, just different). Didn't work, they had to put sockets back, there was just way too much software written around the socket API. I tried to make a more sane interface and never got to something that handled all the edge cases. Did Plan 9 make it sane? If so, care to say how or point me at something like Masscomp's introduction to network programming? On Sun, Mar 08, 2020 at 01:36:14PM +1100, Rob Pike wrote: > Always bemused me that to get a named local I/O connection one ended > up with "Unix domain (what does that even mean?) sockets" rather than > named pipes, especially since sockets are about as natural a Unix > concept as lawn mowers. I've been told, but haven't confirmed, that > early sockets didn't even support read and write. They still don't > support open and close, and never will. > > Networks are not intrinsically more special than any other I/O > peripheral, but they have become gilded unicorns mounted on rotating > hovercrafts compared to the I/O devices Unix supported before them. > > -rob > > On Sun, Mar 8, 2020 at 3:48 AM Derek Fawcus > <dfawcus+lists-tuhs@employees.org> wrote: > > > > On Sat, Mar 07, 2020 at 01:17:09PM +0100, Paul Ruizendaal wrote: > > > > > > Interestingly, Luderer also refers to a 1978 paper by Steve Holmgren (one of the Arpa Unix authors), suggesting ???sockets??? (in today???s parlance) for interproces communication. > > > > Could that simply be bleed over of terminology from the ARPAnet / Internet > > usage, in that "socket" is used to refer to protocol end points? > > > > i.e. see these from 1970: > > > > https://tools.ietf.org/html/rfc54 > > https://tools.ietf.org/html/rfc55 > > https://tools.ietf.org/html/rfc60 > > > > DF -- --- Larry McVoy lm at mcvoy.com http://www.mcvoy.com/lm
On Sun, 8 Mar 2020, Rob Pike wrote: > Always bemused me that to get a named local I/O connection one ended up > with "Unix domain (what does that even mean?) sockets" rather than named > pipes, especially since sockets are about as natural a Unix concept as > lawn mowers. Indeed... > I've been told, but haven't confirmed, that early sockets didn't even > support read and write. They had their own I/O calls such as send()/recv() (and still do). > They still don't support open and close, and never will. Huh; imagine my surprise when I named a function "shutdown()" because it was called at SIGTERM to clean up... > Networks are not intrinsically more special than any other I/O > peripheral, but they have become gilded unicorns mounted on rotating > hovercrafts compared to the I/O devices Unix supported before them. And that's being polite... They are the worst interface that I have ever seen (including OS/360). At a previous $ORKPLACE there was a library that said simply "I am server on port N" and "I want to contact a service on a.b.c.d on port N"; I wish I'd stole^Wborrowed it when we eventually parted company. -- Dave
Rob Pike <robpike@gmail.com> wrote:
> Networks are not intrinsically more special than any other I/O
> peripheral, but they have become gilded unicorns mounted on rotating
> hovercrafts compared to the I/O devices Unix supported before them.
And another one for the fortune file ... :-)
Arnold
Hi Larry, > Did Plan 9 make it sane? If so, care to say how or point me at something > like Masscomp's introduction to network programming? There's Plan 9's dial(2) and friends. http://man.cat-v.org/plan_9/2/dial The ctl file of a connection takes commands like ‘hangup’. http://man.cat-v.org/plan_9/3/ip The idea of dial() just doing the right thing lives on with Go's https://golang.org/pkg/net/#pkg-overview -- Cheers, Ralph.
Ralph Corderoy <ralph@inputplus.co.uk> wrote: > Hi Larry, > > > Did Plan 9 make it sane? If so, care to say how or point me at something > > like Masscomp's introduction to network programming? > > There's Plan 9's dial(2) and friends. > http://man.cat-v.org/plan_9/2/dial > The ctl file of a connection takes commands like ‘hangup’. > http://man.cat-v.org/plan_9/3/ip > > The idea of dial() just doing the right thing lives on with Go's > https://golang.org/pkg/net/#pkg-overview > > -- > Cheers, Ralph. But more than that is that all files are served via the 9P protocol, even local ones. This is a great intro to Plan 9: https://www.youtube.com/watch?v=6m3GuoaxRNM Just over an hour and a quarter, but well worth the time. Arnold
On Sun, Mar 08, 2020 at 01:36:14PM +1100, Rob Pike wrote: > Always bemused me that to get a named local I/O connection one ended > up with "Unix domain (what does that even mean?) sockets" rather than > named pipes, Yeah - I always found that a bit weird, having to use socketpair() to get a bidirectional "pipe". > I've been told, but haven't confirmed, that > early sockets didn't even support read and write. They still don't > support open and close, and never will. Err - granted on the open; but if my memory serves, close() has been supported on them ever since I started using them ('87). Otherwise the normal fork/dup/close/exec pattern for child processes would not work. Now what would have been useful is a way to have distinct fd's for the local read and write end of (e.g.) a TCP socket - such that one direction could be closed w/o closing the other. Or maybe some fcntl() to dup the bidirectional fd in to a pair of unidirectional fds. That way one could dispense with shutdown for closing one direction, making children and fd passed programs socket agnostic. DF
Derek Fawcus: Yeah - I always found that a bit weird, having to use socketpair() to get a bidirectional "pipe". In the Research system, pipes became bidirectional when they became streams. That happened slightly before my time, but so far as I know it broke absolutely nothing. Some time in the late 1980s, the System V people wanted to allow pipes to be streams, but were worried about the bidirectionality. They proposed to have a new system call to make a bidirectional pipe. I attended a meeting with the relevant programmers and program manager to find out why they thought pipes couldn't just be bi-directional, as they had been without fuss in the Research system for some years. They agreed with me that that was how it ought to be; the trouble was that System V releases all had to pass an official System V Verification Suite (reasonable enough), and that suite checked not only that you could read the one pipe file descriptor and write the other, but that you couldn't do it the wrong way. Wait a minute, I said. I'm pretty sure that's not how the official System V Interface Description reads. Anyone got a current copy handy? We found one, and we looked, and sure enough, the official verification suite was wrong. The specification said that data written to fd[1] must be readable from fd[0], but nothing about the other direction: full-duplex pipes were not required but neither were they outlawed! The programming group was delighted: I'd given them the ammo they needed to do it right (make pipes streams, and make them full-duplex by default). I believe that is how it came out, though the only reference I have is Solaris 10, where the manual page specifically says that what pipe(2) makes is full-duplex (and a stream). I wish POSIX and Linux and the BSDs would catch up; that was only 30 years ago. Norman Wilson Toronto ON
Norman Wilson <norman@oclsc.org> wrote: > > I wish POSIX and Linux and the BSDs would catch up; that > was only 30 years ago. After sockets were added to BSD, pipe() was implemented using them. But it was a slightly different implementation than socketpair(). e.g. https://minnie.tuhs.org/cgi-bin/utree.pl?file=4.2BSD/usr/src/sys/sys/uipc_syscalls.c and the amusing comment at the top of https://minnie.tuhs.org/cgi-bin/utree.pl?file=4.2BSD/usr/src/sys/sys/uipc_pipe.c But the socket-backed pipes weren't bidirectional even though they could have been - see this fix in FreeBSD https://svnweb.freebsd.org/base/head/sys/kern/uipc_syscalls.c?r1=12843&r2=13146 Which did not last long because socket-backed pipe were soon replaced with a new (bidirectional) implementation https://svnweb.freebsd.org/base/head/sys/kern/sys_pipe.c?view=log&log_pagestart=200#rev13675 Tony. -- f.anthony.n.finch <dot@dotat.at> http://dotat.at/ Ardnamurchan Point to Cape Wrath: South or southwest 4 or 5, increasing 6 to gale 8, then veering west later. Rough or very rough, occasionally high in west. Rain or drizzle, then showers. Moderate or good, occasionally poor at first.
[-- Attachment #1: Type: text/plain, Size: 1584 bytes --] On 3/6/20 3:44 PM, Noel Chiappa wrote: > a pipe is a FIFO. Hum.... Does that mean that I should be able to replace pipes in my shell commands with a collections of FIFOs? I'm not saying that I would want to. Well, at least not beyond an academic exercise. > RAND ports just allowed (effectively) a pipe to have a name in the > file system. I've got to admit that this statement, particularly the first two words, caused me a lot of trouble. The first few times I read it I kept thinking that RAND ported something from another platform / architecture. I now think it effectively means "(the) port (solution) from RAND". Suffice it to say that many of the emails in this thread didn't make sense with my first, seemingly incorrect, understanding, and made a LOT more sense with my second, seemingly correct, understanding. > The implementation of both is pretty straight-forward. A pipe is > just a file which has a maximum length, after which the writer is > blocked. A port is just a pipe (it uses the pipe code) whose inode > appears in the file system. Intriguing. It is interesting learning the history of things that I've taken for granted for 25 years. This thread aligns quite well with some external reading that I've done trying to learn more about file descriptors and redirection. TL;DR: The redirection symbols (syntax) weren't doing what I thought they were doing. I now believe that they change values of variables used for I/O. Much like DD statements in JCL. -- Grant. . . . unix || die [-- Attachment #2: S/MIME Cryptographic Signature --] [-- Type: application/pkcs7-signature, Size: 4013 bytes --]
[-- Attachment #1: Type: text/plain, Size: 528 bytes --] On 3/8/20 9:13 AM, Derek Fawcus wrote: > Now what would have been useful is a way to have distinct fd's for > the local read and write end of (e.g.) a TCP socket - such that > one direction could be closed w/o closing the other. I believe that this can be done, now. At least I've read that it's possible for one end to close (FIN) a TCP connection without the other end also closing. Thus you end up with the one-way data flow that is still ACKed the way that TCP does. -- Grant. . . . unix || die [-- Attachment #2: S/MIME Cryptographic Signature --] [-- Type: application/pkcs7-signature, Size: 4013 bytes --]
On Mon, Mar 09, 2020 at 05:22:57PM -0600, Grant Taylor via TUHS wrote:
> On 3/8/20 9:13 AM, Derek Fawcus wrote:
> >Now what would have been useful is a way to have distinct fd's for the
> >local read and write end of (e.g.) a TCP socket - such that one direction
> >could be closed w/o closing the other.
>
> I believe that this can be done, now. At least I've read that it's possible
> for one end to close (FIN) a TCP connection without the other end also
> closing. Thus you end up with the one-way data flow that is still ACKed the
> way that TCP does.
Yes, it can be tricky to do it portably but I have.
Grant Taylor via TUHS <tuhs@minnie.tuhs.org> wrote:
> On 3/6/20 3:44 PM, Noel Chiappa wrote:
> > a pipe is a FIFO.
>
> Hum....
>
> Does that mean that I should be able to replace pipes in my shell
> commands with a collections of FIFOs?
Absolutely:
$ mkfifo the_fifo
$ ls -l the_fifo
prw-rw-r-- 1 arnold arnold 0 Mar 10 09:28 the_fifo
$ echo foo > the_fifo & sleep 1 ; cat the_fifo
[1] 3721
foo
[1]+ Done echo foo > the_fifo
As you stated, not that you'd want to do that, but you can.
Arnold
On 3/9/20 7:22 PM, Grant Taylor via TUHS wrote: > On 3/6/20 3:44 PM, Noel Chiappa wrote: >> a pipe is a FIFO. > > Hum.... > > Does that mean that I should be able to replace pipes in my shell commands > with a collections of FIFOs? Yes, absolutely. Just like a pipe is a FIFO, a FIFO is just a pipe with a name that appears in the file system. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRU chet@case.edu http://tiswww.cwru.edu/~chet/
On Tue, 10 Mar 2020, Chet Ramey wrote:
> Yes, absolutely. Just like a pipe is a FIFO, a FIFO is just a pipe with
> a name that appears in the file system.
So they would still be subject to the same 8-block limit? I haven't
delved into the finer points of named pipes as I rarely use them.
Related question: are they the same as Unix-domain sockets? Oddly enough
I haven't used those much either.
-- Dave
On 3/10/20 4:26 PM, Dave Horsfall wrote: > On Tue, 10 Mar 2020, Chet Ramey wrote: > >> Yes, absolutely. Just like a pipe is a FIFO, a FIFO is just a pipe with a >> name that appears in the file system. > > So they would still be subject to the same 8-block limit? I haven't delved > into the finer points of named pipes as I rarely use them. I believe that on Linux, at least, they have the same capacity limits (64K), but you can modify that. I don't know about BSD. > Related question: are they the same as Unix-domain sockets? Oddly enough I > haven't used those much either. They're similar, I guess ("I am but a shadowy reflection of you."). But I haven't used unix domain sockets, either. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRU chet@case.edu http://tiswww.cwru.edu/~chet/
[-- Attachment #1: Type: text/plain, Size: 2361 bytes --] On 3/10/20 1:29 AM, arnold@skeeve.com wrote: > Absolutely: > > $ mkfifo the_fifo > $ ls -l the_fifo > prw-rw-r-- 1 arnold arnold 0 Mar 10 09:28 the_fifo > $ echo foo > the_fifo & sleep 1 ; cat the_fifo > [1] 3721 > foo > [1]+ Done echo foo > the_fifo > > As you stated, not that you'd want to do that, but you can. Thank you for your reply Arnold. As I was reading your reply, I realized that I did not fully convey the question that I was still mulling over in my head. (More in a moment.) This thread is one of about three things happening in my life that have to do with pipes, FIFOs, and file descriptors. I managed to articulate the simpler of the questions while reading Noel's email. The larger more onerous question is could I leverage exec to alter where file descriptors 0 (STDIN), 1 (STDOUT), and 2 (STDERR) are set to, including changing 1 to the value of a FIFO, and 0 of a subsequent command to also be the value of the FIFO, thus have pipe like behavior between two commands without using a pipe or redirection as in ">". This has also gotten me to wonder about the possibility of having multiple commands output to a file descriptor; 1 / 2 / other, that is input to a separate command. Sort of the opposite of tee, in a manner of speaking. I'll try to articulate: $ mkfifo test.fifo $ exec 3>&1 $ exec 1> test.fifo $ for l in {a..z}; do echo $l; sleep 1; done & $ for L in {A..Z}; do echo $L; sleep 1; done & $ for n in {1..100}; do echo $n; sleep 1; done & $ exec 1>&3 $ cat test.fifo This seems special to me in that I have three processes (for loops) writing into what is effectively the same pipe. After having mulled this over for a few days and typing this out, I realize that the "pipe" is really just a fifo and that in this case the fifo is a named pipe on the file system. I could do the same thing with a file. Historically I would have done the same thing with a file. But now I realize that the file is not required and that I can use a fifo which is in memory and never hits the disk. (Save for creating the name interface to the pipe / fifo.) At least, I think that's all accurate. I would be very eager to learn from anyone who is willing to teach me pointers. :-) -- Grant. . . . unix || die [-- Attachment #2: S/MIME Cryptographic Signature --] [-- Type: application/pkcs7-signature, Size: 4013 bytes --]
[-- Attachment #1: Type: text/plain, Size: 445 bytes --] On 3/10/20 2:26 PM, Dave Horsfall wrote: > Related question: are they the same as Unix-domain sockets? Prior to this thread, I would have said that named pipes / FIFOs are unidirectional and that sockets are bi-directional. But there have been a number of things in this thread that call that understanding into question. Now I need to mull things over and get a new working understanding. -- Grant. . . . unix || die [-- Attachment #2: S/MIME Cryptographic Signature --] [-- Type: application/pkcs7-signature, Size: 4013 bytes --]
Hi Grant. Grant Taylor via TUHS <tuhs@minnie.tuhs.org> wrote: > The larger more onerous question is could I leverage exec to alter where > file descriptors 0 (STDIN), 1 (STDOUT), and 2 (STDERR) are set to, > including changing 1 to the value of a FIFO, and 0 of a subsequent > command to also be the value of the FIFO, thus have pipe like behavior > between two commands without using a pipe or redirection as in ">". There's nothing preventing you from doing that. After the fork() and before the exec(), just close() and dup() the relevant fds in the right order and you're set. > This has also gotten me to wonder about the possibility of having > multiple commands output to a file descriptor; 1 / 2 / other, that is > input to a separate command. Sort of the opposite of tee, in a manner > of speaking. I'll try to articulate: > > $ mkfifo test.fifo > $ exec 3>&1 > $ exec 1> test.fifo > $ for l in {a..z}; do echo $l; sleep 1; done & > $ for L in {A..Z}; do echo $L; sleep 1; done & > $ for n in {1..100}; do echo $n; sleep 1; done & > $ exec 1>&3 > $ cat test.fifo I don't think that this is any different from: (for l in {a..z}; do echo $l; sleep 1; done & for L in {A..Z}; do echo $L; sleep 1; done & for n in {1..100}; do echo $n; sleep 1; done &) | cat which reduces to: (for l in {a..z}; do echo $l; sleep 1; done & for L in {A..Z}; do echo $L; sleep 1; done & for n in {1..100}; do echo $n; sleep 1; done &) > /some/file (You might want to background that whole mess given that the final pipeline will sleep for 100 seconds.) > This seems special to me in that I have three processes (for loops) > writing into what is effectively the same pipe. It's not any different than calling `stty -tostop' and then simply backgrounding the three loops at the terminal. Try it! (This is the beauty of the Unix model, an fd is just a data sink, we don't care where it goes.) > I would be very eager to learn from anyone who is willing to teach me > pointers. :-) At the cost of tooting my own horn, I recommend my book "Linux Programming by Example: The Fundamentals", which, despite the "Linux" in the title, covers basic Unix programming, including file descriptor manipulation of the sort under discussion here. HTH, Arnold
[-- Attachment #1: Type: text/plain, Size: 1016 bytes --] On Mon, Mar 09, 2020 at 05:22:57PM -0600, Grant Taylor via TUHS wrote: > On 3/8/20 9:13 AM, Derek Fawcus wrote: > > Now what would have been useful is a way to have distinct fd's for the > > local read and write end of (e.g.) a TCP socket - such that one > > direction could be closed w/o closing the other. > > I believe that this can be done, now. At least I've read that it's possible > for one end to close (FIN) a TCP connection without the other end also > closing. Thus you end up with the one-way data flow that is still ACKed the > way that TCP does. Yep, in the next sentence Derek mentioned "dispense with shutdown", meaning the shutdown(2) syscall that does exactly that. What he meant was, wouldn't it be nice to be able to do that with close(2) instead? G'luck, Peter -- Peter Pentchev roam@{ringlet.net,debian.org,FreeBSD.org} pp@storpool.com PGP key: http://people.FreeBSD.org/~roam/roam.key.asc Key fingerprint 2EE7 A7A5 17FC 124C F115 C354 651E EFB0 2527 DF13 [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --]
On Mon, Mar 23, 2020 at 10:49:43AM +0200, Peter Pentchev wrote:
> On Mon, Mar 09, 2020 at 05:22:57PM -0600, Grant Taylor via TUHS wrote:
> > On 3/8/20 9:13 AM, Derek Fawcus wrote:
> > > Now what would have been useful is a way to have distinct fd's for the
> > > local read and write end of (e.g.) a TCP socket - such that one
> > > direction could be closed w/o closing the other.
> >
> > I believe that this can be done, now. At least I've read that it's possible
> > for one end to close (FIN) a TCP connection without the other end also
> > closing. Thus you end up with the one-way data flow that is still ACKed the
> > way that TCP does.
>
> Yep, in the next sentence Derek mentioned "dispense with shutdown",
> meaning the shutdown(2) syscall that does exactly that. What he meant
> was, wouldn't it be nice to be able to do that with close(2) instead?
Quite.
One point being that one could fork/exec a program with those fd's attached
to stdin/stdout and it could operate as a normal filter, w/o having to
understand it was using a socket.
(i.e. closing stdout [hence triggering a FIN], while still reading from stdin)
Plus various other games achievable by replumbing fd's.
DF
[-- Attachment #1: Type: text/plain, Size: 646 bytes --] On 3/24/20 3:47 AM, Derek Fawcus wrote:> One point being that one could fork/exec a program with those fd's > attached to stdin/stdout and it could operate as a normal filter, w/o > having to understand it was using a socket. Hum. This is quite intriguing. Now I'm wondering what sort of mistchif I could get up to doing things with Bash (et al.) supporting /dev/tcp pseudo files. > (i.e. closing stdout [hence triggering a FIN], while still reading > from stdin) I like it. > Plus various other games achievable by replumbing fd's. Ya. There's quite a bit to think about here. -- Grant. . . . unix || die [-- Attachment #2: S/MIME Cryptographic Signature --] [-- Type: application/pkcs7-signature, Size: 4013 bytes --]
[-- Attachment #1: Type: text/plain, Size: 200 bytes --] > One point being that one could fork/exec a program with those fd's > attached to stdin/stdout and it could operate as a normal filter, w/ Hasn't that pretty much worked ever since BSD wrote inetd? [-- Attachment #2: Type: text/html, Size: 288 bytes --]
[-- Attachment #1: Type: text/plain, Size: 323 bytes --] On 3/25/20 5:47 PM, Richard Salz wrote: > Hasn't that pretty much worked ever since BSD wrote inetd? (x)inetd does allow for the incoming connections (and outgoing replies). But I'm not aware of (x)inetd altering anything for programs initiating new outbound connections. -- Grant. . . . unix || die [-- Attachment #2: S/MIME Cryptographic Signature --] [-- Type: application/pkcs7-signature, Size: 4013 bytes --]
[-- Attachment #1: Type: text/plain, Size: 191 bytes --] > (x)inetd does allow for the incoming connections (and outgoing replies). > But I'm not aware of (x)inetd altering anything for programs > initiating new outbound connections. > netcat ? [-- Attachment #2: Type: text/html, Size: 437 bytes --]
[-- Attachment #1: Type: text/plain, Size: 598 bytes --] On 3/25/20 6:18 PM, Richard Salz wrote: > netcat ? Sure. netcat, et al., do make it possible to initiate outbound connections. Maybe it's the fog in my brain at the moment, but I don't see how to connect netcat's STDIO to another process cleanly. Maybe I need to think more about the following hypothetical example: · netcat's STDOUT to grep's STDIN · grep's STDOUT to netcat's STDIN Where netcat listens on a port and grep does some useful work over the stream coming through netcat. Thank you for prodding my brain Richard. -- Grant. . . . unix || die [-- Attachment #2: S/MIME Cryptographic Signature --] [-- Type: application/pkcs7-signature, Size: 4013 bytes --]
So I fooled around making an HTTP request from bash... (printf 'HEAD / HTTP/1.0\r\nHost: dotat.at\r\n\r\n'; cat 1>&2) \ 0<>/dev/tcp/dotat.at/http 1>&0 And I wondered if it would work with ksh too. It does, but there is weirdness. Bash's <> operator follows POSIX, so in my one-liner the 0 is redundant. https://pubs.opengroup.org/onlinepubs/007908799/xcu/chap2.html#tag_001_007_007 However when I look at a ksh man page I find it says https://github.com/att/ast/blob/master/src/cmd/ksh93/sh.1#L3458 <>word Open file word for reading and writing as standard output. I thought POSIX got features like this from ksh so I'm curious that a weird little incompatibilty like this has crept in. (The copy of ksh.1 I have from the CSRG archives lacks <> so I guess that version was ksh88?) Tony. -- f.anthony.n.finch <dot@dotat.at> http://dotat.at/ Sole: Northerly 6 or 7 at first in west, otherwise easterly or northeasterly 5 or 6. Moderate or rough, occasionally very rough in west. Showers in west. Good.
[-- Attachment #1: Type: text/plain, Size: 294 bytes --] On Wed, 25 Mar 2020, Grant Taylor via TUHS wrote: > · netcat's STDOUT to grep's STDIN > · grep's STDOUT to netcat's STDIN Are you trying to set up a loop of processes or something? I'm not sure if that is even possible, although you can't rule out creative uses of dup2() etc... -- Dave
I can't find it now, but there's a very short rc script that does a
modestly realistic telnet client in Plan 9.
But you know, that's not Unix.
-rob
On Thu, Mar 26, 2020 at 1:39 PM Dave Horsfall <dave@horsfall.org> wrote:
>
> On Wed, 25 Mar 2020, Grant Taylor via TUHS wrote:
>
> > · netcat's STDOUT to grep's STDIN
> > · grep's STDOUT to netcat's STDIN
>
> Are you trying to set up a loop of processes or something? I'm not sure
> if that is even possible, although you can't rule out creative uses of
> dup2() etc...
>
> -- Dave
Options negotiation and the URG/PUSH always freaked me out. PAD (the
X.25 equivalent) was a bugger to work with. From memory, the yorkbox
was a forked pair of processes one to read, one to write. it didn't
work very well to be honest. I tended to convert to half-duplex mode
and construct valid lines of input before sending them (which is not
very editor friendly unelss you like Teco, which I didn't since it was
complicated. I stuck to SOS and ed)
Telnet is pretty much just "read and write for networks" except for
the options. Back in the days of the BBN Butterfly, the gethostaddr()
table for de interwebz was a linear list, and UCL was at the back of
the hosts.txt sort and the time it took the daemon to work out who we
were, for a login: prompt, was 1-2 sec close to the 30 second
drop-link-he's-dead-jim timer in getty or whatever it was then. Sad. I
think we made a lot of drama about read and write for networks.
Really, asynchronous communicating processes is a lot of fun. I went
to Milners lectures on the calculus of communicating systems, it was
also too hard, I lost it.
That was also when the real underlying routing (pre BGP) was a
push-down list, which dropped our routes because we were the boring
british side of things and LRU cache said no. Rebooting the right
fuzzball or BBN box generall brought it back.
X25 was a good fit for PAD. small packets. Enter the ATM cell size
discussion <----here.
On Thu, Mar 26, 2020 at 1:09 PM Rob Pike <robpike@gmail.com> wrote:
>
> I can't find it now, but there's a very short rc script that does a
> modestly realistic telnet client in Plan 9.
>
> But you know, that's not Unix.
>
> -rob
>
> On Thu, Mar 26, 2020 at 1:39 PM Dave Horsfall <dave@horsfall.org> wrote:
> >
> > On Wed, 25 Mar 2020, Grant Taylor via TUHS wrote:
> >
> > > · netcat's STDOUT to grep's STDIN
> > > · grep's STDOUT to netcat's STDIN
> >
> > Are you trying to set up a loop of processes or something? I'm not sure
> > if that is even possible, although you can't rule out creative uses of
> > dup2() etc...
> >
> > -- Dave
[-- Attachment #1: Type: text/plain, Size: 812 bytes --] On 3/25/20 8:38 PM, Dave Horsfall wrote: > Are you trying to set up a loop of processes or something? Nope. I'll try another way +------------------+ +-------------+ --(TCP)-->+ socket stdout +---+ stdin | | nc | | grep | <--(TCP)--| socket stdin +---+ stdout | +------------------+ +-------------+ This example is back to the functionality that (x)inetd would provide. The idea being that grep* would act on the data that came in the TCP connection and send the matching lines out the same TCP connection. *grep is just a hypothetical example here. > I'm not sure if that is even possible, although you can't rule out > creative uses of dup2() etc... ~chuckle~ -- Grant. . . . unix || die [-- Attachment #2: S/MIME Cryptographic Signature --] [-- Type: application/pkcs7-signature, Size: 4013 bytes --]
On Wed, Mar 25, 2020 at 07:47:48PM -0400, Richard Salz wrote:
> > One point being that one could fork/exec a program with those fd's
> > attached to stdin/stdout and it could operate as a normal filter, w/
>
> Hasn't that pretty much worked ever since BSD wrote inetd?
The socket fd is dup'ed and then applied to stdin & stdout for the child
process. So if say the child closes stdout, stdin still has the underlying
'file' open in both read and write mode, hence the reader at the far end
will not receive any form of 'close' notification (e.g. EOF on read).
If a socket fd could be split (or partially dup'ed) in to new read-only
and write-only fds, and the original fd then closed, that 'issue' would
go away. Obviously it hasn't been much of an issue...
DF
[-- Attachment #1: Type: text/plain, Size: 1261 bytes --] On Thu, Mar 26, 2020 at 01:38:17PM +1100, Dave Horsfall wrote: > On Wed, 25 Mar 2020, Grant Taylor via TUHS wrote: > > > · netcat's STDOUT to grep's STDIN > > · grep's STDOUT to netcat's STDIN > > Are you trying to set up a loop of processes or something? I'm not sure if > that is even possible, although you can't rule out creative uses of dup2() > etc... This can't really be done with netcat, but it's quite easy to do with socat; here's an example with a trivial program that reads lines from its standard input and writes a single line to its standard output: [roam@straylight ~]$ socat -v tcp4:nimbus.fccf.net:25 exec:./heysmtp.py > 2020/03/28 13:09:04.005497 length=48 from=0 to=47 220 nimbus.fccf.net ESMTP Postfix (Debian/GNU)\r < 2020/03/28 13:09:04.018931 length=6 from=0 to=5 QUIT\r > 2020/03/28 13:09:04.035387 length=15 from=48 to=62 221 2.0.0 Bye\r [roam@straylight ~]$ All the output was actually from socat because of the "-v" option specified. G'luck, Peter -- Peter Pentchev roam@{ringlet.net,debian.org,FreeBSD.org} pp@storpool.com PGP key: http://people.FreeBSD.org/~roam/roam.key.asc Key fingerprint 2EE7 A7A5 17FC 124C F115 C354 651E EFB0 2527 DF13 [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --]
[-- Attachment #1: Type: text/plain, Size: 1606 bytes --] On Sat, Mar 28, 2020 at 01:12:35PM +0200, Peter Pentchev wrote: > On Thu, Mar 26, 2020 at 01:38:17PM +1100, Dave Horsfall wrote: > > On Wed, 25 Mar 2020, Grant Taylor via TUHS wrote: > > > > > · netcat's STDOUT to grep's STDIN > > > · grep's STDOUT to netcat's STDIN > > > > Are you trying to set up a loop of processes or something? I'm not sure if > > that is even possible, although you can't rule out creative uses of dup2() > > etc... > > This can't really be done with netcat, but it's quite easy to do with > socat; here's an example with a trivial program that reads lines from > its standard input and writes a single line to its standard output: > > [roam@straylight ~]$ socat -v tcp4:nimbus.fccf.net:25 exec:./heysmtp.py > > 2020/03/28 13:09:04.005497 length=48 from=0 to=47 > 220 nimbus.fccf.net ESMTP Postfix (Debian/GNU)\r > < 2020/03/28 13:09:04.018931 length=6 from=0 to=5 > QUIT\r > > 2020/03/28 13:09:04.035387 length=15 from=48 to=62 > 221 2.0.0 Bye\r > [roam@straylight ~]$ > > All the output was actually from socat because of the "-v" option > specified. ...but, of course, this is still not what Derek was talking about earlier - there is no separation of the file descriptors connected to the socket: closing the stdout one would not result in a FIN being sent along the line. G'luck, Peter -- Peter Pentchev roam@{ringlet.net,debian.org,FreeBSD.org} pp@storpool.com PGP key: http://people.FreeBSD.org/~roam/roam.key.asc Key fingerprint 2EE7 A7A5 17FC 124C F115 C354 651E EFB0 2527 DF13 [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --]
Peter Pentchev wrote in <20200328111428.GA1431416@straylight.m.ringlet.net>: |On Sat, Mar 28, 2020 at 01:12:35PM +0200, Peter Pentchev wrote: |> On Thu, Mar 26, 2020 at 01:38:17PM +1100, Dave Horsfall wrote: |>> On Wed, 25 Mar 2020, Grant Taylor via TUHS wrote: |>> |>>> · netcat's STDOUT to grep's STDIN |>>> · grep's STDOUT to netcat's STDIN |>> |>> Are you trying to set up a loop of processes or something? I'm \ |>> not sure if |>> that is even possible, although you can't rule out creative uses \ |>> of dup2() |>> etc... |> |> This can't really be done with netcat, but it's quite easy to do with |> socat; here's an example with a trivial program that reads lines from |> its standard input and writes a single line to its standard output: |> |> [roam@straylight ~]$ socat -v tcp4:nimbus.fccf.net:25 exec:./heysmtp.\ |> py ... perl(1) has IPC::Open2 for that: use IPC::Open2; # We use `csop' for hashing my $MAILX = 'LC_ALL=C s-nail -#:/'; sub hash_em{ die "hash_em: open: $^E" unless my $pid = open2 *RFD, *WFD, $MAILX; foreach my $e (@ENTS){ print WFD "csop hash32 $e->{name}\n"; my $h = <RFD>; chomp $h; $e->{hash} = $h } print WFD "x\n"; waitpid $pid, 0; } hash_em() |...but, of course, this is still not what Derek was talking about |earlier - there is no separation of the file descriptors connected to |the socket: closing the stdout one would not result in a FIN being sent |along the line. Just wanted to add earlier in the thread that on some systems shutdown(2) equals close(2). At least it was like that on Mac OS X (by then) Snow Leopard. --steffen | |Der Kragenbaer, The moon bear, |der holt sich munter he cheerfully and one by one |einen nach dem anderen runter wa.ks himself off |(By Robert Gernhardt)