[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[Douglas McIlroy]

Many tree and dag pipe notations have been proposed. Marc's was one of
the first. I devised one myself, but didn't like it enough to publish
it even as a TM. A recent one is Spinellis's dgsh. More elaborate
networks with feedback loops evolve for power-series computation. The
idea of implementing cellular automata as arrays of processes with
nearest neighbors connected by pipes was suggested early on, but I've
never seen such an arrangement implemented--it would be hideously
slow.

I once wrote a general plumber that worked from a connection list:
connect fd m in process A to fd n in process B. The main challenge was
to find an order of hooking things up so that the number of live file
descriptors in the plumber didn't exceed the system limit.

Doug

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[Ron Natalie]

[-- Attachment #1: Type: text/plain, Size: 1122 bytes --]

NO, the 11/23 has an MMU and was perflectly capable of running  UNIX.   
The 23+ had the extra  address lines  allowing it to address 4 MB like 
the 11/70.


------ Original Message ------
From "Adam Thornton" <athornton@gmail.com>
To "Ron Natalie" <ron@ronnatalie.com>; "The Eunuchs Hysterical Society" 
<tuhs@tuhs.org>
Date 12/5/2024 9:29:24 PM
Subject Re: [TUHS] Re: Pipes (was Re: After 50 years, what has the 
Impact of Unix been?)

>/23+, no?  The basic 23 doesn't have an MMU, I'm pretty sure.
>
>On Thu, Dec 5, 2024 at 11:58 AM Ron Natalie <ron@ronnatalie.com> wrote:
>>I remember on MiniUnix (a stripped down kernel that would run on 
>>PDP11’s
>>without memory management, in our case 11/03, 11/20, and an older 
>>11/40)
>>there were no kernel pipes (and limited process concurrency).
>>The shell used the aforementioned emulation of pipe by just running 
>>the
>>output of one process into a temporary file subsequently loaded as 
>>stdin
>>of the other.
>>
>>We kind of put all that to bed when the 11/23s came out and we could 
>>run
>>our full up kernel on the micros.
>>

[-- Attachment #2: Type: text/html, Size: 2942 bytes --]

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[Chet Ramey via TUHS]

[-- Attachment #1.1: Type: text/plain, Size: 441 bytes --]

On 12/5/24 7:46 PM, Alexis wrote:

> Can anyone point me to any discussions about process substitution 
> potentially becoming part of POSIX? 

It's never come up in any serious way, maybe as a wish list item for
some people on the mailing list.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
		 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 203 bytes --]

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[arnold]

Dan Cross <crossd@gmail.com> wrote:

> My claim is simply that extant shell syntax isn't really amenable to
> this in a natural way. As Chet and Arnold pointed out, process
> substitution as pioneered in ksh gets us a little closer, but it's not
> quite as general (I believe any such graphs would have to be acyclic);
> it's certainly not as syntactically pleasant.
>
> There has been work along these lines; I was sent a reference off-list
> to a paper by Spinellis and Fragkoulis about a DAG-oriented shell:
> https://www.spinellis.gr/sw/dgsh/, which seems relevant.

In the early-to-mid 1980s, the Georgia Tech Software Tools Subsystem
for Prime Computers offered three standard inputs, three standard
outputs, and a shell with syntax to connect things as you desired.
I don't remember the syntax though.  I also don't know how much
this feature was really used.

Arnold

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[arnold]

"Greg A. Woods" <woods@robohack.ca> wrote:

> > Are you sure? I think Tom Duff originated it in his "rc" shell.
>
> The Wikipedia entry for "Process substitution" says:
>
> 	Process substitution was available as a compile-time option for
> 	ksh88, the 1988 version of the KornShell from Bell Labs.  The rc
> 	shell provides the feature as "pipeline branching" in Version 10
> 	Unix, released in 1990.
>
> So, if we believe that then indeed ksh88 pioneered process substitution.

I'd forgotten that it was optional in ksh88. It was there out of
the box in ksh93.

Arnold

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[Diomidis Spinellis]

On 06-Dec-24 01:07, Marc Rochkind wrote:
> I found that 2017 paper "Extending Unix Pipelines to DAGs". It's open 
> access:
> 
> https://ieeexplore.ieee.org/document/7903579 <https:// 
> ieeexplore.ieee.org/document/7903579>
> 
> The open source code itself is here: https://github.com/dspinellis/dgsh 
> <https://github.com/dspinellis/dgsh>
> 
> Maybe an ambitious TUHS contributor can get the code running and give us 
> a report.

I wrote the dgsh code with my co-author Marios Fragkoulis, so I still 
have it running.  Doug McIlroy, who also mentioned dgsh in another 
message, is too modest to say that its design owes much to his input.  I 
asked him for feedback when I was working on it, and over several 
iterations he proposed important (and quite demanding as I recall) 
improvements to its design.

The system allows the concise and readable expression of several graph 
topologies I had in mind when I started working on it, and more [1]. 
However, it hasn't caught on.  I think the main reason is that it is 
based on modified versions of several existing tools (bash, cmp, comm, 
cut, diff, diff3, grep, join, paste, perm, sort) [2].  The modifications 
allow the tools to coordinate between them the setup of pipes when 
placed in a dgsh graph according to available inputs and required 
outputs.  The changes (especially for bash) aren't small, which meant 
that I didn't think it was realistic to push them upstream, which now 
means that the modified tools are out of date and difficult to build. 
Not sure what can be done to address this problem.  It seems that a 
widely adopted system, such as modern Unix/Linux, has too much inertia 
for it to adopt potentially disrupting innovations.

In retrospect, the way we designed the pipe graph setup could also be 
improved.  The current design involves an initial phase where IPC 
messages are circulating around the graph to communicate the I/O 
requirements of each tool, for example that comm(1) should expect input 
from two processes and output to three processes.  The design is brittle 
and difficult to troubleshoot, because coordination happens dynamically 
behind the scenes.  A better design (and one I think Doug was 
advocating) would statically analyze the graph's topology and invoke 
each tool with appropriate parameters or environment variables. 
However, this design would require significantly more extensive 
modifications to bash, or the implementation of a new shell.  Both 
approaches required work for which we didn't have the time and energy at 
the time, and also had their own downsides regarding adoption potential.

[1] https://www.spinellis.gr/sw/dgsh/#examples
[2] https://www.spinellis.gr/sw/dgsh/#tools

Diomidis

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[Adam Thornton]

[-- Attachment #1: Type: text/plain, Size: 651 bytes --]

/23+, no?  The basic 23 doesn't have an MMU, I'm pretty sure.

On Thu, Dec 5, 2024 at 11:58 AM Ron Natalie <ron@ronnatalie.com> wrote:

> I remember on MiniUnix (a stripped down kernel that would run on PDP11’s
> without memory management, in our case 11/03, 11/20, and an older 11/40)
> there were no kernel pipes (and limited process concurrency).
> The shell used the aforementioned emulation of pipe by just running the
> output of one process into a temporary file subsequently loaded as stdin
> of the other.
>
> We kind of put all that to bed when the 11/23s came out and we could run
> our full up kernel on the micros.
>
>

[-- Attachment #2: Type: text/html, Size: 1005 bytes --]

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[Dan Cross]

On Thu, Dec 5, 2024 at 9:02 PM John Levine <johnl@taugh.com> wrote:
> According to Dan Cross <crossd@gmail.com>:
> >> >       diff <(old-program-version) <(new-program-version)
> >> >
> >> > to do simple regression testing.
> >>
> >> And fanout is simply
> >>
> >>         ... | tee >(pipeline1) >(pipeline2)
> >
> >And indeed these things are pretty nifty, but don't they generate
> >trees, and not arbitrary dags? They don't quite capture the full
> >generality of Morrison-style networks since it doesn't seem like
> >there's a way to connect process substitution fan-out with fan-in; at
> >least, not conveniently.
>
> You can use mkfifo to make named pipes and then plumb them as wanted.  It's
> not particularly beautiful but it's as general as you could want.
>
> That's worked since 4.4BSD which was quite a while ago.

Sure, but that's orthogonal since it's not really using shell syntax
to build the thing up in a canonical way. I mean, perhaps in some
literal sense it is (one types the commands into the interpreter...)
but nowhere near the expressiveness of `ls | grep foo | whatever`.

Fundamentally, the questions here are all about syntax.  As I wrote in
my first email, the system calls are very general, and could be used
to build arbitrary directed graphs (not even limited to DAGs) of
processes connected by pipes in all sorts of configurations
(presumably those processes would be written to handle multiple input
and output streams), without having to resort to named pipes. One can
imagine some sort of DSL that describes such graphs of pipes and
processes (Bakul alluded to this), and an interpreter for that DSL
that creates the described set of pipes and processes, all
appropriately connected and running. One could even imagine such a
thing being used as part of a more traditional pipeline.

My claim is simply that extant shell syntax isn't really amenable to
this in a natural way. As Chet and Arnold pointed out, process
substitution as pioneered in ksh gets us a little closer, but it's not
quite as general (I believe any such graphs would have to be acyclic);
it's certainly not as syntactically pleasant.

There has been work along these lines; I was sent a reference off-list
to a paper by Spinellis and Fragkoulis about a DAG-oriented shell:
https://www.spinellis.gr/sw/dgsh/, which seems relevant.

        - Dan C.

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[Steve Nickolas]

On Thu, 5 Dec 2024, Greg A. Woods wrote:

> The Wikipedia entry for "Process substitution" says:
>
> 	Process substitution was available as a compile-time option for
> 	ksh88, the 1988 version of the KornShell from Bell Labs.  The rc
> 	shell provides the feature as "pipeline branching" in Version 10
> 	Unix, released in 1990.
>
> So, if we believe that then indeed ksh88 pioneered process substitution.

Confirming that ksh88d and ksh88i support <(command) at least.

-uso.

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[John Levine]

According to Dan Cross <crossd@gmail.com>:
>> >       diff <(old-program-version) <(new-program-version)
>> >
>> > to do simple regression testing.
>>
>> And fanout is simply
>>
>>         ... | tee >(pipeline1) >(pipeline2)
>
>And indeed these things are pretty nifty, but don't they generate
>trees, and not arbitrary dags? They don't quite capture the full
>generality of Morrison-style networks since it doesn't seem like
>there's a way to connect process substitution fan-out with fan-in; at
>least, not conveniently.

You can use mkfifo to make named pipes and then plumb them as wanted.  It's
not particularly beautiful but it's as general as you could want.

That's worked since 4.4BSD which was quite a while ago.

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[Greg A. Woods]

[-- Attachment #1: Type: text/plain, Size: 1075 bytes --]

At Thu, 5 Dec 2024 19:09:26 -0600, "G. Branden Robinson" <g.branden.robinson@gmail.com> wrote:
Subject: [TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)
>
> [1  <text/plain; us-ascii (7bit)>]
> At 2024-12-05T16:07:10-0700, arnold@skeeve.com wrote:
> > Warner Losh <imp@bsdimp.com> wrote:
> > > The <() , >()  syntax is a bash extension. Not all shells support
> > > it. And I couldn't find them in POSIX Issue 8.
> >
> > It originated in ksh93.
>
> Are you sure? I think Tom Duff originated it in his "rc" shell.

The Wikipedia entry for "Process substitution" says:

	Process substitution was available as a compile-time option for
	ksh88, the 1988 version of the KornShell from Bell Labs.  The rc
	shell provides the feature as "pipeline branching" in Version 10
	Unix, released in 1990.

So, if we believe that then indeed ksh88 pioneered process substitution.

--
					Greg A. Woods <gwoods@acm.org>

Kelowna, BC     +1 250 762-7675           RoboHack <woods@robohack.ca>
Planix, Inc. <woods@planix.com>     Avoncote Farms <woods@avoncote.ca>

[-- Attachment #2: OpenPGP Digital Signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[G. Branden Robinson]

[-- Attachment #1: Type: text/plain, Size: 362 bytes --]

At 2024-12-05T16:07:10-0700, arnold@skeeve.com wrote:
> Warner Losh <imp@bsdimp.com> wrote:
> > The <() , >()  syntax is a bash extension. Not all shells support
> > it. And I couldn't find them in POSIX Issue 8.
>
> It originated in ksh93.

Are you sure? I think Tom Duff originated it in his "rc" shell.

https://archive.org/details/rc-shell

Regards,
Branden

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[Alexis]

Chet Ramey via TUHS <tuhs@tuhs.org> writes:

> On 12/5/24 5:03 PM, Warner Losh wrote:
>
>> The <() , >()  syntax is a bash extension. Not all shells 
>> support
>> it. And
>> I couldn't find them in POSIX Issue 8.
>
> Credit where credit is due: I picked them up from ksh93, and 
> extended
> them to use named pipes on systems where /dev/fd isn't 
> available.

i must say i was moderately surprised to learn a few years ago 
that process substitution wasn't in POSIX - i find it succinct 
syntactic sugar. There's a post on Chris Siebenmann's blog about 
some of the issues involved, including the /dev/fd issue:

  https://utcc.utoronto.ca/~cks/space/blog/unix/ProcessSubstitutionWhyLate

Can anyone point me to any discussions about process substitution 
potentially becoming part of POSIX? This search on the Austin 
Group tracker didn't yield any useful results:

https://austingroupbugs.net/search.php?project_id=0&search=process+substitution&sticky_issues=off&sortby=last_updated&dir=DESC&hide_status_id=-2&FILTER_SEARCH_PLATFORM=


Alexis.

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[Marc Rochkind]

[-- Attachment #1: Type: text/plain, Size: 992 bytes --]

I found that 2017 paper "Extending Unix Pipelines to DAGs". It's open
access:

https://ieeexplore.ieee.org/document/7903579

The open source code itself is here: https://github.com/dspinellis/dgsh

Maybe an ambitious TUHS contributor can get the code running and give us a
report.

Marc Rochkind

On Thu, Dec 5, 2024 at 3:38 PM Chet Ramey via TUHS <tuhs@tuhs.org> wrote:

> On 12/5/24 5:03 PM, Warner Losh wrote:
>
> > The <() , >()  syntax is a bash extension. Not all shells support it. And
> > I couldn't find them in POSIX Issue 8.
>
> Credit where credit is due: I picked them up from ksh93, and extended
> them to use named pipes on systems where /dev/fd isn't available.
>
> --
> ``The lyf so short, the craft so long to lerne.'' - Chaucer
>                  ``Ars longa, vita brevis'' - Hippocrates
> Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/
>


-- 
*My new email address is mrochkind@gmail.com <mrochkind@gmail.com>*

[-- Attachment #2: Type: text/html, Size: 1894 bytes --]

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[arnold]

Warner Losh <imp@bsdimp.com> wrote:

> The <() , >()  syntax is a bash extension. Not all shells support it. And
> I couldn't find them in POSIX Issue 8.
>
> Warner

It originated in ksh93. 30 years have gone by, it's time it
was added to POSIX.

At least, IMHO.

Thanks,

Arnold

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[Chet Ramey via TUHS]

[-- Attachment #1.1: Type: text/plain, Size: 490 bytes --]

On 12/5/24 5:03 PM, Warner Losh wrote:

> The <() , >()  syntax is a bash extension. Not all shells support it. And
> I couldn't find them in POSIX Issue 8.

Credit where credit is due: I picked them up from ksh93, and extended
them to use named pipes on systems where /dev/fd isn't available.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
		 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 203 bytes --]

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[Bakul Shah via TUHS]

On Dec 5, 2024, at 1:13 PM, Dan Cross <crossd@gmail.com> wrote:
> 
> And indeed these things are pretty nifty, but don't they generate
> trees, and not arbitrary dags? They don't quite capture the full
> generality of Morrison-style networks since it doesn't seem like
> there's a way to connect process substitution fan-out with fan-in; at
> least, not conveniently.

Isn’t that a limitation of the shell language(s)? It would be
easier in a 2D graphical shell but that would perhaps make
specifying the more common simpler pipelines harder!

A "linear" syntax would require *naming*. For example,

let-proc a = "foo -a", b = "bar -c -d", c = "zee" in
  a.in1 = file1, a.in2 = file2, b.in1 = a.out, b.in2 = file3,
  c.in1 = b.out1, c.in2 = a.out, c.out = file4;

Here names a, b & c are used to denote *processes*, with
connections specified in the next two lines. Once all the
pipes are created, processes a, b & c can be started giving
them the right endpoints in forked processes prior to execing.
Doable but not really worth it. At this point you may as well
use s-expressions!

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 2077 bytes --]

On Thu, Dec 5, 2024 at 2:13 PM Dan Cross <crossd@gmail.com> wrote:

> On Thu, Dec 5, 2024 at 3:56 PM <arnold@skeeve.com> wrote:
> > Chet Ramey via TUHS <tuhs@tuhs.org> wrote:
> > > On 12/5/24 10:19 AM, Dan Cross wrote:
> > >
> > > > Unix pipelines, on the other hand, tend to be used in a manner that
> is
> > > > strictly linear, without the fan-out and fan-in capabilities
> described
> > > > by Morrison. Of course, nothing prevents one from building a
> > > > Morrison-style "network" from Unix processes and pipes, though it's
> > > > hard to see how that would work without something like `select`,
> which
> > > > didn't yet exist in 1978. Regardless, Unix still doesn't expose a
> > > > particularly convenient syntax for expressing these sorts of
> > > > constructions at the shell.
> > >
> > > Process substitution is about as close as we can get, but most programs
> > > still process their filename arguments one at a time, beginning to end.
> > >
> > > The canonical process substitution example is
> > >
> > >       diff <(old-program-version) <(new-program-version)
> > >
> > > to do simple regression testing.
> >
> > And fanout is simply
> >
> >         ... | tee >(pipeline1) >(pipeline2)
>
> And indeed these things are pretty nifty, but don't they generate
> trees, and not arbitrary dags? They don't quite capture the full
> generality of Morrison-style networks since it doesn't seem like
> there's a way to connect process substitution fan-out with fan-in; at
> least, not conveniently.
>
> It is, perhaps, notable that Go allows me to do this sort of thing
> with channels and goroutines, but it has `select` built into the
> language.
>
> Funny, despite using Unix almost daily for over 30 years now, I don't
> think I've ever felt limited by the power of pipelines. On the
> contrary, I've lost count of the times I've felt limited on systems
> that do Not support pipes.
>

The <() , >()  syntax is a bash extension. Not all shells support it. And
I couldn't find them in POSIX Issue 8.

Warner

[-- Attachment #2: Type: text/html, Size: 2879 bytes --]

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[Marc Rochkind]

[-- Attachment #1.1: Type: text/plain, Size: 3408 bytes --]

Some of the posts in this thread confuse pipes in UNIX shells (the |
symbol) with the pipe system call.

In the shell, how two processes can be connected with a pipe is very
constrained (only one unidirectional pipe). But the pipe system call can be
used to build much more elaborate connections. Back in 1980, when I was
still at Bell Labs, I wrote a shell called 2dsh ("two dimensional shell)
that had a more complex syntax. The memo I wrote, "2DSH—An experimental
shell for connecting processes with multiple data streams",  wasn't
published externally, but exists as a Bell Labs memo. I found a reference
here:
https://scholar.google.com/scholar_lookup?title=2DSH%E2%80%94An+experimental+shell+for+connecting+processes+with+multiple+data+streams&author=M.+J.+Rochkind&publication_year=1980
.

Here are two examples from that memo:
[image: image.png]

I stumbled across another paper from 2017 titled "Extending Unix Pipelines
to DAGs," which references my un-published Bell Labs memo. I haven't read
it since I don't subscribe to IEEE Transactions on Computers.

A while ago Doug McIlroy was kind enough to send me a scan of my memo, but
I don't think I'm allowed to publish it here. In that memo, I credit Doug
for coming with a very similar idea around the same time ("A Notation for
Arboreal Plumbing").

Marc Rochkind

On Thu, Dec 5, 2024 at 2:13 PM Dan Cross <crossd@gmail.com> wrote:

> On Thu, Dec 5, 2024 at 3:56 PM <arnold@skeeve.com> wrote:
> > Chet Ramey via TUHS <tuhs@tuhs.org> wrote:
> > > On 12/5/24 10:19 AM, Dan Cross wrote:
> > >
> > > > Unix pipelines, on the other hand, tend to be used in a manner that
> is
> > > > strictly linear, without the fan-out and fan-in capabilities
> described
> > > > by Morrison. Of course, nothing prevents one from building a
> > > > Morrison-style "network" from Unix processes and pipes, though it's
> > > > hard to see how that would work without something like `select`,
> which
> > > > didn't yet exist in 1978. Regardless, Unix still doesn't expose a
> > > > particularly convenient syntax for expressing these sorts of
> > > > constructions at the shell.
> > >
> > > Process substitution is about as close as we can get, but most programs
> > > still process their filename arguments one at a time, beginning to end.
> > >
> > > The canonical process substitution example is
> > >
> > >       diff <(old-program-version) <(new-program-version)
> > >
> > > to do simple regression testing.
> >
> > And fanout is simply
> >
> >         ... | tee >(pipeline1) >(pipeline2)
>
> And indeed these things are pretty nifty, but don't they generate
> trees, and not arbitrary dags? They don't quite capture the full
> generality of Morrison-style networks since it doesn't seem like
> there's a way to connect process substitution fan-out with fan-in; at
> least, not conveniently.
>
> It is, perhaps, notable that Go allows me to do this sort of thing
> with channels and goroutines, but it has `select` built into the
> language.
>
> Funny, despite using Unix almost daily for over 30 years now, I don't
> think I've ever felt limited by the power of pipelines. On the
> contrary, I've lost count of the times I've felt limited on systems
> that do Not support pipes.
>
>         - Dan C.
>


-- 
*My new email address is mrochkind@gmail.com <mrochkind@gmail.com>*

[-- Attachment #1.2: Type: text/html, Size: 4757 bytes --]

[-- Attachment #2: image.png --]
[-- Type: image/png, Size: 368482 bytes --]

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[Dan Cross]

On Thu, Dec 5, 2024 at 3:56 PM <arnold@skeeve.com> wrote:
> Chet Ramey via TUHS <tuhs@tuhs.org> wrote:
> > On 12/5/24 10:19 AM, Dan Cross wrote:
> >
> > > Unix pipelines, on the other hand, tend to be used in a manner that is
> > > strictly linear, without the fan-out and fan-in capabilities described
> > > by Morrison. Of course, nothing prevents one from building a
> > > Morrison-style "network" from Unix processes and pipes, though it's
> > > hard to see how that would work without something like `select`, which
> > > didn't yet exist in 1978. Regardless, Unix still doesn't expose a
> > > particularly convenient syntax for expressing these sorts of
> > > constructions at the shell.
> >
> > Process substitution is about as close as we can get, but most programs
> > still process their filename arguments one at a time, beginning to end.
> >
> > The canonical process substitution example is
> >
> >       diff <(old-program-version) <(new-program-version)
> >
> > to do simple regression testing.
>
> And fanout is simply
>
>         ... | tee >(pipeline1) >(pipeline2)

And indeed these things are pretty nifty, but don't they generate
trees, and not arbitrary dags? They don't quite capture the full
generality of Morrison-style networks since it doesn't seem like
there's a way to connect process substitution fan-out with fan-in; at
least, not conveniently.

It is, perhaps, notable that Go allows me to do this sort of thing
with channels and goroutines, but it has `select` built into the
language.

Funny, despite using Unix almost daily for over 30 years now, I don't
think I've ever felt limited by the power of pipelines. On the
contrary, I've lost count of the times I've felt limited on systems
that do Not support pipes.

        - Dan C.

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[arnold]

Chet Ramey via TUHS <tuhs@tuhs.org> wrote:

> On 12/5/24 10:19 AM, Dan Cross wrote:
>
> > Unix pipelines, on the other hand, tend to be used in a manner that is
> > strictly linear, without the fan-out and fan-in capabilities described
> > by Morrison. Of course, nothing prevents one from building a
> > Morrison-style "network" from Unix processes and pipes, though it's
> > hard to see how that would work without something like `select`, which
> > didn't yet exist in 1978. Regardless, Unix still doesn't expose a
> > particularly convenient syntax for expressing these sorts of
> > constructions at the shell.
>
> Process substitution is about as close as we can get, but most programs
> still process their filename arguments one at a time, beginning to end.
>
> The canonical process substitution example is
>
> 	diff <(old-program-version) <(new-program-version)
>
> to do simple regression testing.

And fanout is simply

	... | tee >(pipeline1) >(pipeline2)

Arnold

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[Ron Natalie]

I remember on MiniUnix (a stripped down kernel that would run on PDP11’s 
without memory management, in our case 11/03, 11/20, and an older 11/40) 
there were no kernel pipes (and limited process concurrency).
The shell used the aforementioned emulation of pipe by just running the 
output of one process into a temporary file subsequently loaded as stdin 
of the other.

We kind of put all that to bed when the 11/23s came out and we could run 
our full up kernel on the micros.

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[John Levine]

It appears that Marc Rochkind <mrochkind@gmail.com> said:
>-=-=-=-=-=-
>-=-=-=-=-=-
>
>I don't think files as pipes would be "transparent to the user." Reading an
>empty pipe causes a wait until the bytes requested are available, unless
>the pipe is closed first. ...

You should read Meinz' paper.  This was a cut down version of Unix that ran
in 40K bytes on an LSI-11 with a couple of floppy disks.  It ran one program
at a time, since swapping to floppies was absurdly slow.  The pseudo-pipe
ran the first program which wrote the output to a file, then ran the second
program which read it.  It was close enough for most purposes so long as
the disk didn't fill up.

>Still, the same sort of pseudo-pipes were in MS-DOS, and they were
>occasionally useful.

I can think of lots of places that had a way for one program to write a temporary
file that a subsequent one read.  For example, in OS/360 JCL you'd write (well, punch)
something like this for the object output of the Fortran compiler

//SYSLIN DD DSNAME=&OBJECT,DISP=(NEW,PASS),UNIT=SYSSQ

and then in the linker read it in

//SYSIN DD DSNAME=&OBJECT,DISP=(OLD,DELETE)

The & in the name said it was a temp file so make up a unique name.  I probably didn't
get the JCL exactly right since it's been about 50 years since I wrote any.

R's,
John

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[John Cowan]

[-- Attachment #1.1: Type: text/plain, Size: 2889 bytes --]

There was no concurrency in mini-Unix; the first pipeline component ran to
completion, and when it exited the next component ran, and so on.  In this
way a pipeline could be arbitrarily long using just two disk files.

On Thu, Dec 5, 2024, 12:07 PM Marc Rochkind <mrochkind@gmail.com> wrote:

> I don't think files as pipes would be "transparent to the user." Reading
> an empty pipe causes a wait until the bytes requested are available, unless
> the pipe is closed first. Reading to the end of a file results in an
> end-of-file error. This problem is avoided if the source process completes
> before the target process begins, but then there is a different lack of
> transparency, which is that the processes don't run simultaneously. (I
> think this is the case with the implementation that Heinz showed.)
>
> Still, the same sort of pseudo-pipes were in MS-DOS, and they were
> occasionally useful.
>
> Marc
>
> On Thu, Dec 5, 2024 at 9:17 AM Heinz Lycklama <heinz@osta.com> wrote:
>
>> John, thanks for the reminder of the implementation
>> of pipes on a constrained version of UNIX in the early
>> days. The exact implementation is described on page 2095
>> of the BSTJ July-Aug 1978 for interested parties.
>>
>>
>>
>> Heinz
>>
>> On 12/5/2024 8:00 AM, John R Levine wrote:
>>
>> On Thu, 5 Dec 2024, Dan Cross wrote:
>>
>> Pipes were invented at least three times I'm aware of, but what made them
>> work so well in Unix is that they looked to the program the same as a
>> file
>> so any program could use them for input or output without special
>> arrangements,
>> and the shell made it easy to start two programs and pipe them together.
>>
>>
>> Once you have coroutines and queues for passing data between them, a
>> lot of things start to look like pipes.
>>
>>
>> They also can look a lot like temporary files.  Someone, probably Heinz,
>> did a shell for the tiny Unix that ran on floppies so this
>>
>>  foo | bar
>>
>> actually did this
>>
>>  foo > tmpfile ; bar < tmpfile; rm tmpfile
>>
>> to avoid having to swap programs in and out on floppies.  The main
>> disadvantage was that the tmpfile could overflow the tiny disks of the
>> time.
>>
>> They were invented again at IBM in the 1970s and described in this
>> paper.  I wrote
>> them a letter, which they published, saying that Unix pipes did the same
>> thing.
>>
>> https://dl.acm.org/doi/10.1147/sj.174.0383
>>
>>
>> Don't forget CMS pipelines, too!
>>
>> Sadly, the Morrison paper cited above is not easily accessible, though
>>
>>
>> If anyone else needs a copy, just ask.
>>
>> Regards,
>> John Levine, johnl@taugh.com, Taughannock Networks, Trumansburg NY
>> Please consider the environment before reading this e-mail. https://jl.ly
>>
>>
>>
>
> --
> *My new email address is mrochkind@gmail.com <mrochkind@gmail.com>*
>

[-- Attachment #1.2: Type: text/html, Size: 4921 bytes --]

[-- Attachment #2: RGxkomgHdfzyhAUv.png --]
[-- Type: image/png, Size: 138780 bytes --]

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[Chet Ramey via TUHS]

[-- Attachment #1.1: Type: text/plain, Size: 1038 bytes --]

On 12/5/24 10:19 AM, Dan Cross wrote:

> Unix pipelines, on the other hand, tend to be used in a manner that is
> strictly linear, without the fan-out and fan-in capabilities described
> by Morrison. Of course, nothing prevents one from building a
> Morrison-style "network" from Unix processes and pipes, though it's
> hard to see how that would work without something like `select`, which
> didn't yet exist in 1978. Regardless, Unix still doesn't expose a
> particularly convenient syntax for expressing these sorts of
> constructions at the shell.

Process substitution is about as close as we can get, but most programs
still process their filename arguments one at a time, beginning to end.

The canonical process substitution example is

	diff <(old-program-version) <(new-program-version)

to do simple regression testing.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
		 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 203 bytes --]

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[Paul Winalski]

[-- Attachment #1: Type: text/plain, Size: 697 bytes --]

Regarding pipes and pipe-like interprocess communications facilities in
other operating systems, VMS has always had a pipe-like communications
pseudo-devices called mailboxes.  The main difference between Unix pipes
and VMS mailboxes is that pipes have distinct read-only and write-only file
descriptors.  Mailboxes do not--channels (VMS-speak for file descriptors)
assigned to a mailbox can be used for both reading and writing.  This means
that it is not possible to do "broken pipe"-type detection on a mailbox.
This very much restricts the usefulness of mailboxes.

I wrote a true pipe device driver for VMS as part of the DEC Shell product
(a port of the Unix Bourne shell to VMS).

-Paul W.

[-- Attachment #2: Type: text/html, Size: 816 bytes --]

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[Marc Rochkind]

[-- Attachment #1.1: Type: text/plain, Size: 2494 bytes --]

I don't think files as pipes would be "transparent to the user." Reading an
empty pipe causes a wait until the bytes requested are available, unless
the pipe is closed first. Reading to the end of a file results in an
end-of-file error. This problem is avoided if the source process completes
before the target process begins, but then there is a different lack of
transparency, which is that the processes don't run simultaneously. (I
think this is the case with the implementation that Heinz showed.)

Still, the same sort of pseudo-pipes were in MS-DOS, and they were
occasionally useful.

Marc

On Thu, Dec 5, 2024 at 9:17 AM Heinz Lycklama <heinz@osta.com> wrote:

> John, thanks for the reminder of the implementation
> of pipes on a constrained version of UNIX in the early
> days. The exact implementation is described on page 2095
> of the BSTJ July-Aug 1978 for interested parties.
>
>
>
> Heinz
>
> On 12/5/2024 8:00 AM, John R Levine wrote:
>
> On Thu, 5 Dec 2024, Dan Cross wrote:
>
> Pipes were invented at least three times I'm aware of, but what made them
> work so well in Unix is that they looked to the program the same as a file
> so any program could use them for input or output without special
> arrangements,
> and the shell made it easy to start two programs and pipe them together.
>
>
> Once you have coroutines and queues for passing data between them, a
> lot of things start to look like pipes.
>
>
> They also can look a lot like temporary files.  Someone, probably Heinz,
> did a shell for the tiny Unix that ran on floppies so this
>
>  foo | bar
>
> actually did this
>
>  foo > tmpfile ; bar < tmpfile; rm tmpfile
>
> to avoid having to swap programs in and out on floppies.  The main
> disadvantage was that the tmpfile could overflow the tiny disks of the
> time.
>
> They were invented again at IBM in the 1970s and described in this paper.
> I wrote
> them a letter, which they published, saying that Unix pipes did the same
> thing.
>
> https://dl.acm.org/doi/10.1147/sj.174.0383
>
>
> Don't forget CMS pipelines, too!
>
> Sadly, the Morrison paper cited above is not easily accessible, though
>
>
> If anyone else needs a copy, just ask.
>
> Regards,
> John Levine, johnl@taugh.com, Taughannock Networks, Trumansburg NY
> Please consider the environment before reading this e-mail. https://jl.ly
>
>
>

-- 
*My new email address is mrochkind@gmail.com <mrochkind@gmail.com>*

[-- Attachment #1.2: Type: text/html, Size: 4265 bytes --]

[-- Attachment #2: RGxkomgHdfzyhAUv.png --]
[-- Type: image/png, Size: 138780 bytes --]

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[Adam Thornton]

[-- Attachment #1: Type: text/plain, Size: 894 bytes --]

On Thu, Dec 5, 2024 at 8:20 AM Dan Cross <crossd@gmail.com> wrote:

>
> Unix pipelines, on the other hand, tend to be used in a manner that is
> strictly linear, without the fan-out and fan-in capabilities described
> by Morrison. Of course, nothing prevents one from building a
> Morrison-style "network" from Unix processes and pipes, though it's
> hard to see how that would work without something like `select`, which
> didn't yet exist in 1978. Regardless, Unix still doesn't expose a
> particularly convenient syntax for expressing these sorts of
> constructions at the shell.
>
>
Rick Troth has recently published xfl, which is pretty much CMS Pipelines
for Unix.

https://github.com/trothtech/xfl

He's got a slide deck on it at
http://www.casita.net/pub/xfl/pervasive-vmws-2024.pdf .

There are a lot of really cool things you can do with fanin/fanout.

Adam

[-- Attachment #2: Type: text/html, Size: 1464 bytes --]

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[Heinz Lycklama]

[-- Attachment #1: Type: text/plain, Size: 1747 bytes --]

John, thanks for the reminder of the implementation
of pipes on a constrained version of UNIX in the early
days. The exact implementation is described on page 2095
of the BSTJ July-Aug 1978 for interested parties.



Heinz

On 12/5/2024 8:00 AM, John R Levine wrote:
> On Thu, 5 Dec 2024, Dan Cross wrote:
>>> Pipes were invented at least three times I'm aware of, but what made 
>>> them
>>> work so well in Unix is that they looked to the program the same as 
>>> a file
>>> so any program could use them for input or output without special 
>>> arrangements,
>>> and the shell made it easy to start two programs and pipe them 
>>> together.
>>
>> Once you have coroutines and queues for passing data between them, a
>> lot of things start to look like pipes.
>
> They also can look a lot like temporary files.  Someone, probably 
> Heinz, did a shell for the tiny Unix that ran on floppies so this
>
>  foo | bar
>
> actually did this
>
>  foo > tmpfile ; bar < tmpfile; rm tmpfile
>
> to avoid having to swap programs in and out on floppies.  The main 
> disadvantage was that the tmpfile could overflow the tiny disks of the 
> time.
>
>>> They were invented again at IBM in the 1970s and described in this 
>>> paper.  I wrote
>>> them a letter, which they published, saying that Unix pipes did the 
>>> same thing.
>>>
>>> https://dl.acm.org/doi/10.1147/sj.174.0383
>>
>> Don't forget CMS pipelines, too!
>>
>> Sadly, the Morrison paper cited above is not easily accessible, though
>
> If anyone else needs a copy, just ask.
>
> Regards,
> John Levine, johnl@taugh.com, Taughannock Networks, Trumansburg NY
> Please consider the environment before reading this e-mail. https://jl.ly

[-- Attachment #2.1: Type: text/html, Size: 3216 bytes --]

[-- Attachment #2.2: RGxkomgHdfzyhAUv.png --]
[-- Type: image/png, Size: 138780 bytes --]

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[John R Levine]

On Thu, 5 Dec 2024, Dan Cross wrote:
>> Pipes were invented at least three times I'm aware of, but what made them
>> work so well in Unix is that they looked to the program the same as a file
>> so any program could use them for input or output without special arrangements,
>> and the shell made it easy to start two programs and pipe them together.
>
> Once you have coroutines and queues for passing data between them, a
> lot of things start to look like pipes.

They also can look a lot like temporary files.  Someone, probably Heinz, 
did a shell for the tiny Unix that ran on floppies so this

  foo | bar

actually did this

  foo > tmpfile ; bar < tmpfile; rm tmpfile

to avoid having to swap programs in and out on floppies.  The main 
disadvantage was that the tmpfile could overflow the tiny disks of the 
time.

>> They were invented again at IBM in the 1970s and described in this paper.  I wrote
>> them a letter, which they published, saying that Unix pipes did the same thing.
>>
>> https://dl.acm.org/doi/10.1147/sj.174.0383
>
> Don't forget CMS pipelines, too!
>
> Sadly, the Morrison paper cited above is not easily accessible, though

If anyone else needs a copy, just ask.

Regards,
John Levine, johnl@taugh.com, Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[Dan Cross]

On Wed, Dec 4, 2024 at 10:18 PM John Levine <johnl@taugh.com> wrote:
> It appears that Marc Donner <marc.donner@gmail.com> said:
> >With the notion of pipes it became possible to operate on data quickly and
> >flexibly.  There was nothing new from a fundamental capability point of
> >view, but the ease with which one could construct pipelines enabled rapid
> >experimentation and encouraged the development of pipe-able components to
> >add to the tool set.
>
> Pipes were invented at least three times I'm aware of, but what made them
> work so well in Unix is that they looked to the program the same as a file
> so any program could use them for input or output without special arrangements,
> and the shell made it easy to start two programs and pipe them together.

Once you have coroutines and queues for passing data between them, a
lot of things start to look like pipes.

> The Dartmouth Time-Sharing System in the late 1960s had communication files
> which were essentially two-way pipes, but they were asymmetrical. One end, the
> slave end, looked like a file, but the other end, the master end, was different
> and the program had to know it was a com file. They were mostly used to pass
> terminal I/O between user programs at the slave end and SIMON at the master end,
> the terminal monitor that talked to the front end computer than ran the TTYs.

Doug has written at some length on this list about communication files
and their homomorphism to the way Plan 9 presented and handled
resources.

> They were invented again at IBM in the 1970s and described in this paper.  I wrote
> them a letter, which they published, saying that Unix pipes did the same thing.
>
> https://dl.acm.org/doi/10.1147/sj.174.0383

Don't forget CMS pipelines, too!

Sadly, the Morrison paper cited above is not easily accessible, though
I acquired a copy from IEEE; perhaps a sucker's game as it was not
cheap. Your subsequent letter, and part of Morrison's response to you,
however, is available, gratis.

Reading through Morrison, one gets the impression that there are some
substantial differences with Unix pipelines; or rather, the way that
Unix pipelines are usually used. In particular, he describes his
linked streams in terms of a "network", by which he appears to mean an
arbitrary directed graph. Crucially, he describes combining nodes, for
merging data from multiple streams.

Unix pipelines, on the other hand, tend to be used in a manner that is
strictly linear, without the fan-out and fan-in capabilities described
by Morrison. Of course, nothing prevents one from building a
Morrison-style "network" from Unix processes and pipes, though it's
hard to see how that would work without something like `select`, which
didn't yet exist in 1978. Regardless, Unix still doesn't expose a
particularly convenient syntax for expressing these sorts of
constructions at the shell.

As an aside, Morrison has a web page dedicated to "flow-based
programming", that he claims to have invented in the late 1960s.  That
seems like a bit of a tall claim, and I'd wager Doug gives him a run
for his money on that.  https://jpaulm.github.io/fbp/

        - Dan C.