[TUHS] After 50 years, what has the Impact of Unix been?

[TUHS] After 50 years, what has the Impact of Unix been?

[sjenkin]

I was looking at the question of “impact of Unix" and thought that:

	initiating (Portable) Open Source Software including the BSD TCP/IP & Berkeley sockets libraries,
	the Unix -> Minix -> Linux -> Android sequence
	and BSD -> NeXtstep -> OS/X -> iOS sequence,
	plus running the Top 500 supercomputers and most of the Top 500 websites,
	including the infrastructure for trillion dollars companies, Facebook, Amazon (Netflix uses them), and Google
	plus so many embedded Linux / NetBSD based appliances
	even going into space - on small experiments or driving SpaceX’s Falcon 9 & presumably Starship,

would be a slam-dunk for “really high impact” 
	- dominating everywhere important, except Windows desktops.

Unix wasn’t just a ’research project’, it was the result of years of work by a team of very capable, professional programmers,
who weren’t looking for kudos or recognition, nor trying to add every conceivable ‘feature’ possible, but the inverse:

	 how _small_ could it be and still be enough.

When it left the Labs, Unix wasn’t just “Performant”, it came with a very useful set of tools for creating other tools (‘developing’)
and while the kernel wasn’t perfect (some ‘panic’s), it was of “Production Quality”. 

For v6, I believe there were patches for 50-100 bugs circulating, perhaps the first demonstration of “no bug is intractable with ‘many eyeballs’”.

All that in under 5 years of ‘development’, with the “initial release” stable & performant enough for cash-strapped Universities
to gamble their reputations & budgets on running it.
Imagine the grief academics would’ve got if their basic teaching systems failed continuously!

This adoption path pushed Unix through another barrier:

	 ’Security’ - with a lot of bright, bored & curious students banging on it as hard as they could for bragging rights.

How long, in releases or years, did it take for other O/S’s to hit the “very stable” benchmark?

I don’t know enough of Linux to answer that definitively, the *BSD’s grew there through usage and contribution,
while Microsoft NT derivates widely suffered “Blue Screen of Death” for years.

Even now, MS-Windows has serious Security / compromise issues, like the highly visible, global “Crowdstrike” event. 
Not a break-in or takeover, but an own-goal from Security perimeter control.

==========

I now think Unix has had a much larger, direct and important impact

	- the C language and associated tools & libraries
		 that begat modern toolchains and endless portability across platforms.

In 1991, Bill Plauger had a year sabbatical at UNSW in Sydney,
and happened to say :
	“C is wallpaper - people expect it everywhere”.

C gained formal recognition with the POSIX standard, satisfying conservative users / enterprises that it wasn’t the work of a bunch of Hippies or ill-disciplined Hackers.

Even Microsoft wrote 32-bit Windows NT in C, I presume starting by writing it’s own compiler and toolchain to start.

Borland, Watcom and many others - including Microsoft - offered (Visual) C compile & build environments for Windows,
directly responsible for creating the ’shrink-wrap’ third party software market that drove sales of Windows and x86 machines.

Nobody had seen a market for a billion systems before, nor sold 300M+ CPU’s in a single year.

People don’t buy Silicon Chips or nice Boxes, they buy Applications that solve their problems:

	Software drives Sales of Hardware
	 - something that IBM deeply understood first with first the 1401 line, then 360-series.


The other ’small’ achievement of C and Unix was creating the market for RISC chips.
MIPS in the mid-1980’s was only able to design and build the first commercial RISC chip
because it knew it could port Unix to it and find an immediate market 
	- not at zero-cost, but a tiny fraction of what every other Vendor had done before 
		reinventing the wheel from scratch to provide incompatible O/S & tools for their hardware.

Unix on MIPS not only came with a host of proven software, that a large pool of people knew how to use,
but it arrived as “Production Quality” - the porting team had to test their parts - compiler, linker, libraries - hard, but could trust the existing high-quality codebase.

In "A New Golden Age for Computer Architecture”, 2019 by Hennessy & Patterson,
make an aside:

	In today's post-PC era, x86 shipments have fallen almost 10% per year since the peak in 2011,
	while chips with RISC processors have skyrocketed to 20 billion.

	Today, 99% of 32-bit and 64-bit processors are RISC.

i suggest this goes back to PCC followed by the MIPS R2000 - made possible by Dennis’ C language.

The 1977 invention of ‘pcc’ and rewriting of Unix for cross-machine portability was the first time I’m aware of this being done.
 ( Miller @ UoW did a one-off hack, not to devalue his work, he ported, didn’t invent a multi-target portable compiler )

One of the effects of “portable C” was creating whole new industries for third party software developers
or enabling niche products, like CISCO routers and the many embedded devices.

C and Unix came with the tools to create new languages and new tools.
AWK, sed (!) and shells are obvious examples, with Perl, Python & PHP very big in Internet of 2000.

C was a new class of language - a tool to create tools.
It creates a perfect mechanism to bootstrap any new language, tool or product,
allowing to be refined & used enough to become reliable before being made self-hosting.

Very widely used languages such as Python are written in C.
ORACLE achieved its market dominance by providing ‘portability’ - exactly the same on every platform.
Underpinned by portable C.

The original 1127 team went on to create other systems and languages,
not the least being a new Software Engineering tool, “Go” / golang,
addressing a whole slew of deficiencies in the C/C++ approach and 

We’d have no Internet today without Apache written in C and being ported to every environment.

Also, there’s a connection between C and ‘modern’ Software Engineering 
  - distributed Repositories, automated builds & regression tests, and the many toolchains and tools used.

They tended to be built in C to address problems (Open Source) developers were finding with existing toolchains.
‘make’ arose at Bell Labs to automate builds, along with PWB and Writers Workbench.

There’s two questions / observations about 50 years of C in broad use:

	- Just how much C is out there and used ‘in production’?

	- C is ‘obviously’ a product of the 1970’s, not reflecting needs of modern hardware, networks, storage and systems,
		but _what_ can replace it?

		There is simply too much critical code written in C to convert it to another ‘better, modern’ language.
		Any new language that is a simple 1:1 rewriting of C cannot address any of the deficiencies,
		while any incompatible language requires redesign and reimplementation of everything - an unachievable goal.

The Linux Kernel’s “rust” project shows the extent of the problem 
   - even with the best team of the best developers, its a mammoth undertaking, with uncertain payoffs, unquantifiable effort & deadlines.

My thesis is that portable, standard C:

	- not only co-evolved with other tools & needs to create the Modern Software Engineering environment, the basis for multiple Trillion dollar enterprises (FAANG)
but
	- drove the biggest, most profitable software market ever seen (Wintel)
	- which drove sales volume of x86 chips (& DRAM, motherboards, LAN, GPU, monitors, peripherals…) over 2-3 decades,
	- which drove Silicon Valley, paying for new generations of Fabs and lowering chip prices further & further
	- and eventually created the Fabless RISC CPU company,
		which in the Post-PC era absolutely dominates chip sales.

No Software, no Silicon… 

Gordon Moore, in an early comment on his 1968 startup with Robert Noyce, said:

	“we are the real revolutionaries" (vs Hippies & 1967 Summer of Love).

I think Ken & Dennis [ and 1127/ Bell Labs folk ] can say the same.

==========

I’ve written some notes, with links to Programming Languages, especially Jean Sammet’s Histories,
and would like some critiques, suggestions & corrections if people have time and interest.

Unix and C are intimately related - neither was possible or useful without the other.

i think there’s an interesting article in there, but I’m not sure I have what it takes to write it, not in a finite time :)
Very happy to help anyone who does!

Did-C-lang-create-modern-software-industry.txt
	<https://drive.google.com/file/d/1k936sgqHc-vHBvfCdLoSxFhdT9NaijU2/view?usp=sharing>

steve jenkin
04 - dec - 2024

==========
--
Steve Jenkin, IT Systems and Design 
0412 786 915 (+61 412 786 915)
PO Box 38, Kippax ACT 2615, AUSTRALIA

mailto:sjenkin@canb.auug.org.au http://members.tip.net.au/~sjenkin

[TUHS] Re: After 50 years, what has the Impact of Unix been?

[Marc Donner]

[-- Attachment #1: Type: text/plain, Size: 10554 bytes --]

I would add one more important impact: making data a first class component
of the computing environment.

With the notion of pipes it became possible to operate on data quickly and
flexibly.  There was nothing new from a fundamental capability point of
view, but the ease with which one could construct pipelines enabled rapid
experimentation and encouraged the development of pipe-able components to
add to the tool set.

I may not have articulated this as well as it deserves.

Marc
=====
nygeek.net
mindthegapdialogs.com/home <https://www.mindthegapdialogs.com/home>


On Wed, Dec 4, 2024 at 3:43 AM <sjenkin@canb.auug.org.au> wrote:

> I was looking at the question of “impact of Unix" and thought that:
>
>         initiating (Portable) Open Source Software including the BSD
> TCP/IP & Berkeley sockets libraries,
>         the Unix -> Minix -> Linux -> Android sequence
>         and BSD -> NeXtstep -> OS/X -> iOS sequence,
>         plus running the Top 500 supercomputers and most of the Top 500
> websites,
>         including the infrastructure for trillion dollars companies,
> Facebook, Amazon (Netflix uses them), and Google
>         plus so many embedded Linux / NetBSD based appliances
>         even going into space - on small experiments or driving SpaceX’s
> Falcon 9 & presumably Starship,
>
> would be a slam-dunk for “really high impact”
>         - dominating everywhere important, except Windows desktops.
>
> Unix wasn’t just a ’research project’, it was the result of years of work
> by a team of very capable, professional programmers,
> who weren’t looking for kudos or recognition, nor trying to add every
> conceivable ‘feature’ possible, but the inverse:
>
>          how _small_ could it be and still be enough.
>
> When it left the Labs, Unix wasn’t just “Performant”, it came with a very
> useful set of tools for creating other tools (‘developing’)
> and while the kernel wasn’t perfect (some ‘panic’s), it was of “Production
> Quality”.
>
> For v6, I believe there were patches for 50-100 bugs circulating, perhaps
> the first demonstration of “no bug is intractable with ‘many eyeballs’”.
>
> All that in under 5 years of ‘development’, with the “initial release”
> stable & performant enough for cash-strapped Universities
> to gamble their reputations & budgets on running it.
> Imagine the grief academics would’ve got if their basic teaching systems
> failed continuously!
>
> This adoption path pushed Unix through another barrier:
>
>          ’Security’ - with a lot of bright, bored & curious students
> banging on it as hard as they could for bragging rights.
>
> How long, in releases or years, did it take for other O/S’s to hit the
> “very stable” benchmark?
>
> I don’t know enough of Linux to answer that definitively, the *BSD’s grew
> there through usage and contribution,
> while Microsoft NT derivates widely suffered “Blue Screen of Death” for
> years.
>
> Even now, MS-Windows has serious Security / compromise issues, like the
> highly visible, global “Crowdstrike” event.
> Not a break-in or takeover, but an own-goal from Security perimeter
> control.
>
> ==========
>
> I now think Unix has had a much larger, direct and important impact
>
>         - the C language and associated tools & libraries
>                  that begat modern toolchains and endless portability
> across platforms.
>
> In 1991, Bill Plauger had a year sabbatical at UNSW in Sydney,
> and happened to say :
>         “C is wallpaper - people expect it everywhere”.
>
> C gained formal recognition with the POSIX standard, satisfying
> conservative users / enterprises that it wasn’t the work of a bunch of
> Hippies or ill-disciplined Hackers.
>
> Even Microsoft wrote 32-bit Windows NT in C, I presume starting by writing
> it’s own compiler and toolchain to start.
>
> Borland, Watcom and many others - including Microsoft - offered (Visual) C
> compile & build environments for Windows,
> directly responsible for creating the ’shrink-wrap’ third party software
> market that drove sales of Windows and x86 machines.
>
> Nobody had seen a market for a billion systems before, nor sold 300M+
> CPU’s in a single year.
>
> People don’t buy Silicon Chips or nice Boxes, they buy Applications that
> solve their problems:
>
>         Software drives Sales of Hardware
>          - something that IBM deeply understood first with first the 1401
> line, then 360-series.
>
>
> The other ’small’ achievement of C and Unix was creating the market for
> RISC chips.
> MIPS in the mid-1980’s was only able to design and build the first
> commercial RISC chip
> because it knew it could port Unix to it and find an immediate market
>         - not at zero-cost, but a tiny fraction of what every other Vendor
> had done before
>                 reinventing the wheel from scratch to provide incompatible
> O/S & tools for their hardware.
>
> Unix on MIPS not only came with a host of proven software, that a large
> pool of people knew how to use,
> but it arrived as “Production Quality” - the porting team had to test
> their parts - compiler, linker, libraries - hard, but could trust the
> existing high-quality codebase.
>
> In "A New Golden Age for Computer Architecture”, 2019 by Hennessy &
> Patterson,
> make an aside:
>
>         In today's post-PC era, x86 shipments have fallen almost 10% per
> year since the peak in 2011,
>         while chips with RISC processors have skyrocketed to 20 billion.
>
>         Today, 99% of 32-bit and 64-bit processors are RISC.
>
> i suggest this goes back to PCC followed by the MIPS R2000 - made possible
> by Dennis’ C language.
>
> The 1977 invention of ‘pcc’ and rewriting of Unix for cross-machine
> portability was the first time I’m aware of this being done.
>  ( Miller @ UoW did a one-off hack, not to devalue his work, he ported,
> didn’t invent a multi-target portable compiler )
>
> One of the effects of “portable C” was creating whole new industries for
> third party software developers
> or enabling niche products, like CISCO routers and the many embedded
> devices.
>
> C and Unix came with the tools to create new languages and new tools.
> AWK, sed (!) and shells are obvious examples, with Perl, Python & PHP very
> big in Internet of 2000.
>
> C was a new class of language - a tool to create tools.
> It creates a perfect mechanism to bootstrap any new language, tool or
> product,
> allowing to be refined & used enough to become reliable before being made
> self-hosting.
>
> Very widely used languages such as Python are written in C.
> ORACLE achieved its market dominance by providing ‘portability’ - exactly
> the same on every platform.
> Underpinned by portable C.
>
> The original 1127 team went on to create other systems and languages,
> not the least being a new Software Engineering tool, “Go” / golang,
> addressing a whole slew of deficiencies in the C/C++ approach and
>
> We’d have no Internet today without Apache written in C and being ported
> to every environment.
>
> Also, there’s a connection between C and ‘modern’ Software Engineering
>   - distributed Repositories, automated builds & regression tests, and the
> many toolchains and tools used.
>
> They tended to be built in C to address problems (Open Source) developers
> were finding with existing toolchains.
> ‘make’ arose at Bell Labs to automate builds, along with PWB and Writers
> Workbench.
>
> There’s two questions / observations about 50 years of C in broad use:
>
>         - Just how much C is out there and used ‘in production’?
>
>         - C is ‘obviously’ a product of the 1970’s, not reflecting needs
> of modern hardware, networks, storage and systems,
>                 but _what_ can replace it?
>
>                 There is simply too much critical code written in C to
> convert it to another ‘better, modern’ language.
>                 Any new language that is a simple 1:1 rewriting of C
> cannot address any of the deficiencies,
>                 while any incompatible language requires redesign and
> reimplementation of everything - an unachievable goal.
>
> The Linux Kernel’s “rust” project shows the extent of the problem
>    - even with the best team of the best developers, its a mammoth
> undertaking, with uncertain payoffs, unquantifiable effort & deadlines.
>
> My thesis is that portable, standard C:
>
>         - not only co-evolved with other tools & needs to create the
> Modern Software Engineering environment, the basis for multiple Trillion
> dollar enterprises (FAANG)
> but
>         - drove the biggest, most profitable software market ever seen
> (Wintel)
>         - which drove sales volume of x86 chips (& DRAM, motherboards,
> LAN, GPU, monitors, peripherals…) over 2-3 decades,
>         - which drove Silicon Valley, paying for new generations of Fabs
> and lowering chip prices further & further
>         - and eventually created the Fabless RISC CPU company,
>                 which in the Post-PC era absolutely dominates chip sales.
>
> No Software, no Silicon…
>
> Gordon Moore, in an early comment on his 1968 startup with Robert Noyce,
> said:
>
>         “we are the real revolutionaries" (vs Hippies & 1967 Summer of
> Love).
>
> I think Ken & Dennis [ and 1127/ Bell Labs folk ] can say the same.
>
> ==========
>
> I’ve written some notes, with links to Programming Languages, especially
> Jean Sammet’s Histories,
> and would like some critiques, suggestions & corrections if people have
> time and interest.
>
> Unix and C are intimately related - neither was possible or useful without
> the other.
>
> i think there’s an interesting article in there, but I’m not sure I have
> what it takes to write it, not in a finite time :)
> Very happy to help anyone who does!
>
> Did-C-lang-create-modern-software-industry.txt
>         <
> https://drive.google.com/file/d/1k936sgqHc-vHBvfCdLoSxFhdT9NaijU2/view?usp=sharing
> >
>
> steve jenkin
> 04 - dec - 2024
>
> ==========
> --
> Steve Jenkin, IT Systems and Design
> 0412 786 915 (+61 412 786 915)
> PO Box 38, Kippax ACT 2615, AUSTRALIA
>
> mailto:sjenkin@canb.auug.org.au http://members.tip.net.au/~sjenkin
>
>

[-- Attachment #2: Type: text/html, Size: 12374 bytes --]

[TUHS] Re: After 50 years, what has the Impact of Unix been?

[William Cheswick]

This alludes to an important step that should be mentioned.  Plan 9 was a rethinking of
the whole Unix world.  Though it didn’t spread like C, many of its ideas were adopted
elsewhere.  

And its approach to multi-machine compilation and tools is clearly reflected in the design
and implementation of golang.

ches

> On Dec 4, 2024, at 8:05 AM, Marc Donner <marc.donner@gmail.com> wrote:
> 
> 
> The original 1127 team went on to create other systems and languages,
> not the least being a new Software Engineering tool, “Go” / golang,
> addressing a whole slew of deficiencies in the C/C++ approach and 
> 

[TUHS] Re: After 50 years, what has the Impact of Unix been?

[Rich Salz]

[-- Attachment #1: Type: text/plain, Size: 613 bytes --]

>
> I would add one more important impact: making data a first class component
> of the computing environment.
>

Related to that is getting rid of "file types". I remember at a "lessons of
Unix" panel at a Usenix, that someone from the Labs gave an example of a
situation where the operating system treated a file as more than just a set
of bytes (think IBM, VMS, etc).  Read a source file (required them to do a
bunch of file-conversion operations), write an output, perhaps changing a
variable name from 'foo' to 'bar', and then compiling the output, which
also required a bunch of file-conversion operations.

[-- Attachment #2: Type: text/html, Size: 970 bytes --]

[TUHS] Re: After 50 years, what has the Impact of Unix been?

[John Levine]

It appears that Marc Donner <marc.donner@gmail.com> said:
>With the notion of pipes it became possible to operate on data quickly and
>flexibly.  There was nothing new from a fundamental capability point of
>view, but the ease with which one could construct pipelines enabled rapid
>experimentation and encouraged the development of pipe-able components to
>add to the tool set.

Pipes were invented at least three times I'm aware of, but what made them
work so well in Unix is that they looked to the program the same as a file
so any program could use them for input or output without special arrangements,
and the shell made it easy to start two programs and pipe them together.

The Dartmouth Time-Sharing System in the late 1960s had communication files
which were essentially two-way pipes, but they were asymmetrical. One end, the
slave end, looked like a file, but the other end, the master end, was different
and the program had to know it was a com file. They were mostly used to pass
terminal I/O between user programs at the slave end and SIMON at the master end,
the terminal monitor that talked to the front end computer than ran the TTYs.

They were invented again at IBM in the 1970s and described in this paper.  I wrote
them a letter, which they published, saying that Unix pipes did the same thing.

https://dl.acm.org/doi/10.1147/sj.174.0383

R's,
John

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[Dan Cross]

On Wed, Dec 4, 2024 at 10:18 PM John Levine <johnl@taugh.com> wrote:
> It appears that Marc Donner <marc.donner@gmail.com> said:
> >With the notion of pipes it became possible to operate on data quickly and
> >flexibly.  There was nothing new from a fundamental capability point of
> >view, but the ease with which one could construct pipelines enabled rapid
> >experimentation and encouraged the development of pipe-able components to
> >add to the tool set.
>
> Pipes were invented at least three times I'm aware of, but what made them
> work so well in Unix is that they looked to the program the same as a file
> so any program could use them for input or output without special arrangements,
> and the shell made it easy to start two programs and pipe them together.

Once you have coroutines and queues for passing data between them, a
lot of things start to look like pipes.

> The Dartmouth Time-Sharing System in the late 1960s had communication files
> which were essentially two-way pipes, but they were asymmetrical. One end, the
> slave end, looked like a file, but the other end, the master end, was different
> and the program had to know it was a com file. They were mostly used to pass
> terminal I/O between user programs at the slave end and SIMON at the master end,
> the terminal monitor that talked to the front end computer than ran the TTYs.

Doug has written at some length on this list about communication files
and their homomorphism to the way Plan 9 presented and handled
resources.

> They were invented again at IBM in the 1970s and described in this paper.  I wrote
> them a letter, which they published, saying that Unix pipes did the same thing.
>
> https://dl.acm.org/doi/10.1147/sj.174.0383

Don't forget CMS pipelines, too!

Sadly, the Morrison paper cited above is not easily accessible, though
I acquired a copy from IEEE; perhaps a sucker's game as it was not
cheap. Your subsequent letter, and part of Morrison's response to you,
however, is available, gratis.

Reading through Morrison, one gets the impression that there are some
substantial differences with Unix pipelines; or rather, the way that
Unix pipelines are usually used. In particular, he describes his
linked streams in terms of a "network", by which he appears to mean an
arbitrary directed graph. Crucially, he describes combining nodes, for
merging data from multiple streams.

Unix pipelines, on the other hand, tend to be used in a manner that is
strictly linear, without the fan-out and fan-in capabilities described
by Morrison. Of course, nothing prevents one from building a
Morrison-style "network" from Unix processes and pipes, though it's
hard to see how that would work without something like `select`, which
didn't yet exist in 1978. Regardless, Unix still doesn't expose a
particularly convenient syntax for expressing these sorts of
constructions at the shell.

As an aside, Morrison has a web page dedicated to "flow-based
programming", that he claims to have invented in the late 1960s.  That
seems like a bit of a tall claim, and I'd wager Doug gives him a run
for his money on that.  https://jpaulm.github.io/fbp/

        - Dan C.

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[John R Levine]

On Thu, 5 Dec 2024, Dan Cross wrote:
>> Pipes were invented at least three times I'm aware of, but what made them
>> work so well in Unix is that they looked to the program the same as a file
>> so any program could use them for input or output without special arrangements,
>> and the shell made it easy to start two programs and pipe them together.
>
> Once you have coroutines and queues for passing data between them, a
> lot of things start to look like pipes.

They also can look a lot like temporary files.  Someone, probably Heinz, 
did a shell for the tiny Unix that ran on floppies so this

  foo | bar

actually did this

  foo > tmpfile ; bar < tmpfile; rm tmpfile

to avoid having to swap programs in and out on floppies.  The main 
disadvantage was that the tmpfile could overflow the tiny disks of the 
time.

>> They were invented again at IBM in the 1970s and described in this paper.  I wrote
>> them a letter, which they published, saying that Unix pipes did the same thing.
>>
>> https://dl.acm.org/doi/10.1147/sj.174.0383
>
> Don't forget CMS pipelines, too!
>
> Sadly, the Morrison paper cited above is not easily accessible, though

If anyone else needs a copy, just ask.

Regards,
John Levine, johnl@taugh.com, Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[Heinz Lycklama]

[-- Attachment #1: Type: text/plain, Size: 1747 bytes --]

John, thanks for the reminder of the implementation
of pipes on a constrained version of UNIX in the early
days. The exact implementation is described on page 2095
of the BSTJ July-Aug 1978 for interested parties.



Heinz

On 12/5/2024 8:00 AM, John R Levine wrote:
> On Thu, 5 Dec 2024, Dan Cross wrote:
>>> Pipes were invented at least three times I'm aware of, but what made 
>>> them
>>> work so well in Unix is that they looked to the program the same as 
>>> a file
>>> so any program could use them for input or output without special 
>>> arrangements,
>>> and the shell made it easy to start two programs and pipe them 
>>> together.
>>
>> Once you have coroutines and queues for passing data between them, a
>> lot of things start to look like pipes.
>
> They also can look a lot like temporary files.  Someone, probably 
> Heinz, did a shell for the tiny Unix that ran on floppies so this
>
>  foo | bar
>
> actually did this
>
>  foo > tmpfile ; bar < tmpfile; rm tmpfile
>
> to avoid having to swap programs in and out on floppies.  The main 
> disadvantage was that the tmpfile could overflow the tiny disks of the 
> time.
>
>>> They were invented again at IBM in the 1970s and described in this 
>>> paper.  I wrote
>>> them a letter, which they published, saying that Unix pipes did the 
>>> same thing.
>>>
>>> https://dl.acm.org/doi/10.1147/sj.174.0383
>>
>> Don't forget CMS pipelines, too!
>>
>> Sadly, the Morrison paper cited above is not easily accessible, though
>
> If anyone else needs a copy, just ask.
>
> Regards,
> John Levine, johnl@taugh.com, Taughannock Networks, Trumansburg NY
> Please consider the environment before reading this e-mail. https://jl.ly

[-- Attachment #2.1: Type: text/html, Size: 3216 bytes --]

[-- Attachment #2.2: RGxkomgHdfzyhAUv.png --]
[-- Type: image/png, Size: 138780 bytes --]

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[Adam Thornton]

[-- Attachment #1: Type: text/plain, Size: 894 bytes --]

On Thu, Dec 5, 2024 at 8:20 AM Dan Cross <crossd@gmail.com> wrote:

>
> Unix pipelines, on the other hand, tend to be used in a manner that is
> strictly linear, without the fan-out and fan-in capabilities described
> by Morrison. Of course, nothing prevents one from building a
> Morrison-style "network" from Unix processes and pipes, though it's
> hard to see how that would work without something like `select`, which
> didn't yet exist in 1978. Regardless, Unix still doesn't expose a
> particularly convenient syntax for expressing these sorts of
> constructions at the shell.
>
>
Rick Troth has recently published xfl, which is pretty much CMS Pipelines
for Unix.

https://github.com/trothtech/xfl

He's got a slide deck on it at
http://www.casita.net/pub/xfl/pervasive-vmws-2024.pdf .

There are a lot of really cool things you can do with fanin/fanout.

Adam

[-- Attachment #2: Type: text/html, Size: 1464 bytes --]

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[Marc Rochkind]

[-- Attachment #1.1: Type: text/plain, Size: 2494 bytes --]

I don't think files as pipes would be "transparent to the user." Reading an
empty pipe causes a wait until the bytes requested are available, unless
the pipe is closed first. Reading to the end of a file results in an
end-of-file error. This problem is avoided if the source process completes
before the target process begins, but then there is a different lack of
transparency, which is that the processes don't run simultaneously. (I
think this is the case with the implementation that Heinz showed.)

Still, the same sort of pseudo-pipes were in MS-DOS, and they were
occasionally useful.

Marc

On Thu, Dec 5, 2024 at 9:17 AM Heinz Lycklama <heinz@osta.com> wrote:

> John, thanks for the reminder of the implementation
> of pipes on a constrained version of UNIX in the early
> days. The exact implementation is described on page 2095
> of the BSTJ July-Aug 1978 for interested parties.
>
>
>
> Heinz
>
> On 12/5/2024 8:00 AM, John R Levine wrote:
>
> On Thu, 5 Dec 2024, Dan Cross wrote:
>
> Pipes were invented at least three times I'm aware of, but what made them
> work so well in Unix is that they looked to the program the same as a file
> so any program could use them for input or output without special
> arrangements,
> and the shell made it easy to start two programs and pipe them together.
>
>
> Once you have coroutines and queues for passing data between them, a
> lot of things start to look like pipes.
>
>
> They also can look a lot like temporary files.  Someone, probably Heinz,
> did a shell for the tiny Unix that ran on floppies so this
>
>  foo | bar
>
> actually did this
>
>  foo > tmpfile ; bar < tmpfile; rm tmpfile
>
> to avoid having to swap programs in and out on floppies.  The main
> disadvantage was that the tmpfile could overflow the tiny disks of the
> time.
>
> They were invented again at IBM in the 1970s and described in this paper.
> I wrote
> them a letter, which they published, saying that Unix pipes did the same
> thing.
>
> https://dl.acm.org/doi/10.1147/sj.174.0383
>
>
> Don't forget CMS pipelines, too!
>
> Sadly, the Morrison paper cited above is not easily accessible, though
>
>
> If anyone else needs a copy, just ask.
>
> Regards,
> John Levine, johnl@taugh.com, Taughannock Networks, Trumansburg NY
> Please consider the environment before reading this e-mail. https://jl.ly
>
>
>

-- 
*My new email address is mrochkind@gmail.com <mrochkind@gmail.com>*

[-- Attachment #1.2: Type: text/html, Size: 4265 bytes --]

[-- Attachment #2: RGxkomgHdfzyhAUv.png --]
[-- Type: image/png, Size: 138780 bytes --]

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[Paul Winalski]

[-- Attachment #1: Type: text/plain, Size: 697 bytes --]

Regarding pipes and pipe-like interprocess communications facilities in
other operating systems, VMS has always had a pipe-like communications
pseudo-devices called mailboxes.  The main difference between Unix pipes
and VMS mailboxes is that pipes have distinct read-only and write-only file
descriptors.  Mailboxes do not--channels (VMS-speak for file descriptors)
assigned to a mailbox can be used for both reading and writing.  This means
that it is not possible to do "broken pipe"-type detection on a mailbox.
This very much restricts the usefulness of mailboxes.

I wrote a true pipe device driver for VMS as part of the DEC Shell product
(a port of the Unix Bourne shell to VMS).

-Paul W.

[-- Attachment #2: Type: text/html, Size: 816 bytes --]

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[Chet Ramey via TUHS]

[-- Attachment #1.1: Type: text/plain, Size: 1038 bytes --]

On 12/5/24 10:19 AM, Dan Cross wrote:

> Unix pipelines, on the other hand, tend to be used in a manner that is
> strictly linear, without the fan-out and fan-in capabilities described
> by Morrison. Of course, nothing prevents one from building a
> Morrison-style "network" from Unix processes and pipes, though it's
> hard to see how that would work without something like `select`, which
> didn't yet exist in 1978. Regardless, Unix still doesn't expose a
> particularly convenient syntax for expressing these sorts of
> constructions at the shell.

Process substitution is about as close as we can get, but most programs
still process their filename arguments one at a time, beginning to end.

The canonical process substitution example is

	diff <(old-program-version) <(new-program-version)

to do simple regression testing.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
		 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 203 bytes --]

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[John Cowan]

[-- Attachment #1.1: Type: text/plain, Size: 2889 bytes --]

There was no concurrency in mini-Unix; the first pipeline component ran to
completion, and when it exited the next component ran, and so on.  In this
way a pipeline could be arbitrarily long using just two disk files.

On Thu, Dec 5, 2024, 12:07 PM Marc Rochkind <mrochkind@gmail.com> wrote:

> I don't think files as pipes would be "transparent to the user." Reading
> an empty pipe causes a wait until the bytes requested are available, unless
> the pipe is closed first. Reading to the end of a file results in an
> end-of-file error. This problem is avoided if the source process completes
> before the target process begins, but then there is a different lack of
> transparency, which is that the processes don't run simultaneously. (I
> think this is the case with the implementation that Heinz showed.)
>
> Still, the same sort of pseudo-pipes were in MS-DOS, and they were
> occasionally useful.
>
> Marc
>
> On Thu, Dec 5, 2024 at 9:17 AM Heinz Lycklama <heinz@osta.com> wrote:
>
>> John, thanks for the reminder of the implementation
>> of pipes on a constrained version of UNIX in the early
>> days. The exact implementation is described on page 2095
>> of the BSTJ July-Aug 1978 for interested parties.
>>
>>
>>
>> Heinz
>>
>> On 12/5/2024 8:00 AM, John R Levine wrote:
>>
>> On Thu, 5 Dec 2024, Dan Cross wrote:
>>
>> Pipes were invented at least three times I'm aware of, but what made them
>> work so well in Unix is that they looked to the program the same as a
>> file
>> so any program could use them for input or output without special
>> arrangements,
>> and the shell made it easy to start two programs and pipe them together.
>>
>>
>> Once you have coroutines and queues for passing data between them, a
>> lot of things start to look like pipes.
>>
>>
>> They also can look a lot like temporary files.  Someone, probably Heinz,
>> did a shell for the tiny Unix that ran on floppies so this
>>
>>  foo | bar
>>
>> actually did this
>>
>>  foo > tmpfile ; bar < tmpfile; rm tmpfile
>>
>> to avoid having to swap programs in and out on floppies.  The main
>> disadvantage was that the tmpfile could overflow the tiny disks of the
>> time.
>>
>> They were invented again at IBM in the 1970s and described in this
>> paper.  I wrote
>> them a letter, which they published, saying that Unix pipes did the same
>> thing.
>>
>> https://dl.acm.org/doi/10.1147/sj.174.0383
>>
>>
>> Don't forget CMS pipelines, too!
>>
>> Sadly, the Morrison paper cited above is not easily accessible, though
>>
>>
>> If anyone else needs a copy, just ask.
>>
>> Regards,
>> John Levine, johnl@taugh.com, Taughannock Networks, Trumansburg NY
>> Please consider the environment before reading this e-mail. https://jl.ly
>>
>>
>>
>
> --
> *My new email address is mrochkind@gmail.com <mrochkind@gmail.com>*
>

[-- Attachment #1.2: Type: text/html, Size: 4921 bytes --]

[-- Attachment #2: RGxkomgHdfzyhAUv.png --]
[-- Type: image/png, Size: 138780 bytes --]

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[John Levine]

It appears that Marc Rochkind <mrochkind@gmail.com> said:
>-=-=-=-=-=-
>-=-=-=-=-=-
>
>I don't think files as pipes would be "transparent to the user." Reading an
>empty pipe causes a wait until the bytes requested are available, unless
>the pipe is closed first. ...

You should read Meinz' paper.  This was a cut down version of Unix that ran
in 40K bytes on an LSI-11 with a couple of floppy disks.  It ran one program
at a time, since swapping to floppies was absurdly slow.  The pseudo-pipe
ran the first program which wrote the output to a file, then ran the second
program which read it.  It was close enough for most purposes so long as
the disk didn't fill up.

>Still, the same sort of pseudo-pipes were in MS-DOS, and they were
>occasionally useful.

I can think of lots of places that had a way for one program to write a temporary
file that a subsequent one read.  For example, in OS/360 JCL you'd write (well, punch)
something like this for the object output of the Fortran compiler

//SYSLIN DD DSNAME=&OBJECT,DISP=(NEW,PASS),UNIT=SYSSQ

and then in the linker read it in

//SYSIN DD DSNAME=&OBJECT,DISP=(OLD,DELETE)

The & in the name said it was a temp file so make up a unique name.  I probably didn't
get the JCL exactly right since it's been about 50 years since I wrote any.

R's,
John

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[Ron Natalie]

I remember on MiniUnix (a stripped down kernel that would run on PDP11’s 
without memory management, in our case 11/03, 11/20, and an older 11/40) 
there were no kernel pipes (and limited process concurrency).
The shell used the aforementioned emulation of pipe by just running the 
output of one process into a temporary file subsequently loaded as stdin 
of the other.

We kind of put all that to bed when the 11/23s came out and we could run 
our full up kernel on the micros.

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[arnold]

Chet Ramey via TUHS <tuhs@tuhs.org> wrote:

> On 12/5/24 10:19 AM, Dan Cross wrote:
>
> > Unix pipelines, on the other hand, tend to be used in a manner that is
> > strictly linear, without the fan-out and fan-in capabilities described
> > by Morrison. Of course, nothing prevents one from building a
> > Morrison-style "network" from Unix processes and pipes, though it's
> > hard to see how that would work without something like `select`, which
> > didn't yet exist in 1978. Regardless, Unix still doesn't expose a
> > particularly convenient syntax for expressing these sorts of
> > constructions at the shell.
>
> Process substitution is about as close as we can get, but most programs
> still process their filename arguments one at a time, beginning to end.
>
> The canonical process substitution example is
>
> 	diff <(old-program-version) <(new-program-version)
>
> to do simple regression testing.

And fanout is simply

	... | tee >(pipeline1) >(pipeline2)

Arnold

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[Dan Cross]

On Thu, Dec 5, 2024 at 3:56 PM <arnold@skeeve.com> wrote:
> Chet Ramey via TUHS <tuhs@tuhs.org> wrote:
> > On 12/5/24 10:19 AM, Dan Cross wrote:
> >
> > > Unix pipelines, on the other hand, tend to be used in a manner that is
> > > strictly linear, without the fan-out and fan-in capabilities described
> > > by Morrison. Of course, nothing prevents one from building a
> > > Morrison-style "network" from Unix processes and pipes, though it's
> > > hard to see how that would work without something like `select`, which
> > > didn't yet exist in 1978. Regardless, Unix still doesn't expose a
> > > particularly convenient syntax for expressing these sorts of
> > > constructions at the shell.
> >
> > Process substitution is about as close as we can get, but most programs
> > still process their filename arguments one at a time, beginning to end.
> >
> > The canonical process substitution example is
> >
> >       diff <(old-program-version) <(new-program-version)
> >
> > to do simple regression testing.
>
> And fanout is simply
>
>         ... | tee >(pipeline1) >(pipeline2)

And indeed these things are pretty nifty, but don't they generate
trees, and not arbitrary dags? They don't quite capture the full
generality of Morrison-style networks since it doesn't seem like
there's a way to connect process substitution fan-out with fan-in; at
least, not conveniently.

It is, perhaps, notable that Go allows me to do this sort of thing
with channels and goroutines, but it has `select` built into the
language.

Funny, despite using Unix almost daily for over 30 years now, I don't
think I've ever felt limited by the power of pipelines. On the
contrary, I've lost count of the times I've felt limited on systems
that do Not support pipes.

        - Dan C.

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[Marc Rochkind]

[-- Attachment #1.1: Type: text/plain, Size: 3408 bytes --]

Some of the posts in this thread confuse pipes in UNIX shells (the |
symbol) with the pipe system call.

In the shell, how two processes can be connected with a pipe is very
constrained (only one unidirectional pipe). But the pipe system call can be
used to build much more elaborate connections. Back in 1980, when I was
still at Bell Labs, I wrote a shell called 2dsh ("two dimensional shell)
that had a more complex syntax. The memo I wrote, "2DSH—An experimental
shell for connecting processes with multiple data streams",  wasn't
published externally, but exists as a Bell Labs memo. I found a reference
here:
https://scholar.google.com/scholar_lookup?title=2DSH%E2%80%94An+experimental+shell+for+connecting+processes+with+multiple+data+streams&author=M.+J.+Rochkind&publication_year=1980
.

Here are two examples from that memo:
[image: image.png]

I stumbled across another paper from 2017 titled "Extending Unix Pipelines
to DAGs," which references my un-published Bell Labs memo. I haven't read
it since I don't subscribe to IEEE Transactions on Computers.

A while ago Doug McIlroy was kind enough to send me a scan of my memo, but
I don't think I'm allowed to publish it here. In that memo, I credit Doug
for coming with a very similar idea around the same time ("A Notation for
Arboreal Plumbing").

Marc Rochkind

On Thu, Dec 5, 2024 at 2:13 PM Dan Cross <crossd@gmail.com> wrote:

> On Thu, Dec 5, 2024 at 3:56 PM <arnold@skeeve.com> wrote:
> > Chet Ramey via TUHS <tuhs@tuhs.org> wrote:
> > > On 12/5/24 10:19 AM, Dan Cross wrote:
> > >
> > > > Unix pipelines, on the other hand, tend to be used in a manner that
> is
> > > > strictly linear, without the fan-out and fan-in capabilities
> described
> > > > by Morrison. Of course, nothing prevents one from building a
> > > > Morrison-style "network" from Unix processes and pipes, though it's
> > > > hard to see how that would work without something like `select`,
> which
> > > > didn't yet exist in 1978. Regardless, Unix still doesn't expose a
> > > > particularly convenient syntax for expressing these sorts of
> > > > constructions at the shell.
> > >
> > > Process substitution is about as close as we can get, but most programs
> > > still process their filename arguments one at a time, beginning to end.
> > >
> > > The canonical process substitution example is
> > >
> > >       diff <(old-program-version) <(new-program-version)
> > >
> > > to do simple regression testing.
> >
> > And fanout is simply
> >
> >         ... | tee >(pipeline1) >(pipeline2)
>
> And indeed these things are pretty nifty, but don't they generate
> trees, and not arbitrary dags? They don't quite capture the full
> generality of Morrison-style networks since it doesn't seem like
> there's a way to connect process substitution fan-out with fan-in; at
> least, not conveniently.
>
> It is, perhaps, notable that Go allows me to do this sort of thing
> with channels and goroutines, but it has `select` built into the
> language.
>
> Funny, despite using Unix almost daily for over 30 years now, I don't
> think I've ever felt limited by the power of pipelines. On the
> contrary, I've lost count of the times I've felt limited on systems
> that do Not support pipes.
>
>         - Dan C.
>


-- 
*My new email address is mrochkind@gmail.com <mrochkind@gmail.com>*

[-- Attachment #1.2: Type: text/html, Size: 4757 bytes --]

[-- Attachment #2: image.png --]
[-- Type: image/png, Size: 368482 bytes --]

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 2077 bytes --]

On Thu, Dec 5, 2024 at 2:13 PM Dan Cross <crossd@gmail.com> wrote:

> On Thu, Dec 5, 2024 at 3:56 PM <arnold@skeeve.com> wrote:
> > Chet Ramey via TUHS <tuhs@tuhs.org> wrote:
> > > On 12/5/24 10:19 AM, Dan Cross wrote:
> > >
> > > > Unix pipelines, on the other hand, tend to be used in a manner that
> is
> > > > strictly linear, without the fan-out and fan-in capabilities
> described
> > > > by Morrison. Of course, nothing prevents one from building a
> > > > Morrison-style "network" from Unix processes and pipes, though it's
> > > > hard to see how that would work without something like `select`,
> which
> > > > didn't yet exist in 1978. Regardless, Unix still doesn't expose a
> > > > particularly convenient syntax for expressing these sorts of
> > > > constructions at the shell.
> > >
> > > Process substitution is about as close as we can get, but most programs
> > > still process their filename arguments one at a time, beginning to end.
> > >
> > > The canonical process substitution example is
> > >
> > >       diff <(old-program-version) <(new-program-version)
> > >
> > > to do simple regression testing.
> >
> > And fanout is simply
> >
> >         ... | tee >(pipeline1) >(pipeline2)
>
> And indeed these things are pretty nifty, but don't they generate
> trees, and not arbitrary dags? They don't quite capture the full
> generality of Morrison-style networks since it doesn't seem like
> there's a way to connect process substitution fan-out with fan-in; at
> least, not conveniently.
>
> It is, perhaps, notable that Go allows me to do this sort of thing
> with channels and goroutines, but it has `select` built into the
> language.
>
> Funny, despite using Unix almost daily for over 30 years now, I don't
> think I've ever felt limited by the power of pipelines. On the
> contrary, I've lost count of the times I've felt limited on systems
> that do Not support pipes.
>

The <() , >()  syntax is a bash extension. Not all shells support it. And
I couldn't find them in POSIX Issue 8.

Warner

[-- Attachment #2: Type: text/html, Size: 2879 bytes --]

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[Bakul Shah via TUHS]

On Dec 5, 2024, at 1:13 PM, Dan Cross <crossd@gmail.com> wrote:
> 
> And indeed these things are pretty nifty, but don't they generate
> trees, and not arbitrary dags? They don't quite capture the full
> generality of Morrison-style networks since it doesn't seem like
> there's a way to connect process substitution fan-out with fan-in; at
> least, not conveniently.

Isn’t that a limitation of the shell language(s)? It would be
easier in a 2D graphical shell but that would perhaps make
specifying the more common simpler pipelines harder!

A "linear" syntax would require *naming*. For example,

let-proc a = "foo -a", b = "bar -c -d", c = "zee" in
  a.in1 = file1, a.in2 = file2, b.in1 = a.out, b.in2 = file3,
  c.in1 = b.out1, c.in2 = a.out, c.out = file4;

Here names a, b & c are used to denote *processes*, with
connections specified in the next two lines. Once all the
pipes are created, processes a, b & c can be started giving
them the right endpoints in forked processes prior to execing.
Doable but not really worth it. At this point you may as well
use s-expressions!

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[Chet Ramey via TUHS]

[-- Attachment #1.1: Type: text/plain, Size: 490 bytes --]

On 12/5/24 5:03 PM, Warner Losh wrote:

> The <() , >()  syntax is a bash extension. Not all shells support it. And
> I couldn't find them in POSIX Issue 8.

Credit where credit is due: I picked them up from ksh93, and extended
them to use named pipes on systems where /dev/fd isn't available.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
		 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 203 bytes --]

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[arnold]

Warner Losh <imp@bsdimp.com> wrote:

> The <() , >()  syntax is a bash extension. Not all shells support it. And
> I couldn't find them in POSIX Issue 8.
>
> Warner

It originated in ksh93. 30 years have gone by, it's time it
was added to POSIX.

At least, IMHO.

Thanks,

Arnold

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[Marc Rochkind]

[-- Attachment #1: Type: text/plain, Size: 992 bytes --]

I found that 2017 paper "Extending Unix Pipelines to DAGs". It's open
access:

https://ieeexplore.ieee.org/document/7903579

The open source code itself is here: https://github.com/dspinellis/dgsh

Maybe an ambitious TUHS contributor can get the code running and give us a
report.

Marc Rochkind

On Thu, Dec 5, 2024 at 3:38 PM Chet Ramey via TUHS <tuhs@tuhs.org> wrote:

> On 12/5/24 5:03 PM, Warner Losh wrote:
>
> > The <() , >()  syntax is a bash extension. Not all shells support it. And
> > I couldn't find them in POSIX Issue 8.
>
> Credit where credit is due: I picked them up from ksh93, and extended
> them to use named pipes on systems where /dev/fd isn't available.
>
> --
> ``The lyf so short, the craft so long to lerne.'' - Chaucer
>                  ``Ars longa, vita brevis'' - Hippocrates
> Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/
>


-- 
*My new email address is mrochkind@gmail.com <mrochkind@gmail.com>*

[-- Attachment #2: Type: text/html, Size: 1894 bytes --]

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[Alexis]

Chet Ramey via TUHS <tuhs@tuhs.org> writes:

> On 12/5/24 5:03 PM, Warner Losh wrote:
>
>> The <() , >()  syntax is a bash extension. Not all shells 
>> support
>> it. And
>> I couldn't find them in POSIX Issue 8.
>
> Credit where credit is due: I picked them up from ksh93, and 
> extended
> them to use named pipes on systems where /dev/fd isn't 
> available.

i must say i was moderately surprised to learn a few years ago 
that process substitution wasn't in POSIX - i find it succinct 
syntactic sugar. There's a post on Chris Siebenmann's blog about 
some of the issues involved, including the /dev/fd issue:

  https://utcc.utoronto.ca/~cks/space/blog/unix/ProcessSubstitutionWhyLate

Can anyone point me to any discussions about process substitution 
potentially becoming part of POSIX? This search on the Austin 
Group tracker didn't yield any useful results:

https://austingroupbugs.net/search.php?project_id=0&search=process+substitution&sticky_issues=off&sortby=last_updated&dir=DESC&hide_status_id=-2&FILTER_SEARCH_PLATFORM=


Alexis.

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[G. Branden Robinson]

[-- Attachment #1: Type: text/plain, Size: 362 bytes --]

At 2024-12-05T16:07:10-0700, arnold@skeeve.com wrote:
> Warner Losh <imp@bsdimp.com> wrote:
> > The <() , >()  syntax is a bash extension. Not all shells support
> > it. And I couldn't find them in POSIX Issue 8.
>
> It originated in ksh93.

Are you sure? I think Tom Duff originated it in his "rc" shell.

https://archive.org/details/rc-shell

Regards,
Branden

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[Greg A. Woods]

[-- Attachment #1: Type: text/plain, Size: 1075 bytes --]

At Thu, 5 Dec 2024 19:09:26 -0600, "G. Branden Robinson" <g.branden.robinson@gmail.com> wrote:
Subject: [TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)
>
> [1  <text/plain; us-ascii (7bit)>]
> At 2024-12-05T16:07:10-0700, arnold@skeeve.com wrote:
> > Warner Losh <imp@bsdimp.com> wrote:
> > > The <() , >()  syntax is a bash extension. Not all shells support
> > > it. And I couldn't find them in POSIX Issue 8.
> >
> > It originated in ksh93.
>
> Are you sure? I think Tom Duff originated it in his "rc" shell.

The Wikipedia entry for "Process substitution" says:

	Process substitution was available as a compile-time option for
	ksh88, the 1988 version of the KornShell from Bell Labs.  The rc
	shell provides the feature as "pipeline branching" in Version 10
	Unix, released in 1990.

So, if we believe that then indeed ksh88 pioneered process substitution.

--
					Greg A. Woods <gwoods@acm.org>

Kelowna, BC     +1 250 762-7675           RoboHack <woods@robohack.ca>
Planix, Inc. <woods@planix.com>     Avoncote Farms <woods@avoncote.ca>

[-- Attachment #2: OpenPGP Digital Signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[John Levine]

According to Dan Cross <crossd@gmail.com>:
>> >       diff <(old-program-version) <(new-program-version)
>> >
>> > to do simple regression testing.
>>
>> And fanout is simply
>>
>>         ... | tee >(pipeline1) >(pipeline2)
>
>And indeed these things are pretty nifty, but don't they generate
>trees, and not arbitrary dags? They don't quite capture the full
>generality of Morrison-style networks since it doesn't seem like
>there's a way to connect process substitution fan-out with fan-in; at
>least, not conveniently.

You can use mkfifo to make named pipes and then plumb them as wanted.  It's
not particularly beautiful but it's as general as you could want.

That's worked since 4.4BSD which was quite a while ago.

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[Steve Nickolas]

On Thu, 5 Dec 2024, Greg A. Woods wrote:

> The Wikipedia entry for "Process substitution" says:
>
> 	Process substitution was available as a compile-time option for
> 	ksh88, the 1988 version of the KornShell from Bell Labs.  The rc
> 	shell provides the feature as "pipeline branching" in Version 10
> 	Unix, released in 1990.
>
> So, if we believe that then indeed ksh88 pioneered process substitution.

Confirming that ksh88d and ksh88i support <(command) at least.

-uso.

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[Dan Cross]

On Thu, Dec 5, 2024 at 9:02 PM John Levine <johnl@taugh.com> wrote:
> According to Dan Cross <crossd@gmail.com>:
> >> >       diff <(old-program-version) <(new-program-version)
> >> >
> >> > to do simple regression testing.
> >>
> >> And fanout is simply
> >>
> >>         ... | tee >(pipeline1) >(pipeline2)
> >
> >And indeed these things are pretty nifty, but don't they generate
> >trees, and not arbitrary dags? They don't quite capture the full
> >generality of Morrison-style networks since it doesn't seem like
> >there's a way to connect process substitution fan-out with fan-in; at
> >least, not conveniently.
>
> You can use mkfifo to make named pipes and then plumb them as wanted.  It's
> not particularly beautiful but it's as general as you could want.
>
> That's worked since 4.4BSD which was quite a while ago.

Sure, but that's orthogonal since it's not really using shell syntax
to build the thing up in a canonical way. I mean, perhaps in some
literal sense it is (one types the commands into the interpreter...)
but nowhere near the expressiveness of `ls | grep foo | whatever`.

Fundamentally, the questions here are all about syntax.  As I wrote in
my first email, the system calls are very general, and could be used
to build arbitrary directed graphs (not even limited to DAGs) of
processes connected by pipes in all sorts of configurations
(presumably those processes would be written to handle multiple input
and output streams), without having to resort to named pipes. One can
imagine some sort of DSL that describes such graphs of pipes and
processes (Bakul alluded to this), and an interpreter for that DSL
that creates the described set of pipes and processes, all
appropriately connected and running. One could even imagine such a
thing being used as part of a more traditional pipeline.

My claim is simply that extant shell syntax isn't really amenable to
this in a natural way. As Chet and Arnold pointed out, process
substitution as pioneered in ksh gets us a little closer, but it's not
quite as general (I believe any such graphs would have to be acyclic);
it's certainly not as syntactically pleasant.

There has been work along these lines; I was sent a reference off-list
to a paper by Spinellis and Fragkoulis about a DAG-oriented shell:
https://www.spinellis.gr/sw/dgsh/, which seems relevant.

        - Dan C.

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[Adam Thornton]

[-- Attachment #1: Type: text/plain, Size: 651 bytes --]

/23+, no?  The basic 23 doesn't have an MMU, I'm pretty sure.

On Thu, Dec 5, 2024 at 11:58 AM Ron Natalie <ron@ronnatalie.com> wrote:

> I remember on MiniUnix (a stripped down kernel that would run on PDP11’s
> without memory management, in our case 11/03, 11/20, and an older 11/40)
> there were no kernel pipes (and limited process concurrency).
> The shell used the aforementioned emulation of pipe by just running the
> output of one process into a temporary file subsequently loaded as stdin
> of the other.
>
> We kind of put all that to bed when the 11/23s came out and we could run
> our full up kernel on the micros.
>
>

[-- Attachment #2: Type: text/html, Size: 1005 bytes --]

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[Diomidis Spinellis]

On 06-Dec-24 01:07, Marc Rochkind wrote:
> I found that 2017 paper "Extending Unix Pipelines to DAGs". It's open 
> access:
> 
> https://ieeexplore.ieee.org/document/7903579 <https:// 
> ieeexplore.ieee.org/document/7903579>
> 
> The open source code itself is here: https://github.com/dspinellis/dgsh 
> <https://github.com/dspinellis/dgsh>
> 
> Maybe an ambitious TUHS contributor can get the code running and give us 
> a report.

I wrote the dgsh code with my co-author Marios Fragkoulis, so I still 
have it running.  Doug McIlroy, who also mentioned dgsh in another 
message, is too modest to say that its design owes much to his input.  I 
asked him for feedback when I was working on it, and over several 
iterations he proposed important (and quite demanding as I recall) 
improvements to its design.

The system allows the concise and readable expression of several graph 
topologies I had in mind when I started working on it, and more [1]. 
However, it hasn't caught on.  I think the main reason is that it is 
based on modified versions of several existing tools (bash, cmp, comm, 
cut, diff, diff3, grep, join, paste, perm, sort) [2].  The modifications 
allow the tools to coordinate between them the setup of pipes when 
placed in a dgsh graph according to available inputs and required 
outputs.  The changes (especially for bash) aren't small, which meant 
that I didn't think it was realistic to push them upstream, which now 
means that the modified tools are out of date and difficult to build. 
Not sure what can be done to address this problem.  It seems that a 
widely adopted system, such as modern Unix/Linux, has too much inertia 
for it to adopt potentially disrupting innovations.

In retrospect, the way we designed the pipe graph setup could also be 
improved.  The current design involves an initial phase where IPC 
messages are circulating around the graph to communicate the I/O 
requirements of each tool, for example that comm(1) should expect input 
from two processes and output to three processes.  The design is brittle 
and difficult to troubleshoot, because coordination happens dynamically 
behind the scenes.  A better design (and one I think Doug was 
advocating) would statically analyze the graph's topology and invoke 
each tool with appropriate parameters or environment variables. 
However, this design would require significantly more extensive 
modifications to bash, or the implementation of a new shell.  Both 
approaches required work for which we didn't have the time and energy at 
the time, and also had their own downsides regarding adoption potential.

[1] https://www.spinellis.gr/sw/dgsh/#examples
[2] https://www.spinellis.gr/sw/dgsh/#tools

Diomidis

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[arnold]

"Greg A. Woods" <woods@robohack.ca> wrote:

> > Are you sure? I think Tom Duff originated it in his "rc" shell.
>
> The Wikipedia entry for "Process substitution" says:
>
> 	Process substitution was available as a compile-time option for
> 	ksh88, the 1988 version of the KornShell from Bell Labs.  The rc
> 	shell provides the feature as "pipeline branching" in Version 10
> 	Unix, released in 1990.
>
> So, if we believe that then indeed ksh88 pioneered process substitution.

I'd forgotten that it was optional in ksh88. It was there out of
the box in ksh93.

Arnold

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[arnold]

Dan Cross <crossd@gmail.com> wrote:

> My claim is simply that extant shell syntax isn't really amenable to
> this in a natural way. As Chet and Arnold pointed out, process
> substitution as pioneered in ksh gets us a little closer, but it's not
> quite as general (I believe any such graphs would have to be acyclic);
> it's certainly not as syntactically pleasant.
>
> There has been work along these lines; I was sent a reference off-list
> to a paper by Spinellis and Fragkoulis about a DAG-oriented shell:
> https://www.spinellis.gr/sw/dgsh/, which seems relevant.

In the early-to-mid 1980s, the Georgia Tech Software Tools Subsystem
for Prime Computers offered three standard inputs, three standard
outputs, and a shell with syntax to connect things as you desired.
I don't remember the syntax though.  I also don't know how much
this feature was really used.

Arnold

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[Chet Ramey via TUHS]

[-- Attachment #1.1: Type: text/plain, Size: 441 bytes --]

On 12/5/24 7:46 PM, Alexis wrote:

> Can anyone point me to any discussions about process substitution 
> potentially becoming part of POSIX? 

It's never come up in any serious way, maybe as a wish list item for
some people on the mailing list.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
		 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 203 bytes --]

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[Ron Natalie]

[-- Attachment #1: Type: text/plain, Size: 1122 bytes --]

NO, the 11/23 has an MMU and was perflectly capable of running  UNIX.   
The 23+ had the extra  address lines  allowing it to address 4 MB like 
the 11/70.


------ Original Message ------
From "Adam Thornton" <athornton@gmail.com>
To "Ron Natalie" <ron@ronnatalie.com>; "The Eunuchs Hysterical Society" 
<tuhs@tuhs.org>
Date 12/5/2024 9:29:24 PM
Subject Re: [TUHS] Re: Pipes (was Re: After 50 years, what has the 
Impact of Unix been?)

>/23+, no?  The basic 23 doesn't have an MMU, I'm pretty sure.
>
>On Thu, Dec 5, 2024 at 11:58 AM Ron Natalie <ron@ronnatalie.com> wrote:
>>I remember on MiniUnix (a stripped down kernel that would run on 
>>PDP11’s
>>without memory management, in our case 11/03, 11/20, and an older 
>>11/40)
>>there were no kernel pipes (and limited process concurrency).
>>The shell used the aforementioned emulation of pipe by just running 
>>the
>>output of one process into a temporary file subsequently loaded as 
>>stdin
>>of the other.
>>
>>We kind of put all that to bed when the 11/23s came out and we could 
>>run
>>our full up kernel on the micros.
>>

[-- Attachment #2: Type: text/html, Size: 2942 bytes --]

[TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)

[Douglas McIlroy]

Many tree and dag pipe notations have been proposed. Marc's was one of
the first. I devised one myself, but didn't like it enough to publish
it even as a TM. A recent one is Spinellis's dgsh. More elaborate
networks with feedback loops evolve for power-series computation. The
idea of implementing cellular automata as arrays of processes with
nearest neighbors connected by pipes was suggested early on, but I've
never seen such an arrangement implemented--it would be hideously
slow.

I once wrote a general plumber that worked from a connection list:
connect fd m in process A to fd n in process B. The main challenge was
to find an order of hooking things up so that the number of live file
descriptors in the plumber didn't exceed the system limit.

Doug