[TUHS] A few comments on porting the Bourne shell

[TUHS] A few comments on porting the Bourne shell

[Paul Ruizendaal]

London and Reiser report about porting the shell that “it required by far the largest conversion effort of any supposedly portable program, for the simple reason that it is not portable.” By the time of SysIII this is greatly improved, but also in porting the SysIII user land it was the most complex of the set so far.

There were three aspects that I found noteworthy:

1. London/Reiser apparently felt strongly about a property of casts. The code argues that casting an l-value should not convert it into a r-value:

<quote from "mode.h">
/* the following nonsense is required
 * because casts turn an Lvalue
 * into an Rvalue so two cheats
 * are necessary, one for each context.
 */
union { int _cheat;};
#define Lcheat(a)	((a)._cheat)
#define Rcheat(a)	((int)(a))
<endquote>

However, Lcheat is only used in two places (in service.c), to set and to clear a flag in a pointer. Interestingly, the 32V code already replaces one of these instances with a regular r-value cast. So far, I’d never thought about this aspect of casts. I stumbled across it, because the Plan 9 compiler did not accept the Lcheat expansion as valid C.

2. On the history of dup2

The shell code includes the following:

<quote from “io.c”>
rename(f1,f2)
	REG INT		f1, f2;
{
#ifdef RES	/*	research has different sys calls from TS	*/
	IF f1!=f2
	THEN	dup(f1|DUPFLG, f2);
		close(f1);
		IF f2==0 THEN ioset|=1 FI
	FI
#else
	INT	fs;
	IF	f1!=f2
	THEN 	fs = fcntl(f2,1,0);
		close(f2);
		fcntl(f1,0,f2);
		close(f1);
		IF fs==1 THEN fcntl(f2,2,1) FI
		IF f2==0 THEN ioset|=1 FI
	FI
#endif
}
<endquote>

I’ve check the 8th edition source, and indeed it supports using DUPFLG to signal to dup() that it really is dup2(). I had earlier wondered why dup2() did not appear in research until 10th edition, but now that is clear. It would seem that the dup of 8th edition is a direct ancestor to dup() in Plan 9. I wonder why this way of doing things never caught on in the other Unices.

3. Halfway to demand paging

I stumbled across this one because I had a bug in my signal handling. From early days onwards, Unix supported dynamically growing the stack allocation, which arguably is a first step towards building the mechanisms for demand paging. It appears that the Bourne shell made another step, catching page faults and expanding the data/bss allocation dynamically:

<quote from “fault.c”>
VOID	fault(sig)
	REG INT		sig;
{
	signal(sig, fault);
	IF sig==MEMF
	THEN	IF setbrk(brkincr) == -1
		THEN	error(nospace);
		FI
	ELIF ...
<endquote>

This was already present in 7th edition, so it is by no means new in 32V or SysIII -- it had just escaped my attention as a conceptual step in the development of Unix memory handling.

[TUHS] Re: A few comments on porting the Bourne shell

[Chet Ramey]

On 12/30/22 1:25 PM, Paul Ruizendaal wrote:
> 
> London and Reiser report about porting the shell that “it required by far the largest conversion effort of any supposedly portable program, for the simple reason that it is not portable.” By the time of SysIII this is greatly improved, but also in porting the SysIII user land it was the most complex of the set so far.

Have you read

http://www.collyer.net/who/geoff/sh.tour.pdf

and looked at http://www.collyer.net/who/geoff/v7sh.tar ?

In the limited literature on Bourne Shell porting, this is authoritative.

Arnold Robbins built on that work and ported the v8-v10 shells to modern
Linux versions. (I am sorry, I do not have a link right now.)

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
		 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/

[TUHS] Re: A few comments on porting the Bourne shell

[Larry McVoy]

On Fri, Dec 30, 2022 at 02:51:26PM -0500, Chet Ramey wrote:
> On 12/30/22 1:25 PM, Paul Ruizendaal wrote:
> >
> >London and Reiser report about porting the shell that ???it required by far the largest conversion effort of any supposedly portable program, for the simple reason that it is not portable.??? By the time of SysIII this is greatly improved, but also in porting the SysIII user land it was the most complex of the set so far.
> 
> Have you read
> 
> http://www.collyer.net/who/geoff/sh.tour.pdf
> 
> and looked at http://www.collyer.net/who/geoff/v7sh.tar ?
> 
> In the limited literature on Bourne Shell porting, this is authoritative.

Is there are reason to hang on to the Bourne shell?  Maybe shell scripts?
Does it perform better than ksh or bash?

Don't get me wrong, I much prefer the sh syntax over csh syntax, but 
I'd never go back to the Bourne shell as my login shell.  Way too much
useful stuff in ksh/bash.

[TUHS] Re: A few comments on porting the Bourne shell

[Adam Thornton]

> On Dec 30, 2022, at 1:02 PM, Larry McVoy <lm@mcvoy.com> wrote:
>> 
> 
> Is there are reason to hang on to the Bourne shell?  Maybe shell scripts?
> Does it perform better than ksh or bash?

POSIX sh seems like a good lowest-common-denominator shell to write for (sorry, SunOS and xpg4).

> 
> Don't get me wrong, I much prefer the sh syntax over csh syntax, but 
> I'd never go back to the Bourne shell as my login shell.  Way too much
> useful stuff in ksh/bash.

I mean I would assume the reason is that bash takes a whole lot more memory and CPU to do its very convenient magic.  That's an issue if you're trying to run on a PDP-11 and only have 16 bits of address space, but in 64-bit-world, where you almost certainly have at least half a gigabyte of core storage available, and your processor clock is almost certainly in at least the high hundreds of MHz...yeah.

Although (because my daily driver is a Mac) I've finally taken the zsh plunge.  I feel a little dirty, but oh-my-zsh certainly has its attractions.

Adam

[TUHS] Re: A few comments on porting the Bourne shell

[Chet Ramey]

On 12/30/22 3:31 PM, Adam Thornton wrote:

> Although (because my daily driver is a Mac) I've finally taken the zsh plunge.  

You don't have to, you know. ;-)

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
		 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/

[TUHS] Re: A few comments on porting the Bourne shell

[Sven Mascheck]

[-- Attachment #1: Type: text/plain, Size: 1601 bytes --]

Larry McVoy on 30.12.2022 21:02:

> > [SysIII port]
> Is there are reason to hang on to the Bourne shell?  Maybe shell scripts?
> Does it perform better than ksh or bash?
>
> Don't get me wrong, I much prefer the sh syntax over csh syntax, but
> I'd never go back to the Bourne shell as my login shell.  Way too much
> useful stuff in ksh/bash.

I'd like the idea of   "preserving a heirloom in its natural environment"
(and even more effort went in https://heirloom.sourceforge.net/sh.html)
let alone this does not prevent from adding modern shells...

I guess in interactive use most users would only miss one thing: the 
history & line editing capability?

Side notes to that:

  * By intention, the almquist shell (a port due to the Berkeley/ATT
    mess) initially had no history. From the package file DIFFERENCES [1],

    "History.   It seems to me that the csh history mechanism is mostly
    a response to the deficiencies of UNIX terminal I/O. Those of you
    running 4.2 BSD should try out atty (which I am posting to the net
    at the same time as ash) and see if you still want history."

  * and in "ksh - An Extensible High Level Language" David Korn writes:

    "Originally the idea of adding command line editing to ksh was
    rejected in the hope that line editing would move into the terminal
    driver." [2]

I have always wondered, what such a central terminal driver driven 
history/line-editing would have felt like.

[1] https://www.in-ulm.de/~mascheck/various/ash/DIFFERENCES
[2] https://www.in-ulm.de/~mascheck/bourne/korn.html

[-- Attachment #2: Type: text/html, Size: 2451 bytes --]

[TUHS] Re: A few comments on porting the Bourne shell

[Jon Steinhart]

Sven Mascheck writes:
> I guess in interactive use most users would only miss one thing: the
> history & line editing capability?

Job control?

[TUHS] Re: A few comments on porting the Bourne shell

[Sven Mascheck]

Jon Steinhart 30.12.2022 21:48:

>> I guess in interactive use most users would only miss one thing: the
>> history & line editing capability?
> Job control?
well spotted, not available in vanilla sh until SVR4...

[TUHS] Re: A few comments on porting the Bourne shell

[Ralph Corderoy]

Hi Jon,

> > I guess in interactive use most users would only miss one thing:
> > the history & line editing capability?
>
> Job control?

Just as history and line editing were a possible enhancement to the TTY
driver, didn't Pike comment that job control was cooked up in foreign
climes because they couldn't drag out a new TTY window in pixels?

Was something like screen(1)'s ‘Ctrl-A c’ considered in the TTY driver
to spin up another pseudo-TTY rather than suspend the long job with
Ctrl-Z and ‘bg’ it?

-- 
Cheers, Ralph.

[TUHS] Re: A few comments on porting the Bourne shell

[Jon Steinhart]

Ralph Corderoy writes:
> Hi Jon,
>
> > > I guess in interactive use most users would only miss one thing:
> > > the history & line editing capability?
> >
> > Job control?
>
> Just as history and line editing were a possible enhancement to the TTY
> driver, didn't Pike comment that job control was cooked up in foreign
> climes because they couldn't drag out a new TTY window in pixels?
>
> Was something like screen(1)'s ‘Ctrl-A c’ considered in the TTY driver
> to spin up another pseudo-TTY rather than suspend the long job with
> Ctrl-Z and ‘bg’ it?

My memory is really fuzzy on this.  I recall that job control came along
with csh which I used until bash.  At the time there were hardly any
graphics displays used as interactive devices.  I'm talking Berkeley job
control here; I seem to remember that someone got wedged into System V
that was awful and unusable.

Jon

[TUHS] Re: A few comments on porting the Bourne shell

[Clem Cole]

[-- Attachment #1: Type: text/plain, Size: 1370 bytes --]

below

On Sat, Dec 31, 2022 at 1:50 PM Jon Steinhart <jon@fourwinds.com> wrote:

> My memory is really fuzzy on this.  I recall that job control came along
> with csh which I used until bash.

Close.  That was the wide distribution of it.  Kulp created it originally
for his PDP-11 system in the late 1970s.  I'm not sure how, but  wnj saw
 it and imported the idea to UCB.  Joy had stopped hacking on the 11 by
then.  So  he added the kernel support for Job Control to 4.0 and hacked on
his own shell to add the user interface.  This would get widely distributed
as  in 4.1BSD and since job control was in csh, and not in sh (and job had
made the root use csh ) BSD users learned to used job control from there;
since  4.1BSD release that spread with wings.



> At the time there were hardly any
> graphics displays used as interactive devices.  I'm talking Berkeley job
> control here;

Right - which is Kulp job control to be fair.



> I seem to remember that someone got wedged into System V
> that was awful and unusable.
>
IIRC it was in SVR2 that a version of a job control system added some of
the semantics of Kulp's scheme to the kernel, but not all as you point out
and it was pretty disappointing if you had grown up on BSD systems.  Later
POSIX would pick up the Kulp/BSD Job Control definition and by SVR3/SVR4
AT&T fleshed out all all of the semantics.

[-- Attachment #2: Type: text/html, Size: 2598 bytes --]

[TUHS] Re: A few comments on porting the Bourne shell

[Chet Ramey]

On 12/31/22 2:24 PM, Clem Cole wrote:

>     I seem to remember that someone got wedged into System V
>     that was awful and unusable.
> 
> IIRC it was in SVR2 that a version of a job control system added some of 
> the semantics of Kulp's scheme to the kernel, but not all as you point out 
> and it was pretty disappointing if you had grown up on BSD systems.  Later 
> POSIX would pick up the Kulp/BSD Job Control definition and by SVR3/SVR4 
> AT&T fleshed out all all of the semantics.

There is a pretty good discussion of 4.2 BSD-style job control and ways to
make it fit with System V semantics (mostly controlling terminals and, of
course, process groups) at

https://web.physics.ucsb.edu/~pcs/apps/bash/job.control.ps

This is probably the best description of BSD-style job control, and it is
the inspiration for much of the POSIX job control design (e.g., their
invention of "sessions" as the replacement for System V style process
groups).

Chet
-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
		 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/

[TUHS] Re: A few comments on porting the Bourne shell

[Ron Natalie]

Berkeley Job Control was implemented in the kernel and originally only 
supported in CSH.    I detested csh, so I figured out how it worked and 
hacked it into a 5(r0) Bourne Shell.    But he time I did that, all the 
other BRL guys were using tcsh and so to counter, I hacked command line 
editing into the 5R2 Bourne shell.


------ Original Message ------
From "Jon Steinhart" <jon@fourwinds.com>
To tuhs@tuhs.org
Date 12/31/2022 1:49:54 PM
Subject [TUHS] Re: A few comments on porting the Bourne shell

>Ralph Corderoy writes:
>>  Hi Jon,
>>
>>  > > I guess in interactive use most users would only miss one thing:
>>  > > the history & line editing capability?
>>  >
>>  > Job control?
>>
>>  Just as history and line editing were a possible enhancement to the TTY
>>  driver, didn't Pike comment that job control was cooked up in foreign
>>  climes because they couldn't drag out a new TTY window in pixels?
>>
>>  Was something like screen(1)'s ‘Ctrl-A c’ considered in the TTY driver
>>  to spin up another pseudo-TTY rather than suspend the long job with
>>  Ctrl-Z and ‘bg’ it?
>
>My memory is really fuzzy on this.  I recall that job control came along
>with csh which I used until bash.  At the time there were hardly any
>graphics displays used as interactive devices.  I'm talking Berkeley job
>control here; I seem to remember that someone got wedged into System V
>that was awful and unusable.
>
>Jon

[TUHS] Command Line Editing in the Terminal Driver (Was: A few comments on porting the Bourne shell)

[Joseph Holsten]

[-- Attachment #1: Type: text/plain, Size: 2225 bytes --]

On Fri, Dec 30, 2022, at 12:42, Sven Mascheck wrote:
> and in "ksh - An Extensible High Level Language" David Korn writes:
> 
>  *  "Originally the idea of adding command line editing to ksh was rejected in the hope that line editing would move into the terminal driver." [2]
> [2] https://www.in-ulm.de/~mascheck/bourne/korn.html

This phrase has haunted me for years. It’s so clearly the “correct” separation of concerns, until you actually attempt implementing it. And Bourne’s irregular grammar certainly doesn’t help here. I’d prefer to move beyond readline, ideally something like text objects per vim/zsh/treesitter. But having one parser in the interpreter and another in the line editor becomes quite exciting if you want a true bourne or even posix sh.

But the thing that brought be back to playing against this quote again this year was starting to research the QED lineage and discovering helix & kakoune. Honestly, they feels like the most coherent advancements in QED-style editors since sam & acme. (Yes, I’m irrationally excluding vim text-objects, mostly because no one removed editor features that t-o made redundant. It’s as if ed had twice as many commands, one for regexp matches and one for pre-ken-QEDist exact matches.)

Anyway, the only time I’ve really seen “line editing […] move into the terminal driver” sound possible was acme’s win. For those who haven’t had the pleasure, rsc describes it at 15:25 in https://research.swtch.com/acme. “I worked for many years in a shell without history or command line editing, and I never missed it because Acme is providing this for me.”

It’s hard to convey how surprisingly pleasant sh or rc can be within acme win to someone who has only used them within a mere terminal-emulator-emulator. Especially since a greenfield terminal app has more code emulating old xterm behaviors than physical vt100 behaviours. This is especially exhausting when you try to use something like NeoVim’s :term expecting it to just work and discover that buffer-line-wrapping on window-resize hasn’t been implemented.

Anyone seen any other “terminal drivers” that provide a pleasant alternative to line editing?
--
~j

[-- Attachment #2: Type: text/html, Size: 3822 bytes --]

[TUHS] Re: Command Line Editing in the Terminal Driver

[Lars Brinkhoff]

Joseph Holsten wrote:
> Anyone seen any other “terminal drivers” that provide a pleasant
> alternative to line editing?

The Stanford AI lab PDP-10 operating system had line editing built in,
and the E editor integrated with this.

[TUHS] Re: Command Line Editing in the Terminal Driver

[Chris Hanson]

On Jan 2, 2023, at 12:33 PM, Lars Brinkhoff <lars@nocrew.org> wrote:
> 
> Joseph Holsten wrote:
>> Anyone seen any other “terminal drivers” that provide a pleasant
>> alternative to line editing?
> 
> The Stanford AI lab PDP-10 operating system had line editing built in,
> and the E editor integrated with this.

Some operating systems like HP's MPE (and probably also Tandem GUARDIAN) just expected the use of block-mode terminals for purposes of I/O efficiency since interactive use was typically via forms-based applications, and got line editing somewhat as a side-effect.

  -- Chris

[TUHS] Re: Command Line Editing in the Terminal Driver

[Greg 'groggy' Lehey]

[-- Attachment #1: Type: text/plain, Size: 1300 bytes --]

On Tuesday, 10 January 2023 at 18:51:07 -0800, Chris Hanson wrote:
> On Jan 2, 2023, at 12:33 PM, Lars Brinkhoff <lars@nocrew.org> wrote:
>> The Stanford AI lab PDP-10 operating system had line editing built in,
>> and the E editor integrated with this.
>
> Some operating systems like HP's MPE (and probably also Tandem
> GUARDIAN) just expected the use of block-mode terminals for purposes
> of I/O efficiency since interactive use was typically via
> forms-based applications, and got line editing somewhat as a
> side-effect.

Guardian was interesting.  The normal connection (from my point of
view, anyway) was via a TATM (Tandem Asynchronous Terminal
Multiplexer), which buffered the characters, but it didn't interrupt
the processor until one of 4 configurable characters were entered
(typically CR, abort, end of input).  This didn't offer line editing,
and the workarounds were painful.

There was also a block mode, mainly for end user applications, though
the “full screen” editor also used it.

Greg
--
Sent from my desktop computer.
Finger grog@lemis.com for PGP public key.
See complete headers for address and phone numbers.
This message is digitally signed.  If your Microsoft mail program
reports problems, please read http://lemis.com/broken-MUA.php

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 163 bytes --]

[TUHS] Re: Command Line Editing in the Terminal Driver (Was: A few comments on porting the Bourne shell)

[Rob Pike]

[-- Attachment #1: Type: text/plain, Size: 2799 bytes --]

It just seems much more natural to me to have line editing provided at some
fundamental level. Putting it in the shell or some library means some tools
get it but some don't. Why should sh have editing but mail not? But then
there's glob.

Do it once, do it well, and do it for everything, where "it" is a wildcard
that refers to every important detail.

-rob


On Tue, Jan 3, 2023 at 7:03 AM Joseph Holsten <joseph@josephholsten.com>
wrote:

> On Fri, Dec 30, 2022, at 12:42, Sven Mascheck wrote:
>
> and in "ksh - An Extensible High Level Language" David Korn writes:
>
>    -  "Originally the idea of adding command line editing to ksh was
>    rejected in the hope that line editing would move into the terminal
>    driver." [2]
>
> [2] https://www.in-ulm.de/~mascheck/bourne/korn.html
>
>
> This phrase has haunted me for years. It’s so clearly the “correct”
> separation of concerns, until you actually attempt implementing it. And
> Bourne’s irregular grammar certainly doesn’t help here. I’d prefer to move
> beyond readline, ideally something like text objects per
> vim/zsh/treesitter. But having one parser in the interpreter and another in
> the line editor becomes quite exciting if you want a true bourne or even
> posix sh.
>
> But the thing that brought be back to playing against this quote again
> this year was starting to research the QED lineage and discovering helix &
> kakoune. Honestly, they feels like the most coherent advancements in
> QED-style editors since sam & acme. (Yes, I’m irrationally excluding vim
> text-objects, mostly because no one removed editor features that t-o made
> redundant. It’s as if ed had twice as many commands, one for regexp matches
> and one for pre-ken-QEDist exact matches.)
>
> Anyway, the only time I’ve really seen “line editing […] move into the
> terminal driver” sound possible was acme’s win. For those who haven’t had
> the pleasure, rsc describes it at 15:25 in https://research.swtch.com/acme.
> “I worked for many years in a shell without history or command line
> editing, and I never missed it because Acme is providing this for me.”
>
> It’s hard to convey how surprisingly pleasant sh or rc can be within acme
> win to someone who has only used them within a mere
> terminal-emulator-emulator. Especially since a greenfield terminal app
> has more code emulating old xterm behaviors than physical vt100 behaviours.
> This is especially exhausting when you try to use something like NeoVim’s
> :term expecting it to just work and discover that buffer-line-wrapping on
> window-resize hasn’t been implemented.
>
> Anyone seen any other “terminal drivers” that provide a pleasant
> alternative to line editing?
> --
> ~j
>

[-- Attachment #2: Type: text/html, Size: 3987 bytes --]

[TUHS] Re: Command Line Editing in the Terminal Driver (Was: A few comments on porting the Bourne shell)

[Marshall Conover]

[-- Attachment #1: Type: text/plain, Size: 4937 bytes --]

Along these lines but not quite, Jupyter Notebooks have stood out to me as
another approach to this concept, with behavior I'd like to see implemented
in a shell.

With these notebooks, users type text into blocks, and then can have these
blocks evaluated by some underlying engine, with the results of the
evaluation showing up in a block underneath where they entered text.
Results can be not only text, but things like rendered charts and images.
There may be GUI-interactable rendered elements as well, but I'm unsure.

Notably, text-entry blocks are created such that each new block holds onto
the context of the previous blocks. So, a variable defined in the first
eval block will be available in later blocks.

Currently these are used primarily for demonstrating APIs, exploring data
with python, or writing quick PoCs that can then be extracted into an
application. Some companies, such as Netflix, have experimented with using
them entirely as a replacement for shell scripts, which is the sort of
research I'd most love to see.

Fundamentally, I think it's a smell that the way we interact with our
systems is tied to the long-gone hardware requirement of "be usable with a
teletype."  Acme was certainly a move towards recognizing the value of
being able to reuse space that, in TTYs, would be 'dead' text. The ability
to evaluate commands in Acme, and the further inclusion of an underlying
evaluator that holds state by Jupyter were good steps forward, I think.

I'd love to see experimentation with a whole system that takes the jupyter
approach, with nested namespaces forming applications, combined with data
being held in "blocks" as well as code - much like acme opens and edits
files as well as letting you execute either them or snippets in them. I
think there's a chance something could be developed that would better fit
the way we interface with computers today, and the underlying engine
approach would move us toward the "everything speaks one language" design
we lost in the move from shells to standalone GUI applications.

Mostly, I'd be bummed to see yet another shell that pretends it's behind a
"glass teletype", to use the tongue-in-cheek term from "Ed Mastery", or
just continuing elaboration on that design, despite the fact I use shells
daily. It's the best we have now, but I feel like it's the way forward.

Cheers!

Marshall

P.S. - Kakoune is neat and I'm going to have to poke around at that, thanks
for pointing it out.

On Mon, Jan 2, 2023 at 3:03 PM Joseph Holsten <joseph@josephholsten.com>
wrote:

> On Fri, Dec 30, 2022, at 12:42, Sven Mascheck wrote:
>
> and in "ksh - An Extensible High Level Language" David Korn writes:
>
>    -  "Originally the idea of adding command line editing to ksh was
>    rejected in the hope that line editing would move into the terminal
>    driver." [2]
>
> [2] https://www.in-ulm.de/~mascheck/bourne/korn.html
>
>
> This phrase has haunted me for years. It’s so clearly the “correct”
> separation of concerns, until you actually attempt implementing it. And
> Bourne’s irregular grammar certainly doesn’t help here. I’d prefer to move
> beyond readline, ideally something like text objects per
> vim/zsh/treesitter. But having one parser in the interpreter and another in
> the line editor becomes quite exciting if you want a true bourne or even
> posix sh.
>
> But the thing that brought be back to playing against this quote again
> this year was starting to research the QED lineage and discovering helix &
> kakoune. Honestly, they feels like the most coherent advancements in
> QED-style editors since sam & acme. (Yes, I’m irrationally excluding vim
> text-objects, mostly because no one removed editor features that t-o made
> redundant. It’s as if ed had twice as many commands, one for regexp matches
> and one for pre-ken-QEDist exact matches.)
>
> Anyway, the only time I’ve really seen “line editing […] move into the
> terminal driver” sound possible was acme’s win. For those who haven’t had
> the pleasure, rsc describes it at 15:25 in https://research.swtch.com/acme.
> “I worked for many years in a shell without history or command line
> editing, and I never missed it because Acme is providing this for me.”
>
> It’s hard to convey how surprisingly pleasant sh or rc can be within acme
> win to someone who has only used them within a mere
> terminal-emulator-emulator. Especially since a greenfield terminal app
> has more code emulating old xterm behaviors than physical vt100 behaviours.
> This is especially exhausting when you try to use something like NeoVim’s
> :term expecting it to just work and discover that buffer-line-wrapping on
> window-resize hasn’t been implemented.
>
> Anyone seen any other “terminal drivers” that provide a pleasant
> alternative to line editing?
> --
> ~j
>

[-- Attachment #2: Type: text/html, Size: 6059 bytes --]

[TUHS] Re: Command Line Editing in the Terminal Driver (Was: A few comments on porting the Bourne shell)

[Joseph Holsten]

> On Jan 3, 2023, at 08:55, Marshall Conover <marzhall.o@gmail.com> wrote:
> 
> Along these lines but not quite, Jupyter Notebooks have stood out to me as another approach to this concept, with behavior I'd like to see implemented in a shell.
…
> Currently these are used primarily for demonstrating APIs, exploring data with python, or writing quick PoCs that can then be extracted into an application. Some companies, such as Netflix, have experimented with using them entirely as a replacement for shell scripts, which is the sort of research I'd most love to see.
…
> I'd love to see experimentation with a whole system that takes the jupyter approach, with nested namespaces forming applications, combined with data being held in "blocks" as well as code - much like acme opens and edits files as well as letting you execute either them or snippets in them. I think there's a chance something could be developed that would better fit the way we interface with computers today, and the underlying engine approach would move us toward the "everything speaks one language" design we lost in the move from shells to standalone GUI applications.

This sounds tremendously useful. It gets at that feeling I want to have from emacs and literate programming but rarely achieve, the sense of Living Text.

[TUHS] Re: Command Line Editing in the Terminal Driver (Was: A few comments on porting the Bourne shell)

[Chris Hanson]

[-- Attachment #1: Type: text/plain, Size: 1008 bytes --]

On Jan 3, 2023, at 8:53 AM, Marshall Conover <marzhall.o@gmail.com> wrote:
> 
> With these notebooks, users type text into blocks, and then can have these blocks evaluated by some underlying engine, with the results of the evaluation showing up in a block underneath where they entered text. Results can be not only text, but things like rendered charts and images. There may be GUI-interactable rendered elements as well, but I'm unsure.

This is basically the model created by Symbolics for Dynamic Windows, which was the basis for CLIM, and which I believe they also adopted in the Symbolics distribution of MACSYMA. Kalman Reti does a great job demonstrating it under OpenGenera here: https://www.youtube.com/watch?v=o4-YnLpLgtk

(Jupyter notebooks are a derivative of Mathematica notebooks, which bear more than a passing resemblance to the Dynamic Windows stuff just like Mathematica bears more than a passing resemblance to McCarthy's "M-expressions" and the MACSYMA system…)

  -- Chris


[-- Attachment #2: Type: text/html, Size: 1773 bytes --]

[TUHS] Re: A few comments on porting the Bourne shell

[Chet Ramey]

On 12/30/22 3:02 PM, Larry McVoy wrote:
> On Fri, Dec 30, 2022 at 02:51:26PM -0500, Chet Ramey wrote:
>> On 12/30/22 1:25 PM, Paul Ruizendaal wrote:
>>>
>>> London and Reiser report about porting the shell that ???it required by far the largest conversion effort of any supposedly portable program, for the simple reason that it is not portable.??? By the time of SysIII this is greatly improved, but also in porting the SysIII user land it was the most complex of the set so far.
>>
>> Have you read
>>
>> http://www.collyer.net/who/geoff/sh.tour.pdf
>>
>> and looked at http://www.collyer.net/who/geoff/v7sh.tar ?
>>
>> In the limited literature on Bourne Shell porting, this is authoritative.
> 
> Is there are reason to hang on to the Bourne shell?  Maybe shell scripts?
> Does it perform better than ksh or bash?

Historical interest? Software archaeology? Reference behavior?

I don't think anyone is suggesting that we use it as a login shell, in the
same way that no one is suggesting we go back to using v7 as an everyday
computing environment. The world's come too far.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
		 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/

[TUHS] Re: A few comments on porting the Bourne shell

[Luther Johnson]

The Almquist shell (ash) and Debian's version of it, "dash", are 
compatible re-implementations of the Bourne shell, and use much less 
resources, and therefore wind up being faster as processes come and go.

I use csh (tcsh) and bash at the command line, but straight Bourne shell 
language is preferred and recommended by many for shell programming, I 
used to use csh for that and got bitten by all the things that I later 
discovered, those in the know had been warning about for years. Also, 
"bash-isms", syntactic sugary things in bash had led me to use them as a 
crutch, my scripts got simpler and more to the point when I re-wrote 
them for Bourne shell language only. That was my experience. I think 
we'll always have some kind of Bourne shell as the script workhorse, at 
last in Linux/Unix start-up and other blood and guts stuff.

In John Gilmore's paper on porting BSD through gcc to get to 4.4, he 
also remarked on how the original Bourne shell code was difficult to port.

On 12/30/2022 01:02 PM, Larry McVoy wrote:
> On Fri, Dec 30, 2022 at 02:51:26PM -0500, Chet Ramey wrote:
>> On 12/30/22 1:25 PM, Paul Ruizendaal wrote:
>>> London and Reiser report about porting the shell that ???it required by far the largest conversion effort of any supposedly portable program, for the simple reason that it is not portable.??? By the time of SysIII this is greatly improved, but also in porting the SysIII user land it was the most complex of the set so far.
>> Have you read
>>
>> http://www.collyer.net/who/geoff/sh.tour.pdf
>>
>> and looked at http://www.collyer.net/who/geoff/v7sh.tar ?
>>
>> In the limited literature on Bourne Shell porting, this is authoritative.
> Is there are reason to hang on to the Bourne shell?  Maybe shell scripts?
> Does it perform better than ksh or bash?
>
> Don't get me wrong, I much prefer the sh syntax over csh syntax, but
> I'd never go back to the Bourne shell as my login shell.  Way too much
> useful stuff in ksh/bash.
>

[TUHS] Re: A few comments on porting the Bourne shell

[Larry McVoy]

On Fri, Dec 30, 2022 at 05:08:42PM -0700, Luther Johnson wrote:
> I use csh (tcsh) and bash at the command line, but straight Bourne shell
> language is preferred and recommended by many for shell programming, I used
> to use csh for that and got bitten by all the things that I later
> discovered, those in the know had been warning about for years. Also,
> "bash-isms", syntactic sugary things in bash had led me to use them as a
> crutch, my scripts got simpler and more to the point when I re-wrote them
> for Bourne shell language only. That was my experience. I think we'll always
> have some kind of Bourne shell as the script workhorse, at last in
> Linux/Unix start-up and other blood and guts stuff.

When I was running my engineering team I was strict about Bourne syntax
and features only.  I got pushed on like crazy because "bash has this
$GOODNESS whhhhhhhy can't we use it".  Because we were supporting our
product on pretty much every unix and if it wasn't HP-UX that had an
ancient /bin/sh, it was AIX or whoever. 

Over and over, I won the "straight bourne shell only" battle.  So I agree,
if you want /bin/sh to work, Bourne shell for the win.

For a login shell, bash is my shell of choice.  It's bloated but I'm
typing this on a 5 year old Lenova X1 Carbon with 16GB of memory and
4 cores and it's fine.  It was fine a 133mhz Pentium.

[TUHS] Re: A few comments on porting the Bourne shell

[Steve Nickolas]

On Fri, 30 Dec 2022, Larry McVoy wrote:

> When I was running my engineering team I was strict about Bourne syntax
> and features only.  I got pushed on like crazy because "bash has this
> $GOODNESS whhhhhhhy can't we use it".  Because we were supporting our
> product on pretty much every unix and if it wasn't HP-UX that had an
> ancient /bin/sh, it was AIX or whoever.
>
> Over and over, I won the "straight bourne shell only" battle.  So I agree,
> if you want /bin/sh to work, Bourne shell for the win.
>
> For a login shell, bash is my shell of choice.  It's bloated but I'm
> typing this on a 5 year old Lenova X1 Carbon with 16GB of memory and
> 4 cores and it's fine.  It was fine a 133mhz Pentium.

There's some variants of the Almquist shell that have bash-style 
command-line editing, and I can deal with that if that's /bin/sh.  Usually 
I use bash and it's plenty fine.

I generally code shell scripts for Posix sh as my baseline and start my 
scripts with #!/bin/sh - but sometimes I'll use #!/bin/ksh if I don't 
expect them to be used on another machine.  ksh93 is lighter and faster 
than bash and the only things I miss are quirks of the command line 
editor.

-uso.

[TUHS] Re: A few comments on porting the Bourne shell

[Bakul Shah]

On Dec 30, 2022, at 7:59 PM, Larry McVoy <lm@mcvoy.com> wrote:
> 
> For a login shell, bash is my shell of choice.  It's bloated but I'm
> typing this on a 5 year old Lenova X1 Carbon with 16GB of memory and
> 4 cores and it's fine.  It was fine a 133mhz Pentium.

I switched to zsh in 1993-94 when I found out it had sh compatible
syntax but also had some useful features from csh (bang commands,
a{b,c} expansion, aliases etc.). Back then my machine had 16MB of
memory (IIRC a 50Mhz i486). zsh is the Cadillac of shells. But I
also use rc, the VWbug of shells on plan9!

[TUHS] Re: A few comments on porting the Bourne shell

[Larry McVoy]

On Fri, Dec 30, 2022 at 08:18:06PM -0800, Bakul Shah wrote:
> On Dec 30, 2022, at 7:59 PM, Larry McVoy <lm@mcvoy.com> wrote:
> > 
> > For a login shell, bash is my shell of choice.  It's bloated but I'm
> > typing this on a 5 year old Lenova X1 Carbon with 16GB of memory and
> > 4 cores and it's fine.  It was fine a 133mhz Pentium.
> 
> I switched to zsh in 1993-94 when I found out it had sh compatible
> syntax but also had some useful features from csh (bang commands,
> a{b,c} expansion, aliases etc.). 

This should probably move to COFF (which unlike many other lists I've
been on, has critical mass in the "go over there" list).

You are the 2nd to say zsh is your login shell.  I tried it for a while,
my progression was ${I dunno} to csh, actually used Bourne shell as a
login shell for a while around 1985 or so, all of my login scripts, other
shell scripts, were Bourne shell, I think zsh was next, then sighed in
relief with the original ksh release that went out, and when bash came
around with portability to everywhere, it has been bash.  ksh93 was 
interesting but bash was good enough and clearly open source so...

For script portability, pure Bourne shell has been the win for at least
40 years.  I suspect my kids and their kids will be writing Steve's
syntax, that's a legacy.

For general purpose scripting, I've moved on.  I loved Perl4.  It's a
kitchen sink language but it so clearly had roots in the Bourne shell,
in awk, in sed.  It was trying to be all of those things in one language
and it mostly succeeded.  In the 40mhz SPARC days I actually advocated
for rewriting a ton of /usr/bin on SunOS in perl4 because it was so easy
and far more maintainable than the C versions.

These days perl is weird, the kids seem to like Python, I hate Python
because there is no printf in the base language.  How is that a thing?
But people like it.
-- 
---
Larry McVoy           Retired to fishing          http://www.mcvoy.com/lm/boat

[TUHS] Re: A few comments on porting the Bourne shell

[Marc Donner]

[-- Attachment #1: Type: text/plain, Size: 2207 bytes --]

Fascinating.

When I left IBM Research to become a grad student at CMU (1981) the Unix CS
was running did not have history in the shell.  It had just been introduced
in VM/CMS and I loved it.  I nagged the sysadmins in the CS department
about it, and voila, it appeared shortly thereafter.  I presume it was one
of the things discussed here ported to the CMU environment.

I remember that I implemented history for the shell in the PERQ, doing some
nasty stuff to fit a reasonable history length in the weird Pascal they
had, since Pascal didn’t have dynamic strings or real pointers.

Marc
=====
On Fri, Dec 30, 2022 at 10:59 PM Larry McVoy <lm@mcvoy.com> wrote:

> On Fri, Dec 30, 2022 at 05:08:42PM -0700, Luther Johnson wrote:
> > I use csh (tcsh) and bash at the command line, but straight Bourne shell
> > language is preferred and recommended by many for shell programming, I
> used
> > to use csh for that and got bitten by all the things that I later
> > discovered, those in the know had been warning about for years. Also,
> > "bash-isms", syntactic sugary things in bash had led me to use them as a
> > crutch, my scripts got simpler and more to the point when I re-wrote them
> > for Bourne shell language only. That was my experience. I think we'll
> always
> > have some kind of Bourne shell as the script workhorse, at last in
> > Linux/Unix start-up and other blood and guts stuff.
>
> When I was running my engineering team I was strict about Bourne syntax
> and features only.  I got pushed on like crazy because "bash has this
> $GOODNESS whhhhhhhy can't we use it".  Because we were supporting our
> product on pretty much every unix and if it wasn't HP-UX that had an
> ancient /bin/sh, it was AIX or whoever.
>
> Over and over, I won the "straight bourne shell only" battle.  So I agree,
> if you want /bin/sh to work, Bourne shell for the win.
>
> For a login shell, bash is my shell of choice.  It's bloated but I'm
> typing this on a 5 year old Lenova X1 Carbon with 16GB of memory and
> 4 cores and it's fine.  It was fine a 133mhz Pentium.
>
-- 
=====
nygeek.net
mindthegapdialogs.com/home <https://www.mindthegapdialogs.com/home>

[-- Attachment #2: Type: text/html, Size: 3001 bytes --]

[TUHS] Re: A few comments on porting the Bourne shell

[Dave Horsfall]

On Fri, 30 Dec 2022, Larry McVoy wrote:

> When I was running my engineering team I was strict about Bourne syntax 
> and features only.  I got pushed on like crazy because "bash has this 
> $GOODNESS whhhhhhhy can't we use it".  Because we were supporting our 
> product on pretty much every unix and if it wasn't HP-UX that had an 
> ancient /bin/sh, it was AIX or whoever.

I've never bothered to learn those Bash thingies, because "expr" does 
everything that I need and is available on just about all boxes.

> Over and over, I won the "straight bourne shell only" battle.  So I 
> agree, if you want /bin/sh to work, Bourne shell for the win.

Yep; whoever wrote CSH must've been high on something, as the syntax makes 
no sense whatsoever.

> For a login shell, bash is my shell of choice.  It's bloated but I'm 
> typing this on a 5 year old Lenova X1 Carbon with 16GB of memory and 4 
> cores and it's fine.  It was fine a 133mhz Pentium.

I do admit to being a bit of a ZSH user...  I've never bothered to learn 
all its features, but the subset I use gets me through.

-- Dave

[TUHS] Re: A few comments on porting the Bourne shell

[Clem Cole]

[-- Attachment #1: Type: text/plain, Size: 1287 bytes --]

Bourne to program, type with Joy.

On Fri, Dec 30, 2022 at 11:23 PM Dave Horsfall <dave@horsfall.org> wrote:

> On Fri, 30 Dec 2022, Larry McVoy wrote:
>
> > When I was running my engineering team I was strict about Bourne syntax
> > and features only.  I got pushed on like crazy because "bash has this
> > $GOODNESS whhhhhhhy can't we use it".  Because we were supporting our
> > product on pretty much every unix and if it wasn't HP-UX that had an
> > ancient /bin/sh, it was AIX or whoever.
>
> I've never bothered to learn those Bash thingies, because "expr" does
> everything that I need and is available on just about all boxes.
>
> > Over and over, I won the "straight bourne shell only" battle.  So I
> > agree, if you want /bin/sh to work, Bourne shell for the win.
>
> Yep; whoever wrote CSH must've been high on something, as the syntax makes
> no sense whatsoever.
>
> > For a login shell, bash is my shell of choice.  It's bloated but I'm
> > typing this on a 5 year old Lenova X1 Carbon with 16GB of memory and 4
> > cores and it's fine.  It was fine a 133mhz Pentium.
>
> I do admit to being a bit of a ZSH user...  I've never bothered to learn
> all its features, but the subset I use gets me through.
>
> -- Dave
>
-- 
Sent from a handheld expect more typos than usual

[-- Attachment #2: Type: text/html, Size: 1825 bytes --]

[TUHS] Re: A few comments on porting the Bourne shell

[Mary Ann Horton]

[-- Attachment #1: Type: text/plain, Size: 612 bytes --]

On 12/30/22 20:23, Dave Horsfall wrote:
> Yep; whoever wrote CSH must've been high on something, as the syntax makes
> no sense whatsoever.

Bill Joy wrote csh. He based it on the V6 shell.  It made perfect sense 
at the time. This was before the V7 Bourne shell came out.

Thanks,

/Mary Ann Horton/ (she/her/ma'am)
maryannhorton.com <https://maryannhorton.com>

"This is a great book" - Monica Helms

"Brave and Important" - Laura L. Engel

       Available on Amazon and bn.com!

	<https://www.amazon.com/Trailblazer-Lighting-Transgender-Equality-Corporate-ebook/dp/B0B8F2BR9B>




[-- Attachment #2: Type: text/html, Size: 1805 bytes --]

[TUHS] Re: A few comments on porting the Bourne shell

[Clem Cole]

[-- Attachment #1: Type: text/plain, Size: 3158 bytes --]

Be careful in applying modern judgments without remembering
the historical context - below
On Mon, Jan 2, 2023 at 12:12 AM Mary Ann Horton <mah@mhorton.net> wrote:

> On 12/30/22 20:23, Dave Horsfall wrote:
>
> Yep; whoever wrote CSH must've been high on something, as the syntax makes
> no sense whatsoever.
>
> Bill Joy wrote csh. He based it on the V6 shell.  It made perfect sense at
> the time. This was before the V7 Bourne shell came out.
>
Right -- let me add a little detail to this that might help. With the
Fourth, Fifth, and Sixth Edition the default shell from Research is the
Thompson shell.  It's simple and un-adorn.   Get the job done for typing,
but as a formal scripting (programming) language is not easy to use.

In fact, the command 'cd' idiom is still 'chdir' [add it was a typical
local hack that many of did].  If you look at the USENIX tapes in the TUHS
archives, in particular, the Harvard tape I suspect you will find a number
of alternative shells that people had offered up from the different sites.

With 'BSD' (the first UNIX distribution tape from the UCB Industrial
Liasion Office of EECS which we often refer to on this list as 1BSD to
distinguish it from the other releases - also available on TUHS], there is
a new Berkeley shell, that Mary Ann points out Bill had created by
modifying the Thompson shell and added some features to it. By the time of
the 2BSD, the shell has been further been refined and by then renamed the
csh, as the syntax had been made to approximate the C language.

A big difference between the *Thompson Shell begets -> Berkeley Shell ->
begets C Shell *and Bourne's work on his shell, is that Steve started from
scratch. Steve had just come off many years of the Algol 68 project and was
certainly extremely well-versed in the detailed subtleties of the semantics
of language design.  His higher bit was to create something *easier for
people to write scripts using the -> small is beautiful -> Unix philosophy
ideas*.

Bill, on the other hand, is the quintessential hacker (My line is him in
those days used to be: "He types open curly brace, close curly brace and he
patches at 9600 baud.")  Bill was interested in a shell that made typing
easy, was not a load the system, and made it easy for >>him<< to write
simple admin scripts.

I would suggest both were 100% successful at what their purpose was.
Like Larry, I have always been a meany and I have tried to make darned sure
my folks (even to this day - we will see how long that last after I retire)
use V7 syntax scripts - not bash, not even korn.   As I said, the old UNIX
hacker wisdom, has been "*Bourne to Program, Type with **Joy*" and you have
a hard time going wrong.

[FYI - while I have tcsh around, on my Mac these days I finally switched
from tcsh to zsh as my default, but pretty much all of Bill's typing tricks
are supported in zsh so the 'ROMs" in my fingers are happy after >40
years"] but all of my scripts are still V7 syntax - never been a reason to
change that -- bigger issues usually found are GNU/Linux is not UNIX and
Apple had to pee on things also.
ᐧ

[-- Attachment #2: Type: text/html, Size: 5916 bytes --]

[TUHS] Re: A few comments on porting the Bourne shell

[Larry McVoy]

On Mon, Jan 02, 2023 at 11:25:06AM -0500, Clem Cole wrote:
> Like Larry, I have always been a meany and I have tried to make darned sure
> my folks (even to this day - we will see how long that last after I retire)
> use V7 syntax scripts - not bash, not even korn.   

I think it is less of an issue today but if I were still supporting a 
multi platform product, I'd still insist on it.  Ya never know when a
big pile of money is going to show and insist that you work on SCO
based cash registers or something.   Every.  Single.  Time. one of 
my engineers whined that they liked bash better, we'd hit on some
supported platform where /bin/sh is not bash.  And as we shipped
scripts that people expected to work, you can't fix their /bin/sh.

I won that argument over and over and eventually we all instinctively
wrote portable scripts.

[TUHS] Re: A few comments on porting the Bourne shell

[Adam Thornton]

> On Jan 2, 2023, at 9:51 AM, Larry McVoy <lm@mcvoy.com> wrote:
> 
> On Mon, Jan 02, 2023 at 11:25:06AM -0500, Clem Cole wrote:
>> Like Larry, I have always been a meany and I have tried to make darned sure
>> my folks (even to this day - we will see how long that last after I retire)
>> use V7 syntax scripts - not bash, not even korn.   
> And as we shipped
> scripts that people expected to work, you can't fix their /bin/sh.
> 
> I won that argument over and over and eventually we all instinctively
> wrote portable scripts.


I feel like after POSIX.2 had had a few years to settle, it was OK to use *that* as your scripting language (I particularly like $() rather than backticks, because of nestability, although you can certainly argue that if you're regularly nesting your interpolations, it's the design that's broken, not the specific syntax of the command).

At least as long as you put a very visible comment right at the top saying "We're assuming POSIX shell."

But of course the "after it had had a few years to settle" is doing a lot of lifting there.  During the Unix Wars, yeah, clearly using v7 sh as lowest-common-denominator was the right choice.  After the establishment of Linux Hegemony sometime in the early 2000s, I think I remember that you could pretty much count on everyone at least providing a POSIX shell, even if it wasn't the default.  (But of course by then, "everyone" that wasn't Linux or OS X was already down to more-or-less Sun, IBM, HP, and DEC/Compaq.)

These days, if you have a vendor-supplied shell script that DOESN'T work correctly under a POSIX /bin/sh, that should be treated as a defect.

Adam

[TUHS] Re: A few comments on porting the Bourne shell

[Larry McVoy]

On Mon, Jan 02, 2023 at 10:32:50AM -0700, Adam Thornton wrote:
> But of course the "after it had had a few years to settle" is doing
> a lot of lifting there.  During the Unix Wars, yeah, clearly using
> v7 sh as lowest-common-denominator was the right choice.  After the
> establishment of Linux Hegemony sometime in the early 2000s, I think I
> remember that you could pretty much count on everyone at least providing
> a POSIX shell, even if it wasn't the default.  (But of course by then,
> "everyone" that wasn't Linux or OS X was already down to more-or-less Sun,
> IBM, HP, and DEC/Compaq.)

I was supporting a commercial product in the early 2000's and there
were all sorts of systems then that had old shells.  Yes, you could make
everything a 2 level thing where the first level finds the correct shell,
but that's just fuss.  Just make things portable, it's not that hard
and it works everywhere.

When you get to the commercial world, you'd be stunned to see how long
old machines last.  If they are solving some problem, and they aren't
broken, nobody replaces them.  I'll bet you anything there are still
SCO registers out there, I'll bet there are still PDP-11s out there.

If it ain't broke, don't fix it.

[TUHS] Re: A few comments on porting the Bourne shell

[Luther Johnson]

Arnold Schoenberg said of his 12-tone method of music composition, "in 
constraints I find freedom". The best thing about committing to a simple 
style and a small set of tools that span the problem space, is the 
effect it has on my program composition, imho.

On 01/02/2023 10:43 AM, Larry McVoy wrote:
> On Mon, Jan 02, 2023 at 10:32:50AM -0700, Adam Thornton wrote:
>> But of course the "after it had had a few years to settle" is doing
>> a lot of lifting there.  During the Unix Wars, yeah, clearly using
>> v7 sh as lowest-common-denominator was the right choice.  After the
>> establishment of Linux Hegemony sometime in the early 2000s, I think I
>> remember that you could pretty much count on everyone at least providing
>> a POSIX shell, even if it wasn't the default.  (But of course by then,
>> "everyone" that wasn't Linux or OS X was already down to more-or-less Sun,
>> IBM, HP, and DEC/Compaq.)
> I was supporting a commercial product in the early 2000's and there
> were all sorts of systems then that had old shells.  Yes, you could make
> everything a 2 level thing where the first level finds the correct shell,
> but that's just fuss.  Just make things portable, it's not that hard
> and it works everywhere.
>
> When you get to the commercial world, you'd be stunned to see how long
> old machines last.  If they are solving some problem, and they aren't
> broken, nobody replaces them.  I'll bet you anything there are still
> SCO registers out there, I'll bet there are still PDP-11s out there.
>
> If it ain't broke, don't fix it.
>

[TUHS] Re: A few comments on porting the Bourne shell

[G. Branden Robinson]

[-- Attachment #1: Type: text/plain, Size: 2136 bytes --]

At 2023-01-02T10:48:32-0700, Luther Johnson wrote:
> Arnold Schoenberg said of his 12-tone method of music composition, "in
> constraints I find freedom". The best thing about committing to a
> simple style and a small set of tools that span the problem space, is
> the effect it has on my program composition, imho.

Schoenberg didn't have to compose for orchestras such that the Berlin
Philharmonic string section tuned half of the octave meantone with the
other half equal-tempered, whereas the Royal Concertgebouw had the
brasses play an un-notated ascending major third gliss anytime you
approached the tonic from a tritone _below_ (only).

> On 01/02/2023 10:43 AM, Larry McVoy wrote:
> > I was supporting a commercial product in the early 2000's and there
> > were all sorts of systems then that had old shells.  Yes, you could
> > make everything a 2 level thing where the first level finds the
> > correct shell, but that's just fuss.  Just make things portable,
> > it's not that hard and it works everywhere.
> > 
> > When you get to the commercial world, you'd be stunned to see how
> > long old machines last.  If they are solving some problem, and they
> > aren't broken, nobody replaces them.  I'll bet you anything there
> > are still SCO registers out there, I'll bet there are still PDP-11s
> > out there.

The logical consequence of this is to write in Autoconf's recommended
dialect of shell, which has far too many rules to remember because every
Unix vendor added exciting new bugs to their "pure" V7 shells.

https://www.gnu.org/software/autoconf/manual/autoconf-2.60/html_node/Portable-Shell.html

> > If it ain't broke, don't fix it.

They were all broken.  Badly.  If your scripts worked, you got lucky at
Russian Roulette.  No greater claim to robustness can be made.

POSIX shell conformance still proves challenging for vendors, but is an
immense improvement over the status quo ante.

I agree with Robert Elz though that standardizing the alias feature was
nuts.  I want to know who brought the crack cocaine to the conference
table that day.

Regards,
Branden

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[TUHS] Re: A few comments on porting the Bourne shell

[Luther Johnson]

Point taken, if not entirely understood :)

On 01/02/2023 11:00 AM, G. Branden Robinson wrote:
> At 2023-01-02T10:48:32-0700, Luther Johnson wrote:
>> Arnold Schoenberg said of his 12-tone method of music composition, "in
>> constraints I find freedom". The best thing about committing to a
>> simple style and a small set of tools that span the problem space, is
>> the effect it has on my program composition, imho.
> Schoenberg didn't have to compose for orchestras such that the Berlin
> Philharmonic string section tuned half of the octave meantone with the
> other half equal-tempered, whereas the Royal Concertgebouw had the
> brasses play an un-notated ascending major third gliss anytime you
> approached the tonic from a tritone _below_ (only).
>
>> On 01/02/2023 10:43 AM, Larry McVoy wrote:
>>> I was supporting a commercial product in the early 2000's and there
>>> were all sorts of systems then that had old shells.  Yes, you could
>>> make everything a 2 level thing where the first level finds the
>>> correct shell, but that's just fuss.  Just make things portable,
>>> it's not that hard and it works everywhere.
>>>
>>> When you get to the commercial world, you'd be stunned to see how
>>> long old machines last.  If they are solving some problem, and they
>>> aren't broken, nobody replaces them.  I'll bet you anything there
>>> are still SCO registers out there, I'll bet there are still PDP-11s
>>> out there.
> The logical consequence of this is to write in Autoconf's recommended
> dialect of shell, which has far too many rules to remember because every
> Unix vendor added exciting new bugs to their "pure" V7 shells.
>
> https://www.gnu.org/software/autoconf/manual/autoconf-2.60/html_node/Portable-Shell.html
>
>>> If it ain't broke, don't fix it.
> They were all broken.  Badly.  If your scripts worked, you got lucky at
> Russian Roulette.  No greater claim to robustness can be made.
>
> POSIX shell conformance still proves challenging for vendors, but is an
> immense improvement over the status quo ante.
>
> I agree with Robert Elz though that standardizing the alias feature was
> nuts.  I want to know who brought the crack cocaine to the conference
> table that day.
>
> Regards,
> Branden

[TUHS] Re: A few comments on porting the Bourne shell

[Larry McVoy]

On Mon, Jan 02, 2023 at 12:00:20PM -0600, G. Branden Robinson wrote:
> > > If it ain't broke, don't fix it.
> 
> They were all broken.  Badly.  If your scripts worked, you got lucky at
> Russian Roulette.  No greater claim to robustness can be made.
> 
> POSIX shell conformance still proves challenging for vendors, but is an
> immense improvement over the status quo ante.

You are talking to a dude with 40+ years of Unix experience, supporting
commercial products most of that time.  I didn't get "lucky at Russian
Roulette", I wrote scripts that were portable.  I have 40 year old scripts
that _still_ work and they work on virtually every Unix ever built.
How do I know?  I was a contractor for my first job, I got plopped down
in front of every random unix you could imagine and each time I polished
off the warts.

I spent decades supporting my own products on every flavor of Unix and
processors from Arm to System/360.  Oh, and Windows XP and on and MacOS.

My scripts worked with /bin/sh being whatever it was.

It's interesting to me that other old timers, like Clem, are saying
exactly the same thing as I am.  Are we all wrong?

[TUHS] Re: A few comments on porting the Bourne shell

[Clem Cole]

[-- Attachment #1: Type: text/plain, Size: 334 bytes --]

On Mon, Jan 2, 2023 at 1:13 PM Larry McVoy <lm@mcvoy.com> wrote:

>
> My scripts worked with /bin/sh being whatever it was.
>
> It's interesting to me that other old timers, like Clem, are saying
> exactly the same thing as I am.  Are we all wrong?
>
Exactly -- if you are disciplined -- it just works.  It's not magic.
ᐧ

[-- Attachment #2: Type: text/html, Size: 1115 bytes --]

[TUHS] Re: A few comments on porting the Bourne shell

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 1576 bytes --]

On Mon, Jan 2, 2023, 11:18 AM Clem Cole <clemc@ccc.com> wrote:

>
>
> On Mon, Jan 2, 2023 at 1:13 PM Larry McVoy <lm@mcvoy.com> wrote:
>
>>
>> My scripts worked with /bin/sh being whatever it was.
>>
>> It's interesting to me that other old timers, like Clem, are saying
>> exactly the same thing as I am.  Are we all wrong?
>>
> Exactly -- if you are disciplined -- it just works.  It's not magic.
>

I've only got 30 years experience in writing portable shell scripts... But
I have a slightly different take. There is an element of luck involved.
There are a number of constructs that are valid in v7 shell that aren't
completely portable due to bugs in vendor shells. The luck part comes in
either you know what to avoid because you got unlucky and got burned in the
past. Or you got lucky and never thought to write something like that so it
just works. With experience, you know what works...

I notice the old timers hedged a little bit by saying things like
"virtually all" so maybe they had a bit of bad luck now and again...

Also, the really aweful systems are rare enough today that most folks on
this list have had the good luck not to need to run on them... :)

You can get super close though, but I'll bet there's some scripts that work
only because they didn't run on, say, Eunice...

But luck favors the prepared... and making the effort to try to be portable
will help when you have to run on dash, or one of the BSD shells that might
not have $<(cmd) or some other useful but not universal feature...

Warner

> ᐧ
>

[-- Attachment #2: Type: text/html, Size: 3087 bytes --]

[TUHS] Re: A few comments on porting the Bourne shell

[Adam Thornton]

> On Jan 2, 2023, at 12:50 PM, Warner Losh <imp@bsdimp.com> wrote:
> 
> 
> You can get super close though, but I'll bet there's some scripts that work only because they didn't run on, say, Eunice...

"Congratulations!"

> 
> But luck favors the prepared... and making the effort to try to be portable will help when you have to run on dash, or one of the BSD shells that might not have $<(cmd) or some other useful but not universal feature...

Yeah and I think we're basically agreeing here.  The problem isn't *really* whether you use some-odd-number-of-backslashes-and-a-backtick versus dollar-open-paren, the problem is people who put #!/bin/sh at the top of the script when the contents only run under bash.  And there's a lot of that, because just as all the world was VAX once upon a time, all the world is assumed to be an x86_64 Linux where sh is bash.

At least the move towards containerization has largely defeated that assumption now, although in some sense it's also just swept the problem under someone else's rug.

Adam

[TUHS] Re: A few comments on porting the Bourne shell

[G. Branden Robinson]

[-- Attachment #1: Type: text/plain, Size: 2208 bytes --]

At 2023-01-02T10:12:42-0800, Larry McVoy wrote:
> You are talking to a dude with 40+ years of Unix experience,

I'm clocking 30 this year.  A babe in the woods.

> supporting commercial products most of that time.  I didn't get "lucky
> at Russian Roulette", I wrote scripts that were portable.  I have 40
> year old scripts that _still_ work and they work on virtually every
> Unix ever built.  How do I know?  I was a contractor for my first job,
> I got plopped down in front of every random unix you could imagine and
> each time I polished off the warts.

Yes.  That's how the chapter of the GNU Autoconf manual to which I
linked got written--or so I surmise, based simply on how bizarre some of
the bugs are that have to be worked around.

> I spent decades supporting my own products on every flavor of Unix and
> processors from Arm to System/360.  Oh, and Windows XP and on and
> MacOS.
> 
> My scripts worked with /bin/sh being whatever it was.
> 
> It's interesting to me that other old timers, like Clem, are saying
> exactly the same thing as I am.  Are we all wrong?

Over on the groff list you had the opposite advice regarding Make: pick
one implementation and go with it, you said, and other implementations
be damned (very loosely paraphrasing).[1]

I'm not saying you're wrong--I'm saying you're inconsistent.

I reckon you did what you needed to do to keep the lights on and the
engineers fed.  That's fine, but it's also a recipe for passivity.  You
don't have to tell me, but count up the number of times you flexed to
accommodate a buggy implementation versus telling your client that their
shell was just too crap and that they needed to either install one in
which you had confidence, or you'd have to decline the contract.

If their users had demanded more of their commercial Unix vendors, maybe
those Unices would not be regarded as dinosaurs today.

I could harp further on quality of implementation issues, but I'd
probably do better to take up the lotus position on the back of a
motorcycle.

Preferably not while the vehicle is in motion.

Regards,
Branden

[1] https://lists.gnu.org/archive/html/groff/2022-03/msg00084.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[TUHS] Re: A few comments on porting the Bourne shell

[Rich Salz]

[-- Attachment #1: Type: text/plain, Size: 121 bytes --]

The difference is that "make" need only run on your build systems.
Installation scripts need to run on customer systems.

[-- Attachment #2: Type: text/html, Size: 202 bytes --]

[TUHS] Re: A few comments on porting the Bourne shell

[Larry McVoy]

On Mon, Jan 02, 2023 at 02:34:22PM -0500, Rich Salz wrote:
> The difference is that "make" need only run on your build systems.
> Installation scripts need to run on customer systems.

Exactly.  I let my team, against my judgement, use GNU make for our builds.
That was in house, the scripts are customer run.
-- 
---
Larry McVoy           Retired to fishing          http://www.mcvoy.com/lm/boat

[TUHS] Re: A few comments on porting the Bourne shell

[G. Branden Robinson]

[-- Attachment #1: Type: text/plain, Size: 840 bytes --]

At 2023-01-02T14:34:22-0500, Rich Salz wrote:
> The difference is that "make" need only run on your build systems.
> Installation scripts need to run on customer systems.

Surprisingly (not really), I don't agree with this, either.

make, like tsort, is a generally useful tool that people misconceive as
being limited in scope to traditional compile-and-link software
development.

A dab of imagination and familiarity with existing tooling can save one
much wheel reinvention.

That said I don't doubt that some vendors happily took these tools in
particular (along with the compiler, of course) and locked them up
within a "pro" or "development" tier.

But all that teaches us is something we should have already known: free
software is better than proprietary software.  Why _shouldn't_ customer
systems have make(1)?

Regards,
Branden

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[TUHS] Re: A few comments on porting the Bourne shell

[Larry McVoy]

On Mon, Jan 02, 2023 at 02:24:22PM -0600, G. Branden Robinson wrote:
> At 2023-01-02T14:34:22-0500, Rich Salz wrote:
> > The difference is that "make" need only run on your build systems.
> > Installation scripts need to run on customer systems.
> 
> Surprisingly (not really), I don't agree with this, either.
> 
> make, like tsort, is a generally useful tool that people misconceive as
> being limited in scope to traditional compile-and-link software
> development.
> 
> A dab of imagination and familiarity with existing tooling can save one
> much wheel reinvention.

If we wanted to use Makefiles in our customer run product, we would have
shipped it.

[TUHS] Re: A few comments on porting the Bourne shell

[Dan Cross]

On Mon, Jan 2, 2023 at 3:41 PM Larry McVoy <lm@mcvoy.com> wrote:
> On Mon, Jan 02, 2023 at 02:24:22PM -0600, G. Branden Robinson wrote:
> > At 2023-01-02T14:34:22-0500, Rich Salz wrote:
> > > The difference is that "make" need only run on your build systems.
> > > Installation scripts need to run on customer systems.
> >
> > Surprisingly (not really), I don't agree with this, either.
> >
> > make, like tsort, is a generally useful tool that people misconceive as
> > being limited in scope to traditional compile-and-link software
> > development.
> >
> > A dab of imagination and familiarity with existing tooling can save one
> > much wheel reinvention.
>
> If we wanted to use Makefiles in our customer run product, we would have
> shipped it.

Couldn't you also ship a shell? That kinda seems like the simplest solution.

I used to be pretty dogmatic about adherence to 7th Ed tools and
syntax, but I think I'm like the frog in the pot where the temperature
is slowly rising: I suspect I've let things creep in that are not
portable to a universal intersection set, simply because I haven't
noticed, because I'm no longer trying to run things on very old
systems regularly. Will that script I wrote last week run on Ultrix?
Beats me; I haven't powered up the DECstation in 15 years. But
considering the issue raises a related question: practically speaking,
what do you define as a least common denominator? For example, is
`nawk` allowed, or just the traditional 7th Ed awk? How about extended
regular expressions? GNU coreutils `egrep` now spits out a warning
saying, "egrep is obsolescent; using grep -E" (I cannot adequately
express how irritating I find this).  Does `tr` need the square
brackets? Etc. The shell by itself is just part of the equation, and
as a programming environment, /bin/sh is not all that interesting
unless complemented by a bunch of utilities, the portability of which
seems less guaranteed than it once was.

I actually think the shell is less of an issue; C is a bigger one for
me. About a year ago, I set up some emulated systems on my local ham
radio network (connected to AX.25 also!). Among them were Multics and
TOPS-20, but also a VAX running 4.3BSD and 7th Ed on a PDP-11. But
getting anything modern to build on either is a serious pain; speaking
of shells, I was able to find what is (I think) the last version of
tcsh to support 4.3 and get that to build.

Then again, having 7th Ed around forces me to think about shell script
portability again, so...there's that. :-)

        - Dan C.

[TUHS] Re: A few comments on porting the Bourne shell

[Clem Cole]

[-- Attachment #1: Type: text/plain, Size: 247 bytes --]

On Mon, Jan 2, 2023 at 4:03 PM Dan Cross <crossd@gmail.com> wrote:

> Couldn't you also ship a shell? That kinda seems like the simplest
> solution.
>
Sure the problem is -- where do you stop?

The result is the fork/mess called Gnu.
ᐧ

[-- Attachment #2: Type: text/html, Size: 1341 bytes --]

[TUHS] Re: A few comments on porting the Bourne shell

[Dan Cross]

On Mon, Jan 2, 2023 at 4:07 PM Clem Cole <clemc@ccc.com> wrote:
> On Mon, Jan 2, 2023 at 4:03 PM Dan Cross <crossd@gmail.com> wrote:
>> Couldn't you also ship a shell? That kinda seems like the simplest solution.
>
> Sure the problem is -- where do you stop?

Good question. But this sort of suggests that maybe it's not the shell
that's really the problem, but the entire milieu of all the tools that
we use when we write "shell scripts."

> The result is the fork/mess called Gnu.

BUGS: Yes.

        - Dan C.

[TUHS] Re: A few comments on porting the Bourne shell

[segaloco via TUHS]

A shell is only as powerful as the constructs it can interact with.  One can do logic all day and night but if that logic isn't being put to work, it's just an exercise in vanity.  I think that's a good point that scripting problems may be a symptom of the nature of the tools being used in them.

So the day job is pretty much an all Microsoft shop, from .NET code to IIS servers, not a '$' prompt in sight usually.  However, being a scientific operation, we have quite a diverse crew of UNIX machines floating around out there resigned to their fates sitting in the corner running the oldest equipment at their respective labs.  Any time I've had to interact with any of these I've been grateful for not taking various post-Bourne shell quirks for granted.  Not that these interactions are frequent, but I find myself tapped often being "that UNIX guy" on the team.  A couple examples come to mind.

One facility is running two pieces of equipment the software for which runs on Digital UNIX circa mid 90s.  The data transfer mechanism was some FTP thing their previous technical director had cooked up and one morning we had found it went down.  The resident IT guy there knew the basics of how to get in there, but hadn't really put the process down anywhere.  Luckily (security may think otherwise...) I was able to connect over rsh with the password he provided and was able to modify the script for that exporter to self-heal.  I don't know what shell it was running beyond it spitting a '$' out at me, but luckily didn't have to know.

The other is a lab using a data analysis system for, among others, HP-UX.  I don't recall the age of the version they were running on, but the system is called Target and was eventually forked in a few directions when the original company closed up shop, our group inheriting one of the forks.  It's one of those things I hope I can archive publicly someday, but not there yet.  In any case, the technicians all had Windows PCs with an X11 server setup for accessing the HP-UX machine, and it was running CDE if that helps date it (unless that's still default).  That system is in archive now as we've replaced it with the current version of our fork which is now .NET.  I was surprised they didn't have more UNIX going on, but they had managed to get a lot of their older equipment to work with some sort of Windows based solution one way or another.  Life finds a way...

I'm not sure where it stands as far as the sea of shells these days, but one I've opted towards in system builds the past few years is dash.  It's pretty small and seems to hit all the marks as far as basic Bourne-ish behavior and POSIX stuff.  When I'm tinkering on a board and just need to toss a shell on top of a kernel, I go for dash.

- Matt G.

------- Original Message -------
On Monday, January 2nd, 2023 at 1:19 PM, Dan Cross <crossd@gmail.com> wrote:


> On Mon, Jan 2, 2023 at 4:07 PM Clem Cole clemc@ccc.com wrote:
> 
> > On Mon, Jan 2, 2023 at 4:03 PM Dan Cross crossd@gmail.com wrote:
> > 
> > > Couldn't you also ship a shell? That kinda seems like the simplest solution.
> > 
> > Sure the problem is -- where do you stop?
> 
> 
> Good question. But this sort of suggests that maybe it's not the shell
> that's really the problem, but the entire milieu of all the tools that
> we use when we write "shell scripts."
> 
> > The result is the fork/mess called Gnu.
> 
> 
> BUGS: Yes.
> 
> - Dan C.

[TUHS] Re: A few comments on porting the Bourne shell

[Jon Steinhart]

segaloco via TUHS writes:
> I think that's a good point that scripting problems may be
> a symptom of the nature of the tools being used in them.

I think that you're hinting at something different.

To the best of my recollection, scripting languages were originally
intended and used for the automation of repetitive personal tasks;
making it easier for users who found themselves typing the same
stuff over and over again.  Somewhere along the line people forgot
how to use a compiler and began writing large systems in a variety
of roughly equivalent but incompatible interpreted languages.  Can
one even boot linux without having several different incompatible
versions of Python installed today?  So I don't think that it's the
nature of the tools; I think that it's people choosing the wrong
tools for the problems that they're trying to solve.

Jon

[TUHS] Re: A few comments on porting the Bourne shell

[Joseph Holsten]

[-- Attachment #1: Type: text/plain, Size: 1967 bytes --]


> On Jan 2, 2023, at 15:59, Jon Steinhart <jon@fourwinds.com> wrote:
> 
> segaloco via TUHS writes:
>> I think that's a good point that scripting problems may be
>> a symptom of the nature of the tools being used in them.
> 
> To the best of my recollection, scripting languages were originally
> intended and used for the automation of repetitive personal tasks;
> making it easier for users who found themselves typing the same
> stuff over and over again.  Somewhere along the line people forgot
> how to use a compiler and began writing large systems in a variety
> of roughly equivalent but incompatible interpreted languages. 

I used to think this was a somewhat modern thing, at least evident by atrocities performed by tcl & perl in the 1990s. But exposure to this group of weirdos has made me realize that somewhat terrifying amounts of “infrastructure” code is built on little languages like awk or ed scripts. And that any criticism of the horrible things I use Ruby for today only really defends the place for awk and sed that those now hold and where they truly excel.

But then I read https://www.bell-labs.com/usr/dmr/www/qed.html

> For a brief period in the 1970s, the GECOS QED served us as a scripting language; it was in some ways analogous to the Awk or Perl of today. It was used for such tasks as submitting batch jobs, formatting files for the printer, collecting statistics on a file. A collection of macros to do various useful tasks was put in a commonly available place.

I truly hope in my heart of hearts that ken had no inkling that someday QED would be used as the scripting language to power http://www.qef.com/html/quickref.html. Not that QEF is bad, just that any stones thrown at “you shouldn’t use $lang for that!” feel like they must only strike via ricochet off QEF.

I can only imagine the shock and horror Larry McVoy would experience if one of his team suggested replacing make with QEF. 

[-- Attachment #2: Type: text/html, Size: 2933 bytes --]

[TUHS] Re: A few comments on porting the Bourne shell

[Steve Nickolas]

On Mon, 2 Jan 2023, Clem Cole wrote:

> On Mon, Jan 2, 2023 at 4:03 PM Dan Cross <crossd@gmail.com> wrote:
>
>> Couldn't you also ship a shell? That kinda seems like the simplest
>> solution.
>>
> Sure the problem is -- where do you stop?
>
> The result is the fork/mess called Gnu.

GNU is Embrace, Extend and Exterminate applied to Unix.

-uso.

[TUHS] Re: A few comments on porting the Bourne shell

[Joseph Holsten]

On Mon, Jan 2, 2023, at 13:00, Dan Cross wrote:
> On Mon, Jan 2, 2023 at 3:41 PM Larry McVoy <lm@mcvoy.com> wrote:
>>
>> If we wanted to use Makefiles in our customer run product, we would have
>> shipped it.
>
> Couldn't you also ship a shell? That kinda seems like the simplest solution.

Then you have to ship documentation on how to properly install the shell. Also, add guardrails to make sure that the correct shell is being loaded instead of someone getting ambitious with their exec path.

cf every products trying to ship its own python, ruby, etc instead of accepting defeat and shipping a statically build tool in a docker container in an OVA virtual machine image.

Obviously, that’s what should have been done with Bourne shell in the first place.

[TUHS] Re: A few comments on porting the Bourne shell

[Adam Thornton]

> On Jan 2, 2023, at 2:08 PM, Joseph Holsten <joseph@josephholsten.com> wrote:
> 
> On Mon, Jan 2, 2023, at 13:00, Dan Cross wrote:
>> On Mon, Jan 2, 2023 at 3:41 PM Larry McVoy <lm@mcvoy.com> wrote:
>>> 
>>> If we wanted to use Makefiles in our customer run product, we would have
>>> shipped it.
>> 
>> Couldn't you also ship a shell? That kinda seems like the simplest solution.
> 
> Then you have to ship documentation on how to properly install the shell. Also, add guardrails to make sure that the correct shell is being loaded instead of someone getting ambitious with their exec path.
> 
> cf every products trying to ship its own python, ruby, etc instead of accepting defeat and shipping a statically build tool in a docker container in an OVA virtual machine image.
> 
> Obviously, that’s what should have been done with Bourne shell in the first place.

You came very close to owing me a new keyboard there.

Adam

[TUHS] Re: A few comments on porting the Bourne shell

[Adam Thornton]

> On Jan 2, 2023, at 10:43 AM, Larry McVoy <lm@mcvoy.com> wrote:
> 
> When you get to the commercial world, you'd be stunned to see how long
> old machines last.  If they are solving some problem, and they aren't
> broken, nobody replaces them.  I'll bet you anything there are still
> SCO registers out there, I'll bet there are still PDP-11s out there.
> 
> If it ain't broke, don't fix it.

Well, sure, but this also comes down to at what point is losing some customers because they refuse to upgrade to something from this millenium worth the cost you would incur to be able to support their environment?

To be fair, stuff exactly like that was part of the bread-and-butter of Sine Nomine while I was there (for all I know, still is), and I kinda feel like that's the right answer: the places running old technology know they're doing so, and maybe they're doing so for good reasons.  But finding people willing and able to keep new stuff working on those machines and OSes does get more expensive over time.  So there's an aftermarket in consultancies that are willing to take over maintenance of things their vendors have abandoned.  No company concerned with its long-term survival *should* do that without, in parallel, pursuing an exit strategy towards something actually-maintainable at not-exponentially-increasing cost, but of course the inability to look beyond the next quarter is a fundamental flaw of financialization-driven capitalism.

And no one is saying you can't keep running your SCO cash registers forever on your own recognizance, but when they break, you get to keep both pieces, and if they break in such a way that suddenly your entire operation is being held hostage by some Russian organized crime cartel, well, you should have priced that risk appropriately.

Adam

[TUHS] Re: A few comments on porting the Bourne shell

[Clem Cole]

[-- Attachment #1: Type: text/plain, Size: 1045 bytes --]

Maybe this should go to COFF but Adam I fear you are falling into a tap
that is easy to fall into - old == unused

One of my favorite stores in the computer restoration community is from
5-10 years ago and the LCM+L in Seatle was restoring their CDC-6000 that
they got From Purdue.   Core memory is difficult to get, so they made a
card and physical module that could plug into their system that is both
electrically and mechanically equivalent using modern semiconductors.   A
few weeks later they announced that they had the system running and had
built this module. They got approached by the USAF asking if they could get
a copy of the design.  Seems, there was still a least one CDC-6600 running
a particular critical application somewhere.

This is similar to the PDP-11s and Vaxen that are supposed to be still in
the hydroelectric grid [a few years ago the was an ad for an RSX and VMS
programmer to go to Canada running in the Boston newspapers - I know
someone that did a small amount of consulting on that one].
ᐧ

[-- Attachment #2: Type: text/html, Size: 1782 bytes --]

[TUHS] Re: A few comments on porting the Bourne shell

[Dan Cross]

On Mon, Jan 2, 2023 at 1:13 PM Clem Cole <clemc@ccc.com> wrote:
> Maybe this should go to COFF but Adam I fear you are falling into a tap that is easy to fall into - old == unused
>
> One of my favorite stores in the computer restoration community is from 5-10 years ago and the LCM+L in Seatle was restoring their CDC-6000 that they got From Purdue.   Core memory is difficult to get, so they made a card and physical module that could plug into their system that is both electrically and mechanically equivalent using modern semiconductors.   A few weeks later they announced that they had the system running and had built this module. They got approached by the USAF asking if they could get a copy of the design.  Seems, there was still a least one CDC-6600 running a particular critical application somewhere.
>
> This is similar to the PDP-11s and Vaxen that are supposed to be still in the hydroelectric grid [a few years ago the was an ad for an RSX and VMS programmer to go to Canada running in the Boston newspapers - I know someone that did a small amount of consulting on that one].

One of my favorite stories along these lines is the train signalling
system in Melbourne, running on a "PDP-11". I quote PDP-11 because
that is now virtualized:
https://www.equicon.de/images/Virtualisierung/LegacyTrainControlSystemStabilization.pdf

Indeed older systems show up in surprising places. I was once on the
bridge of a US Naval vessel in the late '00s and saw a SPARCstaton 20
running Solaris (with the CDE login screen). I don't recall what it
was doing, but it was a tad surprising.

I do worry about legacy systems in critical situations, but then, I've
been in a firefight when the damned tactical computer with the satcomm
link rebooted and we didn't have VHF comms with the battlespace owner.
That was not particularly fun.

        - Dan C.

[TUHS] Re: A few comments on porting the Bourne shell

[Dan Cross]

[Apologies for resending; I messed up and used the old Minnie address
for COFF in the Cc]

On Mon, Jan 2, 2023 at 1:36 PM Dan Cross <crossd@gmail.com> wrote:
>
> On Mon, Jan 2, 2023 at 1:13 PM Clem Cole <clemc@ccc.com> wrote:
> > Maybe this should go to COFF but Adam I fear you are falling into a tap that is easy to fall into - old == unused
> >
> > One of my favorite stores in the computer restoration community is from 5-10 years ago and the LCM+L in Seatle was restoring their CDC-6000 that they got From Purdue.   Core memory is difficult to get, so they made a card and physical module that could plug into their system that is both electrically and mechanically equivalent using modern semiconductors.   A few weeks later they announced that they had the system running and had built this module. They got approached by the USAF asking if they could get a copy of the design.  Seems, there was still a least one CDC-6600 running a particular critical application somewhere.
> >
> > This is similar to the PDP-11s and Vaxen that are supposed to be still in the hydroelectric grid [a few years ago the was an ad for an RSX and VMS programmer to go to Canada running in the Boston newspapers - I know someone that did a small amount of consulting on that one].
>
> One of my favorite stories along these lines is the train signalling
> system in Melbourne, running on a "PDP-11". I quote PDP-11 because
> that is now virtualized:
> https://www.equicon.de/images/Virtualisierung/LegacyTrainControlSystemStabilization.pdf
>
> Indeed older systems show up in surprising places. I was once on the
> bridge of a US Naval vessel in the late '00s and saw a SPARCstaton 20
> running Solaris (with the CDE login screen). I don't recall what it
> was doing, but it was a tad surprising.
>
> I do worry about legacy systems in critical situations, but then, I've
> been in a firefight when the damned tactical computer with the satcomm
> link rebooted and we didn't have VHF comms with the battlespace owner.
> That was not particularly fun.
>
>         - Dan C.

[TUHS] Re: A few comments on porting the Bourne shell

[Larry McVoy]

On Mon, Jan 02, 2023 at 10:55:51AM -0700, Adam Thornton wrote:
> And no one is saying you can't keep running your SCO cash registers
> forever on your own recognizance, but when they break, you get to keep
> both pieces, and if they break in such a way that suddenly your entire
> operation is being held hostage by some Russian organized crime cartel,
> well, you should have priced that risk appropriately.

Not my problem, I'm not running SCO cash registers, but the customer
might be.  If they have enough money to interest me, then my problem is
having a product that works on their platform.

You argue with me all you want, you aren't going to win.  Writing stuff
in a portable way is not that hard and once you have that figured out,
you just do it.  It's the path to more revenue.

And I've fought this battle with some extremely intelligent and well
meaning engineers, absolutely in the top 1% of software engineers,
and they all caved.  Sometimes it took me letting them off the leash
long enough to make something break, but they all caved eventually.

It's just more pleasant knowing that your stuff will work no matter
what /bin/sh is there.  All the bash-ism in the world are less 
valuable than "it just works".  Don't get me wrong, I use bash-ism
for my personal use but I wouldn't let them get anywhere near a
product I needed to ship.

All that said, I'm done with this topic, feel free to have the last
word, I think I'm boring the smart people at this point.

[TUHS] Re: A few comments on porting the Bourne shell

[John Cowan]

[-- Attachment #1: Type: text/plain, Size: 713 bytes --]

On Mon, Jan 2, 2023 at 12:43 PM Larry McVoy <lm@mcvoy.com> wrote:


> I was supporting a commercial product in the early 2000's and there
> were all sorts of systems then that had old shells.  Yes, you could make
> everything a 2 level thing where the first level finds the correct shell,
> but that's just fuss.  Just make things portable, it's not that hard
> and it works everywhere.
>

Making shell scripts portable means not using pipelines, because given "foo
| bar", kshNN and zsh execute foo in a subshell and bar in the top-level
shell, whereas in other shells, both foo and bar execute in subshells.
(For this reason, Posix allows either behavior.)  Not having pipelines is a
pretty drastic limitation.

[-- Attachment #2: Type: text/html, Size: 1282 bytes --]

[TUHS] Re: A few comments on porting the Bourne shell

[Dan Cross]

On Tue, Jan 3, 2023 at 10:22 PM John Cowan <cowan@ccil.org> wrote:
> Making shell scripts portable means not using pipelines, because given "foo | bar", kshNN and zsh execute foo in a subshell and bar in the top-level shell, whereas in other shells, both foo and bar execute in subshells.  (For this reason, Posix allows either behavior.)  Not having pipelines is a pretty drastic limitation.

This came up at work just the other day:

echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
echo $bummer

The behavior varies between ksh, zsh, bash, sh. Yay.

        - Dan C.

[TUHS] Re: A few comments on porting the Bourne shell

[Bakul Shah]

On Jan 3, 2023, at 7:31 PM, Dan Cross <crossd@gmail.com> wrote:
> 
> On Tue, Jan 3, 2023 at 10:22 PM John Cowan <cowan@ccil.org> wrote:
>> Making shell scripts portable means not using pipelines, because given "foo | bar", kshNN and zsh execute foo in a subshell and bar in the top-level shell, whereas in other shells, both foo and bar execute in subshells.  (For this reason, Posix allows either behavior.)  Not having pipelines is a pretty drastic limitation.
> 
> This came up at work just the other day:
> 
> echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
> echo $bummer
> 
> The behavior varies between ksh, zsh, bash, sh. Yay.

On this example sh, zsh, bash behave the same way on
freebsd. Rather than ban |, I think the lesson is to
*avoid* builtin commands in a pipeline that can affect
shell's environment. I only write /bin/sh scripts in
any case.

It is really unfortunate that infix | is so beguiling!
Except for that a Lisp or Scheme based shell would've
been fine and dandy! :-)/2

[TUHS] Re: A few comments on porting the Bourne shell

[Dan Cross]

On Tue, Jan 3, 2023 at 11:16 PM Bakul Shah <bakul@iitbombay.org> wrote:
> On Jan 3, 2023, at 7:31 PM, Dan Cross <crossd@gmail.com> wrote:
> > On Tue, Jan 3, 2023 at 10:22 PM John Cowan <cowan@ccil.org> wrote:
> >> Making shell scripts portable means not using pipelines, because given "foo | bar", kshNN and zsh execute foo in a subshell and bar in the top-level shell, whereas in other shells, both foo and bar execute in subshells.  (For this reason, Posix allows either behavior.)  Not having pipelines is a pretty drastic limitation.
> >
> > This came up at work just the other day:
> >
> > echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
> > echo $bummer
> >
> > The behavior varies between ksh, zsh, bash, sh. Yay.
>
> On this example sh, zsh, bash behave the same way on
> freebsd. Rather than ban |, I think the lesson is to
> *avoid* builtin commands in a pipeline that can affect
> shell's environment. I only write /bin/sh scripts in
> any case.

Interesting. In my testing, `bash` was the odd man out:

: doctor; zsh
: doctor; echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
echo $bummer

hi
: doctor;
: doctor; bash
: doctor; echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
echo $bummer

: doctor;
exit
: doctor; ksh
: doctor; echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
: doctor; echo $bummer
hi
: doctor;
: doctor; sh
: doctor; echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
: doctor; echo $bummer
hi
: doctor;
: doctor;

That's on illumos.

On OpenBSD, neither bash, ksh, or sh produce output, while zsh does. I
suspect ksh is pdksh and sh is a link to ksh there, though. ksh93
gives "hi".

On FreeBSD, neither bash nor sh produce "hi", but zsh and ksh93 do.
Same on DragonFly, macOS and Arch Linux. On 7th edition, I only tried
sh; empty output. 4.3BSD gives empty output for /bin/sh. For kicks,
neither tcsh nor csh understand `read`; I don't think that *csh has a
read-like builtin.

This is, of course, a silly example. Most script authors would do
something like:

    bummer=`echo ' hi ' | sed 's/^ *//;s/ *$//'`

Or, perhaps even:

    bummer=$(echo ' hi ' | sed 's/^ *//;s/ *$//')

Life's too short and at this point I prefer the latter, but more often
than not my fingers are wired for the former.

> It is really unfortunate that infix | is so beguiling!
> Except for that a Lisp or Scheme based shell would've
> been fine and dandy! :-)/2

Whatever happened to `es` ? I guess it died on the vine. It seemed
like a nifty idea.

I made a concerted effort to use `rc` as my shell on Unix for a little
while. Without all the supporting plan9 machinery, though, I gave up.
Minimalism is great in an environment that can augment it with useful
functionality; outside of that environment, it can be kind of a pain
for work-a-day use.

        - Dan C.

[TUHS] Re: A few comments on porting the Bourne shell

[ron minnich]

[-- Attachment #1: Type: text/plain, Size: 3796 bytes --]

What I see so far from this discussion: Ted has written a lot of useful
scripts, but they are explicitly for one shell (bash) on one kernel
(linux). I.e. portability is not a concern.

People immediately pointed out how easy it is to write non-portable shell
scripts, and others commented that it does not matter (to them).

Kernel portability matters to me. So does code that works without regard to
the shell someone is using. So does robust error handling. So does math.
These are all things that, in my experience, rule against complex shell
scripts.

I still prefer Go as my scripting language, but we all have our
preferences. And, yes, Go gets wordy, but it's a price I'm happy to pay for
good error handling. I much prefer an error message to 100 lines of python
backtrace.

On Wed, Jan 4, 2023 at 8:17 AM Dan Cross <crossd@gmail.com> wrote:

> On Tue, Jan 3, 2023 at 11:16 PM Bakul Shah <bakul@iitbombay.org> wrote:
> > On Jan 3, 2023, at 7:31 PM, Dan Cross <crossd@gmail.com> wrote:
> > > On Tue, Jan 3, 2023 at 10:22 PM John Cowan <cowan@ccil.org> wrote:
> > >> Making shell scripts portable means not using pipelines, because
> given "foo | bar", kshNN and zsh execute foo in a subshell and bar in the
> top-level shell, whereas in other shells, both foo and bar execute in
> subshells.  (For this reason, Posix allows either behavior.)  Not having
> pipelines is a pretty drastic limitation.
> > >
> > > This came up at work just the other day:
> > >
> > > echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
> > > echo $bummer
> > >
> > > The behavior varies between ksh, zsh, bash, sh. Yay.
> >
> > On this example sh, zsh, bash behave the same way on
> > freebsd. Rather than ban |, I think the lesson is to
> > *avoid* builtin commands in a pipeline that can affect
> > shell's environment. I only write /bin/sh scripts in
> > any case.
>
> Interesting. In my testing, `bash` was the odd man out:
>
> : doctor; zsh
> : doctor; echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
> echo $bummer
>
> hi
> : doctor;
> : doctor; bash
> : doctor; echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
> echo $bummer
>
> : doctor;
> exit
> : doctor; ksh
> : doctor; echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
> : doctor; echo $bummer
> hi
> : doctor;
> : doctor; sh
> : doctor; echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
> : doctor; echo $bummer
> hi
> : doctor;
> : doctor;
>
> That's on illumos.
>
> On OpenBSD, neither bash, ksh, or sh produce output, while zsh does. I
> suspect ksh is pdksh and sh is a link to ksh there, though. ksh93
> gives "hi".
>
> On FreeBSD, neither bash nor sh produce "hi", but zsh and ksh93 do.
> Same on DragonFly, macOS and Arch Linux. On 7th edition, I only tried
> sh; empty output. 4.3BSD gives empty output for /bin/sh. For kicks,
> neither tcsh nor csh understand `read`; I don't think that *csh has a
> read-like builtin.
>
> This is, of course, a silly example. Most script authors would do
> something like:
>
>     bummer=`echo ' hi ' | sed 's/^ *//;s/ *$//'`
>
> Or, perhaps even:
>
>     bummer=$(echo ' hi ' | sed 's/^ *//;s/ *$//')
>
> Life's too short and at this point I prefer the latter, but more often
> than not my fingers are wired for the former.
>
> > It is really unfortunate that infix | is so beguiling!
> > Except for that a Lisp or Scheme based shell would've
> > been fine and dandy! :-)/2
>
> Whatever happened to `es` ? I guess it died on the vine. It seemed
> like a nifty idea.
>
> I made a concerted effort to use `rc` as my shell on Unix for a little
> while. Without all the supporting plan9 machinery, though, I gave up.
> Minimalism is great in an environment that can augment it with useful
> functionality; outside of that environment, it can be kind of a pain
> for work-a-day use.
>
>         - Dan C.
>

[-- Attachment #2: Type: text/html, Size: 4847 bytes --]

[TUHS] Re: A few comments on porting the Bourne shell

[Chet Ramey]

On 1/4/23 11:15 AM, Dan Cross wrote:
> On Tue, Jan 3, 2023 at 11:16 PM Bakul Shah <bakul@iitbombay.org> wrote:
>> On Jan 3, 2023, at 7:31 PM, Dan Cross <crossd@gmail.com> wrote:
>>> On Tue, Jan 3, 2023 at 10:22 PM John Cowan <cowan@ccil.org> wrote:
>>>> Making shell scripts portable means not using pipelines, because given "foo | bar", kshNN and zsh execute foo in a subshell and bar in the top-level shell, whereas in other shells, both foo and bar execute in subshells.  (For this reason, Posix allows either behavior.)  Not having pipelines is a pretty drastic limitation.
>>>
>>> This came up at work just the other day:
>>>
>>> echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
>>> echo $bummer
>>>
>>> The behavior varies between ksh, zsh, bash, sh. Yay.
>>
>> On this example sh, zsh, bash behave the same way on
>> freebsd. Rather than ban |, I think the lesson is to
>> *avoid* builtin commands in a pipeline that can affect
>> shell's environment. I only write /bin/sh scripts in
>> any case.
> 
> Interesting. In my testing, `bash` was the odd man out:
> 
> : doctor; zsh
> : doctor; echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
> echo $bummer
> 
> hi
> : doctor;
> : doctor; bash
> : doctor; echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
> echo $bummer
> 
> : doctor;
> exit
> : doctor; ksh
> : doctor; echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
> : doctor; echo $bummer
> hi
> : doctor;
> : doctor; sh
> : doctor; echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
> : doctor; echo $bummer
> hi
> : doctor;
> : doctor;
> 
> That's on illumos.

On Illumos, /bin/sh and /usr/xpg4/bin/sh are both links to ksh93.

ksh88/ksh93 and zsh are the only shells that unconditionally run the last
element of a pipeline in the current shell. Bash will do it if job control
is not enabled and the `lastpipe' option is enabled.

It's usually not a problem, but you do have to be aware of potential side
effects.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
		 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/

[TUHS] Re: A few comments on porting the Bourne shell

[Ralph Corderoy]

Hi John,

> Making shell scripts portable means not using pipelines

No it doesn't...

> because given "foo | bar", kshNN and zsh execute foo in a subshell and
> bar in the top-level shell, whereas in other shells, both foo and bar
> execute in subshells.

...because a lot of the time that doesn't matter.

-- 
Cheers, Ralph.

[TUHS] Re: A few comments on porting the Bourne shell

[Ron Natalie]

Indeed.    MiniUnix had no hardware pipes, but the shell “simulated” it 
by writing to a intermediate temp file.


------ Original Message ------
From "Ralph Corderoy" <ralph@inputplus.co.uk>
To tuhs@tuhs.org
Date 1/4/2023 10:21:02 AM
Subject [TUHS] Re: A few comments on porting the Bourne shell

>Hi John,
>
>>  Making shell scripts portable means not using pipelines
>
>No it doesn't...
>
>>  because given "foo | bar", kshNN and zsh execute foo in a subshell and
>>  bar in the top-level shell, whereas in other shells, both foo and bar
>>  execute in subshells.
>
>...because a lot of the time that doesn't matter.
>
>--
>Cheers, Ralph.

[TUHS] Re: A few comments on porting the Bourne shell

[Clem Cole]

[-- Attachment #1: Type: text/plain, Size: 1641 bytes --]

On Mon, Jan 2, 2023 at 11:51 AM Larry McVoy <lm@mcvoy.com> wrote:

> I think it is less of an issue today but if I were still supporting a multi
> platform product, I'd still insist on it.
>
Yep, and it's so easy to do.  It's just a discipline.  @Adam I helped
create POSIX, so I get it.
But Larry nailed it.  It is just cheaper and easier to be disciplined and
stick with Bourne for your scripts.  *Teach your people the skills and you
save time and money in the long run.*   It's that simple.  As Larry says,
you never know and the problem is - when it happens, it tends to happen on
a short leash.  If you have been disciplined, it's a non-problem.  It's
really not that hard to use the V7 syntax.  Everything you want/need to do
is there.

BTW: At Intel, a couple of years back (less than 3-5 years ago)  we had a
site where we needed things to work on a specific target that was, shall we
say 'a bit custom' - V7 syntax was just fine for the installer - boy folks
were happy a few of us had been on their case to get rid of the bashism the
Millenials had tried to add (I'm not really sure POSIX.2 would have been
good enough -- maybe - but Bourne was fine].

FWIW: In my start-up times, under the same rules of being disciplined, as
VP of Engineering, I insisted, all C and C++ code was required to
'flex-e-lint' warning clean.   I gave my folks a 3-week week slip to clean
everything up.   I was cursed during that time.   But guess what, the
outstanding bug list dropped to ⅒ of what it had been.  Created quite a few
true believers.  And we made those 3 weeks back before we were done.

Clem


ᐧ

[-- Attachment #2: Type: text/html, Size: 3104 bytes --]

[TUHS] Re: A few comments on porting the Bourne shell

[Paul Winalski]

On 1/2/23, Clem Cole <clemc@ccc.com> wrote:
>
> FWIW: In my start-up times, under the same rules of being disciplined, as
> VP of Engineering, I insisted, all C and C++ code was required to
> 'flex-e-lint' warning clean.   I gave my folks a 3-week week slip to clean
> everything up.   I was cursed during that time.   But guess what, the
> outstanding bug list dropped to ⅒ of what it had been.  Created quite a few
> true believers.  And we made those 3 weeks back before we were done.

This was also the policy in DEC's compiler and software development
tools groups.  This was mainly VMS stuff and we didn't have flex and
lint, but as Clem can attest the C and C++ compilers had very
extensive warning capabilities.  It was group policy that all code had
to compile cleanly, without triggering diagnostic messages, before
check-in was allowed.  Once you get through the initial cleanup of
existing code, this policy pays back big time in avoidance of nasty
Heisenbugs.

-Paul W.

[TUHS] Re: A few comments on porting the Bourne shell

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 1481 bytes --]

On Tue, Jan 3, 2023, 10:09 AM Paul Winalski <paul.winalski@gmail.com> wrote:

> On 1/2/23, Clem Cole <clemc@ccc.com> wrote:
> >
> > FWIW: In my start-up times, under the same rules of being disciplined, as
> > VP of Engineering, I insisted, all C and C++ code was required to
> > 'flex-e-lint' warning clean.   I gave my folks a 3-week week slip to
> clean
> > everything up.   I was cursed during that time.   But guess what, the
> > outstanding bug list dropped to ⅒ of what it had been.  Created quite a
> few
> > true believers.  And we made those 3 weeks back before we were done.
>
> This was also the policy in DEC's compiler and software development
> tools groups.  This was mainly VMS stuff and we didn't have flex and
> lint, but as Clem can attest the C and C++ compilers had very
> extensive warning capabilities.  It was group policy that all code had
> to compile cleanly, without triggering diagnostic messages, before
> check-in was allowed.  Once you get through the initial cleanup of
> existing code, this policy pays back big time in avoidance of nasty
> Heisenbugs.
>

Not all fixes to appease warnings fix anything. But enough do that it's
worth it.

The one caveat here is that people must understand the warning and that any
change makes things better. There is nothing worse than just tossing a cast
in to brute force it, only to later discover it's the wrong cast or you
needed a different semantic change.

Warner

>

[-- Attachment #2: Type: text/html, Size: 2210 bytes --]

[TUHS] Re: A few comments on porting the Bourne shell

[Luther Johnson]

[-- Attachment #1: Type: text/plain, Size: 1908 bytes --]

All warnings are worthy of my consideration. I will usually learn 
something about my compiler, my program, and often both. The compiler's 
messages are valuable feedback in my quality work-style loop.

On 01/03/2023 12:19 PM, Warner Losh wrote:
>
>
> On Tue, Jan 3, 2023, 10:09 AM Paul Winalski <paul.winalski@gmail.com 
> <mailto:paul.winalski@gmail.com>> wrote:
>
>     On 1/2/23, Clem Cole <clemc@ccc.com <mailto:clemc@ccc.com>> wrote:
>     >
>     > FWIW: In my start-up times, under the same rules of being
>     disciplined, as
>     > VP of Engineering, I insisted, all C and C++ code was required to
>     > 'flex-e-lint' warning clean.   I gave my folks a 3-week week
>     slip to clean
>     > everything up.   I was cursed during that time.   But guess
>     what, the
>     > outstanding bug list dropped to ⅒ of what it had been.  Created
>     quite a few
>     > true believers.  And we made those 3 weeks back before we were done.
>
>     This was also the policy in DEC's compiler and software development
>     tools groups.  This was mainly VMS stuff and we didn't have flex and
>     lint, but as Clem can attest the C and C++ compilers had very
>     extensive warning capabilities.  It was group policy that all code had
>     to compile cleanly, without triggering diagnostic messages, before
>     check-in was allowed.  Once you get through the initial cleanup of
>     existing code, this policy pays back big time in avoidance of nasty
>     Heisenbugs.
>
>
> Not all fixes to appease warnings fix anything. But enough do that 
> it's worth it.
>
> The one caveat here is that people must understand the warning and 
> that any change makes things better. There is nothing worse than just 
> tossing a cast in to brute force it, only to later discover it's the 
> wrong cast or you needed a different semantic change.
>
> Warner
>


[-- Attachment #2: Type: text/html, Size: 3771 bytes --]

[TUHS] Re: A few comments on porting the Bourne shell

[Dave Horsfall]

[-- Attachment #1: Type: text/plain, Size: 799 bytes --]

On Tue, 3 Jan 2023, Warner Losh wrote:

> Not all fixes to appease warnings fix anything. But enough do that it's 
> worth it.

I consider warnings to be potential errors, and act accordingly; one place 
where I worked insisted that code could not be checked in unless it passed 
"-Wall" (no guarantee of course, but a good start).

> The one caveat here is that people must understand the warning and that 
> any change makes things better. There is nothing worse than just tossing 
> a cast in to brute force it, only to later discover it's the wrong cast 
> or you needed a different semantic change. 

Yeah, that's just painting over the cracks; it's like replacing a fuse 
that keeps blowing, or rebooting a box without fixing the bug (what I call 
the M$ Solution).

-- Dave

[TUHS] Re: A few comments on porting the Bourne shell

[Clem Cole]

[-- Attachment #1: Type: text/plain, Size: 2516 bytes --]

Bring this back to TUHS relevance...

On Tue, Jan 3, 2023 at 2:43 PM Warner Losh <imp@bsdimp.com> wrote:

> The one caveat here is that people must understand the warning and that
> any change makes things better. There is nothing worse than just tossing a
> cast in to brute force it, only to later discover it's the wrong cast or
> you needed a different semantic change.
>
Most certainly --   It's why I >>loved<< Gimpel's flex-e-lint product and
as Paul pointed out, Judy Ward's messages from the DEC Gem compiler - both
were the two best I ever ran into in giving you real information about what
was happening.

I also tell a story from my DEC time.   After the GEM compiler was released
and before most other vendors were 64-bits, the ISVs were first starting to
do their ports to Alpha.   My complaining ensued. We would discover from
numerous ISVs that after the Alpha port was complete, their bug count
dropped - why because the ISV's code has been kinda nasty and the older
compilers had been silent about it they had assumed the ILP32 model.
Alphas, using LP64, could not be.  Judy would find things and say -- what a
minute -- you want me to do what with that and issued a fairly detailed
warning (which was the key - she explained what the issue was).  Often the
32 to 64-bit nature forced the programmers at the ISV's to rethink how there
were actually declaring things to make the code clearer, simpler, better,
*etc*.   The classic rewrite never happens unless you are forced too.

I remember going to a Supercomputer conference and talking with the
developers at one the ISVs who I will not name.  He thanks me.  He said his
team has been arguing with their management for years to redo the UNIX
support library. The Tru64 port was what finally allowed them to do it.
But it took 9 months which pissed off his boss. But when it was completed,
and pass all the tests on the Alpha, it just recompiled on Solaris, AIX,
and HP-UX - which had never happened before. He could not believe what a
great compiler we had.

I've always said the Alpha was the greatest gift to Sun and Intel in the
commercial SW world because it forced the ISV to clean up their act before
they ever saw those processors and Sun/Intel ports were piece of cake.   It
was not that porting to Alpha was difficult -- it was cleaning up your own
mess.

Remember we had already gone through this in the PDP-11 ILP16 to Vax ILP32
transition but it is funny how history repeated itself.
ᐧ

[-- Attachment #2: Type: text/html, Size: 5235 bytes --]

[TUHS] Re: A few comments on porting the Bourne shell

[Clem Cole]

[-- Attachment #1: Type: text/plain, Size: 2741 bytes --]

typo: s/My complaining/Much complaining/
sigh.. dyslex-r-me
ᐧ

On Tue, Jan 3, 2023 at 4:47 PM Clem Cole <clemc@ccc.com> wrote:

>
> Bring this back to TUHS relevance...
>
> On Tue, Jan 3, 2023 at 2:43 PM Warner Losh <imp@bsdimp.com> wrote:
>
>> The one caveat here is that people must understand the warning and that
>> any change makes things better. There is nothing worse than just tossing a
>> cast in to brute force it, only to later discover it's the wrong cast or
>> you needed a different semantic change.
>>
> Most certainly --   It's why I >>loved<< Gimpel's flex-e-lint product and
> as Paul pointed out, Judy Ward's messages from the DEC Gem compiler - both
> were the two best I ever ran into in giving you real information about what
> was happening.
>
> I also tell a story from my DEC time.   After the GEM compiler was
> released and before most other vendors were 64-bits, the ISVs were first
> starting to do their ports to Alpha.   My complaining ensued. We would discover
> from numerous ISVs that after the Alpha port was complete, their bug count
> dropped - why because the ISV's code has been kinda nasty and the older
> compilers had been silent about it they had assumed the ILP32 model.
> Alphas, using LP64, could not be.  Judy would find things and say -- what
> a minute -- you want me to do what with that and issued a fairly detailed
> warning (which was the key - she explained what the issue was).  Often
> the 32 to 64-bit nature forced the programmers at the ISV's to rethink
> how there were actually declaring things to make the code clearer,
> simpler, better, *etc*.   The classic rewrite never happens unless you
> are forced too.
>
> I remember going to a Supercomputer conference and talking with the
> developers at one the ISVs who I will not name.  He thanks me.  He said his
> team has been arguing with their management for years to redo the UNIX
> support library. The Tru64 port was what finally allowed them to do it.
> But it took 9 months which pissed off his boss. But when it was completed,
> and pass all the tests on the Alpha, it just recompiled on Solaris, AIX,
> and HP-UX - which had never happened before. He could not believe what a
> great compiler we had.
>
> I've always said the Alpha was the greatest gift to Sun and Intel in the
> commercial SW world because it forced the ISV to clean up their act before
> they ever saw those processors and Sun/Intel ports were piece of cake.   It
> was not that porting to Alpha was difficult -- it was cleaning up your own
> mess.
>
> Remember we had already gone through this in the PDP-11 ILP16 to Vax
> ILP32  transition but it is funny how history repeated itself.
> ᐧ
>

[-- Attachment #2: Type: text/html, Size: 6172 bytes --]

[TUHS] Re: A few comments on porting the Bourne shell

[Greg 'groggy' Lehey]

[-- Attachment #1: Type: text/plain, Size: 1096 bytes --]

On Friday, 30 December 2022 at 19:59:31 -0800, Larry McVoy wrote:
> Over and over, I won the "straight bourne shell only" battle.  So I agree,
> if you want /bin/sh to work, Bourne shell for the win.

Agreed.

> For a login shell, bash is my shell of choice.  It's bloated but I'm
> typing this on a 5 year old Lenova X1 Carbon with 16GB of memory and
> 4 cores and it's fine.  It was fine a 133mhz Pentium.

I've been using bash since 16 MHz i386s.  People told me that it was
bloated, but somehow I didn't notice.

Somehow this reminds me of the expansion of EMACS: Eight Megabytes And
Continually Swapping.  8 MB?  Nowadays?  It shows how old these
prejudices are.  Now my emacs processes have round 30 MB of resident
memory and are dwarfed by 2 GB web browsers.  bash comes in at a
little over 2 MB, just not worth talking about.

Greg
--
Sent from my desktop computer.
Finger grog@lemis.com for PGP public key.
See complete headers for address and phone numbers.
This message is digitally signed.  If your Microsoft mail program
reports problems, please read http://lemis.com/broken-MUA.php

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 163 bytes --]

[TUHS] Re: A few comments on porting the Bourne shell

[Sven Mascheck]

Chet Ramey on 30.12.2022 20:51:
> Arnold Robbins built on that work and ported the v8-v10 shells to modern
> Linux versions. (I am sorry, I do not have a link right now.)

And btw, 8th ed (http://man.cat-v.org/unix_8th/1/sh) even added a simple 
history mechanism with the "=(1)" command 
(https://www.in-ulm.de/~mascheck/bourne/v8/=.html).

Sven

[TUHS] Re: A few comments on porting the Bourne shell

[Ron Natalie]

I was just happy when VR2 came out and found someone had undone all 
those macros that were in the original code.
I hacked Berkeley Job Control as well as command line editing into it 
(KSH hadn’t seen the light of day outside the labs at that point).
I subsequently had a nice talk with Korn at a USENIX.    I also sat down 
with a couple of the guys trying to implement their own shell 
independent of the ATT code and explained to them how Berkeley job 
control works.   It’s for that reason my name shows up in a lot of the 
early Linux docs.

Amusingly, I’d forgotten all about this stuff until one day I was 
sitting at a MIPS workstation (MIPS branded, not the DEC spim).    
Without thinking about it, I typed “fg” at the shell prompt.

“Job control not Enabled,” it said.   Hey!  That sounds like one of my 
messages.   “set -J” I type.   “Job control enabled.”
Hey!  This is a “Ron shell” as it was known at BRL.   Turns out it went 
out on the Mach distros so I’ve found it on all kinds of things like the 
NeXT etc…


------ Original Message ------
From "Sven Mascheck" <mascheck@in-ulm.de>
To tuhs@tuhs.org
Date 12/30/2022 3:20:28 PM
Subject [TUHS] Re: A few comments on porting the Bourne shell

>Chet Ramey on 30.12.2022 20:51:
>>Arnold Robbins built on that work and ported the v8-v10 shells to modern
>>Linux versions. (I am sorry, I do not have a link right now.)
>
>And btw, 8th ed (http://man.cat-v.org/unix_8th/1/sh) even added a simple history mechanism with the "=(1)" command (https://www.in-ulm.de/~mascheck/bourne/v8/=.html).
>
>Sven
>

[TUHS] Re: A few comments on porting the Bourne shell

[Rob Pike]

[-- Attachment #1: Type: text/plain, Size: 1907 bytes --]

That "someone" was Steve Bourne. At least, the version he handed me when I
built up the v8 shell was from him and clean of ELIFs.

-rob


On Sat, Dec 31, 2022 at 7:50 AM Ron Natalie <ron@ronnatalie.com> wrote:

> I was just happy when VR2 came out and found someone had undone all
> those macros that were in the original code.
> I hacked Berkeley Job Control as well as command line editing into it
> (KSH hadn’t seen the light of day outside the labs at that point).
> I subsequently had a nice talk with Korn at a USENIX.    I also sat down
> with a couple of the guys trying to implement their own shell
> independent of the ATT code and explained to them how Berkeley job
> control works.   It’s for that reason my name shows up in a lot of the
> early Linux docs.
>
> Amusingly, I’d forgotten all about this stuff until one day I was
> sitting at a MIPS workstation (MIPS branded, not the DEC spim).
> Without thinking about it, I typed “fg” at the shell prompt.
>
> “Job control not Enabled,” it said.   Hey!  That sounds like one of my
> messages.   “set -J” I type.   “Job control enabled.”
> Hey!  This is a “Ron shell” as it was known at BRL.   Turns out it went
> out on the Mach distros so I’ve found it on all kinds of things like the
> NeXT etc…
>
>
> ------ Original Message ------
> From "Sven Mascheck" <mascheck@in-ulm.de>
> To tuhs@tuhs.org
> Date 12/30/2022 3:20:28 PM
> Subject [TUHS] Re: A few comments on porting the Bourne shell
>
> >Chet Ramey on 30.12.2022 20:51:
> >>Arnold Robbins built on that work and ported the v8-v10 shells to modern
> >>Linux versions. (I am sorry, I do not have a link right now.)
> >
> >And btw, 8th ed (http://man.cat-v.org/unix_8th/1/sh) even added a simple
> history mechanism with the "=(1)" command (
> https://www.in-ulm.de/~mascheck/bourne/v8/=.html).
> >
> >Sven
> >
>

[-- Attachment #2: Type: text/html, Size: 2908 bytes --]

[TUHS] Re: A few comments on porting the Bourne shell

[Jon Steinhart]

Ron Natalie writes:
> I was just happy when VR2 came out and found someone had undone all 
> those macros that were in the original code.

On an amusing note, a while ago I had polled folks on a different
mailing list on items to include in a starter tool set for my daughter.

Steve:
  Hi Jon, highly recommend DeWalt 20V drill, here is my checklist for my boat
  tools.  Might be something there to add.  Some clearly not but didn't scrub.

Jon:
   Thanks.  I was thinking that I should also get her one of those adapters that
   connects a { to a BEGIN :-)

Steve:
  Looks like this

  :-}

[TUHS] Re: A few comments on porting the Bourne shell

[arnold]

Chet Ramey <chet.ramey@case.edu> wrote:

> On 12/30/22 1:25 PM, Paul Ruizendaal wrote:
> > 
> > London and Reiser report about porting the shell that “it required by
> > far the largest conversion effort of any supposedly portable program,
> > for the simple reason that it is not portable.” By the time of SysIII
> > this is greatly improved, but also in porting the SysIII user land it
> > was the most complex of the set so far.
>
> Have you read
>
> http://www.collyer.net/who/geoff/sh.tour.pdf
>
> and looked at http://www.collyer.net/who/geoff/v7sh.tar ?
>
> In the limited literature on Bourne Shell porting, this is authoritative.
>
> Arnold Robbins built on that work and ported the v8-v10 shells to modern
> Linux versions. (I am sorry, I do not have a link right now.)

Sorry to say, it wasn't me. Geoff Collyer made the v9 sh portable
and it's available from his web site.

I did do stuff with the shell back in the 80s:

- I back-ported Ron Natalie's job control into the 4.2 BSD sh,
  along with a history mechanism I wrote from scratch that was
  inspired by csh. I posted the diffs in comp.sources.unix.

- A few very minor features were added to ksh at my suggestion.
  ISTR that one was `set -A array-name value ...' but I don't see that
  in current ksh93 doc.

- I was a reviewer for both editions of David Korn's book on ksh.

- I contributed to pdksh, although I don't remember what. The pdksh
  doc lists my name though.

- In the early 90s I did a little banging on the vi mode in the
  readline librar so that bash's vi mode would be closer to what
  I'd been used to in ksh.

In the late 80s I got interested in awk and moved almost all my
personal scripting to awk, although I still do shell work at $DAYJOB
or for personal use as needed.

Arnold

[TUHS] Re: A few comments on porting the Bourne shell

[arnold]

I wrote:

> I did do stuff with the shell back in the 80s:

I also recall that there was some USENET discussion in which
Rob Pike described how the 8th edition shell changed the search
order to have functions come before non-special built-ins, with
the addition of "builtin" and "whatis" commands.

Rob was kind enough to send me the sh.1 man page from the 8th edition
(which I still have somewhere), and I made this change in the S5r2
shell. I don't know if I posted diffs or not.

In any case, as part of implementing whatis, I wrote a pretty printer
for shell functions. (Getting 'elif' right needed some extra code.) I
offered it to Rob, who politely declined, indicating that he had his own.

Ah, those were the days... :-)

Arnold

[TUHS] Re: A few comments on porting the Bourne shell

[Chet Ramey]

On 1/1/23 6:28 AM, arnold@skeeve.com wrote:

> I also recall that there was some USENET discussion in which
> Rob Pike described how the 8th edition shell changed the search
> order to have functions come before non-special built-ins, with
> the addition of "builtin" and "whatis" commands.

I have this somewhere. It's a rocking back-and-forth between Rob and
David Korn about shell philosophy. Well worth the read if you can find it.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
		 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/

[TUHS] Re: A few comments on porting the Bourne shell

[Chet Ramey]

On 1/1/23 5:44 AM, arnold@skeeve.com wrote:
> Chet Ramey <chet.ramey@case.edu> wrote:
> 
>> On 12/30/22 1:25 PM, Paul Ruizendaal wrote:
>>>
>>> London and Reiser report about porting the shell that “it required by
>>> far the largest conversion effort of any supposedly portable program,
>>> for the simple reason that it is not portable.” By the time of SysIII
>>> this is greatly improved, but also in porting the SysIII user land it
>>> was the most complex of the set so far.
>>
>> Have you read
>>
>> http://www.collyer.net/who/geoff/sh.tour.pdf
>>
>> and looked at http://www.collyer.net/who/geoff/v7sh.tar ?
>>
>> In the limited literature on Bourne Shell porting, this is authoritative.
>>
>> Arnold Robbins built on that work and ported the v8-v10 shells to modern
>> Linux versions. (I am sorry, I do not have a link right now.)
> 
> Sorry to say, it wasn't me. Geoff Collyer made the v9 sh portable
> and it's available from his web site.

My bad, credit to Geoff.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
		 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/

[TUHS] Re: A few comments on porting the Bourne shell

[segaloco via TUHS]

I'll have to double check later but I'm fairly certain the remaining L/R cheats are gone by SysV.  From what I can tell much of that portability work may have been done prior to the V7 release code base we're familiar with, as I did some comparison and found only one significant change between V7 and 32V code as I know it at least.  Either the claims of portability issues came between 32V and System III (meaning the shell was accepted as "broken"? in 32V) or the code we actually see in V7 has already been tidied up significantly and doesn't represent the "non-portable" version lamented in the famous quote.  Does this observation hold with reality?  Is there an earlier, more PDP-11 bound version of the Bourne Shell out there?  I seem to recall reading something about some bits of it even being in assembly at one point, but can't remember the quote source.

- Matt G.

------- Original Message -------
On Friday, December 30th, 2022 at 10:25 AM, Paul Ruizendaal <pnr@planet.nl> wrote:


> London and Reiser report about porting the shell that “it required by far the largest conversion effort of any supposedly portable program, for the simple reason that it is not portable.” By the time of SysIII this is greatly improved, but also in porting the SysIII user land it was the most complex of the set so far.
> 
> There were three aspects that I found noteworthy:
> 
> 1. London/Reiser apparently felt strongly about a property of casts. The code argues that casting an l-value should not convert it into a r-value:
> 
> <quote from "mode.h">
> 
> /* the following nonsense is required
> * because casts turn an Lvalue
> * into an Rvalue so two cheats
> * are necessary, one for each context.
> */
> union { int _cheat;};
> #define Lcheat(a) ((a)._cheat)
> #define Rcheat(a) ((int)(a))
> <endquote>
> 
> 
> However, Lcheat is only used in two places (in service.c), to set and to clear a flag in a pointer. Interestingly, the 32V code already replaces one of these instances with a regular r-value cast. So far, I’d never thought about this aspect of casts. I stumbled across it, because the Plan 9 compiler did not accept the Lcheat expansion as valid C.
> 
> 2. On the history of dup2
> 
> The shell code includes the following:
> 
> <quote from “io.c”>
> 
> rename(f1,f2)
> REG INT f1, f2;
> {
> #ifdef RES /* research has different sys calls from TS */
> IF f1!=f2
> THEN dup(f1|DUPFLG, f2);
> close(f1);
> IF f2==0 THEN ioset|=1 FI
> FI
> #else
> INT fs;
> IF f1!=f2
> THEN fs = fcntl(f2,1,0);
> close(f2);
> fcntl(f1,0,f2);
> close(f1);
> IF fs==1 THEN fcntl(f2,2,1) FI
> IF f2==0 THEN ioset|=1 FI
> FI
> #endif
> }
> <endquote>
> 
> 
> I’ve check the 8th edition source, and indeed it supports using DUPFLG to signal to dup() that it really is dup2(). I had earlier wondered why dup2() did not appear in research until 10th edition, but now that is clear. It would seem that the dup of 8th edition is a direct ancestor to dup() in Plan 9. I wonder why this way of doing things never caught on in the other Unices.
> 
> 3. Halfway to demand paging
> 
> I stumbled across this one because I had a bug in my signal handling. From early days onwards, Unix supported dynamically growing the stack allocation, which arguably is a first step towards building the mechanisms for demand paging. It appears that the Bourne shell made another step, catching page faults and expanding the data/bss allocation dynamically:
> 
> <quote from “fault.c”>
> 
> VOID fault(sig)
> REG INT sig;
> {
> signal(sig, fault);
> IF sig==MEMF
> THEN IF setbrk(brkincr) == -1
> THEN error(nospace);
> FI
> ELIF ...
> <endquote>
> 
> 
> This was already present in 7th edition, so it is by no means new in 32V or SysIII -- it had just escaped my attention as a conceptual step in the development of Unix memory handling.
>

[TUHS] Re: A few comments on porting the Bourne shell

[Paul Ruizendaal]

The "assembly code in the Bourne shell" comment is in the same London/Reiser paper. The full quote is:

"The (Bourne) shell is the standard user command interpreter. It required by far the largest conversion effort of any supposedly portable program, for the simple reason that it is not portable. Critical portions are coded in assembly language and had to be painstakingly rewritten. The shell uses its own sbrk which is functionally different from the standard routine in libc. The shell wants the routine which fields a signal to be passed a parameter giving the number of the signal being caught; signal was also a private rou- tine. This was handled by having the operating system provide the parameter in the first place, doing away with the private code for signal. The code in fixargs (for constructing the argument list to an exec system call) had to be diddled."

The files in the V7 tree on the Tuhs website are dated January 1979, so it would seem that the fixes for 32V were immediately taken back to Research. As you point out, this means that the comments above do not refer to the well known source code, but to a predecessor of that (which I don’t think survived).

Despite all the criticism voiced above, I think it is well understood that the original Bourne shell is an amazing piece of work that managed to fit an enormous amount of functionality into a cramped address space. Its longevity attests to that. That its internals became difficult to understand is par for the course -- the 1980’s in essence needed a Lions commentary on sh.


> On 30 Dec 2022, at 20:57, segaloco <segaloco@protonmail.com> wrote:
> 
> I'll have to double check later but I'm fairly certain the remaining L/R cheats are gone by SysV.  From what I can tell much of that portability work may have been done prior to the V7 release code base we're familiar with, as I did some comparison and found only one significant change between V7 and 32V code as I know it at least.  Either the claims of portability issues came between 32V and System III (meaning the shell was accepted as "broken"? in 32V) or the code we actually see in V7 has already been tidied up significantly and doesn't represent the "non-portable" version lamented in the famous quote.  Does this observation hold with reality?  Is there an earlier, more PDP-11 bound version of the Bourne Shell out there?  I seem to recall reading something about some bits of it even being in assembly at one point, but can't remember the quote source.
> 
> - Matt G.
> 
> ------- Original Message -------
> On Friday, December 30th, 2022 at 10:25 AM, Paul Ruizendaal <pnr@planet.nl> wrote:
> 
> 
>> London and Reiser report about porting the shell that “it required by far the largest conversion effort of any supposedly portable program, for the simple reason that it is not portable.” By the time of SysIII this is greatly improved, but also in porting the SysIII user land it was the most complex of the set so far.
>> 
>> There were three aspects that I found noteworthy:
>> 
>> 1. London/Reiser apparently felt strongly about a property of casts. The code argues that casting an l-value should not convert it into a r-value:
>> 
>> <quote from "mode.h">
>> 
>> /* the following nonsense is required
>> * because casts turn an Lvalue
>> * into an Rvalue so two cheats
>> * are necessary, one for each context.
>> */
>> union { int _cheat;};
>> #define Lcheat(a) ((a)._cheat)
>> #define Rcheat(a) ((int)(a))
>> <endquote>
>> 
>> 
>> However, Lcheat is only used in two places (in service.c), to set and to clear a flag in a pointer. Interestingly, the 32V code already replaces one of these instances with a regular r-value cast. So far, I’d never thought about this aspect of casts. I stumbled across it, because the Plan 9 compiler did not accept the Lcheat expansion as valid C.
>> 
>> 2. On the history of dup2
>> 
>> The shell code includes the following:
>> 
>> <quote from “io.c”>
>> 
>> rename(f1,f2)
>> REG INT f1, f2;
>> {
>> #ifdef RES /* research has different sys calls from TS */
>> IF f1!=f2
>> THEN dup(f1|DUPFLG, f2);
>> close(f1);
>> IF f2==0 THEN ioset|=1 FI
>> FI
>> #else
>> INT fs;
>> IF f1!=f2
>> THEN fs = fcntl(f2,1,0);
>> close(f2);
>> fcntl(f1,0,f2);
>> close(f1);
>> IF fs==1 THEN fcntl(f2,2,1) FI
>> IF f2==0 THEN ioset|=1 FI
>> FI
>> #endif
>> }
>> <endquote>
>> 
>> 
>> I’ve check the 8th edition source, and indeed it supports using DUPFLG to signal to dup() that it really is dup2(). I had earlier wondered why dup2() did not appear in research until 10th edition, but now that is clear. It would seem that the dup of 8th edition is a direct ancestor to dup() in Plan 9. I wonder why this way of doing things never caught on in the other Unices.
>> 
>> 3. Halfway to demand paging
>> 
>> I stumbled across this one because I had a bug in my signal handling. From early days onwards, Unix supported dynamically growing the stack allocation, which arguably is a first step towards building the mechanisms for demand paging. It appears that the Bourne shell made another step, catching page faults and expanding the data/bss allocation dynamically:
>> 
>> <quote from “fault.c”>
>> 
>> VOID fault(sig)
>> REG INT sig;
>> {
>> signal(sig, fault);
>> IF sig==MEMF
>> THEN IF setbrk(brkincr) == -1
>> THEN error(nospace);
>> FI
>> ELIF ...
>> <endquote>
>> 
>> 
>> This was already present in 7th edition, so it is by no means new in 32V or SysIII -- it had just escaped my attention as a conceptual step in the development of Unix memory handling.
>> 

[TUHS] Re: A few comments on porting the Bourne shell

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 8792 bytes --]

On Sat, Dec 31, 2022 at 5:55 AM Paul Ruizendaal <pnr@planet.nl> wrote:

> The "assembly code in the Bourne shell" comment is in the same
> London/Reiser paper. The full quote is:
>
> "The (Bourne) shell is the standard user command interpreter. It required
> by far the largest conversion effort of any supposedly portable program,
> for the simple reason that it is not portable. Critical portions are coded
> in assembly language and had to be painstakingly rewritten. The shell uses
> its own sbrk which is functionally different from the standard routine in
> libc. The shell wants the routine which fields a signal to be passed a
> parameter giving the number of the signal being caught; signal was also a
> private rou- tine. This was handled by having the operating system provide
> the parameter in the first place, doing away with the private code for
> signal. The code in fixargs (for constructing the argument list to an exec
> system call) had to be diddled."
>
> The files in the V7 tree on the Tuhs website are dated January 1979, so it
> would seem that the fixes for 32V were immediately taken back to Research.
> As you point out, this means that the comments above do not refer to the
> well known source code, but to a predecessor of that (which I don’t think
> survived).
>

We have ample evidence that V7 was really something more akin to a rolling
release. Let me explain: We know from the leaked '50 changes' tape that
many of the features were set earlier rather than later. This leaked in
1978 (if my notes are right), but I found references to it from as early as
November 1976 in
http://www.toad.com/early-usenix-newsletters/197611-unix-news-n11.pdf. This
was 18 months after V6 was released, but over 2 years before V7 was
released. In addition, we know from the AUUS newsletters in the archive
document that the V7 release process process took a while to get through
AT&T's legal department (IIRC a year, but I've not gone back to the AUUS
newsletters to refresh my recollection). A big push of V7 was to make it
portable as well (with AT&T doing an Interdata 8/32 port themselves, as
well as at least looking at the Wollongong Interdata 7/32 port and the
Harvard VM/370 port). In talking to Kirk and others that have been around
from approximately that time, 32V was widely viewed as V7 for Vaxen. We can
see evidence in the surviving 32V files of evolution from the 'PDP-11-like
swapping to a more sophisticated paging algorithm' since we have the
slowsys directory. It's my contention, as someone that coded in the era
before good source code control, that it's evidence that somebody got it
working, then renamed/copied it to slowsys while they got paging working so
they could build either kernel for A/B testing. Kirk has also told me that
the 32V port was started well in advance of V7's release to be both a
useful product inside of Bell Labs (since Vaxen were starting to appear) as
well as to prove that V7 was portable enough. I'll be the first to admit
this is at best conjecture that matches available facts, artifacts and old
timers recollections (sorry Kirk), but that we have no direct evidence for.
It also allowed the 3BSD efforts to get going before the official V7
release due to the close ties between Bell Labs and Berkeley and the DARPA
project around Unix.

I believe that we can conclude that the original 'hard to port' Bourne
shell was produced around the time of the 50 changes tape, give or take.
And that all the unix porting efforts that pre-dated the V7 release rolled
what appeared in 32V into V7 to reduce the amount of pdp-11 assembler. And
those efforts are what we read about in the paper.

It also goes a ways to explain the 32V meme of 'it was pdp11 swapping'
because originally, in a version we no longer have, it was. But many of the
32V tapes that we have represent a later version where that had been
abandoned in favor of what would evolve into System III's and later paging
code.


> Despite all the criticism voiced above, I think it is well understood that
> the original Bourne shell is an amazing piece of work that managed to fit
> an enormous amount of functionality into a cramped address space. Its
> longevity attests to that. That its internals became difficult to
> understand is par for the course -- the 1980’s in essence needed a Lions
> commentary on sh.
>

Totally agree with that...

Warner


>
> > On 30 Dec 2022, at 20:57, segaloco <segaloco@protonmail.com> wrote:
> >
> > I'll have to double check later but I'm fairly certain the remaining L/R
> cheats are gone by SysV.  From what I can tell much of that portability
> work may have been done prior to the V7 release code base we're familiar
> with, as I did some comparison and found only one significant change
> between V7 and 32V code as I know it at least.  Either the claims of
> portability issues came between 32V and System III (meaning the shell was
> accepted as "broken"? in 32V) or the code we actually see in V7 has already
> been tidied up significantly and doesn't represent the "non-portable"
> version lamented in the famous quote.  Does this observation hold with
> reality?  Is there an earlier, more PDP-11 bound version of the Bourne
> Shell out there?  I seem to recall reading something about some bits of it
> even being in assembly at one point, but can't remember the quote source.
> >
> > - Matt G.
> >
> > ------- Original Message -------
> > On Friday, December 30th, 2022 at 10:25 AM, Paul Ruizendaal <
> pnr@planet.nl> wrote:
> >
> >
> >> London and Reiser report about porting the shell that “it required by
> far the largest conversion effort of any supposedly portable program, for
> the simple reason that it is not portable.” By the time of SysIII this is
> greatly improved, but also in porting the SysIII user land it was the most
> complex of the set so far.
> >>
> >> There were three aspects that I found noteworthy:
> >>
> >> 1. London/Reiser apparently felt strongly about a property of casts.
> The code argues that casting an l-value should not convert it into a
> r-value:
> >>
> >> <quote from "mode.h">
> >>
> >> /* the following nonsense is required
> >> * because casts turn an Lvalue
> >> * into an Rvalue so two cheats
> >> * are necessary, one for each context.
> >> */
> >> union { int _cheat;};
> >> #define Lcheat(a) ((a)._cheat)
> >> #define Rcheat(a) ((int)(a))
> >> <endquote>
> >>
> >>
> >> However, Lcheat is only used in two places (in service.c), to set and
> to clear a flag in a pointer. Interestingly, the 32V code already replaces
> one of these instances with a regular r-value cast. So far, I’d never
> thought about this aspect of casts. I stumbled across it, because the Plan
> 9 compiler did not accept the Lcheat expansion as valid C.
> >>
> >> 2. On the history of dup2
> >>
> >> The shell code includes the following:
> >>
> >> <quote from “io.c”>
> >>
> >> rename(f1,f2)
> >> REG INT f1, f2;
> >> {
> >> #ifdef RES /* research has different sys calls from TS */
> >> IF f1!=f2
> >> THEN dup(f1|DUPFLG, f2);
> >> close(f1);
> >> IF f2==0 THEN ioset|=1 FI
> >> FI
> >> #else
> >> INT fs;
> >> IF f1!=f2
> >> THEN fs = fcntl(f2,1,0);
> >> close(f2);
> >> fcntl(f1,0,f2);
> >> close(f1);
> >> IF fs==1 THEN fcntl(f2,2,1) FI
> >> IF f2==0 THEN ioset|=1 FI
> >> FI
> >> #endif
> >> }
> >> <endquote>
> >>
> >>
> >> I’ve check the 8th edition source, and indeed it supports using DUPFLG
> to signal to dup() that it really is dup2(). I had earlier wondered why
> dup2() did not appear in research until 10th edition, but now that is
> clear. It would seem that the dup of 8th edition is a direct ancestor to
> dup() in Plan 9. I wonder why this way of doing things never caught on in
> the other Unices.
> >>
> >> 3. Halfway to demand paging
> >>
> >> I stumbled across this one because I had a bug in my signal handling.
> From early days onwards, Unix supported dynamically growing the stack
> allocation, which arguably is a first step towards building the mechanisms
> for demand paging. It appears that the Bourne shell made another step,
> catching page faults and expanding the data/bss allocation dynamically:
> >>
> >> <quote from “fault.c”>
> >>
> >> VOID fault(sig)
> >> REG INT sig;
> >> {
> >> signal(sig, fault);
> >> IF sig==MEMF
> >> THEN IF setbrk(brkincr) == -1
> >> THEN error(nospace);
> >> FI
> >> ELIF ...
> >> <endquote>
> >>
> >>
> >> This was already present in 7th edition, so it is by no means new in
> 32V or SysIII -- it had just escaped my attention as a conceptual step in
> the development of Unix memory handling.
> >>
>
>

[-- Attachment #2: Type: text/html, Size: 10349 bytes --]

[TUHS] Re: A few comments on porting the Bourne shell

[Jonathan Gray]

On Sat, Dec 31, 2022 at 07:55:28PM -0700, Warner Losh wrote:
> On Sat, Dec 31, 2022 at 5:55 AM Paul Ruizendaal <pnr@planet.nl> wrote:
> 
> > The "assembly code in the Bourne shell" comment is in the same
> > London/Reiser paper. The full quote is:
> >
> > "The (Bourne) shell is the standard user command interpreter. It required
> > by far the largest conversion effort of any supposedly portable program,
> > for the simple reason that it is not portable. Critical portions are coded
> > in assembly language and had to be painstakingly rewritten. The shell uses
> > its own sbrk which is functionally different from the standard routine in
> > libc. The shell wants the routine which fields a signal to be passed a
> > parameter giving the number of the signal being caught; signal was also a
> > private rou- tine. This was handled by having the operating system provide
> > the parameter in the first place, doing away with the private code for
> > signal. The code in fixargs (for constructing the argument list to an exec
> > system call) had to be diddled."
> >
> > The files in the V7 tree on the Tuhs website are dated January 1979, so it
> > would seem that the fixes for 32V were immediately taken back to Research.
> > As you point out, this means that the comments above do not refer to the
> > well known source code, but to a predecessor of that (which I don’t think
> > survived).
> >
> 
> We have ample evidence that V7 was really something more akin to a rolling
> release. Let me explain: We know from the leaked '50 changes' tape that
> many of the features were set earlier rather than later. This leaked in
> 1978 (if my notes are right), but I found references to it from as early as
> November 1976 in
> http://www.toad.com/early-usenix-newsletters/197611-unix-news-n11.pdf. This
> was 18 months after V6 was released, but over 2 years before V7 was
> released. In addition, we know from the AUUS newsletters in the archive
> document that the V7 release process process took a while to get through
> AT&T's legal department (IIRC a year, but I've not gone back to the AUUS
> newsletters to refresh my recollection). A big push of V7 was to make it
> portable as well (with AT&T doing an Interdata 8/32 port themselves, as
> well as at least looking at the Wollongong Interdata 7/32 port and the
> Harvard VM/370 port). In talking to Kirk and others that have been around
> from approximately that time, 32V was widely viewed as V7 for Vaxen. We can
> see evidence in the surviving 32V files of evolution from the 'PDP-11-like
> swapping to a more sophisticated paging algorithm' since we have the
> slowsys directory. It's my contention, as someone that coded in the era
> before good source code control, that it's evidence that somebody got it
> working, then renamed/copied it to slowsys while they got paging working so
> they could build either kernel for A/B testing. Kirk has also told me that
> the 32V port was started well in advance of V7's release to be both a
> useful product inside of Bell Labs (since Vaxen were starting to appear) as
> well as to prove that V7 was portable enough. I'll be the first to admit
> this is at best conjecture that matches available facts, artifacts and old
> timers recollections (sorry Kirk), but that we have no direct evidence for.
> It also allowed the 3BSD efforts to get going before the official V7
> release due to the close ties between Bell Labs and Berkeley and the DARPA
> project around Unix.

"So DEC came to us at Holmdel.  We were clearly the second string.
Tom London and John Reiser were interested and so was Ken Swanson, and
we got the VAX in early '78.  I didn't do any of the technical work.  In
fact, I devoted a lot of energy and time to getting the management to
let us do it.  It wasn't research, you see.  However, they let us take
the time.  And in about three months my small group ported Version 7 to
the VAX.  We got the machine in January, they had it running in April,
and by August it really worked.
...
So, with the blessings of BTL Area 11 management, we sent 32V to
Berkeley.  It was in October or November, 1978"

Charlie Roberts in Salus QCU

"The operating system is Research version 7 as of April 15, 1978.
..
Work on the C compiler began in mid-December 1977.  The hardware arrived
on March 3.  We held a party on May 19 to celebrate successful multiuser
operation of the system."

London and Reiser
A UNIX Operating System for the DEC VAX-11/780 Computer
https://www.bell-labs.com/usr/dmr/www/otherports/32v.html

> 
> I believe that we can conclude that the original 'hard to port' Bourne
> shell was produced around the time of the 50 changes tape, give or take.
> And that all the unix porting efforts that pre-dated the V7 release rolled
> what appeared in 32V into V7 to reduce the amount of pdp-11 assembler. And
> those efforts are what we read about in the paper.

"The Bourne shell work started either in early 1976, or maybe
late 1975. The first version was VERY different"

John Mashey in net.unix-wizards, 18 Mar 86
https://groups.google.com/g/alt.folklore.computers/c/xW3ZgEnFoFs
tuhs/Documentation/AUUGN/AUUGN-V06.6.pdf pg 39

"In 1976, Steve Bourne, who had recently joined 1127, wrote a new shell"
Kernighan, UNIX A History and a Memoir, 5.1 pg 88

Bourne's AsiaBSDCon 2016 talk also lists 1976
and goes on to discuss sbrk() use causing problems with ports
https://youtu.be/7tQ2ftt3LO8?t=715

[TUHS] Re: A few comments on porting the Bourne shell

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 6434 bytes --]

On Sat, Dec 31, 2022, 9:38 PM Jonathan Gray <jsg@jsg.id.au> wrote:

> On Sat, Dec 31, 2022 at 07:55:28PM -0700, Warner Losh wrote:
> > On Sat, Dec 31, 2022 at 5:55 AM Paul Ruizendaal <pnr@planet.nl> wrote:
> >
> > > The "assembly code in the Bourne shell" comment is in the same
> > > London/Reiser paper. The full quote is:
> > >
> > > "The (Bourne) shell is the standard user command interpreter. It
> required
> > > by far the largest conversion effort of any supposedly portable
> program,
> > > for the simple reason that it is not portable. Critical portions are
> coded
> > > in assembly language and had to be painstakingly rewritten. The shell
> uses
> > > its own sbrk which is functionally different from the standard routine
> in
> > > libc. The shell wants the routine which fields a signal to be passed a
> > > parameter giving the number of the signal being caught; signal was
> also a
> > > private rou- tine. This was handled by having the operating system
> provide
> > > the parameter in the first place, doing away with the private code for
> > > signal. The code in fixargs (for constructing the argument list to an
> exec
> > > system call) had to be diddled."
> > >
> > > The files in the V7 tree on the Tuhs website are dated January 1979,
> so it
> > > would seem that the fixes for 32V were immediately taken back to
> Research.
> > > As you point out, this means that the comments above do not refer to
> the
> > > well known source code, but to a predecessor of that (which I don’t
> think
> > > survived).
> > >
> >
> > We have ample evidence that V7 was really something more akin to a
> rolling
> > release. Let me explain: We know from the leaked '50 changes' tape that
> > many of the features were set earlier rather than later. This leaked in
> > 1978 (if my notes are right), but I found references to it from as early
> as
> > November 1976 in
> > http://www.toad.com/early-usenix-newsletters/197611-unix-news-n11.pdf.
> This
> > was 18 months after V6 was released, but over 2 years before V7 was
> > released. In addition, we know from the AUUS newsletters in the archive
> > document that the V7 release process process took a while to get through
> > AT&T's legal department (IIRC a year, but I've not gone back to the AUUS
> > newsletters to refresh my recollection). A big push of V7 was to make it
> > portable as well (with AT&T doing an Interdata 8/32 port themselves, as
> > well as at least looking at the Wollongong Interdata 7/32 port and the
> > Harvard VM/370 port). In talking to Kirk and others that have been around
> > from approximately that time, 32V was widely viewed as V7 for Vaxen. We
> can
> > see evidence in the surviving 32V files of evolution from the
> 'PDP-11-like
> > swapping to a more sophisticated paging algorithm' since we have the
> > slowsys directory. It's my contention, as someone that coded in the era
> > before good source code control, that it's evidence that somebody got it
> > working, then renamed/copied it to slowsys while they got paging working
> so
> > they could build either kernel for A/B testing. Kirk has also told me
> that
> > the 32V port was started well in advance of V7's release to be both a
> > useful product inside of Bell Labs (since Vaxen were starting to appear)
> as
> > well as to prove that V7 was portable enough. I'll be the first to admit
> > this is at best conjecture that matches available facts, artifacts and
> old
> > timers recollections (sorry Kirk), but that we have no direct evidence
> for.
> > It also allowed the 3BSD efforts to get going before the official V7
> > release due to the close ties between Bell Labs and Berkeley and the
> DARPA
> > project around Unix.
>
> "So DEC came to us at Holmdel.  We were clearly the second string.
> Tom London and John Reiser were interested and so was Ken Swanson, and
> we got the VAX in early '78.  I didn't do any of the technical work.  In
> fact, I devoted a lot of energy and time to getting the management to
> let us do it.  It wasn't research, you see.  However, they let us take
> the time.  And in about three months my small group ported Version 7 to
> the VAX.  We got the machine in January, they had it running in April,
> and by August it really worked.
> ...
> So, with the blessings of BTL Area 11 management, we sent 32V to
> Berkeley.  It was in October or November, 1978"
>
> Charlie Roberts in Salus QCU
>
> "The operating system is Research version 7 as of April 15, 1978.
> ..
> Work on the C compiler began in mid-December 1977.  The hardware arrived
> on March 3.  We held a party on May 19 to celebrate successful multiuser
> operation of the system."
>
> London and Reiser
> A UNIX Operating System for the DEC VAX-11/780 Computer
> https://www.bell-labs.com/usr/dmr/www/otherports/32v.html
>
> >
> > I believe that we can conclude that the original 'hard to port' Bourne
> > shell was produced around the time of the 50 changes tape, give or take.
> > And that all the unix porting efforts that pre-dated the V7 release
> rolled
> > what appeared in 32V into V7 to reduce the amount of pdp-11 assembler.
> And
> > those efforts are what we read about in the paper.
>
> "The Bourne shell work started either in early 1976, or maybe
> late 1975. The first version was VERY different"
>
> John Mashey in net.unix-wizards, 18 Mar 86
> https://groups.google.com/g/alt.folklore.computers/c/xW3ZgEnFoFs
> tuhs/Documentation/AUUGN/AUUGN-V06.6.pdf pg 39
>
> "In 1976, Steve Bourne, who had recently joined 1127, wrote a new shell"
> Kernighan, UNIX A History and a Memoir, 5.1 pg 88
>
> Bourne's AsiaBSDCon 2016 talk also lists 1976
> and goes on to discuss sbrk() use causing problems with ports
> https://youtu.be/7tQ2ftt3LO8?t=715


And at 5:18 he says he had a vax lab with three vaxen and the Lab's vax
port didn't have virtual memory. Bill Joy with 3BSD which had virtual
memory. They installed it on the vaxen because they were hitting physical
memory limits for some of their programs....

This suggests that the 32V in the TUHS archive is a later one than this
early port. 3BSD is listed as being released end of 1979...

Still begs the question of when 32v started paging.  :)

But the time lines match up for the 32v effort to be fed back into the
research code base...

Warner

>
>

[-- Attachment #2: Type: text/html, Size: 8139 bytes --]

[TUHS] Re: A few comments on porting the Bourne shell

[Dan Cross]

On Sun, Jan 1, 2023 at 12:27 AM Warner Losh <imp@bsdimp.com> wrote:
> On Sat, Dec 31, 2022, 9:38 PM Jonathan Gray <jsg@jsg.id.au> wrote:
>> [snip]
>> Bourne's AsiaBSDCon 2016 talk also lists 1976
>> and goes on to discuss sbrk() use causing problems with ports
>> https://youtu.be/7tQ2ftt3LO8?t=715
>
> And at 5:18 he says he had a vax lab with three vaxen and the Lab's vax port didn't have virtual memory. Bill Joy with 3BSD which had virtual memory. They installed it on the vaxen because they were hitting physical memory limits for some of their programs....

One wonders what is meant by "virtual memory" in this context. I
contend that Unix has had "virtual memory" since moving off of the
PDP-11/20, in the sense of having a virtual address space that was
mapped onto a (possibly contiguous) physical address space. I think
all of these references mean demand paging, possibly with page
reclamation or whole-process swapping under memory pressure.

        - Dan C.

[TUHS] Re: A few comments on porting the Bourne shell

[G. Branden Robinson]

[-- Attachment #1: Type: text/plain, Size: 2000 bytes --]

At 2023-01-01T00:35:12-0500, Dan Cross wrote:
> On Sun, Jan 1, 2023 at 12:27 AM Warner Losh <imp@bsdimp.com> wrote:
> > On Sat, Dec 31, 2022, 9:38 PM Jonathan Gray <jsg@jsg.id.au> wrote:
> >> [snip]
> >> Bourne's AsiaBSDCon 2016 talk also lists 1976
> >> and goes on to discuss sbrk() use causing problems with ports
> >> https://youtu.be/7tQ2ftt3LO8?t=715
> >
> > And at 5:18 he says he had a vax lab with three vaxen and the Lab's
> > vax port didn't have virtual memory. Bill Joy with 3BSD which had
> > virtual memory. They installed it on the vaxen because they were
> > hitting physical memory limits for some of their programs....
> 
> One wonders what is meant by "virtual memory" in this context. I
> contend that Unix has had "virtual memory" since moving off of the
> PDP-11/20, in the sense of having a virtual address space that was
> mapped onto a (possibly contiguous) physical address space. I think
> all of these references mean demand paging, possibly with page
> reclamation or whole-process swapping under memory pressure.

I apologize if this point is too elementary, but I speculate that one
possible source of confusion comes from a file naming convention: which
of these (multiple) virtual memory or demand-paged VM systems installed
the kernel under the name "vmunix" vs. "unix".

Which ones did and did not?

When I was first learning Unix I asked a local expert why the kernel was
named "vmunix".  They told me that it was because it supported virtual
memory (and explained what that was, because I was even more callow then
than now).

Then I asked where the non-VM kernel was.  I was informed that there
wasn't one--it didn't even exist for modern architectures.  I wondered
then why, if virtual memory was a given, you wouldn't just go back to
using the filename "unix".

I wondered similar things when encountering the "vmlinux" file a couple
of years later.

Reflexive obeisance to traditions has a cost.

Regards,
Branden

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[TUHS] Re: A few comments on porting the Bourne shell

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 2456 bytes --]

On Sat, Dec 31, 2022, 10:53 PM G. Branden Robinson <
g.branden.robinson@gmail.com> wrote:

> At 2023-01-01T00:35:12-0500, Dan Cross wrote:
> > On Sun, Jan 1, 2023 at 12:27 AM Warner Losh <imp@bsdimp.com> wrote:
> > > On Sat, Dec 31, 2022, 9:38 PM Jonathan Gray <jsg@jsg.id.au> wrote:
> > >> [snip]
> > >> Bourne's AsiaBSDCon 2016 talk also lists 1976
> > >> and goes on to discuss sbrk() use causing problems with ports
> > >> https://youtu.be/7tQ2ftt3LO8?t=715
> > >
> > > And at 5:18 he says he had a vax lab with three vaxen and the Lab's
> > > vax port didn't have virtual memory. Bill Joy with 3BSD which had
> > > virtual memory. They installed it on the vaxen because they were
> > > hitting physical memory limits for some of their programs....
> >
> > One wonders what is meant by "virtual memory" in this context. I
> > contend that Unix has had "virtual memory" since moving off of the
> > PDP-11/20, in the sense of having a virtual address space that was
> > mapped onto a (possibly contiguous) physical address space. I think
> > all of these references mean demand paging, possibly with page
> > reclamation or whole-process swapping under memory pressure.
>
> I apologize if this point is too elementary, but I speculate that one
> possible source of confusion comes from a file naming convention: which
> of these (multiple) virtual memory or demand-paged VM systems installed
> the kernel under the name "vmunix" vs. "unix".
>
> Which ones did and did not?
>

3BSD started the vmunix name because it added virtual memory to unix
(paging aspect of virtual memory).

When I was first learning Unix I asked a local expert why the kernel was
> named "vmunix".  They told me that it was because it supported virtual
> memory (and explained what that was, because I was even more callow then
> than now).
>
> Then I asked where the non-VM kernel was.  I was informed that there
> wasn't one--it didn't even exist for modern architectures.  I wondered
> then why, if virtual memory was a given, you wouldn't just go back to
> using the filename "unix".
>

V7 had "unix" for sure. 32V still had unix if my quick grep is right.

I wondered similar things when encountering the "vmlinux" file a couple
> of years later.
>

It was emulation of BSD and/or SunOS...

Reflexive obeisance to traditions has a cost.
>

Yea... There was a thread in FreeBSD about what to name the kernel.  Vmduck
was suggested as a joke...

Warner

> Regards,
> Branden
>

[-- Attachment #2: Type: text/html, Size: 4284 bytes --]

[TUHS] Re: A few comments on porting the Bourne shell

[Rob Pike]

[-- Attachment #1: Type: text/plain, Size: 2336 bytes --]

I distinctly remember I renamed to boot image on our research VAX (alice!)
from "vmunix" to "unix". It had been long enough by that point that the
lingering prefix was starting to sound like a brag.

-rob


On Sun, Jan 1, 2023 at 4:53 PM G. Branden Robinson <
g.branden.robinson@gmail.com> wrote:

> At 2023-01-01T00:35:12-0500, Dan Cross wrote:
> > On Sun, Jan 1, 2023 at 12:27 AM Warner Losh <imp@bsdimp.com> wrote:
> > > On Sat, Dec 31, 2022, 9:38 PM Jonathan Gray <jsg@jsg.id.au> wrote:
> > >> [snip]
> > >> Bourne's AsiaBSDCon 2016 talk also lists 1976
> > >> and goes on to discuss sbrk() use causing problems with ports
> > >> https://youtu.be/7tQ2ftt3LO8?t=715
> > >
> > > And at 5:18 he says he had a vax lab with three vaxen and the Lab's
> > > vax port didn't have virtual memory. Bill Joy with 3BSD which had
> > > virtual memory. They installed it on the vaxen because they were
> > > hitting physical memory limits for some of their programs....
> >
> > One wonders what is meant by "virtual memory" in this context. I
> > contend that Unix has had "virtual memory" since moving off of the
> > PDP-11/20, in the sense of having a virtual address space that was
> > mapped onto a (possibly contiguous) physical address space. I think
> > all of these references mean demand paging, possibly with page
> > reclamation or whole-process swapping under memory pressure.
>
> I apologize if this point is too elementary, but I speculate that one
> possible source of confusion comes from a file naming convention: which
> of these (multiple) virtual memory or demand-paged VM systems installed
> the kernel under the name "vmunix" vs. "unix".
>
> Which ones did and did not?
>
> When I was first learning Unix I asked a local expert why the kernel was
> named "vmunix".  They told me that it was because it supported virtual
> memory (and explained what that was, because I was even more callow then
> than now).
>
> Then I asked where the non-VM kernel was.  I was informed that there
> wasn't one--it didn't even exist for modern architectures.  I wondered
> then why, if virtual memory was a given, you wouldn't just go back to
> using the filename "unix".
>
> I wondered similar things when encountering the "vmlinux" file a couple
> of years later.
>
> Reflexive obeisance to traditions has a cost.
>
> Regards,
> Branden
>

[-- Attachment #2: Type: text/html, Size: 3456 bytes --]

[TUHS] Re: A few comments on porting the Bourne shell

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 1178 bytes --]

On Sat, Dec 31, 2022, 10:35 PM Dan Cross <crossd@gmail.com> wrote:

> On Sun, Jan 1, 2023 at 12:27 AM Warner Losh <imp@bsdimp.com> wrote:
> > On Sat, Dec 31, 2022, 9:38 PM Jonathan Gray <jsg@jsg.id.au> wrote:
> >> [snip]
> >> Bourne's AsiaBSDCon 2016 talk also lists 1976
> >> and goes on to discuss sbrk() use causing problems with ports
> >> https://youtu.be/7tQ2ftt3LO8?t=715
> >
> > And at 5:18 he says he had a vax lab with three vaxen and the Lab's vax
> port didn't have virtual memory. Bill Joy with 3BSD which had virtual
> memory. They installed it on the vaxen because they were hitting physical
> memory limits for some of their programs....
>
> One wonders what is meant by "virtual memory" in this context. I
> contend that Unix has had "virtual memory" since moving off of the
> PDP-11/20, in the sense of having a virtual address space that was
> mapped onto a (possibly contiguous) physical address space. I think
> all of these references mean demand paging, possibly with page
> reclamation or whole-process swapping under memory pressure.
>

It's clear from context it's demand paging and that unlocks processes
larger than physical memory. Imho.

Warner

>

[-- Attachment #2: Type: text/html, Size: 2063 bytes --]

[TUHS] Re: A few comments on porting the Bourne shell

[Ron Natalie]

Paging was what they were referring to.   The Bell releases at the time 
only swapped like they did on the PDP-11.
BSD allowed only fragments of the program to need be resident.

The joke as “It ain’t virtual unless it isn’t all there."


------ Original Message ------
From "Dan Cross" <crossd@gmail.com>
To "Warner Losh" <imp@bsdimp.com>
Cc "Jonathan Gray" <jsg@jsg.id.au>; "Paul Ruizendaal" <pnr@planet.nl>; 
"The Eunuchs Hysterical Society" <tuhs@tuhs.org>; "segaloco" 
<segaloco@protonmail.com>
Date 1/1/2023 12:35:12 AM
Subject [TUHS] Re: A few comments on porting the Bourne shell

>On Sun, Jan 1, 2023 at 12:27 AM Warner Losh <imp@bsdimp.com> wrote:
>>  On Sat, Dec 31, 2022, 9:38 PM Jonathan Gray <jsg@jsg.id.au> wrote:
>>>  [snip]
>>>  Bourne's AsiaBSDCon 2016 talk also lists 1976
>>>  and goes on to discuss sbrk() use causing problems with ports
>>>https://youtu.be/7tQ2ftt3LO8?t=715
>>
>>  And at 5:18 he says he had a vax lab with three vaxen and the Lab's vax port didn't have virtual memory. Bill Joy with 3BSD which had virtual memory. They installed it on the vaxen because they were hitting physical memory limits for some of their programs....
>
>One wonders what is meant by "virtual memory" in this context. I
>contend that Unix has had "virtual memory" since moving off of the
>PDP-11/20, in the sense of having a virtual address space that was
>mapped onto a (possibly contiguous) physical address space. I think
>all of these references mean demand paging, possibly with page
>reclamation or whole-process swapping under memory pressure.
>
>         - Dan C.

[TUHS] Re: A few comments on porting the Bourne shell

[Jonathan Gray]

On Sat, Dec 31, 2022 at 10:25:49PM -0700, Warner Losh wrote:
> And at 5:18 he says he had a vax lab with three vaxen and the Lab's vax
> port didn't have virtual memory. Bill Joy with 3BSD which had virtual
> memory. They installed it on the vaxen because they were hitting physical
> memory limits for some of their programs....
> 
> This suggests that the 32V in the TUHS archive is a later one than this
> early port. 3BSD is listed as being released end of 1979...

It was 4BSD and 1980 according to QCU:

"In 1980 it was Bill Joy who flew to New Jersey and, together with Steve
Bourne, put 4BSD up in Bell Labs. "We did it overnight," Joy told me.
"We decided to do it about 9 p.m.; people showed up in the morning and a
new os was running. It was pretty amazing."

[TUHS] Re: A few comments on porting the Bourne shell

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 924 bytes --]

On Sun, Jan 1, 2023, 12:11 AM Jonathan Gray <jsg@jsg.id.au> wrote:

> On Sat, Dec 31, 2022 at 10:25:49PM -0700, Warner Losh wrote:
> > And at 5:18 he says he had a vax lab with three vaxen and the Lab's vax
> > port didn't have virtual memory. Bill Joy with 3BSD which had virtual
> > memory. They installed it on the vaxen because they were hitting physical
> > memory limits for some of their programs....
> >
> > This suggests that the 32V in the TUHS archive is a later one than this
> > early port. 3BSD is listed as being released end of 1979...
>
> It was 4BSD and 1980 according to QCU:
>
> "In 1980 it was Bill Joy who flew to New Jersey and, together with Steve
> Bourne, put 4BSD up in Bell Labs. "We did it overnight," Joy told me.
> "We decided to do it about 9 p.m.; people showed up in the morning and a
> new os was running. It was pretty amazing."
>

He said 3BSD in the video.... Interesting...

Warner

>

[-- Attachment #2: Type: text/html, Size: 1553 bytes --]

[TUHS] Re: A few comments on porting the Bourne shell

[Paul Ruizendaal]

> On 1 Jan 2023, at 06:25, Warner Losh <imp@bsdimp.com> wrote:
> 
> Still begs the question of when 32v started paging.  :)

The London/Reiser paper includes:

"Work on the C compiler began in mid-December 1977. The hardware arrived on March 3. We held a party on May 19 to celebrate successful multiuser operation of the system."

John Reiser had the following timeline in a mail conversation about a year ago.

"Our VAX-11/780, the first one delivered to a customer, arrived Feb.12, 1978
(Lincoln's Birthday).  Tom and I had been preparing using PDP-11/45 since
December, and we achieved "login: " on the console DECwriter by April 15
(the deadline for US income tax filing).  The rest of 1978 was "tuning",
and preparing for the release of "UNIX-32/V" to UC Berkeley.”

The dates don’t match to the day, but the timeline is essentially the same (moreover, delivery of VAX hardware is not the same as it being operational, achieving login: is but the first step of bringing the user land up).

This sets a minimum timeline for arriving at a first version of “slowsys”. The timestamps on the surviving files of slowsys show January 1979.

Joy and Babaoglu write in “Converting a Swap-Based System to do Paging in an Architecture Lacking Page-Referenced Bits”:

"In the fall of 1978 the Computer Science Division of the University of California at Berkeley purchased a VAX-11/780 and arranged to run an early version of UNIX for the VAX provided by Bell Laboratories under a cooperative research agreement. [...] A subsequent version of the system was capable of loading processes into noncontiguous real memory locations, called scatter loading , and was able to swap only portions of a process, called partial swapping, as deemed necessary by the memory contention. This would become the basis for the paging system development discussed in this paper.”

This places the earliest moment for scatter paging as the Fall of 1978. The timestamps of the surviving files of sys show March 1979, which is the latest moment.

The for the timeline of 32V with true demand paging John Reiser recalled:

"My annual performance review in early 1979 said "Well done; but don't ever do
it again” because it was not regarded as "Research”. So what did I do?  I did
it again; this time, with demand paging. I designed and implemented mmap() based
on experience with PDP-10/Tenex PMAP page mapping system call. I fretted over
introducing the first UNIX system call with 6 arguments.”

John recalls that it was a 6 or 7 months effort, with the last few being a "long tail" of decreasing attention. The would place this third version as completed around the Fall of 1979. This is consistent with Rob Pike’s recollection that he saw it being demonstrated early in 1980.

In 1980 Research had to chose how to progress in the field of memory management. The Reiser system was cool, but not supported by management and the BSD VM system was seen as good enough to use. 3BSD is from early 1980, 4BSD from late 1980. Likely the conflicting recollections of Bill Joy are because it was a snapshot somewhere in between (plus the general issues around recollection). In any case, the 1985 snapshot of 8th edition appears to have been based on 4.1BSD (early 1981), so that is when the systems split.