The Unix Heritage Society mailing list
 help / color / mirror / Atom feed
* [TUHS] A few comments on porting the Bourne shell
@ 2022-12-30 18:25 Paul Ruizendaal
  2022-12-30 19:51 ` [TUHS] " Chet Ramey
  2022-12-30 19:57 ` segaloco via TUHS
  0 siblings, 2 replies; 121+ messages in thread
From: Paul Ruizendaal @ 2022-12-30 18:25 UTC (permalink / raw)
  To: The Eunuchs Hysterical Society


London and Reiser report about porting the shell that “it required by far the largest conversion effort of any supposedly portable program, for the simple reason that it is not portable.” By the time of SysIII this is greatly improved, but also in porting the SysIII user land it was the most complex of the set so far.

There were three aspects that I found noteworthy:

1. London/Reiser apparently felt strongly about a property of casts. The code argues that casting an l-value should not convert it into a r-value:

<quote from "mode.h">
/* the following nonsense is required
 * because casts turn an Lvalue
 * into an Rvalue so two cheats
 * are necessary, one for each context.
 */
union { int _cheat;};
#define Lcheat(a)	((a)._cheat)
#define Rcheat(a)	((int)(a))
<endquote>

However, Lcheat is only used in two places (in service.c), to set and to clear a flag in a pointer. Interestingly, the 32V code already replaces one of these instances with a regular r-value cast. So far, I’d never thought about this aspect of casts. I stumbled across it, because the Plan 9 compiler did not accept the Lcheat expansion as valid C.

2. On the history of dup2

The shell code includes the following:

<quote from “io.c”>
rename(f1,f2)
	REG INT		f1, f2;
{
#ifdef RES	/*	research has different sys calls from TS	*/
	IF f1!=f2
	THEN	dup(f1|DUPFLG, f2);
		close(f1);
		IF f2==0 THEN ioset|=1 FI
	FI
#else
	INT	fs;
	IF	f1!=f2
	THEN 	fs = fcntl(f2,1,0);
		close(f2);
		fcntl(f1,0,f2);
		close(f1);
		IF fs==1 THEN fcntl(f2,2,1) FI
		IF f2==0 THEN ioset|=1 FI
	FI
#endif
}
<endquote>

I’ve check the 8th edition source, and indeed it supports using DUPFLG to signal to dup() that it really is dup2(). I had earlier wondered why dup2() did not appear in research until 10th edition, but now that is clear. It would seem that the dup of 8th edition is a direct ancestor to dup() in Plan 9. I wonder why this way of doing things never caught on in the other Unices.

3. Halfway to demand paging

I stumbled across this one because I had a bug in my signal handling. From early days onwards, Unix supported dynamically growing the stack allocation, which arguably is a first step towards building the mechanisms for demand paging. It appears that the Bourne shell made another step, catching page faults and expanding the data/bss allocation dynamically:

<quote from “fault.c”>
VOID	fault(sig)
	REG INT		sig;
{
	signal(sig, fault);
	IF sig==MEMF
	THEN	IF setbrk(brkincr) == -1
		THEN	error(nospace);
		FI
	ELIF ...
<endquote>

This was already present in 7th edition, so it is by no means new in 32V or SysIII -- it had just escaped my attention as a conceptual step in the development of Unix memory handling.




^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2022-12-30 18:25 [TUHS] A few comments on porting the Bourne shell Paul Ruizendaal
@ 2022-12-30 19:51 ` Chet Ramey
  2022-12-30 20:02   ` Larry McVoy
                     ` (2 more replies)
  2022-12-30 19:57 ` segaloco via TUHS
  1 sibling, 3 replies; 121+ messages in thread
From: Chet Ramey @ 2022-12-30 19:51 UTC (permalink / raw)
  To: Paul Ruizendaal, The Eunuchs Hysterical Society

On 12/30/22 1:25 PM, Paul Ruizendaal wrote:
> 
> London and Reiser report about porting the shell that “it required by far the largest conversion effort of any supposedly portable program, for the simple reason that it is not portable.” By the time of SysIII this is greatly improved, but also in porting the SysIII user land it was the most complex of the set so far.

Have you read

http://www.collyer.net/who/geoff/sh.tour.pdf

and looked at http://www.collyer.net/who/geoff/v7sh.tar ?

In the limited literature on Bourne Shell porting, this is authoritative.

Arnold Robbins built on that work and ported the v8-v10 shells to modern
Linux versions. (I am sorry, I do not have a link right now.)

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
		 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/


^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2022-12-30 18:25 [TUHS] A few comments on porting the Bourne shell Paul Ruizendaal
  2022-12-30 19:51 ` [TUHS] " Chet Ramey
@ 2022-12-30 19:57 ` segaloco via TUHS
  2022-12-31 12:55   ` Paul Ruizendaal
  1 sibling, 1 reply; 121+ messages in thread
From: segaloco via TUHS @ 2022-12-30 19:57 UTC (permalink / raw)
  To: Paul Ruizendaal; +Cc: The Eunuchs Hysterical Society

I'll have to double check later but I'm fairly certain the remaining L/R cheats are gone by SysV.  From what I can tell much of that portability work may have been done prior to the V7 release code base we're familiar with, as I did some comparison and found only one significant change between V7 and 32V code as I know it at least.  Either the claims of portability issues came between 32V and System III (meaning the shell was accepted as "broken"? in 32V) or the code we actually see in V7 has already been tidied up significantly and doesn't represent the "non-portable" version lamented in the famous quote.  Does this observation hold with reality?  Is there an earlier, more PDP-11 bound version of the Bourne Shell out there?  I seem to recall reading something about some bits of it even being in assembly at one point, but can't remember the quote source.

- Matt G.

------- Original Message -------
On Friday, December 30th, 2022 at 10:25 AM, Paul Ruizendaal <pnr@planet.nl> wrote:


> London and Reiser report about porting the shell that “it required by far the largest conversion effort of any supposedly portable program, for the simple reason that it is not portable.” By the time of SysIII this is greatly improved, but also in porting the SysIII user land it was the most complex of the set so far.
> 
> There were three aspects that I found noteworthy:
> 
> 1. London/Reiser apparently felt strongly about a property of casts. The code argues that casting an l-value should not convert it into a r-value:
> 
> <quote from "mode.h">
> 
> /* the following nonsense is required
> * because casts turn an Lvalue
> * into an Rvalue so two cheats
> * are necessary, one for each context.
> */
> union { int _cheat;};
> #define Lcheat(a) ((a)._cheat)
> #define Rcheat(a) ((int)(a))
> <endquote>
> 
> 
> However, Lcheat is only used in two places (in service.c), to set and to clear a flag in a pointer. Interestingly, the 32V code already replaces one of these instances with a regular r-value cast. So far, I’d never thought about this aspect of casts. I stumbled across it, because the Plan 9 compiler did not accept the Lcheat expansion as valid C.
> 
> 2. On the history of dup2
> 
> The shell code includes the following:
> 
> <quote from “io.c”>
> 
> rename(f1,f2)
> REG INT f1, f2;
> {
> #ifdef RES /* research has different sys calls from TS */
> IF f1!=f2
> THEN dup(f1|DUPFLG, f2);
> close(f1);
> IF f2==0 THEN ioset|=1 FI
> FI
> #else
> INT fs;
> IF f1!=f2
> THEN fs = fcntl(f2,1,0);
> close(f2);
> fcntl(f1,0,f2);
> close(f1);
> IF fs==1 THEN fcntl(f2,2,1) FI
> IF f2==0 THEN ioset|=1 FI
> FI
> #endif
> }
> <endquote>
> 
> 
> I’ve check the 8th edition source, and indeed it supports using DUPFLG to signal to dup() that it really is dup2(). I had earlier wondered why dup2() did not appear in research until 10th edition, but now that is clear. It would seem that the dup of 8th edition is a direct ancestor to dup() in Plan 9. I wonder why this way of doing things never caught on in the other Unices.
> 
> 3. Halfway to demand paging
> 
> I stumbled across this one because I had a bug in my signal handling. From early days onwards, Unix supported dynamically growing the stack allocation, which arguably is a first step towards building the mechanisms for demand paging. It appears that the Bourne shell made another step, catching page faults and expanding the data/bss allocation dynamically:
> 
> <quote from “fault.c”>
> 
> VOID fault(sig)
> REG INT sig;
> {
> signal(sig, fault);
> IF sig==MEMF
> THEN IF setbrk(brkincr) == -1
> THEN error(nospace);
> FI
> ELIF ...
> <endquote>
> 
> 
> This was already present in 7th edition, so it is by no means new in 32V or SysIII -- it had just escaped my attention as a conceptual step in the development of Unix memory handling.
>

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2022-12-30 19:51 ` [TUHS] " Chet Ramey
@ 2022-12-30 20:02   ` Larry McVoy
  2022-12-30 20:31     ` Adam Thornton
                       ` (3 more replies)
  2022-12-30 20:20   ` Sven Mascheck
  2023-01-01 10:44   ` arnold
  2 siblings, 4 replies; 121+ messages in thread
From: Larry McVoy @ 2022-12-30 20:02 UTC (permalink / raw)
  To: Chet Ramey; +Cc: Paul Ruizendaal, The Eunuchs Hysterical Society

On Fri, Dec 30, 2022 at 02:51:26PM -0500, Chet Ramey wrote:
> On 12/30/22 1:25 PM, Paul Ruizendaal wrote:
> >
> >London and Reiser report about porting the shell that ???it required by far the largest conversion effort of any supposedly portable program, for the simple reason that it is not portable.??? By the time of SysIII this is greatly improved, but also in porting the SysIII user land it was the most complex of the set so far.
> 
> Have you read
> 
> http://www.collyer.net/who/geoff/sh.tour.pdf
> 
> and looked at http://www.collyer.net/who/geoff/v7sh.tar ?
> 
> In the limited literature on Bourne Shell porting, this is authoritative.

Is there are reason to hang on to the Bourne shell?  Maybe shell scripts?
Does it perform better than ksh or bash?

Don't get me wrong, I much prefer the sh syntax over csh syntax, but 
I'd never go back to the Bourne shell as my login shell.  Way too much
useful stuff in ksh/bash.

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2022-12-30 19:51 ` [TUHS] " Chet Ramey
  2022-12-30 20:02   ` Larry McVoy
@ 2022-12-30 20:20   ` Sven Mascheck
  2022-12-30 20:49     ` Ron Natalie
  2023-01-01 10:44   ` arnold
  2 siblings, 1 reply; 121+ messages in thread
From: Sven Mascheck @ 2022-12-30 20:20 UTC (permalink / raw)
  To: tuhs

Chet Ramey on 30.12.2022 20:51:
> Arnold Robbins built on that work and ported the v8-v10 shells to modern
> Linux versions. (I am sorry, I do not have a link right now.)

And btw, 8th ed (http://man.cat-v.org/unix_8th/1/sh) even added a simple 
history mechanism with the "=(1)" command 
(https://www.in-ulm.de/~mascheck/bourne/v8/=.html).

Sven


^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2022-12-30 20:02   ` Larry McVoy
@ 2022-12-30 20:31     ` Adam Thornton
  2022-12-30 20:49       ` Chet Ramey
  2022-12-30 20:42     ` Sven Mascheck
                       ` (2 subsequent siblings)
  3 siblings, 1 reply; 121+ messages in thread
From: Adam Thornton @ 2022-12-30 20:31 UTC (permalink / raw)
  To: Larry McVoy; +Cc: Paul Ruizendaal, The Eunuchs Hysterical Society



> On Dec 30, 2022, at 1:02 PM, Larry McVoy <lm@mcvoy.com> wrote:
>> 
> 
> Is there are reason to hang on to the Bourne shell?  Maybe shell scripts?
> Does it perform better than ksh or bash?

POSIX sh seems like a good lowest-common-denominator shell to write for (sorry, SunOS and xpg4).

> 
> Don't get me wrong, I much prefer the sh syntax over csh syntax, but 
> I'd never go back to the Bourne shell as my login shell.  Way too much
> useful stuff in ksh/bash.

I mean I would assume the reason is that bash takes a whole lot more memory and CPU to do its very convenient magic.  That's an issue if you're trying to run on a PDP-11 and only have 16 bits of address space, but in 64-bit-world, where you almost certainly have at least half a gigabyte of core storage available, and your processor clock is almost certainly in at least the high hundreds of MHz...yeah.

Although (because my daily driver is a Mac) I've finally taken the zsh plunge.  I feel a little dirty, but oh-my-zsh certainly has its attractions.

Adam


^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2022-12-30 20:02   ` Larry McVoy
  2022-12-30 20:31     ` Adam Thornton
@ 2022-12-30 20:42     ` Sven Mascheck
  2022-12-30 20:48       ` Jon Steinhart
  2023-01-02 20:03       ` [TUHS] Command Line Editing in the Terminal Driver (Was: A few comments on porting the Bourne shell) Joseph Holsten
  2022-12-30 20:47     ` [TUHS] Re: A few comments on porting the Bourne shell Chet Ramey
  2022-12-31  0:08     ` Luther Johnson
  3 siblings, 2 replies; 121+ messages in thread
From: Sven Mascheck @ 2022-12-30 20:42 UTC (permalink / raw)
  To: tuhs

[-- Attachment #1: Type: text/plain, Size: 1601 bytes --]

Larry McVoy on 30.12.2022 21:02:

> > [SysIII port]
> Is there are reason to hang on to the Bourne shell?  Maybe shell scripts?
> Does it perform better than ksh or bash?
>
> Don't get me wrong, I much prefer the sh syntax over csh syntax, but
> I'd never go back to the Bourne shell as my login shell.  Way too much
> useful stuff in ksh/bash.

I'd like the idea of   "preserving a heirloom in its natural environment"
(and even more effort went in https://heirloom.sourceforge.net/sh.html)
let alone this does not prevent from adding modern shells...

I guess in interactive use most users would only miss one thing: the 
history & line editing capability?

Side notes to that:

  * By intention, the almquist shell (a port due to the Berkeley/ATT
    mess) initially had no history. From the package file DIFFERENCES [1],

    "History.   It seems to me that the csh history mechanism is mostly
    a response to the deficiencies of UNIX terminal I/O. Those of you
    running 4.2 BSD should try out atty (which I am posting to the net
    at the same time as ash) and see if you still want history."

  * and in "ksh - An Extensible High Level Language" David Korn writes:

    "Originally the idea of adding command line editing to ksh was
    rejected in the hope that line editing would move into the terminal
    driver." [2]

I have always wondered, what such a central terminal driver driven 
history/line-editing would have felt like.

[1] https://www.in-ulm.de/~mascheck/various/ash/DIFFERENCES
[2] https://www.in-ulm.de/~mascheck/bourne/korn.html

[-- Attachment #2: Type: text/html, Size: 2451 bytes --]

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2022-12-30 20:02   ` Larry McVoy
  2022-12-30 20:31     ` Adam Thornton
  2022-12-30 20:42     ` Sven Mascheck
@ 2022-12-30 20:47     ` Chet Ramey
  2022-12-31  0:08     ` Luther Johnson
  3 siblings, 0 replies; 121+ messages in thread
From: Chet Ramey @ 2022-12-30 20:47 UTC (permalink / raw)
  To: Larry McVoy; +Cc: Paul Ruizendaal, The Eunuchs Hysterical Society

On 12/30/22 3:02 PM, Larry McVoy wrote:
> On Fri, Dec 30, 2022 at 02:51:26PM -0500, Chet Ramey wrote:
>> On 12/30/22 1:25 PM, Paul Ruizendaal wrote:
>>>
>>> London and Reiser report about porting the shell that ???it required by far the largest conversion effort of any supposedly portable program, for the simple reason that it is not portable.??? By the time of SysIII this is greatly improved, but also in porting the SysIII user land it was the most complex of the set so far.
>>
>> Have you read
>>
>> http://www.collyer.net/who/geoff/sh.tour.pdf
>>
>> and looked at http://www.collyer.net/who/geoff/v7sh.tar ?
>>
>> In the limited literature on Bourne Shell porting, this is authoritative.
> 
> Is there are reason to hang on to the Bourne shell?  Maybe shell scripts?
> Does it perform better than ksh or bash?

Historical interest? Software archaeology? Reference behavior?

I don't think anyone is suggesting that we use it as a login shell, in the
same way that no one is suggesting we go back to using v7 as an everyday
computing environment. The world's come too far.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
		 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/


^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2022-12-30 20:42     ` Sven Mascheck
@ 2022-12-30 20:48       ` Jon Steinhart
  2022-12-30 20:51         ` Sven Mascheck
  2022-12-31 11:40         ` Ralph Corderoy
  2023-01-02 20:03       ` [TUHS] Command Line Editing in the Terminal Driver (Was: A few comments on porting the Bourne shell) Joseph Holsten
  1 sibling, 2 replies; 121+ messages in thread
From: Jon Steinhart @ 2022-12-30 20:48 UTC (permalink / raw)
  To: tuhs

Sven Mascheck writes:
> I guess in interactive use most users would only miss one thing: the
> history & line editing capability?

Job control?

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2022-12-30 20:31     ` Adam Thornton
@ 2022-12-30 20:49       ` Chet Ramey
  0 siblings, 0 replies; 121+ messages in thread
From: Chet Ramey @ 2022-12-30 20:49 UTC (permalink / raw)
  To: Adam Thornton, Larry McVoy
  Cc: Paul Ruizendaal, The Eunuchs Hysterical Society

On 12/30/22 3:31 PM, Adam Thornton wrote:

> Although (because my daily driver is a Mac) I've finally taken the zsh plunge.  

You don't have to, you know. ;-)

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
		 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/


^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2022-12-30 20:20   ` Sven Mascheck
@ 2022-12-30 20:49     ` Ron Natalie
  2022-12-30 20:52       ` Rob Pike
  2022-12-30 20:53       ` Jon Steinhart
  0 siblings, 2 replies; 121+ messages in thread
From: Ron Natalie @ 2022-12-30 20:49 UTC (permalink / raw)
  To: Sven Mascheck, tuhs

I was just happy when VR2 came out and found someone had undone all 
those macros that were in the original code.
I hacked Berkeley Job Control as well as command line editing into it 
(KSH hadn’t seen the light of day outside the labs at that point).
I subsequently had a nice talk with Korn at a USENIX.    I also sat down 
with a couple of the guys trying to implement their own shell 
independent of the ATT code and explained to them how Berkeley job 
control works.   It’s for that reason my name shows up in a lot of the 
early Linux docs.

Amusingly, I’d forgotten all about this stuff until one day I was 
sitting at a MIPS workstation (MIPS branded, not the DEC spim).    
Without thinking about it, I typed “fg” at the shell prompt.

“Job control not Enabled,” it said.   Hey!  That sounds like one of my 
messages.   “set -J” I type.   “Job control enabled.”
Hey!  This is a “Ron shell” as it was known at BRL.   Turns out it went 
out on the Mach distros so I’ve found it on all kinds of things like the 
NeXT etc…


------ Original Message ------
From "Sven Mascheck" <mascheck@in-ulm.de>
To tuhs@tuhs.org
Date 12/30/2022 3:20:28 PM
Subject [TUHS] Re: A few comments on porting the Bourne shell

>Chet Ramey on 30.12.2022 20:51:
>>Arnold Robbins built on that work and ported the v8-v10 shells to modern
>>Linux versions. (I am sorry, I do not have a link right now.)
>
>And btw, 8th ed (http://man.cat-v.org/unix_8th/1/sh) even added a simple history mechanism with the "=(1)" command (https://www.in-ulm.de/~mascheck/bourne/v8/=.html).
>
>Sven
>

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2022-12-30 20:48       ` Jon Steinhart
@ 2022-12-30 20:51         ` Sven Mascheck
  2022-12-31 11:40         ` Ralph Corderoy
  1 sibling, 0 replies; 121+ messages in thread
From: Sven Mascheck @ 2022-12-30 20:51 UTC (permalink / raw)
  To: tuhs

Jon Steinhart 30.12.2022 21:48:

>> I guess in interactive use most users would only miss one thing: the
>> history & line editing capability?
> Job control?
well spotted, not available in vanilla sh until SVR4...


^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2022-12-30 20:49     ` Ron Natalie
@ 2022-12-30 20:52       ` Rob Pike
  2022-12-30 20:53       ` Jon Steinhart
  1 sibling, 0 replies; 121+ messages in thread
From: Rob Pike @ 2022-12-30 20:52 UTC (permalink / raw)
  To: Ron Natalie; +Cc: tuhs

[-- Attachment #1: Type: text/plain, Size: 1907 bytes --]

That "someone" was Steve Bourne. At least, the version he handed me when I
built up the v8 shell was from him and clean of ELIFs.

-rob


On Sat, Dec 31, 2022 at 7:50 AM Ron Natalie <ron@ronnatalie.com> wrote:

> I was just happy when VR2 came out and found someone had undone all
> those macros that were in the original code.
> I hacked Berkeley Job Control as well as command line editing into it
> (KSH hadn’t seen the light of day outside the labs at that point).
> I subsequently had a nice talk with Korn at a USENIX.    I also sat down
> with a couple of the guys trying to implement their own shell
> independent of the ATT code and explained to them how Berkeley job
> control works.   It’s for that reason my name shows up in a lot of the
> early Linux docs.
>
> Amusingly, I’d forgotten all about this stuff until one day I was
> sitting at a MIPS workstation (MIPS branded, not the DEC spim).
> Without thinking about it, I typed “fg” at the shell prompt.
>
> “Job control not Enabled,” it said.   Hey!  That sounds like one of my
> messages.   “set -J” I type.   “Job control enabled.”
> Hey!  This is a “Ron shell” as it was known at BRL.   Turns out it went
> out on the Mach distros so I’ve found it on all kinds of things like the
> NeXT etc…
>
>
> ------ Original Message ------
> From "Sven Mascheck" <mascheck@in-ulm.de>
> To tuhs@tuhs.org
> Date 12/30/2022 3:20:28 PM
> Subject [TUHS] Re: A few comments on porting the Bourne shell
>
> >Chet Ramey on 30.12.2022 20:51:
> >>Arnold Robbins built on that work and ported the v8-v10 shells to modern
> >>Linux versions. (I am sorry, I do not have a link right now.)
> >
> >And btw, 8th ed (http://man.cat-v.org/unix_8th/1/sh) even added a simple
> history mechanism with the "=(1)" command (
> https://www.in-ulm.de/~mascheck/bourne/v8/=.html).
> >
> >Sven
> >
>

[-- Attachment #2: Type: text/html, Size: 2908 bytes --]

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2022-12-30 20:49     ` Ron Natalie
  2022-12-30 20:52       ` Rob Pike
@ 2022-12-30 20:53       ` Jon Steinhart
  1 sibling, 0 replies; 121+ messages in thread
From: Jon Steinhart @ 2022-12-30 20:53 UTC (permalink / raw)
  To: tuhs

Ron Natalie writes:
> I was just happy when VR2 came out and found someone had undone all 
> those macros that were in the original code.

On an amusing note, a while ago I had polled folks on a different
mailing list on items to include in a starter tool set for my daughter.

Steve:
  Hi Jon, highly recommend DeWalt 20V drill, here is my checklist for my boat
  tools.  Might be something there to add.  Some clearly not but didn't scrub.

Jon:
   Thanks.  I was thinking that I should also get her one of those adapters that
   connects a { to a BEGIN :-)

Steve:
  Looks like this

  :-}

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2022-12-30 20:02   ` Larry McVoy
                       ` (2 preceding siblings ...)
  2022-12-30 20:47     ` [TUHS] Re: A few comments on porting the Bourne shell Chet Ramey
@ 2022-12-31  0:08     ` Luther Johnson
  2022-12-31  3:59       ` Larry McVoy
  3 siblings, 1 reply; 121+ messages in thread
From: Luther Johnson @ 2022-12-31  0:08 UTC (permalink / raw)
  To: tuhs

The Almquist shell (ash) and Debian's version of it, "dash", are 
compatible re-implementations of the Bourne shell, and use much less 
resources, and therefore wind up being faster as processes come and go.

I use csh (tcsh) and bash at the command line, but straight Bourne shell 
language is preferred and recommended by many for shell programming, I 
used to use csh for that and got bitten by all the things that I later 
discovered, those in the know had been warning about for years. Also, 
"bash-isms", syntactic sugary things in bash had led me to use them as a 
crutch, my scripts got simpler and more to the point when I re-wrote 
them for Bourne shell language only. That was my experience. I think 
we'll always have some kind of Bourne shell as the script workhorse, at 
last in Linux/Unix start-up and other blood and guts stuff.

In John Gilmore's paper on porting BSD through gcc to get to 4.4, he 
also remarked on how the original Bourne shell code was difficult to port.

On 12/30/2022 01:02 PM, Larry McVoy wrote:
> On Fri, Dec 30, 2022 at 02:51:26PM -0500, Chet Ramey wrote:
>> On 12/30/22 1:25 PM, Paul Ruizendaal wrote:
>>> London and Reiser report about porting the shell that ???it required by far the largest conversion effort of any supposedly portable program, for the simple reason that it is not portable.??? By the time of SysIII this is greatly improved, but also in porting the SysIII user land it was the most complex of the set so far.
>> Have you read
>>
>> http://www.collyer.net/who/geoff/sh.tour.pdf
>>
>> and looked at http://www.collyer.net/who/geoff/v7sh.tar ?
>>
>> In the limited literature on Bourne Shell porting, this is authoritative.
> Is there are reason to hang on to the Bourne shell?  Maybe shell scripts?
> Does it perform better than ksh or bash?
>
> Don't get me wrong, I much prefer the sh syntax over csh syntax, but
> I'd never go back to the Bourne shell as my login shell.  Way too much
> useful stuff in ksh/bash.
>


^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2022-12-31  0:08     ` Luther Johnson
@ 2022-12-31  3:59       ` Larry McVoy
  2022-12-31  4:12         ` Steve Nickolas
                           ` (4 more replies)
  0 siblings, 5 replies; 121+ messages in thread
From: Larry McVoy @ 2022-12-31  3:59 UTC (permalink / raw)
  To: Luther Johnson; +Cc: tuhs

On Fri, Dec 30, 2022 at 05:08:42PM -0700, Luther Johnson wrote:
> I use csh (tcsh) and bash at the command line, but straight Bourne shell
> language is preferred and recommended by many for shell programming, I used
> to use csh for that and got bitten by all the things that I later
> discovered, those in the know had been warning about for years. Also,
> "bash-isms", syntactic sugary things in bash had led me to use them as a
> crutch, my scripts got simpler and more to the point when I re-wrote them
> for Bourne shell language only. That was my experience. I think we'll always
> have some kind of Bourne shell as the script workhorse, at last in
> Linux/Unix start-up and other blood and guts stuff.

When I was running my engineering team I was strict about Bourne syntax
and features only.  I got pushed on like crazy because "bash has this
$GOODNESS whhhhhhhy can't we use it".  Because we were supporting our
product on pretty much every unix and if it wasn't HP-UX that had an
ancient /bin/sh, it was AIX or whoever. 

Over and over, I won the "straight bourne shell only" battle.  So I agree,
if you want /bin/sh to work, Bourne shell for the win.

For a login shell, bash is my shell of choice.  It's bloated but I'm
typing this on a 5 year old Lenova X1 Carbon with 16GB of memory and
4 cores and it's fine.  It was fine a 133mhz Pentium.

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2022-12-31  3:59       ` Larry McVoy
@ 2022-12-31  4:12         ` Steve Nickolas
  2022-12-31  4:18         ` Bakul Shah
                           ` (3 subsequent siblings)
  4 siblings, 0 replies; 121+ messages in thread
From: Steve Nickolas @ 2022-12-31  4:12 UTC (permalink / raw)
  To: tuhs

On Fri, 30 Dec 2022, Larry McVoy wrote:

> When I was running my engineering team I was strict about Bourne syntax
> and features only.  I got pushed on like crazy because "bash has this
> $GOODNESS whhhhhhhy can't we use it".  Because we were supporting our
> product on pretty much every unix and if it wasn't HP-UX that had an
> ancient /bin/sh, it was AIX or whoever.
>
> Over and over, I won the "straight bourne shell only" battle.  So I agree,
> if you want /bin/sh to work, Bourne shell for the win.
>
> For a login shell, bash is my shell of choice.  It's bloated but I'm
> typing this on a 5 year old Lenova X1 Carbon with 16GB of memory and
> 4 cores and it's fine.  It was fine a 133mhz Pentium.

There's some variants of the Almquist shell that have bash-style 
command-line editing, and I can deal with that if that's /bin/sh.  Usually 
I use bash and it's plenty fine.

I generally code shell scripts for Posix sh as my baseline and start my 
scripts with #!/bin/sh - but sometimes I'll use #!/bin/ksh if I don't 
expect them to be used on another machine.  ksh93 is lighter and faster 
than bash and the only things I miss are quirks of the command line 
editor.

-uso.

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2022-12-31  3:59       ` Larry McVoy
  2022-12-31  4:12         ` Steve Nickolas
@ 2022-12-31  4:18         ` Bakul Shah
  2022-12-31  4:40           ` Larry McVoy
  2022-12-31  4:19         ` Marc Donner
                           ` (2 subsequent siblings)
  4 siblings, 1 reply; 121+ messages in thread
From: Bakul Shah @ 2022-12-31  4:18 UTC (permalink / raw)
  To: Larry McVoy; +Cc: tuhs

On Dec 30, 2022, at 7:59 PM, Larry McVoy <lm@mcvoy.com> wrote:
> 
> For a login shell, bash is my shell of choice.  It's bloated but I'm
> typing this on a 5 year old Lenova X1 Carbon with 16GB of memory and
> 4 cores and it's fine.  It was fine a 133mhz Pentium.

I switched to zsh in 1993-94 when I found out it had sh compatible
syntax but also had some useful features from csh (bang commands,
a{b,c} expansion, aliases etc.). Back then my machine had 16MB of
memory (IIRC a 50Mhz i486). zsh is the Cadillac of shells. But I
also use rc, the VWbug of shells on plan9!



^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2022-12-31  3:59       ` Larry McVoy
  2022-12-31  4:12         ` Steve Nickolas
  2022-12-31  4:18         ` Bakul Shah
@ 2022-12-31  4:19         ` Marc Donner
  2022-12-31  4:23         ` Dave Horsfall
  2022-12-31  4:41         ` Greg 'groggy' Lehey
  4 siblings, 0 replies; 121+ messages in thread
From: Marc Donner @ 2022-12-31  4:19 UTC (permalink / raw)
  To: Larry McVoy; +Cc: tuhs

[-- Attachment #1: Type: text/plain, Size: 2207 bytes --]

Fascinating.

When I left IBM Research to become a grad student at CMU (1981) the Unix CS
was running did not have history in the shell.  It had just been introduced
in VM/CMS and I loved it.  I nagged the sysadmins in the CS department
about it, and voila, it appeared shortly thereafter.  I presume it was one
of the things discussed here ported to the CMU environment.

I remember that I implemented history for the shell in the PERQ, doing some
nasty stuff to fit a reasonable history length in the weird Pascal they
had, since Pascal didn’t have dynamic strings or real pointers.

Marc
=====
On Fri, Dec 30, 2022 at 10:59 PM Larry McVoy <lm@mcvoy.com> wrote:

> On Fri, Dec 30, 2022 at 05:08:42PM -0700, Luther Johnson wrote:
> > I use csh (tcsh) and bash at the command line, but straight Bourne shell
> > language is preferred and recommended by many for shell programming, I
> used
> > to use csh for that and got bitten by all the things that I later
> > discovered, those in the know had been warning about for years. Also,
> > "bash-isms", syntactic sugary things in bash had led me to use them as a
> > crutch, my scripts got simpler and more to the point when I re-wrote them
> > for Bourne shell language only. That was my experience. I think we'll
> always
> > have some kind of Bourne shell as the script workhorse, at last in
> > Linux/Unix start-up and other blood and guts stuff.
>
> When I was running my engineering team I was strict about Bourne syntax
> and features only.  I got pushed on like crazy because "bash has this
> $GOODNESS whhhhhhhy can't we use it".  Because we were supporting our
> product on pretty much every unix and if it wasn't HP-UX that had an
> ancient /bin/sh, it was AIX or whoever.
>
> Over and over, I won the "straight bourne shell only" battle.  So I agree,
> if you want /bin/sh to work, Bourne shell for the win.
>
> For a login shell, bash is my shell of choice.  It's bloated but I'm
> typing this on a 5 year old Lenova X1 Carbon with 16GB of memory and
> 4 cores and it's fine.  It was fine a 133mhz Pentium.
>
-- 
=====
nygeek.net
mindthegapdialogs.com/home <https://www.mindthegapdialogs.com/home>

[-- Attachment #2: Type: text/html, Size: 3001 bytes --]

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2022-12-31  3:59       ` Larry McVoy
                           ` (2 preceding siblings ...)
  2022-12-31  4:19         ` Marc Donner
@ 2022-12-31  4:23         ` Dave Horsfall
  2022-12-31  4:37           ` Clem Cole
  2023-01-02  5:10           ` Mary Ann Horton
  2022-12-31  4:41         ` Greg 'groggy' Lehey
  4 siblings, 2 replies; 121+ messages in thread
From: Dave Horsfall @ 2022-12-31  4:23 UTC (permalink / raw)
  To: The Eunuchs Hysterical Society

On Fri, 30 Dec 2022, Larry McVoy wrote:

> When I was running my engineering team I was strict about Bourne syntax 
> and features only.  I got pushed on like crazy because "bash has this 
> $GOODNESS whhhhhhhy can't we use it".  Because we were supporting our 
> product on pretty much every unix and if it wasn't HP-UX that had an 
> ancient /bin/sh, it was AIX or whoever.

I've never bothered to learn those Bash thingies, because "expr" does 
everything that I need and is available on just about all boxes.

> Over and over, I won the "straight bourne shell only" battle.  So I 
> agree, if you want /bin/sh to work, Bourne shell for the win.

Yep; whoever wrote CSH must've been high on something, as the syntax makes 
no sense whatsoever.

> For a login shell, bash is my shell of choice.  It's bloated but I'm 
> typing this on a 5 year old Lenova X1 Carbon with 16GB of memory and 4 
> cores and it's fine.  It was fine a 133mhz Pentium.

I do admit to being a bit of a ZSH user...  I've never bothered to learn 
all its features, but the subset I use gets me through.

-- Dave

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2022-12-31  4:23         ` Dave Horsfall
@ 2022-12-31  4:37           ` Clem Cole
  2023-01-02  5:10           ` Mary Ann Horton
  1 sibling, 0 replies; 121+ messages in thread
From: Clem Cole @ 2022-12-31  4:37 UTC (permalink / raw)
  To: Dave Horsfall; +Cc: The Eunuchs Hysterical Society

[-- Attachment #1: Type: text/plain, Size: 1287 bytes --]

Bourne to program, type with Joy.

On Fri, Dec 30, 2022 at 11:23 PM Dave Horsfall <dave@horsfall.org> wrote:

> On Fri, 30 Dec 2022, Larry McVoy wrote:
>
> > When I was running my engineering team I was strict about Bourne syntax
> > and features only.  I got pushed on like crazy because "bash has this
> > $GOODNESS whhhhhhhy can't we use it".  Because we were supporting our
> > product on pretty much every unix and if it wasn't HP-UX that had an
> > ancient /bin/sh, it was AIX or whoever.
>
> I've never bothered to learn those Bash thingies, because "expr" does
> everything that I need and is available on just about all boxes.
>
> > Over and over, I won the "straight bourne shell only" battle.  So I
> > agree, if you want /bin/sh to work, Bourne shell for the win.
>
> Yep; whoever wrote CSH must've been high on something, as the syntax makes
> no sense whatsoever.
>
> > For a login shell, bash is my shell of choice.  It's bloated but I'm
> > typing this on a 5 year old Lenova X1 Carbon with 16GB of memory and 4
> > cores and it's fine.  It was fine a 133mhz Pentium.
>
> I do admit to being a bit of a ZSH user...  I've never bothered to learn
> all its features, but the subset I use gets me through.
>
> -- Dave
>
-- 
Sent from a handheld expect more typos than usual

[-- Attachment #2: Type: text/html, Size: 1825 bytes --]

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2022-12-31  4:18         ` Bakul Shah
@ 2022-12-31  4:40           ` Larry McVoy
  0 siblings, 0 replies; 121+ messages in thread
From: Larry McVoy @ 2022-12-31  4:40 UTC (permalink / raw)
  To: Bakul Shah; +Cc: tuhs

On Fri, Dec 30, 2022 at 08:18:06PM -0800, Bakul Shah wrote:
> On Dec 30, 2022, at 7:59 PM, Larry McVoy <lm@mcvoy.com> wrote:
> > 
> > For a login shell, bash is my shell of choice.  It's bloated but I'm
> > typing this on a 5 year old Lenova X1 Carbon with 16GB of memory and
> > 4 cores and it's fine.  It was fine a 133mhz Pentium.
> 
> I switched to zsh in 1993-94 when I found out it had sh compatible
> syntax but also had some useful features from csh (bang commands,
> a{b,c} expansion, aliases etc.). 

This should probably move to COFF (which unlike many other lists I've
been on, has critical mass in the "go over there" list).

You are the 2nd to say zsh is your login shell.  I tried it for a while,
my progression was ${I dunno} to csh, actually used Bourne shell as a
login shell for a while around 1985 or so, all of my login scripts, other
shell scripts, were Bourne shell, I think zsh was next, then sighed in
relief with the original ksh release that went out, and when bash came
around with portability to everywhere, it has been bash.  ksh93 was 
interesting but bash was good enough and clearly open source so...

For script portability, pure Bourne shell has been the win for at least
40 years.  I suspect my kids and their kids will be writing Steve's
syntax, that's a legacy.

For general purpose scripting, I've moved on.  I loved Perl4.  It's a
kitchen sink language but it so clearly had roots in the Bourne shell,
in awk, in sed.  It was trying to be all of those things in one language
and it mostly succeeded.  In the 40mhz SPARC days I actually advocated
for rewriting a ton of /usr/bin on SunOS in perl4 because it was so easy
and far more maintainable than the C versions.

These days perl is weird, the kids seem to like Python, I hate Python
because there is no printf in the base language.  How is that a thing?
But people like it.
-- 
---
Larry McVoy           Retired to fishing          http://www.mcvoy.com/lm/boat

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2022-12-31  3:59       ` Larry McVoy
                           ` (3 preceding siblings ...)
  2022-12-31  4:23         ` Dave Horsfall
@ 2022-12-31  4:41         ` Greg 'groggy' Lehey
  4 siblings, 0 replies; 121+ messages in thread
From: Greg 'groggy' Lehey @ 2022-12-31  4:41 UTC (permalink / raw)
  To: Larry McVoy; +Cc: tuhs

[-- Attachment #1: Type: text/plain, Size: 1096 bytes --]

On Friday, 30 December 2022 at 19:59:31 -0800, Larry McVoy wrote:
> Over and over, I won the "straight bourne shell only" battle.  So I agree,
> if you want /bin/sh to work, Bourne shell for the win.

Agreed.

> For a login shell, bash is my shell of choice.  It's bloated but I'm
> typing this on a 5 year old Lenova X1 Carbon with 16GB of memory and
> 4 cores and it's fine.  It was fine a 133mhz Pentium.

I've been using bash since 16 MHz i386s.  People told me that it was
bloated, but somehow I didn't notice.

Somehow this reminds me of the expansion of EMACS: Eight Megabytes And
Continually Swapping.  8 MB?  Nowadays?  It shows how old these
prejudices are.  Now my emacs processes have round 30 MB of resident
memory and are dwarfed by 2 GB web browsers.  bash comes in at a
little over 2 MB, just not worth talking about.

Greg
--
Sent from my desktop computer.
Finger grog@lemis.com for PGP public key.
See complete headers for address and phone numbers.
This message is digitally signed.  If your Microsoft mail program
reports problems, please read http://lemis.com/broken-MUA.php

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 163 bytes --]

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2022-12-30 20:48       ` Jon Steinhart
  2022-12-30 20:51         ` Sven Mascheck
@ 2022-12-31 11:40         ` Ralph Corderoy
  2022-12-31 18:49           ` Jon Steinhart
  1 sibling, 1 reply; 121+ messages in thread
From: Ralph Corderoy @ 2022-12-31 11:40 UTC (permalink / raw)
  To: tuhs

Hi Jon,

> > I guess in interactive use most users would only miss one thing:
> > the history & line editing capability?
>
> Job control?

Just as history and line editing were a possible enhancement to the TTY
driver, didn't Pike comment that job control was cooked up in foreign
climes because they couldn't drag out a new TTY window in pixels?

Was something like screen(1)'s ‘Ctrl-A c’ considered in the TTY driver
to spin up another pseudo-TTY rather than suspend the long job with
Ctrl-Z and ‘bg’ it?

-- 
Cheers, Ralph.

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2022-12-30 19:57 ` segaloco via TUHS
@ 2022-12-31 12:55   ` Paul Ruizendaal
  2023-01-01  2:55     ` Warner Losh
  0 siblings, 1 reply; 121+ messages in thread
From: Paul Ruizendaal @ 2022-12-31 12:55 UTC (permalink / raw)
  To: The Unix Historical Society; +Cc: segaloco

The "assembly code in the Bourne shell" comment is in the same London/Reiser paper. The full quote is:

"The (Bourne) shell is the standard user command interpreter. It required by far the largest conversion effort of any supposedly portable program, for the simple reason that it is not portable. Critical portions are coded in assembly language and had to be painstakingly rewritten. The shell uses its own sbrk which is functionally different from the standard routine in libc. The shell wants the routine which fields a signal to be passed a parameter giving the number of the signal being caught; signal was also a private rou- tine. This was handled by having the operating system provide the parameter in the first place, doing away with the private code for signal. The code in fixargs (for constructing the argument list to an exec system call) had to be diddled."

The files in the V7 tree on the Tuhs website are dated January 1979, so it would seem that the fixes for 32V were immediately taken back to Research. As you point out, this means that the comments above do not refer to the well known source code, but to a predecessor of that (which I don’t think survived).

Despite all the criticism voiced above, I think it is well understood that the original Bourne shell is an amazing piece of work that managed to fit an enormous amount of functionality into a cramped address space. Its longevity attests to that. That its internals became difficult to understand is par for the course -- the 1980’s in essence needed a Lions commentary on sh.


> On 30 Dec 2022, at 20:57, segaloco <segaloco@protonmail.com> wrote:
> 
> I'll have to double check later but I'm fairly certain the remaining L/R cheats are gone by SysV.  From what I can tell much of that portability work may have been done prior to the V7 release code base we're familiar with, as I did some comparison and found only one significant change between V7 and 32V code as I know it at least.  Either the claims of portability issues came between 32V and System III (meaning the shell was accepted as "broken"? in 32V) or the code we actually see in V7 has already been tidied up significantly and doesn't represent the "non-portable" version lamented in the famous quote.  Does this observation hold with reality?  Is there an earlier, more PDP-11 bound version of the Bourne Shell out there?  I seem to recall reading something about some bits of it even being in assembly at one point, but can't remember the quote source.
> 
> - Matt G.
> 
> ------- Original Message -------
> On Friday, December 30th, 2022 at 10:25 AM, Paul Ruizendaal <pnr@planet.nl> wrote:
> 
> 
>> London and Reiser report about porting the shell that “it required by far the largest conversion effort of any supposedly portable program, for the simple reason that it is not portable.” By the time of SysIII this is greatly improved, but also in porting the SysIII user land it was the most complex of the set so far.
>> 
>> There were three aspects that I found noteworthy:
>> 
>> 1. London/Reiser apparently felt strongly about a property of casts. The code argues that casting an l-value should not convert it into a r-value:
>> 
>> <quote from "mode.h">
>> 
>> /* the following nonsense is required
>> * because casts turn an Lvalue
>> * into an Rvalue so two cheats
>> * are necessary, one for each context.
>> */
>> union { int _cheat;};
>> #define Lcheat(a) ((a)._cheat)
>> #define Rcheat(a) ((int)(a))
>> <endquote>
>> 
>> 
>> However, Lcheat is only used in two places (in service.c), to set and to clear a flag in a pointer. Interestingly, the 32V code already replaces one of these instances with a regular r-value cast. So far, I’d never thought about this aspect of casts. I stumbled across it, because the Plan 9 compiler did not accept the Lcheat expansion as valid C.
>> 
>> 2. On the history of dup2
>> 
>> The shell code includes the following:
>> 
>> <quote from “io.c”>
>> 
>> rename(f1,f2)
>> REG INT f1, f2;
>> {
>> #ifdef RES /* research has different sys calls from TS */
>> IF f1!=f2
>> THEN dup(f1|DUPFLG, f2);
>> close(f1);
>> IF f2==0 THEN ioset|=1 FI
>> FI
>> #else
>> INT fs;
>> IF f1!=f2
>> THEN fs = fcntl(f2,1,0);
>> close(f2);
>> fcntl(f1,0,f2);
>> close(f1);
>> IF fs==1 THEN fcntl(f2,2,1) FI
>> IF f2==0 THEN ioset|=1 FI
>> FI
>> #endif
>> }
>> <endquote>
>> 
>> 
>> I’ve check the 8th edition source, and indeed it supports using DUPFLG to signal to dup() that it really is dup2(). I had earlier wondered why dup2() did not appear in research until 10th edition, but now that is clear. It would seem that the dup of 8th edition is a direct ancestor to dup() in Plan 9. I wonder why this way of doing things never caught on in the other Unices.
>> 
>> 3. Halfway to demand paging
>> 
>> I stumbled across this one because I had a bug in my signal handling. From early days onwards, Unix supported dynamically growing the stack allocation, which arguably is a first step towards building the mechanisms for demand paging. It appears that the Bourne shell made another step, catching page faults and expanding the data/bss allocation dynamically:
>> 
>> <quote from “fault.c”>
>> 
>> VOID fault(sig)
>> REG INT sig;
>> {
>> signal(sig, fault);
>> IF sig==MEMF
>> THEN IF setbrk(brkincr) == -1
>> THEN error(nospace);
>> FI
>> ELIF ...
>> <endquote>
>> 
>> 
>> This was already present in 7th edition, so it is by no means new in 32V or SysIII -- it had just escaped my attention as a conceptual step in the development of Unix memory handling.
>> 


^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2022-12-31 11:40         ` Ralph Corderoy
@ 2022-12-31 18:49           ` Jon Steinhart
  2022-12-31 19:24             ` Clem Cole
  2023-01-01  1:51             ` Ron Natalie
  0 siblings, 2 replies; 121+ messages in thread
From: Jon Steinhart @ 2022-12-31 18:49 UTC (permalink / raw)
  To: tuhs

Ralph Corderoy writes:
> Hi Jon,
>
> > > I guess in interactive use most users would only miss one thing:
> > > the history & line editing capability?
> >
> > Job control?
>
> Just as history and line editing were a possible enhancement to the TTY
> driver, didn't Pike comment that job control was cooked up in foreign
> climes because they couldn't drag out a new TTY window in pixels?
>
> Was something like screen(1)'s ‘Ctrl-A c’ considered in the TTY driver
> to spin up another pseudo-TTY rather than suspend the long job with
> Ctrl-Z and ‘bg’ it?

My memory is really fuzzy on this.  I recall that job control came along
with csh which I used until bash.  At the time there were hardly any
graphics displays used as interactive devices.  I'm talking Berkeley job
control here; I seem to remember that someone got wedged into System V
that was awful and unusable.

Jon

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2022-12-31 18:49           ` Jon Steinhart
@ 2022-12-31 19:24             ` Clem Cole
  2023-01-03 16:32               ` Chet Ramey
  2023-01-01  1:51             ` Ron Natalie
  1 sibling, 1 reply; 121+ messages in thread
From: Clem Cole @ 2022-12-31 19:24 UTC (permalink / raw)
  To: Jon Steinhart; +Cc: tuhs

[-- Attachment #1: Type: text/plain, Size: 1370 bytes --]

below

On Sat, Dec 31, 2022 at 1:50 PM Jon Steinhart <jon@fourwinds.com> wrote:

> My memory is really fuzzy on this.  I recall that job control came along
> with csh which I used until bash.

Close.  That was the wide distribution of it.  Kulp created it originally
for his PDP-11 system in the late 1970s.  I'm not sure how, but  wnj saw
 it and imported the idea to UCB.  Joy had stopped hacking on the 11 by
then.  So  he added the kernel support for Job Control to 4.0 and hacked on
his own shell to add the user interface.  This would get widely distributed
as  in 4.1BSD and since job control was in csh, and not in sh (and job had
made the root use csh ) BSD users learned to used job control from there;
since  4.1BSD release that spread with wings.



> At the time there were hardly any
> graphics displays used as interactive devices.  I'm talking Berkeley job
> control here;

Right - which is Kulp job control to be fair.



> I seem to remember that someone got wedged into System V
> that was awful and unusable.
>
IIRC it was in SVR2 that a version of a job control system added some of
the semantics of Kulp's scheme to the kernel, but not all as you point out
and it was pretty disappointing if you had grown up on BSD systems.  Later
POSIX would pick up the Kulp/BSD Job Control definition and by SVR3/SVR4
AT&T fleshed out all all of the semantics.

[-- Attachment #2: Type: text/html, Size: 2598 bytes --]

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2022-12-31 18:49           ` Jon Steinhart
  2022-12-31 19:24             ` Clem Cole
@ 2023-01-01  1:51             ` Ron Natalie
  1 sibling, 0 replies; 121+ messages in thread
From: Ron Natalie @ 2023-01-01  1:51 UTC (permalink / raw)
  To: Jon Steinhart, tuhs

Berkeley Job Control was implemented in the kernel and originally only 
supported in CSH.    I detested csh, so I figured out how it worked and 
hacked it into a 5(r0) Bourne Shell.    But he time I did that, all the 
other BRL guys were using tcsh and so to counter, I hacked command line 
editing into the 5R2 Bourne shell.


------ Original Message ------
From "Jon Steinhart" <jon@fourwinds.com>
To tuhs@tuhs.org
Date 12/31/2022 1:49:54 PM
Subject [TUHS] Re: A few comments on porting the Bourne shell

>Ralph Corderoy writes:
>>  Hi Jon,
>>
>>  > > I guess in interactive use most users would only miss one thing:
>>  > > the history & line editing capability?
>>  >
>>  > Job control?
>>
>>  Just as history and line editing were a possible enhancement to the TTY
>>  driver, didn't Pike comment that job control was cooked up in foreign
>>  climes because they couldn't drag out a new TTY window in pixels?
>>
>>  Was something like screen(1)'s ‘Ctrl-A c’ considered in the TTY driver
>>  to spin up another pseudo-TTY rather than suspend the long job with
>>  Ctrl-Z and ‘bg’ it?
>
>My memory is really fuzzy on this.  I recall that job control came along
>with csh which I used until bash.  At the time there were hardly any
>graphics displays used as interactive devices.  I'm talking Berkeley job
>control here; I seem to remember that someone got wedged into System V
>that was awful and unusable.
>
>Jon

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2022-12-31 12:55   ` Paul Ruizendaal
@ 2023-01-01  2:55     ` Warner Losh
  2023-01-01  4:38       ` Jonathan Gray
  0 siblings, 1 reply; 121+ messages in thread
From: Warner Losh @ 2023-01-01  2:55 UTC (permalink / raw)
  To: Paul Ruizendaal; +Cc: The Unix Historical Society, segaloco

[-- Attachment #1: Type: text/plain, Size: 8792 bytes --]

On Sat, Dec 31, 2022 at 5:55 AM Paul Ruizendaal <pnr@planet.nl> wrote:

> The "assembly code in the Bourne shell" comment is in the same
> London/Reiser paper. The full quote is:
>
> "The (Bourne) shell is the standard user command interpreter. It required
> by far the largest conversion effort of any supposedly portable program,
> for the simple reason that it is not portable. Critical portions are coded
> in assembly language and had to be painstakingly rewritten. The shell uses
> its own sbrk which is functionally different from the standard routine in
> libc. The shell wants the routine which fields a signal to be passed a
> parameter giving the number of the signal being caught; signal was also a
> private rou- tine. This was handled by having the operating system provide
> the parameter in the first place, doing away with the private code for
> signal. The code in fixargs (for constructing the argument list to an exec
> system call) had to be diddled."
>
> The files in the V7 tree on the Tuhs website are dated January 1979, so it
> would seem that the fixes for 32V were immediately taken back to Research.
> As you point out, this means that the comments above do not refer to the
> well known source code, but to a predecessor of that (which I don’t think
> survived).
>

We have ample evidence that V7 was really something more akin to a rolling
release. Let me explain: We know from the leaked '50 changes' tape that
many of the features were set earlier rather than later. This leaked in
1978 (if my notes are right), but I found references to it from as early as
November 1976 in
http://www.toad.com/early-usenix-newsletters/197611-unix-news-n11.pdf. This
was 18 months after V6 was released, but over 2 years before V7 was
released. In addition, we know from the AUUS newsletters in the archive
document that the V7 release process process took a while to get through
AT&T's legal department (IIRC a year, but I've not gone back to the AUUS
newsletters to refresh my recollection). A big push of V7 was to make it
portable as well (with AT&T doing an Interdata 8/32 port themselves, as
well as at least looking at the Wollongong Interdata 7/32 port and the
Harvard VM/370 port). In talking to Kirk and others that have been around
from approximately that time, 32V was widely viewed as V7 for Vaxen. We can
see evidence in the surviving 32V files of evolution from the 'PDP-11-like
swapping to a more sophisticated paging algorithm' since we have the
slowsys directory. It's my contention, as someone that coded in the era
before good source code control, that it's evidence that somebody got it
working, then renamed/copied it to slowsys while they got paging working so
they could build either kernel for A/B testing. Kirk has also told me that
the 32V port was started well in advance of V7's release to be both a
useful product inside of Bell Labs (since Vaxen were starting to appear) as
well as to prove that V7 was portable enough. I'll be the first to admit
this is at best conjecture that matches available facts, artifacts and old
timers recollections (sorry Kirk), but that we have no direct evidence for.
It also allowed the 3BSD efforts to get going before the official V7
release due to the close ties between Bell Labs and Berkeley and the DARPA
project around Unix.

I believe that we can conclude that the original 'hard to port' Bourne
shell was produced around the time of the 50 changes tape, give or take.
And that all the unix porting efforts that pre-dated the V7 release rolled
what appeared in 32V into V7 to reduce the amount of pdp-11 assembler. And
those efforts are what we read about in the paper.

It also goes a ways to explain the 32V meme of 'it was pdp11 swapping'
because originally, in a version we no longer have, it was. But many of the
32V tapes that we have represent a later version where that had been
abandoned in favor of what would evolve into System III's and later paging
code.


> Despite all the criticism voiced above, I think it is well understood that
> the original Bourne shell is an amazing piece of work that managed to fit
> an enormous amount of functionality into a cramped address space. Its
> longevity attests to that. That its internals became difficult to
> understand is par for the course -- the 1980’s in essence needed a Lions
> commentary on sh.
>

Totally agree with that...

Warner


>
> > On 30 Dec 2022, at 20:57, segaloco <segaloco@protonmail.com> wrote:
> >
> > I'll have to double check later but I'm fairly certain the remaining L/R
> cheats are gone by SysV.  From what I can tell much of that portability
> work may have been done prior to the V7 release code base we're familiar
> with, as I did some comparison and found only one significant change
> between V7 and 32V code as I know it at least.  Either the claims of
> portability issues came between 32V and System III (meaning the shell was
> accepted as "broken"? in 32V) or the code we actually see in V7 has already
> been tidied up significantly and doesn't represent the "non-portable"
> version lamented in the famous quote.  Does this observation hold with
> reality?  Is there an earlier, more PDP-11 bound version of the Bourne
> Shell out there?  I seem to recall reading something about some bits of it
> even being in assembly at one point, but can't remember the quote source.
> >
> > - Matt G.
> >
> > ------- Original Message -------
> > On Friday, December 30th, 2022 at 10:25 AM, Paul Ruizendaal <
> pnr@planet.nl> wrote:
> >
> >
> >> London and Reiser report about porting the shell that “it required by
> far the largest conversion effort of any supposedly portable program, for
> the simple reason that it is not portable.” By the time of SysIII this is
> greatly improved, but also in porting the SysIII user land it was the most
> complex of the set so far.
> >>
> >> There were three aspects that I found noteworthy:
> >>
> >> 1. London/Reiser apparently felt strongly about a property of casts.
> The code argues that casting an l-value should not convert it into a
> r-value:
> >>
> >> <quote from "mode.h">
> >>
> >> /* the following nonsense is required
> >> * because casts turn an Lvalue
> >> * into an Rvalue so two cheats
> >> * are necessary, one for each context.
> >> */
> >> union { int _cheat;};
> >> #define Lcheat(a) ((a)._cheat)
> >> #define Rcheat(a) ((int)(a))
> >> <endquote>
> >>
> >>
> >> However, Lcheat is only used in two places (in service.c), to set and
> to clear a flag in a pointer. Interestingly, the 32V code already replaces
> one of these instances with a regular r-value cast. So far, I’d never
> thought about this aspect of casts. I stumbled across it, because the Plan
> 9 compiler did not accept the Lcheat expansion as valid C.
> >>
> >> 2. On the history of dup2
> >>
> >> The shell code includes the following:
> >>
> >> <quote from “io.c”>
> >>
> >> rename(f1,f2)
> >> REG INT f1, f2;
> >> {
> >> #ifdef RES /* research has different sys calls from TS */
> >> IF f1!=f2
> >> THEN dup(f1|DUPFLG, f2);
> >> close(f1);
> >> IF f2==0 THEN ioset|=1 FI
> >> FI
> >> #else
> >> INT fs;
> >> IF f1!=f2
> >> THEN fs = fcntl(f2,1,0);
> >> close(f2);
> >> fcntl(f1,0,f2);
> >> close(f1);
> >> IF fs==1 THEN fcntl(f2,2,1) FI
> >> IF f2==0 THEN ioset|=1 FI
> >> FI
> >> #endif
> >> }
> >> <endquote>
> >>
> >>
> >> I’ve check the 8th edition source, and indeed it supports using DUPFLG
> to signal to dup() that it really is dup2(). I had earlier wondered why
> dup2() did not appear in research until 10th edition, but now that is
> clear. It would seem that the dup of 8th edition is a direct ancestor to
> dup() in Plan 9. I wonder why this way of doing things never caught on in
> the other Unices.
> >>
> >> 3. Halfway to demand paging
> >>
> >> I stumbled across this one because I had a bug in my signal handling.
> From early days onwards, Unix supported dynamically growing the stack
> allocation, which arguably is a first step towards building the mechanisms
> for demand paging. It appears that the Bourne shell made another step,
> catching page faults and expanding the data/bss allocation dynamically:
> >>
> >> <quote from “fault.c”>
> >>
> >> VOID fault(sig)
> >> REG INT sig;
> >> {
> >> signal(sig, fault);
> >> IF sig==MEMF
> >> THEN IF setbrk(brkincr) == -1
> >> THEN error(nospace);
> >> FI
> >> ELIF ...
> >> <endquote>
> >>
> >>
> >> This was already present in 7th edition, so it is by no means new in
> 32V or SysIII -- it had just escaped my attention as a conceptual step in
> the development of Unix memory handling.
> >>
>
>

[-- Attachment #2: Type: text/html, Size: 10349 bytes --]

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-01  2:55     ` Warner Losh
@ 2023-01-01  4:38       ` Jonathan Gray
  2023-01-01  5:25         ` Warner Losh
  0 siblings, 1 reply; 121+ messages in thread
From: Jonathan Gray @ 2023-01-01  4:38 UTC (permalink / raw)
  To: Warner Losh; +Cc: Paul Ruizendaal, tuhs, segaloco

On Sat, Dec 31, 2022 at 07:55:28PM -0700, Warner Losh wrote:
> On Sat, Dec 31, 2022 at 5:55 AM Paul Ruizendaal <pnr@planet.nl> wrote:
> 
> > The "assembly code in the Bourne shell" comment is in the same
> > London/Reiser paper. The full quote is:
> >
> > "The (Bourne) shell is the standard user command interpreter. It required
> > by far the largest conversion effort of any supposedly portable program,
> > for the simple reason that it is not portable. Critical portions are coded
> > in assembly language and had to be painstakingly rewritten. The shell uses
> > its own sbrk which is functionally different from the standard routine in
> > libc. The shell wants the routine which fields a signal to be passed a
> > parameter giving the number of the signal being caught; signal was also a
> > private rou- tine. This was handled by having the operating system provide
> > the parameter in the first place, doing away with the private code for
> > signal. The code in fixargs (for constructing the argument list to an exec
> > system call) had to be diddled."
> >
> > The files in the V7 tree on the Tuhs website are dated January 1979, so it
> > would seem that the fixes for 32V were immediately taken back to Research.
> > As you point out, this means that the comments above do not refer to the
> > well known source code, but to a predecessor of that (which I don’t think
> > survived).
> >
> 
> We have ample evidence that V7 was really something more akin to a rolling
> release. Let me explain: We know from the leaked '50 changes' tape that
> many of the features were set earlier rather than later. This leaked in
> 1978 (if my notes are right), but I found references to it from as early as
> November 1976 in
> http://www.toad.com/early-usenix-newsletters/197611-unix-news-n11.pdf. This
> was 18 months after V6 was released, but over 2 years before V7 was
> released. In addition, we know from the AUUS newsletters in the archive
> document that the V7 release process process took a while to get through
> AT&T's legal department (IIRC a year, but I've not gone back to the AUUS
> newsletters to refresh my recollection). A big push of V7 was to make it
> portable as well (with AT&T doing an Interdata 8/32 port themselves, as
> well as at least looking at the Wollongong Interdata 7/32 port and the
> Harvard VM/370 port). In talking to Kirk and others that have been around
> from approximately that time, 32V was widely viewed as V7 for Vaxen. We can
> see evidence in the surviving 32V files of evolution from the 'PDP-11-like
> swapping to a more sophisticated paging algorithm' since we have the
> slowsys directory. It's my contention, as someone that coded in the era
> before good source code control, that it's evidence that somebody got it
> working, then renamed/copied it to slowsys while they got paging working so
> they could build either kernel for A/B testing. Kirk has also told me that
> the 32V port was started well in advance of V7's release to be both a
> useful product inside of Bell Labs (since Vaxen were starting to appear) as
> well as to prove that V7 was portable enough. I'll be the first to admit
> this is at best conjecture that matches available facts, artifacts and old
> timers recollections (sorry Kirk), but that we have no direct evidence for.
> It also allowed the 3BSD efforts to get going before the official V7
> release due to the close ties between Bell Labs and Berkeley and the DARPA
> project around Unix.

"So DEC came to us at Holmdel.  We were clearly the second string.
Tom London and John Reiser were interested and so was Ken Swanson, and
we got the VAX in early '78.  I didn't do any of the technical work.  In
fact, I devoted a lot of energy and time to getting the management to
let us do it.  It wasn't research, you see.  However, they let us take
the time.  And in about three months my small group ported Version 7 to
the VAX.  We got the machine in January, they had it running in April,
and by August it really worked.
...
So, with the blessings of BTL Area 11 management, we sent 32V to
Berkeley.  It was in October or November, 1978"

Charlie Roberts in Salus QCU

"The operating system is Research version 7 as of April 15, 1978.
..
Work on the C compiler began in mid-December 1977.  The hardware arrived
on March 3.  We held a party on May 19 to celebrate successful multiuser
operation of the system."

London and Reiser
A UNIX Operating System for the DEC VAX-11/780 Computer
https://www.bell-labs.com/usr/dmr/www/otherports/32v.html

> 
> I believe that we can conclude that the original 'hard to port' Bourne
> shell was produced around the time of the 50 changes tape, give or take.
> And that all the unix porting efforts that pre-dated the V7 release rolled
> what appeared in 32V into V7 to reduce the amount of pdp-11 assembler. And
> those efforts are what we read about in the paper.

"The Bourne shell work started either in early 1976, or maybe
late 1975. The first version was VERY different"

John Mashey in net.unix-wizards, 18 Mar 86
https://groups.google.com/g/alt.folklore.computers/c/xW3ZgEnFoFs
tuhs/Documentation/AUUGN/AUUGN-V06.6.pdf pg 39

"In 1976, Steve Bourne, who had recently joined 1127, wrote a new shell"
Kernighan, UNIX A History and a Memoir, 5.1 pg 88

Bourne's AsiaBSDCon 2016 talk also lists 1976
and goes on to discuss sbrk() use causing problems with ports
https://youtu.be/7tQ2ftt3LO8?t=715

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-01  4:38       ` Jonathan Gray
@ 2023-01-01  5:25         ` Warner Losh
  2023-01-01  5:35           ` Dan Cross
                             ` (2 more replies)
  0 siblings, 3 replies; 121+ messages in thread
From: Warner Losh @ 2023-01-01  5:25 UTC (permalink / raw)
  To: Jonathan Gray; +Cc: Paul Ruizendaal, The Eunuchs Hysterical Society, segaloco

[-- Attachment #1: Type: text/plain, Size: 6434 bytes --]

On Sat, Dec 31, 2022, 9:38 PM Jonathan Gray <jsg@jsg.id.au> wrote:

> On Sat, Dec 31, 2022 at 07:55:28PM -0700, Warner Losh wrote:
> > On Sat, Dec 31, 2022 at 5:55 AM Paul Ruizendaal <pnr@planet.nl> wrote:
> >
> > > The "assembly code in the Bourne shell" comment is in the same
> > > London/Reiser paper. The full quote is:
> > >
> > > "The (Bourne) shell is the standard user command interpreter. It
> required
> > > by far the largest conversion effort of any supposedly portable
> program,
> > > for the simple reason that it is not portable. Critical portions are
> coded
> > > in assembly language and had to be painstakingly rewritten. The shell
> uses
> > > its own sbrk which is functionally different from the standard routine
> in
> > > libc. The shell wants the routine which fields a signal to be passed a
> > > parameter giving the number of the signal being caught; signal was
> also a
> > > private rou- tine. This was handled by having the operating system
> provide
> > > the parameter in the first place, doing away with the private code for
> > > signal. The code in fixargs (for constructing the argument list to an
> exec
> > > system call) had to be diddled."
> > >
> > > The files in the V7 tree on the Tuhs website are dated January 1979,
> so it
> > > would seem that the fixes for 32V were immediately taken back to
> Research.
> > > As you point out, this means that the comments above do not refer to
> the
> > > well known source code, but to a predecessor of that (which I don’t
> think
> > > survived).
> > >
> >
> > We have ample evidence that V7 was really something more akin to a
> rolling
> > release. Let me explain: We know from the leaked '50 changes' tape that
> > many of the features were set earlier rather than later. This leaked in
> > 1978 (if my notes are right), but I found references to it from as early
> as
> > November 1976 in
> > http://www.toad.com/early-usenix-newsletters/197611-unix-news-n11.pdf.
> This
> > was 18 months after V6 was released, but over 2 years before V7 was
> > released. In addition, we know from the AUUS newsletters in the archive
> > document that the V7 release process process took a while to get through
> > AT&T's legal department (IIRC a year, but I've not gone back to the AUUS
> > newsletters to refresh my recollection). A big push of V7 was to make it
> > portable as well (with AT&T doing an Interdata 8/32 port themselves, as
> > well as at least looking at the Wollongong Interdata 7/32 port and the
> > Harvard VM/370 port). In talking to Kirk and others that have been around
> > from approximately that time, 32V was widely viewed as V7 for Vaxen. We
> can
> > see evidence in the surviving 32V files of evolution from the
> 'PDP-11-like
> > swapping to a more sophisticated paging algorithm' since we have the
> > slowsys directory. It's my contention, as someone that coded in the era
> > before good source code control, that it's evidence that somebody got it
> > working, then renamed/copied it to slowsys while they got paging working
> so
> > they could build either kernel for A/B testing. Kirk has also told me
> that
> > the 32V port was started well in advance of V7's release to be both a
> > useful product inside of Bell Labs (since Vaxen were starting to appear)
> as
> > well as to prove that V7 was portable enough. I'll be the first to admit
> > this is at best conjecture that matches available facts, artifacts and
> old
> > timers recollections (sorry Kirk), but that we have no direct evidence
> for.
> > It also allowed the 3BSD efforts to get going before the official V7
> > release due to the close ties between Bell Labs and Berkeley and the
> DARPA
> > project around Unix.
>
> "So DEC came to us at Holmdel.  We were clearly the second string.
> Tom London and John Reiser were interested and so was Ken Swanson, and
> we got the VAX in early '78.  I didn't do any of the technical work.  In
> fact, I devoted a lot of energy and time to getting the management to
> let us do it.  It wasn't research, you see.  However, they let us take
> the time.  And in about three months my small group ported Version 7 to
> the VAX.  We got the machine in January, they had it running in April,
> and by August it really worked.
> ...
> So, with the blessings of BTL Area 11 management, we sent 32V to
> Berkeley.  It was in October or November, 1978"
>
> Charlie Roberts in Salus QCU
>
> "The operating system is Research version 7 as of April 15, 1978.
> ..
> Work on the C compiler began in mid-December 1977.  The hardware arrived
> on March 3.  We held a party on May 19 to celebrate successful multiuser
> operation of the system."
>
> London and Reiser
> A UNIX Operating System for the DEC VAX-11/780 Computer
> https://www.bell-labs.com/usr/dmr/www/otherports/32v.html
>
> >
> > I believe that we can conclude that the original 'hard to port' Bourne
> > shell was produced around the time of the 50 changes tape, give or take.
> > And that all the unix porting efforts that pre-dated the V7 release
> rolled
> > what appeared in 32V into V7 to reduce the amount of pdp-11 assembler.
> And
> > those efforts are what we read about in the paper.
>
> "The Bourne shell work started either in early 1976, or maybe
> late 1975. The first version was VERY different"
>
> John Mashey in net.unix-wizards, 18 Mar 86
> https://groups.google.com/g/alt.folklore.computers/c/xW3ZgEnFoFs
> tuhs/Documentation/AUUGN/AUUGN-V06.6.pdf pg 39
>
> "In 1976, Steve Bourne, who had recently joined 1127, wrote a new shell"
> Kernighan, UNIX A History and a Memoir, 5.1 pg 88
>
> Bourne's AsiaBSDCon 2016 talk also lists 1976
> and goes on to discuss sbrk() use causing problems with ports
> https://youtu.be/7tQ2ftt3LO8?t=715


And at 5:18 he says he had a vax lab with three vaxen and the Lab's vax
port didn't have virtual memory. Bill Joy with 3BSD which had virtual
memory. They installed it on the vaxen because they were hitting physical
memory limits for some of their programs....

This suggests that the 32V in the TUHS archive is a later one than this
early port. 3BSD is listed as being released end of 1979...

Still begs the question of when 32v started paging.  :)

But the time lines match up for the 32v effort to be fed back into the
research code base...

Warner

>
>

[-- Attachment #2: Type: text/html, Size: 8139 bytes --]

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-01  5:25         ` Warner Losh
@ 2023-01-01  5:35           ` Dan Cross
  2023-01-01  5:52             ` G. Branden Robinson
                               ` (2 more replies)
  2023-01-01  7:11           ` Jonathan Gray
  2023-01-01 10:25           ` Paul Ruizendaal
  2 siblings, 3 replies; 121+ messages in thread
From: Dan Cross @ 2023-01-01  5:35 UTC (permalink / raw)
  To: Warner Losh
  Cc: Jonathan Gray, Paul Ruizendaal, The Eunuchs Hysterical Society, segaloco

On Sun, Jan 1, 2023 at 12:27 AM Warner Losh <imp@bsdimp.com> wrote:
> On Sat, Dec 31, 2022, 9:38 PM Jonathan Gray <jsg@jsg.id.au> wrote:
>> [snip]
>> Bourne's AsiaBSDCon 2016 talk also lists 1976
>> and goes on to discuss sbrk() use causing problems with ports
>> https://youtu.be/7tQ2ftt3LO8?t=715
>
> And at 5:18 he says he had a vax lab with three vaxen and the Lab's vax port didn't have virtual memory. Bill Joy with 3BSD which had virtual memory. They installed it on the vaxen because they were hitting physical memory limits for some of their programs....

One wonders what is meant by "virtual memory" in this context. I
contend that Unix has had "virtual memory" since moving off of the
PDP-11/20, in the sense of having a virtual address space that was
mapped onto a (possibly contiguous) physical address space. I think
all of these references mean demand paging, possibly with page
reclamation or whole-process swapping under memory pressure.

        - Dan C.

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-01  5:35           ` Dan Cross
@ 2023-01-01  5:52             ` G. Branden Robinson
  2023-01-01  6:35               ` Warner Losh
  2023-01-01  6:35               ` Rob Pike
  2023-01-01  6:27             ` Warner Losh
  2023-01-01 14:50             ` Ron Natalie
  2 siblings, 2 replies; 121+ messages in thread
From: G. Branden Robinson @ 2023-01-01  5:52 UTC (permalink / raw)
  To: tuhs

[-- Attachment #1: Type: text/plain, Size: 2000 bytes --]

At 2023-01-01T00:35:12-0500, Dan Cross wrote:
> On Sun, Jan 1, 2023 at 12:27 AM Warner Losh <imp@bsdimp.com> wrote:
> > On Sat, Dec 31, 2022, 9:38 PM Jonathan Gray <jsg@jsg.id.au> wrote:
> >> [snip]
> >> Bourne's AsiaBSDCon 2016 talk also lists 1976
> >> and goes on to discuss sbrk() use causing problems with ports
> >> https://youtu.be/7tQ2ftt3LO8?t=715
> >
> > And at 5:18 he says he had a vax lab with three vaxen and the Lab's
> > vax port didn't have virtual memory. Bill Joy with 3BSD which had
> > virtual memory. They installed it on the vaxen because they were
> > hitting physical memory limits for some of their programs....
> 
> One wonders what is meant by "virtual memory" in this context. I
> contend that Unix has had "virtual memory" since moving off of the
> PDP-11/20, in the sense of having a virtual address space that was
> mapped onto a (possibly contiguous) physical address space. I think
> all of these references mean demand paging, possibly with page
> reclamation or whole-process swapping under memory pressure.

I apologize if this point is too elementary, but I speculate that one
possible source of confusion comes from a file naming convention: which
of these (multiple) virtual memory or demand-paged VM systems installed
the kernel under the name "vmunix" vs. "unix".

Which ones did and did not?

When I was first learning Unix I asked a local expert why the kernel was
named "vmunix".  They told me that it was because it supported virtual
memory (and explained what that was, because I was even more callow then
than now).

Then I asked where the non-VM kernel was.  I was informed that there
wasn't one--it didn't even exist for modern architectures.  I wondered
then why, if virtual memory was a given, you wouldn't just go back to
using the filename "unix".

I wondered similar things when encountering the "vmlinux" file a couple
of years later.

Reflexive obeisance to traditions has a cost.

Regards,
Branden

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-01  5:35           ` Dan Cross
  2023-01-01  5:52             ` G. Branden Robinson
@ 2023-01-01  6:27             ` Warner Losh
  2023-01-01 14:50             ` Ron Natalie
  2 siblings, 0 replies; 121+ messages in thread
From: Warner Losh @ 2023-01-01  6:27 UTC (permalink / raw)
  To: Dan Cross
  Cc: Jonathan Gray, Paul Ruizendaal, The Eunuchs Hysterical Society, segaloco

[-- Attachment #1: Type: text/plain, Size: 1178 bytes --]

On Sat, Dec 31, 2022, 10:35 PM Dan Cross <crossd@gmail.com> wrote:

> On Sun, Jan 1, 2023 at 12:27 AM Warner Losh <imp@bsdimp.com> wrote:
> > On Sat, Dec 31, 2022, 9:38 PM Jonathan Gray <jsg@jsg.id.au> wrote:
> >> [snip]
> >> Bourne's AsiaBSDCon 2016 talk also lists 1976
> >> and goes on to discuss sbrk() use causing problems with ports
> >> https://youtu.be/7tQ2ftt3LO8?t=715
> >
> > And at 5:18 he says he had a vax lab with three vaxen and the Lab's vax
> port didn't have virtual memory. Bill Joy with 3BSD which had virtual
> memory. They installed it on the vaxen because they were hitting physical
> memory limits for some of their programs....
>
> One wonders what is meant by "virtual memory" in this context. I
> contend that Unix has had "virtual memory" since moving off of the
> PDP-11/20, in the sense of having a virtual address space that was
> mapped onto a (possibly contiguous) physical address space. I think
> all of these references mean demand paging, possibly with page
> reclamation or whole-process swapping under memory pressure.
>

It's clear from context it's demand paging and that unlocks processes
larger than physical memory. Imho.

Warner

>

[-- Attachment #2: Type: text/html, Size: 2063 bytes --]

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-01  5:52             ` G. Branden Robinson
@ 2023-01-01  6:35               ` Warner Losh
  2023-01-01  6:35               ` Rob Pike
  1 sibling, 0 replies; 121+ messages in thread
From: Warner Losh @ 2023-01-01  6:35 UTC (permalink / raw)
  To: G. Branden Robinson; +Cc: The Eunuchs Hysterical Society

[-- Attachment #1: Type: text/plain, Size: 2456 bytes --]

On Sat, Dec 31, 2022, 10:53 PM G. Branden Robinson <
g.branden.robinson@gmail.com> wrote:

> At 2023-01-01T00:35:12-0500, Dan Cross wrote:
> > On Sun, Jan 1, 2023 at 12:27 AM Warner Losh <imp@bsdimp.com> wrote:
> > > On Sat, Dec 31, 2022, 9:38 PM Jonathan Gray <jsg@jsg.id.au> wrote:
> > >> [snip]
> > >> Bourne's AsiaBSDCon 2016 talk also lists 1976
> > >> and goes on to discuss sbrk() use causing problems with ports
> > >> https://youtu.be/7tQ2ftt3LO8?t=715
> > >
> > > And at 5:18 he says he had a vax lab with three vaxen and the Lab's
> > > vax port didn't have virtual memory. Bill Joy with 3BSD which had
> > > virtual memory. They installed it on the vaxen because they were
> > > hitting physical memory limits for some of their programs....
> >
> > One wonders what is meant by "virtual memory" in this context. I
> > contend that Unix has had "virtual memory" since moving off of the
> > PDP-11/20, in the sense of having a virtual address space that was
> > mapped onto a (possibly contiguous) physical address space. I think
> > all of these references mean demand paging, possibly with page
> > reclamation or whole-process swapping under memory pressure.
>
> I apologize if this point is too elementary, but I speculate that one
> possible source of confusion comes from a file naming convention: which
> of these (multiple) virtual memory or demand-paged VM systems installed
> the kernel under the name "vmunix" vs. "unix".
>
> Which ones did and did not?
>

3BSD started the vmunix name because it added virtual memory to unix
(paging aspect of virtual memory).

When I was first learning Unix I asked a local expert why the kernel was
> named "vmunix".  They told me that it was because it supported virtual
> memory (and explained what that was, because I was even more callow then
> than now).
>
> Then I asked where the non-VM kernel was.  I was informed that there
> wasn't one--it didn't even exist for modern architectures.  I wondered
> then why, if virtual memory was a given, you wouldn't just go back to
> using the filename "unix".
>

V7 had "unix" for sure. 32V still had unix if my quick grep is right.

I wondered similar things when encountering the "vmlinux" file a couple
> of years later.
>

It was emulation of BSD and/or SunOS...

Reflexive obeisance to traditions has a cost.
>

Yea... There was a thread in FreeBSD about what to name the kernel.  Vmduck
was suggested as a joke...

Warner

> Regards,
> Branden
>

[-- Attachment #2: Type: text/html, Size: 4284 bytes --]

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-01  5:52             ` G. Branden Robinson
  2023-01-01  6:35               ` Warner Losh
@ 2023-01-01  6:35               ` Rob Pike
  1 sibling, 0 replies; 121+ messages in thread
From: Rob Pike @ 2023-01-01  6:35 UTC (permalink / raw)
  To: G. Branden Robinson; +Cc: tuhs

[-- Attachment #1: Type: text/plain, Size: 2336 bytes --]

I distinctly remember I renamed to boot image on our research VAX (alice!)
from "vmunix" to "unix". It had been long enough by that point that the
lingering prefix was starting to sound like a brag.

-rob


On Sun, Jan 1, 2023 at 4:53 PM G. Branden Robinson <
g.branden.robinson@gmail.com> wrote:

> At 2023-01-01T00:35:12-0500, Dan Cross wrote:
> > On Sun, Jan 1, 2023 at 12:27 AM Warner Losh <imp@bsdimp.com> wrote:
> > > On Sat, Dec 31, 2022, 9:38 PM Jonathan Gray <jsg@jsg.id.au> wrote:
> > >> [snip]
> > >> Bourne's AsiaBSDCon 2016 talk also lists 1976
> > >> and goes on to discuss sbrk() use causing problems with ports
> > >> https://youtu.be/7tQ2ftt3LO8?t=715
> > >
> > > And at 5:18 he says he had a vax lab with three vaxen and the Lab's
> > > vax port didn't have virtual memory. Bill Joy with 3BSD which had
> > > virtual memory. They installed it on the vaxen because they were
> > > hitting physical memory limits for some of their programs....
> >
> > One wonders what is meant by "virtual memory" in this context. I
> > contend that Unix has had "virtual memory" since moving off of the
> > PDP-11/20, in the sense of having a virtual address space that was
> > mapped onto a (possibly contiguous) physical address space. I think
> > all of these references mean demand paging, possibly with page
> > reclamation or whole-process swapping under memory pressure.
>
> I apologize if this point is too elementary, but I speculate that one
> possible source of confusion comes from a file naming convention: which
> of these (multiple) virtual memory or demand-paged VM systems installed
> the kernel under the name "vmunix" vs. "unix".
>
> Which ones did and did not?
>
> When I was first learning Unix I asked a local expert why the kernel was
> named "vmunix".  They told me that it was because it supported virtual
> memory (and explained what that was, because I was even more callow then
> than now).
>
> Then I asked where the non-VM kernel was.  I was informed that there
> wasn't one--it didn't even exist for modern architectures.  I wondered
> then why, if virtual memory was a given, you wouldn't just go back to
> using the filename "unix".
>
> I wondered similar things when encountering the "vmlinux" file a couple
> of years later.
>
> Reflexive obeisance to traditions has a cost.
>
> Regards,
> Branden
>

[-- Attachment #2: Type: text/html, Size: 3456 bytes --]

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-01  5:25         ` Warner Losh
  2023-01-01  5:35           ` Dan Cross
@ 2023-01-01  7:11           ` Jonathan Gray
  2023-01-01  7:21             ` Warner Losh
  2023-01-01 10:25           ` Paul Ruizendaal
  2 siblings, 1 reply; 121+ messages in thread
From: Jonathan Gray @ 2023-01-01  7:11 UTC (permalink / raw)
  To: Warner Losh; +Cc: Paul Ruizendaal, tuhs, segaloco

On Sat, Dec 31, 2022 at 10:25:49PM -0700, Warner Losh wrote:
> And at 5:18 he says he had a vax lab with three vaxen and the Lab's vax
> port didn't have virtual memory. Bill Joy with 3BSD which had virtual
> memory. They installed it on the vaxen because they were hitting physical
> memory limits for some of their programs....
> 
> This suggests that the 32V in the TUHS archive is a later one than this
> early port. 3BSD is listed as being released end of 1979...

It was 4BSD and 1980 according to QCU:

"In 1980 it was Bill Joy who flew to New Jersey and, together with Steve
Bourne, put 4BSD up in Bell Labs. "We did it overnight," Joy told me.
"We decided to do it about 9 p.m.; people showed up in the morning and a
new os was running. It was pretty amazing."

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-01  7:11           ` Jonathan Gray
@ 2023-01-01  7:21             ` Warner Losh
  0 siblings, 0 replies; 121+ messages in thread
From: Warner Losh @ 2023-01-01  7:21 UTC (permalink / raw)
  To: Jonathan Gray; +Cc: Paul Ruizendaal, The Eunuchs Hysterical Society, segaloco

[-- Attachment #1: Type: text/plain, Size: 924 bytes --]

On Sun, Jan 1, 2023, 12:11 AM Jonathan Gray <jsg@jsg.id.au> wrote:

> On Sat, Dec 31, 2022 at 10:25:49PM -0700, Warner Losh wrote:
> > And at 5:18 he says he had a vax lab with three vaxen and the Lab's vax
> > port didn't have virtual memory. Bill Joy with 3BSD which had virtual
> > memory. They installed it on the vaxen because they were hitting physical
> > memory limits for some of their programs....
> >
> > This suggests that the 32V in the TUHS archive is a later one than this
> > early port. 3BSD is listed as being released end of 1979...
>
> It was 4BSD and 1980 according to QCU:
>
> "In 1980 it was Bill Joy who flew to New Jersey and, together with Steve
> Bourne, put 4BSD up in Bell Labs. "We did it overnight," Joy told me.
> "We decided to do it about 9 p.m.; people showed up in the morning and a
> new os was running. It was pretty amazing."
>

He said 3BSD in the video.... Interesting...

Warner

>

[-- Attachment #2: Type: text/html, Size: 1553 bytes --]

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-01  5:25         ` Warner Losh
  2023-01-01  5:35           ` Dan Cross
  2023-01-01  7:11           ` Jonathan Gray
@ 2023-01-01 10:25           ` Paul Ruizendaal
  2 siblings, 0 replies; 121+ messages in thread
From: Paul Ruizendaal @ 2023-01-01 10:25 UTC (permalink / raw)
  To: Warner Losh; +Cc: The Eunuchs Hysterical Society


> On 1 Jan 2023, at 06:25, Warner Losh <imp@bsdimp.com> wrote:
> 
> Still begs the question of when 32v started paging.  :)

The London/Reiser paper includes:

"Work on the C compiler began in mid-December 1977. The hardware arrived on March 3. We held a party on May 19 to celebrate successful multiuser operation of the system."

John Reiser had the following timeline in a mail conversation about a year ago.

"Our VAX-11/780, the first one delivered to a customer, arrived Feb.12, 1978
(Lincoln's Birthday).  Tom and I had been preparing using PDP-11/45 since
December, and we achieved "login: " on the console DECwriter by April 15
(the deadline for US income tax filing).  The rest of 1978 was "tuning",
and preparing for the release of "UNIX-32/V" to UC Berkeley.”

The dates don’t match to the day, but the timeline is essentially the same (moreover, delivery of VAX hardware is not the same as it being operational, achieving login: is but the first step of bringing the user land up).

This sets a minimum timeline for arriving at a first version of “slowsys”. The timestamps on the surviving files of slowsys show January 1979.

Joy and Babaoglu write in “Converting a Swap-Based System to do Paging in an Architecture Lacking Page-Referenced Bits”:

"In the fall of 1978 the Computer Science Division of the University of California at Berkeley purchased a VAX-11/780 and arranged to run an early version of UNIX for the VAX provided by Bell Laboratories under a cooperative research agreement. [...] A subsequent version of the system was capable of loading processes into noncontiguous real memory locations, called scatter loading , and was able to swap only portions of a process, called partial swapping, as deemed necessary by the memory contention. This would become the basis for the paging system development discussed in this paper.”

This places the earliest moment for scatter paging as the Fall of 1978. The timestamps of the surviving files of sys show March 1979, which is the latest moment.

The for the timeline of 32V with true demand paging John Reiser recalled:

"My annual performance review in early 1979 said "Well done; but don't ever do
it again” because it was not regarded as "Research”. So what did I do?  I did
it again; this time, with demand paging. I designed and implemented mmap() based
on experience with PDP-10/Tenex PMAP page mapping system call. I fretted over
introducing the first UNIX system call with 6 arguments.”

John recalls that it was a 6 or 7 months effort, with the last few being a "long tail" of decreasing attention. The would place this third version as completed around the Fall of 1979. This is consistent with Rob Pike’s recollection that he saw it being demonstrated early in 1980.

In 1980 Research had to chose how to progress in the field of memory management. The Reiser system was cool, but not supported by management and the BSD VM system was seen as good enough to use. 3BSD is from early 1980, 4BSD from late 1980. Likely the conflicting recollections of Bill Joy are because it was a snapshot somewhere in between (plus the general issues around recollection). In any case, the 1985 snapshot of 8th edition appears to have been based on 4.1BSD (early 1981), so that is when the systems split.






^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2022-12-30 19:51 ` [TUHS] " Chet Ramey
  2022-12-30 20:02   ` Larry McVoy
  2022-12-30 20:20   ` Sven Mascheck
@ 2023-01-01 10:44   ` arnold
  2023-01-01 11:28     ` arnold
  2023-01-03 15:06     ` Chet Ramey
  2 siblings, 2 replies; 121+ messages in thread
From: arnold @ 2023-01-01 10:44 UTC (permalink / raw)
  To: tuhs, pnr, chet.ramey

Chet Ramey <chet.ramey@case.edu> wrote:

> On 12/30/22 1:25 PM, Paul Ruizendaal wrote:
> > 
> > London and Reiser report about porting the shell that “it required by
> > far the largest conversion effort of any supposedly portable program,
> > for the simple reason that it is not portable.” By the time of SysIII
> > this is greatly improved, but also in porting the SysIII user land it
> > was the most complex of the set so far.
>
> Have you read
>
> http://www.collyer.net/who/geoff/sh.tour.pdf
>
> and looked at http://www.collyer.net/who/geoff/v7sh.tar ?
>
> In the limited literature on Bourne Shell porting, this is authoritative.
>
> Arnold Robbins built on that work and ported the v8-v10 shells to modern
> Linux versions. (I am sorry, I do not have a link right now.)

Sorry to say, it wasn't me. Geoff Collyer made the v9 sh portable
and it's available from his web site.

I did do stuff with the shell back in the 80s:

- I back-ported Ron Natalie's job control into the 4.2 BSD sh,
  along with a history mechanism I wrote from scratch that was
  inspired by csh. I posted the diffs in comp.sources.unix.

- A few very minor features were added to ksh at my suggestion.
  ISTR that one was `set -A array-name value ...' but I don't see that
  in current ksh93 doc.

- I was a reviewer for both editions of David Korn's book on ksh.

- I contributed to pdksh, although I don't remember what. The pdksh
  doc lists my name though.

- In the early 90s I did a little banging on the vi mode in the
  readline librar so that bash's vi mode would be closer to what
  I'd been used to in ksh.

In the late 80s I got interested in awk and moved almost all my
personal scripting to awk, although I still do shell work at $DAYJOB
or for personal use as needed.

Arnold

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-01 10:44   ` arnold
@ 2023-01-01 11:28     ` arnold
  2023-01-03 16:34       ` Chet Ramey
  2023-01-03 15:06     ` Chet Ramey
  1 sibling, 1 reply; 121+ messages in thread
From: arnold @ 2023-01-01 11:28 UTC (permalink / raw)
  To: tuhs, pnr, chet.ramey, arnold

I wrote:

> I did do stuff with the shell back in the 80s:

I also recall that there was some USENET discussion in which
Rob Pike described how the 8th edition shell changed the search
order to have functions come before non-special built-ins, with
the addition of "builtin" and "whatis" commands.

Rob was kind enough to send me the sh.1 man page from the 8th edition
(which I still have somewhere), and I made this change in the S5r2
shell. I don't know if I posted diffs or not.

In any case, as part of implementing whatis, I wrote a pretty printer
for shell functions. (Getting 'elif' right needed some extra code.) I
offered it to Rob, who politely declined, indicating that he had his own.

Ah, those were the days... :-)

Arnold

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-01  5:35           ` Dan Cross
  2023-01-01  5:52             ` G. Branden Robinson
  2023-01-01  6:27             ` Warner Losh
@ 2023-01-01 14:50             ` Ron Natalie
  2 siblings, 0 replies; 121+ messages in thread
From: Ron Natalie @ 2023-01-01 14:50 UTC (permalink / raw)
  To: Dan Cross, Warner Losh
  Cc: Jonathan Gray, Paul Ruizendaal, The Eunuchs Hysterical Society, segaloco

Paging was what they were referring to.   The Bell releases at the time 
only swapped like they did on the PDP-11.
BSD allowed only fragments of the program to need be resident.

The joke as “It ain’t virtual unless it isn’t all there."


------ Original Message ------
From "Dan Cross" <crossd@gmail.com>
To "Warner Losh" <imp@bsdimp.com>
Cc "Jonathan Gray" <jsg@jsg.id.au>; "Paul Ruizendaal" <pnr@planet.nl>; 
"The Eunuchs Hysterical Society" <tuhs@tuhs.org>; "segaloco" 
<segaloco@protonmail.com>
Date 1/1/2023 12:35:12 AM
Subject [TUHS] Re: A few comments on porting the Bourne shell

>On Sun, Jan 1, 2023 at 12:27 AM Warner Losh <imp@bsdimp.com> wrote:
>>  On Sat, Dec 31, 2022, 9:38 PM Jonathan Gray <jsg@jsg.id.au> wrote:
>>>  [snip]
>>>  Bourne's AsiaBSDCon 2016 talk also lists 1976
>>>  and goes on to discuss sbrk() use causing problems with ports
>>>https://youtu.be/7tQ2ftt3LO8?t=715
>>
>>  And at 5:18 he says he had a vax lab with three vaxen and the Lab's vax port didn't have virtual memory. Bill Joy with 3BSD which had virtual memory. They installed it on the vaxen because they were hitting physical memory limits for some of their programs....
>
>One wonders what is meant by "virtual memory" in this context. I
>contend that Unix has had "virtual memory" since moving off of the
>PDP-11/20, in the sense of having a virtual address space that was
>mapped onto a (possibly contiguous) physical address space. I think
>all of these references mean demand paging, possibly with page
>reclamation or whole-process swapping under memory pressure.
>
>         - Dan C.

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2022-12-31  4:23         ` Dave Horsfall
  2022-12-31  4:37           ` Clem Cole
@ 2023-01-02  5:10           ` Mary Ann Horton
  2023-01-02 16:25             ` Clem Cole
  1 sibling, 1 reply; 121+ messages in thread
From: Mary Ann Horton @ 2023-01-02  5:10 UTC (permalink / raw)
  To: tuhs

[-- Attachment #1: Type: text/plain, Size: 612 bytes --]

On 12/30/22 20:23, Dave Horsfall wrote:
> Yep; whoever wrote CSH must've been high on something, as the syntax makes
> no sense whatsoever.

Bill Joy wrote csh. He based it on the V6 shell.  It made perfect sense 
at the time. This was before the V7 Bourne shell came out.

Thanks,

/Mary Ann Horton/ (she/her/ma'am)
maryannhorton.com <https://maryannhorton.com>

"This is a great book" - Monica Helms

"Brave and Important" - Laura L. Engel

       Available on Amazon and bn.com!

	<https://www.amazon.com/Trailblazer-Lighting-Transgender-Equality-Corporate-ebook/dp/B0B8F2BR9B>




[-- Attachment #2: Type: text/html, Size: 1805 bytes --]

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-02  5:10           ` Mary Ann Horton
@ 2023-01-02 16:25             ` Clem Cole
  2023-01-02 16:51               ` Larry McVoy
  0 siblings, 1 reply; 121+ messages in thread
From: Clem Cole @ 2023-01-02 16:25 UTC (permalink / raw)
  To: Mary Ann Horton; +Cc: tuhs

[-- Attachment #1: Type: text/plain, Size: 3158 bytes --]

Be careful in applying modern judgments without remembering
the historical context - below
On Mon, Jan 2, 2023 at 12:12 AM Mary Ann Horton <mah@mhorton.net> wrote:

> On 12/30/22 20:23, Dave Horsfall wrote:
>
> Yep; whoever wrote CSH must've been high on something, as the syntax makes
> no sense whatsoever.
>
> Bill Joy wrote csh. He based it on the V6 shell.  It made perfect sense at
> the time. This was before the V7 Bourne shell came out.
>
Right -- let me add a little detail to this that might help. With the
Fourth, Fifth, and Sixth Edition the default shell from Research is the
Thompson shell.  It's simple and un-adorn.   Get the job done for typing,
but as a formal scripting (programming) language is not easy to use.

In fact, the command 'cd' idiom is still 'chdir' [add it was a typical
local hack that many of did].  If you look at the USENIX tapes in the TUHS
archives, in particular, the Harvard tape I suspect you will find a number
of alternative shells that people had offered up from the different sites.

With 'BSD' (the first UNIX distribution tape from the UCB Industrial
Liasion Office of EECS which we often refer to on this list as 1BSD to
distinguish it from the other releases - also available on TUHS], there is
a new Berkeley shell, that Mary Ann points out Bill had created by
modifying the Thompson shell and added some features to it. By the time of
the 2BSD, the shell has been further been refined and by then renamed the
csh, as the syntax had been made to approximate the C language.

A big difference between the *Thompson Shell begets -> Berkeley Shell ->
begets C Shell *and Bourne's work on his shell, is that Steve started from
scratch. Steve had just come off many years of the Algol 68 project and was
certainly extremely well-versed in the detailed subtleties of the semantics
of language design.  His higher bit was to create something *easier for
people to write scripts using the -> small is beautiful -> Unix philosophy
ideas*.

Bill, on the other hand, is the quintessential hacker (My line is him in
those days used to be: "He types open curly brace, close curly brace and he
patches at 9600 baud.")  Bill was interested in a shell that made typing
easy, was not a load the system, and made it easy for >>him<< to write
simple admin scripts.

I would suggest both were 100% successful at what their purpose was.
Like Larry, I have always been a meany and I have tried to make darned sure
my folks (even to this day - we will see how long that last after I retire)
use V7 syntax scripts - not bash, not even korn.   As I said, the old UNIX
hacker wisdom, has been "*Bourne to Program, Type with **Joy*" and you have
a hard time going wrong.

[FYI - while I have tcsh around, on my Mac these days I finally switched
from tcsh to zsh as my default, but pretty much all of Bill's typing tricks
are supported in zsh so the 'ROMs" in my fingers are happy after >40
years"] but all of my scripts are still V7 syntax - never been a reason to
change that -- bigger issues usually found are GNU/Linux is not UNIX and
Apple had to pee on things also.
ᐧ

[-- Attachment #2: Type: text/html, Size: 5916 bytes --]

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-02 16:25             ` Clem Cole
@ 2023-01-02 16:51               ` Larry McVoy
  2023-01-02 17:32                 ` Adam Thornton
  2023-01-02 17:55                 ` Clem Cole
  0 siblings, 2 replies; 121+ messages in thread
From: Larry McVoy @ 2023-01-02 16:51 UTC (permalink / raw)
  To: Clem Cole; +Cc: tuhs

On Mon, Jan 02, 2023 at 11:25:06AM -0500, Clem Cole wrote:
> Like Larry, I have always been a meany and I have tried to make darned sure
> my folks (even to this day - we will see how long that last after I retire)
> use V7 syntax scripts - not bash, not even korn.   

I think it is less of an issue today but if I were still supporting a 
multi platform product, I'd still insist on it.  Ya never know when a
big pile of money is going to show and insist that you work on SCO
based cash registers or something.   Every.  Single.  Time. one of 
my engineers whined that they liked bash better, we'd hit on some
supported platform where /bin/sh is not bash.  And as we shipped
scripts that people expected to work, you can't fix their /bin/sh.

I won that argument over and over and eventually we all instinctively
wrote portable scripts.

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-02 16:51               ` Larry McVoy
@ 2023-01-02 17:32                 ` Adam Thornton
  2023-01-02 17:43                   ` Larry McVoy
  2023-01-02 17:55                 ` Clem Cole
  1 sibling, 1 reply; 121+ messages in thread
From: Adam Thornton @ 2023-01-02 17:32 UTC (permalink / raw)
  To: Larry McVoy; +Cc: The Eunuchs Hysterical Society



> On Jan 2, 2023, at 9:51 AM, Larry McVoy <lm@mcvoy.com> wrote:
> 
> On Mon, Jan 02, 2023 at 11:25:06AM -0500, Clem Cole wrote:
>> Like Larry, I have always been a meany and I have tried to make darned sure
>> my folks (even to this day - we will see how long that last after I retire)
>> use V7 syntax scripts - not bash, not even korn.   
> And as we shipped
> scripts that people expected to work, you can't fix their /bin/sh.
> 
> I won that argument over and over and eventually we all instinctively
> wrote portable scripts.


I feel like after POSIX.2 had had a few years to settle, it was OK to use *that* as your scripting language (I particularly like $() rather than backticks, because of nestability, although you can certainly argue that if you're regularly nesting your interpolations, it's the design that's broken, not the specific syntax of the command).

At least as long as you put a very visible comment right at the top saying "We're assuming POSIX shell."

But of course the "after it had had a few years to settle" is doing a lot of lifting there.  During the Unix Wars, yeah, clearly using v7 sh as lowest-common-denominator was the right choice.  After the establishment of Linux Hegemony sometime in the early 2000s, I think I remember that you could pretty much count on everyone at least providing a POSIX shell, even if it wasn't the default.  (But of course by then, "everyone" that wasn't Linux or OS X was already down to more-or-less Sun, IBM, HP, and DEC/Compaq.)

These days, if you have a vendor-supplied shell script that DOESN'T work correctly under a POSIX /bin/sh, that should be treated as a defect.

Adam

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-02 17:32                 ` Adam Thornton
@ 2023-01-02 17:43                   ` Larry McVoy
  2023-01-02 17:48                     ` Luther Johnson
                                       ` (2 more replies)
  0 siblings, 3 replies; 121+ messages in thread
From: Larry McVoy @ 2023-01-02 17:43 UTC (permalink / raw)
  To: Adam Thornton; +Cc: The Eunuchs Hysterical Society

On Mon, Jan 02, 2023 at 10:32:50AM -0700, Adam Thornton wrote:
> But of course the "after it had had a few years to settle" is doing
> a lot of lifting there.  During the Unix Wars, yeah, clearly using
> v7 sh as lowest-common-denominator was the right choice.  After the
> establishment of Linux Hegemony sometime in the early 2000s, I think I
> remember that you could pretty much count on everyone at least providing
> a POSIX shell, even if it wasn't the default.  (But of course by then,
> "everyone" that wasn't Linux or OS X was already down to more-or-less Sun,
> IBM, HP, and DEC/Compaq.)

I was supporting a commercial product in the early 2000's and there
were all sorts of systems then that had old shells.  Yes, you could make
everything a 2 level thing where the first level finds the correct shell,
but that's just fuss.  Just make things portable, it's not that hard
and it works everywhere.

When you get to the commercial world, you'd be stunned to see how long
old machines last.  If they are solving some problem, and they aren't
broken, nobody replaces them.  I'll bet you anything there are still
SCO registers out there, I'll bet there are still PDP-11s out there.

If it ain't broke, don't fix it.

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-02 17:43                   ` Larry McVoy
@ 2023-01-02 17:48                     ` Luther Johnson
  2023-01-02 18:00                       ` G. Branden Robinson
  2023-01-02 17:55                     ` Adam Thornton
  2023-01-04  3:20                     ` John Cowan
  2 siblings, 1 reply; 121+ messages in thread
From: Luther Johnson @ 2023-01-02 17:48 UTC (permalink / raw)
  To: tuhs

Arnold Schoenberg said of his 12-tone method of music composition, "in 
constraints I find freedom". The best thing about committing to a simple 
style and a small set of tools that span the problem space, is the 
effect it has on my program composition, imho.

On 01/02/2023 10:43 AM, Larry McVoy wrote:
> On Mon, Jan 02, 2023 at 10:32:50AM -0700, Adam Thornton wrote:
>> But of course the "after it had had a few years to settle" is doing
>> a lot of lifting there.  During the Unix Wars, yeah, clearly using
>> v7 sh as lowest-common-denominator was the right choice.  After the
>> establishment of Linux Hegemony sometime in the early 2000s, I think I
>> remember that you could pretty much count on everyone at least providing
>> a POSIX shell, even if it wasn't the default.  (But of course by then,
>> "everyone" that wasn't Linux or OS X was already down to more-or-less Sun,
>> IBM, HP, and DEC/Compaq.)
> I was supporting a commercial product in the early 2000's and there
> were all sorts of systems then that had old shells.  Yes, you could make
> everything a 2 level thing where the first level finds the correct shell,
> but that's just fuss.  Just make things portable, it's not that hard
> and it works everywhere.
>
> When you get to the commercial world, you'd be stunned to see how long
> old machines last.  If they are solving some problem, and they aren't
> broken, nobody replaces them.  I'll bet you anything there are still
> SCO registers out there, I'll bet there are still PDP-11s out there.
>
> If it ain't broke, don't fix it.
>


^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-02 16:51               ` Larry McVoy
  2023-01-02 17:32                 ` Adam Thornton
@ 2023-01-02 17:55                 ` Clem Cole
  2023-01-03 17:08                   ` Paul Winalski
  1 sibling, 1 reply; 121+ messages in thread
From: Clem Cole @ 2023-01-02 17:55 UTC (permalink / raw)
  To: Larry McVoy; +Cc: tuhs

[-- Attachment #1: Type: text/plain, Size: 1641 bytes --]

On Mon, Jan 2, 2023 at 11:51 AM Larry McVoy <lm@mcvoy.com> wrote:

> I think it is less of an issue today but if I were still supporting a multi
> platform product, I'd still insist on it.
>
Yep, and it's so easy to do.  It's just a discipline.  @Adam I helped
create POSIX, so I get it.
But Larry nailed it.  It is just cheaper and easier to be disciplined and
stick with Bourne for your scripts.  *Teach your people the skills and you
save time and money in the long run.*   It's that simple.  As Larry says,
you never know and the problem is - when it happens, it tends to happen on
a short leash.  If you have been disciplined, it's a non-problem.  It's
really not that hard to use the V7 syntax.  Everything you want/need to do
is there.

BTW: At Intel, a couple of years back (less than 3-5 years ago)  we had a
site where we needed things to work on a specific target that was, shall we
say 'a bit custom' - V7 syntax was just fine for the installer - boy folks
were happy a few of us had been on their case to get rid of the bashism the
Millenials had tried to add (I'm not really sure POSIX.2 would have been
good enough -- maybe - but Bourne was fine].

FWIW: In my start-up times, under the same rules of being disciplined, as
VP of Engineering, I insisted, all C and C++ code was required to
'flex-e-lint' warning clean.   I gave my folks a 3-week week slip to clean
everything up.   I was cursed during that time.   But guess what, the
outstanding bug list dropped to ⅒ of what it had been.  Created quite a few
true believers.  And we made those 3 weeks back before we were done.

Clem


ᐧ

[-- Attachment #2: Type: text/html, Size: 3104 bytes --]

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-02 17:43                   ` Larry McVoy
  2023-01-02 17:48                     ` Luther Johnson
@ 2023-01-02 17:55                     ` Adam Thornton
  2023-01-02 18:11                       ` Clem Cole
  2023-01-02 18:18                       ` Larry McVoy
  2023-01-04  3:20                     ` John Cowan
  2 siblings, 2 replies; 121+ messages in thread
From: Adam Thornton @ 2023-01-02 17:55 UTC (permalink / raw)
  To: Larry McVoy; +Cc: The Eunuchs Hysterical Society



> On Jan 2, 2023, at 10:43 AM, Larry McVoy <lm@mcvoy.com> wrote:
> 
> When you get to the commercial world, you'd be stunned to see how long
> old machines last.  If they are solving some problem, and they aren't
> broken, nobody replaces them.  I'll bet you anything there are still
> SCO registers out there, I'll bet there are still PDP-11s out there.
> 
> If it ain't broke, don't fix it.

Well, sure, but this also comes down to at what point is losing some customers because they refuse to upgrade to something from this millenium worth the cost you would incur to be able to support their environment?

To be fair, stuff exactly like that was part of the bread-and-butter of Sine Nomine while I was there (for all I know, still is), and I kinda feel like that's the right answer: the places running old technology know they're doing so, and maybe they're doing so for good reasons.  But finding people willing and able to keep new stuff working on those machines and OSes does get more expensive over time.  So there's an aftermarket in consultancies that are willing to take over maintenance of things their vendors have abandoned.  No company concerned with its long-term survival *should* do that without, in parallel, pursuing an exit strategy towards something actually-maintainable at not-exponentially-increasing cost, but of course the inability to look beyond the next quarter is a fundamental flaw of financialization-driven capitalism.

And no one is saying you can't keep running your SCO cash registers forever on your own recognizance, but when they break, you get to keep both pieces, and if they break in such a way that suddenly your entire operation is being held hostage by some Russian organized crime cartel, well, you should have priced that risk appropriately.

Adam

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-02 17:48                     ` Luther Johnson
@ 2023-01-02 18:00                       ` G. Branden Robinson
  2023-01-02 18:05                         ` Luther Johnson
  2023-01-02 18:12                         ` Larry McVoy
  0 siblings, 2 replies; 121+ messages in thread
From: G. Branden Robinson @ 2023-01-02 18:00 UTC (permalink / raw)
  To: tuhs

[-- Attachment #1: Type: text/plain, Size: 2136 bytes --]

At 2023-01-02T10:48:32-0700, Luther Johnson wrote:
> Arnold Schoenberg said of his 12-tone method of music composition, "in
> constraints I find freedom". The best thing about committing to a
> simple style and a small set of tools that span the problem space, is
> the effect it has on my program composition, imho.

Schoenberg didn't have to compose for orchestras such that the Berlin
Philharmonic string section tuned half of the octave meantone with the
other half equal-tempered, whereas the Royal Concertgebouw had the
brasses play an un-notated ascending major third gliss anytime you
approached the tonic from a tritone _below_ (only).

> On 01/02/2023 10:43 AM, Larry McVoy wrote:
> > I was supporting a commercial product in the early 2000's and there
> > were all sorts of systems then that had old shells.  Yes, you could
> > make everything a 2 level thing where the first level finds the
> > correct shell, but that's just fuss.  Just make things portable,
> > it's not that hard and it works everywhere.
> > 
> > When you get to the commercial world, you'd be stunned to see how
> > long old machines last.  If they are solving some problem, and they
> > aren't broken, nobody replaces them.  I'll bet you anything there
> > are still SCO registers out there, I'll bet there are still PDP-11s
> > out there.

The logical consequence of this is to write in Autoconf's recommended
dialect of shell, which has far too many rules to remember because every
Unix vendor added exciting new bugs to their "pure" V7 shells.

https://www.gnu.org/software/autoconf/manual/autoconf-2.60/html_node/Portable-Shell.html

> > If it ain't broke, don't fix it.

They were all broken.  Badly.  If your scripts worked, you got lucky at
Russian Roulette.  No greater claim to robustness can be made.

POSIX shell conformance still proves challenging for vendors, but is an
immense improvement over the status quo ante.

I agree with Robert Elz though that standardizing the alias feature was
nuts.  I want to know who brought the crack cocaine to the conference
table that day.

Regards,
Branden

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-02 18:00                       ` G. Branden Robinson
@ 2023-01-02 18:05                         ` Luther Johnson
  2023-01-02 18:12                         ` Larry McVoy
  1 sibling, 0 replies; 121+ messages in thread
From: Luther Johnson @ 2023-01-02 18:05 UTC (permalink / raw)
  To: tuhs

Point taken, if not entirely understood :)

On 01/02/2023 11:00 AM, G. Branden Robinson wrote:
> At 2023-01-02T10:48:32-0700, Luther Johnson wrote:
>> Arnold Schoenberg said of his 12-tone method of music composition, "in
>> constraints I find freedom". The best thing about committing to a
>> simple style and a small set of tools that span the problem space, is
>> the effect it has on my program composition, imho.
> Schoenberg didn't have to compose for orchestras such that the Berlin
> Philharmonic string section tuned half of the octave meantone with the
> other half equal-tempered, whereas the Royal Concertgebouw had the
> brasses play an un-notated ascending major third gliss anytime you
> approached the tonic from a tritone _below_ (only).
>
>> On 01/02/2023 10:43 AM, Larry McVoy wrote:
>>> I was supporting a commercial product in the early 2000's and there
>>> were all sorts of systems then that had old shells.  Yes, you could
>>> make everything a 2 level thing where the first level finds the
>>> correct shell, but that's just fuss.  Just make things portable,
>>> it's not that hard and it works everywhere.
>>>
>>> When you get to the commercial world, you'd be stunned to see how
>>> long old machines last.  If they are solving some problem, and they
>>> aren't broken, nobody replaces them.  I'll bet you anything there
>>> are still SCO registers out there, I'll bet there are still PDP-11s
>>> out there.
> The logical consequence of this is to write in Autoconf's recommended
> dialect of shell, which has far too many rules to remember because every
> Unix vendor added exciting new bugs to their "pure" V7 shells.
>
> https://www.gnu.org/software/autoconf/manual/autoconf-2.60/html_node/Portable-Shell.html
>
>>> If it ain't broke, don't fix it.
> They were all broken.  Badly.  If your scripts worked, you got lucky at
> Russian Roulette.  No greater claim to robustness can be made.
>
> POSIX shell conformance still proves challenging for vendors, but is an
> immense improvement over the status quo ante.
>
> I agree with Robert Elz though that standardizing the alias feature was
> nuts.  I want to know who brought the crack cocaine to the conference
> table that day.
>
> Regards,
> Branden


^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-02 17:55                     ` Adam Thornton
@ 2023-01-02 18:11                       ` Clem Cole
  2023-01-02 18:36                         ` Dan Cross
  2023-01-02 18:18                       ` Larry McVoy
  1 sibling, 1 reply; 121+ messages in thread
From: Clem Cole @ 2023-01-02 18:11 UTC (permalink / raw)
  To: Adam Thornton; +Cc: The Eunuchs Hysterical Society

[-- Attachment #1: Type: text/plain, Size: 1045 bytes --]

Maybe this should go to COFF but Adam I fear you are falling into a tap
that is easy to fall into - old == unused

One of my favorite stores in the computer restoration community is from
5-10 years ago and the LCM+L in Seatle was restoring their CDC-6000 that
they got From Purdue.   Core memory is difficult to get, so they made a
card and physical module that could plug into their system that is both
electrically and mechanically equivalent using modern semiconductors.   A
few weeks later they announced that they had the system running and had
built this module. They got approached by the USAF asking if they could get
a copy of the design.  Seems, there was still a least one CDC-6600 running
a particular critical application somewhere.

This is similar to the PDP-11s and Vaxen that are supposed to be still in
the hydroelectric grid [a few years ago the was an ad for an RSX and VMS
programmer to go to Canada running in the Boston newspapers - I know
someone that did a small amount of consulting on that one].
ᐧ

[-- Attachment #2: Type: text/html, Size: 1782 bytes --]

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-02 18:00                       ` G. Branden Robinson
  2023-01-02 18:05                         ` Luther Johnson
@ 2023-01-02 18:12                         ` Larry McVoy
  2023-01-02 18:16                           ` Clem Cole
  2023-01-02 19:21                           ` G. Branden Robinson
  1 sibling, 2 replies; 121+ messages in thread
From: Larry McVoy @ 2023-01-02 18:12 UTC (permalink / raw)
  To: G. Branden Robinson; +Cc: tuhs

On Mon, Jan 02, 2023 at 12:00:20PM -0600, G. Branden Robinson wrote:
> > > If it ain't broke, don't fix it.
> 
> They were all broken.  Badly.  If your scripts worked, you got lucky at
> Russian Roulette.  No greater claim to robustness can be made.
> 
> POSIX shell conformance still proves challenging for vendors, but is an
> immense improvement over the status quo ante.

You are talking to a dude with 40+ years of Unix experience, supporting
commercial products most of that time.  I didn't get "lucky at Russian
Roulette", I wrote scripts that were portable.  I have 40 year old scripts
that _still_ work and they work on virtually every Unix ever built.
How do I know?  I was a contractor for my first job, I got plopped down
in front of every random unix you could imagine and each time I polished
off the warts.

I spent decades supporting my own products on every flavor of Unix and
processors from Arm to System/360.  Oh, and Windows XP and on and MacOS.

My scripts worked with /bin/sh being whatever it was.

It's interesting to me that other old timers, like Clem, are saying
exactly the same thing as I am.  Are we all wrong?

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-02 18:12                         ` Larry McVoy
@ 2023-01-02 18:16                           ` Clem Cole
  2023-01-02 19:50                             ` Warner Losh
  2023-01-02 19:21                           ` G. Branden Robinson
  1 sibling, 1 reply; 121+ messages in thread
From: Clem Cole @ 2023-01-02 18:16 UTC (permalink / raw)
  To: Larry McVoy; +Cc: tuhs

[-- Attachment #1: Type: text/plain, Size: 334 bytes --]

On Mon, Jan 2, 2023 at 1:13 PM Larry McVoy <lm@mcvoy.com> wrote:

>
> My scripts worked with /bin/sh being whatever it was.
>
> It's interesting to me that other old timers, like Clem, are saying
> exactly the same thing as I am.  Are we all wrong?
>
Exactly -- if you are disciplined -- it just works.  It's not magic.
ᐧ

[-- Attachment #2: Type: text/html, Size: 1115 bytes --]

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-02 17:55                     ` Adam Thornton
  2023-01-02 18:11                       ` Clem Cole
@ 2023-01-02 18:18                       ` Larry McVoy
  1 sibling, 0 replies; 121+ messages in thread
From: Larry McVoy @ 2023-01-02 18:18 UTC (permalink / raw)
  To: Adam Thornton; +Cc: The Eunuchs Hysterical Society

On Mon, Jan 02, 2023 at 10:55:51AM -0700, Adam Thornton wrote:
> And no one is saying you can't keep running your SCO cash registers
> forever on your own recognizance, but when they break, you get to keep
> both pieces, and if they break in such a way that suddenly your entire
> operation is being held hostage by some Russian organized crime cartel,
> well, you should have priced that risk appropriately.

Not my problem, I'm not running SCO cash registers, but the customer
might be.  If they have enough money to interest me, then my problem is
having a product that works on their platform.

You argue with me all you want, you aren't going to win.  Writing stuff
in a portable way is not that hard and once you have that figured out,
you just do it.  It's the path to more revenue.

And I've fought this battle with some extremely intelligent and well
meaning engineers, absolutely in the top 1% of software engineers,
and they all caved.  Sometimes it took me letting them off the leash
long enough to make something break, but they all caved eventually.

It's just more pleasant knowing that your stuff will work no matter
what /bin/sh is there.  All the bash-ism in the world are less 
valuable than "it just works".  Don't get me wrong, I use bash-ism
for my personal use but I wouldn't let them get anywhere near a
product I needed to ship.

All that said, I'm done with this topic, feel free to have the last
word, I think I'm boring the smart people at this point.

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-02 18:11                       ` Clem Cole
@ 2023-01-02 18:36                         ` Dan Cross
  2023-01-02 18:48                           ` Dan Cross
  0 siblings, 1 reply; 121+ messages in thread
From: Dan Cross @ 2023-01-02 18:36 UTC (permalink / raw)
  To: Clem Cole; +Cc: COFF

On Mon, Jan 2, 2023 at 1:13 PM Clem Cole <clemc@ccc.com> wrote:
> Maybe this should go to COFF but Adam I fear you are falling into a tap that is easy to fall into - old == unused
>
> One of my favorite stores in the computer restoration community is from 5-10 years ago and the LCM+L in Seatle was restoring their CDC-6000 that they got From Purdue.   Core memory is difficult to get, so they made a card and physical module that could plug into their system that is both electrically and mechanically equivalent using modern semiconductors.   A few weeks later they announced that they had the system running and had built this module. They got approached by the USAF asking if they could get a copy of the design.  Seems, there was still a least one CDC-6600 running a particular critical application somewhere.
>
> This is similar to the PDP-11s and Vaxen that are supposed to be still in the hydroelectric grid [a few years ago the was an ad for an RSX and VMS programmer to go to Canada running in the Boston newspapers - I know someone that did a small amount of consulting on that one].

One of my favorite stories along these lines is the train signalling
system in Melbourne, running on a "PDP-11". I quote PDP-11 because
that is now virtualized:
https://www.equicon.de/images/Virtualisierung/LegacyTrainControlSystemStabilization.pdf

Indeed older systems show up in surprising places. I was once on the
bridge of a US Naval vessel in the late '00s and saw a SPARCstaton 20
running Solaris (with the CDE login screen). I don't recall what it
was doing, but it was a tad surprising.

I do worry about legacy systems in critical situations, but then, I've
been in a firefight when the damned tactical computer with the satcomm
link rebooted and we didn't have VHF comms with the battlespace owner.
That was not particularly fun.

        - Dan C.

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-02 18:36                         ` Dan Cross
@ 2023-01-02 18:48                           ` Dan Cross
  0 siblings, 0 replies; 121+ messages in thread
From: Dan Cross @ 2023-01-02 18:48 UTC (permalink / raw)
  To: Clem Cole; +Cc: COFF

[Apologies for resending; I messed up and used the old Minnie address
for COFF in the Cc]

On Mon, Jan 2, 2023 at 1:36 PM Dan Cross <crossd@gmail.com> wrote:
>
> On Mon, Jan 2, 2023 at 1:13 PM Clem Cole <clemc@ccc.com> wrote:
> > Maybe this should go to COFF but Adam I fear you are falling into a tap that is easy to fall into - old == unused
> >
> > One of my favorite stores in the computer restoration community is from 5-10 years ago and the LCM+L in Seatle was restoring their CDC-6000 that they got From Purdue.   Core memory is difficult to get, so they made a card and physical module that could plug into their system that is both electrically and mechanically equivalent using modern semiconductors.   A few weeks later they announced that they had the system running and had built this module. They got approached by the USAF asking if they could get a copy of the design.  Seems, there was still a least one CDC-6600 running a particular critical application somewhere.
> >
> > This is similar to the PDP-11s and Vaxen that are supposed to be still in the hydroelectric grid [a few years ago the was an ad for an RSX and VMS programmer to go to Canada running in the Boston newspapers - I know someone that did a small amount of consulting on that one].
>
> One of my favorite stories along these lines is the train signalling
> system in Melbourne, running on a "PDP-11". I quote PDP-11 because
> that is now virtualized:
> https://www.equicon.de/images/Virtualisierung/LegacyTrainControlSystemStabilization.pdf
>
> Indeed older systems show up in surprising places. I was once on the
> bridge of a US Naval vessel in the late '00s and saw a SPARCstaton 20
> running Solaris (with the CDE login screen). I don't recall what it
> was doing, but it was a tad surprising.
>
> I do worry about legacy systems in critical situations, but then, I've
> been in a firefight when the damned tactical computer with the satcomm
> link rebooted and we didn't have VHF comms with the battlespace owner.
> That was not particularly fun.
>
>         - Dan C.

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-02 18:12                         ` Larry McVoy
  2023-01-02 18:16                           ` Clem Cole
@ 2023-01-02 19:21                           ` G. Branden Robinson
  2023-01-02 19:34                             ` Rich Salz
  1 sibling, 1 reply; 121+ messages in thread
From: G. Branden Robinson @ 2023-01-02 19:21 UTC (permalink / raw)
  To: tuhs

[-- Attachment #1: Type: text/plain, Size: 2208 bytes --]

At 2023-01-02T10:12:42-0800, Larry McVoy wrote:
> You are talking to a dude with 40+ years of Unix experience,

I'm clocking 30 this year.  A babe in the woods.

> supporting commercial products most of that time.  I didn't get "lucky
> at Russian Roulette", I wrote scripts that were portable.  I have 40
> year old scripts that _still_ work and they work on virtually every
> Unix ever built.  How do I know?  I was a contractor for my first job,
> I got plopped down in front of every random unix you could imagine and
> each time I polished off the warts.

Yes.  That's how the chapter of the GNU Autoconf manual to which I
linked got written--or so I surmise, based simply on how bizarre some of
the bugs are that have to be worked around.

> I spent decades supporting my own products on every flavor of Unix and
> processors from Arm to System/360.  Oh, and Windows XP and on and
> MacOS.
> 
> My scripts worked with /bin/sh being whatever it was.
> 
> It's interesting to me that other old timers, like Clem, are saying
> exactly the same thing as I am.  Are we all wrong?

Over on the groff list you had the opposite advice regarding Make: pick
one implementation and go with it, you said, and other implementations
be damned (very loosely paraphrasing).[1]

I'm not saying you're wrong--I'm saying you're inconsistent.

I reckon you did what you needed to do to keep the lights on and the
engineers fed.  That's fine, but it's also a recipe for passivity.  You
don't have to tell me, but count up the number of times you flexed to
accommodate a buggy implementation versus telling your client that their
shell was just too crap and that they needed to either install one in
which you had confidence, or you'd have to decline the contract.

If their users had demanded more of their commercial Unix vendors, maybe
those Unices would not be regarded as dinosaurs today.

I could harp further on quality of implementation issues, but I'd
probably do better to take up the lotus position on the back of a
motorcycle.

Preferably not while the vehicle is in motion.

Regards,
Branden

[1] https://lists.gnu.org/archive/html/groff/2022-03/msg00084.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-02 19:21                           ` G. Branden Robinson
@ 2023-01-02 19:34                             ` Rich Salz
  2023-01-02 20:12                               ` Larry McVoy
  2023-01-02 20:24                               ` G. Branden Robinson
  0 siblings, 2 replies; 121+ messages in thread
From: Rich Salz @ 2023-01-02 19:34 UTC (permalink / raw)
  To: G. Branden Robinson; +Cc: tuhs

[-- Attachment #1: Type: text/plain, Size: 121 bytes --]

The difference is that "make" need only run on your build systems.
Installation scripts need to run on customer systems.

[-- Attachment #2: Type: text/html, Size: 202 bytes --]

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-02 18:16                           ` Clem Cole
@ 2023-01-02 19:50                             ` Warner Losh
  2023-01-02 20:05                               ` Adam Thornton
  0 siblings, 1 reply; 121+ messages in thread
From: Warner Losh @ 2023-01-02 19:50 UTC (permalink / raw)
  To: Clem Cole; +Cc: The Eunuchs Hysterical Society

[-- Attachment #1: Type: text/plain, Size: 1576 bytes --]

On Mon, Jan 2, 2023, 11:18 AM Clem Cole <clemc@ccc.com> wrote:

>
>
> On Mon, Jan 2, 2023 at 1:13 PM Larry McVoy <lm@mcvoy.com> wrote:
>
>>
>> My scripts worked with /bin/sh being whatever it was.
>>
>> It's interesting to me that other old timers, like Clem, are saying
>> exactly the same thing as I am.  Are we all wrong?
>>
> Exactly -- if you are disciplined -- it just works.  It's not magic.
>

I've only got 30 years experience in writing portable shell scripts... But
I have a slightly different take. There is an element of luck involved.
There are a number of constructs that are valid in v7 shell that aren't
completely portable due to bugs in vendor shells. The luck part comes in
either you know what to avoid because you got unlucky and got burned in the
past. Or you got lucky and never thought to write something like that so it
just works. With experience, you know what works...

I notice the old timers hedged a little bit by saying things like
"virtually all" so maybe they had a bit of bad luck now and again...

Also, the really aweful systems are rare enough today that most folks on
this list have had the good luck not to need to run on them... :)

You can get super close though, but I'll bet there's some scripts that work
only because they didn't run on, say, Eunice...

But luck favors the prepared... and making the effort to try to be portable
will help when you have to run on dash, or one of the BSD shells that might
not have $<(cmd) or some other useful but not universal feature...

Warner

> ᐧ
>

[-- Attachment #2: Type: text/html, Size: 3087 bytes --]

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Command Line Editing in the Terminal Driver (Was: A few comments on porting the Bourne shell)
  2022-12-30 20:42     ` Sven Mascheck
  2022-12-30 20:48       ` Jon Steinhart
@ 2023-01-02 20:03       ` Joseph Holsten
  2023-01-02 20:33         ` [TUHS] Re: Command Line Editing in the Terminal Driver Lars Brinkhoff
                           ` (2 more replies)
  1 sibling, 3 replies; 121+ messages in thread
From: Joseph Holsten @ 2023-01-02 20:03 UTC (permalink / raw)
  To: Tautological Eunuch Horticultural Scythians

[-- Attachment #1: Type: text/plain, Size: 2225 bytes --]

On Fri, Dec 30, 2022, at 12:42, Sven Mascheck wrote:
> and in "ksh - An Extensible High Level Language" David Korn writes:
> 
>  *  "Originally the idea of adding command line editing to ksh was rejected in the hope that line editing would move into the terminal driver." [2]
> [2] https://www.in-ulm.de/~mascheck/bourne/korn.html

This phrase has haunted me for years. It’s so clearly the “correct” separation of concerns, until you actually attempt implementing it. And Bourne’s irregular grammar certainly doesn’t help here. I’d prefer to move beyond readline, ideally something like text objects per vim/zsh/treesitter. But having one parser in the interpreter and another in the line editor becomes quite exciting if you want a true bourne or even posix sh.

But the thing that brought be back to playing against this quote again this year was starting to research the QED lineage and discovering helix & kakoune. Honestly, they feels like the most coherent advancements in QED-style editors since sam & acme. (Yes, I’m irrationally excluding vim text-objects, mostly because no one removed editor features that t-o made redundant. It’s as if ed had twice as many commands, one for regexp matches and one for pre-ken-QEDist exact matches.)

Anyway, the only time I’ve really seen “line editing […] move into the terminal driver” sound possible was acme’s win. For those who haven’t had the pleasure, rsc describes it at 15:25 in https://research.swtch.com/acme. “I worked for many years in a shell without history or command line editing, and I never missed it because Acme is providing this for me.”

It’s hard to convey how surprisingly pleasant sh or rc can be within acme win to someone who has only used them within a mere terminal-emulator-emulator. Especially since a greenfield terminal app has more code emulating old xterm behaviors than physical vt100 behaviours. This is especially exhausting when you try to use something like NeoVim’s :term expecting it to just work and discover that buffer-line-wrapping on window-resize hasn’t been implemented.

Anyone seen any other “terminal drivers” that provide a pleasant alternative to line editing?
--
~j

[-- Attachment #2: Type: text/html, Size: 3822 bytes --]

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-02 19:50                             ` Warner Losh
@ 2023-01-02 20:05                               ` Adam Thornton
  0 siblings, 0 replies; 121+ messages in thread
From: Adam Thornton @ 2023-01-02 20:05 UTC (permalink / raw)
  To: Warner Losh; +Cc: The Eunuchs Hysterical Society



> On Jan 2, 2023, at 12:50 PM, Warner Losh <imp@bsdimp.com> wrote:
> 
> 
> You can get super close though, but I'll bet there's some scripts that work only because they didn't run on, say, Eunice...

"Congratulations!"

> 
> But luck favors the prepared... and making the effort to try to be portable will help when you have to run on dash, or one of the BSD shells that might not have $<(cmd) or some other useful but not universal feature...

Yeah and I think we're basically agreeing here.  The problem isn't *really* whether you use some-odd-number-of-backslashes-and-a-backtick versus dollar-open-paren, the problem is people who put #!/bin/sh at the top of the script when the contents only run under bash.  And there's a lot of that, because just as all the world was VAX once upon a time, all the world is assumed to be an x86_64 Linux where sh is bash.

At least the move towards containerization has largely defeated that assumption now, although in some sense it's also just swept the problem under someone else's rug.

Adam


^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-02 19:34                             ` Rich Salz
@ 2023-01-02 20:12                               ` Larry McVoy
  2023-01-02 20:24                               ` G. Branden Robinson
  1 sibling, 0 replies; 121+ messages in thread
From: Larry McVoy @ 2023-01-02 20:12 UTC (permalink / raw)
  To: Rich Salz; +Cc: tuhs

On Mon, Jan 02, 2023 at 02:34:22PM -0500, Rich Salz wrote:
> The difference is that "make" need only run on your build systems.
> Installation scripts need to run on customer systems.

Exactly.  I let my team, against my judgement, use GNU make for our builds.
That was in house, the scripts are customer run.
-- 
---
Larry McVoy           Retired to fishing          http://www.mcvoy.com/lm/boat

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-02 19:34                             ` Rich Salz
  2023-01-02 20:12                               ` Larry McVoy
@ 2023-01-02 20:24                               ` G. Branden Robinson
  2023-01-02 20:41                                 ` Larry McVoy
  1 sibling, 1 reply; 121+ messages in thread
From: G. Branden Robinson @ 2023-01-02 20:24 UTC (permalink / raw)
  To: tuhs

[-- Attachment #1: Type: text/plain, Size: 840 bytes --]

At 2023-01-02T14:34:22-0500, Rich Salz wrote:
> The difference is that "make" need only run on your build systems.
> Installation scripts need to run on customer systems.

Surprisingly (not really), I don't agree with this, either.

make, like tsort, is a generally useful tool that people misconceive as
being limited in scope to traditional compile-and-link software
development.

A dab of imagination and familiarity with existing tooling can save one
much wheel reinvention.

That said I don't doubt that some vendors happily took these tools in
particular (along with the compiler, of course) and locked them up
within a "pro" or "development" tier.

But all that teaches us is something we should have already known: free
software is better than proprietary software.  Why _shouldn't_ customer
systems have make(1)?

Regards,
Branden

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: Command Line Editing in the Terminal Driver
  2023-01-02 20:03       ` [TUHS] Command Line Editing in the Terminal Driver (Was: A few comments on porting the Bourne shell) Joseph Holsten
@ 2023-01-02 20:33         ` Lars Brinkhoff
  2023-01-11  2:51           ` Chris Hanson
  2023-01-02 21:08         ` [TUHS] Re: Command Line Editing in the Terminal Driver (Was: A few comments on porting the Bourne shell) Rob Pike
  2023-01-03 16:53         ` Marshall Conover
  2 siblings, 1 reply; 121+ messages in thread
From: Lars Brinkhoff @ 2023-01-02 20:33 UTC (permalink / raw)
  To: Joseph Holsten; +Cc: Tautological Eunuch Horticultural Scythians

Joseph Holsten wrote:
> Anyone seen any other “terminal drivers” that provide a pleasant
> alternative to line editing?

The Stanford AI lab PDP-10 operating system had line editing built in,
and the E editor integrated with this.

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-02 20:24                               ` G. Branden Robinson
@ 2023-01-02 20:41                                 ` Larry McVoy
  2023-01-02 21:00                                   ` Dan Cross
  0 siblings, 1 reply; 121+ messages in thread
From: Larry McVoy @ 2023-01-02 20:41 UTC (permalink / raw)
  To: G. Branden Robinson; +Cc: tuhs

On Mon, Jan 02, 2023 at 02:24:22PM -0600, G. Branden Robinson wrote:
> At 2023-01-02T14:34:22-0500, Rich Salz wrote:
> > The difference is that "make" need only run on your build systems.
> > Installation scripts need to run on customer systems.
> 
> Surprisingly (not really), I don't agree with this, either.
> 
> make, like tsort, is a generally useful tool that people misconceive as
> being limited in scope to traditional compile-and-link software
> development.
> 
> A dab of imagination and familiarity with existing tooling can save one
> much wheel reinvention.

If we wanted to use Makefiles in our customer run product, we would have
shipped it.

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-02 20:41                                 ` Larry McVoy
@ 2023-01-02 21:00                                   ` Dan Cross
  2023-01-02 21:06                                     ` Clem Cole
  2023-01-02 21:08                                     ` Joseph Holsten
  0 siblings, 2 replies; 121+ messages in thread
From: Dan Cross @ 2023-01-02 21:00 UTC (permalink / raw)
  To: Larry McVoy; +Cc: tuhs

On Mon, Jan 2, 2023 at 3:41 PM Larry McVoy <lm@mcvoy.com> wrote:
> On Mon, Jan 02, 2023 at 02:24:22PM -0600, G. Branden Robinson wrote:
> > At 2023-01-02T14:34:22-0500, Rich Salz wrote:
> > > The difference is that "make" need only run on your build systems.
> > > Installation scripts need to run on customer systems.
> >
> > Surprisingly (not really), I don't agree with this, either.
> >
> > make, like tsort, is a generally useful tool that people misconceive as
> > being limited in scope to traditional compile-and-link software
> > development.
> >
> > A dab of imagination and familiarity with existing tooling can save one
> > much wheel reinvention.
>
> If we wanted to use Makefiles in our customer run product, we would have
> shipped it.

Couldn't you also ship a shell? That kinda seems like the simplest solution.

I used to be pretty dogmatic about adherence to 7th Ed tools and
syntax, but I think I'm like the frog in the pot where the temperature
is slowly rising: I suspect I've let things creep in that are not
portable to a universal intersection set, simply because I haven't
noticed, because I'm no longer trying to run things on very old
systems regularly. Will that script I wrote last week run on Ultrix?
Beats me; I haven't powered up the DECstation in 15 years. But
considering the issue raises a related question: practically speaking,
what do you define as a least common denominator? For example, is
`nawk` allowed, or just the traditional 7th Ed awk? How about extended
regular expressions? GNU coreutils `egrep` now spits out a warning
saying, "egrep is obsolescent; using grep -E" (I cannot adequately
express how irritating I find this).  Does `tr` need the square
brackets? Etc. The shell by itself is just part of the equation, and
as a programming environment, /bin/sh is not all that interesting
unless complemented by a bunch of utilities, the portability of which
seems less guaranteed than it once was.

I actually think the shell is less of an issue; C is a bigger one for
me. About a year ago, I set up some emulated systems on my local ham
radio network (connected to AX.25 also!). Among them were Multics and
TOPS-20, but also a VAX running 4.3BSD and 7th Ed on a PDP-11. But
getting anything modern to build on either is a serious pain; speaking
of shells, I was able to find what is (I think) the last version of
tcsh to support 4.3 and get that to build.

Then again, having 7th Ed around forces me to think about shell script
portability again, so...there's that. :-)

        - Dan C.

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-02 21:00                                   ` Dan Cross
@ 2023-01-02 21:06                                     ` Clem Cole
  2023-01-02 21:19                                       ` Dan Cross
  2023-01-02 22:43                                       ` Steve Nickolas
  2023-01-02 21:08                                     ` Joseph Holsten
  1 sibling, 2 replies; 121+ messages in thread
From: Clem Cole @ 2023-01-02 21:06 UTC (permalink / raw)
  To: Dan Cross; +Cc: tuhs

[-- Attachment #1: Type: text/plain, Size: 247 bytes --]

On Mon, Jan 2, 2023 at 4:03 PM Dan Cross <crossd@gmail.com> wrote:

> Couldn't you also ship a shell? That kinda seems like the simplest
> solution.
>
Sure the problem is -- where do you stop?

The result is the fork/mess called Gnu.
ᐧ

[-- Attachment #2: Type: text/html, Size: 1341 bytes --]

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-02 21:00                                   ` Dan Cross
  2023-01-02 21:06                                     ` Clem Cole
@ 2023-01-02 21:08                                     ` Joseph Holsten
  2023-01-02 21:15                                       ` Adam Thornton
  1 sibling, 1 reply; 121+ messages in thread
From: Joseph Holsten @ 2023-01-02 21:08 UTC (permalink / raw)
  To: Tautological Eunuch Horticultural Scythians

On Mon, Jan 2, 2023, at 13:00, Dan Cross wrote:
> On Mon, Jan 2, 2023 at 3:41 PM Larry McVoy <lm@mcvoy.com> wrote:
>>
>> If we wanted to use Makefiles in our customer run product, we would have
>> shipped it.
>
> Couldn't you also ship a shell? That kinda seems like the simplest solution.

Then you have to ship documentation on how to properly install the shell. Also, add guardrails to make sure that the correct shell is being loaded instead of someone getting ambitious with their exec path.

cf every products trying to ship its own python, ruby, etc instead of accepting defeat and shipping a statically build tool in a docker container in an OVA virtual machine image.

Obviously, that’s what should have been done with Bourne shell in the first place.

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: Command Line Editing in the Terminal Driver (Was: A few comments on porting the Bourne shell)
  2023-01-02 20:03       ` [TUHS] Command Line Editing in the Terminal Driver (Was: A few comments on porting the Bourne shell) Joseph Holsten
  2023-01-02 20:33         ` [TUHS] Re: Command Line Editing in the Terminal Driver Lars Brinkhoff
@ 2023-01-02 21:08         ` Rob Pike
  2023-01-03 16:53         ` Marshall Conover
  2 siblings, 0 replies; 121+ messages in thread
From: Rob Pike @ 2023-01-02 21:08 UTC (permalink / raw)
  To: Joseph Holsten; +Cc: Tautological Eunuch Horticultural Scythians

[-- Attachment #1: Type: text/plain, Size: 2799 bytes --]

It just seems much more natural to me to have line editing provided at some
fundamental level. Putting it in the shell or some library means some tools
get it but some don't. Why should sh have editing but mail not? But then
there's glob.

Do it once, do it well, and do it for everything, where "it" is a wildcard
that refers to every important detail.

-rob


On Tue, Jan 3, 2023 at 7:03 AM Joseph Holsten <joseph@josephholsten.com>
wrote:

> On Fri, Dec 30, 2022, at 12:42, Sven Mascheck wrote:
>
> and in "ksh - An Extensible High Level Language" David Korn writes:
>
>    -  "Originally the idea of adding command line editing to ksh was
>    rejected in the hope that line editing would move into the terminal
>    driver." [2]
>
> [2] https://www.in-ulm.de/~mascheck/bourne/korn.html
>
>
> This phrase has haunted me for years. It’s so clearly the “correct”
> separation of concerns, until you actually attempt implementing it. And
> Bourne’s irregular grammar certainly doesn’t help here. I’d prefer to move
> beyond readline, ideally something like text objects per
> vim/zsh/treesitter. But having one parser in the interpreter and another in
> the line editor becomes quite exciting if you want a true bourne or even
> posix sh.
>
> But the thing that brought be back to playing against this quote again
> this year was starting to research the QED lineage and discovering helix &
> kakoune. Honestly, they feels like the most coherent advancements in
> QED-style editors since sam & acme. (Yes, I’m irrationally excluding vim
> text-objects, mostly because no one removed editor features that t-o made
> redundant. It’s as if ed had twice as many commands, one for regexp matches
> and one for pre-ken-QEDist exact matches.)
>
> Anyway, the only time I’ve really seen “line editing […] move into the
> terminal driver” sound possible was acme’s win. For those who haven’t had
> the pleasure, rsc describes it at 15:25 in https://research.swtch.com/acme.
> “I worked for many years in a shell without history or command line
> editing, and I never missed it because Acme is providing this for me.”
>
> It’s hard to convey how surprisingly pleasant sh or rc can be within acme
> win to someone who has only used them within a mere
> terminal-emulator-emulator. Especially since a greenfield terminal app
> has more code emulating old xterm behaviors than physical vt100 behaviours.
> This is especially exhausting when you try to use something like NeoVim’s
> :term expecting it to just work and discover that buffer-line-wrapping on
> window-resize hasn’t been implemented.
>
> Anyone seen any other “terminal drivers” that provide a pleasant
> alternative to line editing?
> --
> ~j
>

[-- Attachment #2: Type: text/html, Size: 3987 bytes --]

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-02 21:08                                     ` Joseph Holsten
@ 2023-01-02 21:15                                       ` Adam Thornton
  0 siblings, 0 replies; 121+ messages in thread
From: Adam Thornton @ 2023-01-02 21:15 UTC (permalink / raw)
  To: Joseph Holsten; +Cc: Tautological Eunuch Horticultural Scythians



> On Jan 2, 2023, at 2:08 PM, Joseph Holsten <joseph@josephholsten.com> wrote:
> 
> On Mon, Jan 2, 2023, at 13:00, Dan Cross wrote:
>> On Mon, Jan 2, 2023 at 3:41 PM Larry McVoy <lm@mcvoy.com> wrote:
>>> 
>>> If we wanted to use Makefiles in our customer run product, we would have
>>> shipped it.
>> 
>> Couldn't you also ship a shell? That kinda seems like the simplest solution.
> 
> Then you have to ship documentation on how to properly install the shell. Also, add guardrails to make sure that the correct shell is being loaded instead of someone getting ambitious with their exec path.
> 
> cf every products trying to ship its own python, ruby, etc instead of accepting defeat and shipping a statically build tool in a docker container in an OVA virtual machine image.
> 
> Obviously, that’s what should have been done with Bourne shell in the first place.

You came very close to owing me a new keyboard there.

Adam

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-02 21:06                                     ` Clem Cole
@ 2023-01-02 21:19                                       ` Dan Cross
  2023-01-02 22:54                                         ` segaloco via TUHS
  2023-01-02 22:43                                       ` Steve Nickolas
  1 sibling, 1 reply; 121+ messages in thread
From: Dan Cross @ 2023-01-02 21:19 UTC (permalink / raw)
  To: Clem Cole; +Cc: tuhs

On Mon, Jan 2, 2023 at 4:07 PM Clem Cole <clemc@ccc.com> wrote:
> On Mon, Jan 2, 2023 at 4:03 PM Dan Cross <crossd@gmail.com> wrote:
>> Couldn't you also ship a shell? That kinda seems like the simplest solution.
>
> Sure the problem is -- where do you stop?

Good question. But this sort of suggests that maybe it's not the shell
that's really the problem, but the entire milieu of all the tools that
we use when we write "shell scripts."

> The result is the fork/mess called Gnu.

BUGS: Yes.

        - Dan C.

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-02 21:06                                     ` Clem Cole
  2023-01-02 21:19                                       ` Dan Cross
@ 2023-01-02 22:43                                       ` Steve Nickolas
  1 sibling, 0 replies; 121+ messages in thread
From: Steve Nickolas @ 2023-01-02 22:43 UTC (permalink / raw)
  To: tuhs

On Mon, 2 Jan 2023, Clem Cole wrote:

> On Mon, Jan 2, 2023 at 4:03 PM Dan Cross <crossd@gmail.com> wrote:
>
>> Couldn't you also ship a shell? That kinda seems like the simplest
>> solution.
>>
> Sure the problem is -- where do you stop?
>
> The result is the fork/mess called Gnu.

GNU is Embrace, Extend and Exterminate applied to Unix.

-uso.

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-02 21:19                                       ` Dan Cross
@ 2023-01-02 22:54                                         ` segaloco via TUHS
  2023-01-02 23:58                                           ` Jon Steinhart
  0 siblings, 1 reply; 121+ messages in thread
From: segaloco via TUHS @ 2023-01-02 22:54 UTC (permalink / raw)
  To: Dan Cross; +Cc: tuhs

A shell is only as powerful as the constructs it can interact with.  One can do logic all day and night but if that logic isn't being put to work, it's just an exercise in vanity.  I think that's a good point that scripting problems may be a symptom of the nature of the tools being used in them.

So the day job is pretty much an all Microsoft shop, from .NET code to IIS servers, not a '$' prompt in sight usually.  However, being a scientific operation, we have quite a diverse crew of UNIX machines floating around out there resigned to their fates sitting in the corner running the oldest equipment at their respective labs.  Any time I've had to interact with any of these I've been grateful for not taking various post-Bourne shell quirks for granted.  Not that these interactions are frequent, but I find myself tapped often being "that UNIX guy" on the team.  A couple examples come to mind.

One facility is running two pieces of equipment the software for which runs on Digital UNIX circa mid 90s.  The data transfer mechanism was some FTP thing their previous technical director had cooked up and one morning we had found it went down.  The resident IT guy there knew the basics of how to get in there, but hadn't really put the process down anywhere.  Luckily (security may think otherwise...) I was able to connect over rsh with the password he provided and was able to modify the script for that exporter to self-heal.  I don't know what shell it was running beyond it spitting a '$' out at me, but luckily didn't have to know.

The other is a lab using a data analysis system for, among others, HP-UX.  I don't recall the age of the version they were running on, but the system is called Target and was eventually forked in a few directions when the original company closed up shop, our group inheriting one of the forks.  It's one of those things I hope I can archive publicly someday, but not there yet.  In any case, the technicians all had Windows PCs with an X11 server setup for accessing the HP-UX machine, and it was running CDE if that helps date it (unless that's still default).  That system is in archive now as we've replaced it with the current version of our fork which is now .NET.  I was surprised they didn't have more UNIX going on, but they had managed to get a lot of their older equipment to work with some sort of Windows based solution one way or another.  Life finds a way...

I'm not sure where it stands as far as the sea of shells these days, but one I've opted towards in system builds the past few years is dash.  It's pretty small and seems to hit all the marks as far as basic Bourne-ish behavior and POSIX stuff.  When I'm tinkering on a board and just need to toss a shell on top of a kernel, I go for dash.

- Matt G.

------- Original Message -------
On Monday, January 2nd, 2023 at 1:19 PM, Dan Cross <crossd@gmail.com> wrote:


> On Mon, Jan 2, 2023 at 4:07 PM Clem Cole clemc@ccc.com wrote:
> 
> > On Mon, Jan 2, 2023 at 4:03 PM Dan Cross crossd@gmail.com wrote:
> > 
> > > Couldn't you also ship a shell? That kinda seems like the simplest solution.
> > 
> > Sure the problem is -- where do you stop?
> 
> 
> Good question. But this sort of suggests that maybe it's not the shell
> that's really the problem, but the entire milieu of all the tools that
> we use when we write "shell scripts."
> 
> > The result is the fork/mess called Gnu.
> 
> 
> BUGS: Yes.
> 
> - Dan C.

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-02 22:54                                         ` segaloco via TUHS
@ 2023-01-02 23:58                                           ` Jon Steinhart
  2023-01-04  9:00                                             ` Joseph Holsten
  0 siblings, 1 reply; 121+ messages in thread
From: Jon Steinhart @ 2023-01-02 23:58 UTC (permalink / raw)
  To: tuhs

segaloco via TUHS writes:
> I think that's a good point that scripting problems may be
> a symptom of the nature of the tools being used in them.

I think that you're hinting at something different.

To the best of my recollection, scripting languages were originally
intended and used for the automation of repetitive personal tasks;
making it easier for users who found themselves typing the same
stuff over and over again.  Somewhere along the line people forgot
how to use a compiler and began writing large systems in a variety
of roughly equivalent but incompatible interpreted languages.  Can
one even boot linux without having several different incompatible
versions of Python installed today?  So I don't think that it's the
nature of the tools; I think that it's people choosing the wrong
tools for the problems that they're trying to solve.

Jon

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-01 10:44   ` arnold
  2023-01-01 11:28     ` arnold
@ 2023-01-03 15:06     ` Chet Ramey
  1 sibling, 0 replies; 121+ messages in thread
From: Chet Ramey @ 2023-01-03 15:06 UTC (permalink / raw)
  To: arnold, tuhs, pnr

On 1/1/23 5:44 AM, arnold@skeeve.com wrote:
> Chet Ramey <chet.ramey@case.edu> wrote:
> 
>> On 12/30/22 1:25 PM, Paul Ruizendaal wrote:
>>>
>>> London and Reiser report about porting the shell that “it required by
>>> far the largest conversion effort of any supposedly portable program,
>>> for the simple reason that it is not portable.” By the time of SysIII
>>> this is greatly improved, but also in porting the SysIII user land it
>>> was the most complex of the set so far.
>>
>> Have you read
>>
>> http://www.collyer.net/who/geoff/sh.tour.pdf
>>
>> and looked at http://www.collyer.net/who/geoff/v7sh.tar ?
>>
>> In the limited literature on Bourne Shell porting, this is authoritative.
>>
>> Arnold Robbins built on that work and ported the v8-v10 shells to modern
>> Linux versions. (I am sorry, I do not have a link right now.)
> 
> Sorry to say, it wasn't me. Geoff Collyer made the v9 sh portable
> and it's available from his web site.

My bad, credit to Geoff.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
		 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/


^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2022-12-31 19:24             ` Clem Cole
@ 2023-01-03 16:32               ` Chet Ramey
  0 siblings, 0 replies; 121+ messages in thread
From: Chet Ramey @ 2023-01-03 16:32 UTC (permalink / raw)
  To: Clem Cole, Jon Steinhart; +Cc: tuhs

On 12/31/22 2:24 PM, Clem Cole wrote:

>     I seem to remember that someone got wedged into System V
>     that was awful and unusable.
> 
> IIRC it was in SVR2 that a version of a job control system added some of 
> the semantics of Kulp's scheme to the kernel, but not all as you point out 
> and it was pretty disappointing if you had grown up on BSD systems.  Later 
> POSIX would pick up the Kulp/BSD Job Control definition and by SVR3/SVR4 
> AT&T fleshed out all all of the semantics.

There is a pretty good discussion of 4.2 BSD-style job control and ways to
make it fit with System V semantics (mostly controlling terminals and, of
course, process groups) at

https://web.physics.ucsb.edu/~pcs/apps/bash/job.control.ps

This is probably the best description of BSD-style job control, and it is
the inspiration for much of the POSIX job control design (e.g., their
invention of "sessions" as the replacement for System V style process
groups).

Chet
-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
		 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/


^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-01 11:28     ` arnold
@ 2023-01-03 16:34       ` Chet Ramey
  0 siblings, 0 replies; 121+ messages in thread
From: Chet Ramey @ 2023-01-03 16:34 UTC (permalink / raw)
  To: arnold, tuhs, pnr

On 1/1/23 6:28 AM, arnold@skeeve.com wrote:

> I also recall that there was some USENET discussion in which
> Rob Pike described how the 8th edition shell changed the search
> order to have functions come before non-special built-ins, with
> the addition of "builtin" and "whatis" commands.

I have this somewhere. It's a rocking back-and-forth between Rob and
David Korn about shell philosophy. Well worth the read if you can find it.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
		 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/


^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: Command Line Editing in the Terminal Driver (Was: A few comments on porting the Bourne shell)
  2023-01-02 20:03       ` [TUHS] Command Line Editing in the Terminal Driver (Was: A few comments on porting the Bourne shell) Joseph Holsten
  2023-01-02 20:33         ` [TUHS] Re: Command Line Editing in the Terminal Driver Lars Brinkhoff
  2023-01-02 21:08         ` [TUHS] Re: Command Line Editing in the Terminal Driver (Was: A few comments on porting the Bourne shell) Rob Pike
@ 2023-01-03 16:53         ` Marshall Conover
  2023-01-04  9:18           ` Joseph Holsten
  2023-01-11  2:56           ` Chris Hanson
  2 siblings, 2 replies; 121+ messages in thread
From: Marshall Conover @ 2023-01-03 16:53 UTC (permalink / raw)
  To: Joseph Holsten; +Cc: Tautological Eunuch Horticultural Scythians

[-- Attachment #1: Type: text/plain, Size: 4937 bytes --]

Along these lines but not quite, Jupyter Notebooks have stood out to me as
another approach to this concept, with behavior I'd like to see implemented
in a shell.

With these notebooks, users type text into blocks, and then can have these
blocks evaluated by some underlying engine, with the results of the
evaluation showing up in a block underneath where they entered text.
Results can be not only text, but things like rendered charts and images.
There may be GUI-interactable rendered elements as well, but I'm unsure.

Notably, text-entry blocks are created such that each new block holds onto
the context of the previous blocks. So, a variable defined in the first
eval block will be available in later blocks.

Currently these are used primarily for demonstrating APIs, exploring data
with python, or writing quick PoCs that can then be extracted into an
application. Some companies, such as Netflix, have experimented with using
them entirely as a replacement for shell scripts, which is the sort of
research I'd most love to see.

Fundamentally, I think it's a smell that the way we interact with our
systems is tied to the long-gone hardware requirement of "be usable with a
teletype."  Acme was certainly a move towards recognizing the value of
being able to reuse space that, in TTYs, would be 'dead' text. The ability
to evaluate commands in Acme, and the further inclusion of an underlying
evaluator that holds state by Jupyter were good steps forward, I think.

I'd love to see experimentation with a whole system that takes the jupyter
approach, with nested namespaces forming applications, combined with data
being held in "blocks" as well as code - much like acme opens and edits
files as well as letting you execute either them or snippets in them. I
think there's a chance something could be developed that would better fit
the way we interface with computers today, and the underlying engine
approach would move us toward the "everything speaks one language" design
we lost in the move from shells to standalone GUI applications.

Mostly, I'd be bummed to see yet another shell that pretends it's behind a
"glass teletype", to use the tongue-in-cheek term from "Ed Mastery", or
just continuing elaboration on that design, despite the fact I use shells
daily. It's the best we have now, but I feel like it's the way forward.

Cheers!

Marshall

P.S. - Kakoune is neat and I'm going to have to poke around at that, thanks
for pointing it out.

On Mon, Jan 2, 2023 at 3:03 PM Joseph Holsten <joseph@josephholsten.com>
wrote:

> On Fri, Dec 30, 2022, at 12:42, Sven Mascheck wrote:
>
> and in "ksh - An Extensible High Level Language" David Korn writes:
>
>    -  "Originally the idea of adding command line editing to ksh was
>    rejected in the hope that line editing would move into the terminal
>    driver." [2]
>
> [2] https://www.in-ulm.de/~mascheck/bourne/korn.html
>
>
> This phrase has haunted me for years. It’s so clearly the “correct”
> separation of concerns, until you actually attempt implementing it. And
> Bourne’s irregular grammar certainly doesn’t help here. I’d prefer to move
> beyond readline, ideally something like text objects per
> vim/zsh/treesitter. But having one parser in the interpreter and another in
> the line editor becomes quite exciting if you want a true bourne or even
> posix sh.
>
> But the thing that brought be back to playing against this quote again
> this year was starting to research the QED lineage and discovering helix &
> kakoune. Honestly, they feels like the most coherent advancements in
> QED-style editors since sam & acme. (Yes, I’m irrationally excluding vim
> text-objects, mostly because no one removed editor features that t-o made
> redundant. It’s as if ed had twice as many commands, one for regexp matches
> and one for pre-ken-QEDist exact matches.)
>
> Anyway, the only time I’ve really seen “line editing […] move into the
> terminal driver” sound possible was acme’s win. For those who haven’t had
> the pleasure, rsc describes it at 15:25 in https://research.swtch.com/acme.
> “I worked for many years in a shell without history or command line
> editing, and I never missed it because Acme is providing this for me.”
>
> It’s hard to convey how surprisingly pleasant sh or rc can be within acme
> win to someone who has only used them within a mere
> terminal-emulator-emulator. Especially since a greenfield terminal app
> has more code emulating old xterm behaviors than physical vt100 behaviours.
> This is especially exhausting when you try to use something like NeoVim’s
> :term expecting it to just work and discover that buffer-line-wrapping on
> window-resize hasn’t been implemented.
>
> Anyone seen any other “terminal drivers” that provide a pleasant
> alternative to line editing?
> --
> ~j
>

[-- Attachment #2: Type: text/html, Size: 6059 bytes --]

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-02 17:55                 ` Clem Cole
@ 2023-01-03 17:08                   ` Paul Winalski
  2023-01-03 19:19                     ` Warner Losh
  0 siblings, 1 reply; 121+ messages in thread
From: Paul Winalski @ 2023-01-03 17:08 UTC (permalink / raw)
  To: Clem Cole; +Cc: tuhs

On 1/2/23, Clem Cole <clemc@ccc.com> wrote:
>
> FWIW: In my start-up times, under the same rules of being disciplined, as
> VP of Engineering, I insisted, all C and C++ code was required to
> 'flex-e-lint' warning clean.   I gave my folks a 3-week week slip to clean
> everything up.   I was cursed during that time.   But guess what, the
> outstanding bug list dropped to ⅒ of what it had been.  Created quite a few
> true believers.  And we made those 3 weeks back before we were done.

This was also the policy in DEC's compiler and software development
tools groups.  This was mainly VMS stuff and we didn't have flex and
lint, but as Clem can attest the C and C++ compilers had very
extensive warning capabilities.  It was group policy that all code had
to compile cleanly, without triggering diagnostic messages, before
check-in was allowed.  Once you get through the initial cleanup of
existing code, this policy pays back big time in avoidance of nasty
Heisenbugs.

-Paul W.

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-03 17:08                   ` Paul Winalski
@ 2023-01-03 19:19                     ` Warner Losh
  2023-01-03 19:56                       ` Luther Johnson
                                         ` (2 more replies)
  0 siblings, 3 replies; 121+ messages in thread
From: Warner Losh @ 2023-01-03 19:19 UTC (permalink / raw)
  To: Paul Winalski; +Cc: The Eunuchs Hysterical Society

[-- Attachment #1: Type: text/plain, Size: 1481 bytes --]

On Tue, Jan 3, 2023, 10:09 AM Paul Winalski <paul.winalski@gmail.com> wrote:

> On 1/2/23, Clem Cole <clemc@ccc.com> wrote:
> >
> > FWIW: In my start-up times, under the same rules of being disciplined, as
> > VP of Engineering, I insisted, all C and C++ code was required to
> > 'flex-e-lint' warning clean.   I gave my folks a 3-week week slip to
> clean
> > everything up.   I was cursed during that time.   But guess what, the
> > outstanding bug list dropped to ⅒ of what it had been.  Created quite a
> few
> > true believers.  And we made those 3 weeks back before we were done.
>
> This was also the policy in DEC's compiler and software development
> tools groups.  This was mainly VMS stuff and we didn't have flex and
> lint, but as Clem can attest the C and C++ compilers had very
> extensive warning capabilities.  It was group policy that all code had
> to compile cleanly, without triggering diagnostic messages, before
> check-in was allowed.  Once you get through the initial cleanup of
> existing code, this policy pays back big time in avoidance of nasty
> Heisenbugs.
>

Not all fixes to appease warnings fix anything. But enough do that it's
worth it.

The one caveat here is that people must understand the warning and that any
change makes things better. There is nothing worse than just tossing a cast
in to brute force it, only to later discover it's the wrong cast or you
needed a different semantic change.

Warner

>

[-- Attachment #2: Type: text/html, Size: 2210 bytes --]

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-03 19:19                     ` Warner Losh
@ 2023-01-03 19:56                       ` Luther Johnson
  2023-01-03 20:21                       ` Dave Horsfall
  2023-01-03 21:47                       ` Clem Cole
  2 siblings, 0 replies; 121+ messages in thread
From: Luther Johnson @ 2023-01-03 19:56 UTC (permalink / raw)
  To: tuhs

[-- Attachment #1: Type: text/plain, Size: 1908 bytes --]

All warnings are worthy of my consideration. I will usually learn 
something about my compiler, my program, and often both. The compiler's 
messages are valuable feedback in my quality work-style loop.

On 01/03/2023 12:19 PM, Warner Losh wrote:
>
>
> On Tue, Jan 3, 2023, 10:09 AM Paul Winalski <paul.winalski@gmail.com 
> <mailto:paul.winalski@gmail.com>> wrote:
>
>     On 1/2/23, Clem Cole <clemc@ccc.com <mailto:clemc@ccc.com>> wrote:
>     >
>     > FWIW: In my start-up times, under the same rules of being
>     disciplined, as
>     > VP of Engineering, I insisted, all C and C++ code was required to
>     > 'flex-e-lint' warning clean.   I gave my folks a 3-week week
>     slip to clean
>     > everything up.   I was cursed during that time.   But guess
>     what, the
>     > outstanding bug list dropped to ⅒ of what it had been.  Created
>     quite a few
>     > true believers.  And we made those 3 weeks back before we were done.
>
>     This was also the policy in DEC's compiler and software development
>     tools groups.  This was mainly VMS stuff and we didn't have flex and
>     lint, but as Clem can attest the C and C++ compilers had very
>     extensive warning capabilities.  It was group policy that all code had
>     to compile cleanly, without triggering diagnostic messages, before
>     check-in was allowed.  Once you get through the initial cleanup of
>     existing code, this policy pays back big time in avoidance of nasty
>     Heisenbugs.
>
>
> Not all fixes to appease warnings fix anything. But enough do that 
> it's worth it.
>
> The one caveat here is that people must understand the warning and 
> that any change makes things better. There is nothing worse than just 
> tossing a cast in to brute force it, only to later discover it's the 
> wrong cast or you needed a different semantic change.
>
> Warner
>


[-- Attachment #2: Type: text/html, Size: 3771 bytes --]

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-03 19:19                     ` Warner Losh
  2023-01-03 19:56                       ` Luther Johnson
@ 2023-01-03 20:21                       ` Dave Horsfall
  2023-01-03 21:47                       ` Clem Cole
  2 siblings, 0 replies; 121+ messages in thread
From: Dave Horsfall @ 2023-01-03 20:21 UTC (permalink / raw)
  To: The Eunuchs Hysterical Society

[-- Attachment #1: Type: text/plain, Size: 799 bytes --]

On Tue, 3 Jan 2023, Warner Losh wrote:

> Not all fixes to appease warnings fix anything. But enough do that it's 
> worth it.

I consider warnings to be potential errors, and act accordingly; one place 
where I worked insisted that code could not be checked in unless it passed 
"-Wall" (no guarantee of course, but a good start).

> The one caveat here is that people must understand the warning and that 
> any change makes things better. There is nothing worse than just tossing 
> a cast in to brute force it, only to later discover it's the wrong cast 
> or you needed a different semantic change. 

Yeah, that's just painting over the cracks; it's like replacing a fuse 
that keeps blowing, or rebooting a box without fixing the bug (what I call 
the M$ Solution).

-- Dave

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-03 19:19                     ` Warner Losh
  2023-01-03 19:56                       ` Luther Johnson
  2023-01-03 20:21                       ` Dave Horsfall
@ 2023-01-03 21:47                       ` Clem Cole
  2023-01-03 21:51                         ` Clem Cole
  2 siblings, 1 reply; 121+ messages in thread
From: Clem Cole @ 2023-01-03 21:47 UTC (permalink / raw)
  To: Warner Losh; +Cc: The Eunuchs Hysterical Society

[-- Attachment #1: Type: text/plain, Size: 2516 bytes --]

Bring this back to TUHS relevance...

On Tue, Jan 3, 2023 at 2:43 PM Warner Losh <imp@bsdimp.com> wrote:

> The one caveat here is that people must understand the warning and that
> any change makes things better. There is nothing worse than just tossing a
> cast in to brute force it, only to later discover it's the wrong cast or
> you needed a different semantic change.
>
Most certainly --   It's why I >>loved<< Gimpel's flex-e-lint product and
as Paul pointed out, Judy Ward's messages from the DEC Gem compiler - both
were the two best I ever ran into in giving you real information about what
was happening.

I also tell a story from my DEC time.   After the GEM compiler was released
and before most other vendors were 64-bits, the ISVs were first starting to
do their ports to Alpha.   My complaining ensued. We would discover from
numerous ISVs that after the Alpha port was complete, their bug count
dropped - why because the ISV's code has been kinda nasty and the older
compilers had been silent about it they had assumed the ILP32 model.
Alphas, using LP64, could not be.  Judy would find things and say -- what a
minute -- you want me to do what with that and issued a fairly detailed
warning (which was the key - she explained what the issue was).  Often the
32 to 64-bit nature forced the programmers at the ISV's to rethink how there
were actually declaring things to make the code clearer, simpler, better,
*etc*.   The classic rewrite never happens unless you are forced too.

I remember going to a Supercomputer conference and talking with the
developers at one the ISVs who I will not name.  He thanks me.  He said his
team has been arguing with their management for years to redo the UNIX
support library. The Tru64 port was what finally allowed them to do it.
But it took 9 months which pissed off his boss. But when it was completed,
and pass all the tests on the Alpha, it just recompiled on Solaris, AIX,
and HP-UX - which had never happened before. He could not believe what a
great compiler we had.

I've always said the Alpha was the greatest gift to Sun and Intel in the
commercial SW world because it forced the ISV to clean up their act before
they ever saw those processors and Sun/Intel ports were piece of cake.   It
was not that porting to Alpha was difficult -- it was cleaning up your own
mess.

Remember we had already gone through this in the PDP-11 ILP16 to Vax ILP32
transition but it is funny how history repeated itself.
ᐧ

[-- Attachment #2: Type: text/html, Size: 5235 bytes --]

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-03 21:47                       ` Clem Cole
@ 2023-01-03 21:51                         ` Clem Cole
  0 siblings, 0 replies; 121+ messages in thread
From: Clem Cole @ 2023-01-03 21:51 UTC (permalink / raw)
  To: Warner Losh; +Cc: The Eunuchs Hysterical Society

[-- Attachment #1: Type: text/plain, Size: 2741 bytes --]

typo: s/My complaining/Much complaining/
sigh.. dyslex-r-me
ᐧ

On Tue, Jan 3, 2023 at 4:47 PM Clem Cole <clemc@ccc.com> wrote:

>
> Bring this back to TUHS relevance...
>
> On Tue, Jan 3, 2023 at 2:43 PM Warner Losh <imp@bsdimp.com> wrote:
>
>> The one caveat here is that people must understand the warning and that
>> any change makes things better. There is nothing worse than just tossing a
>> cast in to brute force it, only to later discover it's the wrong cast or
>> you needed a different semantic change.
>>
> Most certainly --   It's why I >>loved<< Gimpel's flex-e-lint product and
> as Paul pointed out, Judy Ward's messages from the DEC Gem compiler - both
> were the two best I ever ran into in giving you real information about what
> was happening.
>
> I also tell a story from my DEC time.   After the GEM compiler was
> released and before most other vendors were 64-bits, the ISVs were first
> starting to do their ports to Alpha.   My complaining ensued. We would discover
> from numerous ISVs that after the Alpha port was complete, their bug count
> dropped - why because the ISV's code has been kinda nasty and the older
> compilers had been silent about it they had assumed the ILP32 model.
> Alphas, using LP64, could not be.  Judy would find things and say -- what
> a minute -- you want me to do what with that and issued a fairly detailed
> warning (which was the key - she explained what the issue was).  Often
> the 32 to 64-bit nature forced the programmers at the ISV's to rethink
> how there were actually declaring things to make the code clearer,
> simpler, better, *etc*.   The classic rewrite never happens unless you
> are forced too.
>
> I remember going to a Supercomputer conference and talking with the
> developers at one the ISVs who I will not name.  He thanks me.  He said his
> team has been arguing with their management for years to redo the UNIX
> support library. The Tru64 port was what finally allowed them to do it.
> But it took 9 months which pissed off his boss. But when it was completed,
> and pass all the tests on the Alpha, it just recompiled on Solaris, AIX,
> and HP-UX - which had never happened before. He could not believe what a
> great compiler we had.
>
> I've always said the Alpha was the greatest gift to Sun and Intel in the
> commercial SW world because it forced the ISV to clean up their act before
> they ever saw those processors and Sun/Intel ports were piece of cake.   It
> was not that porting to Alpha was difficult -- it was cleaning up your own
> mess.
>
> Remember we had already gone through this in the PDP-11 ILP16 to Vax
> ILP32  transition but it is funny how history repeated itself.
> ᐧ
>

[-- Attachment #2: Type: text/html, Size: 6172 bytes --]

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-02 17:43                   ` Larry McVoy
  2023-01-02 17:48                     ` Luther Johnson
  2023-01-02 17:55                     ` Adam Thornton
@ 2023-01-04  3:20                     ` John Cowan
  2023-01-04  3:31                       ` Dan Cross
  2023-01-04 15:21                       ` Ralph Corderoy
  2 siblings, 2 replies; 121+ messages in thread
From: John Cowan @ 2023-01-04  3:20 UTC (permalink / raw)
  To: Larry McVoy; +Cc: The Eunuchs Hysterical Society

[-- Attachment #1: Type: text/plain, Size: 713 bytes --]

On Mon, Jan 2, 2023 at 12:43 PM Larry McVoy <lm@mcvoy.com> wrote:


> I was supporting a commercial product in the early 2000's and there
> were all sorts of systems then that had old shells.  Yes, you could make
> everything a 2 level thing where the first level finds the correct shell,
> but that's just fuss.  Just make things portable, it's not that hard
> and it works everywhere.
>

Making shell scripts portable means not using pipelines, because given "foo
| bar", kshNN and zsh execute foo in a subshell and bar in the top-level
shell, whereas in other shells, both foo and bar execute in subshells.
(For this reason, Posix allows either behavior.)  Not having pipelines is a
pretty drastic limitation.

[-- Attachment #2: Type: text/html, Size: 1282 bytes --]

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-04  3:20                     ` John Cowan
@ 2023-01-04  3:31                       ` Dan Cross
  2023-01-04  4:16                         ` Bakul Shah
  2023-01-04 15:21                       ` Ralph Corderoy
  1 sibling, 1 reply; 121+ messages in thread
From: Dan Cross @ 2023-01-04  3:31 UTC (permalink / raw)
  To: John Cowan; +Cc: The Eunuchs Hysterical Society

On Tue, Jan 3, 2023 at 10:22 PM John Cowan <cowan@ccil.org> wrote:
> Making shell scripts portable means not using pipelines, because given "foo | bar", kshNN and zsh execute foo in a subshell and bar in the top-level shell, whereas in other shells, both foo and bar execute in subshells.  (For this reason, Posix allows either behavior.)  Not having pipelines is a pretty drastic limitation.

This came up at work just the other day:

echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
echo $bummer

The behavior varies between ksh, zsh, bash, sh. Yay.

        - Dan C.

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-04  3:31                       ` Dan Cross
@ 2023-01-04  4:16                         ` Bakul Shah
  2023-01-04 16:15                           ` Dan Cross
  0 siblings, 1 reply; 121+ messages in thread
From: Bakul Shah @ 2023-01-04  4:16 UTC (permalink / raw)
  To: Dan Cross; +Cc: The Eunuchs Hysterical Society

On Jan 3, 2023, at 7:31 PM, Dan Cross <crossd@gmail.com> wrote:
> 
> On Tue, Jan 3, 2023 at 10:22 PM John Cowan <cowan@ccil.org> wrote:
>> Making shell scripts portable means not using pipelines, because given "foo | bar", kshNN and zsh execute foo in a subshell and bar in the top-level shell, whereas in other shells, both foo and bar execute in subshells.  (For this reason, Posix allows either behavior.)  Not having pipelines is a pretty drastic limitation.
> 
> This came up at work just the other day:
> 
> echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
> echo $bummer
> 
> The behavior varies between ksh, zsh, bash, sh. Yay.

On this example sh, zsh, bash behave the same way on
freebsd. Rather than ban |, I think the lesson is to
*avoid* builtin commands in a pipeline that can affect
shell's environment. I only write /bin/sh scripts in
any case.

It is really unfortunate that infix | is so beguiling!
Except for that a Lisp or Scheme based shell would've
been fine and dandy! :-)/2


^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-02 23:58                                           ` Jon Steinhart
@ 2023-01-04  9:00                                             ` Joseph Holsten
  0 siblings, 0 replies; 121+ messages in thread
From: Joseph Holsten @ 2023-01-04  9:00 UTC (permalink / raw)
  To: Jon Steinhart; +Cc: tuhs

[-- Attachment #1: Type: text/plain, Size: 1967 bytes --]


> On Jan 2, 2023, at 15:59, Jon Steinhart <jon@fourwinds.com> wrote:
> 
> segaloco via TUHS writes:
>> I think that's a good point that scripting problems may be
>> a symptom of the nature of the tools being used in them.
> 
> To the best of my recollection, scripting languages were originally
> intended and used for the automation of repetitive personal tasks;
> making it easier for users who found themselves typing the same
> stuff over and over again.  Somewhere along the line people forgot
> how to use a compiler and began writing large systems in a variety
> of roughly equivalent but incompatible interpreted languages. 

I used to think this was a somewhat modern thing, at least evident by atrocities performed by tcl & perl in the 1990s. But exposure to this group of weirdos has made me realize that somewhat terrifying amounts of “infrastructure” code is built on little languages like awk or ed scripts. And that any criticism of the horrible things I use Ruby for today only really defends the place for awk and sed that those now hold and where they truly excel.

But then I read https://www.bell-labs.com/usr/dmr/www/qed.html

> For a brief period in the 1970s, the GECOS QED served us as a scripting language; it was in some ways analogous to the Awk or Perl of today. It was used for such tasks as submitting batch jobs, formatting files for the printer, collecting statistics on a file. A collection of macros to do various useful tasks was put in a commonly available place.

I truly hope in my heart of hearts that ken had no inkling that someday QED would be used as the scripting language to power http://www.qef.com/html/quickref.html. Not that QEF is bad, just that any stones thrown at “you shouldn’t use $lang for that!” feel like they must only strike via ricochet off QEF.

I can only imagine the shock and horror Larry McVoy would experience if one of his team suggested replacing make with QEF. 

[-- Attachment #2: Type: text/html, Size: 2933 bytes --]

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: Command Line Editing in the Terminal Driver (Was: A few comments on porting the Bourne shell)
  2023-01-03 16:53         ` Marshall Conover
@ 2023-01-04  9:18           ` Joseph Holsten
  2023-01-11  2:56           ` Chris Hanson
  1 sibling, 0 replies; 121+ messages in thread
From: Joseph Holsten @ 2023-01-04  9:18 UTC (permalink / raw)
  To: Marshall Conover; +Cc: Tautological Eunuch Horticultural Scythians


> On Jan 3, 2023, at 08:55, Marshall Conover <marzhall.o@gmail.com> wrote:
> 
> Along these lines but not quite, Jupyter Notebooks have stood out to me as another approach to this concept, with behavior I'd like to see implemented in a shell.
> Currently these are used primarily for demonstrating APIs, exploring data with python, or writing quick PoCs that can then be extracted into an application. Some companies, such as Netflix, have experimented with using them entirely as a replacement for shell scripts, which is the sort of research I'd most love to see.
> I'd love to see experimentation with a whole system that takes the jupyter approach, with nested namespaces forming applications, combined with data being held in "blocks" as well as code - much like acme opens and edits files as well as letting you execute either them or snippets in them. I think there's a chance something could be developed that would better fit the way we interface with computers today, and the underlying engine approach would move us toward the "everything speaks one language" design we lost in the move from shells to standalone GUI applications.

This sounds tremendously useful. It gets at that feeling I want to have from emacs and literate programming but rarely achieve, the sense of Living Text.

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-04  3:20                     ` John Cowan
  2023-01-04  3:31                       ` Dan Cross
@ 2023-01-04 15:21                       ` Ralph Corderoy
  2023-01-04 15:54                         ` Ron Natalie
  1 sibling, 1 reply; 121+ messages in thread
From: Ralph Corderoy @ 2023-01-04 15:21 UTC (permalink / raw)
  To: tuhs

Hi John,

> Making shell scripts portable means not using pipelines

No it doesn't...

> because given "foo | bar", kshNN and zsh execute foo in a subshell and
> bar in the top-level shell, whereas in other shells, both foo and bar
> execute in subshells.

...because a lot of the time that doesn't matter.

-- 
Cheers, Ralph.

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-04 15:21                       ` Ralph Corderoy
@ 2023-01-04 15:54                         ` Ron Natalie
  0 siblings, 0 replies; 121+ messages in thread
From: Ron Natalie @ 2023-01-04 15:54 UTC (permalink / raw)
  To: Ralph Corderoy, tuhs


Indeed.    MiniUnix had no hardware pipes, but the shell “simulated” it 
by writing to a intermediate temp file.


------ Original Message ------
From "Ralph Corderoy" <ralph@inputplus.co.uk>
To tuhs@tuhs.org
Date 1/4/2023 10:21:02 AM
Subject [TUHS] Re: A few comments on porting the Bourne shell

>Hi John,
>
>>  Making shell scripts portable means not using pipelines
>
>No it doesn't...
>
>>  because given "foo | bar", kshNN and zsh execute foo in a subshell and
>>  bar in the top-level shell, whereas in other shells, both foo and bar
>>  execute in subshells.
>
>...because a lot of the time that doesn't matter.
>
>--
>Cheers, Ralph.

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-04  4:16                         ` Bakul Shah
@ 2023-01-04 16:15                           ` Dan Cross
  2023-01-04 18:28                             ` ron minnich
  2023-01-04 19:33                             ` Chet Ramey
  0 siblings, 2 replies; 121+ messages in thread
From: Dan Cross @ 2023-01-04 16:15 UTC (permalink / raw)
  To: Bakul Shah; +Cc: The Eunuchs Hysterical Society

On Tue, Jan 3, 2023 at 11:16 PM Bakul Shah <bakul@iitbombay.org> wrote:
> On Jan 3, 2023, at 7:31 PM, Dan Cross <crossd@gmail.com> wrote:
> > On Tue, Jan 3, 2023 at 10:22 PM John Cowan <cowan@ccil.org> wrote:
> >> Making shell scripts portable means not using pipelines, because given "foo | bar", kshNN and zsh execute foo in a subshell and bar in the top-level shell, whereas in other shells, both foo and bar execute in subshells.  (For this reason, Posix allows either behavior.)  Not having pipelines is a pretty drastic limitation.
> >
> > This came up at work just the other day:
> >
> > echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
> > echo $bummer
> >
> > The behavior varies between ksh, zsh, bash, sh. Yay.
>
> On this example sh, zsh, bash behave the same way on
> freebsd. Rather than ban |, I think the lesson is to
> *avoid* builtin commands in a pipeline that can affect
> shell's environment. I only write /bin/sh scripts in
> any case.

Interesting. In my testing, `bash` was the odd man out:

: doctor; zsh
: doctor; echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
echo $bummer

hi
: doctor;
: doctor; bash
: doctor; echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
echo $bummer

: doctor;
exit
: doctor; ksh
: doctor; echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
: doctor; echo $bummer
hi
: doctor;
: doctor; sh
: doctor; echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
: doctor; echo $bummer
hi
: doctor;
: doctor;

That's on illumos.

On OpenBSD, neither bash, ksh, or sh produce output, while zsh does. I
suspect ksh is pdksh and sh is a link to ksh there, though. ksh93
gives "hi".

On FreeBSD, neither bash nor sh produce "hi", but zsh and ksh93 do.
Same on DragonFly, macOS and Arch Linux. On 7th edition, I only tried
sh; empty output. 4.3BSD gives empty output for /bin/sh. For kicks,
neither tcsh nor csh understand `read`; I don't think that *csh has a
read-like builtin.

This is, of course, a silly example. Most script authors would do
something like:

    bummer=`echo ' hi ' | sed 's/^ *//;s/ *$//'`

Or, perhaps even:

    bummer=$(echo ' hi ' | sed 's/^ *//;s/ *$//')

Life's too short and at this point I prefer the latter, but more often
than not my fingers are wired for the former.

> It is really unfortunate that infix | is so beguiling!
> Except for that a Lisp or Scheme based shell would've
> been fine and dandy! :-)/2

Whatever happened to `es` ? I guess it died on the vine. It seemed
like a nifty idea.

I made a concerted effort to use `rc` as my shell on Unix for a little
while. Without all the supporting plan9 machinery, though, I gave up.
Minimalism is great in an environment that can augment it with useful
functionality; outside of that environment, it can be kind of a pain
for work-a-day use.

        - Dan C.

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-04 16:15                           ` Dan Cross
@ 2023-01-04 18:28                             ` ron minnich
  2023-01-04 19:33                             ` Chet Ramey
  1 sibling, 0 replies; 121+ messages in thread
From: ron minnich @ 2023-01-04 18:28 UTC (permalink / raw)
  To: Dan Cross; +Cc: Bakul Shah, The Eunuchs Hysterical Society

[-- Attachment #1: Type: text/plain, Size: 3796 bytes --]

What I see so far from this discussion: Ted has written a lot of useful
scripts, but they are explicitly for one shell (bash) on one kernel
(linux). I.e. portability is not a concern.

People immediately pointed out how easy it is to write non-portable shell
scripts, and others commented that it does not matter (to them).

Kernel portability matters to me. So does code that works without regard to
the shell someone is using. So does robust error handling. So does math.
These are all things that, in my experience, rule against complex shell
scripts.

I still prefer Go as my scripting language, but we all have our
preferences. And, yes, Go gets wordy, but it's a price I'm happy to pay for
good error handling. I much prefer an error message to 100 lines of python
backtrace.

On Wed, Jan 4, 2023 at 8:17 AM Dan Cross <crossd@gmail.com> wrote:

> On Tue, Jan 3, 2023 at 11:16 PM Bakul Shah <bakul@iitbombay.org> wrote:
> > On Jan 3, 2023, at 7:31 PM, Dan Cross <crossd@gmail.com> wrote:
> > > On Tue, Jan 3, 2023 at 10:22 PM John Cowan <cowan@ccil.org> wrote:
> > >> Making shell scripts portable means not using pipelines, because
> given "foo | bar", kshNN and zsh execute foo in a subshell and bar in the
> top-level shell, whereas in other shells, both foo and bar execute in
> subshells.  (For this reason, Posix allows either behavior.)  Not having
> pipelines is a pretty drastic limitation.
> > >
> > > This came up at work just the other day:
> > >
> > > echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
> > > echo $bummer
> > >
> > > The behavior varies between ksh, zsh, bash, sh. Yay.
> >
> > On this example sh, zsh, bash behave the same way on
> > freebsd. Rather than ban |, I think the lesson is to
> > *avoid* builtin commands in a pipeline that can affect
> > shell's environment. I only write /bin/sh scripts in
> > any case.
>
> Interesting. In my testing, `bash` was the odd man out:
>
> : doctor; zsh
> : doctor; echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
> echo $bummer
>
> hi
> : doctor;
> : doctor; bash
> : doctor; echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
> echo $bummer
>
> : doctor;
> exit
> : doctor; ksh
> : doctor; echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
> : doctor; echo $bummer
> hi
> : doctor;
> : doctor; sh
> : doctor; echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
> : doctor; echo $bummer
> hi
> : doctor;
> : doctor;
>
> That's on illumos.
>
> On OpenBSD, neither bash, ksh, or sh produce output, while zsh does. I
> suspect ksh is pdksh and sh is a link to ksh there, though. ksh93
> gives "hi".
>
> On FreeBSD, neither bash nor sh produce "hi", but zsh and ksh93 do.
> Same on DragonFly, macOS and Arch Linux. On 7th edition, I only tried
> sh; empty output. 4.3BSD gives empty output for /bin/sh. For kicks,
> neither tcsh nor csh understand `read`; I don't think that *csh has a
> read-like builtin.
>
> This is, of course, a silly example. Most script authors would do
> something like:
>
>     bummer=`echo ' hi ' | sed 's/^ *//;s/ *$//'`
>
> Or, perhaps even:
>
>     bummer=$(echo ' hi ' | sed 's/^ *//;s/ *$//')
>
> Life's too short and at this point I prefer the latter, but more often
> than not my fingers are wired for the former.
>
> > It is really unfortunate that infix | is so beguiling!
> > Except for that a Lisp or Scheme based shell would've
> > been fine and dandy! :-)/2
>
> Whatever happened to `es` ? I guess it died on the vine. It seemed
> like a nifty idea.
>
> I made a concerted effort to use `rc` as my shell on Unix for a little
> while. Without all the supporting plan9 machinery, though, I gave up.
> Minimalism is great in an environment that can augment it with useful
> functionality; outside of that environment, it can be kind of a pain
> for work-a-day use.
>
>         - Dan C.
>

[-- Attachment #2: Type: text/html, Size: 4847 bytes --]

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-04 16:15                           ` Dan Cross
  2023-01-04 18:28                             ` ron minnich
@ 2023-01-04 19:33                             ` Chet Ramey
  1 sibling, 0 replies; 121+ messages in thread
From: Chet Ramey @ 2023-01-04 19:33 UTC (permalink / raw)
  To: Dan Cross, Bakul Shah; +Cc: The Eunuchs Hysterical Society

On 1/4/23 11:15 AM, Dan Cross wrote:
> On Tue, Jan 3, 2023 at 11:16 PM Bakul Shah <bakul@iitbombay.org> wrote:
>> On Jan 3, 2023, at 7:31 PM, Dan Cross <crossd@gmail.com> wrote:
>>> On Tue, Jan 3, 2023 at 10:22 PM John Cowan <cowan@ccil.org> wrote:
>>>> Making shell scripts portable means not using pipelines, because given "foo | bar", kshNN and zsh execute foo in a subshell and bar in the top-level shell, whereas in other shells, both foo and bar execute in subshells.  (For this reason, Posix allows either behavior.)  Not having pipelines is a pretty drastic limitation.
>>>
>>> This came up at work just the other day:
>>>
>>> echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
>>> echo $bummer
>>>
>>> The behavior varies between ksh, zsh, bash, sh. Yay.
>>
>> On this example sh, zsh, bash behave the same way on
>> freebsd. Rather than ban |, I think the lesson is to
>> *avoid* builtin commands in a pipeline that can affect
>> shell's environment. I only write /bin/sh scripts in
>> any case.
> 
> Interesting. In my testing, `bash` was the odd man out:
> 
> : doctor; zsh
> : doctor; echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
> echo $bummer
> 
> hi
> : doctor;
> : doctor; bash
> : doctor; echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
> echo $bummer
> 
> : doctor;
> exit
> : doctor; ksh
> : doctor; echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
> : doctor; echo $bummer
> hi
> : doctor;
> : doctor; sh
> : doctor; echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
> : doctor; echo $bummer
> hi
> : doctor;
> : doctor;
> 
> That's on illumos.

On Illumos, /bin/sh and /usr/xpg4/bin/sh are both links to ksh93.

ksh88/ksh93 and zsh are the only shells that unconditionally run the last
element of a pipeline in the current shell. Bash will do it if job control
is not enabled and the `lastpipe' option is enabled.

It's usually not a problem, but you do have to be aware of potential side
effects.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
		 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/


^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: Command Line Editing in the Terminal Driver
  2023-01-02 20:33         ` [TUHS] Re: Command Line Editing in the Terminal Driver Lars Brinkhoff
@ 2023-01-11  2:51           ` Chris Hanson
  2023-01-11  3:53             ` Greg 'groggy' Lehey
  0 siblings, 1 reply; 121+ messages in thread
From: Chris Hanson @ 2023-01-11  2:51 UTC (permalink / raw)
  To: Lars Brinkhoff
  Cc: Joseph Holsten, Tautological Eunuch Horticultural Scythians

On Jan 2, 2023, at 12:33 PM, Lars Brinkhoff <lars@nocrew.org> wrote:
> 
> Joseph Holsten wrote:
>> Anyone seen any other “terminal drivers” that provide a pleasant
>> alternative to line editing?
> 
> The Stanford AI lab PDP-10 operating system had line editing built in,
> and the E editor integrated with this.

Some operating systems like HP's MPE (and probably also Tandem GUARDIAN) just expected the use of block-mode terminals for purposes of I/O efficiency since interactive use was typically via forms-based applications, and got line editing somewhat as a side-effect.

  -- Chris


^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: Command Line Editing in the Terminal Driver (Was: A few comments on porting the Bourne shell)
  2023-01-03 16:53         ` Marshall Conover
  2023-01-04  9:18           ` Joseph Holsten
@ 2023-01-11  2:56           ` Chris Hanson
  1 sibling, 0 replies; 121+ messages in thread
From: Chris Hanson @ 2023-01-11  2:56 UTC (permalink / raw)
  To: Marshall Conover
  Cc: Joseph Holsten, Tautological Eunuch Horticultural Scythians

[-- Attachment #1: Type: text/plain, Size: 1008 bytes --]

On Jan 3, 2023, at 8:53 AM, Marshall Conover <marzhall.o@gmail.com> wrote:
> 
> With these notebooks, users type text into blocks, and then can have these blocks evaluated by some underlying engine, with the results of the evaluation showing up in a block underneath where they entered text. Results can be not only text, but things like rendered charts and images. There may be GUI-interactable rendered elements as well, but I'm unsure.

This is basically the model created by Symbolics for Dynamic Windows, which was the basis for CLIM, and which I believe they also adopted in the Symbolics distribution of MACSYMA. Kalman Reti does a great job demonstrating it under OpenGenera here: https://www.youtube.com/watch?v=o4-YnLpLgtk

(Jupyter notebooks are a derivative of Mathematica notebooks, which bear more than a passing resemblance to the Dynamic Windows stuff just like Mathematica bears more than a passing resemblance to McCarthy's "M-expressions" and the MACSYMA system…)

  -- Chris


[-- Attachment #2: Type: text/html, Size: 1773 bytes --]

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: Command Line Editing in the Terminal Driver
  2023-01-11  2:51           ` Chris Hanson
@ 2023-01-11  3:53             ` Greg 'groggy' Lehey
  0 siblings, 0 replies; 121+ messages in thread
From: Greg 'groggy' Lehey @ 2023-01-11  3:53 UTC (permalink / raw)
  To: Chris Hanson; +Cc: Joseph Holsten, Tautological Eunuch Horticultural Scythians

[-- Attachment #1: Type: text/plain, Size: 1300 bytes --]

On Tuesday, 10 January 2023 at 18:51:07 -0800, Chris Hanson wrote:
> On Jan 2, 2023, at 12:33 PM, Lars Brinkhoff <lars@nocrew.org> wrote:
>> The Stanford AI lab PDP-10 operating system had line editing built in,
>> and the E editor integrated with this.
>
> Some operating systems like HP's MPE (and probably also Tandem
> GUARDIAN) just expected the use of block-mode terminals for purposes
> of I/O efficiency since interactive use was typically via
> forms-based applications, and got line editing somewhat as a
> side-effect.

Guardian was interesting.  The normal connection (from my point of
view, anyway) was via a TATM (Tandem Asynchronous Terminal
Multiplexer), which buffered the characters, but it didn't interrupt
the processor until one of 4 configurable characters were entered
(typically CR, abort, end of input).  This didn't offer line editing,
and the workarounds were painful.

There was also a block mode, mainly for end user applications, though
the “full screen” editor also used it.

Greg
--
Sent from my desktop computer.
Finger grog@lemis.com for PGP public key.
See complete headers for address and phone numbers.
This message is digitally signed.  If your Microsoft mail program
reports problems, please read http://lemis.com/broken-MUA.php

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 163 bytes --]

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-05 13:22     ` Tom Ivar Helbekkmo via TUHS
@ 2023-01-05 21:11       ` Rob Pike
  0 siblings, 0 replies; 121+ messages in thread
From: Rob Pike @ 2023-01-05 21:11 UTC (permalink / raw)
  To: Tom Ivar Helbekkmo; +Cc: TUHS main list

[-- Attachment #1: Type: text/plain, Size: 1144 bytes --]

In the v8 shell, the "whatis" command's output is all parseable by the
shell to recreate the item being identified. For instance, "whatis HOME"
would print

   HOME=/usr/rob

while "whatis cd"

would print

   builtin cd

and so on. I failed to convince POSIX or other forces to adopt this, and to
this day remain puzzled why people couldn't see its value, even though of
course not having an editable typescript does limit the flexibility
somewhat.

Plan 9 happened soon after.

-rob


On Fri, Jan 6, 2023 at 12:22 AM Tom Ivar Helbekkmo via TUHS <tuhs@tuhs.org>
wrote:

> Jon Steinhart <jon@fourwinds.com> writes:
>
> > Wow!  Small programs that do one thing and do it well connected together
> to
> > do more complicated things?  What an awesome idea :-)
>
> *grin*
>
> Sometimes people call it "screen scraping", and seem to think it's wrong
> to desire parseable output from commands.  I like to point them at this:
>
> https://www.youtube.com/watch?v=tc4ROCJYbm0&t=358
>
> -tih
> --
> Most people who graduate with CS degrees don't understand the significance
> of Lisp.  Lisp is the most important idea in computer science.  --Alan Kay
>

[-- Attachment #2: Type: text/html, Size: 2841 bytes --]

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-03 17:19   ` Jon Steinhart
@ 2023-01-05 13:22     ` Tom Ivar Helbekkmo via TUHS
  2023-01-05 21:11       ` Rob Pike
  0 siblings, 1 reply; 121+ messages in thread
From: Tom Ivar Helbekkmo via TUHS @ 2023-01-05 13:22 UTC (permalink / raw)
  To: Jon Steinhart; +Cc: TUHS main list

Jon Steinhart <jon@fourwinds.com> writes:

> Wow!  Small programs that do one thing and do it well connected together to
> do more complicated things?  What an awesome idea :-)

*grin*

Sometimes people call it "screen scraping", and seem to think it's wrong
to desire parseable output from commands.  I like to point them at this:

https://www.youtube.com/watch?v=tc4ROCJYbm0&t=358

-tih
-- 
Most people who graduate with CS degrees don't understand the significance
of Lisp.  Lisp is the most important idea in computer science.  --Alan Kay

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-04  1:29   ` Adam Thornton
@ 2023-01-05  1:51     ` Alejandro Colomar
  0 siblings, 0 replies; 121+ messages in thread
From: Alejandro Colomar @ 2023-01-05  1:51 UTC (permalink / raw)
  To: Adam Thornton, Dan Cross; +Cc: Douglas McIlroy, TUHS main list

On 1/4/23 02:29, Adam Thornton wrote:
>> On Jan 3, 2023, at 10:26 AM, Dan Cross <crossd@gmail.com> wrote:
>>
>>
>> A few years ago, I was having lunch with some folks from the Go team
>> and one of them remarked, "you shouldn't write a shell script that's
>> longer than about 10 lines. Once you do, it's time to rewrite it in a
>> real programming language." I was a bit taken aback, but they had a
>> point. I'll note that Go standardized on using bash everywhere.


I have many counterexamples that take more than 10 lines.  I'll show a couple 
that are very useful to me:


#  man_section()  prints specific manual page sections (DESCRIPTION, SYNOPSIS,
# ...) of all manual pages in a directory (or in a single manual page file).
# Usage example:  .../man-pages$ man_section man2 SYNOPSIS 'SEE ALSO';

man_section()
{
	if [ $# -lt 2 ]; then
		>&2 echo "Usage: ${FUNCNAME[0]} <dir> <section>...";
		return $EX_USAGE;
	fi

	local page="$1";
	shift;
	local sect="$*";

	find "$page" -type f \
	|xargs wc -l \
	|grep -v -e '\b1 ' -e '\btotal\b' \
	|awk '{ print $2 }' \
	|sort \
	|while read -r manpage; do
		(sed -n '/^\.TH/,/^\.SH/{/^\.SH/!p}' <"$manpage";
		 for s in $sect; do
			<"$manpage" \
			sed -n \
				-e "/^\.SH $s/p" \
				-e "/^\.SH $s/,/^\.SH/{/^\.SH/!p}";
		 done;) \
		|man -P cat -l - 2>/dev/null;
	done;
}


I don't think it's unreadable by being too long, and in every other language it 
would take a lot more work to implement.  And here's another one, which I used a 
lot until I wrote grepc(1)[1]:


#  grep_syscall()  finds the prototype of a syscall in the kernel sources,
# printing the filename, line number, and the prototype.
# It should be run from the root of the linux kernel source tree.
# Usage example:  .../linux$ grep_syscall openat2;
#
# See also: grepc(1)

grep_syscall()
{
	if [ $# -ne 1 ]; then
		>&2 echo "Usage: ${FUNCNAME[0]} <syscall>";
		return $EX_USAGE;
	fi

	find ./* -type f \
	|grep '\.c$' \
	|xargs grep -l "$1" \
	|sort \
	|xargs pcregrep -Mn "(?s)^\w*SYSCALL_DEFINE.\($1\b.*?\)" \
	|sed -E 's/^[^:]+:[0-9]+:/&\n/';

	find ./* -type f \
	|grep '\.[ch]$' \
	|xargs grep -l "$1" \
	|sort \
	|xargs pcregrep -Mn "(?s)^asmlinkage\s+[\w\s]+\**sys_$1\s*\(.*?\)" \
	|sed -E 's/^[^:]+:[0-9]+:/&\n/';
}


Cheers,
Alex


[1]:  <http://www.alejandro-colomar.es/src/alx/alx/grepc.git/>


> 
> 
> My number is larger than 10, but it's smaller than 100.

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-05  0:06                   ` John Cowan
@ 2023-01-05  0:41                     ` Adam Thornton
  0 siblings, 0 replies; 121+ messages in thread
From: Adam Thornton @ 2023-01-05  0:41 UTC (permalink / raw)
  To: The Eunuchs Hysterical Society

[-- Attachment #1: Type: text/plain, Size: 1158 bytes --]

I was today years old when I learned that GNU sort has a -h option.  Which
doesn't mean "help" but means "use human-friendly units."

Except I sort of derived its existence from first principles.  Someone
wanted a way to find disk hogs (they were on a Linux box), but also wanted
SI units, and I said to myself, "self, doesn't that sound like exactly the
sort of baroque and vaguely nauseating option that GNU would add to sort?"
Turns out it is.

Well, I mean, the MacOS man page just says that -h is an extension to POSIX
(yes, it's there on MacOS too), but I will eat a small hat of mine if it
didn't come to sort via GNU.

Adam

On Wed, Jan 4, 2023 at 5:07 PM John Cowan <cowan@ccil.org> wrote:

>
>
> On Wed, Jan 4, 2023 at 3:47 PM Alejandro Colomar <alx.manpages@gmail.com>
> wrote:
>
> I wish shells didn't supoprt globbing, and that glob(1) would be a
>> standalone
>> program still today.  It would simplify much of the quoting issues if
>> most
>> characters were just characters to the shell.
>>
>
> I like the way the rc shell works: globbing is done, but any word in
> single quotes suppresses all interpretation (double quote has no meaning)
>

[-- Attachment #2: Type: text/html, Size: 2152 bytes --]

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-04 20:46                 ` Alejandro Colomar
@ 2023-01-05  0:06                   ` John Cowan
  2023-01-05  0:41                     ` Adam Thornton
  0 siblings, 1 reply; 121+ messages in thread
From: John Cowan @ 2023-01-05  0:06 UTC (permalink / raw)
  To: Alejandro Colomar; +Cc: Bakul Shah, tuhs

[-- Attachment #1: Type: text/plain, Size: 433 bytes --]

On Wed, Jan 4, 2023 at 3:47 PM Alejandro Colomar <alx.manpages@gmail.com>
wrote:

I wish shells didn't supoprt globbing, and that glob(1) would be a
> standalone
> program still today.  It would simplify much of the quoting issues if most
> characters were just characters to the shell.
>

I like the way the rc shell works: globbing is done, but any word in single
quotes suppresses all interpretation (double quote has no meaning)

[-- Attachment #2: Type: text/html, Size: 1017 bytes --]

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-04 18:01               ` Bakul Shah
@ 2023-01-04 20:46                 ` Alejandro Colomar
  2023-01-05  0:06                   ` John Cowan
  0 siblings, 1 reply; 121+ messages in thread
From: Alejandro Colomar @ 2023-01-04 20:46 UTC (permalink / raw)
  To: Bakul Shah, Ralph Corderoy; +Cc: tuhs

Hello Bakul, Ralph,

On 1/4/23 19:01, Bakul Shah wrote:
> On Jan 4, 2023, at 7:19 AM, Ralph Corderoy <ralph@inputplus.co.uk> wrote:
>>
>> A lot of the time, POSIX find's ‘-exec foo {} +’ suffices and runs
>> foo with as many arguments as will just fit under argv[]s limits,
>> like xargs by default and unlike find's one-at-a-time behaviour with
>> ‘-exec foo {} \;’.
> 
> I often run further transformations before executing
> some command on selected files. For example
> 
> find . -type f -name '*.[csh]' | grep -l foo | xargs wc -l

I find find(1)'s options too complex.  I prefer composing even more.  I very 
often find myself writing:

     find . -type f | grep '\.[ch]$' | grep -l foo | xargs ...

> 
> Composability rules!

Yup it does :)

For me, the only shell feature that matters is the pipe.  I also like using 
bash(1) for it's pipefail feature (and I also usually enable lastpipe).  But 
things like globbing and regex support, I don't like them at all.

I wish shells didn't supoprt globbing, and that glob(1) would be a standalone 
program still today.  It would simplify much of the quoting issues if most 
characters were just characters to the shell.

Cheers,

Alex

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-04 15:19             ` Ralph Corderoy
@ 2023-01-04 18:01               ` Bakul Shah
  2023-01-04 20:46                 ` Alejandro Colomar
  0 siblings, 1 reply; 121+ messages in thread
From: Bakul Shah @ 2023-01-04 18:01 UTC (permalink / raw)
  To: Ralph Corderoy; +Cc: tuhs

On Jan 4, 2023, at 7:19 AM, Ralph Corderoy <ralph@inputplus.co.uk> wrote:
> 
> A lot of the time, POSIX find's ‘-exec foo {} +’ suffices and runs
> foo with as many arguments as will just fit under argv[]s limits,
> like xargs by default and unlike find's one-at-a-time behaviour with
> ‘-exec foo {} \;’.

I often run further transformations before executing
some command on selected files. For example

find . -type f -name '*.[csh]' | grep -l foo | xargs wc -l

Composability rules!

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-04  1:58           ` Adam Thornton
@ 2023-01-04 15:19             ` Ralph Corderoy
  2023-01-04 18:01               ` Bakul Shah
  0 siblings, 1 reply; 121+ messages in thread
From: Ralph Corderoy @ 2023-01-04 15:19 UTC (permalink / raw)
  To: tuhs

Hi Adam,

> Bakul Shah wrote:
> > In fact I use find (or zsh's **/*)

bash(1) also has ‘**’ if ‘shopt -s globstar’.

> I was 50 years old before I learned that sort has a -u, so I could
> have saved the pipe-through-uniq step.

Did you never have a Unix with fine man pages to read?  :-)  Before info(1).
Or ‘The Unix Programming Environment’ mentions sort(1)'s -u on page 106.

> I also keep getting told grep has a -r but my finger macros won't stop
> typing find . -name \*.py | xargs ...

‘... **/*.py’ using the ‘**’ Bakul mentions above.

> (And yes, it's GNU find just about everywhere, so I can do -print0 |
> xargs -0 and not have to get fancy with the -I)

A lot of the time, POSIX find's ‘-exec foo {} +’ suffices and runs
foo with as many arguments as will just fit under argv[]s limits,
like xargs by default and unlike find's one-at-a-time behaviour with
‘-exec foo {} \;’.

> "I've never seen anyone use a GUI just as a way to keep a
>  dozen terminal windows open side by side before."

Clearly, you're a trendsetter.  :-)  Tiling window managers are popular,
e.g. https://en.wikipedia.org/wiki/I3_(window_manager)

-- 
Cheers, Ralph.

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-03 23:03       ` ron minnich
  2023-01-04  1:37         ` Bakul Shah
@ 2023-01-04  5:05         ` Theodore Ts'o
  1 sibling, 0 replies; 121+ messages in thread
From: Theodore Ts'o @ 2023-01-04  5:05 UTC (permalink / raw)
  To: ron minnich; +Cc: TUHS main list

On Tue, Jan 03, 2023 at 03:03:15PM -0800, ron minnich wrote:
> I learned a lesson: people love their shell scripting languages. Nobody
> wanted to script with Go. It made me sad, but that's how it Go-es.

Speaking as someone who has written and maintaining thousands of lines
of bash shell script to support file system testing[1] for Linux,
there's a really simple reason for it.  In order to manipulate GCE
VM's, it's just **way** easier to do it from a shell.

[1] https://thunk.org/gce-xfstests

Compare:

	gcloud compute instances create VM_NAME \
	    [--image=IMAGE | --image-family=IMAGE_FAMILY] \
	    --image-project=IMAGE_PROJECT
	    --machine-type=MACHINE_TYPE

with the equivalent in in other "real languages", per [2].  It's 53
lines of C++, 64 lines of Go code, or 100 lines in Java, or 242(!)
lines in Python.

[2] https://cloud.google.com/compute/docs/instances/create-start-instance#expandable-1

Sure, it might a few thousands lines of codes of bash script.  But it
would probably be tens of thousands of lines in Go or Python.
Furthermore, when I'm trying to experiment with how to experiment with
various GCE SDK interfaces, using gcloud and shell is way, way faster
than trying to experiment with writing go code, multiple rounds of
trying to compile the go code, fixing compile syntax errors, etc ---
versus typing a few hundreds characters of a single gcloud command
line, and adjusting CLI options untll it does what you want.

For my use case, I **really** only care about it running on Linux,
since it's being used to test Linux file system kernel code.  I have a
job I'm trying to get done, which is to write reliable ext4 file
system code.  Implementing kvm-xfstests[2], gce-xfstests[3], and
android-xfstests[4] are just a means to an end.   

[2] https://github.com/tytso/xfstests-bld/blob/master/Documentation/kvm-quickstart.md
[3] https://github.com/tytso/xfstests-bld/blob/master/Documentation/gce-xfstests.md
[4] https://github.com/tytso/xfstests-bld/blob/master/Documentation/android-xfstests.md

My time is valuable, and I'm not getting paid to write a general file
system test framework that could potentially work on FreeBSD or
NetBSD.  If my developer velocity is faster by implementing it in
bash, that's what I'm going to do, since time spent rewriting
gce-xfstests in go or Python is time that I can't spend doing work on
the ext4 file system.

If someone wants to try to reimplementing my bash scripts in go, they
should feel free to send a patch.  :-)

						- Ted


^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-04  1:37         ` Bakul Shah
@ 2023-01-04  1:58           ` Adam Thornton
  2023-01-04 15:19             ` Ralph Corderoy
  0 siblings, 1 reply; 121+ messages in thread
From: Adam Thornton @ 2023-01-04  1:58 UTC (permalink / raw)
  To: Bakul Shah, TUHS main list

[-- Attachment #1: Type: text/plain, Size: 1008 bytes --]



> On Jan 3, 2023, at 6:37 PM, Bakul Shah <bakul@iitbombay.org> wrote:
> 
> I write shell one liners all the time[1].
> 
> Go is just too verbose for scripting.  I use it for larger programs but typically these don't start out as shell scripts.
> 
> [1] In fact I use find (or zsh's **/*),  xargs, grep, sort, uniq, field picking often enough that I wish they were built-in a shell but  I haven't figured out a simple syntax for it as yet.

I was 50 years old before I learned that sort has a -u, so I could have saved the pipe-through-uniq step.

I also keep getting told grep has a -r but my finger macros won't stop typing find . -name \*.py | xargs ...

(And yes, it's GNU find just about everywhere, so I can do -print0 | xargs -0 and not have to get fancy with the -I)

But also, crap, it was 25, count 'em, 25 years ago that a friend looked at my workflow and said "I've never seen anyone use a GUI just as a way to keep a dozen terminal windows open side by side before."

Adam


[-- Attachment #2: Type: text/html, Size: 1717 bytes --]

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-03 23:03       ` ron minnich
@ 2023-01-04  1:37         ` Bakul Shah
  2023-01-04  1:58           ` Adam Thornton
  2023-01-04  5:05         ` Theodore Ts'o
  1 sibling, 1 reply; 121+ messages in thread
From: Bakul Shah @ 2023-01-04  1:37 UTC (permalink / raw)
  To: ron minnich; +Cc: TUHS main list

[-- Attachment #1: Type: text/plain, Size: 6123 bytes --]

I write shell one liners all the time[1].

Go is just too verbose for scripting.  I use it for larger programs but typically these don't start out as shell scripts.

[1] In fact I use find (or zsh's **/*),  xargs, grep, sort, uniq, field picking often enough that I wish they were built-in a shell but  I haven't figured out a simple syntax for it as yet.

> On Jan 3, 2023, at 3:03 PM, ron minnich <rminnich@gmail.com> wrote:
> 
> I apologize for this too long message, but context in this case matters. 
> 
> tl;dr: I agree with the Go folks: past about 10 lines, write a program. But not everybody will agree, as I learned with u-root. Many people love shell programming. I never have.
> 
> u-root is a set of Go programs deployed on a couple million data center systems at many companies, including Google.
> 
> u-root originally used a compile-on-demand model: type a command name, if it's not in /ubin, it gets built. This was fast, in early Go, typically 250 ms or so. And, in the early days of Go, the Go toolchain, u-root, and all support source easily fit in a 16M SPI part.
> 
> The original scripting language for u-root was Go. 
> 
> There were two commands for this: script and builtin. script would run Go fragments; builtin would take supplied commands and build a custom shell with those built in. Each took 250ms to run.
> 
> script took what we called a 'Go fragment', wrapped it with boiler plate, compiled it, and ran it. 
> e.g.
> script fmt.Printf("hi\n")
> would build the program and run that code. 
> So you could, e.g,, do math:
> script fmt.Printf("%d\n", 6*7)
> 
> It could get complex: to see things about interfaces:
> script 'ifaces, _ := net.Interfaces()
> for _, v := range ifaces {
> addrs, _ := v.Addrs()
> Printf("%v has %v", v, addrs)
> }'
> 
> you'd get:
> ip: {1 1500 lo  up|loopback} has [127.0.0.1/8 <http://127.0.0.1/8> ::1/128]
> ip: {5 1500 eth0 fa:42:2c:d4:0e:01 up|broadcast} has [172.17.0.2/16 <http://172.17.0.2/16> fe80::f842:2cff:fed4:e01/64]
> 
> The second command was called builtin. It did not work as other shells do: it built a new shell for you. So, you type:
> bulitin hi fmt.Printf("hi") there fmt.Printf("there\n")
> 
> builtin would convert the Go fragments to functions callable in the u-root shell, build a private name space (on Linux or Plan 9 anyway), rebuild the shell with those new functions, and at that point:
> you type
> hi
> in the shell, and it types
> hi
> back. This was built in to your private shell in your private name space. Once you left the shell, it was gone.
> Again, this process of creating and starting the new shell always took about 250 ms (in Go 1.2 that is).
> 
> I learned a lesson: people love their shell scripting languages. Nobody wanted to script with Go. It made me sad, but that's how it Go-es.
> 
> ron
> p.s. the 'script' command is still available as an experimental u-root command. Source mode is now independent: github.com/u-root/sourcery <http://github.com/u-root/sourcery>
> 
> 
> On Tue, Jan 3, 2023 at 2:35 PM Steffen Nurpmeso <steffen@sdaoden.eu <mailto:steffen@sdaoden.eu>> wrote:
>> Steve Nickolas wrote in
>>  <alpine.DEB.2.21.2301031307001.9988@sd-119843.dedibox.fr <mailto:alpine.DEB.2.21.2301031307001.9988@sd-119843.dedibox.fr>>:
>>  |On Tue, 3 Jan 2023, Dan Cross wrote:
>>  |> Something I've noticed is that lots of people try to increase
>>  |> complexity to solve problems, and it rarely occurs to them to
>>  |> eliminate complexity to solve problems. Sometimes the reasons for this
>>  |> are good; most of the time they are not.
>>  |>
>>  |>        - Dan C.
>>  |
>>  |I think of the saying: "Perfection is not when there is nothing left to 
>>  |add, but when there is nothing left to remove."
>> 
>> He (Exupéry) was then shot down.
>> 
>> I always seem to response this to that.
>> Hmm, openpgp@ietf.org <mailto:openpgp@ietf.org> (to which i have almost zero to add
>> technically shall someone think that, nor do i want) lastly
>> 
>>    |"Perfection is achieved not when there is nothing more to add but when \
>>    |there
>>    |is nothing left to take away" - Antoine de Saint-Exupéry.
>> 
>>   He was then shot down.
>> 
>> But yes, he then really went missing.
>> 
>> The topic ..
>> I do not miss times where suddenly a shell script breaks because
>> ": > FILE" does not work (just recently 'realized from reading code
>> of Paul Eggert of GNU/IANA TZ, "hey, > FILE" is of course
>> sufficient!"), i fixed it via "printf '' > FILE" by then; whatever
>> the reason.  May it be bugs or (local) miscompilations, not
>> detected due to missing unit tests and a too small user base.
>> 
>> Portable?  If i find /usr/xpg4/bin i quickly add it to $PATH for
>> the much better awk (but beware of documented double expansion
>> issues) and the much much better sh(1).
>> Some things just require that, noclobber I/O redirection (set -C)
>> for example.  (mktemp(1) is still not part of the POSIX standard.)
>> 
>> Besides i miss(ed) the history; the author of bmake (and
>> verieexec) just last year asked me why i would use stty for
>> a purpose ("(<&1 >/dev/null stty -a) 2>/dev/null") instead of
>> simply using "[ -t 1 ]", and indeed, i found that as soon as BSD
>> 4.1 and Research V7, but it surprised me.
>> Without an oversight of the history and the lack of many systems
>> to test, perl(1) was omnipresent and if only for OpenSSL and so
>> using it for almost anything seemed save.
>> 
>>   To love is not to look at one another: it is to look,
>>   together, in the same direction.  Antoine de Saint-Exupéry.
>> 
>> A happy and healthy new Year 2023 is now overdue.
>> Even Giants have to die, but with holding hands it can wait a bit
>> longer, i hope.
>> I wish that from Germany to all of you, and deliberately beyond
>> NATO readership.
>> 
>> --steffen
>> |
>> |Der Kragenbaer,                The moon bear,
>> |der holt sich munter           he cheerfully and one by one
>> |einen nach dem anderen runter  wa.ks himself off
>> |(By Robert Gernhardt)


[-- Attachment #2: Type: text/html, Size: 7625 bytes --]

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-03 17:26 ` Dan Cross
  2023-01-03 18:07   ` Steve Nickolas
  2023-01-03 18:19   ` Niklas Karlsson
@ 2023-01-04  1:29   ` Adam Thornton
  2023-01-05  1:51     ` Alejandro Colomar
  2 siblings, 1 reply; 121+ messages in thread
From: Adam Thornton @ 2023-01-04  1:29 UTC (permalink / raw)
  To: Dan Cross; +Cc: Douglas McIlroy, TUHS main list



> On Jan 3, 2023, at 10:26 AM, Dan Cross <crossd@gmail.com> wrote:
> 
> 
> A few years ago, I was having lunch with some folks from the Go team
> and one of them remarked, "you shouldn't write a shell script that's
> longer than about 10 lines. Once you do, it's time to rewrite it in a
> real programming language." I was a bit taken aback, but they had a
> point. I'll note that Go standardized on using bash everywhere.


My number is larger than 10, but it's smaller than 100.

Lord knows I have plenty of things that started as a pipe and-or-for-loop at the command line, that became a shell script, that grew some arguments, that got beefier, and that I then rewrote in Perl or Python and cursed myself for not having done it that way from the start.

The RSP Notebook I mentioned on COFF is about to have its installer--which is a hodgepodge of shell scripts and Python scripts--rewritten as reasonable Python classes with typing and test cases and everything, because it's grown enough that it's hard to understand in its current form.

Adam

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-03 20:19     ` Steffen Nurpmeso
@ 2023-01-03 23:03       ` ron minnich
  2023-01-04  1:37         ` Bakul Shah
  2023-01-04  5:05         ` Theodore Ts'o
  0 siblings, 2 replies; 121+ messages in thread
From: ron minnich @ 2023-01-03 23:03 UTC (permalink / raw)
  To: Steve Nickolas, TUHS main list

[-- Attachment #1: Type: text/plain, Size: 5332 bytes --]

I apologize for this too long message, but context in this case matters.

tl;dr: I agree with the Go folks: past about 10 lines, write a program. But
not everybody will agree, as I learned with u-root. Many people love shell
programming. I never have.

u-root is a set of Go programs deployed on a couple million data center
systems at many companies, including Google.

u-root originally used a compile-on-demand model: type a command name, if
it's not in /ubin, it gets built. This was fast, in early Go, typically 250
ms or so. And, in the early days of Go, the Go toolchain, u-root, and all
support source easily fit in a 16M SPI part.

The original scripting language for u-root was Go.

There were two commands for this: script and builtin. script would run Go
fragments; builtin would take supplied commands and build a custom shell
with those built in. Each took 250ms to run.

script took what we called a 'Go fragment', wrapped it with boiler plate,
compiled it, and ran it.
e.g.
script fmt.Printf("hi\n")
would build the program and run that code.
So you could, e.g,, do math:
script fmt.Printf("%d\n", 6*7)

It could get complex: to see things about interfaces:
script 'ifaces, _ := net.Interfaces()
for _, v := range ifaces {
addrs, _ := v.Addrs()
Printf("%v has %v", v, addrs)
}'

you'd get:
ip: {1 1500 lo  up|loopback} has [127.0.0.1/8 ::1/128]
ip: {5 1500 eth0 fa:42:2c:d4:0e:01 up|broadcast} has [172.17.0.2/16
fe80::f842:2cff:fed4:e01/64]

The second command was called builtin. It did not work as other shells do:
it built a new shell for you. So, you type:
bulitin hi fmt.Printf("hi") there fmt.Printf("there\n")

builtin would convert the Go fragments to functions callable in the u-root
shell, build a private name space (on Linux or Plan 9 anyway), rebuild the
shell with those new functions, and at that point:
you type
hi
in the shell, and it types
hi
back. This was built in to your private shell in your private name space.
Once you left the shell, it was gone.
Again, this process of creating and starting the new shell always took
about 250 ms (in Go 1.2 that is).

I learned a lesson: people love their shell scripting languages. Nobody
wanted to script with Go. It made me sad, but that's how it Go-es.

ron
p.s. the 'script' command is still available as an experimental u-root
command. Source mode is now independent: github.com/u-root/sourcery


On Tue, Jan 3, 2023 at 2:35 PM Steffen Nurpmeso <steffen@sdaoden.eu> wrote:

> Steve Nickolas wrote in
>  <alpine.DEB.2.21.2301031307001.9988@sd-119843.dedibox.fr>:
>  |On Tue, 3 Jan 2023, Dan Cross wrote:
>  |> Something I've noticed is that lots of people try to increase
>  |> complexity to solve problems, and it rarely occurs to them to
>  |> eliminate complexity to solve problems. Sometimes the reasons for this
>  |> are good; most of the time they are not.
>  |>
>  |>        - Dan C.
>  |
>  |I think of the saying: "Perfection is not when there is nothing left to
>  |add, but when there is nothing left to remove."
>
> He (Exupéry) was then shot down.
>
> I always seem to response this to that.
> Hmm, openpgp@ietf.org (to which i have almost zero to add
> technically shall someone think that, nor do i want) lastly
>
>    |"Perfection is achieved not when there is nothing more to add but when
> \
>    |there
>    |is nothing left to take away" - Antoine de Saint-Exupéry.
>
>   He was then shot down.
>
> But yes, he then really went missing.
>
> The topic ..
> I do not miss times where suddenly a shell script breaks because
> ": > FILE" does not work (just recently 'realized from reading code
> of Paul Eggert of GNU/IANA TZ, "hey, > FILE" is of course
> sufficient!"), i fixed it via "printf '' > FILE" by then; whatever
> the reason.  May it be bugs or (local) miscompilations, not
> detected due to missing unit tests and a too small user base.
>
> Portable?  If i find /usr/xpg4/bin i quickly add it to $PATH for
> the much better awk (but beware of documented double expansion
> issues) and the much much better sh(1).
> Some things just require that, noclobber I/O redirection (set -C)
> for example.  (mktemp(1) is still not part of the POSIX standard.)
>
> Besides i miss(ed) the history; the author of bmake (and
> verieexec) just last year asked me why i would use stty for
> a purpose ("(<&1 >/dev/null stty -a) 2>/dev/null") instead of
> simply using "[ -t 1 ]", and indeed, i found that as soon as BSD
> 4.1 and Research V7, but it surprised me.
> Without an oversight of the history and the lack of many systems
> to test, perl(1) was omnipresent and if only for OpenSSL and so
> using it for almost anything seemed save.
>
>   To love is not to look at one another: it is to look,
>   together, in the same direction.  Antoine de Saint-Exupéry.
>
> A happy and healthy new Year 2023 is now overdue.
> Even Giants have to die, but with holding hands it can wait a bit
> longer, i hope.
> I wish that from Germany to all of you, and deliberately beyond
> NATO readership.
>
> --steffen
> |
> |Der Kragenbaer,                The moon bear,
> |der holt sich munter           he cheerfully and one by one
> |einen nach dem anderen runter  wa.ks himself off
> |(By Robert Gernhardt)
>

[-- Attachment #2: Type: text/html, Size: 6704 bytes --]

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-03 18:07   ` Steve Nickolas
@ 2023-01-03 20:19     ` Steffen Nurpmeso
  2023-01-03 23:03       ` ron minnich
  0 siblings, 1 reply; 121+ messages in thread
From: Steffen Nurpmeso @ 2023-01-03 20:19 UTC (permalink / raw)
  To: Steve Nickolas; +Cc: TUHS main list

Steve Nickolas wrote in
 <alpine.DEB.2.21.2301031307001.9988@sd-119843.dedibox.fr>:
 |On Tue, 3 Jan 2023, Dan Cross wrote:
 |> Something I've noticed is that lots of people try to increase
 |> complexity to solve problems, and it rarely occurs to them to
 |> eliminate complexity to solve problems. Sometimes the reasons for this
 |> are good; most of the time they are not.
 |>
 |>        - Dan C.
 |
 |I think of the saying: "Perfection is not when there is nothing left to 
 |add, but when there is nothing left to remove."

He (Exupéry) was then shot down.

I always seem to response this to that.
Hmm, openpgp@ietf.org (to which i have almost zero to add
technically shall someone think that, nor do i want) lastly

   |"Perfection is achieved not when there is nothing more to add but when \
   |there
   |is nothing left to take away" - Antoine de Saint-Exupéry.

  He was then shot down.

But yes, he then really went missing.

The topic ..
I do not miss times where suddenly a shell script breaks because
": > FILE" does not work (just recently 'realized from reading code
of Paul Eggert of GNU/IANA TZ, "hey, > FILE" is of course
sufficient!"), i fixed it via "printf '' > FILE" by then; whatever
the reason.  May it be bugs or (local) miscompilations, not
detected due to missing unit tests and a too small user base.

Portable?  If i find /usr/xpg4/bin i quickly add it to $PATH for
the much better awk (but beware of documented double expansion
issues) and the much much better sh(1).
Some things just require that, noclobber I/O redirection (set -C)
for example.  (mktemp(1) is still not part of the POSIX standard.)

Besides i miss(ed) the history; the author of bmake (and
verieexec) just last year asked me why i would use stty for
a purpose ("(<&1 >/dev/null stty -a) 2>/dev/null") instead of
simply using "[ -t 1 ]", and indeed, i found that as soon as BSD
4.1 and Research V7, but it surprised me.
Without an oversight of the history and the lack of many systems
to test, perl(1) was omnipresent and if only for OpenSSL and so
using it for almost anything seemed save.

  To love is not to look at one another: it is to look,
  together, in the same direction.  Antoine de Saint-Exupéry.

A happy and healthy new Year 2023 is now overdue.
Even Giants have to die, but with holding hands it can wait a bit
longer, i hope.
I wish that from Germany to all of you, and deliberately beyond
NATO readership.

--steffen
|
|Der Kragenbaer,                The moon bear,
|der holt sich munter           he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-03 17:26 ` Dan Cross
  2023-01-03 18:07   ` Steve Nickolas
@ 2023-01-03 18:19   ` Niklas Karlsson
  2023-01-04  1:29   ` Adam Thornton
  2 siblings, 0 replies; 121+ messages in thread
From: Niklas Karlsson @ 2023-01-03 18:19 UTC (permalink / raw)
  To: TUHS main list

[-- Attachment #1: Type: text/plain, Size: 2147 bytes --]

Den tis 3 jan. 2023 kl 18:28 skrev Dan Cross <crossd@gmail.com>:

> A few years ago, I was having lunch with some folks from the Go team
> and one of them remarked, "you shouldn't write a shell script that's
> longer than about 10 lines. Once you do, it's time to rewrite it in a
> real programming language." I was a bit taken aback, but they had a
> point. I'll note that Go standardized on using bash everywhere.
>

They wouldn't like my current project. It's a whole bunch of different
shell scripts (and they use bash-isms, since it's inconceivable that anyone
would use them on a non-Linux non-GNU system) calling each other as
modules. The scripts aren't HUGE, but they're certainly a lot more than 10
lines.

I work for an academic supercomputing facility, and we have a corporate
customer using our systems to do aerodynamics calculations. They want a way
to automatically submit jobs on our systems from their own environment,
complete with transferring the source data files to us and the results
files back to them when they're done. They're integrating that with their
own set of scripts (a lot of Python, I believe) so an aerodynamics engineer
can just pick an option from a menu and everything works automagically. I'd
say bash works pretty well for that application. It's a lot of calling
external programs (scp, sbatch (the command to submit a SLURM job, SLURM
being the batch management facility a lot of large compute clusters use to
manage their jobs), etc).

I inherited this set of scripts from someone who quit before I joined in
2020, and have modified them extensively since then to work well with the
corporate customer's scripts and be more generic. It's in a fully working
state at this point, but I still have some feature requests to take care of.

I am more of a sysadmin than a developer, but I certainly can write code
when the situation warrants. Pure Windows sysadmins often can't write code
at all (though there are certainly some who are very conversant with
PowerShell) but I'd say any UNIX/Linux sysadmin worth their salt can at
least write shell scripts, and often Python, or some years back, Perl.

Niklas

[-- Attachment #2: Type: text/html, Size: 2640 bytes --]

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-03 17:26 ` Dan Cross
@ 2023-01-03 18:07   ` Steve Nickolas
  2023-01-03 20:19     ` Steffen Nurpmeso
  2023-01-03 18:19   ` Niklas Karlsson
  2023-01-04  1:29   ` Adam Thornton
  2 siblings, 1 reply; 121+ messages in thread
From: Steve Nickolas @ 2023-01-03 18:07 UTC (permalink / raw)
  To: TUHS main list

On Tue, 3 Jan 2023, Dan Cross wrote:

> Something I've noticed is that lots of people try to increase
> complexity to solve problems, and it rarely occurs to them to
> eliminate complexity to solve problems. Sometimes the reasons for this
> are good; most of the time they are not.
>
>        - Dan C.

I think of the saying: "Perfection is not when there is nothing left to 
add, but when there is nothing left to remove."

-uso.

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-03 15:08 Douglas McIlroy
  2023-01-03 15:57 ` Alejandro Colomar
@ 2023-01-03 17:26 ` Dan Cross
  2023-01-03 18:07   ` Steve Nickolas
                     ` (2 more replies)
  1 sibling, 3 replies; 121+ messages in thread
From: Dan Cross @ 2023-01-03 17:26 UTC (permalink / raw)
  To: Douglas McIlroy; +Cc: TUHS main list

On Tue, Jan 3, 2023 at 10:09 AM Douglas McIlroy
<douglas.mcilroy@dartmouth.edu> wrote:
> The forgotten compilers were typically used to supply glue
> to paste major tools together. The nature of that glue---often
> simple data reformatting--inspired tools like sed and awk.
> Each use of a tool became a process that saved many minutes
> of work that would in a scriptless world be guided by hand,
> boringly and unreliably.

This, I think, begs the question I raised earlier: what precisely is
"the shell"? The command interpreter by itself isn't that interesting
without a supporting toolset. And now we get into the crux or what it
means to be "portable": what is the minimum set of such tools that
allow us to write portable shell scripts?

Again I ask, am I allowed to use `nawk`? How about '#!' lines? xargs?
cut? paste?

Perhaps that's the benefit of experience: over time, one develops a
sense of what works across the broadest range of systems that one will
likely encounter. But as Warner pointed out, we can be thankful we're
not using Eunice.

A few years ago, I was having lunch with some folks from the Go team
and one of them remarked, "you shouldn't write a shell script that's
longer than about 10 lines. Once you do, it's time to rewrite it in a
real programming language." I was a bit taken aback, but they had a
point. I'll note that Go standardized on using bash everywhere.

> Yet glue processes typically did only microseconds of
> "real" work. In the name of efficiency, the operations began
> to be incorporated directly into the shell. The first
> inklings of this can be seen in "echo" and various forms
> of variable-substitution making their way into the v7
> shell. The phenomenon proliferated into putting what were
> typically canned sed one-liners (but not sed itself) into
> the shell.

Perhaps I've told this story, but 10 or 12 years ago at Google a
colleague was writing a script to find the location of some resource
or other. I no longer recall what, exactly, but he had written this
massively over-structured thing: shell functions that ran one or two
commands, heavily commented, and he was trying to `return` the exit
status from those commands as the result from the function. It wasn't
working because, well, that's just not how such things work. I sat
down, and rewrote it into about 10 lines; the salient part was that
there was a conditional that determined whether the system it was
running on had a 64-bit or 32-bit userland and ran a handful of
commands depending. I showed him the revised script and he was
uncomfortable; after a while he kind of gave me a look and said,
"yeah, I just don't think people conceptualize of problems that way
anymore." Meaning, it was far more natural for him and others to just
write a Python program instead of a simple script. To him, the script
wasn't simple, but the Python program was.

That has stuck with me for a long, long time. Anyway, we checked the
script in and it worked; it's probably still running, though I suspect
most, if not all, 32-bit systems are gone from Google so perhaps it
was long-ago recycled. I have no idea anymore, but it really gave me
pause: had everything I'd learned about Unix in the prior 20 years
become obsolete? Was I an anachronism amongst my colleagues? Were they
right, and I wrong about how to approach problems?

> Lots of specializations crowded out  universality. A side
> effect was an explosion of knowledge required to write
> or understand code. Such is the tragedy of "forgetting
> compilers".

Something I've noticed is that lots of people try to increase
complexity to solve problems, and it rarely occurs to them to
eliminate complexity to solve problems. Sometimes the reasons for this
are good; most of the time they are not.

        - Dan C.

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-03 15:57 ` Alejandro Colomar
@ 2023-01-03 17:19   ` Jon Steinhart
  2023-01-05 13:22     ` Tom Ivar Helbekkmo via TUHS
  0 siblings, 1 reply; 121+ messages in thread
From: Jon Steinhart @ 2023-01-03 17:19 UTC (permalink / raw)
  To: TUHS main list

Alejandro Colomar writes:
> Well, as time goes by, I'm also writing less and less programs.  But only 
> because I find that I can pipe programs together to do what I want without 
> having to write a new one.  I guess that's a fair reason to not compile :)

Wow!  Small programs that do one thing and do it well connected together to
do more complicated things?  What an awesome idea :-)

Jon

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2023-01-03 15:08 Douglas McIlroy
@ 2023-01-03 15:57 ` Alejandro Colomar
  2023-01-03 17:19   ` Jon Steinhart
  2023-01-03 17:26 ` Dan Cross
  1 sibling, 1 reply; 121+ messages in thread
From: Alejandro Colomar @ 2023-01-03 15:57 UTC (permalink / raw)
  To: Douglas McIlroy, TUHS main list

Hi Doug!

On 1/3/23 16:08, Douglas McIlroy wrote:
>> segaloco via TUHS writes:
>>> I think that's a good point that scripting problems may be
>>> a symptom of the nature of the tools being used in them.
> 
>> I think that you're hinting at something different.
> 
>> To the best of my recollection, scripting languages were originally
>> intended and used for the automation of repetitive personal tasks;
>> making it easier for users who found themselves typing the same
>> stuff over and over again.
> 
> Indeed!

Well, as time goes by, I'm also writing less and less programs.  But only 
because I find that I can pipe programs together to do what I want without 
having to write a new one.  I guess that's a fair reason to not compile :)

> 
>> Somewhere along the line people forgot
>> how to use a compiler and began writing large systems in a variety
>> of roughly equivalent but incompatible interpreted languages.  Can
>> one even boot linux without having several different incompatible
>> versions of Python installed today?  So I don't think that it's the
>> nature of the tools; I think that it's people choosing the wrong
>> tools for the problems that they're trying to solve.
> 
>> Jon
> 
> The forgotten compilers were typically used to supply glue
> to paste major tools together. The nature of that glue---often
> simple data reformatting--inspired tools like sed and awk.
> Each use of a tool became a process that saved many minutes
> of work that would in a scriptless world be guided by hand,
> boringly and unreliably.
> 
> Yet glue processes typically did only microseconds of
> "real" work. In the name of efficiency, the operations began
> to be incorporated directly into the shell. The first
> inklings of this can be seen in "echo" and various forms
> of variable-substitution making their way into the v7
> shell. The phenomenon proliferated into putting what were
> typically canned sed one-liners (but not sed itself) into
> the shell.
> 
> Lots of specializations crowded out  universality. A side
> effect was an explosion of knowledge required to write
> or understand code. Such is the tragedy of "forgetting
> compilers".

And a funny thing:

Today in the era of multi-core computers, not only the shell-only scripts are 
unreadable, but they also are slower than scripts with pipes and little-to-no 
shell features.

I tend to write scripts with only pipes.  No shell features.  And I also avoid 
programs with many options (find(1)) when I can split it with xargs(1) and others.

I remember some script I wrote for maintenance of the Linux man-pages, and I 
received a suggestion that I could "simplify" the script considerably and make 
it "faster" by reducing the number of pipes using some features of find(1) or 
sh(1) (I don't remember well).  Well, I tested, and my long list of piped 
programs outperformed by a fair amount the suggested alternative.

I didn't care enough to find out the reason, but I thought it could be because 
with the pipeline, I could run each small process in a different core, so they 
could work all at the same time.  With a single program invocation, you're 
bottle-necked by that program, which is limited to a single core (normally).


Cheers,

Alex

> 
> Doug

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
@ 2023-01-03 15:08 Douglas McIlroy
  2023-01-03 15:57 ` Alejandro Colomar
  2023-01-03 17:26 ` Dan Cross
  0 siblings, 2 replies; 121+ messages in thread
From: Douglas McIlroy @ 2023-01-03 15:08 UTC (permalink / raw)
  To: TUHS main list

> segaloco via TUHS writes:
>> I think that's a good point that scripting problems may be
>> a symptom of the nature of the tools being used in them.

> I think that you're hinting at something different.

> To the best of my recollection, scripting languages were originally
> intended and used for the automation of repetitive personal tasks;
> making it easier for users who found themselves typing the same
> stuff over and over again.

Indeed!

> Somewhere along the line people forgot
> how to use a compiler and began writing large systems in a variety
> of roughly equivalent but incompatible interpreted languages.  Can
> one even boot linux without having several different incompatible
> versions of Python installed today?  So I don't think that it's the
> nature of the tools; I think that it's people choosing the wrong
> tools for the problems that they're trying to solve.

> Jon

The forgotten compilers were typically used to supply glue
to paste major tools together. The nature of that glue---often
simple data reformatting--inspired tools like sed and awk.
Each use of a tool became a process that saved many minutes
of work that would in a scriptless world be guided by hand,
boringly and unreliably.

Yet glue processes typically did only microseconds of
"real" work. In the name of efficiency, the operations began
to be incorporated directly into the shell. The first
inklings of this can be seen in "echo" and various forms
of variable-substitution making their way into the v7
shell. The phenomenon proliferated into putting what were
typically canned sed one-liners (but not sed itself) into
the shell.

Lots of specializations crowded out  universality. A side
effect was an explosion of knowledge required to write
or understand code. Such is the tragedy of "forgetting
compilers".

Doug

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
  2022-12-31 13:26 Douglas McIlroy
@ 2022-12-31 22:19 ` Rob Pike
  0 siblings, 0 replies; 121+ messages in thread
From: Rob Pike @ 2022-12-31 22:19 UTC (permalink / raw)
  To: Douglas McIlroy; +Cc: TUHS main list

[-- Attachment #1: Type: text/plain, Size: 911 bytes --]

Sam to some extent, but the better expression is the Plan 9 window systems 8½
and rio and their embedded interactive terminal. (Plan 9 has no terminal
support in the kernel beyond bootstrappability.) You just edit as you work
and hit newline to commit. There is a feature called "hold mode" that
disabled commit on newline. It was the way you wrote a mail message: Hold,
type a multiline message, edit all you like, release.

-rob


On Sun, Jan 1, 2023 at 12:28 AM Douglas McIlroy <
douglas.mcilroy@dartmouth.edu> wrote:

> > "Originally the idea of adding command line editing to ksh was
> >  rejected in the hope that line editing would move into the terminal
> >  driver." [2]
>
> > I have always wondered, what such a central terminal driver driven
> > history/line-editing would have felt like.
>
> You can get a feel for it in Rob's "sam" editor, which works that way.
>
> Doug
>

[-- Attachment #2: Type: text/html, Size: 1492 bytes --]

^ permalink raw reply	[flat|nested] 121+ messages in thread

* [TUHS] Re: A few comments on porting the Bourne shell
@ 2022-12-31 13:26 Douglas McIlroy
  2022-12-31 22:19 ` Rob Pike
  0 siblings, 1 reply; 121+ messages in thread
From: Douglas McIlroy @ 2022-12-31 13:26 UTC (permalink / raw)
  To: Sven Mascheck, TUHS main list

> "Originally the idea of adding command line editing to ksh was
>  rejected in the hope that line editing would move into the terminal
>  driver." [2]

> I have always wondered, what such a central terminal driver driven
> history/line-editing would have felt like.

You can get a feel for it in Rob's "sam" editor, which works that way.

Doug

^ permalink raw reply	[flat|nested] 121+ messages in thread

end of thread, other threads:[~2023-01-11  3:53 UTC | newest]

Thread overview: 121+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-12-30 18:25 [TUHS] A few comments on porting the Bourne shell Paul Ruizendaal
2022-12-30 19:51 ` [TUHS] " Chet Ramey
2022-12-30 20:02   ` Larry McVoy
2022-12-30 20:31     ` Adam Thornton
2022-12-30 20:49       ` Chet Ramey
2022-12-30 20:42     ` Sven Mascheck
2022-12-30 20:48       ` Jon Steinhart
2022-12-30 20:51         ` Sven Mascheck
2022-12-31 11:40         ` Ralph Corderoy
2022-12-31 18:49           ` Jon Steinhart
2022-12-31 19:24             ` Clem Cole
2023-01-03 16:32               ` Chet Ramey
2023-01-01  1:51             ` Ron Natalie
2023-01-02 20:03       ` [TUHS] Command Line Editing in the Terminal Driver (Was: A few comments on porting the Bourne shell) Joseph Holsten
2023-01-02 20:33         ` [TUHS] Re: Command Line Editing in the Terminal Driver Lars Brinkhoff
2023-01-11  2:51           ` Chris Hanson
2023-01-11  3:53             ` Greg 'groggy' Lehey
2023-01-02 21:08         ` [TUHS] Re: Command Line Editing in the Terminal Driver (Was: A few comments on porting the Bourne shell) Rob Pike
2023-01-03 16:53         ` Marshall Conover
2023-01-04  9:18           ` Joseph Holsten
2023-01-11  2:56           ` Chris Hanson
2022-12-30 20:47     ` [TUHS] Re: A few comments on porting the Bourne shell Chet Ramey
2022-12-31  0:08     ` Luther Johnson
2022-12-31  3:59       ` Larry McVoy
2022-12-31  4:12         ` Steve Nickolas
2022-12-31  4:18         ` Bakul Shah
2022-12-31  4:40           ` Larry McVoy
2022-12-31  4:19         ` Marc Donner
2022-12-31  4:23         ` Dave Horsfall
2022-12-31  4:37           ` Clem Cole
2023-01-02  5:10           ` Mary Ann Horton
2023-01-02 16:25             ` Clem Cole
2023-01-02 16:51               ` Larry McVoy
2023-01-02 17:32                 ` Adam Thornton
2023-01-02 17:43                   ` Larry McVoy
2023-01-02 17:48                     ` Luther Johnson
2023-01-02 18:00                       ` G. Branden Robinson
2023-01-02 18:05                         ` Luther Johnson
2023-01-02 18:12                         ` Larry McVoy
2023-01-02 18:16                           ` Clem Cole
2023-01-02 19:50                             ` Warner Losh
2023-01-02 20:05                               ` Adam Thornton
2023-01-02 19:21                           ` G. Branden Robinson
2023-01-02 19:34                             ` Rich Salz
2023-01-02 20:12                               ` Larry McVoy
2023-01-02 20:24                               ` G. Branden Robinson
2023-01-02 20:41                                 ` Larry McVoy
2023-01-02 21:00                                   ` Dan Cross
2023-01-02 21:06                                     ` Clem Cole
2023-01-02 21:19                                       ` Dan Cross
2023-01-02 22:54                                         ` segaloco via TUHS
2023-01-02 23:58                                           ` Jon Steinhart
2023-01-04  9:00                                             ` Joseph Holsten
2023-01-02 22:43                                       ` Steve Nickolas
2023-01-02 21:08                                     ` Joseph Holsten
2023-01-02 21:15                                       ` Adam Thornton
2023-01-02 17:55                     ` Adam Thornton
2023-01-02 18:11                       ` Clem Cole
2023-01-02 18:36                         ` Dan Cross
2023-01-02 18:48                           ` Dan Cross
2023-01-02 18:18                       ` Larry McVoy
2023-01-04  3:20                     ` John Cowan
2023-01-04  3:31                       ` Dan Cross
2023-01-04  4:16                         ` Bakul Shah
2023-01-04 16:15                           ` Dan Cross
2023-01-04 18:28                             ` ron minnich
2023-01-04 19:33                             ` Chet Ramey
2023-01-04 15:21                       ` Ralph Corderoy
2023-01-04 15:54                         ` Ron Natalie
2023-01-02 17:55                 ` Clem Cole
2023-01-03 17:08                   ` Paul Winalski
2023-01-03 19:19                     ` Warner Losh
2023-01-03 19:56                       ` Luther Johnson
2023-01-03 20:21                       ` Dave Horsfall
2023-01-03 21:47                       ` Clem Cole
2023-01-03 21:51                         ` Clem Cole
2022-12-31  4:41         ` Greg 'groggy' Lehey
2022-12-30 20:20   ` Sven Mascheck
2022-12-30 20:49     ` Ron Natalie
2022-12-30 20:52       ` Rob Pike
2022-12-30 20:53       ` Jon Steinhart
2023-01-01 10:44   ` arnold
2023-01-01 11:28     ` arnold
2023-01-03 16:34       ` Chet Ramey
2023-01-03 15:06     ` Chet Ramey
2022-12-30 19:57 ` segaloco via TUHS
2022-12-31 12:55   ` Paul Ruizendaal
2023-01-01  2:55     ` Warner Losh
2023-01-01  4:38       ` Jonathan Gray
2023-01-01  5:25         ` Warner Losh
2023-01-01  5:35           ` Dan Cross
2023-01-01  5:52             ` G. Branden Robinson
2023-01-01  6:35               ` Warner Losh
2023-01-01  6:35               ` Rob Pike
2023-01-01  6:27             ` Warner Losh
2023-01-01 14:50             ` Ron Natalie
2023-01-01  7:11           ` Jonathan Gray
2023-01-01  7:21             ` Warner Losh
2023-01-01 10:25           ` Paul Ruizendaal
2022-12-31 13:26 Douglas McIlroy
2022-12-31 22:19 ` Rob Pike
2023-01-03 15:08 Douglas McIlroy
2023-01-03 15:57 ` Alejandro Colomar
2023-01-03 17:19   ` Jon Steinhart
2023-01-05 13:22     ` Tom Ivar Helbekkmo via TUHS
2023-01-05 21:11       ` Rob Pike
2023-01-03 17:26 ` Dan Cross
2023-01-03 18:07   ` Steve Nickolas
2023-01-03 20:19     ` Steffen Nurpmeso
2023-01-03 23:03       ` ron minnich
2023-01-04  1:37         ` Bakul Shah
2023-01-04  1:58           ` Adam Thornton
2023-01-04 15:19             ` Ralph Corderoy
2023-01-04 18:01               ` Bakul Shah
2023-01-04 20:46                 ` Alejandro Colomar
2023-01-05  0:06                   ` John Cowan
2023-01-05  0:41                     ` Adam Thornton
2023-01-04  5:05         ` Theodore Ts'o
2023-01-03 18:19   ` Niklas Karlsson
2023-01-04  1:29   ` Adam Thornton
2023-01-05  1:51     ` Alejandro Colomar

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).