[TUHS] History of #! interpretation in Unix

[TUHS] History of #! interpretation in Unix

[Warren Toomey]

Hi all, best wishes for 2011. I had an e-mail from Sven Mascheck asking about
the history of #! interpretation in System V. I couldn't find any #!
code in the kernels before SysVR4. However, I thought I'd pass the
query onto the TUHS list, in case others can shed some light on the question.

Did SysV systems before r4 do #! interpretation, and if so where was it done:
kernel, library, shell? Any code references, e.g. function names etc.?

Many thanks,
	Warren

[TUHS] History of #! interpretation in Unix

[John Cowan]

Warren Toomey scripsit:

> Hi all, best wishes for 2011. I had an e-mail from Sven Mascheck asking about
> the history of #! interpretation in System V. I couldn't find any #!
> code in the kernels before SysVR4. 

That's correct.  The feature was added to the 8th Research edition kernel,
but didn't migrate to the System III/V line until SVR4.  Shebangs began
very early in BSD as a csh-only hack, but appeared as a config option
in kernels as early as 2.8BSD, according to Wikipedia; 4.2BSD was the
first release that turned them on by default.

Perl has always (I think) had its own shebang support; you can get Perl
to exec an arbitrary interpreter for a script provided it has a shebang.
Modern shells will use /bin/sh to run scripts that *don't* have shebangs,
except for ksh which considers itself /bin/sh-compatible, and therefore
runs such scripts itself.

-- 
I am expressing my opinion.  When my            John Cowan
honorable and gallant friend is called,         cowan at ccil.org
he will express his opinion.  This is           http://www.ccil.org/~cowan
the process which we call Debate.                   --Winston Churchill

[TUHS] History of #! interpretation in Unix

[Sven Mascheck]

John Cowan wrote:
> Warren Toomey scripsit:

> > the history of #! interpretation in System V. I couldn't find any #!
> > code in the kernels before SysVR4. 

> The feature was added to the 8th Research edition kernel,
> but didn't migrate to the System III/V line until SVR4.  

Thanks to you all.  Amazing, it came so "late".

> Shebangs began very early in BSD as a csh-only hack, but appeared
> as a config option in kernels as early as 2.8BSD, according to
> Wikipedia; 4.2BSD was the first release that turned them on by
> default.

In a usenet discussion with G.Ritter in '01 I noticed the first detailed
hint about the origin, and since then I've tried my best to document
the issue on http://www.in-ulm.de/~mascheck/various/shebang/#origin

(And although someone added some of the links I collected,
 I believe Wikipedia still has several things mixed up,
 while I'm quite busy to keep my own stuff consistent...)

4.0BSD had incorporated it originally among the BSDs.
2.8BSD came soon, sounds earlier, but after its original release,
2BSD was just a different development line (http://www.levenez.com/unix/unix_a4.pdf).

As you mention "shebangs began .. as csh-only hack":
I still wonder if one may call the BSD csh-hack as origin,
because #! might have been developed independently at Bell Labs.
(although, in turn, the idea might origin from Berkeley, as the
 interesting answer from DMR in the wikipedia article suggests.)
And BTW: it was not "csh-only", but sh was hacked symmetrically :)


> Perl has always (I think) had its own shebang support; you can get Perl
> to exec an arbitrary interpreter for a script provided it has a shebang.

The traditional ash, bash-1 (and -2 ff. as compile time option) also
implement that.


> Modern shells will use /bin/sh to run scripts that *don't* have shebangs,
> except for ksh which considers itself /bin/sh-compatible, and therefore
> runs such scripts itself.

I thought sh-like shells in general try to interprete scripts themselves
upon an ENOEXEC.  Do you know certain shells which actually call "/bin/sh"?

    Sven

[TUHS] History of #! interpretation in Unix

[John Cowan]

Sven Mascheck scripsit:

> As you mention "shebangs began .. as csh-only hack":
> I still wonder if one may call the BSD csh-hack as origin,
> because #! might have been developed independently at Bell Labs.

No, that would be too much of a coincidence to choose exactly the
same characters.  There has to be a single point of origin, from
which it spread by stimulus diffusion (you hear there are telescopes,
you know they use lenses, you build your own telescope).

> I thought sh-like shells in general try to interprete scripts themselves
> upon an ENOEXEC.  Do you know certain shells which actually call "/bin/sh"?

Yes, seemingly bash does that too.  Bogus.

-- 
John Cowan    cowan at ccil.org    http://ccil.org/~cowan
The whole of Gaul is quartered into three halves.
        --Julius Caesar

[TUHS] History of #! interpretation in Unix

[Sven Mascheck]

John Cowan wrote:
> Sven Mascheck scripsit:
> 
> > As you mention "shebangs began .. as csh-only hack":
> > I still wonder if one may call the BSD csh-hack as origin,
> > because #! might have been developed independently at Bell Labs.
> 
> No, that would be too much of a coincidence to choose exactly the
> same characters.  There has to be a single point of origin, from
> which it spread by stimulus diffusion (you hear there are telescopes,
> you know they use lenses, you build your own telescope).

This doesn't sound obligatory to me (I know it's academic).
DMR might have known the csh-hack (and if a comment character is
implemented anywhere, here csh, then it's obvious to use it for #!)


> > I thought sh-like shells in general try to interprete scripts themselves
> > upon an ENOEXEC.  Do you know certain shells which actually call "/bin/sh"?
> 
> Yes, seemingly bash does that too.  Bogus.

How'd you get the impression?  I can't verify this.
bash-1.05:execute_cmd.c executed the commands itself, in a subshell,

" [ errno == ENOEXEC ]

  /* This file is executable.
     If it begins with #!, then help out people with losing
     operating systems.  Otherwise, check to see if it is a binary
     file by seeing if the first line (or upto 30 characters) are in
     the ASCII set.  Execute the contents as shell commands. */ "

and this hasn't been changed (except length or return status)
until the current release.

Perhaps you had another shell in mind?  I believe a shell should
try to execute commands itself instead of calling any other shell.
(That's the only way how executing on in POSIX shell is possible, BTW,
because neither #! nor paths are standardized).

[TUHS] History of #! interpretation in Unix

[John Cowan]

Sven Mascheck scripsit:

> DMR might have known the csh-hack (and if a comment character is
> implemented anywhere, here csh, then it's obvious to use it for #!)

That might account for the "#", but not for "#!" taken together.
Having two different people invent the shebang independently (as opposed
to *implementing* it independently, as has happened many times -- 8th
Edition, SVR4, Linux, etc.) is just too improbable for me to swallow.

> > > I thought sh-like shells in general try to interprete scripts themselves
> > > upon an ENOEXEC.  Do you know certain shells which actually call "/bin/sh"?
> > 
> > Yes, seemingly bash does that too.  Bogus.
> 
> How'd you get the impression?  I can't verify this.

No, I meant that bash does what ksh does: uses itself, which means that a
shebang-free script runs differently on bash, ksh, dash.  This is IMHO
unfortunate.

> (That's the only way how executing on in POSIX shell is possible, BTW,
> because neither #! nor paths are standardized).

Good point.  IMHO the Posix sh definition should be extended so that all
shells claiming Posix-compliance should do shebangs.

-- 
John Cowan  cowan at ccil.org  http://ccil.org/~cowan
And now here I was, in a country where a right to say how the country should
be governed was restricted to six persons in each thousand of its population.
For the nine hundred and ninety-four to express dissatisfaction with the
regnant system and propose to change it, would have made the whole six
shudder as one man, it would have been so disloyal, so dishonorable, such
putrid black treason.  --Mark Twain's Connecticut Yankee

[TUHS] History of #! interpretation in Unix

[Larry McVoy]

On Sun, Jan 16, 2011 at 03:17:46PM -0500, John Cowan wrote:
> Sven Mascheck scripsit:
> 
> > DMR might have known the csh-hack (and if a comment character is
> > implemented anywhere, here csh, then it's obvious to use it for #!)
> 
> That might account for the "#", but not for "#!" taken together.
> Having two different people invent the shebang independently (as opposed
> to *implementing* it independently, as has happened many times -- 8th
> Edition, SVR4, Linux, etc.) is just too improbable for me to swallow.

I grepped 32v, v7, sysIII for it and didn't find it.
-- 
---
Larry McVoy                lm at bitmover.com           http://www.bitkeeper.com

[TUHS] History of #! interpretation in Unix

[Sven Mascheck]

John Cowan wrote:
> Sven Mascheck scripsit:

> > DMR might have known the csh-hack [...]
> 
> Having two different people invent the shebang independently

who (or what system) do you mean was the 2nd?


> > > > > Modern shells will use /bin/sh to run scripts that *don't* have shebangs,
[...]
> No, I meant that bash does what ksh does: uses itself, [...]

With /bin/sh you actually meant any shell calling "itself"?
Otherwise I'm afraid, I have some difficulties following you.


> IMHO the Posix sh definition should be extended so that all
> shells claiming Posix-compliance should do shebangs.

Shells themselves should implement it, not the kernel?

There was a working group resolution to standardize #!, which didn't
make it, http://www.opengroup.org/platform/resolutions/bwg2000-004.html
you could chime in

[TUHS] History of #! interpretation in Unix

[John Cowan]

Sven Mascheck scripsit:

> who (or what system) do you mean was the 2nd?

That was hypothetical.

Trying again:  "It's unlikely that two different people chose #! as the
executable-script mark independently."

> With /bin/sh you actually meant any shell calling "itself"?
> Otherwise I'm afraid, I have some difficulties following you.

I mean that whereas (t)csh uses /bin/sh to run scripts with no
shebangs, the Posix-compatible shells execute scripts with no
shebangs directly.  And this is Bad.

> > IMHO the Posix sh definition should be extended so that all
> > shells claiming Posix-compliance should do shebangs.
> 
> Shells themselves should implement it, not the kernel?

It might be too much to ask the kernel to do, especially on
non-traditional Posix systems like z/OS.

> There was a working group resolution to standardize #!, which didn't
> make it, http://www.opengroup.org/platform/resolutions/bwg2000-004.html
> you could chime in

I see the problem now: a portable awk script, for example, can't assume
that the Posix awk is in /usr/bin/awk, so "#!/usr/bin/awk" might get the
wrong awk.  So the feature is not worth standardizing for Posix.

-- 
[W]hen I wrote it I was more than a little              John Cowan
febrile with foodpoisoning from an antique carrot       cowan at ccil.org
that I foolishly ate out of an illjudged faith          http://ccil.org/~cowan
in the benignancy of vegetables.  --And Rosta

[TUHS] History of #! interpretation in Unix

[Warner Losh]

On 01/16/2011 14:37, John Cowan wrote:
> Sven Mascheck scripsit:
>> With /bin/sh you actually meant any shell calling "itself"?
>> Otherwise I'm afraid, I have some difficulties following you.
> I mean that whereas (t)csh uses /bin/sh to run scripts with no
> shebangs, the Posix-compatible shells execute scripts with no
> shebangs directly.  And this is Bad.

The alternatives are worse.

On a sysIII system, you had the situation where scripts would be 
executed by whatever your shell.  So if you put a script on the system, 
you had to make sure everybody used csh and write it in csh, or you had 
to make sure nobody did and write it in /bin/sh.

The current situation says that 'you must opt-in' to a different shell, 
which leads to more predictable behavior.

Later, #! processing moved into the kernel so that interpreters would 
execute things directly.

To cope with that confusing mess, the perl env hack was born so that the 
perl scripts could execute anywhere, independent of the shell driving it.

Warner

[TUHS] History of #! interpretation in Unix

[John Cowan]

Warner Losh scripsit:

> The alternatives are worse.

You're making my point for me.

> On a sysIII system, you had the situation where scripts would be  
> executed by whatever your shell.  So if you put a script on the system,  
> you had to make sure everybody used csh and write it in csh, or you had  
> to make sure nobody did and write it in /bin/sh.

Right.

> The current situation says that 'you must opt-in' to a different shell,  
> which leads to more predictable behavior.

Unfortunately not.  The current situation is that when a shell sees a
shebang-free script, it decides whether it is Posix enough to execute
that script.  So ash, bash, dash, ksh, zsh will all run shebang-free
scripts themselves, whereas csh, tcsh will defer to /bin/sh.

What I am arguing for is that *all* shells should defer to /bin/sh
to execute shebang-free scripts.  As things stand, the behavior of a
shebang-free script depends on which shell is trying to run it, whereas
it should only depend (says I) on the identity of /bin/sh.

> Later, #! processing moved into the kernel so that interpreters would  
> execute things directly.

Yes, and that's a Good Thing.

-- 
The experiences of the past show                John Cowan
that there has always been a discrepancy        cowan at ccil.org
between plans and performance.                  http://www.ccil.org/~cowan
        --Emperor Hirohito, August 1945

[TUHS] History of #! interpretation in Unix

[Tim Bradshaw]

On 16 Jan 2011, at 20:17, John Cowan wrote:

> That might account for the "#", but not for "#!" taken together.
> Having two different people invent the shebang independently (as opposed
> to *implementing* it independently, as has happened many times -- 8th
> Edition, SVR4, Linux, etc.) is just too improbable for me to swallow.

One possibility is that there was prior art for "!" meaning "run in a shell" which gave various people the same idea independently: in a script it needs to be commented out so the shell will not barf, so you pretty naturally get "#!" as the magic sequence.  That's really arguing that the inventions were not independent but they were both copying from something else.

The obvious place that "!" might have come from is ed: it seems to have been there essentially for ever (the 1st edition manuals have it).

--tim

[TUHS] History of #! interpretation in Unix

[Wilko Bulte]

Hi Warren

I worked on SVR3 maintenance at Philips Information Systems in the dim past. I'm almost certain, like 99%, that #! was not in R3

Wilko

Op 16 jan. 2011 om 09:43 heeft Warren Toomey <wkt at tuhs.org> het volgende geschreven:

> Hi all, best wishes for 2011. I had an e-mail from Sven Mascheck asking about
> the history of #! interpretation in System V. I couldn't find any #!
> code in the kernels before SysVR4. However, I thought I'd pass the
> query onto the TUHS list, in case others can shed some light on the question.
> 
> Did SysV systems before r4 do #! interpretation, and if so where was it done:
> kernel, library, shell? Any code references, e.g. function names etc.?
> 
> Many thanks,
>    Warren
> _______________________________________________
> TUHS mailing list
> TUHS at minnie.tuhs.org
> https://minnie.tuhs.org/mailman/listinfo/tuhs

[TUHS] History of #! interpretation in Unix

[Warner Losh]

On 01/16/2011 03:53, Wilko Bulte wrote:
> Hi Warren
>
> I worked on SVR3 maintenance at Philips Information Systems in the dim past. I'm almost certain, like 99%, that #! was not in R3

That matches my memory of various 3B1 machines I had to use as part of 
my support activities at The Wollognong Group back in 1989.  All the 
SIII machines didn't do #!, while some of the SysV machines did.  I do 
know that r1 didn't have it, but r4 did.

Warner
> Wilko
>
> Op 16 jan. 2011 om 09:43 heeft Warren Toomey<wkt at tuhs.org>  het volgende geschreven:
>
>> Hi all, best wishes for 2011. I had an e-mail from Sven Mascheck asking about
>> the history of #! interpretation in System V. I couldn't find any #!
>> code in the kernels before SysVR4. However, I thought I'd pass the
>> query onto the TUHS list, in case others can shed some light on the question.
>>
>> Did SysV systems before r4 do #! interpretation, and if so where was it done:
>> kernel, library, shell? Any code references, e.g. function names etc.?
>>
>> Many thanks,
>>     Warren
>> _______________________________________________
>> TUHS mailing list
>> TUHS at minnie.tuhs.org
>> https://minnie.tuhs.org/mailman/listinfo/tuhs
> _______________________________________________
> TUHS mailing list
> TUHS at minnie.tuhs.org
> https://minnie.tuhs.org/mailman/listinfo/tuhs
>
>
>

[TUHS] History of #! interpretation in Unix

[Jeremy C. Reed]

4BSD (4.0) usr/src/sys/sys/TODO (of Nov.  9, 1980) says it was planned:

	6. Exec fixes
        	Implement dmr's #! feature; pass string arguments through 
		faster.

And the usr/src/sys/newsys/sys1.c has explanation of it and source code.

http://minnie.tuhs.org/cgi-bin/utree.pl?file=4BSD/usr/src/sys/sys/TODO

http://minnie.tuhs.org/cgi-bin/utree.pl?file=4BSD/usr/src/sys/newsys/sys1.c

This seems to indicate the idea was not developed separately at 
Berkeley.

It was introduced into BSD by April 1981. For 2BSD (2.8) it was added by 
Dec. 16, 1981 when built with MENLO_SCRIPT defined (but I don't see that 
documented or defined).

[TUHS] History of #! interpretation in Unix

[John Cowan]

Jeremy C. Reed scripsit:

> This seems to indicate the idea was not developed separately at 
> Berkeley.

Thanks for doing the digging.  What this shows is that dmr came up with
the idea of putting shebang processing into the kernel, but it does not
show where shebangs as a shell feature came from.  I still think that
has to be the CSRG.

> It was introduced into BSD by April 1981. For 2BSD (2.8) it was added by 
> Dec. 16, 1981 when built with MENLO_SCRIPT defined (but I don't see that 
> documented or defined).

Thanks again for clarifying the ordering here: I had assumed that 2.8
was older than 4.0.

--
We call nothing profound                        cowan at ccil.org
that is not wittily expressed.                  John Cowan
        --Northrop Frye (improved)

[TUHS] History of #! interpretation in Unix

[Jeremy C. Reed]

On Mon, 17 Jan 2011, John Cowan wrote:

> > This seems to indicate the idea was not developed separately at 
> > Berkeley.
> 
> Thanks for doing the digging.  What this shows is that dmr came up with
> the idea of putting shebang processing into the kernel, but it does not
> show where shebangs as a shell feature came from.  I still think that
> has to be the CSRG.

As a similar feature, Joy's experimental "ashell" in the first BSD had a 
feature to run Pascal objects (started with the magic 0404 octal word) 
with the px interpreter. In 2BSD (and then 3BSD), his new csh had the 
OTHERSH feature -- if it started with # it would use csh; if the script 
did not begin with the # it would use /bin/sh.

I still don't see the same #! feature, but the above appear they may be 
inspiration. Also the 1BSD and 2BSD predate CSRG.

> > It was introduced into BSD by April 1981. For 2BSD (2.8) it was added by 
> > Dec. 16, 1981 when built with MENLO_SCRIPT defined (but I don't see that 
> > documented or defined).
> 
> Thanks again for clarifying the ordering here: I had assumed that 2.8
> was older than 4.0.

[TUHS] History of #! interpretation in Unix

[Sven Mascheck]

Jeremy C. Reed wrote:
> 4BSD (4.0) usr/src/sys/sys/TODO (of Nov.  9, 1980) says it was planned:

It was not until now that I noticed that the subject is _that_ general :)

I had already tried to completely collect all the interesting pieces
available through TUHS (and the CSRG archive) on a page as I mentioned
earlier in this thread.  But I'd like to tie in with

> For 2BSD (2.8) it was added by Dec. 16, 1981 when built with
> MENLO_SCRIPT defined (but I don't see that documented or defined).

According to /usr/kernel/src/cmd/standalone/tapes/instructions
these seem to be fixes from the U.S. Geological Survey in Menlo Park,
perhaps by Bill Jolitz.  The code looks different from 4BSD, because
2.8BSD was a 7th ed kernel with several adds+fixes (all to be activated
by macros like the above).  I don't know yet how #! was created/derived
here, exactly - which remindes me that I don't know how the original
implementation (research unix) looked exactly.  I always had the impression
that 4.0BSD newsys/sys1.c already had been adjusted to BSD...

[TUHS] History of #! interpretation in Unix

[Cyrille Lefevre]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 1260 bytes --]


Le 17/01/2011 20:02, Jeremy C. Reed a écrit :
>
> 4BSD (4.0) usr/src/sys/sys/TODO (of Nov.  9, 1980) says it was planned:
>
> 	6. Exec fixes
>          	Implement dmr's #! feature; pass string arguments through
> 		faster.
>
> And the usr/src/sys/newsys/sys1.c has explanation of it and source code.
>
> http://minnie.tuhs.org/cgi-bin/utree.pl?file=4BSD/usr/src/sys/sys/TODO
>
> http://minnie.tuhs.org/cgi-bin/utree.pl?file=4BSD/usr/src/sys/newsys/sys1.c
>
> This seems to indicate the idea was not developed separately at
> Berkeley.
>
> It was introduced into BSD by April 1981. For 2BSD (2.8) it was added by
> Dec. 16, 1981 when built with MENLO_SCRIPT defined (but I don't see that
> documented or defined).

Hi,

yet another reference but more in the spirit of what csh does, it only 
checks for a simple hash (#), no explaim mark (!), and is enclosed in 
UCB_SCRIPT define.

http://minnie.tuhs.org/cgi-bin/utree.pl?file=2.9BSD/usr/src/sys/sys/sys1.c

2.11 BSD seems to have an enhanced version of this feature in the sense 
where the shell path may be followed by some arguments (i.e.: /bin/sh -x)

http://minnie.tuhs.org/cgi-bin/utree.pl?file=2.11BSD/sys/sys/kern_exec.c

Regards,

Cyrille Lefevre
-- 
mailto:Cyrille.Lefevre-lists at laposte.net

[TUHS] History of #! interpretation in Unix

[Sven Mascheck]

Cyrille Lefevre wrote:

> yet another reference but more in the spirit of what csh does, it only  
> checks for a simple hash (#), no explaim mark (!), and is enclosed in  
> UCB_SCRIPT define.
>
> http://minnie.tuhs.org/cgi-bin/utree.pl?file=2.9BSD/usr/src/sys/sys/sys1.c

Perhaps you missed the ! in the macro?
#define SCRMAG '#!'

> 2.11 BSD seems to have an enhanced version of this feature in the sense  
> where the shell path may be followed by some arguments (i.e.: /bin/sh -x)

Yes, #! originally had not implemented arguments at all (this even applies
to 386BSD).  Arguments ("all in one") came with 4.2BSD and later, variations
appeared like splitting up into argv[] or delivering only the "first" argument.
-- 
http://www.in-ulm.de/~mascheck/various/shebang/#results

[TUHS] History of #! interpretation in Unix

[Cyrille Lefevre]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 847 bytes --]


Le 19/01/2011 21:35, Sven Mascheck a écrit :
>
> Cyrille Lefevre wrote:
>
>> yet another reference but more in the spirit of what csh does, it only
>> checks for a simple hash (#), no explaim mark (!), and is enclosed in
>> UCB_SCRIPT define.
>>
>> http://minnie.tuhs.org/cgi-bin/utree.pl?file=2.9BSD/usr/src/sys/sys/sys1.c
>
> Perhaps you missed the ! in the macro?

right %-/

> #define SCRMAG '#!'

multi-character character constant !

u.u_exdata.ux_mag == SCRMAG makes me wonder...

$ cat x.c
#include <stdio.h>
main() { int c = '#!'; printf("%x", c); return 0; }
$ make x
cc     x.c   -o x
x.c:2:18: warning: multi-character character constant
$ ./x
2321
$ man ascii
...
| 20 sp | 21  ! | 22  " | 23  # | 24  $ | 25  % | 26  & | 27  ' |
...

well, it works !


Regards,

Cyrille Lefevre
-- 
mailto:Cyrille.Lefevre-lists at laposte.net

[TUHS] History of #! interpretation in Unix

[Jeremy C. Reed]

On Mon, 17 Jan 2011, Jeremy C. Reed wrote:

> It was introduced into BSD by April 1981. For 2BSD (2.8) it was added by 
> Dec. 16, 1981 when built with MENLO_SCRIPT defined (but I don't see that 
> documented or defined).

Doing some related research, I noticed I had written in my notes (maybe 
over a year ago) that Robert Elz added #! to BSD. Here is the email 
message with the background:

http://mail-index.netbsd.org/netbsd-users/2008/11/10/msg002390.html

[TUHS] History of #! interpretation in Unix

[Sven Mascheck]

Jeremy C. Reed wrote:
> 
> Doing some related research, I noticed I had written in my notes (maybe 
> over a year ago) that Robert Elz added #! to BSD. Here is the email 
> message with the background:
> 
> http://mail-index.netbsd.org/netbsd-users/2008/11/10/msg002390.html

Thanks for digging that out. I was confused at first, because 4.0 BSD
implemented it without arguments, while the above article tells

"When I added it to BSD, I made a conscious decision to permit
 exactly one arg - not a general command line."

But the first regular occurence (active by default) in 4.2BSD
is implemented like that.

Do you feel like providing access to more of your research?