[TUHS] Shell control through external commands

[TUHS] Shell control through external commands

[Norman Wilson]

I remember reading about #! in the early 1980s, and
having mixed feelings about it, as I still do.  The
basic idea is fine, if annoyingly limited; but that
the kernel has to decide, in effect, whether to treat
a header as binary or text bothers me.  Were I designing
a new system from scratch today, I'd just make the
header all text; the small extra space and time for
the kernel to parse that for binaries doesn't matter
any more.  It certainly did when #! was invented,
though.

I had the impression at the time that it came from
Berkeley, but I think I later heard from the horse's
mouth that it was originally Dennis's idea.

I don't think anyone has yet laid out the complete
story of what came before:

1.  Originally, the shell would exec(file), and if
exec returned ENOEXEC, would open the file and treat
it as shell commands.

2.  Then came the C shell, and a problem: did file
contain commands for csh or sh?  A hack emerged:
if csh encountered a script file, it would read
the first character; if that was '#' it was a
csh script, otherwise it handed off to /bin/sh.

None of this helped when some program other than
the shell called exec on a shell script.  That's one
reason execlp and execvp appeared.  (The other is that
they observe $PATH if the command pathname has a
single element.)

I don't know offhand whether there was ever an execlp/vp
that implemented the #-means-csh convention.  Anyone
else remember?

Norman Wilson
Toronto ON

[TUHS] Shell control through external commands

[Joerg Schilling]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 611 bytes --]

norman at oclsc.org (Norman Wilson) wrote:

> I don't know offhand whether there was ever an execlp/vp
> that implemented the #-means-csh convention.  Anyone
> else remember?


I've never seen such a userland implementation outside csh, but UNOS did 
implement the #! feature in the command interpreter (not in the kernel) around 
1983/84.


Jörg

-- 
 EMail:joerg at schily.net                  (home) Jörg Schilling D-13353 Berlin
       joerg.schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL:  http://cdrecord.org/private/ http://sourceforge.net/projects/schilytools/files/'

[TUHS] Shell control through external commands

[Joerg Schilling]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 1207 bytes --]

Sven Mascheck <mascheck at in-ulm.de> wrote:

> On Mon, Sep 12, 2016 at 02:50:10PM +0200, Joerg Schilling wrote:
> > > > The colon was introduced by AT&T around 1983.
> > > 
> > > It's a builtin in the v7 Bourne shell - see SYSNULL in msg.c (which
> > > has the command name table) and in xec.c (which has the implementation)
> > 
> > You are missinterpeting things.
> > 
> > The colon in line one as a hint to a modified csh to call the Bourne Shell  
> > first appeared around 1983.
>
> I'm still confused (you're short with context): what does "introduced by AT&T" mean?
>
> I only know the #-hacked csh from 2BSD+ ('79), and the #-hacked sh from 3BSD+ ('80).
> How would ":" as a hint in this respect show up on other systems?

This is long ago, so my memory is not good anymore.

Aprox. 30 years ago, I ported my shell to a svr3 system and discovered problems.
AFAIR, this system had special treatment for scripts starting with ":".

Jörg

-- 
 EMail:joerg at schily.net                  (home) Jörg Schilling D-13353 Berlin
       joerg.schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL:  http://cdrecord.org/private/ http://sourceforge.net/projects/schilytools/files/

[TUHS] Shell control through external commands

[Sven Mascheck]

Sven Mascheck wrote:
> Joerg Schilling wrote:

> > > > The colon was introduced by AT&T around 1983.
> [...]
> I'm still confused (you're short with context): what does "introduced by AT&T" mean?

Ok, so they included csh with SysV, but SVR1 had no #! yet.
Less confused but still curious: what problems did show up with : then?

-Sven

[TUHS] Shell control through external commands

[Sven Mascheck]

On Mon, Sep 12, 2016 at 02:50:10PM +0200, Joerg Schilling wrote:
> > > The colon was introduced by AT&T around 1983.
> > 
> > It's a builtin in the v7 Bourne shell - see SYSNULL in msg.c (which
> > has the command name table) and in xec.c (which has the implementation)
> 
> You are missinterpeting things.
> 
> The colon in line one as a hint to a modified csh to call the Bourne Shell  
> first appeared around 1983.

I'm still confused (you're short with context): what does "introduced by AT&T" mean?

I only know the #-hacked csh from 2BSD+ ('79), and the #-hacked sh from 3BSD+ ('80).
How would ":" as a hint in this respect show up on other systems?


BTW: academic but funny side effect of : as no-op instead of real comment:

: `echo output 1>&2`

actually writes to stderr.


Normal Wilson wrote:

> [...] A hack emerged: if csh encountered a script file, it would read the
> first character; if that was '#' it was a csh script, otherwise it handed
> off to /bin/sh.

lesser known fact: even sh was hacked on BSDs (since 3BSD) with this #-csh magic,
and also exec'ed to csh. This then co-existed with the soon coming implementation
of "#" as comment character.

-Sven

[TUHS] Shell control through external commands

[Dave Horsfall]

On Tue, 13 Sep 2016, Dave Horsfall wrote:

> Minor nit: ":" is not precisely a comment, but an empty label.  Ano no, 
> it's not particularly magic in any way, as I recall.

Oops - I didn't know that CSH treated it specially...  I'm not a CSH 
programmer, never have been, and never will be (it ought to attract the 
death penalty).

-- 
Dave Horsfall DTM (VK2KFU)  "Those who don't understand security will suffer."

[TUHS] Shell control through external commands

[Dave Horsfall]

On Mon, 12 Sep 2016, markus schnalke wrote:

> I think I've seen many eary (pre-Shebang and even pre-#-comment) shell 
> scripts start with a colon. Is this only a coincidence with having a 
> comment at the beginning or was the colon character magic in some way?

Minor nit: ":" is not precisely a comment, but an empty label.  Ano no, 
it's not particularly magic in any way, as I recall.

-- 
Dave Horsfall DTM (VK2KFU)  "Those who don't understand security will suffer."

[TUHS] Shell control through external commands

[Norman Wilson]

It sounds like my understanding of the different 4.1x versions is
just mistaken.  If 4.1c had FFS and sockets, the kernel on which
V8 was built must have been an earlier 4.1x.

I believe the reason for adopting that kernel, rather than sticking
with 32/V, was exactly to get paging support.  There was a competing
32/V descendant with paging, written by John Reiser at Holmdel, which
I gather was thought by many to be much cleaner technically; e.g. he
unified the block-device buffer cache and the memory-page cache, and
implemented copy-on-write paging rather than resorting to the messy
vfork.  I have heard that there was considerable argument and
hand-wringing over whether to use Reiser's kernel or the BSD one.
It all happened well before I arrived, and I don't know what the
tradeoffs were, though one was certainly that Reiser's management
didn't support his work and nobody in 1127 was keen to have to take
it over.

Norman Wilson
Toronto ON

[TUHS] Shell control through external commands

[Norman Wilson]

Joerg Schilling:

  The colon was introduced by AT&T around 1983. It was used for Bourne Shell
  scripts. Some of these scripts made it into SVr4 and caused problems with
  non-Bourne compatible other shells.

====

Interesting.  I never knew of that convention.  I remember seeing
shell scripts with a : at the front, but thought that was just to
make sure the first character wasn't # even if the script began with
a comment.

Since some here had never heard of the #-means-csh convention, I
should perhaps explain about :.  In pre-Bourne shells that used the
simple external control-flow mechanisms that I think were discussed
here a few months ago, : was used to mark a label: goto(1) would
seek to the beginning of its standard input, then read until it
encountered a line of the form
	: label
with the desired label, then exit with the seek pointer at the first
character of the following line.

: was a no-op command; I forget whether it was implemented within the
shell or externally.  Either way, that made it useful as a comment
character, but somewhat clumsy: it was just a command, with no
special parsing rules attached.  A comment using : had to begin at
a command boundary, and its arguments were parsed in the normal way:
	rm -rf * : you don't want to do this
was probably not what you wanted, instead you had to type
	rm -rf * ; : "you don't want to do this"
or the like.

csh used # as a comment character from the beginning.  Bourne
adopted it too.

Norman Wilson
Toronto ON

[TUHS] Shell control through external commands

[Joerg Schilling]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 695 bytes --]

Tony Finch <dot at dotat.at> wrote:

> Joerg Schilling <schily at schily.net> wrote:
> >
> > The colon was introduced by AT&T around 1983.
>
> It's a builtin in the v7 Bourne shell - see SYSNULL in msg.c (which
> has the command name table) and in xec.c (which has the implementation)

You are missinterpeting things.

The colon in line one as a hint to a modified csh to call the Bourne Shell  
first appeared around 1983.


Jörg

-- 
 EMail:joerg at schily.net                  (home) Jörg Schilling D-13353 Berlin
       joerg.schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL:  http://cdrecord.org/private/ http://sourceforge.net/projects/schilytools/files/'

[TUHS] Shell control through external commands

[Tony Finch]

Joerg Schilling <schily at schily.net> wrote:
>
> The colon was introduced by AT&T around 1983.

It's a builtin in the v7 Bourne shell - see SYSNULL in msg.c (which
has the command name table) and in xec.c (which has the implementation)

http://www.tuhs.org/Archive/PDP-11/Trees/V7/usr/src/cmd/sh/

Tony.
-- 
f.anthony.n.finch  <dot at dotat.at>  http://dotat.at/  -  I xn--zr8h punycode
Southeast Fitzroy: Southerly becoming cyclonic, 5 to 7, occasionally 4 later.
Moderate or rough, occasionally very rough at first. Rain or showers. Good,
occasionally poor.

[TUHS] Shell control through external commands

[Jacob Goense]

On 2016-09-10 19:45, Dario Niedermann wrote:
> TIL! Still, it seems to me that there is no way the `cd' script can
> actually accomplish anything, even when called by exec().

It can give you a useful return code.

[TUHS] Shell control through external commands

[Joerg Schilling]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 780 bytes --]

norman at oclsc.org (Norman Wilson) wrote:

> I don't think the BSD kernel when adopted had much, if any,
> of sockets, Berkeley's TCP/IP, McKusick's FFS; if it did,
> they were excised.  Likewise any remaining trace of V7's
> mpx(2) multiplexed-file IPC.

From looking at the CSRG sources, it seems that the filesystem project has been 
founded by Bill Joy and Kirk approached the project a year (or more) later and 
imlemented ony the parts that are related to the block allocation.

Does someone know more?

Jörg

-- 
 EMail:joerg at schily.net                  (home) Jörg Schilling D-13353 Berlin
       joerg.schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL:  http://cdrecord.org/private/ http://sourceforge.net/projects/schilytools/files/'

[TUHS] Shell control through external commands

[Joerg Schilling]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 724 bytes --]

markus schnalke <meillo at marmaro.de> wrote:

> I think I've seen many eary (pre-Shebang and even pre-#-comment)
> shell scripts start with a colon. Is this only a coincidence with
> having a comment at the beginning or was the colon character magic
> in some way?

The colon was introduced by AT&T around 1983. It was used for Bourne Shell 
scripts. Some of these scripts made it into SVr4 and caused problems with 
non-Bourne compatible other shells.

Jörg

-- 
 EMail:joerg at schily.net                  (home) Jörg Schilling D-13353 Berlin
       joerg.schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL:  http://cdrecord.org/private/ http://sourceforge.net/projects/schilytools/files/'

[TUHS] Shell control through external commands

[markus schnalke]

[2016-09-10 13:22] arnold at skeeve.com
> Dario Niedermann <dnied at tiscali.it> wrote:
> >
> > I never knew that a shell script could work without a shebang line.
>
> That was the simplicity of the original system, where you just made the
> file executable. The shell would fork and exec as usual.  When the exec
> failed, the shell noticed that errno was ENOEXEC (not a runnable file)
> and started interpreting the script itself.

I think I've seen many eary (pre-Shebang and even pre-#-comment)
shell scripts start with a colon. Is this only a coincidence with
having a comment at the beginning or was the colon character magic
in some way?


meillo

[TUHS] Shell control through external commands

[Dave Horsfall]

On Sat, 10 Sep 2016, arnold at skeeve.com wrote:

> > Correct; if the exec failed then the invoking shell assumed that it 
> > was a shell script, and spawned a shell to run it.
> 
> I think this happened in the "spawned" shell - so the exec would fail and
> then this child shell would just run the script.

Oops - you're right.  My organic bit-rot is worse than I thought...

-- 
Dave Horsfall DTM (VK2KFU)  "Those who don't understand security will suffer."

[TUHS] Shell control through external commands

[Norman Wilson]

Doug McIlroy:

  "Re-port" may be a bit strong. Internet stuff from Berkeley
  was folded into the research code (for a huge increase in
  kernel size). But I think it was done by pasting Berkeley
  code into local source, rather than the other way around.

====

Actually it was more nearly:

-- Adopt 4.1c BSD kernel
-- Graft in Research-specific things it was important to
keep: in particular Dennis's stream subsystem, Tom Killian's
original /proc, Peter Weinberger's early network file system
client code (the server was just a user-mode program) and
simple hackery to speed up the file system without great
fuss (make the block size 4KiB and move the free list to
a bitmap; no cylinder groups or other radical changes).
Also device drivers to support Datakit, at the time our
workhorse network.  I think a file-system switch went
in early as well, spurred by having both /proc and
pjw's network file system; it wasn't used to support
multiple disk-file-system layouts, though it could have
been.
-- Outside the kernel, keep pretty much the then-current
Research commands, including Blit/5620 support, the
cleaned up and slightly-extended Bourne shell, and whatnot.

I don't think the BSD kernel when adopted had much, if any,
of sockets, Berkeley's TCP/IP, McKusick's FFS; if it did,
they were excised.  Likewise any remaining trace of V7's
mpx(2) multiplexed-file IPC.

I'm going by the state the system was in when I arrived
in August 1984, plus a short note written by Weinberger
that I came across later.

TCP/IP support didn't show up until later, I think summer
1985, though it might have been a year later.  The first
cut was done by Robert T. Morris (later famous for a buggy
program that broke the Internet), who did several summers
as an intern; he took the code from (I think) 4.2 BSD,
and constructed some shims to fit it into the stream world.
Paul Glick later cleaned it up a lot, removing the need
for most of the shimmery.

Further evolution followed, of course, including a
complete rewrite of the interface between drivers
(device, file system, and stream) and the rest of the
system, which made configuration much more straightforward.
Also a rampage on my part to identify code that was no
longer useful and kick it out; I took special pleasure
in removing ptrace(2) (even though I had to change adb
and sdb first to use /proc).

But that was all later.

Norman Wilson
Toronto ON

[TUHS] Shell control through external commands

[Doug McIlroy]

> 8th edition was essentially a re-port of 4.1c BSD, correct?

"Re-port" may be a bit strong. Internet stuff from Berkeley
was folded into the research code (for a huge increase in
kernel size). But I think it was done by pasting Berkeley
code into local source, rather than the other way around.
But, since much of the rest of the BSD kernel was Bell
Labs code, it's probably right that the result of the
merge had more code in common with BSD than with Research.
If you ask, though, what fraction of Research code
survived the merge, it was probably larger than the
surviving fraction of the total BSD code.

Doug

[TUHS] Shell control through external commands

[Random832]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 2059 bytes --]

On Sun, Sep 11, 2016, at 14:50, Sven Mascheck wrote:
> I tried to follow all the traces of #! and collected the results on
> http://www.in-ulm.de/~mascheck/various/shebang/
> I'd be happy to add and fix stuff, if somebody spots anything.

DEMOS (the soviet version of Unix) seems to have an independent
implementation that supports some features no-one else does

- use $* to define where the passed-in arguments [including the script
name] go between the interpreter-arguments specified on the #!-line
- alternate magic "/*#!"

I don't remember where I downloaded it from, I think maybe someone
posted it here a few years ago.

Relevant bits from sys1.c:

# define SCRMAG  "#!"
# define SCRMAG2 "/*#!"
# define ARGPLACE "$*"

     /* формат:
         пусть есть файл   /.../.../x
         со следующим содержимым:
         -------------------------------
         |#!CMD A1 A2 A3
         | .......

        Вызываем команду:
                x B1 B2

        Запустится на самом деле такая команда:
                CMD A1 A2 A3 /.../.../x B1 B2

        Один из аргументов Ai может иметь вид $*
        Тогда аргументы запуска подставятся на это место, а не в конец:
         -------------------------------
         |#!CMD A1 $* A2 A3
         | .......

        Вызываем:
                x B1 B2

        Запустится:
                CMD A1 /.../.../x B1 B2 A2 A3
     */

There's no comment describing the SCRMAG2 feature but you can see it
used in code:

        if (indir < NSYMLNK && u.u_offset > (off_t)2 ){
            cp = (char *) &u.u_exdata;
            if( !bcmp(cp, SCRMAG, 2 ))
                off = (off_t) 2;
            else if( u.u_offset > (off_t)4 && !bcmp( cp, SCRMAG2, 4))
                off = (off_t) 4;


Sole uses of these features in the source tree:

cmd/OPQRS/spline.c,v:@/*#!/bin/cc -O $* -o spline
cmd/EFGHIJK/glob.c,v:@/*#!/bin/cc -o glob

[TUHS] Shell control through external commands

[Sven Mascheck]

On Sun, Sep 11, 2016 at 02:27:48PM -0400, Dan Cross wrote:
> On Sun, Sep 11, 2016 at 1:39 PM, Doug McIlroy <doug at cs.dartmouth.edu> wrote:
> 
> > > IIRC #! originated at Bell Labs but it got out to the world via BSD.
> > > Perhaps Dr. McIlroy could confirm / deny / expand upon the details
> > (please?)
> >
> > I recall Dennis discussing the feature at some length before installing it.
> > So the exact semantics, especially the injected argument, are almost]
> > certainly his. I don't know whether he built on a model from elsewhere.
> > #! appeared between v7 (1979) and v8 (1985). As v8 was never released,
> > it clearly made its way into the world via BSD and USG. BSD, being
> > more nimble, was likely first.
> 
> Also, 8th edition was essentially a re-port of 4.1c BSD, correct? If it
> were already in BSD, it could conceivably come back into 8th edition via
> that route. I found the following in /usr/src/sys/sys/TODO for 4.0BSD:
> 
> 6. Exec fixes
> >         Implement dmr's #! feature; pass string arguments through faster.

See 4.0BSD /usr/src/sys/newsys/sys1.c
http://minnie.tuhs.org/cgi-bin/utree.pl?file=4BSD/usr/src/sys/newsys/sys1.c
for two mails from Dennis to Berkeley, sending the info and code.

> >From dmr Thu Jan 10 04:25:49 1980 remote from research
> The system has been changed so that if a file being executed
> begins with the magic characters #! , [...]

I tried to follow all the traces of #! and collected the results on
http://www.in-ulm.de/~mascheck/various/shebang/
I'd be happy to add and fix stuff, if somebody spots anything.

Warren, one has to be careful not to have trailing slashes in web archive URLs.
(e.g., after modifying URLs by hand). Example: calling
http://minnie.tuhs.org/cgi-bin/utree.pl?file=4BSD/usr/src/sys/
eventually results in the link
http://minnie.tuhs.org/cgi-bin/utree.pl?file=4BSD/usr/src/sys//sys/TODO
which yields "...is not a known file.", while
http://minnie.tuhs.org/cgi-bin/utree.pl?file=4BSD/usr/src/sys/sys/TODO
is valid.

-Sven

[TUHS] Shell control through external commands

[Dan Cross]

On Sun, Sep 11, 2016 at 1:39 PM, Doug McIlroy <doug at cs.dartmouth.edu> wrote:

> > IIRC #! originated at Bell Labs but it got out to the world via BSD.
> > Perhaps Dr. McIlroy could confirm / deny / expand upon the details
> (please?)
>
> I recall Dennis discussing the feature at some length before installing it.
> So the exact semantics, especially the injected argument, are almost]
> certainly his. I don't know whether he built on a model from elsewhere.
> #! appeared between v7 (1979) and v8 (1985). As v8 was never released,
> it clearly made its way into the world via BSD and USG. BSD, being
> more nimble, was likely first.
>

Also, 8th edition was essentially a re-port of 4.1c BSD, correct? If it
were already in BSD, it could conceivably come back into 8th edition via
that route. I found the following in /usr/src/sys/sys/TODO for 4.0BSD:

6. Exec fixes
>         Implement dmr's #! feature; pass string arguments through faster.


That would strongly imply that the feature originated with Dennis and was
known in Berkeley around the time for 4BSD.

4.0's 'getxfile()' in /usr/src/sys/sys/sys1.c does not recognize the '#!'
sequence as a magic number, but 'exece()' in /usr/src/sys/newsys/sys1.c
does. No mention of it came up in my cursory scan of 3bsd. So it was
certain present by the time of 4.1c, but not before 4BSD that I found.

        - Dan C.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://minnie.tuhs.org/pipermail/tuhs/attachments/20160911/0110691d/attachment-0001.html>

[TUHS] Shell control through external commands

[Clem Cole]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 1761 bytes --]

On Sun, Sep 11, 2016 at 1:39 PM, Doug McIlroy <doug at cs.dartmouth.edu> wrote:

> I recall Dennis discussing the feature at some length before installing it.
> So the exact semantics, especially the injected argument, are almost]
> certainly his. I don't know whether he built on a model from elsewhere.
> #! appeared between v7 (1979) and v8 (1985). As v8 was never released,
> it clearly made its way into the world via BSD and USG. BSD, being
> more nimble, was likely first.
>

​Well that tracks my memory. I also thought it was from BSD, although it
might have come from some place else like Harvard, MIT, Ill, Rice or the
like.  It is not in the V7 sources in Warren's code repository but I do
remember it in my customized V7 (polite form of hacking)​ at Tektronix on
our 11/70 and 11/44 in late 1979/early 1980 before I went to UCB.  But I do
not remember that extension @ CMU (the tapes which I have had a hard time
getting read so I can not check).

I'm a little hazy, but I think it means that Mark Bales/Jim Kleckner gave
it to me with BSD stuff when we picked up csh, vi and pas @ Tek - which we
originally did in drips and drabs - during their time as summer students.
 (I would not get an official set of 1BSD or 2BSD tapes for Tek until right
before I left for Berkeley a couple of years later).

BTW: It's also possible that's how fsck originally made it back to UCB (but
tjk might have given it to Joy directly), as Mark and Jim took back my
collection of CMU hacks at the end of that summer which was either 79 or
80.  I do remember Mark being impressed with fsck.

Clem
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://minnie.tuhs.org/pipermail/tuhs/attachments/20160911/837222fd/attachment.html>

[TUHS] Shell control through external commands

[Doug McIlroy]

> IIRC #! originated at Bell Labs but it got out to the world via BSD.
> Perhaps Dr. McIlroy could confirm / deny / expand upon the details (please?)

I recall Dennis discussing the feature at some length before installing it.
So the exact semantics, especially the injected argument, are almost]
certainly his. I don't know whether he built on a model from elsewhere.
#! appeared between v7 (1979) and v8 (1985). As v8 was never released,
it clearly made its way into the world via BSD and USG. BSD, being
more nimble, was likely first.

doug

[TUHS] Shell control through external commands

[Diomidis Spinellis]

On 10/09/2016 20:45, Dario Niedermann wrote:
> TIL! Still, it seems to me that there is no way the `cd' script can
> actually accomplish anything, even when called by exec().

# Time directory access
/usr/bin/time cd /mounts/foo

# Verify that directories can be cd'd
find /mounts/foo -type d -exec cd '{}' \; && echo OK

[TUHS] Shell control through external commands

[arnold]

Dave Horsfall <dave at horsfall.org> wrote:

> On Sat, 10 Sep 2016, arnold at skeeve.com wrote:
>
> > I think that the way the shell "recognized" scripts was by having exec 
> > fail, otherwise how would it know?  I think we'll have to go grubbing in 
> > the source archives to be sure.
>
> Correct; if the exec failed then the invoking shell assumed that it was a 
> shell script, and spawned a shell to run it.

I think this happened in the "spawned" shell - so the exec would fail and
then this child shell would just run the script.

	if ((pid = fork()) != 0)
		wait for pid
	else {
		/* in the child */
		exec /path/to/program
		if (errno == ENOEXEC)	/* or whatever it is, I don't remember */
			interpret the script directly
	}

Arnold

[TUHS] Shell control through external commands

[Dave Horsfall]

On Sat, 10 Sep 2016, arnold at skeeve.com wrote:

> I think that the way the shell "recognized" scripts was by having exec 
> fail, otherwise how would it know?  I think we'll have to go grubbing in 
> the source archives to be sure.

Correct; if the exec failed then the invoking shell assumed that it was a 
shell script, and spawned a shell to run it.

With the #! hack, it had to be followed by the full path name (white space 
optional) and the kernel ran it directly.

-- 
Dave Horsfall DTM (VK2KFU)  "Those who don't understand security will suffer."

[TUHS] Shell control through external commands

[Ronald Natalie]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 932 bytes --]


> On Sep 10, 2016, at 10:39 PM, arnold at skeeve.com wrote:
> 
> "Ron Natalie" <ron at ronnatalie.com> wrote:
> 
>> Traditionally, the shell recognized shell scripts and ran them in a new
>> instance of the shell rather than calling exec on them.
> 
> I think that the way the shell "recognized" scripts was by having
> exec fail, otherwise how would it know?  I think we'll have to go
> grubbing in the source archives to be sure.
You are correct.

> 
>> It was a Berkeley hack to add a new exec magic number that happened to
>> correspond to the characters #!.
> 
> ISTR reading on this list that #! was a post-V7 Bell Labs invention
> that the BSD guys went and implemented themselves...

That’s possible.   We jumped from V7 to the BSD distributions because about the same time we were going from PDP-11’s to VAXen of various flavors
(actually our first VAX was a copy of one of the George Gobel dual processor 780’s).

[TUHS] Shell control through external commands

[arnold]

"Ron Natalie" <ron at ronnatalie.com> wrote:

> Traditionally, the shell recognized shell scripts and ran them in a new
> instance of the shell rather than calling exec on them.

I think that the way the shell "recognized" scripts was by having
exec fail, otherwise how would it know?  I think we'll have to go
grubbing in the source archives to be sure.

> It was a Berkeley hack to add a new exec magic number that happened to
> correspond to the characters #!.

ISTR reading on this list that #! was a post-V7 Bell Labs invention
that the BSD guys went and implemented themselves...

Can someone who was there please clarify?

Thanks,

Arnold

[TUHS] Shell control through external commands

[Ron Natalie]

Traditionally, the shell recognized shell scripts and ran them in a new
instance of the shell rather than calling exec on them.
It was a Berkeley hack to add a new exec magic number that happened to
correspond to the characters #!.    The shells typically still recognized
scripts (the #!   was syntactically a comment) and invoked them, but if they
were passed to exec (i.e., from something other than the shell that was to
interpret the), the kernel recognized the magic number and invoked what
followed as the name of an interpreter to handle the file.

Not only did this allow arbitrary interpretters (other than /bin/sh or any
shell), but it also allowed things like setuid bits to work, and scripts to
be invoked by things that didn't just feed the file name to a shell to
invoke.

-----Original Message-----
From: TUHS [mailto:tuhs-bounces@minnie.tuhs.org] On Behalf Of Dave Horsfall
Sent: Saturday, September 10, 2016 6:44 PM
To: The Eunuchs Hysterical Society
Subject: Re: [TUHS] Shell control through external commands

On Sat, 10 Sep 2016, Dario Niedermann wrote:

> I never knew that a shell script could work without a shebang line.

I believe it was used to specify an interpreter other than /bin/sh (the
default).

--
Dave Horsfall DTM (VK2KFU)  "Those who don't understand security will
suffer."

[TUHS] Shell control through external commands

[Dave Horsfall]

On Sat, 10 Sep 2016, Dario Niedermann wrote:

> I never knew that a shell script could work without a shebang line.

I believe it was used to specify an interpreter other than /bin/sh (the 
default).

-- 
Dave Horsfall DTM (VK2KFU)  "Those who don't understand security will suffer."

[TUHS] Shell control through external commands

[Dario Niedermann]

Il 10/09/2016 alle 22:02, Chet Ramey ha scritto:

> Use it as a permissions check.


Il 10/09/2016 alle 21:53, Dan Cross ha scritto:

> It can return an exit status: this makes it mostly equivalent to something
> like 'test -d /foo && test -x /foo'.


Yeah, good point.

[TUHS] Shell control through external commands

[Chet Ramey]

On 9/10/16 1:45 PM, Dario Niedermann wrote:

> 
> TIL! Still, it seems to me that there is no way the `cd' script can
> actually accomplish anything, even when called by exec().

Use it as a permissions check.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
		 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet at case.edu    http://cnswww.cns.cwru.edu/~chet/

[TUHS] Shell control through external commands

[Dan Cross]

On Sat, Sep 10, 2016 at 1:45 PM, Dario Niedermann <dnied at tiscali.it> wrote:

> Il 10/09/2016 alle 11:41, Joerg Schilling ha scritto:
>
> > POSIX requires some commands to be callable via exec().
>
> Il 10/09/2016 alle 19:06, Random832 ha scritto:
>
> > something that is technically required by POSIX. There is a general
> > requirement that "regular builtin" commands [with no exception for cd]
> > "shall be implemented in a manner so that they can be accessed via the
> > exec family of functions as defined in the System Interfaces volume of
> > POSIX.1-2008 and can be invoked directly by those standard utilities
> > that require it (env, find, nice, nohup, time, xargs)."
>
>
> TIL! Still, it seems to me that there is no way the `cd' script can
> actually accomplish anything, even when called by exec().
>

It can return an exit status: this makes it mostly equivalent to something
like 'test -d /foo && test -x /foo'.

        - Dan C.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://minnie.tuhs.org/pipermail/tuhs/attachments/20160910/e92e2ced/attachment.html>

[TUHS] Shell control through external commands

[arnold]

Dario Niedermann <dnied at tiscali.it> wrote:

> I never knew that a shell script could work without a shebang line.

That was the simplicity of the original system, where you just made the
file executable. The shell would fork and exec as usual.  When the exec
failed, the shell noticed that errno was ENOEXEC (not a runnable file)
and started interpreting the script itself.

This broke down as soon as there were two shells (/bin/sh and csh) since
if you tried to run a csh script from the Bourne shell or vice versa,
things weirded out pretty quickly.

IIRC #! originated at Bell Labs but it got out to the world via BSD.
Perhaps Dr. McIlroy could confirm / deny / expand upon the details (please?)

Thanks,

Arnold

[TUHS] Shell control through external commands

[Dario Niedermann]

Il 10/09/2016 alle 11:41, Joerg Schilling ha scritto:

> POSIX requires some commands to be callable via exec().


Il 10/09/2016 alle 19:06, Random832 ha scritto:

> something that is technically required by POSIX. There is a general
> requirement that "regular builtin" commands [with no exception for cd]
> "shall be implemented in a manner so that they can be accessed via the
> exec family of functions as defined in the System Interfaces volume of
> POSIX.1-2008 and can be invoked directly by those standard utilities
> that require it (env, find, nice, nohup, time, xargs)."


TIL! Still, it seems to me that there is no way the `cd' script can
actually accomplish anything, even when called by exec().

[TUHS] Shell control through external commands

[Dario Niedermann]

Il 15/07/2016 alle 18:47, Doug McIlroy ha scritto:

> Gerard Holzmann took the true and false commands as
> the jumping-off point for "Code Inflation", an
> installment of his "Reliable Code" blog and column
> in IEE Software. An informative, but depressing, read:
> http://spinroot.com/gerard/pdf/Code_Inflation.pdf

I just reclaimed 56K of disk space by replacing the GNU versions of
/bin/true and /bin/false with hand-made reproductions of the earliest
versions.

I never knew that a shell script could work without a shebang line.

[TUHS] Shell control through external commands

[Random832]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 937 bytes --]

On Sat, Sep 10, 2016, at 05:13, Michael Kjörling wrote:
> On 10 Sep 2016 09:45 +0200, from dnied at tiscali.it (Dario Niedermann):
> > Il 15/07/2016 alle 14:27, Norman Wilson ha scritto:
> >> lu$ cat /bin/cd
> >> #!/bin/sh
> >> builtin cd "$@"
> >> lu$ 
> > 
> > But doesn't this change the current dir only in the child shell?
> > Which then exits right after the second line, parent shell's $PWD
> > unaffected. I really don't see how this script is useful.

I've wondered about this in the past, the conclusion was that it is
something that is technically required by POSIX. There is a general
requirement that "regular builtin" commands [with no exception for cd]
"shall be implemented in a manner so that they can be accessed via the
exec family of functions as defined in the System Interfaces volume of
POSIX.1-2008 and can be invoked directly by those standard utilities
that require it (env, find, nice, nohup, time, xargs)."

[TUHS] Shell control through external commands

[Nemo]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 1188 bytes --]

On 10 September 2016 at 05:41, Joerg Schilling <schily at schily.net> wrote:
> Michael Kjörling <michael at kjorling.se> wrote:
>
>> On 10 Sep 2016 09:45 +0200, from dnied at tiscali.it (Dario Niedermann):
>> > Il 15/07/2016 alle 14:27, Norman Wilson ha scritto:
>> >> lu$ cat /bin/cd
>> >> #!/bin/sh
>> >> builtin cd "$@"
>> >> lu$
>> >
>> > But doesn't this change the current dir only in the child shell?
>> > Which then exits right after the second line, parent shell's $PWD
>> > unaffected. I really don't see how this script is useful.
>>
>> It does appear rather useless. Curiously, Debian (checked on Wheezy =
>> bash 4.2+dfsg-0.1+deb7u3 and Jessie = bash 4.3-11+b1) seems to not
>> supply anything like that, so it would appear to be some kind of
>> Fedora-ism rather than a part of anything upstream; that, or the
>> Debian folks are actually paying attention to what they ship onto
>> users' systems.
>
> POSIX requires some commands to be callable via exec().

Solaris 10 has the following amusing implementation (/usr/bin/cd):

#!/bin/ksh -p
#
#ident  "@(#)alias.sh   1.2     00/02/15 SMI"
#
# Copyright (c) 1995 by Sun Microsystems, Inc.
#
cmd=`basename $0`
$cmd "$@"

N.

[TUHS] Shell control through external commands

[Joerg Schilling]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 1145 bytes --]

Michael Kjörling <michael at kjorling.se> wrote:

> On 10 Sep 2016 09:45 +0200, from dnied at tiscali.it (Dario Niedermann):
> > Il 15/07/2016 alle 14:27, Norman Wilson ha scritto:
> >> lu$ cat /bin/cd
> >> #!/bin/sh
> >> builtin cd "$@"
> >> lu$ 
> > 
> > But doesn't this change the current dir only in the child shell?
> > Which then exits right after the second line, parent shell's $PWD
> > unaffected. I really don't see how this script is useful.
>
> It does appear rather useless. Curiously, Debian (checked on Wheezy =
> bash 4.2+dfsg-0.1+deb7u3 and Jessie = bash 4.3-11+b1) seems to not
> supply anything like that, so it would appear to be some kind of
> Fedora-ism rather than a part of anything upstream; that, or the
> Debian folks are actually paying attention to what they ship onto
> users' systems.

POSIX requires some commands to be callable via exec().

Jörg

-- 
 EMail:joerg at schily.net                  (home) Jörg Schilling D-13353 Berlin
       joerg.schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL:  http://cdrecord.org/private/ http://sourceforge.net/projects/schilytools/files/'

[TUHS] Shell control through external commands

[Michael Kjörling]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 1825 bytes --]

On 10 Sep 2016 09:45 +0200, from dnied at tiscali.it (Dario Niedermann):
> Il 15/07/2016 alle 14:27, Norman Wilson ha scritto:
>> lu$ cat /bin/cd
>> #!/bin/sh
>> builtin cd "$@"
>> lu$ 
> 
> But doesn't this change the current dir only in the child shell?
> Which then exits right after the second line, parent shell's $PWD
> unaffected. I really don't see how this script is useful.

It does appear rather useless. Curiously, Debian (checked on Wheezy =
bash 4.2+dfsg-0.1+deb7u3 and Jessie = bash 4.3-11+b1) seems to not
supply anything like that, so it would appear to be some kind of
Fedora-ism rather than a part of anything upstream; that, or the
Debian folks are actually paying attention to what they ship onto
users' systems.

Even more curious, check this out from Debian Wheezy (I suspect the
Jessie system would behave identically):

$ cat > /michael/home/bin/cd
#!/bin/sh
builtin cd "$@"
^D
$ chmod +x bin/cd
$ echo $PATH
/michael/home/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
$ type cd
cd is a shell builtin
$

so not even the shell will use the script in the first place!

If I force the shell to use the script I created (by saying ~/bin/cd
instead of just cd), after fixing /bin/sh to /bin/bash because when
run as sh the shell has no concept of "builtin", it behaves like you'd
expect: the working directory changes _within_ the script, but $PWD is
unchanged in the parent shell once the script finishes, turning it
into a somewhat expensive no-op.

Unless Fedora's shell has been specifically modified to pay attention
to the child process' $PWD, of course...

-- 
Michael Kjörling • https://michael.kjorling.se • michael at kjorling.se
                 “People who think they know everything really annoy
                 those of us who know we don’t.” (Bjarne Stroustrup)

[TUHS] Shell control through external commands

[Dario Niedermann]

Il 15/07/2016 alle 14:27, Norman Wilson ha scritto:

> Just for the record, Fedora 21 supplies /bin/cd, as part
> of package bash-4.3.42-1.  Interestingly, it is a shell
> script:
> 
> lu$ cat /bin/cd
> #!/bin/sh
> builtin cd "$@"
> lu$ 

But doesn't this change the current dir only in the child shell?
Which then exits right after the second line, parent shell's $PWD
unaffected. I really don't see how this script is useful.

[TUHS] Shell control through external commands

[Dave Horsfall]

On Thu, 14 Jul 2016, arnold at skeeve.com wrote:

> Consider
> 
> 	execlp("[", "-f", "/some/file", "]", NULL);
> 
> Then [ and test need to live in the filesystem.

Ah yes, of course...  Thanks.

-- 
Dave Horsfall DTM (VK2KFU)  "Those who don't understand security will suffer."

[TUHS] Shell control through external commands

[Doug McIlroy]

Gerard Holzmann took the true and false commands as
the jumping-off point for "Code Inflation", an
installment of his "Reliable Code" blog and column
in IEE Software. An informative, but depressing, read:
http://spinroot.com/gerard/pdf/Code_Inflation.pdf

Doug

[TUHS] Shell control through external commands

[Chet Ramey]

On 7/14/16 11:59 PM, Dave Horsfall wrote:
> On Thu, 14 Jul 2016, Random832 wrote:
> 
>> My Mac even has /usr/bin/cd.
> 
> After which, of course, the shell returns to where it was...  Or does
> it do a getcwd() after each command or something?

Consider something like `find . -type d -exec cd {} \;' to test for
directory search permission.  That's the only use I've ever seen advocated.


-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
		 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet at case.edu    http://cnswww.cns.cwru.edu/~chet/

[TUHS] Shell control through external commands

[Norman Wilson]

Random832:

  The existence of cd as a real command is a bit silly (Ubuntu doesn't
  seem to bother with it), but it is technically required by the standard.

===

Just for the record, Fedora 21 supplies /bin/cd, as part
of package bash-4.3.42-1.  Interestingly, it is a shell
script:

lu$ cat /bin/cd
#!/bin/sh
builtin cd "$@"
lu$ 

As has been said here, it's hard to see the functional point.

Others have remarked on the continued life of /bin/true and
/bin/false.  There are some who use those as shells in /etc/passwd
for logins that should never actually be allowed to do anything
directly.  I have no strong personal feeling about that, I'm just
reporting.

And to be fair (as has also already been displayed here), the
copyright notice inserted in the once-empty /bin/true was hundreds
of bytes long, not thousands.  Let us call out silliness, but let
us not make it out as any sillier than it actually is.

Norman Wilson
Toronto ON
UNIX old fart and amateur pedant

[TUHS] Shell control through external commands

[Joerg Schilling]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 801 bytes --]

Marc Rochkind <rochkind at basepath.com> wrote:

> Thought I'd mention here that the OP was asking about the original shell,
> and that the Mashey and Bourne shells were much newer than that.

BTW: last Year there was a talk from Stephen Bourne af BSDcon where he 
explained how the Bourne Shell started. It seems that Mashey, Bourne and csh 
all started with the Thompson Shell code.

...and from a todays view, the time between 1970 anf 1976 is a short time 
compared to the fact that the Bourne Shell now exists since 40 years.

Jörg

-- 
 EMail:joerg at schily.net                  (home) Jörg Schilling D-13353 Berlin
       joerg.schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL:  http://cdrecord.org/private/ http://sourceforge.net/projects/schilytools/files/'

[TUHS] Shell control through external commands

[Tony Finch]

Dave Horsfall <dave at horsfall.org> wrote:
> On Thu, 14 Jul 2016, Clem Cole wrote:
>
> > Could you be confusing the fact the true and false were implemented by
> > external commands in some early shell's
>
> And on the Mac and FreeBSD, they still are (as well as being builtins).
> At one time they were shell scripts e.g. "exit 0" for "true".

This isn't a BSDism: POSIX requires that most built-in commands also exist
as executables on the PATH. The exceptions are the "special built-in
commands" which manipulate the shell command language state, e.g. break
and set.

http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_14

Tony.
-- 
f.anthony.n.finch  <dot at dotat.at>  http://dotat.at/  -  I xn--zr8h punycode
Humber, Thames, Dover: West or southwest 4 or 5, increasing 6 at times. Slight
or moderate. Mainly fair. Good.

[TUHS] Shell control through external commands

[Rudi Blom]

Just for the fun of it results on a few UNIX versions

Intel - SCO UNIX 3.2V4.2
/bin/true:		empty file
/bin/false:		contains "exit 255"

> file $(which true) $(which false)

Alpha - Digital Unix 4.0G
/usr/bin/true:	COFF format alpha dynamically linked, demand paged
executable or object module stripped - version 3.11-10
/usr/bin/false:	COFF format alpha dynamically linked, demand paged
executable or object module stripped - version 3.11-10

Alpha - Tru64 V5.1B
/usr/bin/true:	COFF format alpha dynamically linked, demand paged
executable or object module stripped - version 3.13-14
/usr/bin/false:	COFF format alpha dynamically linked, demand paged
executable or object module stripped - version 3.13-14

Itanium - HP-UX 11i B.11.23 & HP-UX 11i B.11.31
/sbin/true:		ELF-32 executable object file - IA64
/sbin/false:	ELF-32 executable object file - IA64

[TUHS] Shell control through external commands

[Random832]

On Thu, Jul 14, 2016, at 23:59, Dave Horsfall wrote:
> On Thu, 14 Jul 2016, Random832 wrote:
> 
> > My Mac even has /usr/bin/cd.
> 
> After which, of course, the shell returns to where it was...  Or does
> it do a getcwd() after each command or something?

The existence of cd as a real command is a bit silly (Ubuntu doesn't
seem to bother with it), but it is technically required by the standard.

[TUHS] Shell control through external commands

[Dave Horsfall]

On Thu, 14 Jul 2016, Random832 wrote:

> My Mac even has /usr/bin/cd.

After which, of course, the shell returns to where it was...  Or does
it do a getcwd() after each command or something?

-- 
Dave Horsfall DTM (VK2KFU)  "Those who don't understand security will suffer."

[TUHS] Shell control through external commands

[Random832]

On Thu, Jul 14, 2016, at 19:56, shawn wilson wrote:
> I've always been curious though - what was the reason behind
> implementing /bin/[ ? IDK any shell where this isn't implemented - I
> always assumed it's a POSIX compatibility stopgap older systems needed
> to stay compliant with their shipped shell.

It's required because it might be executed by non-shell programs.

http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap01.html#tag_17_06

My Mac even has /usr/bin/cd.

The "special builtins" which aren't required to exist as binaries are:  
  break : continue .  eval exec exit export readonly return set shift
times trap unset

[TUHS] Shell control through external commands

[arnold]

Dave Horsfall <dave at horsfall.org> wrote:

> On Thu, 14 Jul 2016, shawn wilson wrote:
>
> > I've always been curious though - what was the reason behind 
> > implementing /bin/[ ? IDK any shell where this isn't implemented - I 
> > always assumed it's a POSIX compatibility stopgap older systems needed 
> > to stay compliant with their shipped shell.
>
> To implement:
>
>     [ blah ]
>
> when it wasn't a built-in (there were few built-ins in those days).
>
> I have no idea why it's still there...  A Posix requirement, along with 
> really unsafe functions?

Consider

	execlp("[", "-f", "/some/file", "]", NULL);

Then [ and test need to live in the filesystem.

Arnold

[TUHS] Shell control through external commands

[Nemo]

On 14 July 2016 at 19:10, Ronald Natalie <ron at ronnatalie.com> wrote:
[...]
> Actually, one of the UNIX releases had NO EXECUTABLE CODE in /bin/true.   The default is to return the true.
> There was 20 lines of copyright/rights notice in commends.

https://www.computer.org/csdl/mags/so/2015/02/mso2015020010.pdf

[TUHS] Shell control through external commands

[Steve Nickolas]

On Fri, 15 Jul 2016, Dave Horsfall wrote:

> On Thu, 14 Jul 2016, shawn wilson wrote:
>
>> I've always been curious though - what was the reason behind
>> implementing /bin/[ ? IDK any shell where this isn't implemented - I
>> always assumed it's a POSIX compatibility stopgap older systems needed
>> to stay compliant with their shipped shell.
>
> To implement:
>
>    [ blah ]
>
> when it wasn't a built-in (there were few built-ins in those days).
>
> I have no idea why it's still there...  A Posix requirement, along with
> really unsafe functions?
>
> By the way, on both Mac and FreeBSD (I can't be bothered checking my
> Penguin box) "true" and "false" are binaries (and aren't even linked).
>
>

$ file $(which true) $(which false)
/bin/true:  ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), 
dynamically linked (uses shared libs), for GNU/Linux 2.6.26, 
BuildID[sha1]=0x347aaa1fa815d6689723e6f8aa0f7207566b6aba, stripped
/bin/false: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), 
dynamically linked (uses shared libs), for GNU/Linux 2.6.26, 
BuildID[sha1]=0x821c6a0a4331413fb4b562b623c6bd5ca0a707f6, stripped

-uso.

[TUHS] Shell control through external commands

[Dave Horsfall]

On Thu, 14 Jul 2016, shawn wilson wrote:

> I've always been curious though - what was the reason behind 
> implementing /bin/[ ? IDK any shell where this isn't implemented - I 
> always assumed it's a POSIX compatibility stopgap older systems needed 
> to stay compliant with their shipped shell.

To implement:

    [ blah ]

when it wasn't a built-in (there were few built-ins in those days).

I have no idea why it's still there...  A Posix requirement, along with 
really unsafe functions?

By the way, on both Mac and FreeBSD (I can't be bothered checking my
Penguin box) "true" and "false" are binaries (and aren't even linked).

-- 
Dave Horsfall DTM (VK2KFU)  "Those who don't understand security will suffer."

[TUHS] Shell control through external commands

[shawn wilson]

On Jul 14, 2016 7:01 PM, "Peter Jeremy" <peter at rulingia.com> wrote:
>
> On 2016-Jul-15 08:36:56 +1000, Dave Horsfall <dave at horsfall.org> wrote:
> >On Thu, 14 Jul 2016, Clem Cole wrote:

> >And on the Mac and FreeBSD, they still are (as well as being builtins).
>
> FreeBSD provides a convenient list of what commands are (currently)
builtin
> to the provided shells and available externally:
> https://www.freebsd.org/cgi/man.cgi?builtin
>

Bash man page does as well along with command -v (and hash IIRC) letting
you know.

I've always been curious though - what was the reason behind implementing
/bin/[ ? IDK any shell where this isn't implemented - I always assumed it's
a POSIX compatibility stopgap older systems needed to stay compliant with
their shipped shell.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://minnie.tuhs.org/pipermail/tuhs/attachments/20160714/95d9b674/attachment.html>

[TUHS] Shell control through external commands

[Marc Rochkind]

Thought I'd mention here that the OP was asking about the original shell,
and that the Mashey and Bourne shells were much newer than that.

On Thu, Jul 14, 2016 at 4:49 PM, Peter Jeremy <peter at rulingia.com> wrote:

> On 2016-Jul-15 08:36:56 +1000, Dave Horsfall <dave at horsfall.org> wrote:
> >On Thu, 14 Jul 2016, Clem Cole wrote:
> >> Could you be confusing the fact the true and false were implemented by
> >> external commands in some early shell's
>
> I was always amused by AT&T adding a multi-kB copyright notice to what
> had previously been an empty file.
>
> >And on the Mac and FreeBSD, they still are (as well as being builtins).
>
> FreeBSD provides a convenient list of what commands are (currently) builtin
> to the provided shells and available externally:
> https://www.freebsd.org/cgi/man.cgi?builtin
>
> --
> Peter Jeremy
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://minnie.tuhs.org/pipermail/tuhs/attachments/20160714/7a40d2fa/attachment.html>

[TUHS] Shell control through external commands

[Ronald Natalie]

> On Jul 14, 2016, at 6:36 PM, Dave Horsfall <dave at horsfall.org> wrote:
> 
> On Thu, 14 Jul 2016, Clem Cole wrote:
> 
>> Could you be confusing the fact the true and false were implemented by
>> external commands in some early shell's
> 
> And on the Mac and FreeBSD, they still are (as well as being builtins).
> 
> At one time they were shell scripts e.g. "exit 0" for "true".

Actually, one of the UNIX releases had NO EXECUTABLE CODE in /bin/true.   The default is to return the true.
There was 20 lines of copyright/rights notice in commends.

[TUHS] Shell control through external commands

[Peter Jeremy]

On 2016-Jul-15 08:36:56 +1000, Dave Horsfall <dave at horsfall.org> wrote:
>On Thu, 14 Jul 2016, Clem Cole wrote:
>> Could you be confusing the fact the true and false were implemented by
>> external commands in some early shell's

I was always amused by AT&T adding a multi-kB copyright notice to what
had previously been an empty file.

>And on the Mac and FreeBSD, they still are (as well as being builtins).

FreeBSD provides a convenient list of what commands are (currently) builtin
to the provided shells and available externally:
https://www.freebsd.org/cgi/man.cgi?builtin

-- 
Peter Jeremy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 949 bytes
Desc: not available
URL: <http://minnie.tuhs.org/pipermail/tuhs/attachments/20160715/4a9466d1/attachment.sig>

[TUHS] Shell control through external commands

[Corey Lindsly]

> 
> At one time they were shell scripts e.g. "exit 0" for "true".
> 
> -- 
> Dave Horsfall DTM (VK2KFU)  "Those who don't understand security will suffer."
> 

Indeed. Here's what they look like in Solaris 8 .. /usr/bin/true is just 
comments.

lod# cat /usr/bin/true
#!/usr/bin/sh
#       Copyright (c) 1984, 1986, 1987, 1988, 1989 AT&T
#         All Rights Reserved

#       THIS IS UNPUBLISHED PROPRIETARY SOURCE CODE OF AT&T
#       The copyright notice above does not evidence any
#       actual or intended publication of such source code.

#ident  "@(#)true.sh    1.6     93/01/11 SMI"   /* SVr4.0 1.4   */


lod# cat /usr/bin/false
#!/usr/bin/sh
#       Copyright (c) 1984, 1986, 1987, 1988, 1989 AT&T
#         All Rights Reserved

#       THIS IS UNPUBLISHED PROPRIETARY SOURCE CODE OF AT&T
#       The copyright notice above does not evidence any
#       actual or intended publication of such source code.

#ident  "@(#)false.sh   1.6     93/01/11 SMI"   /* SVr4.0 1.3   */
exit 255


lod# uname -a
SunOS lod 5.8 Generic_108528-05 sun4u sparc SUNW,UltraAX-i2
lod# 


--corey

[TUHS] Shell control through external commands

[Dave Horsfall]

On Thu, 14 Jul 2016, Clem Cole wrote:

> Could you be confusing the fact the true and false were implemented by
> external commands in some early shell's

And on the Mac and FreeBSD, they still are (as well as being builtins).

At one time they were shell scripts e.g. "exit 0" for "true".

-- 
Dave Horsfall DTM (VK2KFU)  "Those who don't understand security will suffer."

[TUHS] Shell control through external commands

[Dave Horsfall]

On Thu, 14 Jul 2016, Diomidis Spinellis wrote:

[...]

> The only external command I found was glob, which performed wildcard
> expansion.

Well, "test" (and "[") are still external commands...

-- 
Dave Horsfall DTM (VK2KFU)  "Those who don't understand security will suffer."

[TUHS] Shell control through external commands

[Random832]

On Thu, Jul 14, 2016, at 11:46, John Cowan wrote:
> Between the 6e shell and the Bourne shell there was the Mashey, or PWB,
> shell, which was the first to have them built in.  So if they existed,
> they were probably separate commands on 6e-ish systems that didn't reach
> the outside world.  It's important to remember that the whole terminology
> of editions and releases didn't apply within Bell Labs itself.

They're present in 6e, they're just not in the shell source code because
they don't require any special support from the shell.

[TUHS] Shell control through external commands

[John Cowan]

Diomidis Spinellis scripsit:

> The 7th Edition Bourne shell has these commands built-in
> (usr/src/cmd/sh/cmd.c), while the 6th Edition (usr/source/s2/sh.c)
> seems to lack them completely.

Between the 6e shell and the Bourne shell there was the Mashey, or PWB,
shell, which was the first to have them built in.  So if they existed,
they were probably separate commands on 6e-ish systems that didn't reach
the outside world.  It's important to remember that the whole terminology
of editions and releases didn't apply within Bell Labs itself.

-- 
John Cowan          http://www.ccil.org/~cowan        cowan at ccil.org
"The serene chaos that is Courage, and the phenomenon of Unopened
Consciousness have been known to the Great World eons longer than Extaboulism."
"Why is that?" the woman inquired.
"Because I just made that word up", the Master said wisely.
        --Kehlog Albran, The Profit

[TUHS] Shell control through external commands

[Diomidis Spinellis]

On 14/07/2016 17:11, schoedel at kw.igs.net wrote:
> On Thu, 14 Jul 2016 16:18:43 +0300, Diomidis Spinellis wrote
>> I remember hearing that originally the Unix shell had control structures
>> (e.g. if, while, case) implemented through external commands.  However,
>>  I can't see this reflected in the source code.  The 7th Edition
>> Bourne shell has these commands built-in (usr/src/cmd/sh/cmd.c), while
>> the 6th Edition (usr/source/s2/sh.c) seems to lack them completely.
>
> http://www.tuhs.org/cgi-bin/utree.pl?file=V6/usr/source/s1/if.c
> http://www.tuhs.org/cgi-bin/utree.pl?file=V6/usr/source/s1/goto.c

Thank you!  So:
- if(1) gets as arguments an expression to evaluate and the command to 
execute if the expression is true, and
- goto(1) changes the seek offset of the shared standard input file 
descriptor to match the label's position in the file.
This is a much simpler implementation that what I thought would be the 
case, but, true to the spirit of Unix, remarkably effective.

[TUHS] Shell control through external commands

[Mary Ann Horton]

I thought the V6 Mashey shell didn't have anything built in, but there 
were external commands such as goto that would seek the open file 
descriptor to a location matching a label, which would be a comment 
(beginning with a colon, which was a no-op command.)  There would also 
have been an "if" command or option to goto or something.

On 07/14/2016 06:23 AM, Clem Cole wrote:
> Could you be confusing the fact the true and false were implemented by 
> external commands in some early shell's
>
> On Thu, Jul 14, 2016 at 9:18 AM, Diomidis Spinellis <dds at aueb.gr 
> <mailto:dds at aueb.gr>> wrote:
>
>     I remember hearing that originally the Unix shell had control
>     structures (e.g. if, while, case) implemented through external
>     commands.  However, I can't see this reflected in the source
>     code.  The 7th Edition Bourne shell has these commands built-in
>     (usr/src/cmd/sh/cmd.c), while the 6th Edition (usr/source/s2/sh.c)
>     seems to lack them completely.
>
>     The only external command I found was glob, which performed
>     wildcard expansion.
>
>     Am I missing something?  Was this implemented in a version that
>     was never released?  If so, does anyone know how this
>     implementation worked?  (Nested commands might require holding
>     some sort of globally accessible stack.)
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://minnie.tuhs.org/pipermail/tuhs/attachments/20160714/e6d40b7b/attachment.html>

[TUHS] Shell control through external commands

[schoedel]

On Thu, 14 Jul 2016 16:18:43 +0300, Diomidis Spinellis wrote
> I remember hearing that originally the Unix shell had control structures
> (e.g. if, while, case) implemented through external commands.  However,
>  I can't see this reflected in the source code.  The 7th Edition 
> Bourne shell has these commands built-in (usr/src/cmd/sh/cmd.c), while 
> the 6th Edition (usr/source/s2/sh.c) seems to lack them completely.

http://www.tuhs.org/cgi-bin/utree.pl?file=V6/usr/source/s1/if.c
http://www.tuhs.org/cgi-bin/utree.pl?file=V6/usr/source/s1/goto.c

-- 
Kevin Schoedel <schoedel at kw.igs.net> VA3TCS

[TUHS] Shell control through external commands

[Clem Cole]

Could you be confusing the fact the true and false were implemented by
external commands in some early shell's

On Thu, Jul 14, 2016 at 9:18 AM, Diomidis Spinellis <dds at aueb.gr> wrote:

> I remember hearing that originally the Unix shell had control structures
> (e.g. if, while, case) implemented through external commands.  However, I
> can't see this reflected in the source code.  The 7th Edition Bourne shell
> has these commands built-in (usr/src/cmd/sh/cmd.c), while the 6th Edition
> (usr/source/s2/sh.c) seems to lack them completely.
>
> The only external command I found was glob, which performed wildcard
> expansion.
>
> Am I missing something?  Was this implemented in a version that was never
> released?  If so, does anyone know how this implementation worked?  (Nested
> commands might require holding some sort of globally accessible stack.)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://minnie.tuhs.org/pipermail/tuhs/attachments/20160714/b761d5f3/attachment.html>

[TUHS] Shell control through external commands

[Diomidis Spinellis]

I remember hearing that originally the Unix shell had control structures 
(e.g. if, while, case) implemented through external commands.  However, 
I can't see this reflected in the source code.  The 7th Edition Bourne 
shell has these commands built-in (usr/src/cmd/sh/cmd.c), while the 6th 
Edition (usr/source/s2/sh.c) seems to lack them completely.

The only external command I found was glob, which performed wildcard 
expansion.

Am I missing something?  Was this implemented in a version that was 
never released?  If so, does anyone know how this implementation worked? 
  (Nested commands might require holding some sort of globally 
accessible stack.)