[9fans] typed sh (was: what features would you like in a shell?)

[9fans] typed sh (was: what features would you like in a shell?)

[fgergo]

On Thu, Apr 2, 2009 at 8:41 PM, John Stalker <stalker@maths.tcd.ie> wrote:
> What I most often miss in shell programming is a proper type system.
You should have a look at alphabet. It is cool.
http://www.vitanuova.com/inferno/man/1/sh-alphabet.html

Re: [9fans] typed sh (was: what features would you like in a shell?)

[roger peppe]

2009/4/2  <fgergo@gmail.com>:
> On Thu, Apr 2, 2009 at 8:41 PM, John Stalker <stalker@maths.tcd.ie> wrote:
>> What I most often miss in shell programming is a proper type system.
> You should have a look at alphabet. It is cool.
> http://www.vitanuova.com/inferno/man/1/sh-alphabet.html

i certainly enjoyed creating it. unfortunately it's
unfinished - i ran into unwarranted-complexity problems
trying to simulate polymorphism with processes and channels...
limbo's not a great language for writing interpreters in
when you don't know all your types in advance.

for those thinking of ideas for new shells,
i still think there's mileage in some of alphebet's ideas,
principally the way it lives in a half-way house between almost
no types (sh) and unrestricted types (most other languages).
that idea came from the thought that part of what makes
sh so powerful is the fact that so many commands
use the same types (string, stream of bytes + exit status).

i wanted to go a little beyond sh while stopping
short of the type profligacy of most other languages,
hoping to create a situation where many commands
used exactly the same types, and hence were
viable to pipeline together.

a pipeline is an amazingly powerful thing considering
that it's not a turing-complete abstraction.

alphabet was actually a generalisation of the fs command
(see http://www.vitanuova.com/inferno/man/1/fs.html),
that i'd found really useful (and more powerful than other
filesystem traversal tools i've seen, for minimal code).

some time i'll finish it!

Re: [9fans] typed sh (was: what features would you like in a shell?)

[tlaronde]

I don't know if others have already hit this kind of problematic, but I
was dealing with a fair amount of C code, usable both as a library and
accessible by a shell. Plus debugging needs. So I was, again and again,
writing a wrapper to access a C function from the shell.

So I ended concluding that I needed a kind of C interpreter as a shell.

(I have an implementation, but it is not pure C---sentential calculus is
distinct; it's a 4 values logic (NONSENSE, TRUE, FALSE, UNDECIDABLE)
that has already real application in geometrical calculus; and integer
and real calculus is added too for mathematical tasks---but it is not
ready for prime time and I have still unanswered questions for special
things I want to be present.)

typedef, i.e. the ability to define other types above primary ones is
perhaps what you are looking for?

ISTR that on the early PCees, there was a basic interpreter in BIOS to
let you play with the almost bare machine. I'd like to have a shell that
lets me play with the bare OS.
--
Thierry Laronde (Alceste) <tlaronde +AT+ polynum +dot+ com>
                 http://www.kergis.com/
Key fingerprint = 0FF7 E906 FBAF FE95 FD89  250D 52B1 AE95 6006 F40C

Re: [9fans] typed sh (was: what features would you like in a shell?)

[Bakul Shah]

On Thu, 02 Apr 2009 20:28:57 BST roger peppe <rogpeppe@gmail.com>  wrote:
> 2009/4/2  <fgergo@gmail.com>:
> i wanted to go a little beyond sh while stopping
> short of the type profligacy of most other languages,
> hoping to create a situation where many commands
> used exactly the same types, and hence were
> viable to pipeline together.

Nitpick: the output type of one command and the input type of
the next command in the pipeline has to match, not every
command.

> a pipeline is an amazingly powerful thing considering
> that it's not a turing-complete abstraction.

"f | g" is basically function composition, where f and g are
stream functions. Of course, this simple analogy breaks down
the moment we add more input/output channels -- may be that
is why anything beyond a simple pipeline seems to get people
in trouble (see the rc output redirection thread).

To go beyond simple char streams, one can for example build a
s-expr pipeline: a stream of self identifying objects of a
few types (chars, numbers, symbols, lists, vectors). In Q
(from kx.com) over an IPC connection you can send strings,
vectors, dictionaries, tables, or arbitray Q expressions. But
there the model is more of a client/server.

Re: [9fans] typed sh (was: what features would you like in a shell?)

[erik quanstrom]

> Nitpick: the output type of one command and the input type of
> the next command in the pipeline has to match, not every
> command.

i think this is wrong.  there's no requirement
that the programs participating in a pipeline are compatable
at all; that's the beauty of pipes.  you can do things
that were not envisioned at the time the programs were
written.

> To go beyond simple char streams, one can for example build a
> s-expr pipeline: a stream of self identifying objects of a
> few types (chars, numbers, symbols, lists, vectors). In Q
> (from kx.com) over an IPC connection you can send strings,
> vectors, dictionaries, tables, or arbitray Q expressions. But
> there the model is more of a client/server.

or ntfs where files are databases.  not sure if streams
can look the same way.

- erik

Re: [9fans] typed sh (was: what features would you like in a shell?)

[Bakul Shah]

On Mon, 06 Apr 2009 07:09:47 EDT erik quanstrom <quanstro@quanstro.net>  wrote:
> > Nitpick: the output type of one command and the input type of
> > the next command in the pipeline has to match, not every
> > command.
>
> i think this is wrong.  there's no requirement
> that the programs participating in a pipeline are compatable
> at all; that's the beauty of pipes.

If program A outputs numbers in big-endian order and B
expects input in little-endian order, A|B won't do the "right
thing".  Even for programs like wc have a concept of a
'character' and if the prev prog. produces something else you
will be counting something meaningless.

Perhaps it is impossible to capture such type compatibility
in anything but runtime IO routines but the concept exists.

>                                      you can do things
> that were not envisioned at the time the programs were
> written.

That comes from composability.

> > To go beyond simple char streams, one can for example build a
> > s-expr pipeline: a stream of self identifying objects of a
> > few types (chars, numbers, symbols, lists, vectors). In Q
> > (from kx.com) over an IPC connection you can send strings,
> > vectors, dictionaries, tables, or arbitray Q expressions. But
> > there the model is more of a client/server.
>
> or ntfs where files are databases.  not sure if streams
> can look the same way.

May be not.  What I was getting at was that one can do a lot
with a small number of IO types -- no need for "type
profligacy"!  [Unless your definition of profligacy is
anything more than one.]  The nice thing about s-expr is that
you have a syntax for structured IO and its printer, parser
are already written for you.  Anyway, a typed sh would be an
interesting experiment.

Re: [9fans] typed sh (was: what features would you like in a shell?)

[erik quanstrom]

> If program A outputs numbers in big-endian order and B
> expects input in little-endian order, A|B won't do the "right
> thing".

non-marshaled data considered harmful.  film at 11.  ☺

what i said was not that A|B "makes sense" for all A and B
and for any data but rather that using text streams makes
A|B possible for any A and any B and any input.  the output
might not be useful, but that is a problem on a completely
different semantic level, one that computers are usually no good at.
alsi, i don't think that "type compatability" is sufficient
to insure that the output "makes sense".  what if A produces
big-endian times in ms while B expects big-endian times in µs.

> Even for programs like wc have a concept of a
> 'character' and if the prev prog. produces something else you
> will be counting something meaningless.

that's why plan 9 uses a single character set.

but forcing compability seems worse.  where are these decisions
centralized?  how do you change decisions?  can you override
these decisions (cast)?  how does the output of, say, awk get
typed?

- erik

Re: [9fans] typed sh

[maht]

>  but rather that using text streams makes
> A|B possible for any A and any B and any input.
What is this "text" of which you speak ? ASCII EBCDIC UTF-16 UTF-8
ISO8859 etc. etc. etc.

Re: [9fans] typed sh (was: what features would you like in a shell?)

[Bakul Shah]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 2104 bytes --]

On Mon, 06 Apr 2009 12:02:21 EDT erik quanstrom <quanstro@quanstro.net>  wrote:
> > If program A outputs numbers in big-endian order and B
> > expects input in little-endian order, A|B won't do the "right
> > thing".
>
> non-marshaled data considered harmful.  film at 11.  ☺

In effect you are imposing a constraint (a type discipline).
Even if the programs themselves check such constraints, the
compatibility idea exists.

> what i said was not that A|B "makes sense" for all A and B
> and for any data but rather that using text streams makes
> A|B possible for any A and any B and any input.  the output
> might not be useful, but that is a problem on a completely
> different semantic level, one that computers are usually no good at.
> alsi, i don't think that "type compatability" is sufficient
> to insure that the output "makes sense".  what if A produces
> big-endian times in ms while B expects big-endian times in µs.

In effect you are saying that text streams allow nonsensical
pipelines as well as sensible ones and anything other than
text streams would imply giving up freedom to create sensible
pipelines as yet unthought of.  No disagreement there but see
below.

> > Even for programs like wc have a concept of a
> > 'character' and if the prev prog. produces something else you
> > will be counting something meaningless.
>
> that's why plan 9 uses a single character set.
>
> but forcing compability seems worse.  where are these decisions
> centralized?  how do you change decisions?  can you override
> these decisions (cast)?  how does the output of, say, awk get
> typed?

I am not suggesting forcing anything; I am suggesting
experimenting with s-expr streams (in the context of "typed
sh" idea). I don't know if that buys you anything more or if
you give up any essential freedom.  My guess is you'd build
something more scalable, more composable but I wouldn't
really know until it is tried.  I imagine s-expr-{grep,awk}
would look quite different from {grep,awk}.  May be you'd end
up with something like a Lisp machine.

Re: [9fans] typed sh

[erik quanstrom]

> >  but rather that using text streams makes
> > A|B possible for any A and any B and any input.
> What is this "text" of which you speak ? ASCII EBCDIC UTF-16 UTF-8
> ISO8859 etc. etc. etc.

there's got to be a latin term parallel to reducto ad absurdum that
means the opposite.  make the problem gratituiously harder
until no sensible statements can be made at all.  or maybe it's
the opposite of divide and conquer.  accrete and be subjugated?

☺.

by the way, tcs works fine for me.

- erik

Re: [9fans] typed sh (was: what features would you like in a shell?)

[John Stalker]

> but forcing compability seems worse.  where are these decisions
> centralized?  how do you change decisions?  can you override
> these decisions (cast)?  how does the output of, say, awk get
> typed?

The output of awk is a byte stream, same as its input.  The
same holds for any program.  If you want to give it some other
type then you need at least a casting mechanism.  You probably
also want a splitting mechanism, with a regular expression for
the field separator.
--
John Stalker
School of Mathematics
Trinity College Dublin
tel +353 1 896 1983
fax +353 1 896 2282

Re: [9fans] typed sh (was: what features would you like in a shell?)

[roger peppe]

2009/4/6 Bakul Shah <bakul+plan9@bitblocks.com>:
> On Thu, 02 Apr 2009 20:28:57 BST roger peppe <rogpeppe@gmail.com>  wrote:
>> a pipeline is an amazingly powerful thing considering
>> that it's not a turing-complete abstraction.
>
> "f | g" is basically function composition, where f and g are
> stream functions. Of course, this simple analogy breaks down
> the moment we add more input/output channels -- may be that
> is why anything beyond a simple pipeline seems to get people
> in trouble (see the rc output redirection thread).

actually, the analogy works fine if we add more
input channels - it's multiple output channels
that make things hard, as they mean that you have
an arbitrary directed graph rather than a tree, which doesn't
have such a convenient textual representation
and is harder to comprehend to boot.

in alphabet, i had a diagnostic channel, which was strictly
textual and not easily accessible from the language, which
was arguably not the best solution, but i didn't want things
to get too complex.

> To go beyond simple char streams, one can for example build a
> s-expr pipeline: a stream of self identifying objects of a
> few types (chars, numbers, symbols, lists, vectors).

the difficulty with s-exprs (and most nested structures, e.g. XML)
from a pipeline point of view is
that their nested nature means that any branch might contain unlimited
quantities
of stuff, so you can't always process in O(1) space, which is one of the
things i really like about pipeline processing.

i found a nice counter-example in the fs stuff - the fundamental type
was based around a "conditional-push" protocol for sending trees
of files - the sender sends some information on a file/directory
and the receiver replies whether to descend into that file or
not. the tree had a canonical order (alphabetical on name), so
tree merging could be done straightforwardly in O(1) space.

this kind of streaming "feels" like a regular pipeline, but you can't
do this with a regular pipeline. for instance, a later element in the
pipeline can prevent an earlier from descending into a part
of the file system that might block indefinitely.

every language has a trade-off between typed and untyped representations;
with alphabet i was trying to create something where it was *possible*
to create new kinds of types where necessary (as in the fs example),
but where it wouldn't be customary or necessary to do so in the
vast majority of cases.

perhaps it was folly, but i still think it was an interesting experiment,
and i don't know of anything similar.

Re: [9fans] typed sh (was: what features would you like in a shell?)

[Bakul Shah]

On Thu, 16 Apr 2009 18:24:36 BST roger peppe <rogpeppe@gmail.com>  wrote:
> 2009/4/6 Bakul Shah <bakul+plan9@bitblocks.com>:
> > On Thu, 02 Apr 2009 20:28:57 BST roger peppe <rogpeppe@gmail.com> =C2=A0w=
> rote:
> >> a pipeline is an amazingly powerful thing considering
> >> that it's not a turing-complete abstraction.
> >
> > "f | g" is basically function composition, where f and g are
> > stream functions. Of course, this simple analogy breaks down
> > the moment we add more input/output channels -- may be that
> > is why anything beyond a simple pipeline seems to get people
> > in trouble (see the rc output redirection thread).
>
> actually, the analogy works fine if we add more
> input channels - it's multiple output channels
> that make things hard, as they mean that you have
> an arbitrary directed graph rather than a tree, which doesn't
> have such a convenient textual representation
> and is harder to comprehend to boot.

True in general but certain graphs are relatively easy to
comprehend depending on what you are doing (trees, hub &
spokes, rings). Shells don't provide you a convenient
mechanism for constructing these graphs (I'd use macros in
Scheme/Lisp, or a graphics editor).

For DAGs you can use something like the example below but it
doesn't have the nice aesthetics of a pipeline!

let s0,s1 = function-with-two-output-streams
  function-with-two-input-streams(f0(s0), f1(s1), ...)

> > To go beyond simple char streams, one can for example build a
> > s-expr pipeline: a stream of self identifying objects of a
> > few types (chars, numbers, symbols, lists, vectors).
>
> the difficulty with s-exprs (and most nested structures, e.g. XML)
> from a pipeline point of view is
> that their nested nature means that any branch might contain unlimited
> quantities
> of stuff, so you can't always process in O(1) space, which is one of the
> things i really like about pipeline processing.

You can have arbitrarily long lines in a text file so if you
operate on lines, you need arbitrary buffer space. It is the
same problem.

Also note that I was talking about a stream of s-exprs, not
one s-expr as a stream (which makes no sense).  For example,

    (attach ...) (walk ...) (open ...) (read ...) (clunk ...)

> i found a nice counter-example in the fs stuff - the fundamental type
> was based around a "conditional-push" protocol for sending trees
> of files - the sender sends some information on a file/directory
> and the receiver replies whether to descend into that file or
> not. the tree had a canonical order (alphabetical on name), so
> tree merging could be done straightforwardly in O(1) space.
>
> this kind of streaming "feels" like a regular pipeline, but you can't
> do this with a regular pipeline. for instance, a later element in the
> pipeline can prevent an earlier from descending into a part
> of the file system that might block indefinitely.
>
> every language has a trade-off between typed and untyped representations;
> with alphabet i was trying to create something where it was *possible*
> to create new kinds of types where necessary (as in the fs example),
> but where it wouldn't be customary or necessary to do so in the
> vast majority of cases.
>
> perhaps it was folly, but i still think it was an interesting experiment,
> and i don't know of anything similar.