Re: [9fans] dataflow programming from shell interpreter

[Sam Watkins <sam@nipl.net>]

On Mon, Jan 18, 2010 at 10:58:30AM +0000, Tim Climber wrote:
> Is this possible for UNIX philosophy to develop further? Let's say,
> XML-coded trees or graphs instead of one-line strings in stdin/
> stdout.Or LISP S-expressions. New set of utilities for filtering such
> streams, grep for XML trees, etc. Building environment for dataflow
> programming from shell interpreter.
> Any interesting papers exist on this topic?

This is interesting.  I'm not keen on XML (except for actual markup of
documents) but dataflow programming is interesting and worthwhile.

I like graphical programming (with nodes and arcs) for this, but programs can
also be represented as plain text.  I haven't implemented much to do with this
yet, but I have thought about it at length, determined what basic operators and
forms can be used for lambda, math, sets, lists etc.

I'm interested in relational / logic / dataflow programming where for example:

  a + b c

expresses a relationship.

If b and c are known, a can be calculated:  a = b + c
If a and b are known, c can be calculated:  c = b - a

This can lead to much shorter code.  Immediately you can see there is no need
for a separate subtraction operator, it is just a different mode of addition.
Likewise / is only a mode of *.  Raising to the power, taking a root,
exponentiation and logarithms are different modes of a single operator.

  + - * / ^ log   becomes   + * ^

One basic operator is needed for sets:  the discrete union, which is like a sum:

  A B # C

means  A OR B = C,  A AND B = empty


Various list relationships such as reversal, membership, head, tail, sublist,
etc.  can be represented simply with a single list symbol.

Operators can be defined using lambda closure (a box).  An operator over
paramters A, B, C, D is effectively (from some point of view) a set of all
tuples satisfying the contained relationships  { (A,B,C,D) }

So the "lambda" box thing creates a set of possible values from a free
variable.  It can be used in reverse to produce a free variable that is a
member of a set.  (also, an instance of an operator).  Lambda, application, set
definition, and set membership are the same thing (expressed by a box).

A single processor might perform both splitting and joining of text, depending
which direction you feed the data to it, etc.  Or parsing and formatting, maybe
encoding and decoding video, etc.  There will be some components of the program
used only for encoding, some only for decoding, but much can be shared between
them.

I am not fully sure how to integrate the dataflow system with the relational
system, but it seems like these models would go very well together.  Perhaps an
abstract model of a stream (like in Haskell) would be the way, perhaps a lazy
list achieves this already.  I need to think more about this.


Sam