[9fans] A personal history of pipe multiplexing

[9fans] A personal history of pipe multiplexing

[mycroftiv]

A short personal history of Plan 9 pipe muxing

This post is purely personal, historical, technical, and makes zero
comment on anything other than my own personal history of software
development.

In the summer of 2009, I'd been working with Plan 9 for slightly less
than a year, and I was mostly working on learning how to program C, by
working through k&r along with Nemo's "Intro to OS Abstractions" book.
My personal work on pipes in Plan 9 started with the "window" script
and section 12.9 of Nemo's book, on Rio.  The "window" script and the
demos showed by Nemo show how Rio is just a fileserver and the stuff
printed on the windows comes from /dev/cons.

I thought all of this was really cool, and I played with a lot, and
tried to find new tricks to do.  I found some things I thought were
interesting and possibly useful and I made this blog post on the old
9gridchan site:

http://www.9gridchan.org/blog/2009/06/17/1_blog_post/

This blog post is basically a tiny evolution of the window scripts and
Nemo's demos with one VERY signifacnt difference.  The "tee" command
is inserted into the pipelines, like this:

exec rc <{tee /dev/cons} |tee /dev/cons

So this already "muxes the pipes" in a completely traditional way.  It
makes sense as something to do in Plan 9 because of how Rio works, you
can make a new blank window and have the same shell write to 2 places.
This seemed so easy and awesome to me, and I kept working, and the
very next day you can see a blog post about doing "screen" style shell
persistence with these tricks.

http://www.9gridchan.org/blog/2009/06/18/1_blog_post/

Now this was fun but it was hardly robust and really usable.  Maybe I
could build on this and make a usable "screen" for Plan 9 with a
little bit of C code to glue it together.

http://www.9gridchan.org/older_stuff/alphasrvrc/

I don't know exactly when it was from - it was obviously before the
"real" Iosrv was released, but you can see that this program is both a
clear evolution of the earlier simple "tee" tricks and also moves the
implementation to something a lot more sophisticated - but it was
STILL basically a hack job, even though I felt like I had written a
"real" c program now, not just a beginner's exercise.

I continued working intensively with Nemo's book on os abstractions,
and you can see very easily that iosrv's parallelized lock-based
design was worked out by following along with Nemo's examples on
locking and the "bounded buffer" problem in his work.  I had heard
that writing parallel code was hard, and it sure was - if you look at
the final Iosrv, you can see an incredibly heavy debugging interface.
I remember countless hours smashing data throught the pipes, watching
everything come crashing to a halt as my locking implementation
tangled itself into a heap, and then trying to figure out why Lock B
wasn't released even though process C should have hit Lock A and...

So anyway after working very hard to get things basically usable, I
released Iosrv which is still up on sources.  Iosrv is a weird way of
implementing a 9pfs, because I didn't know how to write a 9pfs then!
It makes pipes with the pipe device, creates buffers around them, and
turns itself into a 9pfs just by wrapping the whole thing inside
exportfs and telling exportfs: "You make me a 9p fs, I don't know
how!"

Iosrv also had this extraordinarily ugly command language.  There was
a lot of power in it, and things like starting/stopping all the pipes,
individually killing attached processes, are all there - but it wasn't
graceful to use, and it seemed like I was making a ridiculous amount
of exportf processes for what the program was doing.

So, I decided to learn to make a REAL 9p fs which implemented the
exact same abstractions and ideas.  I wanted to do it in the simplest
way possible so I worked by modifying the tiny example ramfs.c in
/sys/src/lib9p which is mostly a demo of how to use 9pfile.  I worked
to make hubfs pretty much a direct "port" of iosrv to being a real
9pfs, and I also worked very hard on making the usage model much more
graceful and less awkward than Iosrv's per-pipe command language.

So anyway, that was how I ended up writing hubfs.  You can see even
from my original post that I was very conscious that what I was doing
was not a "new invention" - in fact, I was making a claim of
UN-originality, because I was linking quotes from interviews showing
that I believed hubfs was actually exactly what McIlroy had originally
described, freeform pipe muxing.

Even within the world of Plan 9, before I was doing pipe muxing, there
has been aux/consolefs forever, and Inferno has drawmux which RSC
mentioned as an idea for future Plan 9 work after I posted about
Iosrv.  I wanted to trace my own history in detail just to make it
exactly explicit how I came to write Iosrv and Hubfs, and I also think
it is an interesting story of how a beginner - me - was able to
gradually develop skill in programming and using an operating system,
and develop a piece of software that has some nice uses, because I
still use hubfs in almost exactly the same form as it was initially
released, and I think it is a nice simple tool.

Thanks for your time!  Peace and love to all Plan 9 users and
developers,

Ben Kidwell
"mycroftiv"