Re: [9fans] 'stacking' user-level fileservers to add 'attributes'?

Re: [9fans] 'stacking' user-level fileservers to add 'attributes'?

[rog]

> 1. dependfs
> 2. filterfs

are either of these available externally?

it's always nice to see more interesting filesystem implementations...

Re: [9fans] 'stacking' user-level fileservers to add 'attributes'?

[Eric Grosse]

> > Depend has been around for years;  see /sys/src/cmd/aux/depend.c
> > in the distribution.
> so it has! (it must have been one of the ealier alef programs?)
> interesting: though it's not immediately obvious what the various "F",
> "D" and "R" options in the .depend file do...

This seemed too parochial to be worth documenting in man pages.  It was
something I wrote back when netlib was on Unix as an email server only.
The ftp, tar-on-the-fly, 9P version was something Presotto and I did later.

For the curious, a paragraph
	F rst.f
	D rst_
	R imtql1_
	R imtql2_
in /netlib/eispack/.depend means that file eispack/rst.f defines external
symbol rst_ and that module requests external symbols imtql1_ and imtql2_.
This file is, naturally, created by a sed script applied to linker maps
when compiling the source.  It's just an automated and compact way to let
people extract useful subsets of large libraries, without hassle to either
the user or the administrator.  Sorry to drift off-topic.

Eric

[Also, sorry to describe the imaps server just now when the query was
about the client.  My face is an appropriate shade of red.]

Re: [9fans] 'stacking' user-level fileservers to add 'attributes'?

[rog]

> Depend has been around for years;  see /sys/src/cmd/aux/depend.c
> in the distribution.

so it has! (it must have been one of the ealier alef programs?)
interesting: though it's not immediately obvious what the various "F",
"D" and "R" options in the .depend file do...

  rog.

Re: [9fans] 'stacking' user-level fileservers to add 'attributes'?

[Eric Grosse]

> From: rog@vitanuova.com
> Date: Mon, 1 Oct 2001 18:01:15 +0100
>
> > 1. dependfs
> > 2. filterfs
>
> are either of these available externally?

Depend has been around for years;  see /sys/src/cmd/aux/depend.c
in the distribution.

Filterfs is newer and will be in the 9P2000 release.  It is
basically a variation on exportfs.

Eric

Re: [9fans] 'stacking' user-level fileservers to add 'attributes'?

[William Josephson]

On Mon, Oct 01, 2001 at 12:58:37PM -0400, William Josephson wrote:
> On Mon, Oct 01, 2001 at 06:10:32PM +0200, Axel Belinfante wrote:
> > Question in short: how to easily 'lift' the idea of:
> > 	producer | filter | consumer
> > to something equivalent on/using fileserver programs?
> 
> Netlib uses at least one to extract the bits of code a given routine
> depends upon into a tarball for download.  I don't think it is
> distributable, although I don't recall the reason.

Damn.  Looks as though I need a mail delivery system with lower
variance.

 -WJ

Re: [9fans] 'stacking' user-level fileservers to add 'attributes'?

[William Josephson]

On Mon, Oct 01, 2001 at 06:10:32PM +0200, Axel Belinfante wrote:
> Question in short: how to easily 'lift' the idea of:
> 	producer | filter | consumer
> to something equivalent on/using fileserver programs?

Netlib uses at least one to extract the bits of code a given routine
depends upon into a tarball for download.  I don't think it is
distributable, although I don't recall the reason.

 -WJ

Re: [9fans] 'stacking' user-level fileservers to add 'attributes'?

[Eric Grosse]

There are two vaguely related file tree filters underlying our Plan 9 web
server in support of netlib.

1. dependfs

If you look at one of the numerical libraries like eispack
   http://plan9.bell-labs.com/netlib/eispack/index.html
you see that there are lots of interrelated functions, whose call
graph forms a dag.  The "with dependencies" links on that index
page point to a file system that computes the transitive closure
starting from a node on that dag and bundles up (as a tar file)
exactly the subset of eispack that you need to make the one call.
This dependfs file system is bound into the namespace seen by
our ftp server.

2. filterfs

Exploiting the Plan 9 file system, we offer a daily snapshot of netlib
going back eight years.  See
   http://plan9.bell-labs.com/historic/index.html
Because one can't rewrite history on the dump and because sometimes
things got published by accident that shouldn't have, we inserted a
filterfs file system between the actual dump and the namespace seen
by the http server.   This filterfs also takes care of publishing
just /n/dump/*/*/netlib/pub/*, which would not be simple to do with
unadorned bind.

Eric

[9fans] 'stacking' user-level fileservers to add 'attributes'?

[Axel Belinfante]

Question in short: how to easily 'lift' the idea of:
	producer | filter | consumer
to something equivalent on/using fileserver programs?

The idea is that the producer produces a file tree which
is 'butchered' by intermediate filters that implement succesive
stages of an algorithm, by adding (and maybe also filtering (hiding?))
files, and the resulting file tree is presented to the consumer
(which might be the next filter in the chain).

Context:
I have a fileserver program that gives me (demand-driven) access
to a dag-like data structure (a labelled transition system);
each dag node is represented by a numbered directory containing
files for (e.g.) the label, the parent node number, and for the
succesor node numbers.
Reading the successor file yields the successor numbers -
if necessary it expands the tree, creating new numbered directories.
Removing a numbered directory removes a node of the dag.

The program that uses this needs for each node (i.e. for each
numbered directory) some information that can be derived from
the label. Because the computation of this information does
concern neither the dag-server nor the user program I was
thinking about having a second fileserver that extends the
file tree procduced by the dag-server with a new file in
each numbered directory of the dag-server. Effectively,
the exentension-server would be like a kind of 'filter' -
but on a file tree, instead of on text.
Does this sound like a reasonable approach?

If so, what would be the best (easiest?) way to implement this?
With the 'extender' between the user program and the dag-server
a la iostats? Or could a union mount be used where the extender
just offers the directories containing the 'extension' files -
but still the 'extender' should (lazily) follow the creation
and removal of the numbered directories, which probably breaks
this idea.
In a next step I might indeed not only extend the dag-server
with new files, but also 'filter' some files offered by the
dag-server, which seems to suggest an iostats like approach.
I wrote the dag-server prototype using lib 9p. Could that be
used here, or would it be better to use the structure of iostats?
Are there other examples of this approach in the distribution?

I'm sorry about the length; couldn't find a shorter way to
present the idea.

Axel.