[9fans] Python filesystem

[9fans] Python filesystem

[Doug McIlroy]

I
I don't know what it has to do with Python, but
here's a little about the half-baked idea referenced in
the attached message.
In its simplest form, the idea was for the metadata
to be a single line of text, with all the metadata
in a file system unioned into a single file in which
each line begins with the (or a) name of the
associated data file.  What descriptors might
go in the metadata is left arbitrary.
To ask about a file, one would just look at the metadata.
To find a file given some descriptors, one could grep
the union file.
I never thought of an appealing way to maintain the
union file.  How, for example, can a useful name
be determined for a file, short of some ugly AI
like pwd?  What happens to the metadata when
a file gets recreated?  Assuming only one copy
of metadata exists for a file with two names, how
do you treat the metadata on unlink?
One can come up with answers to these questions, but
can one come up with a solution that will run fast
and be accurate?
Glimpse, for example, provides (a different sort of)
index to the contents of a file system by periodically
reading the whole file system.  Hence it cannot be
accurate.
The way metadata is stored matters, too.  Imagine
trying to grep the union of 10,000 files, were
each metadata item in a separate file.
If anybody thinks up a good way to overcome the
problems, I'd love to hear about it.

Doug McIlroy


>From: skipt@real.com
>Date: Thu Nov 29 14:50:26 EST 2001
>To: 9fans@cse.psu.edu
>Subject: Re: [9fans] Python filesystem
>
>In a thread long ago, Rob mentioned an idea proposed by Doug McIlroy for an
>indexed/annotated filesystem that would keep an annotation file for each
>regular file. It would seem like the right idea for keeping the mod
>documentation, etc.
>
>BTW, I've not been able to find any reference to the annotation/fs idea.
>Was anything written up (that could be shared)?
>
>>What do we lack then?  Locking and management of metadata?  There's
>>probably a way around those, as well.  The revision control FS isn't
>>well formed for saving; maybe a better solution would be to echo a
>>revision number into a ctl or new file, and then have that create a new
>>delta.  Write the file into it, and let the FS take care of picking out
>>the delta and storing it.  Perhaps a metadata file could be associated
>>with every file.  eg, foo.c;meta, foo.c;1.0, etc.
>

Re: [9fans] Python filesystem

[geoff]

An afterthought on Tenex-style version numbers: I don't believe any
implementations of file version numbers permit directories to have
versions, only plain files.  If directories had versions, and they
were kept around for a lot longer than a day, these systems would be
more usable (except for the added clutter of the directory versions)
and you probably wouldn't want nor need version numbers on plain
files.

Further, if the directory version number were incremented (and a new
directory created) only upon user request (ick), these systems would
be closer to the file server dumps:

	cd '.;3'
or
	cd '/sys/src/cmd;57'

though still rather nasty.  These systems cut the interaction of time
and the file system namespace one way, putting the
not-terribly-meaningful version information at the end using odd
syntax, which results in clutter between PURGEs and possibly after.
The file server dumps cut another way, with time-based version
information at the start, using uniform syntax, and gathering related
versions of files together (which minimises clutter), rather than
gathering related names (file;*) in a cluttered directory.

Imagine trying to go back and gather up related source files using
file version numbers as they currently stand.  You are in a cluttered
directory of files and versions.  You can't see far.  You get to pick
up files, examine them, put them down or in your backpack, and wander
around, looking at dates or diffing adjacent versions, trying to
figure out which files and versions form a consistent set, and keeping
a list as you go.  Let's see: it's fs.h;203, main.c;173, cfs.c;159,
9p.c;2, mkfile;17.  Bletch.

I just don't see the attraction.  You get ugly file names with odd
syntax, no convenient way to back up to a given time, and your
directories are always full of trash.  Such a deal.  Is there
*anything* that people find attractive about file version numbers?

I understand that people are proposing active file servers based on
Tenex-style version numbers, but I would propose to start from a solid
foundation rather than the shakey house of cards that is file version
numbers.

Re: [9fans] Python filesystem

[William Josephson]

On Thu, Nov 29, 2001 at 04:11:31PM -0800, geoff@collyer.net wrote:
> An afterthought on Tenex-style version numbers: I don't believe any
> implementations of file version numbers permit directories to have
> versions, only plain files.  If directories had versions, and they
> were kept around for a lot longer than a day, these systems would be
> more usable (except for the added clutter of the directory versions)
> and you probably wouldn't want nor need version numbers on plain
> files.

> I understand that people are proposing active file servers based on
> Tenex-style version numbers, but I would propose to start from a solid
> foundation rather than the shakey house of cards that is file version
> numbers.

There's always Elephant.... ;-/

  -WJ

Re: [9fans] Python filesystem

[geoff]

C News has the shell script that locks the news system and augments
PATH.  INN tends more to the monolithic end of the program-design
spectrum.

Version numbers in file names have some well known problems.  They
actually come from systems older than VMS, including at least
RS(u)X-11 and Tenex/TOPS-20, which takes us back about 30 years.  All
the implementations I've seen use a semi-colon to separate the real
file name from the version number.  Among other warts, this means you
have to quote them when referring to them from pretty much any shell:

	rm 'foo.c;1'
or
	rm 'foo.c;'*

In addition, the parsing of these names has traditionally been weird.
For example, foo.c means foo.c;0 (or foo.c;-1, I can't recall) which
means the most recent version of foo.c, perhaps foo.c;6.  I think the
other one (;-1 vs.  ;0) means the oldest version of foo.c still on
disk, perhaps foo.c;5.  Rob's and pjw's ``The Hideous Name'' paper
elaborates on these and other naming botches.

The reality is that you end up with a directory full of dreck rather
quickly, then the nightly PURGE command runs and clobbers all but the
most recent two or three versions (it's settable by your local
administrator and can be as low as one, if memory serves), so you
don't end up with a full history on disk, which is just as well
because you wouldn't be able to find anything.  Version numbers of
related files are not kept in synch, so you can't just

	cp *';6' /tmp/compile

to get a consistent snapshot.  File server dumps get this right:

; cd /n/dump/2001/1127/sys/src/cmd/disk/rkfs
; lc
all.h       dat.c       errno.h     ialloc.c    mkfile      sub.c
auth.c      dat.h       fcall.c     iobuf.c     portdat.h   uid.c
chk.c       dentry.c    fcallconv.c lib.h       portfns.h   utime.c
con.c       devmulti.c  fns.h       main.c      porttime.c
console.c   devwren.c   fs.c        misc.c      print.c
; mkdir /tmp/compile
; bind -bc /tmp/compile .
; mk
8c -FVw fs.c
8c -FVw sub.c
[...]
8c -FVw devwren.c
8l  -o 8.out fs.8 sub.8 porttime.8 fcall.8 iobuf.8 dat.8 main.8 dentry.8 fcallconv.8 ialloc.8 misc.8 con.8 console.8 chk.8 uid.8 auth.8 devwren.8

It's possible that Dan's comment about granularity referred to space,
not time.  The various source control systems just store diffs between
versions, but if you edit (and thus rewrite) a source file, the whole
file gets written to the dump.  Given the low cost of storage, that
seems quite acceptable.  Storing diffs to save space probably mattered
more in the mid-1970s when SCCS was written.

Re: [9fans] Python filesystem

[George Michaelson]

Its probably my naievity, but I always read the hideous name as applying
recursively to the entire path to an object, not just its terminal name
within context, which might explain why I have such a hard time with
namespaces which don't lie in the /path/to/file model. I used to have
arguments with my dad about this, he was entirely comfortable with VMS,
Tops-10, EMAS, CP/M, MS/DOS all exposing [device]:[uid,gid]/path/to/file
as part of the name construct. It seemed to me it was bad bad bad.

Having said which, putting the use of semicolon to one side, I don't know
its inherently evil to denote *SOME* token to mean 'what follows is version
information' such that a reference to a file either walks the revision
engine buried in the directory name structure logic, or walks something
like a time-based file nametree to find the best fit. That at least admits
of things like the mh cur:-10 which for MH means a range, but could mean
this thing, walk back 10 from its current state.

Clearly, its useless to do this in userspace. Either its applicable globally
for all processes which can be given objects to work on, or its really not
very useful. So VMS got it right, in as much as it DTRT for you. But yes,
they chose entirely the wrong token to use.

Anybody remember trying to live with Eunice under this stuff?

cheers
	-George
--
George Michaelson       |  APNIC
Email: ggm@apnic.net    |  PO Box 2131 Milton QLD 4064
Phone: +61 7 3367 0490  |  Australia
  Fax: +61 7 3367 0482  |  http://www.apnic.net

Re: [9fans] Python filesystem

[Russ Cox]

> cvshell?  I've noticed a trend in Plan 9 to write compressed word
> sequences using all letters, eg, cvsshell.

As opposed to all numbers?  I'm confused.
I suppose I could have written cvs.shell or cvs_shell
but that removes the ambiguity and that was most
of the fun.

> Now the question arises, is
> this a Freudian slip, or an intentional jab at CVS?  :-)

I ported CVS to Plan 9.  Does that make me more
or less likely to jab at it?  </unhelpful>

A constructive note: the biggest problem with making CVS
a file system is that it's not at all clear how to make
it present _useful_ views of the world.  The dump is nice
because the versioning is at the top of the tree, so that
each directory is internally consistent.  If you did the
same with CVS, I can easily see being overwhelemed with
needless detail in order to present consistent pictures.

Russ

Re: [9fans] Python filesystem

[Dan Cross]

In article <20011129200126.8C925199BB@mail.cse.psu.edu> you write:
>> cvshell?  I've noticed a trend in Plan 9 to write compressed word
>> sequences using all letters, eg, cvsshell.
>
>As opposed to all numbers?

Aren't all letters also numbers?  Certainly, that's true in the semetic
languages.  One can think of the Roman letters of English as being
numbered; a is 1, b is 2, etc.  Then of course there are the character
encoding systems such as ASCII, EBCDIC, Unicode, Baudot, etc.

>I'm confused.

It's a natural state for lots of people.  I estimate that I spend
90%-95% of my time hopelessly confused.  :-)

>I suppose I could have written cvs.shell or cvs_shell
>but that removes the ambiguity and that was most
>of the fun.

And that's what I meant: cvshell; is that CVS Hell, or CVS Shell with
an attempt to save a character?

>> Now the question arises, is
>> this a Freudian slip, or an intentional jab at CVS?  :-)
>
>I ported CVS to Plan 9.  Does that make me more
>or less likely to jab at it?  </unhelpful>

Well....  If the things Boyd says about CVS are any indication, it
probably makes you more likely to jab at it.  But then there's the
possibility that you did so subconsiously.  Note that I didn't leave
the option of it being a typo.  :-)

>A constructive note: the biggest problem with making CVS
>a file system is that it's not at all clear how to make
>it present _useful_ views of the world.  The dump is nice
>because the versioning is at the top of the tree, so that
>each directory is internally consistent.  If you did the
>same with CVS, I can easily see being overwhelemed with
>needless detail in order to present consistent pictures.

This is why I was looking at what VMS did; by appending the version
number on the end of the file, and then adding hooks such that opening
foo.dat by default opened the latest version of foo.dat, they
sidestepped much of the detail.  Creating a file automatically created
a new version.  I know several VMS programmers who never bothered with
revision control; the filesystem did it for them.

The downside was that you could have huge numbers of old files hanging
around, and the entire file was copied each time the editor saved a new
version, even if you only changed one character.  That was a pain, and
lead to the necessity for the PURGE command to delete old copies (I
suppose you could have done the same thing with DELETE, but it was a
pain).

At anyrate, I wonder if a two-filesystem model would work; the lower
one holds deltas, etc, and the higher one provides a useful view onto
that.
	- Dan C.

Re: [9fans] Python filesystem

[Boyd Roberts]

> Well....  If the things Boyd says about CVS are any indication, it
> probably makes you more likely to jab at it.

We use CVS at Strakt -- you have no choice.  Revision control is
a must and CVS is the your least worst choice.

> Creating a file automatically created
> a new version.  I know several VMS programmers who never bothered with
> revision control; the filesystem did it for them.

I heard the VMS system hackers developed on UNIX, 'cos VMS was too
'orrible :)

I could tell you a story about ULTRIX revision control, but that
could get me into some amount of trouble, given I signed my life
away some 10 years back :)

Re: [9fans] Python filesystem

[plan9]

On Thu, Nov 29, 2001 at 09:49:58PM +0100, Boyd Roberts wrote:
>
> I could tell you a story about ULTRIX revision control, but that
> could get me into some amount of trouble, given I signed my life
> away some 10 years back :)

Who would sue you?  Compaq?  Intel?  HP?  I'm sure the paper with your
signature on it is hopelessly lost in the Black Hole of Mergers, from
which even a coherent business plan can't escape.

Re: [9fans] Python filesystem

[Boyd Roberts]

> Who would sue you?

I signed the paper(s).  I am ethically bound (to my mind) and
legally bound.

And AFAIK rob fought hard to get the 3rd release out.

What I mean to say is that lawyers are a problem.

Sure, ULTRIX is dead, but I am am still bound by
the papers I signed.  I you have a reason to the
contrary I suggest you contact Paul Vixie.

Re: [9fans] Python filesystem

[Russ Cox]

#!/bin/rc

if(! ~ $#* 0){
	echo 'usage: cvshell' >[1=2]
	exit usage
}

while(echo -n 'cvs> '; r=`{read})
	rc -c 'cvs '^$r

sorry.  couldn't help myself.

Re: [9fans] Python filesystem

[Dan Cross]

In article <20011129144905.6695D19A9F@mail.cse.psu.edu> you write:
>	echo 'usage: cvshell' >[1=2]

cvshell?  I've noticed a trend in Plan 9 to write compressed word
sequences using all letters, eg, cvsshell.  Now the question arises, is
this a Freudian slip, or an intentional jab at CVS?  :-)

I was thinking about this problem, and one nice thing about RCS/CVS/
whatever is that they store deltas of a file up to a current version.
This is what /n/dump does, execpt that /n/dump has a non-arbitrary
level of granularity, which seems to be what is missing.  Labelling
versions could be handled by a meta-database of files at another level;
eg, have a directory versions/ with sudirectories and files
corresponding to the versions of files that make up a release.

	versions/1/2/1/vers

The files have standard namespace(6) format.  Or maybe even be mkfs
prototypes.  Anyway, there's the ability to make release from arbitrary
things on the dump; just create a namespace with all the named files in
it, and tar it up.

But what about arbitrary commitment to the dump?  It struck me that an
filesystem that mimmicked VMS file versions could give you that.
create(2)'ing a file would make a new version, which was really a delta
on the old.  If the deltas were kept in normal RCS-like files, and
reconsitituted on the fly by the revison control fs, you'd get
basically RCS/CVS, but with the ability to edit things using `normal'
tools; no more CVS commands or strange tools to manipulate the
repository.  Files would be named, eg, file.c;1.4.3.

What do we lack then?  Locking and management of metadata?  There's
probably a way around those, as well.  The revision control FS isn't
well formed for saving; maybe a better solution would be to echo a
revision number into a ctl or new file, and then have that create a new
delta.  Write the file into it, and let the FS take care of picking out
the delta and storing it.  Perhaps a metadata file could be associated
with every file.  eg, foo.c;meta, foo.c;1.0, etc.

Maybe I'm smoking my hair; these thoughts are only half formed.

	- Dan C.

Re: [9fans] Python filesystem

[Skip Tavakkolian]

In a thread long ago, Rob mentioned an idea proposed by Doug McIlroy for an
indexed/annotated filesystem that would keep an annotation file for each
regular file. It would seem like the right idea for keeping the mod
documentation, etc.

BTW, I've not been able to find any reference to the annotation/fs idea.
Was anything written up (that could be shared)?

>What do we lack then?  Locking and management of metadata?  There's
>probably a way around those, as well.  The revision control FS isn't
>well formed for saving; maybe a better solution would be to echo a
>revision number into a ctl or new file, and then have that create a new
>delta.  Write the file into it, and let the FS take care of picking out
>the delta and storing it.  Perhaps a metadata file could be associated
>with every file.  eg, foo.c;meta, foo.c;1.0, etc.

Re: [9fans] Python filesystem

[Boyd Roberts]

> I was thinking about this problem, and one nice thing about RCS/CVS/
> whatever is that they store deltas of a file up to a current version.

So does /n/dump as it only makes copies of blocks that have been modified.
The various versions are held together by the tree.  An unmodified block
is shared by all 'versions' of the file.

> This is what /n/dump does, execpt that /n/dump has a non-arbitrary
> level of granularity, which seems to be what is missing.

IIRC it does:

    /n/dump/2001/1129[a-z]

> Labelling versions could be handled by a meta-database of files at
> another level; eg, have a directory versions/ with sudirectories and files
> corresponding to the versions of files that make up a release.
>
> versions/1/2/1/vers

No this is horrible.  It falls straight out of /n/dump.

> The files have standard namespace(6) format.

Yes this is right.  You have file(s) that build a namespace(s)
and then you type:

    mk [ dist ]

and it just does it.

> Or maybe even be mkfs prototypes.  Anyway, there's the ability to make
> release from arbitrary things on the dump; just create a namespace with
> all the named files in it, and tar it up.

Err, I'd stick to small n formats otherwise it just complicates the problem.

> But what about arbitrary commitment to the dump?  It struck me that an
> filesystem that mimmicked VMS file versions could give you that.
> create(2)'ing a file would make a new version, which was really a delta
> on the old.

/n/dump already does it during the dump.  Just turn it on via bind(1).

> If the deltas were kept in normal RCS-like files, and
> reconsitituted on the fly by the revison control fs, you'd get
> basically RCS/CVS, but with the ability to edit things using `normal'
> tools; no more CVS commands or strange tools to manipulate the
> repository.  Files would be named, eg, file.c;1.4.3.

No /n/dump already does it.  VMS file versions are a disaster.  Qids
are a much better idea, but it is a slightly different problem.

> What do we lack then?

We lack a mod to bind(1)/9P, some namespace files and some mkfiles.

It all falls straight out of /n/dump.

I'm not sure how a copy-on-write bind would fit into 9P, because
I haven't looked at 9P for a long time, but I'm assuming it's
doable; a bit like writable union directories.

Re: [9fans] Python filesystem

[rog]

> the meta anti-spam RFC 822 header parser version

cool! do they do these in Woolworth's?

Re: [9fans] Python filesystem

[Boyd Roberts]

rog wrote:
> cool! do they do these in Woolworth's?

They might do soon, given the heir to the Woolworth
fortune is a mate of mine.

Re: [9fans] Python filesystem

[David Gordon Hogan]

> I'm surprised more of the command line tools aren't daemonised actually.

Perl is frequently demonized.

Re: [9fans] Python filesystem

[Russ Cox]

why not just post a connection to a real shell
in /srv and open it each time you get a request?
then you have a persistent environment.

if you were worried about concurrent access
then i guess i could see something like /net/cs
that just serves conversations, but otherwise
a file system seems like overkill.

Re: [9fans] Python filesystem

[Matt]

On Wednesday 28 November 2001 18:54, you wrote:
> why not just post a connection to a real shell
> in /srv and open it each time you get a request?
> then you have a persistent environment.

tbh I don't know :) that's why I was asking
I don't my head is fully round /srv just yet

> if you were worried about concurrent access
nope :) not yet

> a file system seems like overkill.
maybe but not necessarily. exposing the local variables as files seems
interesting

I'm surprised more of the command line tools aren't daemonised actually.

mounting things like awk & sed

%echo '/^something/ {print $2}' > /n/awk/prog
%cat text > /n/awk/stdin
%cat /n/awk/stdout
%cat moretext > /n/awk/stdin
%cat /n/awk/stdout

maybe my head is too full of something :)

If I stop thinking maybe it will go away

M

Re: [9fans] Python filesystem

[Boyd Roberts]

I am really unconvinced why you would want to do this.

However, you could access python data structures through a
f/s, but I'm not sure it would give you anything useful.

Re: [9fans] Python filesystem

[Matt]

On Wednesday 28 November 2001 21:46, you wrote:
> I am really unconvinced why you would want to do this.
>
> However, you could access python data structures through a
> f/s, but I'm not sure it would give you anything useful.

ok then here goes my rambling idea

I've been working with Apache
Apache has a thing called "the request loop" which is (mostly)

Request Received
URI Translation
Access Control
Authentication
Authorisation
Response

with mod_perl & mod_snake one can add 0 or more custom handlers at each stage
of the loop to alter the default behavior

I was day dreaming about what a plan9 approach to httpd would be and thought
that the Apache model would suit as a good starting point.
As I've been studying file servers for my other project I came up with an
httpd fileserver for which one could attach a list of commands to execute at
each stage of the loop. These commands could alter the values of the
request/response objects as exposed by the httpdfs.

but I was concerned by two things
1. lots of process forks to spawn the commands
2. persistence

both of which seem to be cured by having the commands daemonised

putting a channel in /srv is one way I guess but there would still need to be
some glue for the /srver to write back to the request/response object esp. if
one was to go for multiple httpd processes and therefore possibly multiple
/servers doing the same job for different httpds. With httpd using multiple
threads I can feel the projects hair growing and mine getting pulled out.

M

Re: [9fans] Python filesystem

[Lucio De Re]

On Wed, Nov 28, 2001 at 07:09:58PM +0000, Matt wrote:
>
> I'm surprised more of the command line tools aren't daemonised actually.
>
My thinking (just to show how muddled one can get) was to turn
environments into shells, instead.  Take CVS, for example:

cvs login
cvs co
cvs update
etc.

I'd have a CVS shell accepting all sort of commands:

% CVS
cvs> login
cvs> co
...

where the commands are scripts and executables built into the shell (or
not, as one sees fit) and bound to /bin (the traditional home for them)
in as restricted a namespace as one finds necessary.

Very, very vague, I fear.

++L

Re: [9fans] Python filesystem

[Boyd Roberts]

> My thinking (just to show how muddled one can get) was to turn
> environments into shells, instead.  Take CVS, for example:

No, this is RAND [MH] mail hell.

CVS has some good ideas, but it leaves you with a polluted
source tree.

/n/dump is pretty cool, but I don't see it releasing coherent
releases.  At 1127 it's probably fine, but all the world is
not 1127.

It's a tricky problem.  Version control is absolutely necessary,
but without the pollution.

This was never commercialised, and it had its problems, but I
think it has part of the solution.

    Prusker Francis J. and Wobber Edward P. The Siphon: Managing Distant
    Replicated Repositories. PRL Research Report #7, Nov 1990

Hmm, perhaps a copy-on-write option to bind with a later replace?

Re: [9fans] Python filesystem

[George Michaelson]

> > My thinking (just to show how muddled one can get) was to turn
> > environments into shells, instead.  Take CVS, for example:
>
> No, this is RAND [MH] mail hell.

I'm missing something. there is no magic binding glue in MH. mail is files
and thats it.

whats hellish about MH?

-George
--
George Michaelson       |  APNIC
Email: ggm@apnic.net    |  PO Box 2131 Milton QLD 4064
Phone: +61 7 3367 0490  |  Australia
  Fax: +61 7 3367 0482  |  http://www.apnic.net

Re: [9fans] Python filesystem

[Boyd Roberts]

> whats hellish about MH?

There is a fundemental design flaw in 'comp'.  When you quit the
editor it says (IIRC):

    What now?

Well this is just no good.  Composition and delivery should be
decoupled and you should be able to edit multiple messages at
once and deliver them at will.

This is what I mean:

#!/bin/sh

# dws [ box ] [ id ]
#
# die worthless spammer

myname="`basename \"$0\"`"
spam=+spam

case "`box`" in
$spam)
 echo "$myname: Replying to a spam in \"$spam\"?" 1>&2
 exit 1
 ;;
esac

case $# in
0|1|2)
 mov ${1+"$@"} "$spam" || exit $?
 ;;

*)
 echo "usage: $myname [ box ] [ id ]" 1>&2
 exit 1
 ;;
esac

box="`box`"

box "$spam" || exit $?

# construct reply
(
 EDITOR='sam -d' rep -i > /dev/null 2>&1 <<'!'
/^To:.*\n(    .*\n)+/
x/\n    /c/ /
/^To:.*\n/
.t.
x/[\-a-zA-Z0-9._&]+@/c/postmaster@/
/^To:.*\n/
/^To:.*\n/
s/^To:/Cc:/
,x/^Cc: \n/d
,x/^Bcc: \n/d
,x/^Subject: \n/d
1,/^\n/
a
die, worthless spammer.

postmaster: check out the Mail Abuse Protection System (MAPS)
     http://maps.vix.com

.
1,/^$/p
w
q
!
# log reply and deliver
) || exit 1

case "$myname" in
dws)
 echo "$myname: Spam returned to `msg | 822flatten | sed -e
^[TC][oc][  ]*:[  ]*/!d' -e 's///' | tr '\012' ' '`" 1>&2
 del && box "$box"
 ;;

rws)
 med
 ;;
esac

----

Unfortunately I just realised that I had broken the 'mace' bundle because
I put up the meta anti-spam RFC 822 header parser version that was still
in test.  Unfortunately the Received: headers have a context sensitive
grammar which yacc is not quite up to.

My plan was to walk the Received: headers and automatically copy
abuse/postmaster
at all the sites the spam had passed through.  When I have a spare
nanosecond
I will fix it, because I can't stand these broken GUI mailers.  Yes, I will
have
to deal with MIME, but I can creep up on that with a tools approach.

Une chose � la fois.

Re: [9fans] Python filesystem

[Sam Holden]

On Thu, 29 Nov 2001 07:18:45 GMT, Boyd Roberts <boyd@fr.inter.net> wrote:
>> whats hellish about MH?
>
>There is a fundemental design flaw in 'comp'.  When you quit the
>editor it says (IIRC):
>
>    What now?

'comp' doesn't do that. 'whatnow' does that (well on my version comp
pretends to be whatnow if whatnowproc is called whatnow - but that's a
bug imho).

>Well this is just no good.  Composition and delivery should be
>decoupled and you should be able to edit multiple messages at
>once and deliver them at will.

You can.

That's what the -draftfolder and -draftmessage switches are for.

I compose messages for later sending quite often using mh (well nmh). Not
for doing automated script things I asmit (I just send as I go) but when
composing a message which I wish to put off for a few hours/days. I've had
more than one of these at the same time, and happily sent and recieved other
mail in the meantime...

--
Sam Holden

Re: [9fans] Python filesystem

[Ralph Corderoy]

> There is a fundemental design flaw in 'comp'.  When you quit the
> editor it says (IIRC):
>
>     What now?

It's whatnow(1) that says that, not comp(1).  That's why you get the
same `whatnow' prompt from repl(1), dist(1), etc.

> Well this is just no good.  Composition and delivery should be
> decoupled and you should be able to edit multiple messages at once
> and deliver them at will.

comp is targetted at the interactive user.  Perhaps you want MH's
send(1) and post(8) programs instead?


Ralph.

Re: [9fans] Python filesystem

[Boyd Roberts]

> comp is targetted at the interactive user.  Perhaps you want MH's
> send(1) and post(8) programs instead?

I understand what you are saying, but it's not what I want or
wanted so I wrote it myself.  Unfortunately I put a WIP on the
web.  I have to do a small amount of work to fix it.

RFC 822 goes to all this trouble to have this horrendously,
useless complexity for the addresses and then decides on
a context sensitive grammar for the Received: fields.

Being able to walk them (which was my plan) and to send mail
to abuse or postmaster for every host that touched the spam
was my idea of backpressure :)

Re: [9fans] Python filesystem

[Lucio De Re]

On Thu, Nov 29, 2001 at 07:30:25AM +0100, Boyd Roberts wrote:
>
> > My thinking (just to show how muddled one can get) was to turn
> > environments into shells, instead.  Take CVS, for example:
>
> No, this is RAND [MH] mail hell.
>
I'd forgotten about MH.  Yes, that's precisely the model, but with Plan
9's private namespaces instead of an arbitrary collection of badly
named modules.

MH struck me as clumsy more because of the selection of module
functions and names than out of a failing in the concept.  After all,
it is the nature of Unix to have simple commands that can be strung
together to produce complex results, where does MH's concept fail?

As another example, was it C News or INN that had a shell environment that
locked the news system while providing a more practical "path"?

Those are half-baked ideas that might have a useful eventual
resolution.  Plan 9's user namespaces make that much more practical.

++L

Re: [9fans] Python filesystem

[Boyd Roberts]

> MH struck me as clumsy more because of the selection of module
> functions and names than out of a failing in the concept.  After all,
> it is the nature of Unix to have simple commands that can be strung
> together to produce complex results, where does MH's concept fail?

MH is too interactive.

Without trickery you can't really build meta-tools with it.

You know, the power of the shell/rc and the pipeline etc...

IIRC you couldn't use B as your $EDITOR with MH comp.

With my cut down, simpler version:

    EDITOR=B
    com
    <edit with sam>
    del

With sam you could have multiple messages being edited at once
and you del[iver] them as needed.

Re: [9fans] Python filesystem

[Ralph Corderoy]

Hi Boyd,

> MH is too interactive.

I agree its interface isn't great for scripting.

> IIRC you couldn't use B as your $EDITOR with MH comp.

You use its -editor option?

    % comp -editor true

    What now? q -d

> With sam you could have multiple messages being edited at once and
> you del[iver] them as needed.

With MH you have a folder called drafts and comp starts a new draft or
you can say `comp -u 4' to resume draft number four.  As it's just
another mail folder in other respects, commands like rmm work on it
too.

    % comp -editor prargv
    0 '/home/ralph/bin/prargv'
    1 '/home/ralph/mail/drafts/5'

    What now? q -d

Cheers,


Ralph.

Re: [9fans] Python filesystem

[Skip Tavakkolian]

CVS or a derivative thereof, should be a filesystem. It seems to me that
something like ftpfs is very close to what a CVS fs could be. Assuming a
pserver is managing the sources, the cvsfs connects and shows the user the
source repository.  The user can copy files out of and into of the
filesystem (causing checkouts and checkins).  I think starting very simple
(just checkins and checkouts) would still be very useful.

At 07:49 AM 11/29/2001 +0200, Lucio De Re wrote:
>On Wed, Nov 28, 2001 at 07:09:58PM +0000, Matt wrote:
>>
>> I'm surprised more of the command line tools aren't daemonised actually.
>>
>My thinking (just to show how muddled one can get) was to turn
>environments into shells, instead.  Take CVS, for example:
>
>cvs login
>cvs co
>cvs update
>etc.
>
>I'd have a CVS shell accepting all sort of commands:
>
>% CVS
>cvs> login
>cvs> co
>...
>
>where the commands are scripts and executables built into the shell (or
>not, as one sees fit) and bound to /bin (the traditional home for them)
>in as restricted a namespace as one finds necessary.
>
>Very, very vague, I fear.
>
>++L
>
>

Re: [9fans] Python filesystem

[Steve Kilbane]

> CVS or a derivative thereof, should be a filesystem. It seems to me that
> something like ftpfs is very close to what a CVS fs could be. Assuming a
> pserver is managing the sources, the cvsfs connects and shows the user the
> source repository.  The user can copy files out of and into of the
> filesystem (causing checkouts and checkins).  I think starting very simple
> (just checkins and checkouts) would still be very useful.

Quite probably, although I think you'd want to run through the basic set of
operations and work out how they'd function, before doing anything at all.
To work as a file server, it would need to support the common activities in
a natural manner. Otherwise, users would have to keep returning to the usual
interface, and would be frustrated.

Re: [9fans] Python filesystem

[Laura Creighton]

Filesystems are very nice things, but before you spend a lot of time
inventing an improved CVS consider -- is the _file_ the basic unit
you wish in developing software?  I want something smaller.

I would dearly like a way to indicate that this file has now become
2 files, and that class no longer lives in this one.  So I do not
wish to see old hacked versions of that class magically reappearing
in this file every month as more people check in code.

The other great problem that I have is that in the middle of hacking up
something, I discover a memory leak.  Usually when I find one, I
find a pattern that happens all over the code base.  I now want to
stop whatever I am doing, get a new fresh version of the code base,
and fix the memory leak once and for all throughout everything.  I do
not wish either to lose my current hacking, or have to finish them
before I can go kill that memory leak.

Currently, I mostly cheat.  I go to another machine, and log in as
another user I have created for that purpose, and nail the memory leak.
Then I go back to being me, and back to what I was hacking on.  That
I find this the most convenient way to solve a problem I have all the
time is an indication that my usual work habits and the work habits
assumed by cvs do not mesh well.

I think we had better figure out what we want in a CVS replacement before
we start replacing it.  I want to tag things at the per object level.
What do the rest of you want?

Laura Creighton

Re: [9fans] Python filesystem

[Ralph Corderoy]

Hi Laura,

> The other great problem that I have is that in the middle of hacking
> up something, I discover a memory leak.  Usually when I find one, I
> find a pattern that happens all over the code base.  I now want to
> stop whatever I am doing, get a new fresh version of the code base,
> and fix the memory leak once and for all throughout everything.  I do
> not wish either to lose my current hacking, or have to finish them
> before I can go kill that memory leak.
>
> Currently, I mostly cheat.  I go to another machine, and log in as
> another user I have created for that purpose, and nail the memory
> leak.  Then I go back to being me, and back to what I was hacking on.
> That I find this the most convenient way to solve a problem I have
> all the time is an indication that my usual work habits and the work

And this is with CVS?  If so, why not check out another copy of the
source to work on?

    mkdir work1 && cd work1
    cvs co myproject
    # work away, spot problem
    mkdir ~/work2 && cd ~/work2
    cvs co myproject
    # fix widespread leak, altering many files
    cvs ci -m'fix ...'
    cd
    rm -rf work2
    cd work1
    cvs update   # to merge in all the changes committed from work2

Unlike with SCCS and RCS you can have multiple copies of the CVS
repository contents to work on at once as the `history files', e.g.
*,v, are separated from the working area.  (I know this is possible
with SCCS and RCS, but not their normal manner of working.)

Cheers,


Ralph.

Re: [9fans] Python filesystem

[Laura Creighton]

Ralph Corderoy <ralph@inputplus.demon.co.uk> explains to me how
to use cvs without hopping around like a frog in the fire.  Hmmm.
I tried to do this once, botched it, and stupidly concluded that
it couldn't be done.  Thank you for enlightening me.  My coworkers
who will be pleased to see that I am not hogging 3 or 4 terminals in
the main terminal room will also bless the day you took the time to
post this.

Laura

Re: [9fans] Python filesystem

[Boyd Roberts]

> CVS or a derivative thereof, should be a filesystem. It seems to me that
> something like ftpfs is very close to what a CVS fs could be.

It should probably be a filesystem, but nothing like ftpfs.  FTP
is there to copy files.  CVS is just a meta RCS which is not a good
thing.  RCS is useful and simple but it's useless if you want to use
it for serial #'s in DNS zone files (SCCS is great for that, but not
good for much else).

I think /n/dump needs a layer or a concept of grouping a chunk of stuff
together which constitutes a release.  Now, let's not go mad and go the
whole hog as Vesta did.

One day they found a serious design flaw in Vesta -- it ran out of space,
apart from he fact it was excruciatingly slow.  Instead of building a list
of things to blow away [failsafe] it built a list of things to keep.  The
trouble was that the 'list' was written to a file, but the file-system(s)
were full -- so, it created the null list of things to keep.

This resulted in that it went ahead and it started to delete everything.

FYI: Vesta was a project from SRC.  It sort of did /n/dump but instead
     of bind-ing the universe together it would just re-create it from
     scratch -- ouch.

     Vesta may have been able to re-create the universe in small n days,
     but it was extremely efficient in destroying in fewer.

Re: [9fans] Python filesystem

[Christopher Nielsen]

On Thu, Nov 29, 2001 at 08:37:21AM +0100, Boyd Roberts wrote:

> > CVS or a derivative thereof, should be a filesystem. It seems to me that
> > something like ftpfs is very close to what a CVS fs could be.
>
> It should probably be a filesystem, but nothing like ftpfs.  FTP
> is there to copy files.  CVS is just a meta RCS which is not a good
> thing.  RCS is useful and simple but it's useless if you want to use
> it for serial #'s in DNS zone files (SCCS is great for that, but not
> good for much else).
>
> I think /n/dump needs a layer or a concept of grouping a chunk of stuff
> together which constitutes a release.

When I brought up subversion in an earlier thread, using some of
its ideas was more my intent instead of porting the tools with
which it seems tightly coupled. I didn't really have the time
to respond properly before, but Boyd seems to have articulated
some of what I wanted to say.

I like the idea of source control being an fs. It seems natural.
/n/dump is a very cool idea, but it is lacking in the scm arena.
That's not surprising, since it seems to me that it was designed
more for backups.

My suggestion is to take a look at the architecture of subversion
and find the bits that look useful and seem natural. Maybe we can
come up with a better scm system.

I'll be happy to work on such a beast, when/if I have the time.

--
Christopher Nielsen - Metal-wielding pyro techie
cnielsen@pobox.com
"Those who are willing to trade freedom for security deserve
 neither freedom nor security." --Benjamin Franklin

Re: [9fans] Python filesystem

[Skip Tavakkolian]

I was assuming that releases would all be under one directory. I see that
it is flawed.

At 08:37 AM 11/29/2001 +0100, Boyd Roberts wrote:
>
>> CVS or a derivative thereof, should be a filesystem. It seems to me that
>> something like ftpfs is very close to what a CVS fs could be.
>
>It should probably be a filesystem, but nothing like ftpfs.  FTP
>is there to copy files.  CVS is just a meta RCS which is not a good
>thing.  RCS is useful and simple but it's useless if you want to use
>it for serial #'s in DNS zone files (SCCS is great for that, but not
>good for much else).
>
>I think /n/dump needs a layer or a concept of grouping a chunk of stuff
>together which constitutes a release.  Now, let's not go mad and go the
>whole hog as Vesta did.
>
>One day they found a serious design flaw in Vesta -- it ran out of space,
>apart from he fact it was excruciatingly slow.  Instead of building a list
>of things to blow away [failsafe] it built a list of things to keep.  The
>trouble was that the 'list' was written to a file, but the file-system(s)
>were full -- so, it created the null list of things to keep.
>
>This resulted in that it went ahead and it started to delete everything.
>
>FYI: Vesta was a project from SRC.  It sort of did /n/dump but instead
>     of bind-ing the universe together it would just re-create it from
>     scratch -- ouch.
>
>     Vesta may have been able to re-create the universe in small n days,
>     but it was extremely efficient in destroying in fewer.
>
>
>

Re: [9fans] Python filesystem

[John Murdie]

On 29 Nov, Lucio De Re wrote:
> On Wed, Nov 28, 2001 at 07:09:58PM +0000, Matt wrote:
>>
>> I'm surprised more of the command line tools aren't daemonised actually.
>>
> My thinking (just to show how muddled one can get) was to turn
> environments into shells, instead.  Take CVS, for example:
>
> cvs login
> cvs co
> cvs update
> etc.
>
> I'd have a CVS shell accepting all sort of commands:
>
> % CVS
> cvs> login
> cvs> co
> ...
>
> where the commands are scripts and executables built into the shell (or
> not, as one sees fit) and bound to /bin (the traditional home for them)
> in as restricted a namespace as one finds necessary.
>
> Very, very vague, I fear.
>
> ++L

And very, very, retro, and contrary to the Unix (and Plan 9)
`philosophy' of putting commonly-required facilities in (just) one
place. If you did the above, wouldn't you have to add all the non-CVS
facilities of a shell to CVS (and to everything else you turned into a
shell)?

I'm sure a lot of us remember DEC's PIP; file manipulation in a
closed-off environment which had separate and very different grammar
rules from the shell. Uggh!
--

John A. Murdie
Experimental Officer (Software)
Department of Computer Science
University of York
England

Re: [9fans] Python filesystem

[Boyd Roberts]

> I'm sure a lot of us remember DEC's PIP; file manipulation in a
> closed-off environment which had separate and very different grammar
> rules from the shell. Uggh!

Oh yes, PIP.  Get those arguments wrong and kiss your data goodbye.

Re: [9fans] Python filesystem

[Lucio De Re]

On Thu, Nov 29, 2001 at 10:08:30AM +0000, John Murdie wrote:
>
> And very, very, retro, and contrary to the Unix (and Plan 9)
> `philosophy' of putting commonly-required facilities in (just) one
> place. If you did the above, wouldn't you have to add all the non-CVS
> facilities of a shell to CVS (and to everything else you turned into a
> shell)?
>
A valid point, but one that may apply to shared objects too  :-)
:-)  :-)

My intention was to cause the shell to build a namespace with the
necessary (and only the necessary) tools in place.  Those that are
unique to the shell would be built (in a funny sense) into the
shell and "exported" (that's an interesting solution to the lack
of private environment variables in rc - just don't add them to
/env, which means serving /env within the shell, doesn't it?) into
the namespace as appropriate.

I'm kind of looking for seamlessness between the shell and the
namespace and, for the sake of protection, limiting the namespace
to specific instances of modules that may have been sanitised for
the purpose.  Or extended, for that matter.

I like embedded languages and my limited efforts with the namespace
library made me think on how best to combine embedding with the
more practical aspects of a shell and the environment it provides.
But it is just rambles, right now.

++L

Re: [9fans] Python filesystem

[Russ Cox]

> python is not just a script interpreter but it is also a kind of shell
> fire it up without a script and you can execute commands 'interactively'
> which means it's quite a candidate for a file system

you lost me here.  rc is not a file system.

russ

Re: [9fans] Python filesystem

[Matt]

On Wednesday 28 November 2001 18:21, you wrote:
> > python is not just a script interpreter but it is also a kind of shell
> > fire it up without a script and you can execute commands 'interactively'
> > which means it's quite a candidate for a file system
>
> you lost me here.  rc is not a file system.
>

maybe the shell word was a bad choice

run the pyfs

%echo 'print "hello"' > /n/pyfs/stdin
%cat /n/pyfs/stdout
hello
%

once could also have

%ls n/pyfs/locals
and get a list of local variables and their values (possibly writeable)

the real reason I'm thinking about it is to use it as CGI along the lines of
mod_perl or mod_snake where the environment is persistent across HTTP requests
but I didn't really want to introduce HTTP into the conversation just yet :)

[9fans] Python filesystem

[Matt]

I know the answer to this mail should be :

"send us the code when it's finished"

but I've been musing the past couple of days about a python filesystem

python is not just a script interpreter but it is also a kind of shell

fire it up without a script and you can execute commands 'interactively'

which means it's quite a candidate for a file system

is it a technique that is used anywhere else?

the ability to embed a programming language that an application can send code
to progressively sounds quite interesting

any thoughts?

M

Re: [9fans] Python filesystem

[Quinn Dunkan]

> I know the answer to this mail should be :
>
> "send us the code when it's finished"
>
> but I've been musing the past couple of days about a python filesystem
>
> python is not just a script interpreter but it is also a kind of shell

Yes, most languages have REPLs.

> fire it up without a script and you can execute commands 'interactively'
>
> which means it's quite a candidate for a file system
>
> is it a technique that is used anywhere else?

plumber?

> the ability to embed a programming language that an application can send code
>
> to progressively sounds quite interesting

Isn't that just like creating a named pipe to a shell or REPL?  Isn't that
what acme's win does?

A while back I wrote a lua program that served lua variables---a table becomes
a directory and strings become files, and functions became files containing
whatever string the function returns when passed nbytes read (bytes written
are passed as a string).  And since most everything in lua is a table,
including the global namespace, you could assign variables by writing to
files.  It presented a possibly interesting debugging interface, but not
practical.  Type information is lost, because of table <-> directory and
everything else <-> bytes in file.  And it's purely value-oriented, so as long
as you're treating data functionally you're fine, but if you want to alias you
can't.

But it's a fun way to throw up a filesystem fast (or interactively, by
assigning to the served table from the REPL).