[9fans] everything is a directory

[9fans] everything is a directory

[jsnx]

I'm not sure if this is the right place to post this, but it seems
like a good fit. What better forum for deep thought on the meaning of
files and directories than the Plan 9 news group?

There would be great utility in merging files and directories into a
composite content/container object that respond 'read' and 'write' for
file ops and 'list', 'add', 'delete' for directory ops. For example, a
disk drive could respond to 'read' with a bunch of stuff on the disk,
and respond to 'list' with a listing of it's hardware settings, which
could be set with a 'write'. Merged file/directories also make a lot
of sense when you think about languages with hierarchical modules --
instead of having naming conventions to find a sub-module, you just
look it up and read it. Similarly, hierarchical documents map straight
on to the mixed file/folder -- you put the intro in the head and its
components under the head.

I'm sure this idea has come up in the past; many of my ideas are like
that. The 'everything is a file' model is proverbial, but it was not
so once upon a time. I'm sure the 'everything is a directory' model
had its proponents in days gone by, just as functional languages did
(and will again!). In fact, 'everything is a directory' is the man
behind the curtain in LDAP.

In the considered opinion of the list, is "everything is a directory"
a big mess, a resource wasting fantasy, an idea whose time has come?

Re: [9fans] everything is a directory

[maht]

We already have composite container / content objects, they are called 
directories

"list" gives you a list of their files

reading the files gives you the data available

new semantics like yours will need a new raft of tools, protocols and 
all the shite that goes with it



> is "everything is a directory"
> a big mess, a resource wasting fantasy, an idea whose time has come?
>
>   

just the first two

Re: [9fans] everything is a directory

[Charles Forsyth]

>I'm sure this idea has come up in the past; many of my ideas are like

i'm afraid that directory-with-contents does indeed reappear from time to time.
it's not obvious (at least to me) that it ever made a big difference,
but it does add ambiguity.  a version and configuration system made use of one implementation,
so that directory "xyz" might contain the portable source of something, and "xyz/special-variant"
might have a modified version or a machine-specific part of the general
"xyz" (eg, "console", "console/dgnova", "console/ti990").
there was a bit more to it than that, but in the end on other systems "xyz" was
a directory, "xyz/port" had the general version, and other substructure gave the variants.
so the directory-with-content didn't really seem to make things simpler.

at the moment in Plan 9, /dev/sdC0/data has the drive's data and /dev/sdC0/ctl
shows you its attributes and writes control them,
and ls /dev/sdC0 shows you ctl, data and others.
/dev/sdC0 refers to that set (as distinct from the data, which is in "data").
this seems clear and adequate.  i'd have said it was a simpler model.

Re: [9fans] everything is a directory

[Francisco J Ballesteros]

In the first edition of Plan B we had "boxes". They are generic containers that
behave either as files or as directories depending on what op. you
use. In general,
a box is a typed container that has inner boxes. After having then
working, the lack
of applications made us drop the idea and switch back to files. A
brief description is
at "The Box, a replacement for files" paper in lsub.org. Also, I think
I still have the source
of that plan b, with a prefix mount table included and a box library,
but would have to
dig in the worm to re-locate it.

On 8/16/07, jsnx <jason.dusek@gmail.com> wrote:
> I'm not sure if this is the right place to post this, but it seems
> like a good fit. What better forum for deep thought on the meaning of
> files and directories than the Plan 9 news group?
>
> There would be great utility in merging files and directories into a
> composite content/container object that respond 'read' and 'write' for
> file ops and 'list', 'add', 'delete' for directory ops. For example, a
> disk drive could respond to 'read' with a bunch of stuff on the disk,
> and respond to 'list' with a listing of it's hardware settings, which
> could be set with a 'write'. Merged file/directories also make a lot
> of sense when you think about languages with hierarchical modules --
> instead of having naming conventions to find a sub-module, you just
> look it up and read it. Similarly, hierarchical documents map straight
> on to the mixed file/folder -- you put the intro in the head and its
> components under the head.
>
> I'm sure this idea has come up in the past; many of my ideas are like
> that. The 'everything is a file' model is proverbial, but it was not
> so once upon a time. I'm sure the 'everything is a directory' model
> had its proponents in days gone by, just as functional languages did
> (and will again!). In fact, 'everything is a directory' is the man
> behind the curtain in LDAP.
>
> In the considered opinion of the list, is "everything is a directory"
> a big mess, a resource wasting fantasy, an idea whose time has come?
>

Re: [9fans] everything is a directory

[David Leimbach]

[-- Attachment #1: Type: text/plain, Size: 2951 bytes --]

Unix already has this, and their called extended attributes, and I hate them
with a burning passion.  Rsync, cp, any tool that manipulates files (tar for
example) has to be able to capture this data, and just reading the file
won't do it anymore.
Oh and to make things more "fun" FreeBSD, Linux and Mac OS X at least have
different ways of dealing with them, and max size limits etc.

Where are people picturing the great utility of them?  I've found it to be
the worst thing invented since symlinks (except maybe relative symlinks that
use environment variables to complete part of the path... Makes everything
way more complex than it needs to be)

Note that Mac OS X (and GNUStep?) already does treat certain directories as
a single unit, and the structure is what makes up a framework or a .app
application, or even .pkg files.  It is clear that there are other ways to
achieve this sort of abstraction without changing file/directory semantics
that we already have any more than people already have done.

Having personally written parts an archiver that captures these adorable and
non-standardized bits of information  somewhat well hidden in files, and
having used the less than ideal APIs to access them, well lets just say I'd
rather throw out my computer and play Nintendo Wii for the rest of my
life...

My 2 cents...  It's probably worth less than that though

On 8/16/07, jsnx <jason.dusek@gmail.com> wrote:
>
> I'm not sure if this is the right place to post this, but it seems
> like a good fit. What better forum for deep thought on the meaning of
> files and directories than the Plan 9 news group?
>
> There would be great utility in merging files and directories into a
> composite content/container object that respond 'read' and 'write' for
> file ops and 'list', 'add', 'delete' for directory ops. For example, a
> disk drive could respond to 'read' with a bunch of stuff on the disk,
> and respond to 'list' with a listing of it's hardware settings, which
> could be set with a 'write'. Merged file/directories also make a lot
> of sense when you think about languages with hierarchical modules --
> instead of having naming conventions to find a sub-module, you just
> look it up and read it. Similarly, hierarchical documents map straight
> on to the mixed file/folder -- you put the intro in the head and its
> components under the head.
>
> I'm sure this idea has come up in the past; many of my ideas are like
> that. The 'everything is a file' model is proverbial, but it was not
> so once upon a time. I'm sure the 'everything is a directory' model
> had its proponents in days gone by, just as functional languages did
> (and will again!). In fact, 'everything is a directory' is the man
> behind the curtain in LDAP.
>
> In the considered opinion of the list, is "everything is a directory"
> a big mess, a resource wasting fantasy, an idea whose time has come?
>

[-- Attachment #2: Type: text/html, Size: 3645 bytes --]

Re: [9fans] everything is a directory

[erik quanstrom]

that's a quality rant.

- erik

Re: [9fans] everything is a directory

[maht]

> Unix already has this, and their called extended attributes, and I 
> hate them with a burning passion.  Rsync, cp, any tool that 
> manipulates files (tar for example) has to be able to capture this 
> data, and just reading the file won't do it anymore.  
>
> Oh and to make things more "fun" FreeBSD, Linux and Mac OS X at least 
> have different ways of dealing with them, and max size limits etc.  

you forgot NTFS's version

http://www.wikistc.org/wiki/Alternate_data_streams

echo plain > t.xt
echo hidden > t.txt:hidden.txt

%type t.txt
plain
% type t.txt:hidden.txt
The filename, directory name, or volume label syntax is incorrect.

% notepad t.txt:hidden.txt
hidden

Re: [9fans] everything is a directory

[David Leimbach]

[-- Attachment #1: Type: text/plain, Size: 901 bytes --]

On 8/16/07, maht <mattmobile@proweb.co.uk> wrote:
>
>
> > Unix already has this, and their called extended attributes, and I
> > hate them with a burning passion.  Rsync, cp, any tool that
> > manipulates files (tar for example) has to be able to capture this
> > data, and just reading the file won't do it anymore.
> >
> > Oh and to make things more "fun" FreeBSD, Linux and Mac OS X at least
> > have different ways of dealing with them, and max size limits etc.
>
> you forgot NTFS's version
>
> http://www.wikistc.org/wiki/Alternate_data_streams
>
> echo plain > t.xt
> echo hidden > t.txt:hidden.txt
>
> %type t.txt
> plain
> % type t.txt:hidden.txt
> The filename, directory name, or volume label syntax is incorrect.
>
> % notepad t.txt:hidden.txt
> hidden


So there we have it... all the major OSes I know of have turned files into
directories already... *sigh*

[-- Attachment #2: Type: text/html, Size: 1343 bytes --]

Re: [9fans] everything is a directory

[ron minnich]

On 8/16/07, David Leimbach <leimy2k@gmail.com> wrote:

> So there we have it... all the major OSes I know of have turned files into
> directories already... *sigh*

yes, it's a measure of how bad a given os is ... a simple metric.

ron

Re: [9fans] everything is a directory

[Robert Sherwood]

[-- Attachment #1: Type: text/plain, Size: 566 bytes --]

I'm confused. Is it the concept that's awful, or the implimentation? Is the
problem that visibiliby of the attributes to tools is inconsistant and
they're managed with entirely different symantics? Or is the whole idea of
arbitrary file attributes a tool of the devil?

Certainly, some file formats (I'm thinking of MP3) use file attributes to
great effect, though of course they're specially implemented in the context
of the files' use case. Is it appropriate to implement a general file
tagging capability that is file independant and managed by the OS?

[-- Attachment #2: Type: text/html, Size: 627 bytes --]

Re: [9fans] everything is a directory

[David Leimbach]

[-- Attachment #1: Type: text/plain, Size: 2442 bytes --]

On 8/16/07, Robert Sherwood <robert.sherwood@gmail.com> wrote:
>
> I'm confused. Is it the concept that's awful, or the implimentation? Is
> the problem that visibiliby of the attributes to tools is inconsistant and
> they're managed with entirely different symantics? Or is the whole idea of
> arbitrary file attributes a tool of the devil?
>

Both?

There's no standardized interface across the board that's guaranteed to work
for all platforms that are supposed to be more or less compatible.  And the
fact that the implementation exists means my fundamental abstractions that
made Unix Unix to begin with are changing underneath the tools and
applications I have to work with them.

I think the root of the issue, for me, is that we had a definition for what
a file was in an OS and the world was for the most part simple.  People
stored things like a tar table of contents in the file and defined formats
on a per-file basis, and people realized that if they wanted to manipulate
such files, that all that was needed was to use some well defined structures
and APIs to manage them.  For things that were common to "files" we could
copy them, delete them, rename them with no expectations of side-effects or
missed metadata.

At a high level, copying a file is defined as copying the whole file - all
the data on the disk associated with that particular blob, because that's
what a file is, a representation of the data for humans to deal with, not
just a portion of the data.  When things like extended attributes were
added, that was no longer the case (and even earlier in things like Mac OS
Resource Forks).  Now I've lost my original abstraction that I loved so
much, because someone didn't figure out another way to store an icon for my
program that didn't break the semantics of keeping all the file's data in
one abstracted "place".

Sure people fixed tar, rsync and (hopefully) all the standard faire to deal
with these new and wonderful attributes, but couldn't we have found a better
way?  For some things, like access control lists, maybe not (I'm still
unconvinced that it had to be done with extended attributes).  But let's
just say it doesn't always feel that it's worth the pain involved to break
fundamental abstractions like "file" and "directory" to get these new
features.


One could argue if we must break those abstractions, that maybe it's time
for a new OS.

Dave

[-- Attachment #2: Type: text/html, Size: 3176 bytes --]

Re: [9fans] everything is a directory

[Uriel]

On 8/16/07, David Leimbach <leimy2k@gmail.com> wrote:
> Unix already has this, and their called extended attributes, and I hate them
> with a burning passion.  Rsync, cp, any tool that manipulates files (tar for
> example) has to be able to capture this data, and just reading the file
> won't do it anymore.
>
> Oh and to make things more "fun" FreeBSD, Linux and Mac OS X at least have
> different ways of dealing with them, and max size limits etc.
>
>  Where are people picturing the great utility of them?  I've found it to be
> the worst thing invented since symlinks (except maybe relative symlinks that
> use environment variables to complete part of the path... Makes everything
> way more complex than it needs to be)

Amen to all this, for once we agree completely. The sad thing is that
while they added all this wonderful 'features', they could not be
bothered to implement things as basic as proper union mounts and bind
operations, which Plan 9 has had for eons.

I'm waiting for some genius to come up with the idea of extended
attributes for extended attributes... The 'files as directories'
paradigm (which seems like the linux guys are moving towards) is a
really scary semantic minefield which makes symlink-induced nightmares
look like a walk around Disneyworld. 'Magic symlinks' are, as you
point out, even worse.

But lets not complain, think of how many jobs all this extra
complexity is helping create, I'm sure the good shape of the world
economy is in good part thanks to the great progress in our wonderful
software industry.

Best wishes

uriel

Re: [9fans] everything is a directory

[Skip Tavakkolian]

if the goal of the extra metadata is to somehow help in the selection
of the data (i.e.  file), then something like what abhey presented at
iwp9 will fit the problem.

http://www.cs.bell-labs.com/sources/contrib/iwp9-2006/papers/abheyshahplan9.pdf

> I'm confused. Is it the concept that's awful, or the implimentation? Is the
> problem that visibiliby of the attributes to tools is inconsistant and
> they're managed with entirely different symantics? Or is the whole idea of
> arbitrary file attributes a tool of the devil?
> 
> Certainly, some file formats (I'm thinking of MP3) use file attributes to
> great effect, though of course they're specially implemented in the context
> of the files' use case. Is it appropriate to implement a general file
> tagging capability that is file independant and managed by the OS?

[9fans] Re: everything is a directory

[jsnx]

On Aug 16, 4:48 am, fors...@terzarima.net (Charles Forsyth) wrote:
> >I'm sure this idea has come up in the past; many of my ideas are like
>
> i'm afraid that directory-with-contents does indeed reappear from time to time.
> it's not obvious (at least to me) that it ever made a big difference,
> but it does add ambiguity.

How does it add ambiguity? The example you give is really no different
from putting the portable code for a foo in foo/code.c and the arch
specific versions in foo/i386/code.c, except for it doesn't have the
baggage of a file naming convention.

[9fans] Re: everything is a directory

[jsnx]

On Aug 16, 10:46 am, leim...@gmail.com (David Leimbach) wrote:
> So there we have it... all the major OSes I know of have turned files into
> directories already... *sigh*

But what about making directories into files? We should get rid of one
or the other, is my point. Give all filesystem objects children (which
are other filesystem objects) and then as many 'channels' as they
like, for app and OS specific things (ACL channel, owner channel, byte
channel, diff channel...).

[9fans] Re: everything is a directory

[jsnx]

On Aug 16, 5:40 am, n...@lsub.org (Francisco J Ballesteros) wrote:
> In the first edition of Plan B we had "boxes".

After reading through the paper, I realize that my directories-as-
files idea gets a little strange when you have union mounts. Just
joining putting the bytes in row one after the other is liable to
result in an invalid file for many file types.

Re: [9fans] everything is a directory

[Douglas A. Gwyn]

What do you mean by "extended attributes"?
I haven't noticed them on the Unix systems I use.

Re: [9fans] everything is a directory

[matt]

You can't argue against the principle, we take files as directories all 
the time and one level or another.
So, for me, it's botched implementation. Just wait for extended 
attribute directories to be added on with something like 
file.txt:meta.zip (but not zip files, something new [that's like zip but 
slightly different - maybe even encrypted])
They already have synthetic file systems built into NT!

Re: [9fans] everything is a directory

[erik quanstrom]

> You can't argue against the principle, we take files as directories all 
> the time and one level or another.
> So, for me, it's botched implementation. Just wait for extended 
> attribute directories to be added on with something like 
> file.txt:meta.zip (but not zip files, something new [that's like zip but 
> slightly different - maybe even encrypted])
> They already have synthetic file systems built into NT!

i think it's worse thank that.  attributes and their ilk essentially
add methods to a filesystem.

if you want an object system, you probablly don't want plan 9,
windows or linux.

- erik

Re: [9fans] everything is a directory

[ron minnich]

On 8/17/07, Douglas A. Gwyn <DAGwyn@null.net> wrote:
> What do you mean by "extended attributes"?
> I haven't noticed them on the Unix systems I use.
>
maybe I'm missing the question, but on my linux:

man -k extended | grep attrib | wc
    21     183    1417
libattr             (rpm) - Dynamic library for extended attribute support
fgetxattr [getxattr] (2)  - retrieve an extended attribute value
flistxattr [listxattr] (2)  - list extended attribute names
(etc. It's a lot like read but, as usual, it needs  a new set of
system calls ... 5 of them.
And ls can't show them,  but ... we have getfattr/setfattr
       The output format of getfattr -d is as follows:
               1:  # file: somedir/
               2:  user.name0="value0"
               3:  user.name1="value1"
               4:  user.name2="value2"
               5:  ...


it's everywhere. But it was too hard to put in in some normal way, so
it went in from the side.

ron

Re: [9fans] everything is a directory

[erik quanstrom]

hey, how do you grep for stuff in extended attributes?

- erik


> On 8/17/07, Douglas A. Gwyn <DAGwyn@null.net> wrote:
>> What do you mean by "extended attributes"?
>> I haven't noticed them on the Unix systems I use.
>>
> maybe I'm missing the question, but on my linux:
> 
> man -k extended | grep attrib | wc
>     21     183    1417
> libattr             (rpm) - Dynamic library for extended attribute support
> fgetxattr [getxattr] (2)  - retrieve an extended attribute value
> flistxattr [listxattr] (2)  - list extended attribute names
> (etc. It's a lot like read but, as usual, it needs  a new set of
> system calls ... 5 of them.
> And ls can't show them,  but ... we have getfattr/setfattr
>        The output format of getfattr -d is as follows:
>                1:  # file: somedir/
>                2:  user.name0="value0"
>                3:  user.name1="value1"
>                4:  user.name2="value2"
>                5:  ...
> 
> 
> it's everywhere. But it was too hard to put in in some normal way, so
> it went in from the side.
> 
> ron

Re: [9fans] Re: everything is a directory

[erik quanstrom]

> But what about making directories into files? We should get rid of one
> or the other, is my point. Give all filesystem objects children (which
> are other filesystem objects) and then as many 'channels' as they
> like, for app and OS specific things (ACL channel, owner channel, byte
> channel, diff channel...).

of course directories are files.  the whole file system is a file.  filesystems
role in life is to turn a linear array of blocks into something with heirarchy.
often that linear array of blocks is known as a disk drive.

(mbox format is a directory masquerading as a file and so is /adm/keys.)

- erik

Re: [9fans] everything is a directory

[plan9]

On Fri, Aug 17, 2007 at 09:24:26AM -0400, erik quanstrom wrote:
> ron wrote:
>> On 8/17/07, Douglas A. Gwyn <DAGwyn@null.net> wrote:
>>> What do you mean by "extended attributes"?
>>> I haven't noticed them on the Unix systems I use.
>>>
>> maybe I'm missing the question, but on my linux:
>> 
>> man -k extended | grep attrib | wc
>>     21     183    1417
>> libattr             (rpm) - Dynamic library for extended attribute support
>> fgetxattr [getxattr] (2)  - retrieve an extended attribute value
>> flistxattr [listxattr] (2)  - list extended attribute names
>> (etc. It's a lot like read but, as usual, it needs  a new set of
>> system calls ... 5 of them.
>> And ls can't show them,  but ... we have getfattr/setfattr
>>        The output format of getfattr -d is as follows:
>>                1:  # file: somedir/
>>                2:  user.name0="value0"
>>                3:  user.name1="value1"
>>                4:  user.name2="value2"
>>                5:  ...
>> 
>> 
>> it's everywhere. But it was too hard to put in in some normal way, so
>> it went in from the side.
>> 
>> ron
> 
> hey, how do you grep for stuff in extended attributes?
> 
> - erik

grepfattr, I should think.  (Now with ANSI color support!)

Re: [9fans] everything is a directory

[ron minnich]

I just read this great article: http://www.linux.com/articles/114027

This quote says it all: "By the time you finish reading this article,
you will be able to take advantage of the coolest file system feature
to be implemented since the symbolic link.  - CLI Magic: Use Extended
Attributes for better file management By Ryan Paul"

So, I guess he's right. We need to get this in to plan 9. The system
calls should not be too hard to add, and then we can bring the
programs over. Anyone want to do it? We could even implement the
symlinks as a kind of extended attribute.

ron

Re: [9fans] everything is a directory

[David Leimbach]

[-- Attachment #1: Type: text/plain, Size: 300 bytes --]

/usr/include/sys/xattr.h (on linux anyway)
man setxattr

On FreeBSD it's /usr/include/sys/extattr.h

What Unix system are you using?

On 8/17/07, Douglas A. Gwyn <DAGwyn@null.net> wrote:
>
> What do you mean by "extended attributes"?
> I haven't noticed them on the Unix systems I use.
>

[-- Attachment #2: Type: text/html, Size: 802 bytes --]

Re: [9fans] everything is a directory

[Richard Miller]

> I wonder what happened to my request for a UTF encoding for sarcasm...

0x263A

Re: [9fans] everything is a directory

[David Leimbach]

[-- Attachment #1: Type: text/plain, Size: 749 bytes --]

I wonder what happened to my request for a UTF encoding for sarcasm...

On 8/17/07, ron minnich <rminnich@gmail.com> wrote:
>
> I just read this great article: http://www.linux.com/articles/114027
>
> This quote says it all: "By the time you finish reading this article,
> you will be able to take advantage of the coolest file system feature
> to be implemented since the symbolic link.  - CLI Magic: Use Extended
> Attributes for better file management By Ryan Paul"
>
> So, I guess he's right. We need to get this in to plan 9. The system
> calls should not be too hard to add, and then we can bring the
> programs over. Anyone want to do it? We could even implement the
> symlinks as a kind of extended attribute.
>
>
> ron
>

[-- Attachment #2: Type: text/html, Size: 1077 bytes --]

Re: [9fans] everything is a directory

[ron minnich]

On 8/17/07, David Leimbach <leimy2k@gmail.com> wrote:
> I wonder what happened to my request for a UTF encoding for sarcasm...

OK, you win the prize! I am a lousy troll ...

ron

Re: [9fans] everything is a directory

[David Leimbach]

[-- Attachment #1: Type: text/plain, Size: 1806 bytes --]

On 8/17/07, ron minnich <rminnich@gmail.com> wrote:
>
> On 8/17/07, David Leimbach <leimy2k@gmail.com> wrote:
> > I wonder what happened to my request for a UTF encoding for sarcasm...
>
> OK, you win the prize! I am a lousy troll ...
>
> ron
>

Nah I just thought you were joking about the article because the author
starts by singing the praises of the symbolic link.  :-)

My objections to their well established existence are based on feeling
uneasy about the answers to the following questions:

What happens to file metadata that you
1. Put on NFS?
2. Access with 9p?
3. Copy from one filesystem to another?
4. Copy from one OS to another?
5. Archive one one and move to another OS then extract? (smart archiver?)

Somehow I don't feel like I need this feature to do anything and that any
data I put in there is destined to be lost on backup and restore which is
part of why I worked on xar to do extended attributes for Mac OS X... I
really wanted to be able to backup Microsoft Office and have it restore and
actually work, "tar" couldn't do this, "pax" couldn't do this (well at the
time anyway they might now....).

See for yourself, there's at least 3 implementations of extended attribute
"backends" in xar.

(fbsdattr, darwinattr, linuxattr)
http://xar.googlecode.com/svn/trunk/xar/lib/


Like em or not, they're here... and we have to deal.  I just don't want to
see Plan 9 adopt them because it seems possible to do everything they enable
us to do by fixing file formats.  Clearly I can design one format that
archives them from 3 different OSes (and yeah, we used XML for that cuz
everything is better with XML!!!).

At any rate, it certainly increases job security for those who know about
them and how to use em! :-)

Dave






Dave

[-- Attachment #2: Type: text/html, Size: 3890 bytes --]

Re: [9fans] everything is a directory

[Charles Forsyth]

> At any rate, it certainly increases job security for those who know about
> them and how to use em! :-)

just blame the stock market crash, errr... `correction'
on extended attributes, in a blog somewhere
and soon there will be hearings about them.

[9fans] Re: everything is a directory

[jsnx]

On Aug 17, 5:06 am, quans...@quanstro.net (erik quanstrom) wrote:
> > They already have synthetic file systems built into NT!
>
> i think it's worse thank that.  attributes and their ilk essentially
> add methods to a filesystem.

That's overstatement -- attributes add 'user defined' types to the
filesystem, but that's not the same thing as giving each filesystem
object procedures. It might require polymorphic versions of ls, cat,
&c. -- but probably not, since the extra fields aren't of interest to
them.

I've seen a lot of criticism of extended attributes on this thread,
but no one has stepped up with a solution that addresses the problem
they solve. Application specific data should go in the file -- we all
agree about that. The file's position in the heirarchy is modeled by
directories, which also carry OS specific metadata -- permissions,
ownership. But where do the oddball intermediaries put their metadata?
For example, where should the desktop environment put file specific
icons or annotations? It can't very well stuff icons into your
doom .wad files. In principle, the desktop could put that stuff in a
mockup of the real file-system hierarchy, but having a /gnome/icons
tree that had to be kept in synchrony with the 'real' filesystem
invites a lot of semantic confusion, as well as an implementation
hassle (broken links, new versions of 'mv', something else?).

The fathers of Unix saw many things, but who's to say they saw all the
metadata we will ever need?

Re: [9fans] everything is a directory

[Douglas A. Gwyn]

ron minnich wrote:
> On 8/17/07, Douglas A. Gwyn <DAGwyn@null.net> wrote:
> > What do you mean by "extended attributes"?
> > I haven't noticed them on the Unix systems I use.
> maybe I'm missing the question, but on my linux:
> man -k extended | grep attrib | wc
>     21     183    1417

On Solaris 8 there is only one line, for user_attr(4),
which pertains to users, not to files.
However, on Solaris 10 it turns up fsattr(5),
which seems to be the feature under discussion.
The Solaris 10 manual page for fsattr(5) describes
nicely how to access the attribute information, but
it fails to explain why one would want to use that
feature in the first place.

> it's everywhere. But it was too hard to put in in some normal way, so
> it went in from the side.

At least on Solaris 10, there is an openat() syscall
that opens the attributes as a normal fd starting at
the (possibly ordinary) fd supplied as an argument,
which acts "sort of" like a current-working-directory.

I'm still wondering what the motivation was.

Re: [9fans] everything is a directory

[Douglas A. Gwyn]

Uriel wrote:
> I'm waiting for some genius to come up with the idea of extended
> attributes for extended attributes...

According to the Solaris 10 fsattr(5) that's already allowed,
although some filesystems (inclding UFS) don't support it.

Re: [9fans] everything is a directory

[Douglas A. Gwyn]

"erik quanstrom" <quanstro@quanstro.net> wrote in message 
news:198ff93d3223123a0e103cbcc4755bc4@quanstro.net...
> hey, how do you grep for stuff in extended attributes?

grep -@, assuming that grep has been extended like other utilities (tar, 
etc.)

One would have thought that a better design would have been
    /path/file/.attributes/...
Who makes these decisions, anyway??

Re: [9fans] Re: everything is a directory

[Francisco J Ballesteros]

Do you prefer embedding the "face" in each mail as metadata instead of
keeping a faces db?

I suffer the same thing with mp3 art, embedded in the mp3s. Lots of dup. data
plus some players that cannot cope with such decorated mp3 files,
including the one
in my car :(

On 8/20/07, jsnx <jason.dusek@gmail.com> wrote:
> On Aug 17, 5:06 am, quans...@quanstro.net (erik quanstrom) wrote:
> > > They already have synthetic file systems built into NT!
> >
> > i think it's worse thank that.  attributes and their ilk essentially
> > add methods to a filesystem.
>
> That's overstatement -- attributes add 'user defined' types to the
> filesystem, but that's not the same thing as giving each filesystem
> object procedures. It might require polymorphic versions of ls, cat,
> &c. -- but probably not, since the extra fields aren't of interest to
> them.
>
> I've seen a lot of criticism of extended attributes on this thread,
> but no one has stepped up with a solution that addresses the problem
> they solve. Application specific data should go in the file -- we all
> agree about that. The file's position in the heirarchy is modeled by
> directories, which also carry OS specific metadata -- permissions,
> ownership. But where do the oddball intermediaries put their metadata?
> For example, where should the desktop environment put file specific
> icons or annotations? It can't very well stuff icons into your
> doom .wad files. In principle, the desktop could put that stuff in a
> mockup of the real file-system hierarchy, but having a /gnome/icons
> tree that had to be kept in synchrony with the 'real' filesystem
> invites a lot of semantic confusion, as well as an implementation
> hassle (broken links, new versions of 'mv', something else?).
>
> The fathers of Unix saw many things, but who's to say they saw all the
> metadata we will ever need?
>

Re: [9fans] Re: everything is a directory

[erik quanstrom]

> The fathers of Unix saw many things, but who's to say they saw all the
> metadata we will ever need?

who's to say they didn't.  (i hope we're talking about plan 9 here, not
unix.  in plan 9 many cumbersome traditions were broken.  if
they had wanted attributes, they would have added them.)

i don't see a specific problem in plan 9 that extra metadata solves.  
nemo makes a great point.  the only place that plan 9 has icons, they
are kept in /lib/face as files.

- erik

Re: [9fans] Re: everything is a directory

[Tom Lieber]

On 8/20/07, Francisco J Ballesteros <nemo@lsub.org> wrote:
> Do you prefer embedding the "face" in each mail as metadata instead of
> keeping a faces db?

Why would he? He argued for keeping icons in sync with files -- that's
because files change and move. Once you link an address with a face,
there are few reasons why that linkage would ever change.

> I suffer the same thing with mp3 art, embedded in the mp3s. Lots of dup. data
> plus some players that cannot cope with such decorated mp3 files,
> including the one
> in my car :(

If the problem is stuffing data into a file that was never meant to be
in the file, attributes are a solution. After all, you quote from jsnx
did say, "But where do the oddball intermediaries put their metadata?
... [They] can't very well stuff album art into your .mp3 files" (or
something like that...).

-- 
Tom Lieber
http://AllTom.com/

Re: [9fans] Re: everything is a directory

[erik quanstrom]

> On 8/20/07, Francisco J Ballesteros <nemo@lsub.org> wrote:
> > Do you prefer embedding the "face" in each mail as metadata instead of
> > keeping a faces db?
> 
> Why would he? He argued for keeping icons in sync with files -- that's
> because files change and move. Once you link an address with a face,
> there are few reasons why that linkage would ever change.

plan 9 doesn't have icons.

- erik

Re: [9fans] Re: everything is a directory

[Uriel]

>From #plan9 some years ago:

<mhat> the only places for icons is in a church, a burning church at that

On 8/20/07, erik quanstrom <quanstro@quanstro.net> wrote:
> > On 8/20/07, Francisco J Ballesteros <nemo@lsub.org> wrote:
> > > Do you prefer embedding the "face" in each mail as metadata instead of
> > > keeping a faces db?
> >
> > Why would he? He argued for keeping icons in sync with files -- that's
> > because files change and move. Once you link an address with a face,
> > there are few reasons why that linkage would ever change.
>
> plan 9 doesn't have icons.
>
> - erik
>

Re: [9fans] everything is a directory

[Douglas A. Gwyn]

I was thinking more about the subject, and extended attributes.
Here is an alternative proposal to implement extended attributes:

(1) Everything stays the same as it is (Plan 9 or old-Unix)
except:
(2) An attempt to read(2) or write(2) a directory will succeed
if and only if there is a .data entry, in which case it will
access contents of the .data object.

None of the existing tools need to be changed.

To add attributes (all done in user space):
(a) Move the current file (terminal node) to a temp name.
(b) Make a directory in its place, with the original name.
(c) Move the temp name to original_name/.data .
(d) Create a file (subdirectory?) .attributes .
(e) Populate the .attributes .

Re: [9fans] everything is a directory

[Francisco J Ballesteros]

But why don't just keep a foo.attr side by side with the file then?
Should you need that, I mean.

On 8/20/07, Douglas A. Gwyn <DAGwyn@null.net> wrote:
> I was thinking more about the subject, and extended attributes.
> Here is an alternative proposal to implement extended attributes:
>
> (1) Everything stays the same as it is (Plan 9 or old-Unix)
> except:
> (2) An attempt to read(2) or write(2) a directory will succeed
> if and only if there is a .data entry, in which case it will
> access contents of the .data object.
>
> None of the existing tools need to be changed.
>
> To add attributes (all done in user space):
> (a) Move the current file (terminal node) to a temp name.
> (b) Make a directory in its place, with the original name.
> (c) Move the temp name to original_name/.data .
> (d) Create a file (subdirectory?) .attributes .
> (e) Populate the .attributes .
>

Re: [9fans] everything is a directory

[Rob Pike]

This is only possible because someone said it was OK to
break directories in the first place.  Not too long ago, if you
wanted to read a directory you could use a read system call.
I see no functional reason that that symmetry was worth
breaking.

Also, I dispute your claim that tools have to change.
For instance, ls will misbehave: it will think your file
is a directory.  And there will be lots more instances
because, fundamentally, stat will be misleading.
Then all the existing tools will be changed because
people will want to use them on attributed files and
their attributes.

It reminds me of symlinks, which I detest. I understand
why they were created - cross-device links didn't work -
but they broke just about everything by introducing a
second kind of file. Half the time you want the file,
half the time you want the file it describes. They're
pointers, and deciding whether the operation wants the
pointer's value or referent means a new system call
shows up, a new flag appears everywhere, and one
usually guesses wrong the first time.

Attributes, as created or as posited here, are better
only because for the most part one can ignore them
altogether. Except when you can't.

If you want attributes, rethink the system, don't hack
it.

-rob



On 8/20/07, Douglas A. Gwyn <DAGwyn@null.net> wrote:
> I was thinking more about the subject, and extended attributes.
> Here is an alternative proposal to implement extended attributes:
>
> (1) Everything stays the same as it is (Plan 9 or old-Unix)
> except:
> (2) An attempt to read(2) or write(2) a directory will succeed
> if and only if there is a .data entry, in which case it will
> access contents of the .data object.
>
> None of the existing tools need to be changed.
>
> To add attributes (all done in user space):
> (a) Move the current file (terminal node) to a temp name.
> (b) Make a directory in its place, with the original name.
> (c) Move the temp name to original_name/.data .
> (d) Create a file (subdirectory?) .attributes .
> (e) Populate the .attributes .
>

Re: [9fans] Re: everything is a directory

[Paweł Lasek]

On 8/20/07, Tom Lieber <alltom@gmail.com> wrote:
> On 8/20/07, Francisco J Ballesteros <nemo@lsub.org> wrote:

> If the problem is stuffing data into a file that was never meant to be
> in the file, attributes are a solution. After all, you quote from jsnx
> did say, "But where do the oddball intermediaries put their metadata?
> ... [They] can't very well stuff album art into your .mp3 files" (or
> something like that...).

While I don't give a damn about files being directories, I pretty much
like Extended Attributes or NTFS files (in NTFS, file is something
different than in *nix-like FS or FAT).

While I don't like stuffing "persistent" data into EA's (like album
art data, icons or similar things), they are fairly useful from system
daemon perspective. Since you want examples, I'll give those that I
know:

* in XFS, several tools use EA's to assign attributes like "DO NOT
DUMP" or hierarchical file management data. This data *should* be
discarded by normal tools (since cp is creating a new file, and mv
only modifes directories but not i-node's EAs) and are of interest
only to those system daemons

* (IIRC) Beagle daemon uses extended attributes to track files that
were scanned and are not modified - they contain part of it's detected
metadata, hash and DB pointer

* NTFS discards the Unix concept of a file entirely - what was shown
before as trouble with Alternate Data Streams is in fact trouble of
keeping backward compatibility (Internally, NT doesn't have drive
letters or any of the DOS stuff). A File in NTFS is a MFT entry with a
bunch of attributes. Those attributes have several types/names
defined, like DATA for file contents, one for keeping filename (for
which there are three namespaces: DOS8.3, Windows, POSIX), one for
legacy DOS attributes (used by the official, backward compatible API),
one for security data, and others used mostly by system daemons
(Although NT5+ explorer uses one to hold various comments about files.
Rarely used by people I think).

I especially like the NTFS implementation, because it allows to use
NTFS for most Operating Systems without modifying internal structure,
while making it easy to add "compatibility layer"

As for things that need to work with cp/mv/others, I suggest making
better file formats or adding metadata in different file. It's not the
smartest use of Extended Attributes/NTFS attributes :-)

> --
> Tom Lieber
> http://AllTom.com/
>


-- 
Paul Lasek

Re: [9fans] Re: everything is a directory

[Roman Shaposhnick]

On Mon, 2007-08-20 at 08:47 +0000, jsnx wrote:
> On Aug 17, 5:06 am, quans...@quanstro.net (erik quanstrom) wrote:
> > > They already have synthetic file systems built into NT!
> >
> > i think it's worse thank that.  attributes and their ilk essentially
> > add methods to a filesystem.
> 
> That's overstatement -- attributes add 'user defined' types to the
> filesystem, but that's not the same thing as giving each filesystem
> object procedures. It might require polymorphic versions of ls, cat,
> &c. -- but probably not, since the extra fields aren't of interest to
> them.
> 
> I've seen a lot of criticism of extended attributes on this thread,
> but no one has stepped up with a solution that addresses the problem
> they solve. Application specific data should go in the file -- we all
> agree about that. The file's position in the heirarchy is modeled by
> directories, which also carry OS specific metadata -- permissions,
> ownership. But where do the oddball intermediaries put their metadata?

  If that's the only need that can justify extended file attributes my
personal reaction would be to quote Al Viro (who, I believe, was
paraphrasing Ken, but I'm not certain): "Out-of-band == should be on a
separate channel..." 

  To me extended attributes are no better than ioctls and I hope we
all share a certain deep and profound feeling towards them.

  One last observation. I would argue that the very same reason that
makes symlinks semi-interisting on UNIX-like systems makes extended 
attributes pop-up in the same context as well: it is difficult to
manipulate your personal file namespace there and create these
"separate channels" on demand.

Thanks,
Roman.

Re: [9fans] Re: everything is a directory

[Uriel]

>   If that's the only need that can justify extended file attributes my
> personal reaction would be to quote Al Viro (who, I believe, was
> paraphrasing Ken, but I'm not certain): "Out-of-band == should be on a
> separate channel..."

I would love to find the original quote from ken (the one from Al is
quite good though). I personally like to put it this way:

"Metadata is just data with a stupid interface."

>   To me extended attributes are no better than ioctls and I hope we
> all share a certain deep and profound feeling towards them.

Very true.

>   One last observation. I would argue that the very same reason that
> makes symlinks semi-interisting on UNIX-like systems makes extended
> attributes pop-up in the same context as well: it is difficult to
> manipulate your personal file namespace there and create these
> "separate channels" on demand.

Sad thing is that when you try to convince Unix people of the power of
private namespaces, they complain that they seem 'too confusing'...
yea, because symlinks, extended attributes and other such hideous
hacks are so much nicer... and certainly create many more jobs, so
they keep adding layers of crud but wont implement the solution that
has been around for ages... has anyone learned anything from plan9?
I'm starting to doubt it.

Why is it that every new software feature people come up with makes me
think of the "I did it for you all" interview?

uriel

Re: [9fans] Re: everything is a directory

[erik quanstrom]

> * (IIRC) Beagle daemon uses extended attributes to track files that
> were scanned and are not modified - they contain part of it's detected
> metadata, hash and DB pointer

so, to summarize.  beagle is using attributes to extend the file object.
i think the main benefit files (and a filesystem) is exactly that they
are not objects and can't be extended.

> 
> * NTFS discards the Unix concept of a file entirely - what was shown
> before as trouble with Alternate Data Streams is in fact trouble of
> keeping backward compatibility (Internally, NT doesn't have drive
> letters or any of the DOS stuff). A File in NTFS is a MFT entry with a
> bunch of attributes. Those attributes have several types/names
> defined, like DATA for file contents, one for keeping filename (for
> which there are three namespaces: DOS8.3, Windows, POSIX), one for
> legacy DOS attributes (used by the official, backward compatible API),
> one for security data, and others used mostly by system daemons
> (Although NT5+ explorer uses one to hold various comments about files.
> Rarely used by people I think).
> 
> I especially like the NTFS implementation, because it allows to use
> NTFS for most Operating Systems without modifying internal structure,
> while making it easy to add "compatibility layer"

so how does one create a sensible grep utility that operates on an arbitrary
mft entry?

> As for things that need to work with cp/mv/others, I suggest making
> better file formats or adding metadata in different file. It's not the
> smartest use of Extended Attributes/NTFS attributes :-)

i think you're missing the real difficulty with attributres.  the tools approach
works precicely because files are not objects and can't be extended.

if you want a system with extensible object storage and not a filesystem,
i think you need to think about how to add the extension to all tools at
the same time.  (or maybe you dispense with the tools approach.)  this
doesn't seem like a promising route to me.

- erik

Re: [9fans] Re: everything is a directory

[Paweł Lasek]

On 8/20/07, erik quanstrom <quanstro@coraid.com> wrote:
> > * (IIRC) Beagle daemon uses extended attributes to track files that
> > were scanned and are not modified - they contain part of it's detected
> > metadata, hash and DB pointer
>
> so, to summarize.  beagle is using attributes to extend the file object.
> i think the main benefit files (and a filesystem) is exactly that they
> are not objects and can't be extended.

The beagle example is the weakest, since I don't use it ( it's written
in .NET, uses a bunch of weird tools and AFAIK doesn't play well with
external storage -- shortly, I'd do it differently :D).

> >
> > * NTFS discards the Unix concept of a file entirely - what was shown
> > before as trouble with Alternate Data Streams is in fact trouble of
> > keeping backward compatibility (Internally, NT doesn't have drive
> > letters or any of the DOS stuff). A File in NTFS is a MFT entry with a
> > bunch of attributes. Those attributes have several types/names
> > defined, like DATA for file contents, one for keeping filename (for
> > which there are three namespaces: DOS8.3, Windows, POSIX), one for
> > legacy DOS attributes (used by the official, backward compatible API),
> > one for security data, and others used mostly by system daemons
> > (Although NT5+ explorer uses one to hold various comments about files.
> > Rarely used by people I think).
> >
> > I especially like the NTFS implementation, because it allows to use
> > NTFS for most Operating Systems without modifying internal structure,
> > while making it easy to add "compatibility layer"
>
> so how does one create a sensible grep utility that operates on an arbitrary
> mft entry?

I'm not WinAPI expert, so I can' tell you implementation details on
Windows, but if you are writing your own API, all you need is to open
attribute/iterate over attributes instead of fopen/fread/... As I
wrote, they weren't exactly made "for the users" and as long as you
don't put vital user data inside, it should not pose problems. The
only thing I'd add is that I think NTFS driver should allow
'open'/'read'/'write'/etc on directory with directory structure as
it's data field (directories have different type in NTFS, so they
don't have the canonical DATA field)

If attributes are used properly on NTFS, your grep question changes to
"How does one create a sensible grep for file access rights?" in
plan9/unix context. AFAIK everyone uses 'stat' for such things instead
of grepping FS structures?

The biggest problem is exactly the same as the one with Unix signals -
people make wrong use of the tools - most of the time, NTFS attributes
are only accessed through API calls for the data that is stored inside
or by FS driver itself, not directly by apps. On the other hand, it
means that I can implement a Plan 9 NTFS variant without remaking FS
structures and if I put some dummy data in MFT entries, It will be
readable (even writable) by NT, linux, etc.

> > As for things that need to work with cp/mv/others, I suggest making
> > better file formats or adding metadata in different file. It's not the
> > smartest use of Extended Attributes/NTFS attributes :-)
>
> i think you're missing the real difficulty with attributres.  the tools approach
> works precicely because files are not objects and can't be extended.
>
> if you want a system with extensible object storage and not a filesystem,
> i think you need to think about how to add the extension to all tools at
> the same time.  (or maybe you dispense with the tools approach.)  this
> doesn't seem like a promising route to me.

I said that what is put into attributes should be easily discarded
away, so while tools might support copying them, it's up to you to
specify that they should. Like the xfstools example before - none of
them use eas the way that you should need to modify basic tools (at
least i don't see any reason to use them that way).

> - erik
>

I'd sum it this way: I'm all for *wisely* used external attributes -
the trouble is the *wisely* part :-)

-- 
Paul Lasek

Re: [9fans] Re: everything is a directory

[Dave Eckhardt]

> In XFS, several tools use EA's to assign attributes like "DO NOT
> DUMP" or hierarchical file management data. This data *should* be
> discarded by normal tools (since cp is creating a new file, and mv
> only modifes directories but not i-node's EAs) and are of interest
> only to those system daemons

Some attributes (icon) "*should*" be copied.

Other attributes (don't back me up) "*should* be discarded".

New attributes will doubtless be added every week.

How's a tool to know the right way to treat each attribute?

Dave Eckhardt

Re: [9fans] Re: everything is a directory

[Robert William Fuller]

Dave Eckhardt wrote:
>> In XFS, several tools use EA's to assign attributes like "DO NOT
>> DUMP" or hierarchical file management data. This data *should* be
>> discarded by normal tools (since cp is creating a new file, and mv
>> only modifes directories but not i-node's EAs) and are of interest
>> only to those system daemons
> 
> Some attributes (icon) "*should*" be copied.
> 
> Other attributes (don't back me up) "*should* be discarded".
> 
> New attributes will doubtless be added every week.
> 
> How's a tool to know the right way to treat each attribute?

The answer to that seems obvious.  A single bit could determine whether
or not an attribute is transient or more permanent.

Having worked on various data replication and backup products, I have
observed real problems copying data from one type of file system to
another because not every file system supports the same attributes or
the same characters in filenames.  Extended attributes seem like a
convenient place to stuff such data, but the fact that not all file
systems support extended attributes poses another problem, unless you
chose to ignore all file systems that do not support extended attributes.

Having said that, I have to say that I'm not a fan of extended
attributes, for a couple of reasons.  First, they lack visibility.  I
suppose that to some extent, this is merely a problem of interfaces,
both user interfaces and programming interfaces.  Second, they require
tools to handle two types of attributes, those "built-in" to the file
system, as well as those stored in extended attributes.  Again, this is
probably merely a matter of interface.  There is no reason the same
interface could not be used to enumerate both built-in and extended
attributes.  Perhaps my objections to extended attributes are all
addressable?  Still, I have to wonder if there isn't a better solution.

Finally, to argue that files are not objects seems silly.  They ARE
objects.  They have properties.  They have well defined interfaces for
manipulating those properties.  A more reasonable argument may be that
they are not object oriented since they lack certain prerequisites such
as inheritance and abstraction, both mechanisms of extensibility.

What a way to introduce myself to the list.  I'm sure I'm making friends
fast :-p

Re: [9fans] Re: everything is a directory

[erik quanstrom]

> Finally, to argue that files are not objects seems silly.  They ARE
> objects.  They have properties.  They have well defined interfaces for
> manipulating those properties.  A more reasonable argument may be that
> they are not object oriented since they lack certain prerequisites such
> as inheritance and abstraction, both mechanisms of extensibility.

so files are non-object-oriented objects?

i bet you can't say that without smiling.

- erik

Re: [9fans] Re: everything is a directory

[Uriel]

> Finally, to argue that files are not objects seems silly.  They ARE
> objects.  They have properties.  They have well defined interfaces for
> manipulating those properties.  A more reasonable argument may be that
> they are not object oriented since they lack certain prerequisites such
> as inheritance and abstraction, both mechanisms of extensibility.

In Plan 9 (and originally in Unix) files are the main abstraction, so
I'm not sure what you mean when you say they lack it; but they do lack
inheritance, and that is a good thing, I have never seen any good come
from inheritance.

> What a way to introduce myself to the list.  I'm sure I'm making friends
> fast :-p

Just be glad that boyd is not around anymore ;)

uriel

Re: [9fans] Re: everything is a directory

[Robert William Fuller]

erik quanstrom wrote:
>> Finally, to argue that files are not objects seems silly.  They ARE
>> objects.  They have properties.  They have well defined interfaces for
>> manipulating those properties.  A more reasonable argument may be that
>> they are not object oriented since they lack certain prerequisites such
>> as inheritance and abstraction, both mechanisms of extensibility.
> 
> so files are non-object-oriented objects?
> 
> i bet you can't say that without smiling.

Troll.  And I am smiling :-p.

Objects are defined as data structures with associated methods for
manipulating them.

Object oriented programming requires four key attributes:
encapsulation, inheritance, abstraction, and polymorphism.  Object
oriented programming is a more sophisticated programming discipline.

Objects do not imply object oriented programming, although object
oriented programming implies objects.

You're merely confounding the issue by playing a naive semantical game
based on the commonality of the word "object."

Re: [9fans] Re: everything is a directory

[Robert William Fuller]

Uriel wrote:
>> Finally, to argue that files are not objects seems silly.  They ARE
>> objects.  They have properties.  They have well defined interfaces for
>> manipulating those properties.  A more reasonable argument may be that
>> they are not object oriented since they lack certain prerequisites such
>> as inheritance and abstraction, both mechanisms of extensibility.
> 
> In Plan 9 (and originally in Unix) files are the main abstraction, so
> I'm not sure what you mean when you say they lack it; but they do lack
> inheritance, and that is a good thing, I have never seen any good come
> from inheritance.

This is true.  Files are an abstraction.  However, we are talking about
two different levels of abstraction.  You are talking about the level of
abstraction at the operating system interface.  I'm talking about the
level of abstraction available within the file construct.

>> What a way to introduce myself to the list.  I'm sure I'm making friends
>> fast :-p
> 
> Just be glad that boyd is not around anymore ;)

Should I search the archives for boyd for my entertainment :-)

Re: [9fans] Re: everything is a directory

[David Leimbach]

[-- Attachment #1: Type: text/plain, Size: 955 bytes --]

On 8/20/07, erik quanstrom <quanstro@coraid.com> wrote:
>
> > Finally, to argue that files are not objects seems silly.  They ARE
> > objects.  They have properties.  They have well defined interfaces for
> > manipulating those properties.  A more reasonable argument may be that
> > they are not object oriented since they lack certain prerequisites such
> > as inheritance and abstraction, both mechanisms of extensibility.
>
> so files are non-object-oriented objects?
>
> i bet you can't say that without smiling.
>
> - erik
>

I think the better way to think of a file is an interface... don't think of
it as an object.  The data it refers to might be thought of as an object
with properties and metadata (permissions and such).  Files are a way to get
to such resources.


Files for devices, files for directories, files for data on disk.  All exist
in Plan 9.

They're names with paths... and that's about it (well to me).

[-- Attachment #2: Type: text/html, Size: 1464 bytes --]

Re: [9fans] Re: everything is a directory

[dave.l]

> The answer to that seems obvious.  A single bit could determine  
> whether
> or not an attribute is transient or more permanent.

So now our attributes have an attribute.
How kewl is that?
What about the state of the attribute's attribute:
do you copy that when you copy the attribute?
Is it optional or mandatory?
What's the default, Kenneth?

As has been stated before,
it's then non-extensibility of the file system and file access
protocols that makes them wonderful.

Poor analogy:
the file interface is like having a standardised range of nuts and  
bolts:
you can build wonderful things with them simply;
attributes is like having loads of optional options on your nuts and  
bolts
that you don't need and just get in the way.


> Finally, to argue that files are not objects seems silly.  They ARE
> objects.

"You're merely confounding the issue by playing a naive semantical  
game":-).
Files are objects in the same sense that C-style arrays-of-characters  
are string objects.


> Should I search the archives for boyd for my entertainment :-)

You could do that.
Alternatively, you could keep on talking about <spit>file attributes</ 
spit>:
I can sense him spinning already.
If this nonsense goes on he'll probably start channelling through
some poor unfortunate random 9fan sometime soon.

I don't know ...
9fans is so full of noise these days ...
If it's not C++ it's eye candy and if it's not eye candy it's file  
system abuse.

It's the best OS in the known universe:
why can't people try to improve it instead of making it look like
all the other pukebucket OSes out there?
(Exeunt left mumbling and shaking head).

DaveL.

P.S. Did you mean "transient" ("lasting only for a short time")?

Re: [9fans] Re: everything is a directory

[David Leimbach]

Hmmm who's Dave L?  I thought I was Dave L, but this isn't me Dave L.

Super :-)

On 8/20/07, dave.l@mac.com <dave.l@mac.com> wrote:
> > The answer to that seems obvious.  A single bit could determine
> > whether
> > or not an attribute is transient or more permanent.
>
> So now our attributes have an attribute.
> How kewl is that?
> What about the state of the attribute's attribute:
> do you copy that when you copy the attribute?
> Is it optional or mandatory?
> What's the default, Kenneth?
>
> As has been stated before,
> it's then non-extensibility of the file system and file access
> protocols that makes them wonderful.
>
> Poor analogy:
> the file interface is like having a standardised range of nuts and
> bolts:
> you can build wonderful things with them simply;
> attributes is like having loads of optional options on your nuts and
> bolts
> that you don't need and just get in the way.
>
>
> > Finally, to argue that files are not objects seems silly.  They ARE
> > objects.
>
> "You're merely confounding the issue by playing a naive semantical
> game":-).
> Files are objects in the same sense that C-style arrays-of-characters
> are string objects.
>
>
> > Should I search the archives for boyd for my entertainment :-)
>
> You could do that.
> Alternatively, you could keep on talking about <spit>file attributes</
> spit>:
> I can sense him spinning already.
> If this nonsense goes on he'll probably start channelling through
> some poor unfortunate random 9fan sometime soon.
>
> I don't know ...
> 9fans is so full of noise these days ...
> If it's not C++ it's eye candy and if it's not eye candy it's file
> system abuse.
>
> It's the best OS in the known universe:
> why can't people try to improve it instead of making it look like
> all the other pukebucket OSes out there?
> (Exeunt left mumbling and shaking head).
>
> DaveL.
>
> P.S. Did you mean "transient" ("lasting only for a short time")?
>
>

Re: [9fans] Re: everything is a directory

[Uriel]

I guess robert is not the only one who has to do some reading of 9fans
archives...

uriel

On 8/21/07, David Leimbach <leimy2k@gmail.com> wrote:
> Hmmm who's Dave L?  I thought I was Dave L, but this isn't me Dave L.
>
> Super :-)
>
> On 8/20/07, dave.l@mac.com <dave.l@mac.com> wrote:
> > > The answer to that seems obvious.  A single bit could determine
> > > whether
> > > or not an attribute is transient or more permanent.
> >
> > So now our attributes have an attribute.
> > How kewl is that?
> > What about the state of the attribute's attribute:
> > do you copy that when you copy the attribute?
> > Is it optional or mandatory?
> > What's the default, Kenneth?
> >
> > As has been stated before,
> > it's then non-extensibility of the file system and file access
> > protocols that makes them wonderful.
> >
> > Poor analogy:
> > the file interface is like having a standardised range of nuts and
> > bolts:
> > you can build wonderful things with them simply;
> > attributes is like having loads of optional options on your nuts and
> > bolts
> > that you don't need and just get in the way.
> >
> >
> > > Finally, to argue that files are not objects seems silly.  They ARE
> > > objects.
> >
> > "You're merely confounding the issue by playing a naive semantical
> > game":-).
> > Files are objects in the same sense that C-style arrays-of-characters
> > are string objects.
> >
> >
> > > Should I search the archives for boyd for my entertainment :-)
> >
> > You could do that.
> > Alternatively, you could keep on talking about <spit>file attributes</
> > spit>:
> > I can sense him spinning already.
> > If this nonsense goes on he'll probably start channelling through
> > some poor unfortunate random 9fan sometime soon.
> >
> > I don't know ...
> > 9fans is so full of noise these days ...
> > If it's not C++ it's eye candy and if it's not eye candy it's file
> > system abuse.
> >
> > It's the best OS in the known universe:
> > why can't people try to improve it instead of making it look like
> > all the other pukebucket OSes out there?
> > (Exeunt left mumbling and shaking head).
> >
> > DaveL.
> >
> > P.S. Did you mean "transient" ("lasting only for a short time")?
> >
> >
>

Re: [9fans] Re: everything is a directory

[Roman Shaposhnick]

On Tue, 2007-08-21 at 01:37 +0100, dave.l@mac.com wrote:
> It's the best OS in the known universe:

  True enough. But community makes half the value. I'd hate
to see it deteriorate to the point where every new member
gets flamed to death.

> why can't people try to improve it instead of making it look like
> all the other pukebucket OSes out there?
> (Exeunt left mumbling and shaking head).

  Well, see that's the problem -- even the best kind of art might
need a bit of introductory appreciation lessons.

Thanks,
Roman.

P.S. boyd, is this you? :-)

Re: [9fans] Re: everything is a directory

[Uriel]

On 8/21/07, Roman Shaposhnick <rvs@sun.com> wrote:
> On Tue, 2007-08-21 at 01:37 +0100, dave.l@mac.com wrote:
> > It's the best OS in the known universe:
>
>   True enough. But community makes half the value. I'd hate
> to see it deteriorate to the point where every new member
> gets flamed to death.

This is Plan 9, when have new members *not* been flamed to death? It
is part of the initiation ritual...

>
> > why can't people try to improve it instead of making it look like
> > all the other pukebucket OSes out there?
> > (Exeunt left mumbling and shaking head).
>
>   Well, see that's the problem -- even the best kind of art might
> need a bit of introductory appreciation lessons.
>
> Thanks,
> Roman.
>
> P.S. boyd, is this you? :-)

Nah, not harsh enough, but I can feel the ground shaking from the
spinning somewhere...

uriel

Re: [9fans] Re: everything is a directory

[David Leimbach]

Nope, already lost interest.

On 8/20/07, Uriel <uriel99@gmail.com> wrote:
> I guess robert is not the only one who has to do some reading of 9fans
> archives...
>
> uriel
>
> On 8/21/07, David Leimbach <leimy2k@gmail.com> wrote:
> > Hmmm who's Dave L?  I thought I was Dave L, but this isn't me Dave L.
> >
> > Super :-)
> >
> > On 8/20/07, dave.l@mac.com <dave.l@mac.com> wrote:
> > > > The answer to that seems obvious.  A single bit could determine
> > > > whether
> > > > or not an attribute is transient or more permanent.
> > >
> > > So now our attributes have an attribute.
> > > How kewl is that?
> > > What about the state of the attribute's attribute:
> > > do you copy that when you copy the attribute?
> > > Is it optional or mandatory?
> > > What's the default, Kenneth?
> > >
> > > As has been stated before,
> > > it's then non-extensibility of the file system and file access
> > > protocols that makes them wonderful.
> > >
> > > Poor analogy:
> > > the file interface is like having a standardised range of nuts and
> > > bolts:
> > > you can build wonderful things with them simply;
> > > attributes is like having loads of optional options on your nuts and
> > > bolts
> > > that you don't need and just get in the way.
> > >
> > >
> > > > Finally, to argue that files are not objects seems silly.  They ARE
> > > > objects.
> > >
> > > "You're merely confounding the issue by playing a naive semantical
> > > game":-).
> > > Files are objects in the same sense that C-style arrays-of-characters
> > > are string objects.
> > >
> > >
> > > > Should I search the archives for boyd for my entertainment :-)
> > >
> > > You could do that.
> > > Alternatively, you could keep on talking about <spit>file attributes</
> > > spit>:
> > > I can sense him spinning already.
> > > If this nonsense goes on he'll probably start channelling through
> > > some poor unfortunate random 9fan sometime soon.
> > >
> > > I don't know ...
> > > 9fans is so full of noise these days ...
> > > If it's not C++ it's eye candy and if it's not eye candy it's file
> > > system abuse.
> > >
> > > It's the best OS in the known universe:
> > > why can't people try to improve it instead of making it look like
> > > all the other pukebucket OSes out there?
> > > (Exeunt left mumbling and shaking head).
> > >
> > > DaveL.
> > >
> > > P.S. Did you mean "transient" ("lasting only for a short time")?
> > >
> > >
> >
>

[9fans] Re: everything is a directory

[jsnx]

On Aug 20, 1:48 am, "Douglas A. Gwyn" <DAG...@null.net> wrote:
> "erik quanstrom" <quans...@quanstro.net> wrote in message
>
> news:198ff93d3223123a0e103cbcc4755bc4@quanstro.net...
>
> > hey, how do you grep for stuff in extended attributes?
>
> grep -@, assuming that grep has been extended like other utilities (tar,
> etc.)
>
> One would have thought that a better design would have been
>     /path/file/.attributes/...
> Who makes these decisions, anyway??

Putting attributes in the same namespace as files introduces a whole
new problem -- just as storing ownership in foo/.owners would.

[9fans] Re: everything is a directory

[jsnx]

On Aug 20, 7:55 am, n...@lsub.org (Francisco J Ballesteros) wrote:
> But why don't just keep a foo.attr side by side with the file then?

That would poison the file namespace.

[9fans] Re: everything is a directory

[jsnx]

On Aug 20, 4:41 am, quans...@quanstro.net (erik quanstrom) wrote:
> plan 9 doesn't have icons.

Neither does Linux or FreeBSD -- what's your point? It's the desktop
environment, not the OS, that is responsible for managing icons.

[9fans] Re: everything is a directory

[jsnx]

On Aug 20, 2:01 am, n...@lsub.org (Francisco J Ballesteros) wrote:
> Do you prefer embedding the "face" in each mail as metadata instead of
> keeping a faces db?

That doesn't have anything to do with my point about icons for window
managers, since mail messages are not files, generally speaking. They
may be stored as files, in which case the mail tool that stores them
may also include the face as a file attribute -- probably just a uuid
into the faces db, not the whole image.

Re: [9fans] everything is a directory

[Douglas A. Gwyn]

"Rob Pike" <robpike@gmail.com> wrote in message 
news:7359f0490708200802g2ba8abc0jd61b4f9550bc382f@mail.gmail.com...
> This is only possible because someone said it was OK to
> break directories in the first place.  Not too long ago, if you
> wanted to read a directory you could use a read system call.
> I see no functional reason that that symmetry was worth
> breaking.

That was already changed long before "extended attributes"
entered the picture.  On modern Unix systems (since NFS
or slightly before) and even on Plan 9 one doesn't normally
read and parse actual directory content; the OS/library has
a different interface for directory interrogation.

> Also, I dispute your claim that tools [don't] have to change.
> For instance, ls will misbehave: it will think your file
> is a directory.

But under my proposal that is correct.  An attributed file
*is* an ordinary directory (with ordinary files and/or other
directories as members).  But if you "cat" or "grep" the file
name, you access its .data member.  Since these days the
actual directory content isn't meaningfully accessed, that frees
it up for some other interpretation, such as what I proposed.
(You can also access the file data as path/name/.data, so
there is nothing misleading about this use of the existing
filesystem structure.)

(An implementation detail I didn't mention: I recommend
that the .data member always be the first entry for speed
of access, thus no "." or ".." entries [unlike Solaris-10
extended attributes]; those should be handled by the OS
path interpreter, as on Plan 9.)

>  And there will be lots more instances
> because, fundamentally, stat will be misleading.
> Then all the existing tools will be changed because
> people will want to use them on attributed files and
> their attributes.

There is no reason why "attribute aware" tools shouldn't
be developed, if attributes are indeed useful (something
I'm not yet convinced of).  My purpose was to figure
out a way of implementating extended file attributes with
minimal disruption to the usual Unix/Plan 9 file model.
Note that the Linux/Solaris-10 (Posix?) implementation
requires separate system calls and extensive changes to
standard tools *and* to the way those tools are invoked.

> It reminds me of symlinks, which I detest. ...

Yes, the analogy is close.

> but they broke just about everything by introducing a
> second kind of file. ...

Which is what we want to avoid this time.

> Attributes, as created or as posited here, are better
> only because for the most part one can ignore them
> altogether. Except when you can't.

Unfortunately, if you forget about attributes on Linux/
Solaris-10, you far too easily do what users would
definitely consider to be "the wrong thing".  That's
*why* so many standard tools had to be modified
using their approach to extended attributes.

> If you want attributes, rethink the system, don't hack
> it.

So what is a better proposal?

[9fans] Re: everything is a directory

[app]

I would just like to remind, that directories with union mounts do
present "inheritance". Because inheritance is the most defining
characteristic of the OO paradigm, is is quite fair to say that file
systems in Plan 9 are (persistent, networked) "objects" in the
classical OO sense.

You can take a standard /net and bind your own modified /net as an
union with it, hiding some files, replacing some files, and leaving
some files intact. That is, you can take a father object, inherit some
of its functionality to the child object (the union) untouched, modify
some functionality (perhaps filtering ctrl files and writing back to
original ones), and remove some. Exactly like in SmallTalk or C++ or
Java.

Re: [9fans] Re: everything is a directory

[Uriel]

On 8/21/07, jsnx <jason.dusek@gmail.com> wrote:
> On Aug 20, 2:01 am, n...@lsub.org (Francisco J Ballesteros) wrote:
> > Do you prefer embedding the "face" in each mail as metadata instead of
> > keeping a faces db?
>
> That doesn't have anything to do with my point about icons for window
> managers, since mail messages are not files, generally speaking.

upasfs(4) In Plan 9 everything is a file (server).

And that is one more reason why extended attributes make no sense in
Plan 9, because if you need them it means you have not done a good job
designing your file server interface.

uriel

Re: [9fans] Re: everything is a directory

[Brantley Coile]

Erik is too tall and thin to be a troll.  And
I've never seen him under a bridge.

Robert William Fuller wrote:
> erik quanstrom wrote:
>>> Finally, to argue that files are not objects seems silly.  They ARE
>>> objects.  They have properties.  They have well defined interfaces for
>>> manipulating those properties.  A more reasonable argument may be that
>>> they are not object oriented since they lack certain prerequisites such
>>> as inheritance and abstraction, both mechanisms of extensibility.
>> so files are non-object-oriented objects?
>>
>> i bet you can't say that without smiling.
> 
> Troll.  And I am smiling :-p.
> 
> Objects are defined as data structures with associated methods for
> manipulating them.
> 
> Object oriented programming requires four key attributes:
> encapsulation, inheritance, abstraction, and polymorphism.  Object
> oriented programming is a more sophisticated programming discipline.
> 
> Objects do not imply object oriented programming, although object
> oriented programming implies objects.
> 
> You're merely confounding the issue by playing a naive semantical game
> based on the commonality of the word "object."
> 

Re: [9fans] Re: everything is a directory

[Skip Tavakkolian]

>> Do you prefer embedding the "face" in each mail as metadata instead of
>> keeping a faces db?
> 
> That doesn't have anything to do with my point about icons for window
> managers, since mail messages are not files, generally speaking. They
> may be stored as files, in which case the mail tool that stores them
> may also include the face as a file attribute -- probably just a uuid
> into the faces db, not the whole image.

plan9 already supports extended attributes.  e.g. upasfs(4) and
lnfs(4).  there are also tools that let you access extended
attributes created on other systems: aux/olefs

maybe you should ask the "right" fs:

% upas/fs
% grep jason /mail/fs/mbox/*/from

or

% lnfs /
% touch /usr/glenda/other_examples_of_file_as_a_filesystem_include_MS_OLE_structured_storage

Re: [9fans] Re: everything is a directory

[Robert Raschke]

Robert William Fuller wrote:
> Object oriented programming requires four key attributes:
> encapsulation, inheritance, abstraction, and polymorphism.  Object
> oriented programming is a more sophisticated programming discipline.

cpu% dict sophisticated
1	
  { So*phis"ti*cate (?), So*phis"ti*ca`ted (?) }, a. Adulterated;
not pure; not genuine.
  So truth, while only one supplied the state,
   Grew scare and dear, and yet sophisticate. Dryden.
 

Re: [9fans] Re: everything is a directory

[andrey mirtchovski]

In the spirit of "Eschew obfuscation", here's a new fortune:

Eschew [artificially] extended attributes in Women, Men, and File Systems alike.

[9fans] Re: everything is a directory

[jsnx]

On Aug 20, 3:05 pm, hydrologiccy...@gmail.com (Robert William Fuller)
wrote:
> Objects are defined as data structures with associated methods for
> manipulating them.

What is the difference between structures and objects, then?

[9fans] Re: everything is a directory

[jsnx]

On Aug 21, 1:48 am, app <porttik...@gmail.com> wrote:
> I would just like to remind, that directories with union mounts do
> present "inheritance"...Exactly like in SmallTalk or C++ or
> Java.

Well, it's not exactly like in Java, since you are composing *two*
bases to make a child -- that's multiple inheritance, which is not
permitted in Java.

There are many programming paradigms that support module composition
-- OO is not the only one.

[9fans] Re: everything is a directory

[app]

On Aug 22, 11:47 am, jsnx <jason.du...@gmail.com> wrote:
> On Aug 21, 1:48 am, app <porttik...@gmail.com> wrote:
>
> > I would just like to remind, that directories with union mounts do
> > present "inheritance"...Exactly like in SmallTalk or C++ or
> > Java.
>
> Well, it's not exactly like in Java, since you are composing *two*
> bases to make a child -- that's multiple inheritance, which is not
> permitted in Java.

In OO you are _always_ composing stuff from at least two sets of data
and functions: the data and functions of parent, and the data and
functions of the new child you are creating. In Plan 9, the standard /
net might represent the parent, and /my/net that you bind on top of it
represents the new data and new functions you create, in purpose of
modifying the parent.

Don't you agree that the semantics are roughly the same as in
classical OO, even though the syntax, type checking and run-time vs.
compile-time trade offs are different?

Benefits in Plan 9 approach are numerous: language independence, free
serialization, optional persistence, networking, object naming and
object permissions provided automatically by the system, not by
programmers.

Re: [9fans] Re: everything is a directory

[roger peppe]

> Don't you agree that the semantics are roughly the same as in
> classical OO, even though the syntax, type checking and run-time vs.
> compile-time trade offs are different?

no, union binding in plan 9 does not really give inheritance
semantics in general. for some cases it can (e.g. /net), but usually when
it's already been anticipated. try combining two instances of upas/fs,
for example.

to me, it seems more like the C library system than an object inheritance
system - you can combine and override elements, but not much else.

Re: [9fans] Re: everything is a directory

[erik quanstrom]

> In OO you are _always_ composing stuff from at least two sets of data
> and functions: the data and functions of parent, and the data and
> functions of the new child you are creating. In Plan 9, the standard /
> net might represent the parent, and /my/net that you bind on top of it
> represents the new data and new functions you create, in purpose of
> modifying the parent.

i think you're missing some important bits, and i think you misunderstand
the culture.  you might want to read ip(3) for information on how /net
is really structured.  even supposing that each process had it's own network
stack (which they do not), it's doubtful that /my/net would be the choice.
namespace(6) describes how /lib/namespace is interpreted to build an
initial namespace.

- erik

[9fans] Re: everything is a directory

[jsnx]

On Aug 22, 2:54 am, app <porttik...@gmail.com> wrote:
> On Aug 22, 11:47 am, jsnx <jason.du...@gmail.com> wrote:
>
> > On Aug 21, 1:48 am, app <porttik...@gmail.com> wrote:
>
> > > I would just like to remind, that directories with union mounts do
> > > present "inheritance"...Exactly like in SmallTalk or C++ or
> > > Java.
>
> > Well, it's not exactly like in Java, since you are composing *two*
> > bases to make a child -- that's multiple inheritance, which is not
> > permitted in Java.
>
> In OO you are _always_ composing stuff from at least two sets of
> data...

The reason I'm calling it 'multiple inheritance' is because there are
two distinct bases which you are extending to create a child. In
single inheritance, there is one base that you extend to create a
childhood.

The only reason I bring it up is to point out the loose coupling of OO
languages and the notion you are talking about. It is much more like
mixins, a feature in Ruby, than it is like inheritance in Java.

Why do I stress this point? Because of my Jihad against object-
oriented programming in particular, and my distress at the general
willingness of folks in the computing community to compare good
features in any system to bad features in object-oriented programming.

By noting the connection between inheritance and unions, you have not
seen into inheritance but rather through it -- out the other side to a
rich world of architectural approaches.

> Benefits in Plan 9 approach are numerous: language independence, free
> serialization, optional persistence, networking, object naming and
> object permissions provided automatically by the system, not by
> programmers.

Indeed, I agree -- using the filesystem is the "way of the future"^tm
for module composition, document structure, laying out web sites,
managing users...

We can only hope for the day when people realize that text streams are
the real Common Language Runtime!

[9fans] Re: everything is a directory

[jsnx]

On Aug 20, 10:53 am, urie...@gmail.com (Uriel) wrote:
> Sad thing is that when you try to convince Unix people of the power of
> private namespaces, they complain that they seem 'too confusing'...
> yea, because symlinks, extended attributes and other such hideous
> hacks are so much nicer...

Let's return to my original post and one of the use cases I mentioned
there. Say I have TeX document with innumerable sections in it,
sections that I would like to handle 'file style', for good
interaction with version control. I do not want a lot of include
directives and other crap, though -- I don't want to actually make the
new files. Let us imagine a TeX filesystem:

  sec/
    matter
    images
    sec0/
      matter
    sec1/
    ...and so on...

This filesystem has two forms -- serialized form and hierarchical
form. In serialized form, it is a TeX file; I mount this file to get
the example above.

Now, here is the tough part, I think -- say I want to use version
control with the above example. I check the directories into my
version control system, which pollutes the TeX fs with a bunch of .svn
directories. This makes my filesystem puke; so I rewrite it to ignore
the .svn thingies. Very good; but, I'm still not finished. Because I
checked the dirs, and not their serialized master, into version
control, I have big problem when I check the thing back out -- it is
now a normal filesystem, full of files and dirs, that has forgotten
it's TeX nature. I have to do a cast -- Iinstantiate it as a normal fs
and then cast it to a TeX fs.

That seems pretty unworkable to me, but I'm very new to Plan 9 and
there's probably something here I'm missing. With attributes and my
dir/file system, there is only one way to mount something, so I am in
no danger of losing my metadata when I try to compose filesystems (in
this case, I am composing an ad-hoc 'version controlled filesystem'
with a TeX filesytem). I can use more than one kind of metadata at a
time. However, in the metadata-in-fs approach, a resource can only be
governed by metadata from one or another filesystem at any given time;
a real problem for the application I am considering.

[9fans] Re: everything is a directory

[Douglas A. Gwyn]

"jsnx" <jason.dusek@gmail.com> wrote in message 
news:1187630509.315942.311670@m37g2000prh.googlegroups.com...
> Putting attributes in the same namespace as files introduces a whole
> new problem -- just as storing ownership in foo/.owners would.

I don't see why.  Attributes have to be attached to a specific file,
so why not attach them "in the usual way" rather than inventing a
new mechanism?  If .owners was enforced by the OS then why
couldn't it be attached in the usual namespace?

Re: [9fans] Re: everything is a directory

[Steve Simon]

Do the arguments for extended attributes stem from a desire to store
info for backups/HSM and for version control systems?

Icons are the responsibility of the desktop manager.

Faces are stored in their own "database" on plan9.

We don't have resource forks - the window managers (rio and acme)
keep all the widgets need to display any plan9 program internally (some
colours, textures, cursors, menuhit tables).

Perhaps the reason plan9 doesn't see the need for extended attributes
is the same as the reason it doesn't need backup systems, HSM, and
version control software - has fossil and venti (or kenfs depending on
your taste) solved this problem for us?

Maybe there are other examples of where extended attributes are really
useful?

I'am not trolling, I just don't see their efficacy in plan9.

-Steve

[9fans] Re: everything is a directory

[Douglas A. Gwyn]

jsnx wrote:
> Now, here is the tough part, I think -- say I want to use version
> control with the above example. ...
> That seems pretty unworkable to me, ...

The problem seems to me to be in the design of that version control
system.

You might work around it by keeping the version archives in one place
and the working copies you actually use in another place.

[9fans] Re: everything is a directory

[jsnx]

On Aug 23, 1:50 am, "Douglas A. Gwyn" <DAG...@null.net> wrote:
> If .owners was enforced by the OS then why
> couldn't it be attached in the usual namespace?

Well, that could lead to a lot of 'reserved words' in the filesystem.
Down that road lay the IANA...

[9fans] Re: everything is a directory

[jsnx]

On Aug 23, 3:35 am, st...@quintile.net (Steve Simon) wrote:
> I'am not trolling, I just don't see their efficacy in plan9.

I don't see how to architect the system I discussed without attributes.

Re: [9fans] Re: everything is a directory

[Martin Neubauer]

* jsnx (jason.dusek@gmail.com) wrote:
> I don't see how to architect the system I discussed without attributes.

It's not hard to conceive a system that requires _any- set of features. Not
impressed.

Re: [9fans] Re: everything is a directory

[David Leimbach]

[-- Attachment #1: Type: text/plain, Size: 1188 bytes --]

On 8/24/07, jsnx <jason.dusek@gmail.com> wrote:
>
> On Aug 23, 3:35 am, st...@quintile.net (Steve Simon) wrote:
> > I'am not trolling, I just don't see their efficacy in plan9.
>
> I don't see how to architect the system I discussed without attributes.
>

I can...

Build yourself a file server that provides the environment you want with
attributes... don't inject it into the core system.  Store it for real in
fossil files with a certain format.

Done?

Plan 9's ability to have synthetic filesystems seems to do away with the
need to change the way the core filesystems work.

If you need the filesystem to behave or to have different semantics it can
clearly be done.  Seen  the TCP interface for instance?  sshnet?  Sure
sometimes we have to make small compromises in the way the system works to
achieve some new functionality, but when we do, we seem to be buying a lot
for our trade up.  (my understanding is sshnet did require some changes to
the way the system worked before sshnet was proposed.)

Plan 9 seems to have always been about trying to solve problems in a way
that's simpler than the way things seem to have turned out in Linux/Unix
etc.

[-- Attachment #2: Type: text/html, Size: 1823 bytes --]

[9fans] Re: everything is a directory

[jsnx]

On Aug 24, 3:55 am, leim...@gmail.com (David Leimbach) wrote:
> On 8/24/07, jsnx <jason.du...@gmail.com> wrote:
> > On Aug 23, 3:35 am, st...@quintile.net (Steve Simon) wrote:
> > > I'am not trolling, I just don't see their efficacy in plan9.
> > I don't see how to architect the system I discussed without attributes.
> Build yourself a file server that provides the environment you want with
> attributes... don't inject it into the core system.  Store it for real in
> fossil files with a certain format.
>
> Done?

If I want to have a system with the features of two distinct
filesystems, what then? Attributes allow for composition, whereas the
FS oriented approach appears not to.

I would have to write (or obtain) a TeX fs, a SVN fs and then a TeX-
SVN fs if I wanted to have those feature sets independently, if I am
understanding you correctly.

[9fans] Re: everything is a directory

[jsnx]

On Aug 24, 1:58 am, m...@gmx.net (Martin Neubauer) wrote:
> * jsnx (jason.du...@gmail.com) wrote:
> > I don't see how to architect the system I discussed without attributes.
>
> It's not hard to conceive a system that requires _any- set of features. Not
> impressed.

Martin, your posts on this thread have been just awesome so far. Keep
up the good work!

[9fans] Re: everything is a directory

[Douglas A. Gwyn]

"jsnx" <jason.dusek@gmail.com> wrote in message 
news:1187900988.333897.37270@e9g2000prf.googlegroups.com...
> On Aug 23, 1:50 am, "Douglas A. Gwyn" <DAG...@null.net> wrote:
>> If .owners was enforced by the OS then why
>> couldn't it be attached in the usual namespace?
> Well, that could lead to a lot of 'reserved words' in the filesystem.

Probably it should be .attributes/owner, no new added word beyond
what would be needed for extended attributes anyway.

Re: [9fans] Re: everything is a directory

[Uriel]

Fossil already provides something like 'svnfs' (but much simpler and
saner), as far as I can see your proposed texfs is no different
conceptually from upasfs, and nothing keeps you from using both
together.

And I certainly fail to see how adding attributes into the mix makes
composition any easier, they just add another meta-namespace with its
own ambiguities and clashes.

In the end, it is also good to remember that file servers, while
incredibly powerful, are not the perfect abstraction for everything,
when it comes to composition the tool/text-strem original unix
philosophy (this days lost everywhere except in Plan 9) is still the
most powerful, file servers simply and transparently expand the
environment where tools live and interact.

Not everything needs to be a file server, but everything should handle
text streams (ie., pipes or files).

Best wishes

uriel

On 8/29/07, jsnx <jason.dusek@gmail.com> wrote:
> On Aug 24, 3:55 am, leim...@gmail.com (David Leimbach) wrote:
> > On 8/24/07, jsnx <jason.du...@gmail.com> wrote:
> > > On Aug 23, 3:35 am, st...@quintile.net (Steve Simon) wrote:
> > > > I'am not trolling, I just don't see their efficacy in plan9.
> > > I don't see how to architect the system I discussed without attributes.
> > Build yourself a file server that provides the environment you want with
> > attributes... don't inject it into the core system.  Store it for real in
> > fossil files with a certain format.
> >
> > Done?
>
> If I want to have a system with the features of two distinct
> filesystems, what then? Attributes allow for composition, whereas the
> FS oriented approach appears not to.
>
> I would have to write (or obtain) a TeX fs, a SVN fs and then a TeX-
> SVN fs if I wanted to have those feature sets independently, if I am
> understanding you correctly.
>

[9fans] Re: everything is a directory

[jsnx]

On Aug 29, 2:06 am, "Douglas A. Gwyn" <DAG...@null.net> wrote:
> "jsnx" <jason.du...@gmail.com> wrote in message
> > On Aug 23, 1:50 am, "Douglas A. Gwyn" <DAG...@null.net> wrote:
> >> If .owners was enforced by the OS...
> > ...that could lead to a lot of 'reserved words'...
>
> Probably it should be .attributes/owner, no new added word beyond
> what would be needed for extended attributes anyway.

That's a good idea, and I think that brings us full circle. If we
allow for a
.attributes/content which is accepted by all the major tools, then
what I want
to do is done.

Re: [9fans] Re: everything is a directory

[David Leimbach]

[-- Attachment #1: Type: text/plain, Size: 1488 bytes --]

On 8/30/07, jsnx <jason.dusek@gmail.com> wrote:
>
> On Aug 29, 2:06 am, "Douglas A. Gwyn" <DAG...@null.net> wrote:
> > "jsnx" <jason.du...@gmail.com> wrote in message
> > > On Aug 23, 1:50 am, "Douglas A. Gwyn" <DAG...@null.net> wrote:
> > >> If .owners was enforced by the OS...
> > > ...that could lead to a lot of 'reserved words'...
> >
> > Probably it should be .attributes/owner, no new added word beyond
> > what would be needed for extended attributes anyway.
>
> That's a good idea, and I think that brings us full circle. If we
> allow for a
> .attributes/content which is accepted by all the major tools, then
> what I want
> to do is done.
>

I'm a bit confused at this point... What is this proposal and why is it
attractive?

On the surface it looks like you could get everything you want using a pair
of files to me.  One to tell you the start of the file's normal data (or the
end), and the other to let you have scratch space for arbitrary key/value
pairs.

The secondary file is just then a table of contents for the original file
that just happens to hold both the core file data and the other gunk that
goes with it.... which brings me back to the idea that these problems should
have been solved in the file formats themselves (and again, not terribly
sure that ACLs or such could have been safely implemented in that way, my
understanding is that is how the extended attributes can be put to good use
on other filesystems)


Dave

[-- Attachment #2: Type: text/html, Size: 2121 bytes --]

Re: [9fans] Re: everything is a directory

[Douglas A. Gwyn]

On 8/30/07, jsnx <jason.dusek@gmail.com> wrote:

> On Aug 29, 2:06 am, "Douglas A. Gwyn" <DAG...@null.net> wrote:
> > "jsnx" <jason.du...@gmail.com> wrote in message 
> > > On Aug 23, 1:50 am, "Douglas A. Gwyn" <DAG...@null.net> wrote:
> > >> If .owners was enforced by the OS...
> > > ...that could lead to a lot of 'reserved words'... 
> >
> > Probably it should be .attributes/owner, no new added word beyond
> > what would be needed for extended attributes anyway.

> That's a good idea, and I think that brings us full circle. If we allow for a
> .attributes/content which is accepted by all the major tools, then what I want
> to do is done.

> I'm a bit confused at this point... What is this proposal and why is it attractive? 

> On the surface it looks like you could get everything you want using a pair of files to me.  One to tell you the start of
> the file's normal data (or the end), and the other to let you have scratch space for arbitrary key/value pairs.

> ...

The proposal, as modified, provides a mechanism that can be used to
add extended attributes to files, to encode existing attributes for
files, and to associate all kinds of other files with given files.

The attribute names (analogous to Linux/Solaris extended attributes)
would all be filenames within the .attributes directory attached to
the target file, and their values would be the contents of those
named files.

My proposed implementation, for any Unix-like system including
Plan9, is to change read() on a directory to instead read from the
directoryname/.data file, if it exists, or report an error if it
doesn't exist.  This change should cause no disruption to existing
file structures or applications, since as it stands there is no
purpose in read()ing the contents of a directory (these days we use
walk(), getdents(), etc. instead).  Almost all utilities, including
"tar", would continue to do "the right thing".  Probably "cp" would
need to be changed so it copies directory trees as well as ordinary
files.  Note that there would still be "leaf" (non-directory) files
under my proposal, particularly .data files and not-yet-converted
existing ordinary files.  If one wanted to totally convert the
storage filesystem, then .data entries could be hidden and then all
visible files would indeed be directories.

The primary purpose is to provide support for extended attributes
in a way that works within the traditional Unix hierarchical file
model, without requiring extensive kludges like the Linux/Solaris
implementations have.  A two-file model would also require a large
amount of ad-hoc kludgery to work well enough.

Re: [9fans] Re: everything is a directory

[Joel C. Salomon]

On 8/31/07, Douglas A. Gwyn <DAGwyn@null.net> wrote:
> My proposed implementation, for any Unix-like system including
> Plan9, is to change read() on a directory to instead read from the
> directoryname/.data file, if it exists, or report an error if it
> doesn't exist.  This change should cause no disruption to existing
> file structures or applications, since as it stands there is no
> purpose in read()ing the contents of a directory (these days we use
> walk(), getdents(), etc. instead).

See /sys/src/libc/9sys/dirread.c and read(5).  Plan 9 doesn't have a
distinct system call for reading directory entries.

--Joel

Re: [9fans] Re: everything is a directory

[Uriel]

With this way of thinking we will never catch up with lunix's 400
syscalls (and counting, not to mention the ioctls)!

uriel

> See /sys/src/libc/9sys/dirread.c and read(5).  Plan 9 doesn't have a
> distinct system call for reading directory entries.
>
> --Joel
>

Re: [9fans] Re: everything is a directory

[Joel C. Salomon]

On 9/1/07, Uriel <uriel99@gmail.com> wrote:
> With this way of thinking we will never catch up with lunix's 400
> syscalls (and counting, not to mention the ioctls)!

We're partly there in spirit; what fraction of the 4e kernel's system
calls are there for backwards compatibility?  ;)

Back on topic, I wasn't proposing a readdir() syscall but pointing out
that Douglas's suggestion would not in fact be painless or transparent
under Plan 9.

--Joel

Re: [9fans] Re: everything is a directory

[erik quanstrom]

> We're partly there in spirit; what fraction of the 4e kernel's system
> calls are there for backwards compatibility?  ;)

i don't know if that's a rhetorical question or not.  it's not hard to answer.

/sys/src/libc/9syscall/sys.h lists 51 system calls.  ten have underscores.
OSEEK is also obsolete.  that would make 22%.

- erik

Re: [9fans] Re: everything is a directory

[Anthony Martin]

> /sys/src/libc/9syscall/sys.h lists 51 system calls.
> ten have underscores. OSEEK is also obsolete.
> that would make 22%.

If you count DEATH and PASSFD in sysproc.c and
sysfile.c, respectively, then it's 25%. They
are the missing syscall numbers 48 and 49.

--
"Christ, what an imagination I've got." - Shalmaneser

Re: [9fans] Re: everything is a directory

[Uriel]

On 9/2/07, Joel C. Salomon <joelcsalomon@gmail.com> wrote:
> On 9/1/07, Uriel <uriel99@gmail.com> wrote:
> > With this way of thinking we will never catch up with lunix's 400
> > syscalls (and counting, not to mention the ioctls)!
>
> We're partly there in spirit; what fraction of the 4e kernel's system
> calls are there for backwards compatibility?  ;)
>
> Back on topic, I wasn't proposing a readdir() syscall but pointing out
> that Douglas's suggestion would not in fact be painless or transparent
> under Plan 9.

Indeed, and I would say that is a feature, not a bug. (Adding an extra
syscall to read dirs would be a bug). But maybe my sarcasm got lost
along the way.

And actually, I think one could have something similar to Douglas
suggestion in Plan 9 without changing the kernel or the vfs, or even
the file servers, just have a stackable file server which for every
original file /foo.txt allows you to access a /foo.txt@extendedjunk/
dir where all the extended attributes or whatever can live, that would
even keep backwards compatibility with all existing tools (tools that
don't know about @extendedjunk/ dirs would not even see them unless
they explicitly walk to them, so you could use cd
/foo.txt@extenedjunk/, followed of ls and cat to inspect the
attributes, but if you do cat /* or ls / you would get a sensible
output.

Anyway, I still think it would be a waste of time, but like unix in
its day, plan9 makes adding new 'features' rather easy, whatever they
are worthy or another symlink-hell awaiting to happen is another
question.

Best wishes

uriel

P.S.: The Plan 9 industry is not creating enough jobs, we need more
syscalls! And dynamic linking!

Re: [9fans] Re: everything is a directory

[Joel C. Salomon]

On 9/2/07, Uriel <uriel99@gmail.com> wrote:
> And actually, I think one could have something similar to Douglas
> suggestion in Plan 9 without changing the kernel or the vfs, or even
> the file servers, just have a stackable file server which for every
> original file /foo.txt allows you to access a /foo.txt@extendedjunk/
> dir where all the extended attributes or whatever can live, that would
> even keep backwards compatibility with all existing tools (tools that
> don't know about @extendedjunk/ dirs would not even see them unless
> they explicitly walk to them, so you could use cd
> /foo.txt@extenedjunk/, followed of ls and cat to inspect the
> attributes, but if you do cat /* or ls / you would get a sensible
> output.

Or how about this:  Just as you don't need read permissions on a
directory to walk it -- so long as you know the name of the file you
want -- how about 9p allowing walks to ordinary files.  /foo.txt will
exist, and reads work as usual, but /foo.txt/metadata.xml (of course
it's all buzzword-compliant XML!) can be walked to and manipulated.
Disgusting, eh?

--Joel

Re: [9fans] Re: everything is a directory

[Uriel]

Disgusting indeed, but it might even work with no changes to 9P; it
might confuse some clients, but doesn't sound easily doable given some
basic naming conventions (of course, then you lose the possibility to
cd into that dir and run ls there, which is the main reason I thought
an extension to the file name would make more sense)

uriel

On 9/2/07, Joel C. Salomon <joelcsalomon@gmail.com> wrote:
> On 9/2/07, Uriel <uriel99@gmail.com> wrote:
> > And actually, I think one could have something similar to Douglas
> > suggestion in Plan 9 without changing the kernel or the vfs, or even
> > the file servers, just have a stackable file server which for every
> > original file /foo.txt allows you to access a /foo.txt@extendedjunk/
> > dir where all the extended attributes or whatever can live, that would
> > even keep backwards compatibility with all existing tools (tools that
> > don't know about @extendedjunk/ dirs would not even see them unless
> > they explicitly walk to them, so you could use cd
> > /foo.txt@extenedjunk/, followed of ls and cat to inspect the
> > attributes, but if you do cat /* or ls / you would get a sensible
> > output.
>
> Or how about this:  Just as you don't need read permissions on a
> directory to walk it -- so long as you know the name of the file you
> want -- how about 9p allowing walks to ordinary files.  /foo.txt will
> exist, and reads work as usual, but /foo.txt/metadata.xml (of course
> it's all buzzword-compliant XML!) can be walked to and manipulated.
> Disgusting, eh?
>
> --Joel
>

Re: [9fans] Re: everything is a directory

[Douglas A. Gwyn]

"Joel C. Salomon" <joelcsalomon@gmail.com> wrote ...
> Back on topic, I wasn't proposing a readdir() syscall but pointing out
> that Douglas's suggestion would not in fact be painless or transparent
> under Plan 9.

Since I was unable to get Plan9 installed successfully when I set up
my home systems, and can't use it at work, I don't have direct
experience with its implementation.  I had been thinking that the
Twalk part of 9P was connected with getting directory entries.
However, if dirread() is used instead, then the implementation of
dirread() would of course have to change (turn into a syscall,
with stuff added to 9P and the various file servers, alas).  That
would indeed require a lot of work.  Unix already made the
change a couple of decades ago.

Re: [9fans] Re: everything is a directory

[Charles Forsyth]

> Unix already made the
> change a couple of decades ago.

plan 9's definition of read on a directory, and the definition of the data
thereby returned, is different from Unix's and there was no
need to make a change.  for instance, the data returned is unrelated
to any underlying storage structure, which is just as well, because it's
usually just some data structures in a program.

actually, since Unix's data from reading a directory was essentially a sequence of (ino, name) tuples,
that independence might have been true of Unix as well, but previously-used slots
were revealed, and the names were fixed length, because it really was just the data read
from the disc representation.

Re: [9fans] Re: everything is a directory

[Joel C. Salomon]

On 9/3/07, Uriel <uriel99@gmail.com> wrote:
> Disgusting indeed, but it might even work with no changes to 9P; it
> might confuse some clients, but doesn't sound easily doable given some
> basic naming conventions (of course, then you lose the possibility to
> cd into that dir and run ls there, which is the main reason I thought
> an extension to the file name would make more sense)

You don't want to run ls, remember.  The point of this exercise is to
conceptually add some form of metadata to files.  All that's needed
(if that!) is a set of attribute=value pairs, which can all be kept in
a single 'file' with a known name and predetermined format.  (Or a
database with multiple file-like 'views', so meta.txt, meta.csv, and
meta.xml are all available.  Ick.)

--Joel

Re: [9fans] Re: everything is a directory

[Uriel]

Well, I expected people who wants extended attributes would want
resource forks (double yuck) and who knows what else.

Actually I think something like this was russ' suggestion at IWP9 to
handle things like locking, symlinks and other PoSix junk.

All that would be needed is a well defined convention, and then
clients who cared about such extra 'features' could use them without
changing 9p, existing servers or clients (which don't care about this
'features').

uriel

On 9/3/07, Joel C. Salomon <joelcsalomon@gmail.com> wrote:
> On 9/3/07, Uriel <uriel99@gmail.com> wrote:
> > Disgusting indeed, but it might even work with no changes to 9P; it
> > might confuse some clients, but doesn't sound easily doable given some
> > basic naming conventions (of course, then you lose the possibility to
> > cd into that dir and run ls there, which is the main reason I thought
> > an extension to the file name would make more sense)
>
> You don't want to run ls, remember.  The point of this exercise is to
> conceptually add some form of metadata to files.  All that's needed
> (if that!) is a set of attribute=value pairs, which can all be kept in
> a single 'file' with a known name and predetermined format.  (Or a
> database with multiple file-like 'views', so meta.txt, meta.csv, and
> meta.xml are all available.  Ick.)
>
> --Joel
>

Re: [9fans] Re: everything is a directory

[Roman Shaposhnik]

[-- Attachment #1: Type: text/plain, Size: 949 bytes --]

On Aug 24, 2007, at 3:55 AM, David Leimbach wrote:

>
>
> On 8/24/07, jsnx <jason.dusek@gmail.com> wrote:
> On Aug 23, 3:35 am, st...@quintile.net (Steve Simon) wrote:
> > I'am not trolling, I just don't see their efficacy in plan9.
>
> I don't see how to architect the system I discussed without
> attributes.
>
> I can...
>
> Build yourself a file server that provides the environment you want
> with attributes... don't inject it into the core system.  Store it
> for real in fossil files with a certain format.
>
> Done?
>
> Plan 9's ability to have synthetic filesystems seems to do away
> with the need to change the way the core filesystems work.

   Speaking of which -- is there an easy way of implementing
filtering filesystems? Something that would give
a filter-server N existing [sub]trees and a set of instructions on
how to reshuffle their content and represent
it as a mountable output.

Thanks,
Roman.

[-- Attachment #2: Type: text/html, Size: 1702 bytes --]

Re: [9fans] Re: everything is a directory

[Roman Shaposhnik]

On Aug 29, 2007, at 2:06 AM, jsnx wrote:

> On Aug 24, 3:55 am, leim...@gmail.com (David Leimbach) wrote:
>> On 8/24/07, jsnx <jason.du...@gmail.com> wrote:
>>> On Aug 23, 3:35 am, st...@quintile.net (Steve Simon) wrote:
>>>> I'am not trolling, I just don't see their efficacy in plan9.
>>> I don't see how to architect the system I discussed without
>>> attributes.
>> Build yourself a file server that provides the environment you
>> want with
>> attributes... don't inject it into the core system.  Store it for
>> real in
>> fossil files with a certain format.
>>
>> Done?
>
> If I want to have a system with the features of two distinct
> filesystems, what then? Attributes allow for composition, whereas the
> FS oriented approach appears not to.

   I apologize for coming into the discussion a bit late. But I
always wanted
to share the following observation: metadata seems to be very
confusing to
an average joe. And when I say metadata I mean all
metadata, not just resource forks and attributes. Time and again I've
seen PC users name their digital picture (or at least directories with
the digital pictures) <date>_<place_taken>. Why? Well, because
even though date and time are part of the metadata in pretty much
every filesystem these days it is extremely difficult to make sure that
all the tools handle them and preserve them just the way you want it
to be
(think about the time picture was taken vs. the time the file was
created vs. the time it was last updated or even accessed).

   On top of the constant paranoia (Quiz-1: what's the option for cp to
preserve the timestamp?) the metadata approach makes it very
difficult to manipulate collections of files. Quiz-2: what's the
moral equivalent
of doing:

   $ cp 09_*_2007*.jpg ~/USA_tour

if I didn't have date as part of the name?

   These usability issues with even the most established bits of
file metadata such as date, time, owner, etc. show up often enough
to make one think of if not abolishing metadata all together, at least
not proliferating it any further.

>
> I would have to write (or obtain) a TeX fs, a SVN fs and then a TeX-
> SVN fs if I wanted to have those feature sets independently, if I am
> understanding you correctly.

   The problem you seem to be alluding to is genuine and I'll try to
paraphrase it in order to see whether we're talking about the same
thing. Suppose you have a collection of leaf-node objects that
you can group together in a number of different ways. Suppose
also that these groupings are mostly orthogonal to each other.
Now, if you express objects as files -- what is the most natural
way of expressing various groupings? To make it even more clear
here's a practical example: in my build system I have to produce
binaries for SPARC and x86 running under Solaris and Linux
and each of those binaries either built for debugging or with
maximum optimization. My objects are, obviously, files
but how am I supposed to structure the groupings? It looks like
three levels of subdirectories should do the trick giving me
names like:
      sparc/linux/opt/binary
Now, of course, even if we put aside questions like why not
linux/sparc/opt/binary, we still have to figure out how to handle
the case where the very same physical binary file happens to
be a fat binary that can run on both SPARC and x86 Solaris
OSes. Where should it go? Should we symlink to it from
"the other" place? Questions like these seem to call for tagging and
filter filesystems (see my previous question in this very thread).
The scariest part is that they also come dangerously close to the
oldest argument in the OS design history: filesystem abstraction
vs. database abstraction for managing persistent data.

Your hypothetical problem seems to fit exactly the same profile: your
leaf-node objects are lines (or perhaps even sets of TeX lexems)
and the grouping can either be determined based on which SCM
delta a particular object belongs to or where it fits in a tree-like
TeX document structure.

The problem is genuine, and I don't know the right solution for
it, but somehow I don't think the metadata would help here. It seems
to me that you could be confusing a building block for a proper
solution (e.g. tagging) with the solution itself. After all, if I can
convince you that extended attributes ARE just a building block
it shouldn't matter how they are implemented as long as the end result
works properly.

And as for implementation -- David's proposal seems to fit the bill
perfectly.

Thanks,
Roman.

Re: [9fans] Re: everything is a directory

[roger peppe]

> Or how about this:  Just as you don't need read permissions on a
> directory to walk it -- so long as you know the name of the file you
> want -- how about 9p allowing walks to ordinary files.

you could do this (macos x does something very similar), but it would
still have exactly the same problems that most of these schemes have;
i.e. the metadata will be lost unless all tools in the path from source
to destination are aware of its existence.