[TUHS] Re: Quick Question: Early Filesystems and Name/Metadata Separation?

[TUHS] Re: Quick Question: Early Filesystems and Name/Metadata Separation?

[Noel Chiappa via TUHS]

    > From: Warren Toomey

    > Was Unix the first to separate a file's name from the other file metadata

UNIX was the first filesystem I know of where file metadata was not kept in
the directory. So, yes.

	Noel

[TUHS] Re: Quick Question: Early Filesystems and Name/Metadata Separation?

[Clem Cole via TUHS]

below...

On Mon, Sep 15, 2025 at 8:10 PM Warren Toomey via TUHS <tuhs@tuhs.org>
wrote:

> Hi all, a quick question. Was Unix the first to separate
> a file's name from the other file metadata (thus allowing
> hard links where no filename is "better" than the others)?
>
I'm not 100% I'm following you.  As far as I know, a significant innovation
in UNIC from its contemporaries, when Ken and Dennis were first considering
file systems, was the realization that humans and computers have distinct needs
for the file store.   So Ken created a first-level (lower)  file system
that mapped well to the computer itself, which is 100% flat. A numerical
index identifies files in that linear list or "map" that is common to all
files.  Furthermore, the metadata associated with each file is stored in
that single, internal list.   But that low-level file system is never
exposed to anything but the OS kernel.  A human-oriented file system is
layered on top, where everything is a stream of bytes, and files are given
human-understandable names that are associated with the actual file in the
low-level store.   The OS then defines a structure for some of those files
and gives them types (stored in the metadata).

What changed over time with UNIC to UNIX was how the upper-level file
system was exposed to the user programs.  The current
hierarchical structure emerged after trying a few others, which was the
beauty of the design.  What the kernbel uses is the "i-number," while
humans use a path.

This two-level store was unique as far as I know, so I would believe that
it was indeed first

[TUHS] Re: Quick Question: Early Filesystems and Name/Metadata Separation?

[Dave Horsfall via TUHS]

On Mon, 15 Sep 2025, Noel Chiappa via TUHS wrote:

> UNIX was the first filesystem I know of where file metadata was not kept 
> in the directory. So, yes.

I still recall being surprised in the early 70s upon learning that the 
filename was not part of the file (I came from OS/360 and RSX-11); I 
thought it was odd, then realised the sheer genius of it.

-- Dave

[TUHS] Re: Quick Question: Early Filesystems and Name/Metadata Separation?

[Dan Cross via TUHS]

On Mon, Sep 15, 2025 at 8:19 PM Warren Toomey via TUHS <tuhs@tuhs.org> wrote:
> Hi all, a quick question. Was Unix the first to separate
> a file's name from the other file metadata (thus allowing
> hard links where no filename is "better" than the others)?

I wondered if perhaps Multics had the concept, but it appears that it
does not, if I am ready section 2.1 of
https://multicians.org/fjcc4.html correctly.  It does have symbolic
links, but those seem to be of the conventional variety we are
familiar with from Unix, and the names associated with a link are
distinguished from the "primary name", which is special.  Multics
directory entries of the "branch" variety carry a file's metadata in
addition to the primary name; entries of the "link" variety point to
other entries.  Note that, unlike Unix, where inodes contain physical
disk addresses for the file's data (and indirect/doubly-indirect
blocks), Multics directory entries include a VTOC index,

However, it appears that the Berkeley Timesharing System for the
SDS-930 may have had something like hard links.  Section 12.9 of
https://bitsavers.org/pdf/sds/9xx/940/ucbProjectGenie/R-21_Time-Sharing_System_Reference_Oct68.pdf
describes the directory structure for that system, and notes that what
we'd call a directory entry is a 3-word record in a hash table
(presumably indexed by file name), with the first two words containing
"string pointers to the file name" and the 4th word containing "a
pointer to a 4-word 'description block'".  The "description block" is
described as something containing metadata and sounds an awful lot
like an inode.  That document goes on to say, "notice that several
entries in the hash table may point to a single description block; the
associated names are then synonyms for the same object, which can be
referenced by any of them."

The next paragraph continues: "The command DEFINE NAME creates a new
name to point to an existing description block; conversely DELETE NAME
detaches the name from its description block, the description block
itself is lost only if this was the only name pointing to it."

So the Project GENIE OS at least separated file names and metadata,
though it appears that each user had a single directory level.
Regardless, I'd say that meets the requested criteria.  I wonder what
DTSS did in this area, if anything.

        - Dan C.

[TUHS] Re: Quick Question: Early Filesystems and Name/Metadata Separation?

[Tom Perrine via TUHS]

I think that was also true of Multics.


On Mon, Sep 15, 2025 at 5:30 PM Noel Chiappa via TUHS <tuhs@tuhs.org> wrote:

>     > From: Warren Toomey
>
>     > Was Unix the first to separate a file's name from the other file
> metadata
>
> UNIX was the first filesystem I know of where file metadata was not kept in
> the directory. So, yes.
>
>         Noel
>

[TUHS] Re: Quick Question: Early Filesystems and Name/Metadata Separation?

[Bakul Shah via TUHS]

In case anyone is further interested in Strachey & Stoy's OS6....

https://www.cs.ox.ac.uk/files/3230/PRG08.pdf

Excerpt:
A single file may be associated with different pairs of
names in different indexes, or even in the same index. This is
one form of sharing of files: two different entries point to the
same file.

The file here is equivalent to an inode. "Pairs of name" can be file
name and file type but the system doesn't care which meaning a
user attaches. There is no link count and the system has to GC.
There is even a link command!

Theoretically one can construct a directory tree (an index table).
The paper is an interesting & easy read! I wonder how much of
the OS6 & Unix designs were influenced by the respective culture's
sensibilities (British and American as well as University vs Bell Labs)!

Bakul

> On Sep 15, 2025, at 7:08 PM, Douglas McIlroy via TUHS <tuhs@tuhs.org> wrote:
> 
> Strachey and Stoy independently invented the same idea at essentially
> the same time.
> Their operating system, OS1, went into operation before Unix but
> didn't get a file system until OS4, in April 1970. So Unix beat it by
> six calendar months, but neither influenced the other.
> 
> OS6 was published in The Computer Journal in 1972, ahead of Unix.
> Source with commentary a la Lions was available. Yet Unix got the
> world's attention. Unix was compiled to machine  language for the
> PDP11, while OS6 was compiled to interpreted code for the Modular One
> made by Control Technologies Limited. Unix had time-sharing, while OS6
> was single user. And finally Unix was American, while OS6 was British.
> 
> Later there was a similarly accidental "race" to port the two
> operating systems. I'm not sure whether Wollongong or Oxford won.
> 
> Doug

[TUHS] Re: Quick Question: Early Filesystems and Name/Metadata Separation?

[George Michaelson via TUHS]

Is the return of a stat() call from the directory block not file "metadata"?

G

On Tue, 16 Sept 2025, 2:21 pm Tom Perrine via TUHS, <tuhs@tuhs.org> wrote:

> I think that was also true of Multics.
>
>
> On Mon, Sep 15, 2025 at 5:30 PM Noel Chiappa via TUHS <tuhs@tuhs.org>
> wrote:
>
> >     > From: Warren Toomey
> >
> >     > Was Unix the first to separate a file's name from the other file
> > metadata
> >
> > UNIX was the first filesystem I know of where file metadata was not kept
> in
> > the directory. So, yes.
> >
> >         Noel
> >
>

[TUHS] Re: Quick Question: Early Filesystems and Name/Metadata Separation?

[Noel Chiappa via TUHS]

    > From: Dan Cross

    > I wondered if perhaps Multics had the concept

You have no idea of the depth of the tarpit you just stepped into! Making it
even deeper, Multics has two quite different generations of file systems;
read about the second one (reasons why, etc) here:

   https://www.multicians.org/nss.html

File metadata was kept in the directory in the first; in the second, much
metadata _was_ moved to a thing called a 'Volume Table of Contents' (VTOC),
which is sort of like the ilist (the above says "[w]e were aware of the Unix
file system design, where there was a similar split between naming data in
directories and physical data in the inode"). However, even in the New
Storage System, _some_ metadata (ACLs, etc) remained in the directories, see:

  https://www.multicians.org/mtbs/MTB-220.pdf

Just to increase the complexity, in both systems, in addition to the
segment's primary name (sort of a 'hard link'), and logical ('soft') links,
segments could have additional secondary names (much used for short names for
commands - the familiar ls, cp, mv, etc were all short names), kept in the
same directory as the primary name:

  https://multicians.org/mga.html#additionalnames

(Please excuse Bernie.) Are we having fun yet? :-)


    > it appears that the Berkeley Timesharing System for the SDS-930 may
    > have had something like hard links

I was wondering if BTSS (on which, of course, Ken worked) had such, but was
too lazy to look. Thanks!


    > From: George Michaelson

    > Is the return of a stat() call from the directory block not file
    > "metadata"?

stat() mostly return information from the inode.

	Noel

[TUHS] Re: Quick Question: Early Filesystems and Name/Metadata Separation?

[George Michaelson via TUHS]

stat() mostly return information from the inode

Which is not the contents of the file. It's data about the file. It's
attributes, such as creation and modification times, its owner and group,
pointers to extended attributes.

Its... metadata.

G

[TUHS] Re: Quick Question: Early Filesystems and Name/Metadata Separation?

[Noel Chiappa via TUHS]

    > From: George Michaelson

    > Which is not the contents of the file. It's data about the file.
    > It's attributes, such as creation and modification times, its owner
    > and group, pointers to extended attributes.
    > Its... metadata.

You said:

    >>> Is the return of a stat() call from the directory block not file
    >>> "metadata"?

I replied:

    >> stat() mostly return information from the inode.

Note the "from the directory block".

	Noel
!h

[TUHS] Re: Quick Question: Early Filesystems and Name/Metadata Separation?

[Dan Cross via TUHS]

On Tue, Sep 16, 2025 at 2:15 AM George Michaelson via TUHS
<tuhs@tuhs.org> wrote:
> Is the return of a stat() call from the directory block not file "metadata"?

As Noel pointed out, there's no real metadata in the directory entry
itself.  The directory entry is really just a pair, consisting of a
string (the path name component in the entry) and an inode number.
Critically, the inode is stored separately, and holds all of the
metadata describing a file.  So while `stat()` may take a file name as
an argument, internally the kernel uses that to look up the associated
inode and populates the structure it returns from _that_, not from the
directory entry.

But as usual, it's slightly more complicated than that: on the PDP-11
Unix directory entries were fixed size: 16 bytes; 2 bytes for the
inode number (interpreted as an unsigned 16-bit integer) and 14 for
the file name part.  Working with these is incredibly simple.  But
Berkeley expanded this a bit with UFS, allowing 4 bytes (interpreted
as an unsigned 32-bit integer) for the inode number and up to 255
characters for the file name.  But since most file names were
well-short of 255 characters, storing them as such was wasteful, so
they added two other fields: a record length and a name length.  This
enabled them to compress entries down to a minimal record size,
aligned to a 4-byte boundary, and thus pack entries together rather
more tightly than if the entire 256 bytes (255 + 1 for the terminating
NUL byte) were stored directly.  Lookups---the most common
operation---remained simple: to retrieve the "next" entry in a
directory, one need only add the offset from the current entry to that
entry's position relative to the start of the directory file, but that
was an extra step compared to what workaday programmers were used to
doing (which was just opening the directory file read-only and, well,
reading from it), so the `opendir`/`readdir`/`closedir` interfaces
were added to hide this in the kernel and mimic the more traditional
open/read/close loop.

Unfortunately, adding and removing entries became significantly more
complex than in earlier file system versions.  Regardless, the
additional data added to directory entries are really more details of
the implementation of directories themselves, and not metadata about
the file an entry points to.

        - Dan C.

> On Tue, 16 Sept 2025, 2:21 pm Tom Perrine via TUHS, <tuhs@tuhs.org> wrote:
> > I think that was also true of Multics.
> >
> >
> > On Mon, Sep 15, 2025 at 5:30 PM Noel Chiappa via TUHS <tuhs@tuhs.org>
> > wrote:
> >
> > >     > From: Warren Toomey
> > >
> > >     > Was Unix the first to separate a file's name from the other file
> > > metadata
> > >
> > > UNIX was the first filesystem I know of where file metadata was not kept
> > in
> > > the directory. So, yes.
> > >
> > >         Noel
> > >
> >

[TUHS] Re: Quick Question: Early Filesystems and Name/Metadata Separation?

[Paul McJones via TUHS]

> On 15 Sep 2025 21:40:06 -0700,Bakul Shah <bakul@iitbombay.org> wrote:
> 
> In case anyone is further interested in Strachey & Stoy's OS6....
> 
> https://www.cs.ox.ac.uk/files/3230/PRG08.pdf
> 
> Excerpt:
> A single file may be associated with different pairs of
> names in different indexes, or even in the same index. This is
> one form of sharing of files: two different entries point to the
> same file.

Perhaps also of interest:

Christopher Strachey and Joseph Stoy. The text of OS6Pub.
PRG-09, Programming Research Group, Oxford University Computer Laboratory, July 1972.
http://www.cs.ox.ac.uk/files/3231/PRG09.pdf

Christopher Strachey and Joseph Stoy. The text of OS6Pub (Commentary).
PRG-09 (c), Programming Research Group, Oxford University Computer Laboratory, July 1972.
https://www.cs.ox.ac.uk/files/4632/PRG-9%28c%29.pdf 

[TUHS] Re: Quick Question: Early Filesystems and Name/Metadata Separation?

[Dan Cross via TUHS]

On Tue, Sep 16, 2025 at 4:44 AM Noel Chiappa via TUHS <tuhs@tuhs.org> wrote:
>     > From: Dan Cross
>
>     > I wondered if perhaps Multics had the concept
>
> You have no idea of the depth of the tarpit you just stepped into! Making it
> even deeper, Multics has two quite different generations of file systems;
> read about the second one (reasons why, etc) here:
>
>    https://www.multicians.org/nss.html
>
> File metadata was kept in the directory in the first; in the second, much
> metadata _was_ moved to a thing called a 'Volume Table of Contents' (VTOC),
> which is sort of like the ilist (the above says "[w]e were aware of the Unix
> file system design, where there was a similar split between naming data in
> directories and physical data in the inode"). However, even in the New
> Storage System, _some_ metadata (ACLs, etc) remained in the directories, see:
>
>   https://www.multicians.org/mtbs/MTB-220.pdf
>
> Just to increase the complexity, in both systems, in addition to the
> segment's primary name (sort of a 'hard link'), and logical ('soft') links,
> segments could have additional secondary names (much used for short names for
> commands - the familiar ls, cp, mv, etc were all short names), kept in the
> same directory as the primary name:
>
>   https://multicians.org/mga.html#additionalnames
>
> (Please excuse Bernie.) Are we having fun yet? :-)

Ooops, yup: I had looked at the VTOC code in Multics, but sent before
I finished editing my reply to mention it beyond a half-finished
sentence; mea culpa, there!  FWIW from what I saw, the VTOC entry
contains physical disk addresses for files, and is indexed by a "VTOCE
index", which is an integer that is stored in branch directory
entries.  However, most of the metadata about the file, such as
timestamps, ACL pointers, the primary name, size, etc remains in
branch entries.

For those interested in the gory details, the definition of a Multics
directory entry, as of MR12.8 is online:
https://dps8m.gitlab.io/sb/MR12.8/library_dir_dir/include/dir_entry.incl.pl1.html

(No problem about Bernie; he's a character.)

>     > it appears that the Berkeley Timesharing System for the SDS-930 may
>     > have had something like hard links
>
> I was wondering if BTSS (on which, of course, Ken worked) had such, but was
> too lazy to look. Thanks!

Sure thing!

        - Dan C.

[TUHS] Re: Quick Question: Early Filesystems and Name/Metadata Separation?

[Leah Neukirchen via TUHS]

Dan Cross via TUHS <tuhs@tuhs.org> writes:

> Lookups---the most common
> operation---remained simple: to retrieve the "next" entry in a
> directory, one need only add the offset from the current entry to that
> entry's position relative to the start of the directory file, but that
> was an extra step compared to what workaday programmers were used to
> doing (which was just opening the directory file read-only and, well,
> reading from it), so the `opendir`/`readdir`/`closedir` interfaces
> were added to hide this in the kernel and mimic the more traditional
> open/read/close loop.

And then we had to wait until POSIX.1-2024 until the "old" interface
became standardized (which is useful because opendir doesn't let you
specify buffer size).

https://pubs.opengroup.org/onlinepubs/9799919799/functions/posix_getdents.html

-- 
Leah Neukirchen  <leah@vuxu.org>  https://leahneukirchen.org/

[TUHS] Re: Quick Question: Early Filesystems and Name/Metadata Separation?

[Steffen Nurpmeso via TUHS]

Leah Neukirchen via TUHS wrote in
 <87y0qdfku1.fsf@vuxu.org>:
 |Dan Cross via TUHS <tuhs@tuhs.org> writes:
 |
 |> Lookups---the most common
 |> operation---remained simple: to retrieve the "next" entry in a
 |> directory, one need only add the offset from the current entry to that
 |> entry's position relative to the start of the directory file, but that
 |> was an extra step compared to what workaday programmers were used to
 |> doing (which was just opening the directory file read-only and, well,
 |> reading from it), so the `opendir`/`readdir`/`closedir` interfaces
 |> were added to hide this in the kernel and mimic the more traditional
 |> open/read/close loop.
 |
 |And then we had to wait until POSIX.1-2024 until the "old" interface
 |became standardized (which is useful because opendir doesn't let you
 |specify buffer size).

And with the (yet only "encouraged") DT_FORCE_TYPE flag for

 |https://pubs.opengroup.org/onlinepubs/9799919799/functions/posix_getdent\
 |s.html

a painful, racy / unsafe programming could finally iterate out.
Just last week, due to a thread on a FreeBSD list i read, i
looked into an implementation that is the quarter of a century
old; it contains several (preprocessor conditionalized) code
paths, dependend upon availability of certain *at() series
functions (not to talk about ENOSYS handling, which is done
badly), and it was a pain, and very unsafe.  No, it effectively
is broken to create absolute (real)paths and/or chdir()
(multi-threading was a thing, that is locked..) for querying
file statuses.
Hoping for this.  (And posix_devctl() instead of ioctl(), though
i think .. the Unix world becomes somewhat easy, anyhow.)

Plan9port that is talked about here at the moment went into
the other direction five years ago, and dropped all the getdents /
getdirentries (maybe) shims to end up only with opendir.
'Thing was also that opendir() stuff performs allocations, and
thus sucks in the entire memory cache layer, and that sucks in
threading stuff ...  All that does no longer matter with all the
init, fini etc sections, ifunc's, TLS (ThreadSpecificData), of
modern times.  Of course.

P.S.: i would at least remove the defunct header fields if you do
not want to fix it, Warren.  Isn't that nothing more than a sed(1)
invocation?  It is so .. an ugly alien spot, and it cannot even be
healed with tea tree oil.  (Sigh.)

--steffen
|
|Der Kragenbaer,                The moon bear,
|der holt sich munter           he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)