The Unix Heritage Society mailing list
 help / color / mirror / Atom feed
* Re: [TUHS] moving directories in svr2
@ 2021-12-29 19:13 Douglas McIlroy
  2021-12-29 19:37 ` Dan Cross
  0 siblings, 1 reply; 71+ messages in thread
From: Douglas McIlroy @ 2021-12-29 19:13 UTC (permalink / raw)
  To: TUHS main list

> what was the last Unix version
> that let users make arbitrary links, such that the file system was no
> longer a DAG? I recall in v6 days hearing that earlier Unix allowed
> this, and that cleanup (via icheck and friends) got to be near
> impossible.

From v1 on (I'm not sure about PDP-7 Unix) only the superuser could do
that, so what you heard strikes me as urban legend. Perhaps some
installations abused root privilege to scramble the file system, but I
certainly didn't see that happen in Research systems.

Doug

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2021-12-29 19:13 [TUHS] moving directories in svr2 Douglas McIlroy
@ 2021-12-29 19:37 ` Dan Cross
  2021-12-29 20:15   ` Dan Cross
  0 siblings, 1 reply; 71+ messages in thread
From: Dan Cross @ 2021-12-29 19:37 UTC (permalink / raw)
  To: Douglas McIlroy; +Cc: TUHS main list

On Wed, Dec 29, 2021 at 2:14 PM Douglas McIlroy
<douglas.mcilroy@dartmouth.edu> wrote:
> > what was the last Unix version
> > that let users make arbitrary links, such that the file system was no
> > longer a DAG? I recall in v6 days hearing that earlier Unix allowed
> > this, and that cleanup (via icheck and friends) got to be near
> > impossible.
>
> From v1 on (I'm not sure about PDP-7 Unix) only the superuser could do
> that, so what you heard strikes me as urban legend. Perhaps some
> installations abused root privilege to scramble the file system, but I
> certainly didn't see that happen in Research systems.

I recall doing this on some system via a lesser-known flag to `ln`.
But I believe that a) you had to be root, and b) fsck sure was unhappy
after the fact. I gather it was one of those, "wizard sysadmin invokes
the deep magic from single user mode to fix the filesystem" and like
other such arcana was mostly for dealing with seriously perverse
failure modes, possibly in concert with the `unlink` command.

https://www.freebsd.org/cgi/man.cgi?query=ln&apropos=0&sektion=0&manpath=SunOS+4.1.3&arch=default&format=html
says that the `-f` flag to `ln` would force a hard link to a directory
on (at least) SunOS 4.1.3.

The 4.3BSD man page has this gem: "Hard links may not refer to
directories (unless the proper incantations are supplied) and may not
span file systems." And indeed, a quick look at /usr/src/bin/ln.c on a
VAX running 4.3 shows that it has an undocumented `-f` flag:
presumably the "proper incantation" mentioned in the man page.

        - Dan C.

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2021-12-29 19:37 ` Dan Cross
@ 2021-12-29 20:15   ` Dan Cross
  2021-12-29 20:42     ` Richard Salz
  0 siblings, 1 reply; 71+ messages in thread
From: Dan Cross @ 2021-12-29 20:15 UTC (permalink / raw)
  To: Douglas McIlroy; +Cc: TUHS main list

On Wed, Dec 29, 2021 at 2:37 PM Dan Cross <crossd@gmail.com> wrote:
> The 4.3BSD man page has this gem: "Hard links may not refer to
> directories (unless the proper incantations are supplied) and may not
> span file systems." And indeed, a quick look at /usr/src/bin/ln.c on a
> VAX running 4.3 shows that it has an undocumented `-f` flag:
> presumably the "proper incantation" mentioned in the man page.

On a lark, I decided to check for this on 7th Edition as well. The man
page says, quite clearly: "It is forbidden to link to a directory or
to link across file systems." However, /usr/src/cmd/ln.c has the
undocumented `-f` flag, and if given, will elide checking for whether
the "name1" argument is a directory or not. Looking at the kernel, as
Doug mentioned, one must be the superuser to link a directory.

        - Dan C.

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2021-12-29 20:15   ` Dan Cross
@ 2021-12-29 20:42     ` Richard Salz
  2021-12-29 20:58       ` Dan Cross
  2021-12-30  3:15       ` Bakul Shah
  0 siblings, 2 replies; 71+ messages in thread
From: Richard Salz @ 2021-12-29 20:42 UTC (permalink / raw)
  To: Dan Cross; +Cc: TUHS main list, Douglas McIlroy

[-- Attachment #1: Type: text/plain, Size: 988 bytes --]

https://dsf.berkeley.edu/cs262/unix.pdf section 3.2 ends with:

Each directory always has at least two entries. The
name "." in each directory refers to the directory itself. Thus a
program may read the current directory under the name “.”
without knowing its complete path name. The name “..” by
convention refers to the parent of the directory in which it
appears, that is, to the directory in which it was created.
The directory structure is constrained to have the form
of a rooted tree. Except for the special entries “.” and “..”,
each directory must appear as an entry in exactly one other,
which is its parent. The reason for this is to simplify the
writing of programs which visit subtrees of the directory
structure, and more important, to avoid the separation of
portions of the hierarchy. If arbitrary links to directories
were permitted, it would be quite difficult to detect when
the last connection from the root to a directory was severed

[-- Attachment #2: Type: text/html, Size: 2749 bytes --]

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2021-12-29 20:42     ` Richard Salz
@ 2021-12-29 20:58       ` Dan Cross
  2021-12-29 21:20         ` Clem Cole
  2021-12-30  3:15       ` Bakul Shah
  1 sibling, 1 reply; 71+ messages in thread
From: Dan Cross @ 2021-12-29 20:58 UTC (permalink / raw)
  To: Richard Salz; +Cc: TUHS main list, Douglas McIlroy

Partitioning the tree sure seems like it would lead to a bad day, doesn't it?

I expect the undocumented arcana related to linking directories was
always an escape hatch for a skilled administrator to rescue a sick
system. One can imagine any number of ways in which directory files
could be corrupted; sufficient inspection of disk contents might give
someone with the requisite low-level knowledge sufficient information
that they could, say, reattach a pruned subtree with clever uses of
`ln -f` and `unlink`. Did `icheck` and friends have something
analogous to the `lost+found` directory created by `fsck`? I could
imagine (somewhat painfully) executing those commands while in
single-user mode on the console.

        - Dan C.

On Wed, Dec 29, 2021 at 3:42 PM Richard Salz <rich.salz@gmail.com> wrote:
>
> https://dsf.berkeley.edu/cs262/unix.pdf section 3.2 ends with:
>
> Each directory always has at least two entries. The
> name "." in each directory refers to the directory itself. Thus a
> program may read the current directory under the name “.”
> without knowing its complete path name. The name “..” by
> convention refers to the parent of the directory in which it
> appears, that is, to the directory in which it was created.
> The directory structure is constrained to have the form
> of a rooted tree. Except for the special entries “.” and “..”,
> each directory must appear as an entry in exactly one other,
> which is its parent. The reason for this is to simplify the
> writing of programs which visit subtrees of the directory
> structure, and more important, to avoid the separation of
> portions of the hierarchy. If arbitrary links to directories
> were permitted, it would be quite difficult to detect when
> the last connection from the root to a directory was severed

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2021-12-29 20:58       ` Dan Cross
@ 2021-12-29 21:20         ` Clem Cole
  0 siblings, 0 replies; 71+ messages in thread
From: Clem Cole @ 2021-12-29 21:20 UTC (permalink / raw)
  To: Dan Cross; +Cc: TUHS main list, Douglas McIlroy

[-- Attachment #1: Type: text/plain, Size: 1412 bytes --]

On Wed, Dec 29, 2021 at 3:59 PM Dan Cross <crossd@gmail.com> wrote:

> Did `icheck` and friends have something analogous to the `lost+found`
> directory created by `fsck`?
>
The lost+found directory was a tjk-ism although I can get a very small
level of credit/inspiration.  It was born of need.   Ted was explaining to
me one night that one of the issues when you had identified a file that no
directory was claiming was what to do with it.  Originally, Ted linked lost
files into root because there was usually only a couple of files, but ...
we startiong seeing a situation where often the root directory did not have
free directory slots.

Some of the folks (I want to say it was either bio-med or architecture
dept) had a recurring crash that occurred after a bunch of intermediate
files had been created on a long running program.  If the program stopped
we had a way of restarting it part way using them.   The problem was that
people did not clean up the /tmp directory so the program unlinked them
after creating them.  But if it crashed, the files were still there but
just connected to anything.   Ted could find them with f*ck [it had a
different name then].

I remember that we talked more about the issue and he came up with the idea
of creating the directory lost+found and then preallocating a bunch of
slots with a shell script instead of hoping the root directory was going to
being available.

[-- Attachment #2: Type: text/html, Size: 2337 bytes --]

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2021-12-29 20:42     ` Richard Salz
  2021-12-29 20:58       ` Dan Cross
@ 2021-12-30  3:15       ` Bakul Shah
  1 sibling, 0 replies; 71+ messages in thread
From: Bakul Shah @ 2021-12-30  3:15 UTC (permalink / raw)
  To: TUHS main list



> On Dec 29, 2021, at 12:42 PM, Richard Salz <rich.salz@gmail.com> wrote:
> 
> https://dsf.berkeley.edu/cs262/unix.pdf section 3.2 ends with:
> 
> Each directory always has at least two entries. The
> name "." in each directory refers to the directory itself. Thus a
> program may read the current directory under the name “.”
> without knowing its complete path name. The name “..” by
> convention refers to the parent of the directory in which it
> appears, that is, to the directory in which it was created.
> The directory structure is constrained to have the form
> of a rooted tree. Except for the special entries “.” and “..”,

Note that "." or ".." entries are not needed if you store the
entire path of the current working dir. in a process'es kernel
state. My guess is *not* storing a path instead of a ptr to
the inode was done to save on memory.

> each directory must appear as an entry in exactly one other,
> which is its parent. The reason for this is to simplify the
> writing of programs which visit subtrees of the directory
> structure, and more important, to avoid the separation of
> portions of the hierarchy. If arbitrary links to directories
> were permitted, it would be quite difficult to detect when
> the last connection from the root to a directory was severed


Every inode has a linkcount so detecting when the last conn.
is severed not a problem. The separation *can* occur in any
case if the system crashes at the wrong time and you can
find #inode-number named directories in /lost+found.

plan9 folks finessed the problem by not storing the linkcount.
This is why moving a tree is a copy and then rm -rf of the old
location. You win some, you lose some.

Clem wrote:
> Late binding to names (like symlinks) are a dynamic (runtime idea).  Bakul points out that by using the per process u area, the dynamic context can be retained.  The observation is that .. (like symlinks) tend to be a runtime (dynamic) notion, although I'm not sure how you keep consistency in the static FS if you don't store the link in the inode.

Symlinks are not a runtime idea but $PWD is strictly a runtime
thing. With symlinks perhaps 4.xBSD took the easy way out by
decreeing that only 8(?) symlink may be traversed at runtime
in interpreting a path for open etc, perhaps to save space but
detecting loops can be done by storing <fsid,inum> per component
and checking that. The same algorithm would help if multiple dir.
hard links were allowed. Sure it has O(n) behavior but the user
pays for it, or you can put a liberal limit on this array.

I don't have much experience with crash recovery on plan9 so
don't know if not having linkcounts is a problem. I haven't
thought through about whether there are crash or concurrency
related issues with allowin arbitray graphs by dir linking.
But probably fixable by writing an intent log ala DBMS.

> At the risk of kicking a dead horse too much 

I think it helps to once in a while wonder about why/how some
system decisions were made and learn something from that; or
gain a deeper understanding. At least that is my excuse!

Thanks for all the responses. My take away is that while it
is possible to allow arbitrary dir links, it would increase
implementation complexity while the added benefit of it are
few if any.

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2022-01-03 22:26                           ` Theodore Ts'o
  2022-01-03 23:10                             ` Dan Cross
  2022-01-04 15:45                             ` Chet Ramey
@ 2022-01-09 19:28                             ` Larry McVoy
  2 siblings, 0 replies; 71+ messages in thread
From: Larry McVoy @ 2022-01-09 19:28 UTC (permalink / raw)
  To: Theodore Ts'o; +Cc: The Eunuchs Hysterical Society

On Mon, Jan 03, 2022 at 05:26:33PM -0500, Theodore Ts'o wrote:
> Larry has told a similar story about the advantages of using a "dead"
> language like Tcl for Bitkeeper, since he didn't have to deal with
> gratuitous version skew problems caused by backwards incomaptible
> changes in Perl or Python.  

Well, we also made very substantial changes to the Tcl source tree,
resulting in http://little-lang.org which tosses Tcl's tortured 
"can't decide if I'm lisp or not" syntax in favor of something 
that looks like a super pleasant superset of C.

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2022-01-03 22:26                           ` Theodore Ts'o
  2022-01-03 23:10                             ` Dan Cross
@ 2022-01-04 15:45                             ` Chet Ramey
  2022-01-09 19:28                             ` Larry McVoy
  2 siblings, 0 replies; 71+ messages in thread
From: Chet Ramey @ 2022-01-04 15:45 UTC (permalink / raw)
  To: Theodore Ts'o, Dan Cross; +Cc: The Eunuchs Hysterical Society

On 1/3/22 5:26 PM, Theodore Ts'o wrote:

> you can just choose *not* to upgrade to newer versions of bash or
> network utilities.  Who *says* you have to use bash 5.1?  Or switch to
> ip/ss?

It's always a trade-off, isn't it? If you don't use the current version of
something, you assume the burden of maintenance. As we've seen, vendors
vary as to the attention they give old versions of software (e.g., bash-4.2
still shipping as part of RHEL 7 a decade after bash-4.3 was released).
Google can make that choice for itself, I suppose.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
		 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2022-01-03 23:37                             ` Adam Thornton
@ 2022-01-04 14:49                               ` Stuart Remphrey
  0 siblings, 0 replies; 71+ messages in thread
From: Stuart Remphrey @ 2022-01-04 14:49 UTC (permalink / raw)
  To: Adam Thornton; +Cc: The Eunuchs Hysterical Society

[-- Attachment #1: Type: text/plain, Size: 1406 bytes --]

Oh, that reminded me of SunOS 4.0 that introduced streams. Broke
So I used to pine for 3.5!

Agree re 2.4/2.5.1 having gotten an O/S back to being solid and very usable
-- IIRC we had customers query whether they had good distribution media,
because the "patch" directory was empty. No patches were introduced during
beta.


On Tue, 4 Jan 2022, 07:38 Adam Thornton, <athornton@gmail.com> wrote:

> 2.5.1 was the first Solaris that didn't often make me scream that I wanted
> SunOS 4.1 back.
>
> Adam
>
>
> On Mon, Jan 3, 2022 at 4:32 PM Doug McIntyre <merlyn@geeks.org> wrote:
>
>> On Mon, Jan 03, 2022 at 04:15:08PM -0500, Dan Cross wrote:
>> > On Mon, Jan 3, 2022 at 3:23 PM Theodore Ts'o <tytso@mit.edu> wrote:
>>
>> > > Yeah, to be fair, by the time Solaris 2.3 or 2.4 came around, it was
>> > > mostly up to par.  (Or maybe it was because Moore's law meant that we
>> > > didn't care any more.  :-)
>> >
>> > I have some vague memories that we had to do something like double the
>> > RAM in our SPARCstations to make Solaris 2 feel comfortable. At the
>> > time, that was a pretty serious outlay in an academic department.
>> > 2.5.1 felt like the first version that was _truly_ usable.
>>
>> I'd agree, 2.4 was pretty slow and chunky, 2.5 was alright, but 2.5.1 was
>> quite usable and stable.
>> Also by this time, the hardware was going in directions that SunOS
>> wouldn't keep up with.
>>
>>
>>

[-- Attachment #2: Type: text/html, Size: 2274 bytes --]

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2022-01-04  2:28                               ` Theodore Ts'o
@ 2022-01-04  2:42                                 ` Larry McVoy
  0 siblings, 0 replies; 71+ messages in thread
From: Larry McVoy @ 2022-01-04  2:42 UTC (permalink / raw)
  To: Theodore Ts'o; +Cc: The Eunuchs Hysterical Society

On Mon, Jan 03, 2022 at 09:28:02PM -0500, Theodore Ts'o wrote:
> On Mon, Jan 03, 2022 at 03:44:11PM -0800, Larry McVoy wrote:
> > On Mon, Jan 03, 2022 at 05:21:51PM -0600, Doug McIntyre wrote:
> > > I'd agree, 2.4 was pretty slow and chunky, 2.5 was alright, but 2.5.1 was quite usable and stable.
> > > Also by this time, the hardware was going in directions that SunOS wouldn't keep up with.
> > 
> > Yeah, Doug is right, SunOS was pretty simple, it didn't really take advantage
> > of SMP, Greg Limes tried to thread it but it was too big a job for one guy.
> > 
> > That's not to say that SunOS couldn't have evolved into SMP, I'm 100%
> > sure it could have.  It just didn't.  It's a shame.
> 
> I would have thought that if we consider the amount of engineering
> resources to that was invested to get from Solaris 2.0 to 2.5.1, if
> that had instead been invested into making SunOS a scalable SMP OS,
> that it would have been doable.  Do you think that's fair?

Yes, absolutely.  I worked in the kernel group in building 5.  That was
pretty much the best group of people I have been a part of.  That team
could have done anything, SMP was not a problem for them.  They for sure
could have made SunOS scale.  The people who didn't leave were the people
who made Solaris work.

That said, I have to give credit to the group of people I built who
did BitKeeper.  I'd stand them up against the Sun kernel team and they
would win.  Smaller group, stellar bunch of people.

I'm happy that I got to be part of both teams.

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2022-01-03 23:44                             ` Larry McVoy
  2022-01-03 23:56                               ` Warner Losh
@ 2022-01-04  2:28                               ` Theodore Ts'o
  2022-01-04  2:42                                 ` Larry McVoy
  1 sibling, 1 reply; 71+ messages in thread
From: Theodore Ts'o @ 2022-01-04  2:28 UTC (permalink / raw)
  To: Larry McVoy; +Cc: The Eunuchs Hysterical Society

On Mon, Jan 03, 2022 at 03:44:11PM -0800, Larry McVoy wrote:
> On Mon, Jan 03, 2022 at 05:21:51PM -0600, Doug McIntyre wrote:
> > I'd agree, 2.4 was pretty slow and chunky, 2.5 was alright, but 2.5.1 was quite usable and stable.
> > Also by this time, the hardware was going in directions that SunOS wouldn't keep up with.
> 
> Yeah, Doug is right, SunOS was pretty simple, it didn't really take advantage
> of SMP, Greg Limes tried to thread it but it was too big a job for one guy.
> 
> That's not to say that SunOS couldn't have evolved into SMP, I'm 100%
> sure it could have.  It just didn't.  It's a shame.

I would have thought that if we consider the amount of engineering
resources to that was invested to get from Solaris 2.0 to 2.5.1, if
that had instead been invested into making SunOS a scalable SMP OS,
that it would have been doable.  Do you think that's fair?

Of course, that's assuming that Sun could have stayed afloat without
that injection of cash from AT&T....

						- Ted

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2022-01-03 23:44                             ` Larry McVoy
@ 2022-01-03 23:56                               ` Warner Losh
  2022-01-04  2:28                               ` Theodore Ts'o
  1 sibling, 0 replies; 71+ messages in thread
From: Warner Losh @ 2022-01-03 23:56 UTC (permalink / raw)
  To: Larry McVoy; +Cc: The Eunuchs Hysterical Society

[-- Attachment #1: Type: text/plain, Size: 1513 bytes --]

On Mon, Jan 3, 2022, 4:44 PM Larry McVoy <lm@mcvoy.com> wrote:

> On Mon, Jan 03, 2022 at 05:21:51PM -0600, Doug McIntyre wrote:
> > On Mon, Jan 03, 2022 at 04:15:08PM -0500, Dan Cross wrote:
> > > On Mon, Jan 3, 2022 at 3:23 PM Theodore Ts'o <tytso@mit.edu> wrote:
> >
> > > > Yeah, to be fair, by the time Solaris 2.3 or 2.4 came around, it was
> > > > mostly up to par.  (Or maybe it was because Moore's law meant that we
> > > > didn't care any more.  :-)
> > >
> > > I have some vague memories that we had to do something like double the
> > > RAM in our SPARCstations to make Solaris 2 feel comfortable. At the
> > > time, that was a pretty serious outlay in an academic department.
> > > 2.5.1 felt like the first version that was _truly_ usable.
> >
> > I'd agree, 2.4 was pretty slow and chunky, 2.5 was alright, but 2.5.1
> was quite usable and stable.
> > Also by this time, the hardware was going in directions that SunOS
> wouldn't keep up with.
>
> Yeah, Doug is right, SunOS was pretty simple, it didn't really take
> advantage
> of SMP, Greg Limes tried to thread it but it was too big a job for one guy.
>
> That's not to say that SunOS couldn't have evolved into SMP, I'm 100%
> sure it could have.  It just didn't.  It's a shame.
>

Solbourne had it's OS/MP which did do SMP with a subsystem locking
strategy. This worked reasonably well for the time. So it was quite
possible. Solbourne died, though because it couldn't get Solaris sources
soon enough to do a port that mattered...

Warner

>

[-- Attachment #2: Type: text/html, Size: 2311 bytes --]

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2022-01-03 23:21                           ` Doug McIntyre
  2022-01-03 23:37                             ` Adam Thornton
@ 2022-01-03 23:44                             ` Larry McVoy
  2022-01-03 23:56                               ` Warner Losh
  2022-01-04  2:28                               ` Theodore Ts'o
  1 sibling, 2 replies; 71+ messages in thread
From: Larry McVoy @ 2022-01-03 23:44 UTC (permalink / raw)
  To: Doug McIntyre; +Cc: The Eunuchs Hysterical Society

On Mon, Jan 03, 2022 at 05:21:51PM -0600, Doug McIntyre wrote:
> On Mon, Jan 03, 2022 at 04:15:08PM -0500, Dan Cross wrote:
> > On Mon, Jan 3, 2022 at 3:23 PM Theodore Ts'o <tytso@mit.edu> wrote:
> 
> > > Yeah, to be fair, by the time Solaris 2.3 or 2.4 came around, it was
> > > mostly up to par.  (Or maybe it was because Moore's law meant that we
> > > didn't care any more.  :-)
> > 
> > I have some vague memories that we had to do something like double the
> > RAM in our SPARCstations to make Solaris 2 feel comfortable. At the
> > time, that was a pretty serious outlay in an academic department.
> > 2.5.1 felt like the first version that was _truly_ usable.
> 
> I'd agree, 2.4 was pretty slow and chunky, 2.5 was alright, but 2.5.1 was quite usable and stable.
> Also by this time, the hardware was going in directions that SunOS wouldn't keep up with.

Yeah, Doug is right, SunOS was pretty simple, it didn't really take advantage
of SMP, Greg Limes tried to thread it but it was too big a job for one guy.

That's not to say that SunOS couldn't have evolved into SMP, I'm 100%
sure it could have.  It just didn't.  It's a shame.

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2022-01-03 23:21                           ` Doug McIntyre
@ 2022-01-03 23:37                             ` Adam Thornton
  2022-01-04 14:49                               ` Stuart Remphrey
  2022-01-03 23:44                             ` Larry McVoy
  1 sibling, 1 reply; 71+ messages in thread
From: Adam Thornton @ 2022-01-03 23:37 UTC (permalink / raw)
  To: Doug McIntyre, The Eunuchs Hysterical Society

[-- Attachment #1: Type: text/plain, Size: 979 bytes --]

2.5.1 was the first Solaris that didn't often make me scream that I wanted
SunOS 4.1 back.

Adam


On Mon, Jan 3, 2022 at 4:32 PM Doug McIntyre <merlyn@geeks.org> wrote:

> On Mon, Jan 03, 2022 at 04:15:08PM -0500, Dan Cross wrote:
> > On Mon, Jan 3, 2022 at 3:23 PM Theodore Ts'o <tytso@mit.edu> wrote:
>
> > > Yeah, to be fair, by the time Solaris 2.3 or 2.4 came around, it was
> > > mostly up to par.  (Or maybe it was because Moore's law meant that we
> > > didn't care any more.  :-)
> >
> > I have some vague memories that we had to do something like double the
> > RAM in our SPARCstations to make Solaris 2 feel comfortable. At the
> > time, that was a pretty serious outlay in an academic department.
> > 2.5.1 felt like the first version that was _truly_ usable.
>
> I'd agree, 2.4 was pretty slow and chunky, 2.5 was alright, but 2.5.1 was
> quite usable and stable.
> Also by this time, the hardware was going in directions that SunOS
> wouldn't keep up with.
>
>
>

[-- Attachment #2: Type: text/html, Size: 1463 bytes --]

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2022-01-03 21:15                         ` Dan Cross
  2022-01-03 22:26                           ` Theodore Ts'o
@ 2022-01-03 23:21                           ` Doug McIntyre
  2022-01-03 23:37                             ` Adam Thornton
  2022-01-03 23:44                             ` Larry McVoy
  1 sibling, 2 replies; 71+ messages in thread
From: Doug McIntyre @ 2022-01-03 23:21 UTC (permalink / raw)
  To: The Eunuchs Hysterical Society

On Mon, Jan 03, 2022 at 04:15:08PM -0500, Dan Cross wrote:
> On Mon, Jan 3, 2022 at 3:23 PM Theodore Ts'o <tytso@mit.edu> wrote:

> > Yeah, to be fair, by the time Solaris 2.3 or 2.4 came around, it was
> > mostly up to par.  (Or maybe it was because Moore's law meant that we
> > didn't care any more.  :-)
> 
> I have some vague memories that we had to do something like double the
> RAM in our SPARCstations to make Solaris 2 feel comfortable. At the
> time, that was a pretty serious outlay in an academic department.
> 2.5.1 felt like the first version that was _truly_ usable.

I'd agree, 2.4 was pretty slow and chunky, 2.5 was alright, but 2.5.1 was quite usable and stable.
Also by this time, the hardware was going in directions that SunOS wouldn't keep up with.



^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2022-01-03 22:26                           ` Theodore Ts'o
@ 2022-01-03 23:10                             ` Dan Cross
  2022-01-04 15:45                             ` Chet Ramey
  2022-01-09 19:28                             ` Larry McVoy
  2 siblings, 0 replies; 71+ messages in thread
From: Dan Cross @ 2022-01-03 23:10 UTC (permalink / raw)
  To: Theodore Ts'o; +Cc: The Eunuchs Hysterical Society

On Mon, Jan 3, 2022 at 5:26 PM Theodore Ts'o <tytso@mit.edu> wrote:
> On Mon, Jan 03, 2022 at 04:15:08PM -0500, Dan Cross wrote:
> > > For that matter, you
> > > probably don't need to use ifconfig/route --- just let the DHCP client
> > > server of your choice take care of setting up the network, and you're
> > > done.
> >
> > That'll work on a laptop or on my home network (where I set up the
> > DHCP server). In a large-scale datacenter environment, maybe not so
> > much.
>
> I've seen a number of large-scale data center environments that use
> DHCP; and those that don't in Linux-land generally just edit a config
> file in /etc/network/interfaces or /etc/network/interfaces.d --- and
> then the init scripts would just use "ifup -a" which would bring up
> the network either using DHCP or the hard-coded values in
> /etc/network/interaces.  No need to have sysadmins manually typing
> "route add ..." commands except in extreme debugging situations.

So, would you say that those are...supported by a bunch of shell
scripts that would have to be changed if those systems migrated to the
new tools for whatever reason? :-D

> > I would suggest that the number of users who want to run MySQL is much
> > smaller than the number who want to have a functioning network. But
> > you're right; it's not that hard to adapt. That was kind of the point;
> > there have been cases where Linux users have adapted to one degree or
> > another.
>
> I actually suspect the number of users who want to run MySQL is
> actually larger than the number of users who need to manually
> configure a network using /sbin/ifconfig and /sbin/route....  :-)

Touche.

> > The `apt install net-tools` thing is a red-herring, though: that's
> > explicitly why I mentioned Google prod. What works on a single system
> > doesn't necessarily scale to O(10^6) machines supporting O(10^7)
> > separately running instances of O(10^4) services in O(10) globally
> > distributed datacenters, that's just a single organization.
>
> Google prod's easy --- Google maintains its own minimal prod
> distribution where a small set of packages (less than a few dozen)
> that were originally forked from Debian a number of years ago.  So
> it's trivial for us, because net-tools is installed by default and
> there is **no** engineering upside for us to switch to ip/ss --- so we
> haven't.  :-)

Recall that I was saying that the effort if Google decided, for
whatever reason, that it wanted to switch, would be multiyear. Giving
into the inertia of staying where Google is today is the easy part. As
you yourself acknowledge, the engineering effort required to switch to
the newfangled tools is non-zero. However, this creates a conundrum;
if Google wanted (again for whatever reason) to avail itself of the
new functionality provided by the new tools, it now has a choice:
either roll the new tools out, or extend the existing tools to provide
the required functionality. This is not something I would describe as
"easy." Sure, rolling the new commands out to machines would be
relatively straight-forward (though I've worked on those systems and
getting things into the production image was non-trivial politically,
if not technically), but transitioning to use them not so much.

This sort of begs the question: why not default to shipping the old
tools (route, netstat, ifconfig) and let people install the _new_
tools if they want their Linux machine to behave like a cisco router?
Google made this choice and sure: as you say, this is a distro-level
question. But it sure seems like the broad consensus around the major
distros is the opposite.

> Similarly, we're still using a *very* old version of bash (having
> stayed back across multiple major releases) because it's easier to
> stick with an old version of bash and keep up with the security fixes,
> than it would be to audit a gazillion shell scripts and shell script
> fragments for the various potential and subtle backwards-incomaptible
> changes that bash has made over the past decade.

This is an aside, but one of my continual gripes about Google was that
it took monumental efforts to deprecate anything (lookin' at you, disk
device names), often because no one had paid attention to trying to
maintain portability between different systems, let alone different
versions of the same system.

But bear in mind that even Google's monorepo doesn't exist in a
hermetic bubble: third party code and upstream dependencies for
externally imposed requirements; in some cases, that forces upgrades
of libraries, toolchains, and yes, even userspace tools.

Incidentally, this was one of the big arguments for getting the Rust
toolchain pulled into google3: even though Google wasn't authoring
first-party Rust code for prod, we _knew_ there were third-party
dependencies that would force us to have a toolchain for the same
reason we have a FORTRAN toolchain. That I was one of the people who
benefited from a Rust toolchain in g3 wasn't entirely coincidental, as
our team finally forced language tools' hands there.

> Larry has told a similar story about the advantages of using a "dead"
> language like Tcl for Bitkeeper, since he didn't have to deal with
> gratuitous version skew problems caused by backwards incomaptible
> changes in Perl or Python.  But if you manage your own userspace,
> including affording to do your own FEDRAMP-compliant security updates,
> you can just choose *not* to upgrade to newer versions of bash or
> network utilities.  Who *says* you have to use bash 5.1?  Or switch to
> ip/ss?

The interesting thing about this is that it ignores the unseen costs
of staying with the older versions of things. The kernel was a great
example, actually, for reasons you know well (though perhaps not super
appropriate for discussion here). This creates a negative feedback
loop: we can't upgrade anything because something will break, so we
stay with the old stuff until forced to move.

> On the other hand, we did spend untold engineering hours migrating
> from Python 2.7 to 3.x (and *wow* was that painful) because in that
> case, that was considered less work and more secure than independently
> supporting Python 2.7 ad infinitum (also, we have fewer Python
> programs than we have shell script fragments in config files like
> Makefiles as well actual shell scripts).

...and also because third-party dependencies started dropping support
for Python 2.

I would love to see language stats for the claim that there's more
shell scripts than Python over there. I'd be mildly though perhaps not
completely surprised if that were true. I suppose if "shell script
fragments" includes `a && b` then I could see it. I'm not cleared for
that anymore though. But I also remember when /usr/bin/python2.4 was
really python2.7. :-(

> So the decision about whether to follow breaking interfaces/language
> changes is an engineering decision that's made on a case by case
> basis, as it should be for all organizations.  In the case of bash and
> net-tools, one decision was made.

Inertia is a hell of a drug.

> In the case of Python, a difference
> decision was made --- although a lot of people really weren't happy
> that the Python developers didn't appreciate the cost of breaking
> backwards compatibility (or rather, they decided that they didn't care
> and prioritized their own convenience over that of their users).
>
> This is why I'm a big supporter of Linus's, "Thou Shalt Not Break
> Userspace" rule; backwards compatibility at the lowest levels is
> important!  I'd draw that line higher than some people, but hey, if
> you don't like the instability of Python, you can do what Bitkeeper
> did and base your extension language/scripts on something like Tcl
> instead.

I guess what I'm saying is that "thou shalt not break userspace" is a
bit of a misnomer; perhaps it should be rephrased as, "thou shalt not
change the kernel in such a way that userspace code no longer works."
He is, after all, referring specifically to the kernel interface,
right? But just as sufficiently advanced technology is said to be
indistinguishable from magic, a missing command like `ifconfig`
_might_ be indistinguishable from an incompatible system interface as
far as a user is concerned.

> P.S.  If you want to see the horror of trying to support a Python
> program that has to be able to run on a wide variety of Linux distros,
> running different versions of Python (from RHEL 6 and Python 2.3 up to
> the newest bleeding edge Python 3.x), take a look at the Google's
> gcloud program.  Alas, unlike the Google prod environment, we can't
> dictate what distro our Cloud customers might choose to use.
>
> The gcloud command (part of the Google Cloud SDK) has to play all
> sorts of portability games, and people need to continually test across
> a wide range of Python versions and Linux distros to provide assurance
> that it continues to correctly as new features are added and bugs are
> fixed.  I *completely* understand why Larry chose to implement
> Bitkeeper extensions in Tcl, since it avoids this problem.

No thank you! I'm good. :-)

> Bt yeah, it's why I wouldn't want to change dot-dot, since having seen
> the pain the Python developers inflected on me, I wouldn't want to
> inflict it on others...

 Hmm. I wonder if I have any 386 a.out binaries floating around. :-)

        - Dan C.

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2022-01-03 20:23                       ` Theodore Ts'o
  2022-01-03 20:45                         ` Warner Losh
  2022-01-03 21:15                         ` Dan Cross
@ 2022-01-03 22:57                         ` Phil Budne
  2 siblings, 0 replies; 71+ messages in thread
From: Phil Budne @ 2022-01-03 22:57 UTC (permalink / raw)
  To: tytso, crossd; +Cc: tuhs

Theodore Ts'o wrote:
> Well, take a look at the ip-route man page.  The BSD route command
> assumes fundamentally there is a single routing table that you can
> update.  In Linux, there are multiple routing tables --- to support
> NAT, VRF (virtual routing and forwarding), etc.

Non-exhaustive research (didn't look at NetBSD, OpenBSD or Dragonfly):
Looks like the FreeBSD route got the -fib argument in November 2012,
the setfib command (execute a command with an alternate routing table)
appeared in 2008.

Looks like the ip command appeared in Linux 2.2 (end of 1999), so it
doesn't look like there was a lead to follow.

Now don't get me wrong, I have no lost love for the BSD route command,
it almost always takes me at least four tries; one for a syntax error,
one to add a bad route, and another to remove the bad route!

BUT, as someone who has been using BSD based systems since 4.2, it's
hard not to feel like GNU and Linux folks have evidenced a pride that
they're better than Un*x (I remember at a Usenix (Boston in '94?)
overhearing a VA Linux employee boasting how their goal was to beat
out commercial Un*x vendors.  Too bad they weren't aiming at the
Windows desktop...) and missed the boat on "Unix philosophy", but I'm
well aware that what I remember with fondness as simpler times was
contemporaneously viewed by others as "beginning to smell really bad".

Finally, I'm new here, but I can't help feeling that both complaining
and boasting about Linux aren't within the realm of "Unix Heritage"?

But discussion of lessons learned by Bell Research post 7th Edition,
are of greater interest to me.

Keeping ".." working is certainly a major pain in the a**, and I can
understand wanting to throw it in the ditch, but I came from the DEC
36-bit world (TOPS-10/20 and ITS) which didn't present a single,
uniform file name space (full paths started with "DEVICE:", and
different device handlers could treat the following name space
differently), and I've never been convinced that the Unix single
namespace hasn't brought just as much pain as benefit.

ITS in particular had device handlers that were dynamically loaded
into another process and commuinicated thru a special device handler
handler (the JOB device) that translated system calls to messages.
Listing the directory of a spooled print device would show the queue,
and there was also network file access to other ITS machines).

The DEC systems had system and user defined "logical devices", which I
do remember fondly as causing less pain than symlinks.  ITS may have
had both symlink and logical device-like things...


^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2022-01-03 21:15                         ` Dan Cross
@ 2022-01-03 22:26                           ` Theodore Ts'o
  2022-01-03 23:10                             ` Dan Cross
                                               ` (2 more replies)
  2022-01-03 23:21                           ` Doug McIntyre
  1 sibling, 3 replies; 71+ messages in thread
From: Theodore Ts'o @ 2022-01-03 22:26 UTC (permalink / raw)
  To: Dan Cross; +Cc: The Eunuchs Hysterical Society

On Mon, Jan 03, 2022 at 04:15:08PM -0500, Dan Cross wrote:
> > For that matter, you
> > probably don't need to use ifconfig/route --- just let the DHCP client
> > server of your choice take care of setting up the network, and you're
> > done.
> 
> That'll work on a laptop or on my home network (where I set up the
> DHCP server). In a large-scale datacenter environment, maybe not so
> much.

I've seen a number of large-scale data center environments that use
DHCP; and those that don't in Linux-land generally just edit a config
file in /etc/network/interfaces or /etc/network/interfaces.d --- and
then the init scripts would just use "ifup -a" which would bring up
the network either using DHCP or the hard-coded values in
/etc/network/interaces.  No need to have sysadmins manually typing
"route add ..." commands except in extreme debugging situations.

> > That's a distro-level choice.  And for most users, their networking is
> > automatically brought up using NetworkManager, or some such, so they
> > probably don't care.  And it's not like installing a package is that
> > painful.  I don't see users of say, mysql complaining that they have
> > to install that package should they want to use it.
> 
> I would suggest that the number of users who want to run MySQL is much
> smaller than the number who want to have a functioning network. But
> you're right; it's not that hard to adapt. That was kind of the point;
> there have been cases where Linux users have adapted to one degree or
> another.

I actually suspect the number of users who want to run MySQL is
actually larger than the number of users who need to manually
configure a network using /sbin/ifconfig and /sbin/route....  :-)

> The `apt install net-tools` thing is a red-herring, though: that's
> explicitly why I mentioned Google prod. What works on a single system
> doesn't necessarily scale to O(10^6) machines supporting O(10^7)
> separately running instances of O(10^4) services in O(10) globally
> distributed datacenters, that's just a single organization.

Google prod's easy --- Google maintains its own minimal prod
distribution where a small set of packages (less than a few dozen)
that were originally forked from Debian a number of years ago.  So
it's trivial for us, because net-tools is installed by default and
there is **no** engineering upside for us to switch to ip/ss --- so we
haven't.  :-)

Similarly, we're still using a *very* old version of bash (having
stayed back across multiple major releases) because it's easier to
stick with an old version of bash and keep up with the security fixes,
than it would be to audit a gazillion shell scripts and shell script
fragments for the various potential and subtle backwards-incomaptible
changes that bash has made over the past decade.

Larry has told a similar story about the advantages of using a "dead"
language like Tcl for Bitkeeper, since he didn't have to deal with
gratuitous version skew problems caused by backwards incomaptible
changes in Perl or Python.  But if you manage your own userspace,
including affording to do your own FEDRAMP-compliant security updates,
you can just choose *not* to upgrade to newer versions of bash or
network utilities.  Who *says* you have to use bash 5.1?  Or switch to
ip/ss?

On the other hand, we did spend untold engineering hours migrating
from Python 2.7 to 3.x (and *wow* was that painful) because in that
case, that was considered less work and more secure than independently
supporting Python 2.7 ad infinitum (also, we have fewer Python
programs than we have shell script fragments in config files like
Makefiles as well actual shell scripts).

So the decision about whether to follow breaking interfaces/language
changes is an engineering decision that's made on a case by case
basis, as it should be for all organizations.  In the case of bash and
net-tools, one decision was made.  In the case of Python, a difference
decision was made --- although a lot of people really weren't happy
that the Python developers didn't appreciate the cost of breaking
backwards compatibility (or rather, they decided that they didn't care
and prioritized their own convenience over that of their users).

This is why I'm a big supporter of Linus's, "Thou Shalt Not Break
Userspace" rule; backwards compatibility at the lowest levels is
important!  I'd draw that line higher than some people, but hey, if
you don't like the instability of Python, you can do what Bitkeeper
did and base your extension language/scripts on something like Tcl
instead.

					- Ted

P.S.  If you want to see the horror of trying to support a Python
program that has to be able to run on a wide variety of Linux distros,
running different versions of Python (from RHEL 6 and Python 2.3 up to
the newest bleeding edge Python 3.x), take a look at the Google's
gcloud program.  Alas, unlike the Google prod environment, we can't
dictate what distro our Cloud customers might choose to use.

The gcloud command (part of the Google Cloud SDK) has to play all
sorts of portability games, and people need to continually test across
a wide range of Python versions and Linux distros to provide assurance
that it continues to correctly as new features are added and bugs are
fixed.  I *completely* understand why Larry chose to implement
Bitkeeper extensions in Tcl, since it avoids this problem.

Bt yeah, it's why I wouldn't want to change dot-dot, since having seen
the pain the Python developers inflected on me, I wouldn't want to
inflict it on others...

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2022-01-03 20:23                       ` Theodore Ts'o
  2022-01-03 20:45                         ` Warner Losh
@ 2022-01-03 21:15                         ` Dan Cross
  2022-01-03 22:26                           ` Theodore Ts'o
  2022-01-03 23:21                           ` Doug McIntyre
  2022-01-03 22:57                         ` Phil Budne
  2 siblings, 2 replies; 71+ messages in thread
From: Dan Cross @ 2022-01-03 21:15 UTC (permalink / raw)
  To: Theodore Ts'o; +Cc: The Eunuchs Hysterical Society

On Mon, Jan 3, 2022 at 3:23 PM Theodore Ts'o <tytso@mit.edu> wrote:
> On Mon, Jan 03, 2022 at 08:35:24AM -0500, Dan Cross wrote:
> > That it was, at least initially. It's actually pretty good now, but it
> > took a _long_ time to get there, and the forced incompatibilities
> > caused a lot of pain in the process, which was my original thesis.
> > [snip]
>
> Yeah, to be fair, by the time Solaris 2.3 or 2.4 came around, it was
> mostly up to par.  (Or maybe it was because Moore's law meant that we
> didn't care any more.  :-)

I have some vague memories that we had to do something like double the
RAM in our SPARCstations to make Solaris 2 feel comfortable. At the
time, that was a pretty serious outlay in an academic department.
2.5.1 felt like the first version that was _truly_ usable.

> > Are there _distros_ that don't supply those things? Probably; I really
> > have no idea. Are there mainstream distros that do not? I doubt it.
> > However, they have to be installed, which is an additional step that
> > has to at least be accounted for. At scale, that's a pain: I imagine
> > that if, say, Google wanted to move to `ip` in lieu of `ifconfig` et
> > al in prod, it would be a multiyear process, including sunsetting the
> > older tools. Just identifying every use of `ifconfig` in a shell
> > script somewhere would be a pretty major undertaking.
>
> Sure, but there's no *point* to sunset the old tools.  The git tree
> for net-tools is still being actively developed (the last commit was
> dated December 12, 2021).  And the kernel interfaces are not going to
> be disappear, because of the prime directive: Thou Shalt Not Break
> Userspace.

Within a single organization, two semi-compatible ways of doing things
seems suboptimal for a number of reasons.

> > [snip]
> > Surely the programmatic interfaces are separate from their realization
> > in a tool? I can understand the rigidity of some `ioctl` based
> > interface that's a pain to work around; I find it harder to believe
> > that plugging some other interface into `ifconfig` in a relatively
> > graceful way is untractible. Surely, in the limit, one could extend
> > ifconfig with a new verb as the first positional argument that expands
> > to the argument set of `ip`: `ifconfig ip address ...` etc. Maybe that
> > was considered and rejected by the maintainers.

Jon Steinhart reminds me that he said largely the same thing last
week; credit where it's due!

> Well, take a look at the ip-route man page.  The BSD route command
> assumes fundamentally there is a single routing table that you can
> update.  In Linux, there are multiple routing tables --- to support
> NAT, VRF (virtual routing and forwarding), etc.

As Warner mentioned, the BSDs also support multiple routing tables,
with the `route` command. To bring it back to ham radio, I
not-so-humbly offer up
https://wiki.ampr.org/wiki/Setting_up_a_gateway_on_OpenBSD as a rather
pedestrian example of their utility.

> I suspect the other consideration was that all of this extra
> functionality and complexity were done by folks who wanted the Linux
> networking stack to essentially have pretty much all of the
> functionality of a Cisco Router.  So it made sense to create a new
> user interface interface that was inspired by the Cisco IOS
> configuration language.

I would suggest that the latter does not necessarily follow from the
former. But if people want to evolve things, that's ok: after all, the
genesis of this discussion is that we know how to handle breaking
changes.

> Now, if you weren't trying to do something
> ala a router in the default-free zone, and were just simply doing what
> most leaf nodes on the internet (99.99999% of the hosts), there really
> is no reason to need to use the ip/ss interface.

Except that's what a lot of documentation tells you to do. :-) Really,
`ip` and `ss` are basically fine. The overriding point is that when
one says on the one hand, "we cannot change the way `..` works because
people's scripts might break..." but we fundamentally change the
default network commands on the other, the former seems to be
self-invalidating.

> For that matter, you
> probably don't need to use ifconfig/route --- just let the DHCP client
> server of your choice take care of setting up the network, and you're
> done.

That'll work on a laptop or on my home network (where I set up the
DHCP server). In a large-scale datacenter environment, maybe not so
much.

> > Well, you kind of have. It's a small thing to install another package,
> > sure, but still something that must be done if you want the old tools.
>
> That's a distro-level choice.  And for most users, their networking is
> automatically brought up using NetworkManager, or some such, so they
> probably don't care.  And it's not like installing a package is that
> painful.  I don't see users of say, mysql complaining that they have
> to install that package should they want to use it.

I would suggest that the number of users who want to run MySQL is much
smaller than the number who want to have a functioning network. But
you're right; it's not that hard to adapt. That was kind of the point;
there have been cases where Linux users have adapted to one degree or
another.

> I'm old school, and since I generally tend to install BIND, that will
> drag in net-tools as dependency, so all my systems have ifconfig
> installed.  But I'm not going to have a lot of sympathy for someone
> who thinks that "sudo apt-get install net-tools" is massive
> inconvenience.

Since shells often have custom (and occasionally inconsistent)
handling of `..`, I'm not sure I'd have a lot of sympathy for that one
either. Why one and not the other? I suspect Clem got it right: it has
to do with the perceived value of the change. The networking stuff had
sufficient value that it was worth the cost incurred (which is low,
but non-zero). Perhaps changing `..` just wouldn't have the benefit. I
argue that it's hard to know those things beforehand.

The `apt install net-tools` thing is a red-herring, though: that's
explicitly why I mentioned Google prod. What works on a single system
doesn't necessarily scale to O(10^6) machines supporting O(10^7)
separately running instances of O(10^4) services in O(10) globally
distributed datacenters, that's just a single organization.

        - Dan C.

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2022-01-03 20:23                       ` Theodore Ts'o
@ 2022-01-03 20:45                         ` Warner Losh
  2022-01-03 21:15                         ` Dan Cross
  2022-01-03 22:57                         ` Phil Budne
  2 siblings, 0 replies; 71+ messages in thread
From: Warner Losh @ 2022-01-03 20:45 UTC (permalink / raw)
  To: Theodore Ts'o; +Cc: The Eunuchs Hysterical Society

[-- Attachment #1: Type: text/plain, Size: 4979 bytes --]

On Mon, Jan 3, 2022 at 1:24 PM Theodore Ts'o <tytso@mit.edu> wrote:

> On Mon, Jan 03, 2022 at 08:35:24AM -0500, Dan Cross wrote:
> > > (No
> > > matter that Solaris 2 was a major step *backwards* in terms of
> > > performance and stability compared to Sun OS....)
> >
> > That it was, at least initially. It's actually pretty good now, but it
> > took a _long_ time to get there, and the forced incompatibilities
> > caused a lot of pain in the process, which was my original thesis.
> > Even now, though, I find some things gratuitously different than other
> > versions of Unix (network administration, for example). Managing NFS
> > is still something of a mystery to me.
>
> Yeah, to be fair, by the time Solaris 2.3 or 2.4 came around, it was
> mostly up to par.  (Or maybe it was because Moore's law meant that we
> didn't care any more.  :-)
>

For my corner of the world, 2.3 was stable enough to run in production. 2.4
was fast enough :)


> > Are there _distros_ that don't supply those things? Probably; I really
> > have no idea. Are there mainstream distros that do not? I doubt it.
> > However, they have to be installed, which is an additional step that
> > has to at least be accounted for. At scale, that's a pain: I imagine
> > that if, say, Google wanted to move to `ip` in lieu of `ifconfig` et
> > al in prod, it would be a multiyear process, including sunsetting the
> > older tools. Just identifying every use of `ifconfig` in a shell
> > script somewhere would be a pretty major undertaking.
>
> Sure, but there's no *point* to sunset the old tools.  The git tree
> for net-tools is still being actively developed (the last commit was
> dated December 12, 2021).  And the kernel interfaces are not going to
> be disappear, because of the prime directive: Thou Shalt Not Break
> Userspace.
>
> > > The reason why ip and ss were added was because there was a
> > > significant amount of new functionality that was added to the Linux
> > > networking stack (especially relating to routing and address aliasing)
> > > that couldn't be expressed using the older C programming interfaces as
> > > well as the ifconfig/route shell commands.
> >
> > Surely the programmatic interfaces are separate from their realization
> > in a tool? I can understand the rigidity of some `ioctl` based
> > interface that's a pain to work around; I find it harder to believe
> > that plugging some other interface into `ifconfig` in a relatively
> > graceful way is untractible. Surely, in the limit, one could extend
> > ifconfig with a new verb as the first positional argument that expands
> > to the argument set of `ip`: `ifconfig ip address ...` etc. Maybe that
> > was considered and rejected by the maintainers.
>
> Well, take a look at the ip-route man page.  The BSD route command
> assumes fundamentally there is a single routing table that you can
> update.  In Linux, there are multiple routing tables --- to support
> NAT, VRF (virtual routing and forwarding), etc.
>

The 4.4BSD route command does this. The various BSDs now support multiple
routing tables through an evolved interface...


> I suspect the other consideration was that all of this extra
> functionality and complexity were done by folks who wanted the Linux
> networking stack to essentially have pretty much all of the
> functionality of a Cisco Router.  So it made sense to create a new
> user interface interface that was inspired by the Cisco IOS
> configuration language.  Now, if you weren't trying to do something
> ala a router in the default-free zone, and were just simply doing what
> most leaf nodes on the internet (99.99999% of the hosts), there really
> is no reason to need to use the ip/ss interface.  For that matter, you
> probably don't need to use ifconfig/route --- just let the DHCP client
> server of your choice take care of setting up the network, and you're
> done.
>

True. The vast majority of installs today are just "use DHCP and be happy"
though somebody has to run the dhcp server :)


> > Well, you kind of have. It's a small thing to install another package,
> > sure, but still something that must be done if you want the old tools.
>
> That's a distro-level choice.  And for most users, their networking is
> automatically brought up using NetworkManager, or some such, so they
> probably don't care.  And it's not like installing a package is that
> painful.  I don't see users of say, mysql complaining that they have
> to install that package should they want to use it.
>
> I'm old school, and since I generally tend to install BIND, that will
> drag in net-tools as dependency, so all my systems have ifconfig
> installed.  But I'm not going to have a lot of sympathy for someone
> who thinks that "sudo apt-get install net-tools" is massive
> inconvenience.
>

For machines you manage, sure. It's easy. Try it with the linksys
router you can log into and it starts to get much harder... though
with those loading new firmware isn't super hard.

Warner

[-- Attachment #2: Type: text/html, Size: 6366 bytes --]

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2022-01-03 13:35                     ` Dan Cross
@ 2022-01-03 20:23                       ` Theodore Ts'o
  2022-01-03 20:45                         ` Warner Losh
                                           ` (2 more replies)
  0 siblings, 3 replies; 71+ messages in thread
From: Theodore Ts'o @ 2022-01-03 20:23 UTC (permalink / raw)
  To: Dan Cross; +Cc: The Eunuchs Hysterical Society

On Mon, Jan 03, 2022 at 08:35:24AM -0500, Dan Cross wrote:
> > (No
> > matter that Solaris 2 was a major step *backwards* in terms of
> > performance and stability compared to Sun OS....)
> 
> That it was, at least initially. It's actually pretty good now, but it
> took a _long_ time to get there, and the forced incompatibilities
> caused a lot of pain in the process, which was my original thesis.
> Even now, though, I find some things gratuitously different than other
> versions of Unix (network administration, for example). Managing NFS
> is still something of a mystery to me.

Yeah, to be fair, by the time Solaris 2.3 or 2.4 came around, it was
mostly up to par.  (Or maybe it was because Moore's law meant that we
didn't care any more.  :-)

> Are there _distros_ that don't supply those things? Probably; I really
> have no idea. Are there mainstream distros that do not? I doubt it.
> However, they have to be installed, which is an additional step that
> has to at least be accounted for. At scale, that's a pain: I imagine
> that if, say, Google wanted to move to `ip` in lieu of `ifconfig` et
> al in prod, it would be a multiyear process, including sunsetting the
> older tools. Just identifying every use of `ifconfig` in a shell
> script somewhere would be a pretty major undertaking.

Sure, but there's no *point* to sunset the old tools.  The git tree
for net-tools is still being actively developed (the last commit was
dated December 12, 2021).  And the kernel interfaces are not going to
be disappear, because of the prime directive: Thou Shalt Not Break
Userspace.

> > The reason why ip and ss were added was because there was a
> > significant amount of new functionality that was added to the Linux
> > networking stack (especially relating to routing and address aliasing)
> > that couldn't be expressed using the older C programming interfaces as
> > well as the ifconfig/route shell commands.
> 
> Surely the programmatic interfaces are separate from their realization
> in a tool? I can understand the rigidity of some `ioctl` based
> interface that's a pain to work around; I find it harder to believe
> that plugging some other interface into `ifconfig` in a relatively
> graceful way is untractible. Surely, in the limit, one could extend
> ifconfig with a new verb as the first positional argument that expands
> to the argument set of `ip`: `ifconfig ip address ...` etc. Maybe that
> was considered and rejected by the maintainers.

Well, take a look at the ip-route man page.  The BSD route command
assumes fundamentally there is a single routing table that you can
update.  In Linux, there are multiple routing tables --- to support
NAT, VRF (virtual routing and forwarding), etc.

I suspect the other consideration was that all of this extra
functionality and complexity were done by folks who wanted the Linux
networking stack to essentially have pretty much all of the
functionality of a Cisco Router.  So it made sense to create a new
user interface interface that was inspired by the Cisco IOS
configuration language.  Now, if you weren't trying to do something
ala a router in the default-free zone, and were just simply doing what
most leaf nodes on the internet (99.99999% of the hosts), there really
is no reason to need to use the ip/ss interface.  For that matter, you
probably don't need to use ifconfig/route --- just let the DHCP client
server of your choice take care of setting up the network, and you're
done.

> Well, you kind of have. It's a small thing to install another package,
> sure, but still something that must be done if you want the old tools.

That's a distro-level choice.  And for most users, their networking is
automatically brought up using NetworkManager, or some such, so they
probably don't care.  And it's not like installing a package is that
painful.  I don't see users of say, mysql complaining that they have
to install that package should they want to use it.

I'm old school, and since I generally tend to install BIND, that will
drag in net-tools as dependency, so all my systems have ifconfig
installed.  But I'm not going to have a lot of sympathy for someone
who thinks that "sudo apt-get install net-tools" is massive
inconvenience.

						- Ted

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2022-01-01  0:09                   ` Theodore Ts'o
@ 2022-01-03 13:35                     ` Dan Cross
  2022-01-03 20:23                       ` Theodore Ts'o
  0 siblings, 1 reply; 71+ messages in thread
From: Dan Cross @ 2022-01-03 13:35 UTC (permalink / raw)
  To: Theodore Ts'o; +Cc: The Eunuchs Hysterical Society

On Fri, Dec 31, 2021 at 7:09 PM Theodore Ts'o <tytso@mit.edu> wrote:
> On Fri, Dec 31, 2021 at 01:17:18PM -0500, Dan Cross wrote:
> > On Fri, Dec 31, 2021, 10:54 AM Adam Thornton <athornton@gmail.com> wrote:
> > > Slightly older, but also slightly more fundamental to the system, you need
> > > look no farther than Solaris's `/bin/sh` for an illustrated example of the
> > > pros and cons of maintaining backwards compatibility. [snip]
> >
> > Sun is not the exemplar here: the move from SunOS 4's BSD userland to
> > Solaris 2's SVR4 broke tons of things. They didn't seem to mind that their
> > customers had to pay the cost of adaptation.
>
> I'm sure that there were people at Sun who *did* care.

Oh I think we know from first-hand accounts that there were _people_
there who cared. When I wrote "they" earlier I was referring to the
corporation (since those are people now, you know!) which clearly
didn't; or at least made a decision that indicated that they thought
the cost was worth it.

> The story I
> had heard was that it was a decision made at the C-suite level, and
> was a quid-pro-quo where to get that sweet, sweet, cash from AT&T so
> Sun could stay afloat, they had to switch over to System V.

I wouldn't kick AT&T's $100 million out of the house for running System V.

> (No
> matter that Solaris 2 was a major step *backwards* in terms of
> performance and stability compared to Sun OS....)

That it was, at least initially. It's actually pretty good now, but it
took a _long_ time to get there, and the forced incompatibilities
caused a lot of pain in the process, which was my original thesis.
Even now, though, I find some things gratuitously different than other
versions of Unix (network administration, for example). Managing NFS
is still something of a mystery to me.

> > The Linux example is also a bit strange. The move from e.g. `ifconfig` and
> > `netstat to `ip` and `ss` required lots of local retooling (I suppose some
> > distros retain the older tools or let you install them as an option. I
> > suppose one could always install `bash` on Solaris as a shell lingua
> > franca, as well). Not to mention systemd. The point is, breaking changes
> > are introduced all the time.
>
> Are there distros who are no longer supplying ifconfig/netstat/route,
> at least as an optional package?  That's surprising if so.

Are there _distros_ that don't supply those things? Probably; I really
have no idea. Are there mainstream distros that do not? I doubt it.
However, they have to be installed, which is an additional step that
has to at least be accounted for. At scale, that's a pain: I imagine
that if, say, Google wanted to move to `ip` in lieu of `ifconfig` et
al in prod, it would be a multiyear process, including sunsetting the
older tools. Just identifying every use of `ifconfig` in a shell
script somewhere would be a pretty major undertaking.

> All of the kernel interfaces to allow the old-style net-tools packages
> to work, as well as the BSD-style ioctls/setsockopt, etc., are still
> around, and fully supported.  At least on my systems, I still install
> net-tools because my finger macros are still used to using ifconfig,
> netstat, and friends.

By virtue of the existence of that additional step, however, there is
an incompatibility with the "older" way of doing things. By no longer
being the default, it is (in perhaps a minor way) a breaking change.

> The reason why ip and ss were added was because there was a
> significant amount of new functionality that was added to the Linux
> networking stack (especially relating to routing and address aliasing)
> that couldn't be expressed using the older C programming interfaces as
> well as the ifconfig/route shell commands.

Surely the programmatic interfaces are separate from their realization
in a tool? I can understand the rigidity of some `ioctl` based
interface that's a pain to work around; I find it harder to believe
that plugging some other interface into `ifconfig` in a relatively
graceful way is untractible. Surely, in the limit, one could extend
ifconfig with a new verb as the first positional argument that expands
to the argument set of `ip`: `ifconfig ip address ...` etc. Maybe that
was considered and rejected by the maintainers.

> There were two north star
> principles about the new networking interfaces:
>
> 1) The old interfaces were always supposed to continue to work, and if
> you didn't need the new functionality, there was no reason to use the
> newer interfaces.
>
> 2) The new interfaces were *supposed* to be a strict superset of the
> old interfaces.
>
> If in fact ip and ss don't support AX.25, or other "exotic address
> families" --- that's a bug, and should be reported as such.

This is an aside, but I'd suggest taking the opposite approach: rip
AX.25 out. It has serious bugs (including, apparently, leaking kernel
memory into transmitted packets?!) that have gone unfixed for years.
My AX.25 machine panics on reboot if there are active netrom
connections, and netrom packet state isn't cleaned up properly. The
maintainers seem to be out to lunch and are unresponsive on
linux-hams.

Personally, I'd prefer to see a FUSE-like mechanism for implementing
things like AX.25 in user-space: that would make it much easier to fix
things like that. On plan9, I would implement AX.25 as a userspace
program that exported a small filesystem and bind-mount it onto /net.
On Unix, a similar thing could be done with Unix-domain sockets and
daemons muxing the radio, but it wouldn't integrate nicely with the
existing tools. That's a problem with doing everything in a tightly
coupled, monolithic way.

> That
> being said, if you don't need the fancy new features, there's no
> reason to switch away from ifconfig.  The whole *point* of the first
> principle was that we didn't want to force users to do a forced Python
> 2.7 -> Python 3 style "long march" migration.

Well, you kind of have. It's a small thing to install another package,
sure, but still something that must be done if you want the old tools.

        - Dan C.

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2021-12-31 18:17                 ` Dan Cross
  2021-12-31 18:23                   ` Larry McVoy
  2021-12-31 18:29                   ` Arthur Krewat
@ 2022-01-01  0:09                   ` Theodore Ts'o
  2022-01-03 13:35                     ` Dan Cross
  2 siblings, 1 reply; 71+ messages in thread
From: Theodore Ts'o @ 2022-01-01  0:09 UTC (permalink / raw)
  To: Dan Cross; +Cc: The Eunuchs Hysterical Society

On Fri, Dec 31, 2021 at 01:17:18PM -0500, Dan Cross wrote:
> On Fri, Dec 31, 2021, 10:54 AM Adam Thornton <athornton@gmail.com> wrote:
> 
> > Slightly older, but also slightly more fundamental to the system, you need
> > look no farther than Solaris's `/bin/sh` for an illustrated example of the
> > pros and cons of maintaining backwards compatibility. [snip]
> >
> 
> Sun is not the exemplar here: the move from SunOS 4's BSD userland to
> Solaris 2's SVR4 broke tons of things. They didn't seem to mind that their
> customers had to pay the cost of adaptation.

I'm sure that there were people at Sun who *did* care.  The story I
had heard was that it was a decision made at the C-suite level, and
was a quid-pro-quo where to get that sweet, sweet, cash from AT&T so
Sun could stay afloat, they had to switch over to System V.  (No
matter that Solaris 2 was a major step *backwards* in terms of
performance and stability compared to Sun OS....)

> The Linux example is also a bit strange. The move from e.g. `ifconfig` and
> `netstat to `ip` and `ss` required lots of local retooling (I suppose some
> distros retain the older tools or let you install them as an option. I
> suppose one could always install `bash` on Solaris as a shell lingua
> franca, as well). Not to mention systemd. The point is, breaking changes
> are introduced all the time.

Are there distros who are no longer supplying ifconfig/netstat/route,
at least as an optional package?  That's surprising if so.

All of the kernel interfaces to allow the old-style net-tools packages
to work, as well as the BSD-style ioctls/setsockopt, etc., are still
around, and fully supported.  At least on my systems, I still install
net-tools because my finger macros are still used to using ifconfig,
netstat, and friends.

The reason why ip and ss were added was because there was a
significant amount of new functionality that was added to the Linux
networking stack (especially relating to routing and address aliasing)
that couldn't be expressed using the older C programming interfaces as
well as the ifconfig/route shell commands.  There were two north star
principles about the new networking interfaces:

1) The old interfaces were always supposed to continue to work, and if
you didn't need the new functionality, there was no reason to use the
newer interfaces.

2) The new interfaces were *supposed* to be a strict superset of the
old interfaces.

If in fact ip and ss don't support AX.25, or other "exotic address
families" --- that's a bug, and should be reported as such.  That
being said, if you don't need the fancy new features, there's no
reason to switch away from ifconfig.  The whole *point* of the first
principle was that we didn't want to force users to do a forced Python
2.7 -> Python 3 style "long march" migration.

							- Ted

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2021-12-31  0:43     ` Bakul Shah
  2021-12-31  1:00       ` Rob Pike
@ 2021-12-31 18:56       ` Chet Ramey
  1 sibling, 0 replies; 71+ messages in thread
From: Chet Ramey @ 2021-12-31 18:56 UTC (permalink / raw)
  To: Bakul Shah, Dan Cross; +Cc: TUHS main list, Noel Chiappa

On 12/30/21 7:43 PM, Bakul Shah wrote:

> Without the ".." entry you can't map a dir inode back to a path.
> Note that something similar can happen even today:
> 
> $ mkdir ~/a; cd ~/a; rm -rf ~/a; cd ..
> cd: no such file or directory: ..
> 
> $ mkdir -p ~/a/b; ln -s ~/a/b b; cd b; mv ~/a/b ~/a/c; cd ../b
> ls: ../b: No such file or directory

Modern POSIX shells attempt to disguise this for cd and pwd, using $PWD and
a canonicalization algorithm that essentially backs up one component when
it encounters `..' and removes `.' (with verification at each step and
error handling of various quality). It's the default behavior; the
traditional physical behavior is available with -P. It's a variant of the
ksh schme that Rob's lexical names paper explores.

It has the usual weaknesses: what works for cd doesn't work for, say, ls
or any other utilities that don't understand this particular scheme.

It produces the expected behavior for the first case above, but can be
fooled by file system changes that invalidate $PWD, as in the second case.
(I assume the `ls' error message was either a typo or an artifact of a
function or alias for `cd').

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
		 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2021-12-31 18:23                   ` Larry McVoy
@ 2021-12-31 18:37                     ` Dan Cross
  0 siblings, 0 replies; 71+ messages in thread
From: Dan Cross @ 2021-12-31 18:37 UTC (permalink / raw)
  To: Larry McVoy; +Cc: The Eunuchs Hysterical Society

[-- Attachment #1: Type: text/plain, Size: 1791 bytes --]

On Fri, Dec 31, 2021, 1:23 PM Larry McVoy <lm@mcvoy.com> wrote:

> On Fri, Dec 31, 2021 at 01:17:18PM -0500, Dan Cross wrote:
> > On Fri, Dec 31, 2021, 10:54 AM Adam Thornton <athornton@gmail.com>
> wrote:
> >
> > > Slightly older, but also slightly more fundamental to the system, you
> need
> > > look no farther than Solaris's `/bin/sh` for an illustrated example of
> the
> > > pros and cons of maintaining backwards compatibility. [snip]
> > >
> >
> > Sun is not the exemplar here: the move from SunOS 4's BSD userland to
> > Solaris 2's SVR4 broke tons of things. They didn't seem to mind that
> their
> > customers had to pay the cost of adaptation.
>
> I was there for that and I minded like crazy.  It was a hugely bad move
> in my opinion.  Time has not changed that.
>

Yeah. I work on Illumos now for Oxide, and it's...fine? But it took a long
time to get there.

When the SunOS->Solaris changeover happened, the thing that really tripped
me up was `awk`. SunOS 4's awk was nawk, but Solaris 2's awk was 7th Ed awk
and lacked functions and so forth. nawk was available as `nawk`, but it
seemed like a gratuitous regression and we had to modify a lot of local
shell scripts and so on.

> The Linux example is also a bit strange. The move from e.g. `ifconfig` and
> > `netstat to `ip` and `ss` required lots of local retooling
>
> I never understood why Linux did that.
>

I've been dealing with a couple of Linux machines over the break, and
trying to do things the canonical way instead of running back to the
familiar tools, but some things just aren't supported. `ss` doesn't know
about some of the more exotic address families, like AX.25 and NET/ROM. I
can certainly understand why (talk about a hobbyist niche...) but the irony
is that I had to install `netstat` anyway.

>

[-- Attachment #2: Type: text/html, Size: 2728 bytes --]

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2021-12-31 18:17                 ` Dan Cross
  2021-12-31 18:23                   ` Larry McVoy
@ 2021-12-31 18:29                   ` Arthur Krewat
  2022-01-01  0:09                   ` Theodore Ts'o
  2 siblings, 0 replies; 71+ messages in thread
From: Arthur Krewat @ 2021-12-31 18:29 UTC (permalink / raw)
  To: tuhs

On 12/31/2021 1:17 PM, Dan Cross wrote:
> I suppose one could always install `bash` on Solaris as a shell lingua 
> franca, as well

zeus# uname -a
SunOS zeus 5.10 Generic_147441-10 i86pc i386 i86pc
zeus# which bash
/usr/bin/bash

Twas installed by default with Solaris 10, at least later versions that 
I know of. I would usually change root's login shell to /bin/bash and 
never had an issue.

art k.


^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2021-12-31 18:17                 ` Dan Cross
@ 2021-12-31 18:23                   ` Larry McVoy
  2021-12-31 18:37                     ` Dan Cross
  2021-12-31 18:29                   ` Arthur Krewat
  2022-01-01  0:09                   ` Theodore Ts'o
  2 siblings, 1 reply; 71+ messages in thread
From: Larry McVoy @ 2021-12-31 18:23 UTC (permalink / raw)
  To: Dan Cross; +Cc: The Eunuchs Hysterical Society

On Fri, Dec 31, 2021 at 01:17:18PM -0500, Dan Cross wrote:
> On Fri, Dec 31, 2021, 10:54 AM Adam Thornton <athornton@gmail.com> wrote:
> 
> > Slightly older, but also slightly more fundamental to the system, you need
> > look no farther than Solaris's `/bin/sh` for an illustrated example of the
> > pros and cons of maintaining backwards compatibility. [snip]
> >
> 
> Sun is not the exemplar here: the move from SunOS 4's BSD userland to
> Solaris 2's SVR4 broke tons of things. They didn't seem to mind that their
> customers had to pay the cost of adaptation.

I was there for that and I minded like crazy.  It was a hugely bad move
in my opinion.  Time has not changed that.

> The Linux example is also a bit strange. The move from e.g. `ifconfig` and
> `netstat to `ip` and `ss` required lots of local retooling 

I never understood why Linux did that.

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2021-12-31 15:53               ` Adam Thornton
  2021-12-31 16:13                 ` Arthur Krewat
@ 2021-12-31 18:17                 ` Dan Cross
  2021-12-31 18:23                   ` Larry McVoy
                                     ` (2 more replies)
  1 sibling, 3 replies; 71+ messages in thread
From: Dan Cross @ 2021-12-31 18:17 UTC (permalink / raw)
  To: Adam Thornton; +Cc: The Eunuchs Hysterical Society

[-- Attachment #1: Type: text/plain, Size: 896 bytes --]

On Fri, Dec 31, 2021, 10:54 AM Adam Thornton <athornton@gmail.com> wrote:

> Slightly older, but also slightly more fundamental to the system, you need
> look no farther than Solaris's `/bin/sh` for an illustrated example of the
> pros and cons of maintaining backwards compatibility. [snip]
>

Sun is not the exemplar here: the move from SunOS 4's BSD userland to
Solaris 2's SVR4 broke tons of things. They didn't seem to mind that their
customers had to pay the cost of adaptation.

The Linux example is also a bit strange. The move from e.g. `ifconfig` and
`netstat to `ip` and `ss` required lots of local retooling (I suppose some
distros retain the older tools or let you install them as an option. I
suppose one could always install `bash` on Solaris as a shell lingua
franca, as well). Not to mention systemd. The point is, breaking changes
are introduced all the time.

        - Dan C.

[-- Attachment #2: Type: text/html, Size: 1543 bytes --]

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2021-12-31 15:53               ` Adam Thornton
@ 2021-12-31 16:13                 ` Arthur Krewat
  2021-12-31 18:17                 ` Dan Cross
  1 sibling, 0 replies; 71+ messages in thread
From: Arthur Krewat @ 2021-12-31 16:13 UTC (permalink / raw)
  To: tuhs



On 12/31/2021 10:53 AM, Adam Thornton wrote:
> Slightly older, but also slightly more fundamental to the system, you 
> need look no farther than Solaris's `/bin/sh` for an illustrated 
> example of the pros and cons of maintaining backwards compatibility.  
> It was a pain by, say, 2007, to have to check if you were running on 
> SunOS and use `/usr/xpg4/bin/sh` (IIRC) if you wanted a POSIX 
> shell--by that point the other major vendors (and Linux) had something 
> POSIX as the default shell.  But Sun had an installed user base with a 
> bunch of shell scripts that depended on pre-POSIX semantics.

Did some checking, at the risk of going off-topic:

Solaris 11.4 (current patch level, 11.4.40.107.3, December 2021): 
/bin/sh looks to be Korn shell.

root@mercury:~# ls -l /bin/sh
-r-xr-xr-x   5 root     bin      2946936 Dec 22 10:46 /bin/sh
root@mercury:~# strings /bin/sh | grep Korn | head -1
][-author?David Korn <dgk@research.att.com>][-copyright?Copyright (c) 
1982-2021 AT&T Intellectual 
Property][-license?http://www.eclipse.org/org/documents/epl-v10.html][--catalog?libshell][+NAME?
root@mercury:~# sh --version
   version         sh (AT&T Research) 93u+ 2012-08-01

Solaris 11.3: /bin/sh seems to be Korn shell.

medusa# ls -l /bin/sh
lrwxrwxrwx   1 root     root          11 Jun 11  2016 /bin/sh -> amd64/ksh93
medusa# ls -lL /bin/sh
-r-xr-xr-x   8 root     bin      2601344 Jul  1  2016 /bin/sh
medusa# strings !$ | grep dgk | head -1
strings /bin/sh | grep dgk | head -1
][-author?David Korn <dgk@research.att.com>][-copyright?Copyright (c) 
1982-2016 AT&T Intellectual 
Property][-license?http://www.eclipse.org/org/documents/epl-v10.html][--catalog?libshell][+NAME?
medusa# sh --version
   version         sh (AT&T Research) 93u+ 2012-08-01


Solaris 10: (seems /bin/sh is still Bourne at this point):

zeus<@medusa.kilonet.net># ls -l /bin/sh
lrwxrwxrwx   1 root     root          13 Jun  6  2014 /bin/sh -> 
../../sbin/sh
zeus<@medusa.kilonet.net># ls -lL /bin/sh
-r-xr-xr-x   1 root     root       82456 Sep 22  2010 /bin/sh

sh --version just opens a new shell.


^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2021-12-31 13:32             ` Michael Kjörling
@ 2021-12-31 15:53               ` Adam Thornton
  2021-12-31 16:13                 ` Arthur Krewat
  2021-12-31 18:17                 ` Dan Cross
  0 siblings, 2 replies; 71+ messages in thread
From: Adam Thornton @ 2021-12-31 15:53 UTC (permalink / raw)
  To: Michael Kjörling, The Eunuchs Hysterical Society

[-- Attachment #1: Type: text/plain, Size: 2446 bytes --]

Slightly older, but also slightly more fundamental to the system, you need
look no farther than Solaris's `/bin/sh` for an illustrated example of the
pros and cons of maintaining backwards compatibility.  It was a pain by,
say, 2007, to have to check if you were running on SunOS and use
`/usr/xpg4/bin/sh` (IIRC) if you wanted a POSIX shell--by that point the
other major vendors (and Linux) had something POSIX as the default shell.
But Sun had an installed user base with a bunch of shell scripts that
depended on pre-POSIX semantics.

Of course, people don't do a particularly good job of writing POSIX shell
either.  Just this past week one of our refactoring projects at the Rubin
Observatory had some difficulty because Debian uses dash as its
non-interactive POSIX shell.  Some bit of some conda package that we need
claimed to work with dash (and POSIX generally) but in reality didn't.  But
it always worked fine on bash and zsh, so no one had gotten bitten until we
did.

Adam

On Fri, Dec 31, 2021 at 6:40 AM Michael Kjörling <michael@kjorling.se>
wrote:

> On 31 Dec 2021 16:55 +1100, from robpike@gmail.com (Rob Pike):
> > And of course you're right, but we've known how to roll out breaking
> > changes in software for a very long time now. And we do so often. Few
> Unix
> > programs from the past would compile and run today without being updated
> to
> > "modern" interfaces and compilers.
> >
> > I believe it could be done, and I believe it should be done. Rolled out
> > carefully and explained well it would please far more than it offends,
> and
> > is arguably far closer to correct than the current situation.
>
> Seems to me that we have a decent, relatively current case study of
> this with Python 2 vs Python 3.
>
> Which is probably a good bit easier actually, because maintaining the
> ability to run Python 2 applications while introducing the ability to
> run Python 3 applications is likely easier than introducing changes to
> how something as relatively fundamental as how ".." behaves in paths
> while at the same time maintaining backwards compatibility with the
> old behavior, where different users and different applications on the
> same system might very well expect different behavior...
>
> --
> Michael Kjörling • https://michael.kjorling.se • michael@kjorling.se
>  “Remember when, on the Internet, nobody cared that you were a dog?”
>
>

[-- Attachment #2: Type: text/html, Size: 3068 bytes --]

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2021-12-31  5:55           ` Rob Pike
@ 2021-12-31 13:32             ` Michael Kjörling
  2021-12-31 15:53               ` Adam Thornton
  0 siblings, 1 reply; 71+ messages in thread
From: Michael Kjörling @ 2021-12-31 13:32 UTC (permalink / raw)
  To: tuhs

On 31 Dec 2021 16:55 +1100, from robpike@gmail.com (Rob Pike):
> And of course you're right, but we've known how to roll out breaking
> changes in software for a very long time now. And we do so often. Few Unix
> programs from the past would compile and run today without being updated to
> "modern" interfaces and compilers.
> 
> I believe it could be done, and I believe it should be done. Rolled out
> carefully and explained well it would please far more than it offends, and
> is arguably far closer to correct than the current situation.

Seems to me that we have a decent, relatively current case study of
this with Python 2 vs Python 3.

Which is probably a good bit easier actually, because maintaining the
ability to run Python 2 applications while introducing the ability to
run Python 3 applications is likely easier than introducing changes to
how something as relatively fundamental as how ".." behaves in paths
while at the same time maintaining backwards compatibility with the
old behavior, where different users and different applications on the
same system might very well expect different behavior...

-- 
Michael Kjörling • https://michael.kjorling.se • michael@kjorling.se
 “Remember when, on the Internet, nobody cared that you were a dog?”


^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2021-12-31  5:16         ` Theodore Ts'o
  2021-12-31  5:21           ` Dan Cross
@ 2021-12-31  5:55           ` Rob Pike
  2021-12-31 13:32             ` Michael Kjörling
  1 sibling, 1 reply; 71+ messages in thread
From: Rob Pike @ 2021-12-31  5:55 UTC (permalink / raw)
  To: Theodore Ts'o; +Cc: TUHS main list, Noel Chiappa

[-- Attachment #1: Type: text/plain, Size: 4153 bytes --]

Thanks for your thoughtful response (and for reading the paper).

This for me is the catch:

*But there is a whole ecosystem of shell scripts, programs, user's finger
macros, etc., that would break if we "made things better" or made things
simpler.*

And of course you're right, but we've known how to roll out breaking
changes in software for a very long time now. And we do so often. Few Unix
programs from the past would compile and run today without being updated to
"modern" interfaces and compilers.

I believe it could be done, and I believe it should be done. Rolled out
carefully and explained well it would please far more than it offends, and
is arguably far closer to correct than the current situation.

Of course, I am not volunteering, sorry. Easy for me to snipe, I admit, but
then I paid my OS dues long ago.

-rob


On Fri, Dec 31, 2021 at 4:16 PM Theodore Ts'o <tytso@mit.edu> wrote:

> On Fri, Dec 31, 2021 at 02:23:15PM +1100, Rob Pike wrote:
> > In broad strokes, that's exactly what we did in Plan 9, with great
> results,
> > as was said here a while back. The paper ended with a plea to do the same
> > to Unix, which I think is quite doable.
> >
> > https://9p.io/sys/doc/lexnames.html
>
> I'm sure it *could* be done.  In fact, Linux has a number of the
> pieces of the implementations already, which could be reused for a
> Plan-9-style lexical namespace.  Linux has bind mounts, and a dentry is
> essentially a cached directory lookup (it contains a pointer to its
> parent dentry, the filename used to do the lookup and a pointer to the
> resulting inode).  And a struct file contains a struct dentry, which
> means in Linux it is possible to determine the path used to open a
> particular file.  There are some differences of course; so it's not
> exact, but I agree with you that it is quite "doable".  (In
> particular, how Linux dentries and symlinks interact appears to be
> different from how Plan 9 and Channels did things, if I understand the
> paper correctly.)
>
> The question though, is whether Linux *should* do it.  Or rather,
> *could* we do it without breaking backwards compatibility with users
> and application which expect that dot-dot works the way it does in
> classical Unix implementations for the past 4 decades or so.
>
> Granted that the combination of symlinks, and hard links, and in
> Linux, bind mounts (not to mention bind namespaces, chroots, and
> pivot_root operations), can be confusing.  But there is a whole
> ecosystem of shell scripts, programs, user's finger macros, etc., that
> would break if we "made things better" or "made things simpler".
>
> One of the benefits of research OS's is that you can experiment with
> things without having to worry about the howls of angry users when you
> break backwards compatibility, or eliminate some particular awkward
> API or feature.
>
>                                            - Ted
>
> P.S.  For example, how I wish I could eliminate telldir/seekdir, or at
> least change things so that telldir/seekdir used a substantially
> larger "cookie" to memoize a position in a directory's readdir stream
> which has to be valid effectively forever.  Today, you can pass a
> 32-bit telldir cookie to seekdir, and readdir MUST not return a
> directory entry that it had already returned, and MUST not skip a
> directory entry that was present at the time of the telldir().  Thus
> spake POSIX, and there are applications which depend on
> telldir/seekdir/readdir having this precise set of semantics
>
> If your directory is stored internally as a linear stream, that's not
> too bad; but if you're using a more complex data structure, life gets
> interesting.  I know of one file system (JFS) which added an extra,
> separate, on-disk b-tree structure to each directory just to be able
> to provide stable telldir/seekdir cookies.  (And of course, there's a
> performance impact to needing to update an extra b-tree each time a
> directory entry is added or removed.)
>
> With a research OS, it's at least possible to say, "Well, *that* was a
> mistake; let's pretend it never happened and let's see how much better
> life would be."  :-)
>

[-- Attachment #2: Type: text/html, Size: 5038 bytes --]

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2021-12-31  5:16         ` Theodore Ts'o
@ 2021-12-31  5:21           ` Dan Cross
  2021-12-31  5:55           ` Rob Pike
  1 sibling, 0 replies; 71+ messages in thread
From: Dan Cross @ 2021-12-31  5:21 UTC (permalink / raw)
  To: Theodore Ts'o; +Cc: TUHS main list

[-- Attachment #1: Type: text/plain, Size: 410 bytes --]

On Fri, Dec 31, 2021, 12:16 AM Theodore Ts'o <tytso@mit.edu> wrote:

> [snip]
> One of the benefits of research OS's is that you can experiment with
> things without having to worry about the howls of angry users when you
> break backwards compatibility, or eliminate some particular awkward
> API or feature.
>

Ah, I see you mercifully avoiding spending a lot of time on the 9fans list
...

        - Dan C.

[-- Attachment #2: Type: text/html, Size: 941 bytes --]

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2021-12-31  3:23       ` Rob Pike
@ 2021-12-31  5:16         ` Theodore Ts'o
  2021-12-31  5:21           ` Dan Cross
  2021-12-31  5:55           ` Rob Pike
  0 siblings, 2 replies; 71+ messages in thread
From: Theodore Ts'o @ 2021-12-31  5:16 UTC (permalink / raw)
  To: Rob Pike; +Cc: TUHS main list, Noel Chiappa

On Fri, Dec 31, 2021 at 02:23:15PM +1100, Rob Pike wrote:
> In broad strokes, that's exactly what we did in Plan 9, with great results,
> as was said here a while back. The paper ended with a plea to do the same
> to Unix, which I think is quite doable.
> 
> https://9p.io/sys/doc/lexnames.html

I'm sure it *could* be done.  In fact, Linux has a number of the
pieces of the implementations already, which could be reused for a
Plan-9-style lexical namespace.  Linux has bind mounts, and a dentry is
essentially a cached directory lookup (it contains a pointer to its
parent dentry, the filename used to do the lookup and a pointer to the
resulting inode).  And a struct file contains a struct dentry, which
means in Linux it is possible to determine the path used to open a
particular file.  There are some differences of course; so it's not
exact, but I agree with you that it is quite "doable".  (In
particular, how Linux dentries and symlinks interact appears to be
different from how Plan 9 and Channels did things, if I understand the
paper correctly.)

The question though, is whether Linux *should* do it.  Or rather,
*could* we do it without breaking backwards compatibility with users
and application which expect that dot-dot works the way it does in
classical Unix implementations for the past 4 decades or so.

Granted that the combination of symlinks, and hard links, and in
Linux, bind mounts (not to mention bind namespaces, chroots, and
pivot_root operations), can be confusing.  But there is a whole
ecosystem of shell scripts, programs, user's finger macros, etc., that
would break if we "made things better" or "made things simpler".

One of the benefits of research OS's is that you can experiment with
things without having to worry about the howls of angry users when you
break backwards compatibility, or eliminate some particular awkward
API or feature.

      	      	   	       	     	   - Ted

P.S.  For example, how I wish I could eliminate telldir/seekdir, or at
least change things so that telldir/seekdir used a substantially
larger "cookie" to memoize a position in a directory's readdir stream
which has to be valid effectively forever.  Today, you can pass a
32-bit telldir cookie to seekdir, and readdir MUST not return a
directory entry that it had already returned, and MUST not skip a
directory entry that was present at the time of the telldir().  Thus
spake POSIX, and there are applications which depend on
telldir/seekdir/readdir having this precise set of semantics

If your directory is stored internally as a linear stream, that's not
too bad; but if you're using a more complex data structure, life gets
interesting.  I know of one file system (JFS) which added an extra,
separate, on-disk b-tree structure to each directory just to be able
to provide stable telldir/seekdir cookies.  (And of course, there's a
performance impact to needing to update an extra b-tree each time a
directory entry is added or removed.)

With a research OS, it's at least possible to say, "Well, *that* was a
mistake; let's pretend it never happened and let's see how much better
life would be."  :-)

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2021-12-31  3:08     ` Theodore Ts'o
  2021-12-31  3:23       ` Rob Pike
@ 2021-12-31  5:12       ` Bakul Shah
  1 sibling, 0 replies; 71+ messages in thread
From: Bakul Shah @ 2021-12-31  5:12 UTC (permalink / raw)
  To: Theodore Ts'o; +Cc: TUHS main list, Noel Chiappa



> On Dec 30, 2021, at 7:08 PM, Theodore Ts'o <tytso@mit.edu> wrote:
> 
> On Thu, Dec 30, 2021 at 05:31:09PM -0500, Dan Cross wrote:
>> On Thu, Dec 30, 2021 at 11:41 AM Theodore Ts'o <tytso@mit.edu> wrote:
>>> On Wed, Dec 29, 2021 at 10:45:12PM -0500, Noel Chiappa wrote:
>>>>> From: Bakul Shah
>>>> 
>>>>> My guess is *not* storing a path instead of a ptr to the inode was done
>>>>> to save on memory.
>>>> 
>>>> More probably speed; those old disks were not fast, and on a PDP-11, disk
>>>> caches were so small that converting the path to the current directory to its
>>>> in memory inode could take a bunch of disk reads.
>>> 
>>> The other problem with storing the path as a string is that if
>>> higher-level directories get renamed, the path would become
>>> invalidated.  If you store the cwd as "/foo/bar/baz/quux", and someone
>>> renames "/foo/bar" to "/foo/sadness" the cwd-stored-as-a-string would
>>> become invalidated.
>> 
>> Why? Presumably as you traversed the filesystem, you'd cache, (path
>> component, inode) pairs and keep a ref on the inode.  For any given
>> file, including $CWD, you'd know it's pathname from the root as you
>> accessed it, but if it got renamed, it wouldn't matter because you'd
>> have cached a reference to the inode.
> 
> I was responding to Bakul's suggestion that the original Unix
> could/should have stored the cwd as a string, instead of a pointer to
> a directory inode.  If you stored the cwd as a string, then you could
> interpret .. by chopping off the last file name component of the
> string, and so now you could have hard links to directories and the
> file system tree could then be a file system DAG.
> 
> I don't think Bakul's proposal would have worked --- although I
> suppose if you disallowed directory renames, I guess it could.  

Note that this is already broken!

$ mkdir -p ~/a/b; cd ~/a/b; mv ~/a /tmp # /tmp is on a diff filesystem
$ cd ..
cd: no such file or directory: .. 

At least with the change[1] I was talking about, cd .. would not be
an error! At least with this change the behavior is more consistent!

Now if you want bug for bug compatibility then yes, you can't make any
changes but usually it is more a cost/benefit tradeoff as far as any
breaking changes are concerned. NFS broke a few things for example
and most of us put up with that.

[1] cwd as string, not allowing multiple links to a dir. I speculated
about the latter but any benefit is not worth the cost.
not worth it.




^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2021-12-31  3:08     ` Theodore Ts'o
@ 2021-12-31  3:23       ` Rob Pike
  2021-12-31  5:16         ` Theodore Ts'o
  2021-12-31  5:12       ` Bakul Shah
  1 sibling, 1 reply; 71+ messages in thread
From: Rob Pike @ 2021-12-31  3:23 UTC (permalink / raw)
  To: Theodore Ts'o; +Cc: TUHS main list, Noel Chiappa

[-- Attachment #1: Type: text/plain, Size: 2404 bytes --]

In broad strokes, that's exactly what we did in Plan 9, with great results,
as was said here a while back. The paper ended with a plea to do the same
to Unix, which I think is quite doable.

https://9p.io/sys/doc/lexnames.html

-rob


On Fri, Dec 31, 2021 at 2:09 PM Theodore Ts'o <tytso@mit.edu> wrote:

> On Thu, Dec 30, 2021 at 05:31:09PM -0500, Dan Cross wrote:
> > On Thu, Dec 30, 2021 at 11:41 AM Theodore Ts'o <tytso@mit.edu> wrote:
> > > On Wed, Dec 29, 2021 at 10:45:12PM -0500, Noel Chiappa wrote:
> > > >     > From: Bakul Shah
> > > >
> > > >     > My guess is *not* storing a path instead of a ptr to the inode
> was done
> > > >     > to save on memory.
> > > >
> > > > More probably speed; those old disks were not fast, and on a PDP-11,
> disk
> > > > caches were so small that converting the path to the current
> directory to its
> > > > in memory inode could take a bunch of disk reads.
> > >
> > > The other problem with storing the path as a string is that if
> > > higher-level directories get renamed, the path would become
> > > invalidated.  If you store the cwd as "/foo/bar/baz/quux", and someone
> > > renames "/foo/bar" to "/foo/sadness" the cwd-stored-as-a-string would
> > > become invalidated.
> >
> > Why? Presumably as you traversed the filesystem, you'd cache, (path
> > component, inode) pairs and keep a ref on the inode.  For any given
> > file, including $CWD, you'd know it's pathname from the root as you
> > accessed it, but if it got renamed, it wouldn't matter because you'd
> > have cached a reference to the inode.
>
> I was responding to Bakul's suggestion that the original Unix
> could/should have stored the cwd as a string, instead of a pointer to
> a directory inode.  If you stored the cwd as a string, then you could
> interpret .. by chopping off the last file name component of the
> string, and so now you could have hard links to directories and the
> file system tree could then be a file system DAG.
>
> I don't think Bakul's proposal would have worked --- although I
> suppose if you disallowed directory renames, I guess it could.  Noel's
> objection above was a performance one, in that if you had to parse the
> path every single time, you'd be much more dependent on the namei
> cache --- and that PDP-11 didn't have the memory to support a
> sufficiently large cache.
>
>                                                         - Ted
>
>

[-- Attachment #2: Type: text/html, Size: 3238 bytes --]

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2021-12-30 22:31   ` Dan Cross
  2021-12-31  0:43     ` Bakul Shah
@ 2021-12-31  3:08     ` Theodore Ts'o
  2021-12-31  3:23       ` Rob Pike
  2021-12-31  5:12       ` Bakul Shah
  1 sibling, 2 replies; 71+ messages in thread
From: Theodore Ts'o @ 2021-12-31  3:08 UTC (permalink / raw)
  To: Dan Cross; +Cc: TUHS main list, Noel Chiappa

On Thu, Dec 30, 2021 at 05:31:09PM -0500, Dan Cross wrote:
> On Thu, Dec 30, 2021 at 11:41 AM Theodore Ts'o <tytso@mit.edu> wrote:
> > On Wed, Dec 29, 2021 at 10:45:12PM -0500, Noel Chiappa wrote:
> > >     > From: Bakul Shah
> > >
> > >     > My guess is *not* storing a path instead of a ptr to the inode was done
> > >     > to save on memory.
> > >
> > > More probably speed; those old disks were not fast, and on a PDP-11, disk
> > > caches were so small that converting the path to the current directory to its
> > > in memory inode could take a bunch of disk reads.
> >
> > The other problem with storing the path as a string is that if
> > higher-level directories get renamed, the path would become
> > invalidated.  If you store the cwd as "/foo/bar/baz/quux", and someone
> > renames "/foo/bar" to "/foo/sadness" the cwd-stored-as-a-string would
> > become invalidated.
> 
> Why? Presumably as you traversed the filesystem, you'd cache, (path
> component, inode) pairs and keep a ref on the inode.  For any given
> file, including $CWD, you'd know it's pathname from the root as you
> accessed it, but if it got renamed, it wouldn't matter because you'd
> have cached a reference to the inode.

I was responding to Bakul's suggestion that the original Unix
could/should have stored the cwd as a string, instead of a pointer to
a directory inode.  If you stored the cwd as a string, then you could
interpret .. by chopping off the last file name component of the
string, and so now you could have hard links to directories and the
file system tree could then be a file system DAG.

I don't think Bakul's proposal would have worked --- although I
suppose if you disallowed directory renames, I guess it could.  Noel's
objection above was a performance one, in that if you had to parse the
path every single time, you'd be much more dependent on the namei
cache --- and that PDP-11 didn't have the memory to support a
sufficiently large cache.

							- Ted


^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2021-12-31  1:45         ` Bakul Shah
@ 2021-12-31  2:23           ` Adam Thornton
  0 siblings, 0 replies; 71+ messages in thread
From: Adam Thornton @ 2021-12-31  2:23 UTC (permalink / raw)
  To: Bakul Shah, Computer Old Farts Followers

[-- Attachment #1: Type: text/plain, Size: 4676 bytes --]

Moving to COFF, perhaps prematurely, but...

It feels weird to be a Unix native (which I consider myself: got my first
taste of Irix and SVR3 in 1989, went to college where it was a Sun-mostly
environment, started running Linux on my own machines in 1992 and never
stopped).  (For purposes of this discussion, of course Linux is Unix.)

It feels weird the same way it was weird when I was working for Express
Scripts, and then ESRX bought Medco, and all of a sudden we were the 500-lb
Gorilla.  That's why I left: we (particularly my little group) had been
doing some fairly cool and innovative stuff, and after that deal closed, we
switched over entirely to playing defense, and it got really boring really
fast.  My biggest win after that was showing that Pega ran perfectly fine
on Tomcat, which caused IBM to say something like "oh did we say $5 million
a year to license Websphere App Server?  Uh...we meant $50K."  So I saved
them a lot of money but it sucked to watch several months' work flushed
down the toilet, even though the savings to the company was many times my
salary for those months.

But the weird part is similar: Unix won.  Windows *lost*.  Sure, corporate
desktops still mostly run Windows, and those people who use it mostly hate
it.  But people who like using computers...use Macs (or, sure, Linux, and
then there are those weirdos like me who enjoy running all sorts of
ancient-or-niche-systems, many of which are Unix).  And all the people who
don't care do computing tasks on their phones, which are running either
Android--a Unix--or iOS--also a Unix.  It's ubiquitous.  It's the air you
breathe.  It's no longer strange to be a Unix user, it means you use a
21st-century electronic device.

And, sure, it's got its warts, but it's still basically the least-worst
thing out there.  And it continues to flabbergast me that a typesetting
system designed to run on single-processor 16-bit machines has, basically,
conquered the world.

Adam

P.S.  It's also about time, he said with a sigh of relief, having been an
OS/2 partisan, and a BeOS partisan, back in the day.  Nice to back a
winning horse for once.

On Thu, Dec 30, 2021 at 6:46 PM Bakul Shah <bakul@iitbombay.org> wrote:

> ?
>
> I was just explaining Ts'o's point, not agreeing with it. The first
> example I
> gave works just fine on plan9 (unlike on unix). And since it doesn't allow
> renames, the scenario T'so outlines can't happen there! But we were
> discussing Unix here.
>
> As for symlinks, if we have to have them, storing a path actually makes
> their
> use less surprising.
>
> We're in the 6th decade of Unix and we still suffer from unintended,
> fixable consequences of decisions made long long ago.
>
>
> No argument here. Perhaps you can suggest a path for fixing?
>
> On Dec 30, 2021, at 5:00 PM, Rob Pike <robpike@gmail.com> wrote:
>
> Grumpy hat on.
>
> Sometimes the Unix community suffers from the twin attitudes of a)
> believing if it can't be done perfectly, any improvement shouldn't be
> attempted at all and b) it's already done as well as is possible anyway.
>
> I disagree with both of these positions, obviously, but have given up
> pushing against them.
>
> We're in the 6th decade of Unix and we still suffer from unintended,
> fixable consequences of decisions made long long ago.
>
> Grumpy hat off.
>
> -rob
>
>
> On Fri, Dec 31, 2021 at 11:44 AM Bakul Shah <bakul@iitbombay.org> wrote:
>
>> On Dec 30, 2021, at 2:31 PM, Dan Cross <crossd@gmail.com> wrote:
>> >
>> > On Thu, Dec 30, 2021 at 11:41 AM Theodore Ts'o <tytso@mit.edu> wrote:
>> >>
>> >> The other problem with storing the path as a string is that if
>> >> higher-level directories get renamed, the path would become
>> >> invalidated.  If you store the cwd as "/foo/bar/baz/quux", and someone
>> >> renames "/foo/bar" to "/foo/sadness" the cwd-stored-as-a-string would
>> >> become invalidated.
>> >
>> > Why? Presumably as you traversed the filesystem, you'd cache, (path
>> > component, inode) pairs and keep a ref on the inode.  For any given
>> > file, including $CWD, you'd know it's pathname from the root as you
>> > accessed it, but if it got renamed, it wouldn't matter because you'd
>> > have cached a reference to the inode.
>>
>> Without the ".." entry you can't map a dir inode back to a path.
>> Note that something similar can happen even today:
>>
>> $ mkdir ~/a; cd ~/a; rm -rf ~/a; cd ..
>> cd: no such file or directory: ..
>>
>> $ mkdir -p ~/a/b; ln -s ~/a/b b; cd b; mv ~/a/b ~/a/c; cd ../b
>> ls: ../b: No such file or directory
>>
>> You can't protect the user from every such case. Storing a path
>> instead of the cwd inode simply changes the symptoms.
>>
>>
>>
>

[-- Attachment #2: Type: text/html, Size: 6339 bytes --]

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2021-12-31  1:00       ` Rob Pike
@ 2021-12-31  1:45         ` Bakul Shah
  2021-12-31  2:23           ` Adam Thornton
  0 siblings, 1 reply; 71+ messages in thread
From: Bakul Shah @ 2021-12-31  1:45 UTC (permalink / raw)
  To: Rob Pike; +Cc: TUHS main list, Noel Chiappa

[-- Attachment #1: Type: text/plain, Size: 2553 bytes --]

?

I was just explaining Ts'o's point, not agreeing with it. The first example I
gave works just fine on plan9 (unlike on unix). And since it doesn't allow
renames, the scenario T'so outlines can't happen there! But we were
discussing Unix here.

As for symlinks, if we have to have them, storing a path actually makes their
use less surprising.

> We're in the 6th decade of Unix and we still suffer from unintended, fixable consequences of decisions made long long ago.


No argument here. Perhaps you can suggest a path for fixing?

> On Dec 30, 2021, at 5:00 PM, Rob Pike <robpike@gmail.com> wrote:
> 
> Grumpy hat on.
> 
> Sometimes the Unix community suffers from the twin attitudes of a) believing if it can't be done perfectly, any improvement shouldn't be attempted at all and b) it's already done as well as is possible anyway.
> 
> I disagree with both of these positions, obviously, but have given up pushing against them.
> 
> We're in the 6th decade of Unix and we still suffer from unintended, fixable consequences of decisions made long long ago.
> 
> Grumpy hat off.
> 
> -rob
> 
> 
> On Fri, Dec 31, 2021 at 11:44 AM Bakul Shah <bakul@iitbombay.org <mailto:bakul@iitbombay.org>> wrote:
> On Dec 30, 2021, at 2:31 PM, Dan Cross <crossd@gmail.com <mailto:crossd@gmail.com>> wrote:
> > 
> > On Thu, Dec 30, 2021 at 11:41 AM Theodore Ts'o <tytso@mit.edu <mailto:tytso@mit.edu>> wrote:
> >> 
> >> The other problem with storing the path as a string is that if
> >> higher-level directories get renamed, the path would become
> >> invalidated.  If you store the cwd as "/foo/bar/baz/quux", and someone
> >> renames "/foo/bar" to "/foo/sadness" the cwd-stored-as-a-string would
> >> become invalidated.
> > 
> > Why? Presumably as you traversed the filesystem, you'd cache, (path
> > component, inode) pairs and keep a ref on the inode.  For any given
> > file, including $CWD, you'd know it's pathname from the root as you
> > accessed it, but if it got renamed, it wouldn't matter because you'd
> > have cached a reference to the inode.
> 
> Without the ".." entry you can't map a dir inode back to a path.
> Note that something similar can happen even today:
> 
> $ mkdir ~/a; cd ~/a; rm -rf ~/a; cd ..
> cd: no such file or directory: ..
> 
> $ mkdir -p ~/a/b; ln -s ~/a/b b; cd b; mv ~/a/b ~/a/c; cd ../b
> ls: ../b: No such file or directory
> 
> You can't protect the user from every such case. Storing a path
> instead of the cwd inode simply changes the symptoms. 
> 
> 


[-- Attachment #2: Type: text/html, Size: 4397 bytes --]

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2021-12-31  0:43     ` Bakul Shah
@ 2021-12-31  1:00       ` Rob Pike
  2021-12-31  1:45         ` Bakul Shah
  2021-12-31 18:56       ` Chet Ramey
  1 sibling, 1 reply; 71+ messages in thread
From: Rob Pike @ 2021-12-31  1:00 UTC (permalink / raw)
  To: Bakul Shah; +Cc: TUHS main list, Noel Chiappa

[-- Attachment #1: Type: text/plain, Size: 1772 bytes --]

Grumpy hat on.

Sometimes the Unix community suffers from the twin attitudes of a)
believing if it can't be done perfectly, any improvement shouldn't be
attempted at all and b) it's already done as well as is possible anyway.

I disagree with both of these positions, obviously, but have given up
pushing against them.

We're in the 6th decade of Unix and we still suffer from unintended,
fixable consequences of decisions made long long ago.

Grumpy hat off.

-rob


On Fri, Dec 31, 2021 at 11:44 AM Bakul Shah <bakul@iitbombay.org> wrote:

> On Dec 30, 2021, at 2:31 PM, Dan Cross <crossd@gmail.com> wrote:
> >
> > On Thu, Dec 30, 2021 at 11:41 AM Theodore Ts'o <tytso@mit.edu> wrote:
> >>
> >> The other problem with storing the path as a string is that if
> >> higher-level directories get renamed, the path would become
> >> invalidated.  If you store the cwd as "/foo/bar/baz/quux", and someone
> >> renames "/foo/bar" to "/foo/sadness" the cwd-stored-as-a-string would
> >> become invalidated.
> >
> > Why? Presumably as you traversed the filesystem, you'd cache, (path
> > component, inode) pairs and keep a ref on the inode.  For any given
> > file, including $CWD, you'd know it's pathname from the root as you
> > accessed it, but if it got renamed, it wouldn't matter because you'd
> > have cached a reference to the inode.
>
> Without the ".." entry you can't map a dir inode back to a path.
> Note that something similar can happen even today:
>
> $ mkdir ~/a; cd ~/a; rm -rf ~/a; cd ..
> cd: no such file or directory: ..
>
> $ mkdir -p ~/a/b; ln -s ~/a/b b; cd b; mv ~/a/b ~/a/c; cd ../b
> ls: ../b: No such file or directory
>
> You can't protect the user from every such case. Storing a path
> instead of the cwd inode simply changes the symptoms.
>
>
>

[-- Attachment #2: Type: text/html, Size: 2547 bytes --]

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2021-12-30 22:31   ` Dan Cross
@ 2021-12-31  0:43     ` Bakul Shah
  2021-12-31  1:00       ` Rob Pike
  2021-12-31 18:56       ` Chet Ramey
  2021-12-31  3:08     ` Theodore Ts'o
  1 sibling, 2 replies; 71+ messages in thread
From: Bakul Shah @ 2021-12-31  0:43 UTC (permalink / raw)
  To: Dan Cross; +Cc: TUHS main list, Noel Chiappa

On Dec 30, 2021, at 2:31 PM, Dan Cross <crossd@gmail.com> wrote:
> 
> On Thu, Dec 30, 2021 at 11:41 AM Theodore Ts'o <tytso@mit.edu> wrote:
>> 
>> The other problem with storing the path as a string is that if
>> higher-level directories get renamed, the path would become
>> invalidated.  If you store the cwd as "/foo/bar/baz/quux", and someone
>> renames "/foo/bar" to "/foo/sadness" the cwd-stored-as-a-string would
>> become invalidated.
> 
> Why? Presumably as you traversed the filesystem, you'd cache, (path
> component, inode) pairs and keep a ref on the inode.  For any given
> file, including $CWD, you'd know it's pathname from the root as you
> accessed it, but if it got renamed, it wouldn't matter because you'd
> have cached a reference to the inode.

Without the ".." entry you can't map a dir inode back to a path.
Note that something similar can happen even today:

$ mkdir ~/a; cd ~/a; rm -rf ~/a; cd ..
cd: no such file or directory: ..

$ mkdir -p ~/a/b; ln -s ~/a/b b; cd b; mv ~/a/b ~/a/c; cd ../b
ls: ../b: No such file or directory

You can't protect the user from every such case. Storing a path
instead of the cwd inode simply changes the symptoms. 



^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2021-12-30 18:02           ` John Cowan
@ 2021-12-30 23:04             ` Richard Salz
  0 siblings, 0 replies; 71+ messages in thread
From: Richard Salz @ 2021-12-30 23:04 UTC (permalink / raw)
  To: John Cowan; +Cc: TUHS main list

[-- Attachment #1: Type: text/plain, Size: 308 bytes --]

> In the new days, rsync -a dir1/ dir2 is the ticket ...
> If you rsync more than once, only the files in dir1 that have changed are
> copied.
>

More then that.  It uses a clever algorithm to send over only changes.  It
was the basis of Tridgedell's PhD thesis.
https://www.samba.org/~tridge/phd_thesis.pdf

[-- Attachment #2: Type: text/html, Size: 822 bytes --]

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2021-12-30 16:40 ` Theodore Ts'o
@ 2021-12-30 22:31   ` Dan Cross
  2021-12-31  0:43     ` Bakul Shah
  2021-12-31  3:08     ` Theodore Ts'o
  0 siblings, 2 replies; 71+ messages in thread
From: Dan Cross @ 2021-12-30 22:31 UTC (permalink / raw)
  To: Theodore Ts'o; +Cc: TUHS main list, Noel Chiappa

On Thu, Dec 30, 2021 at 11:41 AM Theodore Ts'o <tytso@mit.edu> wrote:
> On Wed, Dec 29, 2021 at 10:45:12PM -0500, Noel Chiappa wrote:
> >     > From: Bakul Shah
> >
> >     > My guess is *not* storing a path instead of a ptr to the inode was done
> >     > to save on memory.
> >
> > More probably speed; those old disks were not fast, and on a PDP-11, disk
> > caches were so small that converting the path to the current directory to its
> > in memory inode could take a bunch of disk reads.
>
> The other problem with storing the path as a string is that if
> higher-level directories get renamed, the path would become
> invalidated.  If you store the cwd as "/foo/bar/baz/quux", and someone
> renames "/foo/bar" to "/foo/sadness" the cwd-stored-as-a-string would
> become invalidated.

Why? Presumably as you traversed the filesystem, you'd cache, (path
component, inode) pairs and keep a ref on the inode.  For any given
file, including $CWD, you'd know it's pathname from the root as you
accessed it, but if it got renamed, it wouldn't matter because you'd
have cached a reference to the inode.

> Sure, you could scan the pwd of all the processes running on the
> system and do edit all of the strings, but once you go SMP, the
> potential locking pitfalls would be numerous.

I don't see the need for that. What am I missing?

        - Dan C.

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2021-12-30  5:14         ` Dan Stromberg
  2021-12-30 16:22           ` Clem Cole
@ 2021-12-30 18:02           ` John Cowan
  2021-12-30 23:04             ` Richard Salz
  1 sibling, 1 reply; 71+ messages in thread
From: John Cowan @ 2021-12-30 18:02 UTC (permalink / raw)
  To: Dan Stromberg; +Cc: TUHS main list

[-- Attachment #1: Type: text/plain, Size: 785 bytes --]

On Thu, Dec 30, 2021 at 12:15 AM Dan Stromberg <drsalists@gmail.com> wrote:

FWIW: in the old days, I just used  tar cf - . | (cd some_place_else; tar
>> xvpf - )  preserving permissions.
>>
>
In the new days, rsync -a dir1/ dir2 is the ticket (derived from "is
etiquette", if you can believe it).  You need the terminating slash to
prevent dir1 from becoming a subdirectory of dir2; dir1/* would of course
not copy dotfiles.  (I grew weary of typing "cp -r dir1/.[a-zA-Z]* dir2" to
copy dotfiles and dotdirectories left behind.)  The -a flag ensures
permissions, symlinks, etc. are preserved.

If you rsync more than once, only the files in dir1 that have changed are
copied.  You can also use user@host:path as either argument to make it
work over ssh, which is the common use case.

[-- Attachment #2: Type: text/html, Size: 1938 bytes --]

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2021-12-30  3:45 Noel Chiappa
  2021-12-30  4:02 ` Bakul Shah
@ 2021-12-30 16:40 ` Theodore Ts'o
  2021-12-30 22:31   ` Dan Cross
  1 sibling, 1 reply; 71+ messages in thread
From: Theodore Ts'o @ 2021-12-30 16:40 UTC (permalink / raw)
  To: Noel Chiappa; +Cc: tuhs

On Wed, Dec 29, 2021 at 10:45:12PM -0500, Noel Chiappa wrote:
>     > From: Bakul Shah
> 
>     > My guess is *not* storing a path instead of a ptr to the inode was done
>     > to save on memory.
> 
> More probably speed; those old disks were not fast, and on a PDP-11, disk
> caches were so small that converting the path to the current directory to its
> in memory inode could take a bunch of disk reads.

The other problem with storing the path as a string is that if
higher-level directories get renamed, the path would become
invalidated.  If you store the cwd as "/foo/bar/baz/quux", and someone
renames "/foo/bar" to "/foo/sadness" the cwd-stored-as-a-string would
become invalidated.

Sure, you could scan the pwd of all the processes running on the
system and do edit all of the strings, but once you go SMP, the
potential locking pitfalls would be numerous.

						- Ted

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2021-12-30  5:14         ` Dan Stromberg
@ 2021-12-30 16:22           ` Clem Cole
  2021-12-30 18:02           ` John Cowan
  1 sibling, 0 replies; 71+ messages in thread
From: Clem Cole @ 2021-12-30 16:22 UTC (permalink / raw)
  To: Dan Stromberg; +Cc: TUHS main list

[-- Attachment #1: Type: text/plain, Size: 745 bytes --]

sure - but old habits die hard (and Ken's man page for tar shows the form I
use.  I programmed my fingers to his example in 1978 I suspect).

On Thu, Dec 30, 2021 at 12:14 AM Dan Stromberg <drsalists@gmail.com> wrote:

>
> On Wed, Dec 29, 2021 at 8:18 AM Clem Cole <clemc@ccc.com> wrote:
>
>> Where and when did mvdir(1) appear?  It's not in Research or BSD.
>>
>> FWIW: in the old days, I just used  tar cf - . | (cd some_place_else;
>> tar xvpf - )  preserving permissions.  The hardlink scheme of cpio was
>> the one thing it could do that the original tar did not.
>>
>
> I'm partial to:
> (cd /source/dir && tar cf - .) | (cd /dest/dir && tar xfp -)
>
> The &&'s make it so a typo is less likely to do something odd or even
> damaging.
>
>

[-- Attachment #2: Type: text/html, Size: 1898 bytes --]

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2021-12-29 16:17       ` Clem Cole
  2021-12-29 16:58         ` Richard Salz
@ 2021-12-30  5:14         ` Dan Stromberg
  2021-12-30 16:22           ` Clem Cole
  2021-12-30 18:02           ` John Cowan
  1 sibling, 2 replies; 71+ messages in thread
From: Dan Stromberg @ 2021-12-30  5:14 UTC (permalink / raw)
  To: Clem Cole; +Cc: TUHS main list

[-- Attachment #1: Type: text/plain, Size: 501 bytes --]

On Wed, Dec 29, 2021 at 8:18 AM Clem Cole <clemc@ccc.com> wrote:

> Where and when did mvdir(1) appear?  It's not in Research or BSD.
>
> FWIW: in the old days, I just used  tar cf - . | (cd some_place_else; tar
> xvpf - )  preserving permissions.  The hardlink scheme of cpio was the
> one thing it could do that the original tar did not.
>

I'm partial to:
(cd /source/dir && tar cf - .) | (cd /dest/dir && tar xfp -)

The &&'s make it so a typo is less likely to do something odd or even
damaging.

[-- Attachment #2: Type: text/html, Size: 1198 bytes --]

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2021-12-30  3:45 Noel Chiappa
@ 2021-12-30  4:02 ` Bakul Shah
  2021-12-30 16:40 ` Theodore Ts'o
  1 sibling, 0 replies; 71+ messages in thread
From: Bakul Shah @ 2021-12-30  4:02 UTC (permalink / raw)
  To: Noel Chiappa; +Cc: tuhs

On Dec 29, 2021, at 7:45 PM, Noel Chiappa <jnc@mercury.lcs.mit.edu> wrote:
> 
>> From: Bakul Shah
> 
>> My guess is *not* storing a path instead of a ptr to the inode was done
>> to save on memory.
> 
> More probably speed; those old disks were not fast, and on a PDP-11, disk
> caches were so small that converting the path to the current directory to its
> in memory inode could take a bunch of disk reads.

Good point!

>> Every inode has a linkcount so detecting when the last conn. is severed
>> not a problem.
> 
> Depends; if a directory _has_ to be empty before it can be deleted, maybe; but
> if not, no. (Consider if /a/b/c/d exists, and /a/b is removed; the tree
> underneath it has to be walked and the components deleted. That could take a
> while...) In the general case (e.g. without the restriction to a tree), it's
> basically the same problem as garbage collection in LISP.

Indeed. You can detect circularly linked nodes when you try to delete
sub directories. Then you have to ask the user whether to still delete it!
In Lisp you can simply NIL a node and let the GC take care of it.... May be
you can do the same with "rm -rf" but it may have a surprising/unintended
outcome! An important point I missed. Thanks.

> 
> 	  Noel


^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
@ 2021-12-30  3:45 Noel Chiappa
  2021-12-30  4:02 ` Bakul Shah
  2021-12-30 16:40 ` Theodore Ts'o
  0 siblings, 2 replies; 71+ messages in thread
From: Noel Chiappa @ 2021-12-30  3:45 UTC (permalink / raw)
  To: tuhs; +Cc: jnc

    > From: Bakul Shah

    > My guess is *not* storing a path instead of a ptr to the inode was done
    > to save on memory.

More probably speed; those old disks were not fast, and on a PDP-11, disk
caches were so small that converting the path to the current directory to its
in memory inode could take a bunch of disk reads.

    > Every inode has a linkcount so detecting when the last conn. is severed
    > not a problem.

Depends; if a directory _has_ to be empty before it can be deleted, maybe; but
if not, no. (Consider if /a/b/c/d exists, and /a/b is removed; the tree
underneath it has to be walked and the components deleted. That could take a
while...) In the general case (e.g. without the restriction to a tree), it's
basically the same problem as garbage collection in LISP.

	  Noel

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2021-12-29 19:33 Noel Chiappa
@ 2021-12-30  3:40 ` Jay Logue via TUHS
  0 siblings, 0 replies; 71+ messages in thread
From: Jay Logue via TUHS @ 2021-12-30  3:40 UTC (permalink / raw)
  To: tuhs

[-- Attachment #1: Type: text/plain, Size: 2727 bytes --]

v6 mv is indeed limited to renaming directories.  However, mv on v7 is 
able to move directories. Considering the similarity between the code 
bases, I think it would be fairly straightforward to support directory 
move on v6 in the same manner as was done in v7.  (On a related tangent, 
I found it fairly straightforward to implement modern rename(2) 
semantics on top of both code bases in my retro-fuse filesystem [e.g. 
v6fs.c:559 
<https://github.com/jaylogue/retro-fuse/blob/b300865c1aa4c38930adea66de364f182c73b3b5/src/v6fs.c#L599>].  
Of course I had the benefit of a single-threaded execution environment.)

Given the way mkdir and directory renaming worked in v6/v7, I'm not sure 
the lack of atomicity in directory move would have made things any worse 
for operators. It's interesting though that dcheck does not look for 
malformed directory links.

--Jay

On 12/29/2021 11:33 AM, Noel Chiappa wrote:
>      > From: Clem Cole
>
>      > Try it on V6 or V7 and you will get 'directory exists' as an error.
>
> The V6 'mv' is prepared to move a directory _within_ a directory (i.e.
> 'rename' functionality). I'm not sure why it's not prepared to move within
> a partition; probably whoever wrote it wasn't prepared to deal with all the
> extra work for that (unlink from the old '..', then link to the '..' in the
> new directory, etc, etc).
>
> (The MIT PWB1 had a 'mvdir' written by Jeff Schiller, so PWB1 didn't have
> 'move directory' functionality before that. MIT must have been using the PWB1
> system for 6.031, which I didn't recall; the comments in 'mvdir' refer to it
> being used there.)
>
> The V6 'mv' is fairly complicated (as I found out when I tried to modify it
> to use 'smdate()', so that moving a file didn't change its 'last write'
> date). Oddly enough, it is prepared to do cross-partition 'moves' (it forks a
> 'cp' to do the move). Although on V6, 'cp' only does one file; 'cp *.c
> {dest}' was not supported, there was 'cpall' for that. (Why no 'mvall', I
> wonder? It would have been trivial to clone 'cpall'.)
>
> Run fact; the V6 'mv' is the place that has the famous (?)  "values of B will
>   give rise to dom!" error message (in the directory-moing section).
>
>      > if the BSD mv command for 4.1 supported it. If it did then it was not
>      > atomic -- it would have had to create the new directory, move the
>      > contents independently and then remove the old one.
>
> Speaking of atomic operation, in V6 'mkdir' (not being a system call) was
> not atomic, so if interrupted at 'just the right time', it could leave
> the FS in an inconsistent state. That's the best reason I've come across
> to make 'mkdir' a system call - it can be made atomic that way.
>
> 	Noel

[-- Attachment #2: Type: text/html, Size: 3169 bytes --]

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2021-12-29 18:27       ` ron minnich
@ 2021-12-29 20:59         ` Rob Pike
  0 siblings, 0 replies; 71+ messages in thread
From: Rob Pike @ 2021-12-29 20:59 UTC (permalink / raw)
  To: ron minnich; +Cc: bakul, tuhs

[-- Attachment #1: Type: text/plain, Size: 4243 bytes --]

Not an answer, but relevant: Apple's Time Machine works by hard-linking
directories to make the forest of old roots.

-rob


On Thu, Dec 30, 2021 at 5:28 AM ron minnich <rminnich@gmail.com> wrote:

> since .. came up, nobody mentioned it yet,
> https://plan9.io/sys/doc/lexnames.pdf
>
> but the subject line is "moving directories" and, as pointed out, on
> old school file systems with hard links, it's not a move, just an ln,
> more or less. And in the easy case, it's easy; and in the not so easy
> case, it can be impossible ...
>
> On newer systems, which don't have such notions as hard links, it's a
> different game, and on many, it may not be possible; dircp to the
> rescue, with all the pain that implies (doesn't go well when you are
> in CA and the server is in NJ, trust me :-)
>
> But this brings up a question I forgot: what was the last Unix version
> that let users make arbitrary links, such that the file system was no
> longer a DAG? I recall in v6 days hearing that earlier Unix allowed
> this, and that cleanup (via icheck and friends) got to be near
> impossible.
>
> On Wed, Dec 29, 2021 at 9:50 AM Brantley Coile <brantley@coraid.com>
> wrote:
> >
> > Plan 9 can't move directories with mv. I will only change the name of
> them.
> > (If this is the question. I was only half paying attention to the
> thread. Sorry)
> >
> >         --bwc
> >
> > cessna% mkdir dira
> > cessna% mkdir dirb
> > cessna% touch dira/a
> > cessna% touch dirb/b
> > cessna% mv dira dirb
> > mv: can't remove ./dirb: remove -- directory not empty
> >
> > To move contents of directories we use dircp.
> >
> > cessna% man dircp
> >
> >      TAR(1)                                                     TAR(1)
> >
> >      NAME
> >           tar, dircp - archiver
> >
> >      SYNOPSIS
> >           tar key [ file ... ]
> >
> >           dircp fromdir todir
> >
> >      DESCRIPTION
> >           Tar saves and restores file trees.  It is most often used to
> >           transport a tree of files from one system to another.  The
> >
> >         ...
> >                and .tz.  If no extension matches, gzip is used.  The z
> >                flag is unnecessary (but allowed) when using the t and
> >                x verbs on archives with recognized extensions.
> >
> >      EXAMPLES
> >           Tar can be used to copy hierarchies thus:
> >
> >                @{cd fromdir && tar c .} | @{cd todir && tar xT}
> >
> >           Dircp does this.
> >
> >      SOURCE
> >           /sys/src/cmd/tar.c
> >           /rc/bin/dircp
> >
> >      SEE ALSO
> >           ar(1), bundle(1), tapefs(4), mkfs(8)
> >
> >
> >
> > cessna%
> > > On Dec 29, 2021, at 12:38 PM, Clem Cole <clemc@ccc.com> wrote:
> > >
> > > At the risk of kicking a dead horse too much ...
> > >
> > > On Wed, Dec 29, 2021 at 12:14 PM <arnold@skeeve.com> wrote:
> > > Plan 9 eventually did something like this. I don't remember the
> details.
> > > Arnold - point taken but ... unfortunately, I think that the
> comparison is a little difficult because Plan9's concepts of namespaces is
> a tad different than UNIX's.  But I'll let Rob or Ken comment as they lived
> and developed both systems.
> > >
> > > FWIW: An object store is something that retains information after the
> processes that operates on it complete - i.e. its a static entity.  Links
> were (are) also a static concept.   Late binding to names (like symlinks)
> are a dynamic (runtime idea).  Bakul points out that by using the per
> process u area, the dynamic context can be retained.  The observation is
> that .. (like symlinks) tend to be a runtime (dynamic) notion, although I'm
> not sure how you keep consistency in the static FS if you don't store the
> link in the inode.  As someone that did, I suggest - try writing fsck if
> you are using dynamic content.   How do you know?  I'd still claim it is
> the same issue.
> > >
> > > Anyway, I suppose that like context sensitive symlinks (which I sorely
> miss), you could do this and keep a list of the N inodes for the N parents
> and then like CDSL's keep the one you used to get there in the u area so
> that .. picks the proper one on the way out and you can still have the
> static notion which something like fsck can check off line.
> >
>

[-- Attachment #2: Type: text/html, Size: 5519 bytes --]

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
@ 2021-12-29 19:33 Noel Chiappa
  2021-12-30  3:40 ` Jay Logue via TUHS
  0 siblings, 1 reply; 71+ messages in thread
From: Noel Chiappa @ 2021-12-29 19:33 UTC (permalink / raw)
  To: tuhs; +Cc: jnc

    > From: Clem Cole

    > Try it on V6 or V7 and you will get 'directory exists' as an error.

The V6 'mv' is prepared to move a directory _within_ a directory (i.e.
'rename' functionality). I'm not sure why it's not prepared to move within
a partition; probably whoever wrote it wasn't prepared to deal with all the
extra work for that (unlink from the old '..', then link to the '..' in the
new directory, etc, etc).

(The MIT PWB1 had a 'mvdir' written by Jeff Schiller, so PWB1 didn't have
'move directory' functionality before that. MIT must have been using the PWB1
system for 6.031, which I didn't recall; the comments in 'mvdir' refer to it
being used there.)

The V6 'mv' is fairly complicated (as I found out when I tried to modify it
to use 'smdate()', so that moving a file didn't change its 'last write'
date). Oddly enough, it is prepared to do cross-partition 'moves' (it forks a
'cp' to do the move). Although on V6, 'cp' only does one file; 'cp *.c
{dest}' was not supported, there was 'cpall' for that. (Why no 'mvall', I
wonder? It would have been trivial to clone 'cpall'.)

Run fact; the V6 'mv' is the place that has the famous (?)  "values of B will
 give rise to dom!" error message (in the directory-moing section).

    > if the BSD mv command for 4.1 supported it. If it did then it was not
    > atomic -- it would have had to create the new directory, move the
    > contents independently and then remove the old one.

Speaking of atomic operation, in V6 'mkdir' (not being a system call) was
not atomic, so if interrupted at 'just the right time', it could leave
the FS in an inconsistent state. That's the best reason I've come across
to make 'mkdir' a system call - it can be made atomic that way.

	Noel

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2021-12-29 17:49     ` Brantley Coile
@ 2021-12-29 18:27       ` ron minnich
  2021-12-29 20:59         ` Rob Pike
  0 siblings, 1 reply; 71+ messages in thread
From: ron minnich @ 2021-12-29 18:27 UTC (permalink / raw)
  To: Brantley Coile; +Cc: bakul, tuhs

since .. came up, nobody mentioned it yet, https://plan9.io/sys/doc/lexnames.pdf

but the subject line is "moving directories" and, as pointed out, on
old school file systems with hard links, it's not a move, just an ln,
more or less. And in the easy case, it's easy; and in the not so easy
case, it can be impossible ...

On newer systems, which don't have such notions as hard links, it's a
different game, and on many, it may not be possible; dircp to the
rescue, with all the pain that implies (doesn't go well when you are
in CA and the server is in NJ, trust me :-)

But this brings up a question I forgot: what was the last Unix version
that let users make arbitrary links, such that the file system was no
longer a DAG? I recall in v6 days hearing that earlier Unix allowed
this, and that cleanup (via icheck and friends) got to be near
impossible.

On Wed, Dec 29, 2021 at 9:50 AM Brantley Coile <brantley@coraid.com> wrote:
>
> Plan 9 can't move directories with mv. I will only change the name of them.
> (If this is the question. I was only half paying attention to the thread. Sorry)
>
>         --bwc
>
> cessna% mkdir dira
> cessna% mkdir dirb
> cessna% touch dira/a
> cessna% touch dirb/b
> cessna% mv dira dirb
> mv: can't remove ./dirb: remove -- directory not empty
>
> To move contents of directories we use dircp.
>
> cessna% man dircp
>
>      TAR(1)                                                     TAR(1)
>
>      NAME
>           tar, dircp - archiver
>
>      SYNOPSIS
>           tar key [ file ... ]
>
>           dircp fromdir todir
>
>      DESCRIPTION
>           Tar saves and restores file trees.  It is most often used to
>           transport a tree of files from one system to another.  The
>
>         ...
>                and .tz.  If no extension matches, gzip is used.  The z
>                flag is unnecessary (but allowed) when using the t and
>                x verbs on archives with recognized extensions.
>
>      EXAMPLES
>           Tar can be used to copy hierarchies thus:
>
>                @{cd fromdir && tar c .} | @{cd todir && tar xT}
>
>           Dircp does this.
>
>      SOURCE
>           /sys/src/cmd/tar.c
>           /rc/bin/dircp
>
>      SEE ALSO
>           ar(1), bundle(1), tapefs(4), mkfs(8)
>
>
>
> cessna%
> > On Dec 29, 2021, at 12:38 PM, Clem Cole <clemc@ccc.com> wrote:
> >
> > At the risk of kicking a dead horse too much ...
> >
> > On Wed, Dec 29, 2021 at 12:14 PM <arnold@skeeve.com> wrote:
> > Plan 9 eventually did something like this. I don't remember the details.
> > Arnold - point taken but ... unfortunately, I think that the comparison is a little difficult because Plan9's concepts of namespaces is a tad different than UNIX's.  But I'll let Rob or Ken comment as they lived and developed both systems.
> >
> > FWIW: An object store is something that retains information after the processes that operates on it complete - i.e. its a static entity.  Links were (are) also a static concept.   Late binding to names (like symlinks) are a dynamic (runtime idea).  Bakul points out that by using the per process u area, the dynamic context can be retained.  The observation is that .. (like symlinks) tend to be a runtime (dynamic) notion, although I'm not sure how you keep consistency in the static FS if you don't store the link in the inode.  As someone that did, I suggest - try writing fsck if you are using dynamic content.   How do you know?  I'd still claim it is the same issue.
> >
> > Anyway, I suppose that like context sensitive symlinks (which I sorely miss), you could do this and keep a list of the N inodes for the N parents and then like CDSL's keep the one you used to get there in the u area so that .. picks the proper one on the way out and you can still have the static notion which something like fsck can check off line.
>

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2021-12-29 17:38   ` Clem Cole
@ 2021-12-29 17:49     ` Brantley Coile
  2021-12-29 18:27       ` ron minnich
  0 siblings, 1 reply; 71+ messages in thread
From: Brantley Coile @ 2021-12-29 17:49 UTC (permalink / raw)
  To: Clem Cole; +Cc: bakul, tuhs

Plan 9 can't move directories with mv. I will only change the name of them.
(If this is the question. I was only half paying attention to the thread. Sorry)

	--bwc

cessna% mkdir dira
cessna% mkdir dirb
cessna% touch dira/a
cessna% touch dirb/b
cessna% mv dira dirb
mv: can't remove ./dirb: remove -- directory not empty

To move contents of directories we use dircp.

cessna% man dircp

     TAR(1)                                                     TAR(1)

     NAME
          tar, dircp - archiver

     SYNOPSIS
          tar key [ file ... ]

          dircp fromdir todir

     DESCRIPTION
          Tar saves and restores file trees.  It is most often used to
          transport a tree of files from one system to another.  The

 	...
               and .tz.  If no extension matches, gzip is used.  The z
               flag is unnecessary (but allowed) when using the t and
               x verbs on archives with recognized extensions.

     EXAMPLES
          Tar can be used to copy hierarchies thus:

               @{cd fromdir && tar c .} | @{cd todir && tar xT}

          Dircp does this.

     SOURCE
          /sys/src/cmd/tar.c
          /rc/bin/dircp

     SEE ALSO
          ar(1), bundle(1), tapefs(4), mkfs(8)



cessna% 
> On Dec 29, 2021, at 12:38 PM, Clem Cole <clemc@ccc.com> wrote:
> 
> At the risk of kicking a dead horse too much ...
> 
> On Wed, Dec 29, 2021 at 12:14 PM <arnold@skeeve.com> wrote:
> Plan 9 eventually did something like this. I don't remember the details.
> Arnold - point taken but ... unfortunately, I think that the comparison is a little difficult because Plan9's concepts of namespaces is a tad different than UNIX's.  But I'll let Rob or Ken comment as they lived and developed both systems.
> 
> FWIW: An object store is something that retains information after the processes that operates on it complete - i.e. its a static entity.  Links were (are) also a static concept.   Late binding to names (like symlinks) are a dynamic (runtime idea).  Bakul points out that by using the per process u area, the dynamic context can be retained.  The observation is that .. (like symlinks) tend to be a runtime (dynamic) notion, although I'm not sure how you keep consistency in the static FS if you don't store the link in the inode.  As someone that did, I suggest - try writing fsck if you are using dynamic content.   How do you know?  I'd still claim it is the same issue.
> 
> Anyway, I suppose that like context sensitive symlinks (which I sorely miss), you could do this and keep a list of the N inodes for the N parents and then like CDSL's keep the one you used to get there in the u area so that .. picks the proper one on the way out and you can still have the static notion which something like fsck can check off line.


^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2021-12-29 17:14 ` arnold
@ 2021-12-29 17:38   ` Clem Cole
  2021-12-29 17:49     ` Brantley Coile
  0 siblings, 1 reply; 71+ messages in thread
From: Clem Cole @ 2021-12-29 17:38 UTC (permalink / raw)
  To: arnold; +Cc: bakul, tuhs

[-- Attachment #1: Type: text/plain, Size: 1456 bytes --]

At the risk of kicking a dead horse too much ...

On Wed, Dec 29, 2021 at 12:14 PM <arnold@skeeve.com> wrote:

> Plan 9 eventually did something like this. I don't remember the details.
>
Arnold - point taken but ... unfortunately, I think that the comparison is
a little difficult because Plan9's concepts of namespaces is a tad
different than UNIX's.  But I'll let Rob or Ken comment as they lived and
developed both systems.

FWIW: An object store is something that retains information after the
processes that operates on it complete -* i.e. *its a *static entity*.
Links were (are) also a static concept.   Late binding to names (like
symlinks) are a dynamic (runtime idea).  Bakul points out that by using the
per process u area, the dynamic context can be retained.  The observation
is that .. (like symlinks) tend to be a runtime (dynamic) notion, although
I'm not sure how you keep consistency in the static FS if you don't store
the link in the inode.  As someone that did, I suggest - try writing fsck
if you are using dynamic content.   How do you know?  I'd still claim it is
the same issue.

Anyway, I suppose that like context sensitive symlinks (which I sorely
miss), you could do this and keep a list of the N inodes for the N parents
and then like CDSL's keep the one you used to get there in the u area so
that .. picks the proper one on the way out and you can still have the
static notion which something like fsck can check off line.

[-- Attachment #2: Type: text/html, Size: 2736 bytes --]

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2021-12-29 16:59 Bakul Shah
  2021-12-29 17:04 ` Clem Cole
@ 2021-12-29 17:14 ` arnold
  2021-12-29 17:38   ` Clem Cole
  1 sibling, 1 reply; 71+ messages in thread
From: arnold @ 2021-12-29 17:14 UTC (permalink / raw)
  To: clemc, bakul; +Cc: tuhs

Bakul Shah <bakul@iitbombay.org> wrote:

> 
> On Dec 29, 2021, at 8:01 AM, Bakul Shah <bakul@iitbombay.org> wrote:
> > On Dec 29, 2021, at 7:18 AM, Clem Cole <clemc@ccc.com> wrote:
> >> 
> >> Think about the UNIX FS and the link system call. How is mv implemented?   You link the file to the new directory and the unlink it from the old one.   But a directory file can not be in two directories at the same time as the .. link would fail.
> > 
> > Don’t see why linking a dir in two places is a problem.
>
> To expand on this a bit, the “cd ..” case can be handled by not storing a ..
> link in a dir. in the first place! Store the $PWD path in the u struct. Then
> cd .. would simply lop off the last component, if one exists. Thus .. takes
> you back only on the path you used! This also takes care of the issue with
> symlinks (& does what csh did in user code).

Plan 9 eventually did something like this. I don't remember the
details.

Arnold

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
@ 2021-12-29 17:12 Douglas McIlroy
  0 siblings, 0 replies; 71+ messages in thread
From: Douglas McIlroy @ 2021-12-29 17:12 UTC (permalink / raw)
  To: TUHS main list

The first specific mention of moving directories in Research is in
v10, but I'm sure that was implemented considerably earlier. The only
things special about moving a directory were that it needed root
privilege and checked against moving a directory into itself. As with
ordinary files, copying (with loss of hard links) happened only when
moving between different file systems. As far as I know, no atomicity
precautions were taken.

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2021-12-29 16:59 Bakul Shah
@ 2021-12-29 17:04 ` Clem Cole
  2021-12-29 17:14 ` arnold
  1 sibling, 0 replies; 71+ messages in thread
From: Clem Cole @ 2021-12-29 17:04 UTC (permalink / raw)
  To: Bakul Shah; +Cc: TUHS main list

[-- Attachment #1: Type: text/plain, Size: 599 bytes --]

On Wed, Dec 29, 2021 at 11:59 AM Bakul Shah <bakul@iitbombay.org> wrote:

> To expand on this a bit, the “cd ..” case can be handled by not storing a
> ..
> link in a dir. in the first place! Store the $PWD path in the u struct.
> Then
> cd .. would simply lop off the last component, if one exists. Thus .. takes
> you back only on the path you used! This also takes care of the issue with
> symlinks (& does what csh did in user code).

Which is exactly the problem  ...   Your solution works as a runtime fix,
sometimes.  It's hardly sufficient and why I hate symlinks thank you.

[-- Attachment #2: Type: text/html, Size: 1274 bytes --]

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
@ 2021-12-29 16:59 Bakul Shah
  2021-12-29 17:04 ` Clem Cole
  2021-12-29 17:14 ` arnold
  0 siblings, 2 replies; 71+ messages in thread
From: Bakul Shah @ 2021-12-29 16:59 UTC (permalink / raw)
  To: Clem Cole; +Cc: TUHS main list


On Dec 29, 2021, at 8:01 AM, Bakul Shah <bakul@iitbombay.org> wrote:
> On Dec 29, 2021, at 7:18 AM, Clem Cole <clemc@ccc.com> wrote:
>> 
>> Think about the UNIX FS and the link system call. How is mv implemented?   You link the file to the new directory and the unlink it from the old one.   But a directory file can not be in two directories at the same time as the .. link would fail.
> 
> Don’t see why linking a dir in two places is a problem.

To expand on this a bit, the “cd ..” case can be handled by not storing a ..
link in a dir. in the first place! Store the $PWD path in the u struct. Then
cd .. would simply lop off the last component, if one exists. Thus .. takes
you back only on the path you used! This also takes care of the issue with
symlinks (& does what csh did in user code).

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2021-12-29 16:17       ` Clem Cole
@ 2021-12-29 16:58         ` Richard Salz
  2021-12-30  5:14         ` Dan Stromberg
  1 sibling, 0 replies; 71+ messages in thread
From: Richard Salz @ 2021-12-29 16:58 UTC (permalink / raw)
  To: Clem Cole; +Cc: TUHS main list

[-- Attachment #1: Type: text/plain, Size: 103 bytes --]

Where and when did mvdir(1) appear?  It's not in Research or BSD.
>

IIRC /etc/mvdir in early SysV?

>

[-- Attachment #2: Type: text/html, Size: 855 bytes --]

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2021-12-29 16:10     ` Clem Cole
@ 2021-12-29 16:33       ` Warner Losh
  0 siblings, 0 replies; 71+ messages in thread
From: Warner Losh @ 2021-12-29 16:33 UTC (permalink / raw)
  To: Clem Cole; +Cc: TUHS main list

[-- Attachment #1: Type: text/plain, Size: 2804 bytes --]

Kirk has said it took decades to get rename(2) right... it was by far the
hardest system call to get right, especially in the face of concurrency.

Warner

Warner

On Wed, Dec 29, 2021, 9:11 AM Clem Cole <clemc@ccc.com> wrote:

> Just get the 4.1 man pages and it says: "Directories may only be moved
> within the same parent directory."
>
> On Wed, Dec 29, 2021 at 10:44 AM Will Senn <will.senn@gmail.com> wrote:
>
>> On 12/29/21 9:17 AM, Clem Cole wrote:
>>
>> Not until the BSD rename system call,.
>>
>> Try it on V6 or V7 and you will get 'directory exists' as an error.
>>
>> Sure enough. I never really noticed. I guess cuz I was careful about
>> creating directories in the first place.
>>
>> Think about the UNIX FS and the link system call. How is mv implemented?
>>   You link the file to the new directory and the unlink it from the old
>> one.   But a directory file can not be in two directories at the same time
>> as the .. link would fail.      When Joy created the rename system call it
>> became possible.  Until System V picked it up (IIRC was SVR3), only system
>> that supported the BSD world.
>>
>> Nice summary. I will add it to my systems programming investigations
>> list.
>>
>> What I have forgotten is if the BSD mv command for 4.1 supported it.  If
>> it did then it was not atomic -- it would have had to create the new
>> directory, move the contents independently and then remove the old one.
>>
>> FWIW: when we did the first SMP UNIX at Masscomp we had supported the BSD
>> FS and the 4.1c/4.2 system calls.   Joy's implementation of rename was a
>> real mess.   Making it atomic, supporting an SMP and deal with all the
>> recovery paths on an error took some work.  It's funny, the rename system
>> call is a simple idea, but on a failure when partially thru it, makes
>> unwinding the partial completion an interesting problem.
>>
>> It's impressive what early work was done and how few today really
>> understand the challenges that were met and overcome to give us our cushy
>> oses.
>>
>>
>> Clem
>>
>> On Wed, Dec 29, 2021 at 9:34 AM Will Senn <will.senn@gmail.com> wrote:
>>
>>> I'm a little flummoxed in trying to move some directories around in
>>> svr2. Shouldn't the following work?
>>>
>>>     mkdir a
>>>     mkdir b
>>>     mv a b
>>>
>>> I get the following error:
>>>     mv: b exists
>>>
>>> I tried many of the possible variants including:
>>>
>>> mv a b/
>>> mv: b/ exists
>>> mv a b/a
>>> mv: directory rename only
>>> cd b
>>> mv ../a .
>>> mv: . exists
>>> mv ../a ./
>>> mv: ./ exists
>>> mv ../a ./a
>>> mv: directory rename only
>>>
>>>
>>> If moving directories into existing directories wasn't allowed in those
>>> days, 1) how were directories managed? and 2) when did moving directories
>>> into directories become a thing?
>>>
>>>
>>>
>>

[-- Attachment #2: Type: text/html, Size: 6377 bytes --]

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2021-12-29 15:44     ` Richard Salz
@ 2021-12-29 16:17       ` Clem Cole
  2021-12-29 16:58         ` Richard Salz
  2021-12-30  5:14         ` Dan Stromberg
  0 siblings, 2 replies; 71+ messages in thread
From: Clem Cole @ 2021-12-29 16:17 UTC (permalink / raw)
  To: Richard Salz; +Cc: TUHS main list

[-- Attachment #1: Type: text/plain, Size: 402 bytes --]

Where and when did mvdir(1) appear?  It's not in Research or BSD.

FWIW: in the old days, I just used  tar cf - . | (cd some_place_else; tar
xvpf - )  preserving permissions.  The hardlink scheme of cpio was the one
thing it could do that the original tar did not.

Clem

On Wed, Dec 29, 2021 at 10:46 AM Richard Salz <rich.salz@gmail.com> wrote:

> Wasn't some of that handled by the mvdir command?
>

[-- Attachment #2: Type: text/html, Size: 1261 bytes --]

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2021-12-29 15:44   ` Will Senn
@ 2021-12-29 16:10     ` Clem Cole
  2021-12-29 16:33       ` Warner Losh
  0 siblings, 1 reply; 71+ messages in thread
From: Clem Cole @ 2021-12-29 16:10 UTC (permalink / raw)
  To: Will Senn; +Cc: TUHS main list

[-- Attachment #1: Type: text/plain, Size: 2493 bytes --]

Just get the 4.1 man pages and it says: "Directories may only be moved
within the same parent directory."

On Wed, Dec 29, 2021 at 10:44 AM Will Senn <will.senn@gmail.com> wrote:

> On 12/29/21 9:17 AM, Clem Cole wrote:
>
> Not until the BSD rename system call,.
>
> Try it on V6 or V7 and you will get 'directory exists' as an error.
>
> Sure enough. I never really noticed. I guess cuz I was careful about
> creating directories in the first place.
>
> Think about the UNIX FS and the link system call. How is mv implemented?
> You link the file to the new directory and the unlink it from the old one.
>   But a directory file can not be in two directories at the same time as
> the .. link would fail.      When Joy created the rename system call it
> became possible.  Until System V picked it up (IIRC was SVR3), only system
> that supported the BSD world.
>
> Nice summary. I will add it to my systems programming investigations list.
>
> What I have forgotten is if the BSD mv command for 4.1 supported it.  If
> it did then it was not atomic -- it would have had to create the new
> directory, move the contents independently and then remove the old one.
>
> FWIW: when we did the first SMP UNIX at Masscomp we had supported the BSD
> FS and the 4.1c/4.2 system calls.   Joy's implementation of rename was a
> real mess.   Making it atomic, supporting an SMP and deal with all the
> recovery paths on an error took some work.  It's funny, the rename system
> call is a simple idea, but on a failure when partially thru it, makes
> unwinding the partial completion an interesting problem.
>
> It's impressive what early work was done and how few today really
> understand the challenges that were met and overcome to give us our cushy
> oses.
>
>
> Clem
>
> On Wed, Dec 29, 2021 at 9:34 AM Will Senn <will.senn@gmail.com> wrote:
>
>> I'm a little flummoxed in trying to move some directories around in svr2.
>> Shouldn't the following work?
>>
>>     mkdir a
>>     mkdir b
>>     mv a b
>>
>> I get the following error:
>>     mv: b exists
>>
>> I tried many of the possible variants including:
>>
>> mv a b/
>> mv: b/ exists
>> mv a b/a
>> mv: directory rename only
>> cd b
>> mv ../a .
>> mv: . exists
>> mv ../a ./
>> mv: ./ exists
>> mv ../a ./a
>> mv: directory rename only
>>
>>
>> If moving directories into existing directories wasn't allowed in those
>> days, 1) how were directories managed? and 2) when did moving directories
>> into directories become a thing?
>>
>>
>>
>

[-- Attachment #2: Type: text/html, Size: 5726 bytes --]

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2021-12-29 15:17 ` Clem Cole
  2021-12-29 15:44   ` Will Senn
@ 2021-12-29 16:01   ` Bakul Shah
  1 sibling, 0 replies; 71+ messages in thread
From: Bakul Shah @ 2021-12-29 16:01 UTC (permalink / raw)
  To: Clem Cole; +Cc: TUHS main list

On Dec 29, 2021, at 7:18 AM, Clem Cole <clemc@ccc.com> wrote:
> 
> Think about the UNIX FS and the link system call. How is mv implemented?   You link the file to the new directory and the unlink it from the old one.   But a directory file can not be in two directories at the same time as the .. link would fail.

Don’t see why linking a dir in two places is a problem. 

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2021-12-29 15:17 ` Clem Cole
@ 2021-12-29 15:44   ` Will Senn
  2021-12-29 16:10     ` Clem Cole
  2021-12-29 16:01   ` Bakul Shah
  1 sibling, 1 reply; 71+ messages in thread
From: Will Senn @ 2021-12-29 15:44 UTC (permalink / raw)
  To: Clem Cole; +Cc: TUHS main list

[-- Attachment #1: Type: text/plain, Size: 2428 bytes --]

On 12/29/21 9:17 AM, Clem Cole wrote:
> Not until the BSD rename system call,.
>
> Try it on V6 or V7 and you will get 'directory exists' as an error.
>
Sure enough. I never really noticed. I guess cuz I was careful about 
creating directories in the first place.

> Think about the UNIX FS and the link system call. How is mv 
> implemented?   You link the file to the new directory and the unlink 
> it from the old one.   But a directory file can not be in two 
> directories at the same time as the .. link would fail.      When Joy 
> created the rename system call it became possible.  Until System V 
> picked it up (IIRC was SVR3), only system that supported the BSD world.
>
Nice summary. I will add it to my systems programming investigations list.
> What I have forgotten is if the BSD mv command for 4.1 supported it.  
> If it did then it was not atomic -- it would have had to create the 
> new directory, move the contents independently and then remove the old 
> one.
>
> FWIW: when we did the first SMP UNIX at Masscomp we had supported the 
> BSD FS and the 4.1c/4.2 system calls.   Joy's implementation of rename 
> was a real mess.   Making it atomic, supporting an SMP and deal with 
> all the recovery paths on an error took some work.  It's funny, the 
> rename system call is a simple idea, but on a failure when 
> partially thru it, makes unwinding the partial completion an 
> interesting problem.
It's impressive what early work was done and how few today really 
understand the challenges that were met and overcome to give us our 
cushy oses.
>
> Clem
>
> On Wed, Dec 29, 2021 at 9:34 AM Will Senn <will.senn@gmail.com> wrote:
>
>     I'm a little flummoxed in trying to move some directories around
>     in svr2. Shouldn't the following work?
>
>         mkdir a
>         mkdir b
>         mv a b
>
>     I get the following error:
>     mv: b exists
>
>     I tried many of the possible variants including:
>
>         mv a b/
>         mv: b/ exists
>         mv a b/a
>         mv: directory rename only
>         cd b
>         mv ../a .
>         mv: . exists
>         mv ../a ./
>         mv: ./ exists
>         mv ../a ./a
>         mv: directory rename only
>
>
>     If moving directories into existing directories wasn't allowed in
>     those days, 1) how were directories managed? and 2) when did
>     moving directories into directories become a thing?
>
>

[-- Attachment #2: Type: text/html, Size: 5647 bytes --]

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2021-12-29 15:38   ` Will Senn
@ 2021-12-29 15:44     ` Richard Salz
  2021-12-29 16:17       ` Clem Cole
  0 siblings, 1 reply; 71+ messages in thread
From: Richard Salz @ 2021-12-29 15:44 UTC (permalink / raw)
  To: Will Senn; +Cc: TUHS main list

[-- Attachment #1: Type: text/plain, Size: 50 bytes --]

Wasn't some of that handled by the mvdir command?

[-- Attachment #2: Type: text/html, Size: 76 bytes --]

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2021-12-29 15:02 ` arnold
@ 2021-12-29 15:38   ` Will Senn
  2021-12-29 15:44     ` Richard Salz
  0 siblings, 1 reply; 71+ messages in thread
From: Will Senn @ 2021-12-29 15:38 UTC (permalink / raw)
  To: arnold, tuhs

That's fascinating. I always wondered how oldtimers were so facile with 
cpio and its ilk. If they had to use if for stuff like this, it's no 
wonder they built some muscle memory on it.

Thanks,

Will

On 12/29/21 9:02 AM, arnold@skeeve.com wrote:
> System V of that era didn't allow moving directories. You can copy them
> recursively with find and cpio with the option that makes hard links
> and then remove the old directory, or use a standard tar pipeline to
> copy the directory tree.
>
> BSD has always allowed moving directories; I *think* that descends
> from Research Unix but I don't remember for sure.
>
> HTH,
>
> Arnold
>
> Will Senn <will.senn@gmail.com> wrote:
>
>> I'm a little flummoxed in trying to move some directories around in
>> svr2. Shouldn't the following work?
>>
>>       mkdir a
>>       mkdir b
>>       mv a b
>>
>> I get the following error:
>> mv: b exists
>>
>> I tried many of the possible variants including:
>>
>>      mv a b/
>>      mv: b/ exists
>>      mv a b/a
>>      mv: directory rename only
>>      cd b
>>      mv ../a .
>>      mv: . exists
>>      mv ../a ./
>>      mv: ./ exists
>>      mv ../a ./a
>>      mv: directory rename only
>>
>>
>> If moving directories into existing directories wasn't allowed in those
>> days, 1) how were directories managed? and 2) when did moving
>> directories into directories become a thing?
>>


^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2021-12-29 14:33 Will Senn
  2021-12-29 15:02 ` arnold
@ 2021-12-29 15:17 ` Clem Cole
  2021-12-29 15:44   ` Will Senn
  2021-12-29 16:01   ` Bakul Shah
  1 sibling, 2 replies; 71+ messages in thread
From: Clem Cole @ 2021-12-29 15:17 UTC (permalink / raw)
  To: Will Senn; +Cc: TUHS main list

[-- Attachment #1: Type: text/plain, Size: 1842 bytes --]

Not until the BSD rename system call,.

Try it on V6 or V7 and you will get 'directory exists' as an error.

Think about the UNIX FS and the link system call. How is mv implemented?
You link the file to the new directory and the unlink it from the old one.
  But a directory file can not be in two directories at the same time as
the .. link would fail.      When Joy created the rename system call it
became possible.  Until System V picked it up (IIRC was SVR3), only system
that supported the BSD world.

What I have forgotten is if the BSD mv command for 4.1 supported it.  If it
did then it was not atomic -- it would have had to create the new
directory, move the contents independently and then remove the old one.

FWIW: when we did the first SMP UNIX at Masscomp we had supported the BSD
FS and the 4.1c/4.2 system calls.   Joy's implementation of rename was a
real mess.   Making it atomic, supporting an SMP and deal with all the
recovery paths on an error took some work.  It's funny, the rename system
call is a simple idea, but on a failure when partially thru it, makes
unwinding the partial completion an interesting problem.

Clem

On Wed, Dec 29, 2021 at 9:34 AM Will Senn <will.senn@gmail.com> wrote:

> I'm a little flummoxed in trying to move some directories around in svr2.
> Shouldn't the following work?
>
>     mkdir a
>     mkdir b
>     mv a b
>
> I get the following error:
>     mv: b exists
>
> I tried many of the possible variants including:
>
> mv a b/
> mv: b/ exists
> mv a b/a
> mv: directory rename only
> cd b
> mv ../a .
> mv: . exists
> mv ../a ./
> mv: ./ exists
> mv ../a ./a
> mv: directory rename only
>
>
> If moving directories into existing directories wasn't allowed in those
> days, 1) how were directories managed? and 2) when did moving directories
> into directories become a thing?
>
>
>

[-- Attachment #2: Type: text/html, Size: 3692 bytes --]

^ permalink raw reply	[flat|nested] 71+ messages in thread

* Re: [TUHS] moving directories in svr2
  2021-12-29 14:33 Will Senn
@ 2021-12-29 15:02 ` arnold
  2021-12-29 15:38   ` Will Senn
  2021-12-29 15:17 ` Clem Cole
  1 sibling, 1 reply; 71+ messages in thread
From: arnold @ 2021-12-29 15:02 UTC (permalink / raw)
  To: will.senn, tuhs

System V of that era didn't allow moving directories. You can copy them
recursively with find and cpio with the option that makes hard links
and then remove the old directory, or use a standard tar pipeline to
copy the directory tree.

BSD has always allowed moving directories; I *think* that descends
from Research Unix but I don't remember for sure.

HTH,

Arnold

Will Senn <will.senn@gmail.com> wrote:

> I'm a little flummoxed in trying to move some directories around in 
> svr2. Shouldn't the following work?
>
>      mkdir a
>      mkdir b
>      mv a b
>
> I get the following error:
> mv: b exists
>
> I tried many of the possible variants including:
>
>     mv a b/
>     mv: b/ exists
>     mv a b/a
>     mv: directory rename only
>     cd b
>     mv ../a .
>     mv: . exists
>     mv ../a ./
>     mv: ./ exists
>     mv ../a ./a
>     mv: directory rename only
>
>
> If moving directories into existing directories wasn't allowed in those 
> days, 1) how were directories managed? and 2) when did moving 
> directories into directories become a thing?
>

^ permalink raw reply	[flat|nested] 71+ messages in thread

* [TUHS] moving directories in svr2
@ 2021-12-29 14:33 Will Senn
  2021-12-29 15:02 ` arnold
  2021-12-29 15:17 ` Clem Cole
  0 siblings, 2 replies; 71+ messages in thread
From: Will Senn @ 2021-12-29 14:33 UTC (permalink / raw)
  To: TUHS main list

[-- Attachment #1: Type: text/plain, Size: 621 bytes --]

I'm a little flummoxed in trying to move some directories around in 
svr2. Shouldn't the following work?

     mkdir a
     mkdir b
     mv a b

I get the following error:
mv: b exists

I tried many of the possible variants including:

    mv a b/
    mv: b/ exists
    mv a b/a
    mv: directory rename only
    cd b
    mv ../a .
    mv: . exists
    mv ../a ./
    mv: ./ exists
    mv ../a ./a
    mv: directory rename only


If moving directories into existing directories wasn't allowed in those 
days, 1) how were directories managed? and 2) when did moving 
directories into directories become a thing?


[-- Attachment #2: Type: text/html, Size: 1270 bytes --]

^ permalink raw reply	[flat|nested] 71+ messages in thread

end of thread, other threads:[~2022-01-09 19:28 UTC | newest]

Thread overview: 71+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-12-29 19:13 [TUHS] moving directories in svr2 Douglas McIlroy
2021-12-29 19:37 ` Dan Cross
2021-12-29 20:15   ` Dan Cross
2021-12-29 20:42     ` Richard Salz
2021-12-29 20:58       ` Dan Cross
2021-12-29 21:20         ` Clem Cole
2021-12-30  3:15       ` Bakul Shah
  -- strict thread matches above, loose matches on Subject: below --
2021-12-30  3:45 Noel Chiappa
2021-12-30  4:02 ` Bakul Shah
2021-12-30 16:40 ` Theodore Ts'o
2021-12-30 22:31   ` Dan Cross
2021-12-31  0:43     ` Bakul Shah
2021-12-31  1:00       ` Rob Pike
2021-12-31  1:45         ` Bakul Shah
2021-12-31  2:23           ` Adam Thornton
2021-12-31 18:56       ` Chet Ramey
2021-12-31  3:08     ` Theodore Ts'o
2021-12-31  3:23       ` Rob Pike
2021-12-31  5:16         ` Theodore Ts'o
2021-12-31  5:21           ` Dan Cross
2021-12-31  5:55           ` Rob Pike
2021-12-31 13:32             ` Michael Kjörling
2021-12-31 15:53               ` Adam Thornton
2021-12-31 16:13                 ` Arthur Krewat
2021-12-31 18:17                 ` Dan Cross
2021-12-31 18:23                   ` Larry McVoy
2021-12-31 18:37                     ` Dan Cross
2021-12-31 18:29                   ` Arthur Krewat
2022-01-01  0:09                   ` Theodore Ts'o
2022-01-03 13:35                     ` Dan Cross
2022-01-03 20:23                       ` Theodore Ts'o
2022-01-03 20:45                         ` Warner Losh
2022-01-03 21:15                         ` Dan Cross
2022-01-03 22:26                           ` Theodore Ts'o
2022-01-03 23:10                             ` Dan Cross
2022-01-04 15:45                             ` Chet Ramey
2022-01-09 19:28                             ` Larry McVoy
2022-01-03 23:21                           ` Doug McIntyre
2022-01-03 23:37                             ` Adam Thornton
2022-01-04 14:49                               ` Stuart Remphrey
2022-01-03 23:44                             ` Larry McVoy
2022-01-03 23:56                               ` Warner Losh
2022-01-04  2:28                               ` Theodore Ts'o
2022-01-04  2:42                                 ` Larry McVoy
2022-01-03 22:57                         ` Phil Budne
2021-12-31  5:12       ` Bakul Shah
2021-12-29 19:33 Noel Chiappa
2021-12-30  3:40 ` Jay Logue via TUHS
2021-12-29 17:12 Douglas McIlroy
2021-12-29 16:59 Bakul Shah
2021-12-29 17:04 ` Clem Cole
2021-12-29 17:14 ` arnold
2021-12-29 17:38   ` Clem Cole
2021-12-29 17:49     ` Brantley Coile
2021-12-29 18:27       ` ron minnich
2021-12-29 20:59         ` Rob Pike
2021-12-29 14:33 Will Senn
2021-12-29 15:02 ` arnold
2021-12-29 15:38   ` Will Senn
2021-12-29 15:44     ` Richard Salz
2021-12-29 16:17       ` Clem Cole
2021-12-29 16:58         ` Richard Salz
2021-12-30  5:14         ` Dan Stromberg
2021-12-30 16:22           ` Clem Cole
2021-12-30 18:02           ` John Cowan
2021-12-30 23:04             ` Richard Salz
2021-12-29 15:17 ` Clem Cole
2021-12-29 15:44   ` Will Senn
2021-12-29 16:10     ` Clem Cole
2021-12-29 16:33       ` Warner Losh
2021-12-29 16:01   ` Bakul Shah

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).