Re: [9fans] mv vs cp

Re: [9fans] mv vs cp

[rob pike]

> walk away from symlinks because it was hard to get right, more than
> because  philosophically they were 'wrong'? 

Symlinks were put in to make it possible to link directories from user
mode and to have cross-file system links.  Plan 9 does both those
things another way.  I have never felt the need for symbolic links in
Plan 9.

They also have some ugly properties.  I find the delayed evaluation
mystifying sometimes.  Also, although they are used to implement
cross-system links, they live on a single file system.  For example,
/usr/rob might be a link to /home/rob even though /usr and /home live
on different disks or even different servers, which means /usr has a
file that talks about /home, a file on a different machine.  Besides
the creepiness of that, the delayed evaluation can bite you hard, the
permanence of the link can be troublesome, and this little piece of
name space stored as a time bomb somewhere in the network rather than
in the client's representation of its resources is poor compartment-
alization and therefore lousy design.  And why on earth does ls
default to showing you the link rather than what it points to?

This does not imply that Plan 9's name space was designed in reaction
to symbolic links, of course, but at least some of us did observe that
symbolic links were little more than a disgusting hack and that there
had to be a better way.  Ken was already thinking about that way when
all this was happening back in the early '80s.  I believe that the
design he came up with, Plan 9's name space, is indeed better,
although it has some problems of its own and doesn't exactly cover
symlinks (the lack of permanence of the mount table comes to mind).
The unification of all naming operations into a single data structure
(the mount table), the well-defined evaluation, and the clean
separation of file servers and name spaces, and most important the
user-modifiable, per-process name space were all improvements.

As I said in my `dot-dot' paper, Unix still hasn't recovered from all
the problems wrought by the introduction of symbolic links.  It is a
lesson, although not a new one, that the ugly, supposedly temporary
hack of symlinks became a permanent fixture of the system, and one
that people argue for passionately, mistaking the addressing of the
issue for a true solution to the problem (c.f.  Plan 9's # notation).

-rob

Re: [9fans] mv vs cp

[Thomas Bushnell, BSG]

rob@plan9.bell-labs.com (rob pike) writes:

[about symlinks]

> and this little piece of
> name space stored as a time bomb somewhere in the network rather than
> in the client's representation of its resources is poor compartment-
> alization and therefore lousy design

This makes me think of something.

Plan 9 puts the mount hierarchy over in the per-process namespace,
which is a nifty idea.

Rob points out here that symlinks also really belong in per-process
places (and the same effect can be more cleanly realized with
"bind").  So clearly, there is no need for symlinks in Plan 9.

(One thing; please clear up my ignorance: sometimes a symlink is used
for the benefit of all the users; where in a Plan 9 system do I put a
"bind" such that it is the automatic default for all users in that
way?)

It seems to me that the same could be true of directory renames, if
the entire naming hierarchy were regarded as a per-process namespace,
rather than just the "mount" portions.  Then a given user could move,
mount, unlink, etc, without changing anything on the fileserver at all
(except a ref count).  [Actually, let it become more liberal and a ref
count is probably not sufficient any more.]

Re: [9fans] mv vs cp

[Thomas Bushnell, BSG]

davel@luchie-chowchows.demon.co.uk writes:

> My point is that
> a) we shouldn't be putting in hacks to ameliorate user mistakes:
>    do you want undelete in there too?

Plan 9 already *has* that feature.  Of course, it would be even nicer
if it had a complete version history, instead of "yesterday's
snapshot", but that's pretty good too (better than Unix!).

Yes, a good system should be making automatic backups.  And even
having convenient commands to recover files from the backup in the
event of mistakes is a good idea.

Re: [9fans] mv vs cp

[davel]

On Thu, Oct 11, 2001 at 09:10:20AM +0000, Thomas Bushnell, BSG wrote:

> Ah!  The job of software is not to make users' lives easier.  Not to
> provide them tools to be more productive.

What has that got to do with putting in hacks to get round mistakes?

> No--the job of the computer is to administer punishments for their
> errors.
> 
> Are you really serious?

Are YOU really serious?  Who said anything about punishment?

My point is that
a) we shouldn't be putting in hacks to ameliorate user mistakes:
   do you want undelete in there too?
b) Plan9 already has this nice-flexible FS architecture
   which gets around lots of the directory shuffling
   necessary on other OSs.

> There are reasons to have a feature, and reasons not to.  But "we want
> to punish people for making mistakes" is never a reason not to, and it
> sounds like you have run out of steam when that's all you muster.

a) It sounds like you have run out of steam if all you can do is misconstrue
   my meaning.
b) You make me sound like a sadist according to your logic,
   when in fact I'm a sado-masochist, since I'm punishing myself too!

Can we now please agree to differ?
This has gone on long enough.

Cheers,
	Dave.

Re: [9fans] mv vs cp

[Thomas Bushnell, BSG]

davel@luchie-chowchows.demon.co.uk writes:

> On Wed, Oct 10, 2001 at 08:49:41AM +0000, Thomas Bushnell, BSG wrote:
> > davel@luchie-chowchows.demon.co.uk writes:
> > 
> > > Also, to be brutal for a minute,
> > > why *do* people want to move directories around:
> > > why not just put them in the right place to start with?
> > 
> > Unlike people here, apparently, I sometimes make mistakes.  
> 
> So do I, and I expect to suffer for them,
> not have kernel hacks put in place to make them easier to bodge around.

Ah!  The job of software is not to make users' lives easier.  Not to
provide them tools to be more productive.

No--the job of the computer is to administer punishments for their
errors.

Are you really serious?

There are reasons to have a feature, and reasons not to.  But "we want
to punish people for making mistakes" is never a reason not to, and it
sounds like you have run out of steam when that's all you muster.

Re: [9fans] mv vs cp

[anothy]

yeah, but whoever's _got_ your fileserver can do
pretty much whatever they please.

Re: [9fans] mv vs cp

[okamoto]

Do you have some special Plan 9 file server kernel?

Kenji

Re: [9fans] mv vs cp

[Russ Cox]

That's okay, whoever runs your file server
can read your mail box too.

Russ

Re: [9fans] mv vs cp

[okamoto]

>Before you start saying that shared /tmp or all-mighty root are bad ideas,

I hate all-mighty root user because s/he can read my mail box.

Kenji

Re: [9fans] mv vs cp

[forsyth]

[-- Attachment #1: Type: text/plain, Size: 1157 bytes --]

yes, i don't think the desire to move directory contents was at issue,
for that and other examples.
the question was whether to implement an optimisation for the
case where source and target names are on the same file store.

since in both the Plan 9 and Unix cases, it's necessary to copy,
plan 9 settles for renaming files and directories (in their containing directories),
but does not allow either files or directories to be moved without copying.
(depending on what the move operation means in the presence of
arbitrary per-process binds and mounts, it might be impossible to
provide a sensible definition of move from a user's point of view anyway.)

some versions of unix attempt to allow directories to be moved,
that typically doesn't work across file systems, so mv ends
up copying anyway, doing pretty much what plan 9 would do.
on unix, i might like to do {mv /tmp/superbigvideofile $home/videofarm/birthday}
without having the system copy a big file, but that usually doesn't work.
so on unix i probably would do {recordvideo $home/videofarm/tmp.$pid}
and rename it later, which is how i'd get round the problem on plan 9.


[-- Attachment #2: Type: message/rfc822, Size: 1761 bytes --]

To: 9fans@cse.psu.edu
Subject: Re: [9fans] mv vs cp
Date: Wed, 10 Oct 2001 08:49:09 GMT
Message-ID: <9pvcjf$k1f$1@inputplus.demon.co.uk>

Hi Dave,

> Also, to be brutal for a minute,
> why *do* people want to move directories around:

Typical reason is wanting to re-arrange a directory structure after a
better layout becomes clearer with use.


Ralph.

Re: [9fans] mv vs cp

[Borja Marcos]

On Wednesday 10 October 2001 13:50, you wrote:
> 	And you had a directory called "/some/directory/.." which didn't actually
> point to "/some/directory", but was a new directory. It was a great way for

	TYPO: I meant point to "/some", obviously ;-)



	Borja.

Re: [9fans] mv vs cp

[Borja Marcos]

On Wednesday 10 October 2001 10:57, you wrote:
> George Michaelson wrote:
> > Surely hard links couldn't point to directories, ...
>
> Yes, they could.  It was a common form of corruption in 6th Edition
> UNIX.

	You could even do weird things. For example, this worked under SCO Unix.

	cd /some/directory
	mkdir another
	unlink ..
	link another ..
	unlink another

	And you had a directory called "/some/directory/.." which didn't actually 
point to "/some/directory", but was a new directory. It was a great way for 
hackers to hide things in a machine.



	Borja.

Re: [9fans] mv vs cp

[Alexander Viro]

On Wed, 10 Oct 2001, Thomas Bushnell, BSG wrote:

> viro@math.psu.edu (Alexander Viro) writes:
> 
> > BTW, while we are at it, check what does GNU find(1) (which is the
> > default on Hurd, IIRC) do when somebody moves a directory while
> > find is walking through its subdirectories.  And think about the
> > implications.  What was that about correctness and completeness, again?
> 
> find does not necessarily traverse every file on an active partition.  
> 
> But that has nothing to do with the moving of directories?  The same
> thing happens with the moving of single files too.

GNU find traverses the tree with chdir() into subdirectories and _blind_
chdir("..") to go back.  Cwd follows directory when you do rename(2).
It does no such thing if you do mkdir() + recursive copy + recursive remove +
rmdir().  See the problem now?  "Find something under /tmp and do something
with all such objects" can be subverted into "find something under / and
do something with all such objects" by attacker that has access only to
subdirectories of /tmp.

And no, it's not only GNU.  Solaris find(1) has the same lovely property
(at least had it not too long ago - I hadn't checked the recent versions).
And it's not only find(1) - other tree-walkers are often vulnerable to
the same kind of crap.

I don't think that security implications really need to be explained.
Before you start saying that shared /tmp or all-mighty root are bad ideas,
think for a minute and see how the scenario can be modified for situations
other than one described above.

Notice that "generate pathnames and don't bother with chdir() at all" is
also not an answer.  E.g. Plan 9 rm(1) would become a glaring security
hole if ported on any recent UNIX.  Symlink attacks would make it very
interesting...  Something like dup, walk, clunk, stat and remove exported
to userland would allow to do such stuff more or less elegantly, but
without that you need fchdir() to avoid symlink attacks.

Anyway, the point is that cross-directory rename(2) leads to userland bugs
that are really nasty and, as experience shows, not obvious for a lot of
people.  Actually you've just demonstrated that.

It's not about kernel simplicity; userland code becomes much more complicated
to be able to cope with effects of having cross-directory rename(2) around.

Re: [9fans] mv vs cp

[davel]

On Wed, Oct 10, 2001 at 08:49:41AM +0000, Thomas Bushnell, BSG wrote:
> davel@luchie-chowchows.demon.co.uk writes:
> 
> > Also, to be brutal for a minute,
> > why *do* people want to move directories around:
> > why not just put them in the right place to start with?
> 
> Unlike people here, apparently, I sometimes make mistakes.  

So do I, and I expect to suffer for them,
not have kernel hacks put in place to make them easier to bodge around.

Cheers,
	Dave.

Re: [9fans] mv vs cp

[Douglas A. Gwyn]

George Michaelson wrote:
> Surely hard links couldn't point to directories, ...

Yes, they could.  It was a common form of corruption in 6th Edition
UNIX.

Re: [9fans] mv vs cp

[Thomas Bushnell, BSG]

viro@math.psu.edu (Alexander Viro) writes:

> BTW, while we are at it, check what does GNU find(1) (which is the
> default on Hurd, IIRC) do when somebody moves a directory while
> find is walking through its subdirectories.  And think about the
> implications.  What was that about correctness and completeness, again?

find does not necessarily traverse every file on an active partition.  

But that has nothing to do with the moving of directories?  The same
thing happens with the moving of single files too.

Re: [9fans] mv vs cp

[Thomas Bushnell, BSG]

forsyth@vitanuova.com writes:

> >>Is this just an example of the "New Jersey" preference for simplicity
> >>of implementation over the "Cambridge" preference for correctness and
> >>completeness?
> 
> ``correctness and completeness''?  this sounds like the result of
> a self-assessment exercise.

Have you read the "wrong is better" paper?

Re: [9fans] mv vs cp

[Thomas Bushnell, BSG]

davel@luchie-chowchows.demon.co.uk writes:

> Also, to be brutal for a minute,
> why *do* people want to move directories around:
> why not just put them in the right place to start with?

Unlike people here, apparently, I sometimes make mistakes.  

Re: [9fans] mv vs cp

[Ralph Corderoy]

Hi Dave,

> Also, to be brutal for a minute,
> why *do* people want to move directories around:

Typical reason is wanting to re-arrange a directory structure after a
better layout becomes clearer with use.


Ralph.

Re: [9fans] mv vs cp

[davel]

On Tue, Oct 09, 2001 at 09:05:08PM -0400, erik quanstrom wrote:
> plan 9, an operating system for people who never change their minds 
> or have new ideas.

(I almost ignored this as a troll)

Just to be explicit:
* The _old_ idea here is *being* able to rename directories.
* The _new_ idea here is *not being* able to rename directories.

Also, I have yet to see anyone explain *why* moving directories around
is such a big win.  All I hear is
"Well, I can do it on this other system ...".

As I said
> > Plan9's ability to present a flexible view of the file system
> > probably obviates most reasons to want to shuffle directories around. 

Cheers,
	Dave.

Re: [9fans] mv vs cp

[Skip Tavakkolian]

and the minimalist home page. what? no javascript?!!!!

At 10:15 PM 10/9/2001 -0400, david presotto wrote:
>Hence my domain name.  Right on!
>----- Original Message ----- 
>From: erik quanstrom <quanstro@speakeasy.net>
>To: <9fans@cse.psu.edu>
>Sent: Tuesday, October 09, 2001 9:05 PM
>Subject: Re: [9fans] mv vs cp
>
>
>> >Also, to be brutal for a minute,
>> >why *do* people want to move directories around:
>> >why not just put them in the right place to start with?
>> 
>> plan 9, an operating system for people who never change their minds 
>> or have new ideas.
>> 
>> erik
>> 
>
>

Re: [9fans] mv vs cp

[david presotto]

Hence my domain name.  Right on!
----- Original Message ----- 
From: erik quanstrom <quanstro@speakeasy.net>
To: <9fans@cse.psu.edu>
Sent: Tuesday, October 09, 2001 9:05 PM
Subject: Re: [9fans] mv vs cp


> >Also, to be brutal for a minute,
> >why *do* people want to move directories around:
> >why not just put them in the right place to start with?
> 
> plan 9, an operating system for people who never change their minds 
> or have new ideas.
> 
> erik
> 

Re: [9fans] mv vs cp

[okamoto]

[-- Attachment #1: Type: text/plain, Size: 222 bytes --]

I responded your comment before I read another comment on EROS, 
which said EROS is persistent.  Maybe it indicates it has no concept 
of directory graph.   If so, I understand your behaviour here.

Thanks

Kenji


[-- Attachment #2: Type: message/rfc822, Size: 2215 bytes --]

From: Richard <greon@best.com>
To: 9fans@cse.psu.edu
Subject: Re: [9fans] mv vs cp
Date: Mon, 08 Oct 2001 22:46:41 -0700
Message-ID: <E15qpiz-0000EM-00@localhost>

okamoto@granite.cias.osakafu-u.ac.jp writes:

>I don't understand your stance anymore...
>Aren't you who recommended us EROS?
>Isn't the file system of EROS directory graph?

my stance is that OS designers should learn about
capability OSes like EROS and that Plan 9 should 
be made more attractive to people used to Unix's mv
command.  do you detect an inconsistency in
that?

Re: [9fans] mv vs cp

[erik quanstrom]

>Also, to be brutal for a minute,
>why *do* people want to move directories around:
>why not just put them in the right place to start with?

plan 9, an operating system for people who never change their minds 
or have new ideas.

erik

Re: [9fans] mv vs cp

[presotto]

On Tue Oct  9 12:35:49 EDT 2001, tb+usenet@becket.net wrote:
> 
> (One thing; please clear up my ignorance: sometimes a symlink is used
> for the benefit of all the users; where in a Plan 9 system do I put a
> "bind" such that it is the automatic default for all users in that
> way?)

That's what /lib/namespace is.  It represents a global namespace for
all users on a machine.  Of course, the user can then do whatever
he/she wants but serves as the default namespace.

/lib/namespace and your profile also represent persistence which is
another useful property you get from symlinks.

Re: [9fans] mv vs cp

[rog]

> I'm not expert in Plan 9, but it seems to me that it breaks it for the
> user.

surely it's "broken for the user" the moment the user can't move a
file from one place to another for any reason at all (e.g.  because
the file is on a different disk, or a different location on the
network, or whatever).

under Windows, where disks and networks are visible at the top level
of the file tree to the user, perhaps this is slightly more
understandable, but where the file tree is unified, the difference
between "can't because it's on a different disk" and "can't because
it's in a different directory" is surely one of degree not of kind?

  rog.

Re: [9fans] mv vs cp

[Alexander Viro]

On Tue, 9 Oct 2001, Thomas Bushnell, BSG wrote:

> Is this just an example of the "New Jersey" preference for simplicity
> of implementation over the "Cambridge" preference for correctness and
> completeness?

Taste vs. GNU?  That too, but there's more to it.

BTW, while we are at it, check what does GNU find(1) (which is the
default on Hurd, IIRC) do when somebody moves a directory while
find is walking through its subdirectories.  And think about the
implications.  What was that about correctness and completeness, again?

Re: [9fans] mv vs cp

[davel]

On Tue, Oct 09, 2001 at 03:58:47PM +0000, Thomas Bushnell, BSG wrote:
> I'm not expert in Plan 9, but it seems to me that it breaks it for the
> user.

"The user"?
You mean there's only one?!?:-)
Most of the people who have been complaining are not typical "users".

In reality, I doubt that most users want to move large directories around.
(In my experience, most Windows users just leave files wherever
  the application drops them and then complain when they can't find them ...)

Plan9's ability to present a flexible view of the file system
probably obviates most reasons to want to shuffle directories around.

Also, to be brutal for a minute,
why *do* people want to move directories around:
why not just put them in the right place to start with?

The only time I can remember needing to move large directories
was when i needed to balance NFS servers,
i.e. relocating them from one file server or volume to another ...

> Is this just an example of the "New Jersey" preference for simplicity 
> of implementation over the "Cambridge" preference for correctness and
> completeness?

No polite comment available:-).

Cheers,
	Dave.

Re: [9fans] mv vs cp

[forsyth]

>>Is this just an example of the "New Jersey" preference for simplicity
>>of implementation over the "Cambridge" preference for correctness and
>>completeness?

``correctness and completeness''?  this sounds like the result of
a self-assessment exercise.

Re: [9fans] mv vs cp

[Thomas Bushnell, BSG]

viro@math.psu.edu (Alexander Viro) writes:

> Again, I've really been there and done that.  It wasn't pretty and I
> would be much happier if rename() couldn't change the topology.  We
> couldn't do that since it would break a lot of userland stuff;  Plan 9
> has a luxury of dropping annoying crap and being able to find and fix
> resulting breakage in userland.  In this case crap is not there in the
> first place, so introducing it would cause a lot of ugliness in the kernel
> for no good reason.

I'm not expert in Plan 9, but it seems to me that it breaks it for the
user.

Is this just an example of the "New Jersey" preference for simplicity
of implementation over the "Cambridge" preference for correctness and
completeness?

Re: [9fans] mv vs cp

[forsyth]

>>Surely hard links couldn't point to directories, so it was only about
>>alternate paths to leaf objects. Thats not the same as a GOTO loop.

they could, but (at least by 5th eidtion) it was restricted to the super-user.
in fact, that was how directory rename was implemented, by setuid mv using
link and unlink.  races?  argument checking?  ``values of [beta] will give rise to dom!''.
the . and .. names were actual links in the directory, put there by the mkdir command.
the rmdir command did the 3 unlinks (for ., .., and the name itself).

Re: [9fans] mv vs cp

[George Michaelson]

> path.  Even on pre-7th Edition Unix, it was fairly easy to make
> a cyclic structure using (hard) links.  But tree-walking has
> been common and useful for decades despite those opportunities
> for cyclic behavior.

Surely hard links couldn't point to directories, so it was only about
alternate paths to leaf objects. Thats not the same as a GOTO loop.

> 
> As the WWW has shown, structures don't have to be perfect to
> be very useful.  We're still waiting for Xanadu..

I entirely agree. I think that symlinks can be viewed as runtime
evaluated conditionals in the filespace. That it means perfoming
more work than traversing the hard link is true. That it lets
you perform a large number of tasks that want to walk into partitioned
spaces it also true I think. 

That the only extant implementation has severe flaws and inconsistencies
is a damn shame.

Didn't the guys one stack down on the shoulders of the giants decide to
walk away from symlinks because it was hard to get right, more than
because  philosophically they were 'wrong'? Unlike with dead physicists,
we get to ask these questions and sometimes get answers more than conjecture.

-George

Re: [9fans] mv vs cp

[Alexander Viro]

On Tue, 9 Oct 2001, Thomas Bushnell, BSG wrote:

> It's adequate to do this on a per-filesystem basis (assuming you
> prohibit cross-device renames), so the Hurd can conveniently do it in

BTDT (see fs/namei.c::vfs_rename_dir() in Linux 2.4.x).  But that's far
from the only annoying piece of shit in rename(2) semantics.  E.g. you
are getting very ugly locking rules if you want to handle the case of
rename() removing an empty directory correctly.

Again, I've really been there and done that.  It wasn't pretty and I
would be much happier if rename() couldn't change the topology.  We
couldn't do that since it would break a lot of userland stuff;  Plan 9
has a luxury of dropping annoying crap and being able to find and fix
resulting breakage in userland.  In this case crap is not there in the
first place, so introducing it would cause a lot of ugliness in the kernel
for no good reason.

> individual servers (and does); Plan 9 could do the same I would
> suspect.

I suspect that Hurd has slightly, erm, different design philosophy (and
that's about the only printable comment I can make on the thing, so let's
not go there).

Re: [9fans] mv vs cp

[Douglas A. Gwyn]

rob pike wrote:
> This is because the file system isn't a tree ...

? It has a root, and in every directory including the root there
are names that refer to things that are either directories or not.
The only thing that would keep those properties from imposing a
tree structure would be the existence of cycles.  Sure, it is
possible to construct Moebius poiuyts, but most of the time if
someone wants to invoke a tree walker it is because he *knows*
that no such shenanigans are present.  On UNIX variants these
days, there are symbolic links that allow cycles, but all that
is necessary to keep them from causing problems is to flag an
error when path-walking revisits a node that is in the parent
path.  Even on pre-7th Edition Unix, it was fairly easy to make
a cyclic structure using (hard) links.  But tree-walking has
been common and useful for decades despite those opportunities
for cyclic behavior.

As the WWW has shown, structures don't have to be perfect to
be very useful.  We're still waiting for Xanadu..

Re: [9fans] mv vs cp

[Thomas Bushnell, BSG]

viro@math.psu.edu (Alexander Viro) writes:

> I'm pretty sure in the opposite and unfortunately *BSD source
> happens to agree with me.  Consider the scenario I've described
> upthread (s/ELOOP/EINVAL/ - sorry about the braino) and think what
> happens if both renames get to ufs_checkpath() simultaneously.  See
> /src/ufs/ufs/ufs_vnode.c::ufs_rename() for details.  Notice that
> none of the lookups is going to come anywhere near the vnodes
> affected by another rename() and ufs_checkpath() is called with fvp
> unlocked and it unlocks dp (parent of fvp).

It would certainly be a bug if two processes got to ufs_checkpath
simultaneously.  

My recollection is that BSD 4.3 contained code to serialize directory
renames, which solves the problem quite nicely.  I can't find an
analog in NetBSD (which of course has totally changed the organization
of the filesystem code since BSD 4.3).

It's adequate to do this on a per-filesystem basis (assuming you
prohibit cross-device renames), so the Hurd can conveniently do it in
individual servers (and does); Plan 9 could do the same I would
suspect.

Thomas

Re: [9fans] mv vs cp

[Richard]

okamoto@granite.cias.osakafu-u.ac.jp writes:

>I don't understand your stance anymore...
>Aren't you who recommended us EROS?
>Isn't the file system of EROS directory graph?

my stance is that OS designers should learn about
capability OSes like EROS and that Plan 9 should 
be made more attractive to people used to Unix's mv
command.  do you detect an inconsistency in
that?

Re: [9fans] mv vs cp

[okamoto]

[-- Attachment #1: Type: text/plain, Size: 139 bytes --]

I don't understand your stance anymore...
Aren't you who recommended us EROS?
Isn't the file system of EROS directory graph?

Kenji


[-- Attachment #2: Type: message/rfc822, Size: 2501 bytes --]

From: Richard <greon@best.com>
To: 9fans@cse.psu.edu
Subject: Re: [9fans] mv vs cp
Date: Mon, 08 Oct 2001 08:37:59 -0700
Message-ID: <E15qcTf-0007iJ-00@localhost>

jmk@plan9.bell-labs.com writes:

>How often do you do a move of a whole tree, not just a rename?
>I can't remember the last time I did other than a rename or a copy
>of a few files. But I'm probably not the average user and I don't
>ever use a local disc.

I do it all the time (at least 5, 10 times a month) on Linux, because
I'm kind of compulsive about maintaining just the right names for data
files I care about.

The point is not that it is necessary, but rather that millions are
already used to being able to do it, and letting them do it on Plan 9
(provided of course it is implemented cleanly) will lower the obstacle
to their become Plan 9 users.

Re: [9fans] mv vs cp

[presotto]

There are a number of solutions being aired here:

1) avoiding a copy the data and perhaps some of the metadata
2) on a recursive call, avoiding a message per file
3) getting it right if two people/programs do it simultaneously

The first 2 are performance, the last is adding something that
you can't do with our current mv.

(3) seems to be a solving a problem in the wrong place.
If two people simultaneously move a graph, the result is going to
be confusing/scarey even if everything resides in the same file
server and we somehow manage to lock the whole file server while
it happens, since in the absolute best case one is going to see
his graph inexpicably disappear.  This is more a social problem than
a technical one.  If programs are going to do it, they'ld better
set locks somewhere and have some agreement on what name space is
being looked at.

In plan 9, as rob pointed out, there's no way
for the file server to know what your name space is.
Therefore, the application or kernel has to walk the whole
subspace being moved and then try to create a set of requests
to the relevant file servers.  Not only will this set not 
be locked but you'll have done a large amount of walking
to figure out what to do.  Because of this, I think we're stuck with
forgetting about (2) and (3) without majorly changing how mounts
work.

(1), on the otherhand, may be possible.  It still requires that
you walk both paths and make sure they end up on the same
server.  Then you can have a message that says 'mv fid1 to fid2'.
I think that that's the best you can do, i.e., file by file.
You might improve that to a directory mv but you'ld have to
know that nothing was mounted below the source directory.
That is probably more trouble than its worth.

The file server can always say 'error - unimplemented' and
you can fall back to the create/copy/rm.  That way you
don't have to require exportfs, ftpfs, upas/fs and a
slurry of other impossible to understand servers to
change.

By the way, think about the following scenario:

	mkdir x
	> x/y
	bind -a x .
	mv y z

What do you want to happen here?  Should y change directory when
renamed to z or not?

On the whole, I don't think it worth the trouble.  As the above
example points out, the result of mv is already pretty confusing
in our namespace.  Whatever solution you choose should make things
less confusing, not more.

Re: [9fans] mv vs cp

[Lucio De Re]

On Mon, Oct 08, 2001 at 10:46:44AM -0400, rob pike wrote:
> 
> Do you?  A mv-tree thingy would require the server to know the name
> space of the client to get this right.  The server doesn't know that
> one of the files in the client's tree is somewhere else.  I honestly
> don't see a reasonable way to do this right, even if we don't worry
> about race conditions (and we do).
> 
Wait, I'm being obtuse here.  Do you see a "mv-tree thingy" as
essential to a "rename" function?  I'm going along with Dave's
suggestion that the kernel determine the fids of source and
destination (and I'm using fids loosely because I'm not really
familiar with them) and then submit then (Tmv or, my preference,
Trename) to the fileserver for actioning.

Moves that require data to be tranferred are forbidden in my model
(they can be handled by user-space operations, I suppose) and damage
to the namespace is the user's responsibility.

Again, in my ignorance, I assume that one would create an anonymous
node at the destination and point it (in some fashion I really
don't know enough about, but I'll be pleased to have my nose rubbed
into the details of, whichever way) to the fileserver entity
originally pointed at by the source.  The original connection would
then be removed and name transfer take place (roughly).  Please
educate me on this, reading the documents and man pages didn't make
the picture any clearer.

I certainly can't agree that the fileserver won't know that a file
is elsewhere: if the nameserver can't tell which files it serves
we have a serious problem.  Now, if by the time the fileserver
considers a file it has an id that looks like a local one, we
certainly have a crisis, but I'm hoping that the kernel can prevent
this _at_the_root_ of the rename, yet I hear continuous references
to the remainder of the graph, which puzzles me because I don't
see rename navigating the graph at all.  I'm sure I'm wrong, so
please tell me where.

As for Alexander's example, I really need to look at it more
carefully, I don't understand the details at all.

++L

Re: [9fans] mv vs cp

[anothy]

// But the idea of a filename, booked into the filename space which says
// 
// 	"I may not resolve all the time, but if I do, I point to <x>"
// 
// Isn't so inherently evil to me.

it is to me. how do i know if it does or doesn't? what's the condition?
what's the behaviour if it doesn't? hell, what's the behaviour if it _does_
resolve? questions of whether to follow links or not... it's just never
seemed worth it. the _only_ time i've missed symlinks in Plan 9 is in
importing packages from elsewhere. this simpler version of what you
have above, which _wouldn't_ seem inherently evil to me, would be:
	"I point to <x>, where x is a set of data."
much simpler, much easier to implement (uh, like, files?), much more
consistant, and thus easier to understand. even multiple hard links
under unix satisfy this: if it's there, it points to a certain set of data.
no condidtions on that.

// Pointing out M$ has .LNK isn't going to help my cause much is it

no, not likely. it's been a while since i've poked at any Windows box,
but under at least '95 and '98revA, drop into a shell prompt and try
to cat^Wtype foo.lnk. it's not even built into the FS, it's interpreted by
the tools, like explorer.
-α.

Re: [9fans] mv vs cp

[William Josephson]

On Mon, Oct 08, 2001 at 08:37:59AM -0700, Richard wrote:
> I do it all the time (at least 5, 10 times a month) on Linux, because
> I'm kind of compulsive about maintaining just the right names for data
> files I care about.
> 
> The point is not that it is necessary, but rather that millions are
> already used to being able to do it, and letting them do it on Plan 9
> (provided of course it is implemented cleanly) will lower the obstacle
> to their become Plan 9 users.

But Plan 9 is not Unix and the more one tries to make it look like
Unix, feel like Unix, or quack like Unix, the more grief one comes to.
Under Unix I use mv to move trees around fairly frequently (as often
as once or twice a day), although at least half the time it is across
partitions so the point is moot anyway.  The urge to do this
disappeared pretty quickly when I started using Plan 9 to the
exclusion of Unix this summer.  I'm more of a neat-freak when it comes
to directory structure than most and I still didn't feel the loss.
Perhaps having a real fileserver makes the difference, but I'd suggest
living for a while with Plan 9 as Plan 9 and not Unix and see if you
still find it intolerable.  I think the biggest obstacle to using Plan
9 is knowing that it was built in the same room as Unix.

  -WJ

Re: [9fans] mv vs cp

[Richard]

jmk@plan9.bell-labs.com writes:

>How often do you do a move of a whole tree, not just a rename?
>I can't remember the last time I did other than a rename or a copy
>of a few files. But I'm probably not the average user and I don't
>ever use a local disc.

I do it all the time (at least 5, 10 times a month) on Linux, because
I'm kind of compulsive about maintaining just the right names for data
files I care about.

The point is not that it is necessary, but rather that millions are
already used to being able to do it, and letting them do it on Plan 9
(provided of course it is implemented cleanly) will lower the obstacle
to their become Plan 9 users.

Re: [9fans] mv vs cp

[Markus Friedl]

On Mon, Oct 08, 2001 at 11:00:24AM -0400, Alexander Viro wrote:
> Actually, there's another fun issue here:
> 
> bind /foo/bar/baz /quux
> cd /quux/crap
> mv /foo/bar/baz/crap /foo/bar/crap
> cd ..
> 
> Where should we end up?  I have a somewhat reasonable answer for our
> semantics of bindings, but I don't see it for Plan 9 one.

there are probably many ways to mess with your own
namespace. but why should the system care? after
all it's your own process-local namespace.

Re: [9fans] mv vs cp

[Alexander Viro]

On Mon, 8 Oct 2001, rob pike wrote:

> > We accept that the "trees" (you're perfectly right about directed
> > graphs) are rooted at the same fileserver, I think.
> 
> Do you?  A mv-tree thingy would require the server to know the name
> space of the client to get this right.  The server doesn't know that
> one of the files in the client's tree is somewhere else.  I honestly
> don't see a reasonable way to do this right, even if we don't worry
> about race conditions (and we do).

Actually, there's another fun issue here:

bind /foo/bar/baz /quux
cd /quux/crap
mv /foo/bar/baz/crap /foo/bar/crap
cd ..

Where should we end up?  I have a somewhat reasonable answer for our
semantics of bindings, but I don't see it for Plan 9 one.

Re: [9fans] mv vs cp

[rob pike]

> We accept that the "trees" (you're perfectly right about directed
> graphs) are rooted at the same fileserver, I think.

Do you?  A mv-tree thingy would require the server to know the name
space of the client to get this right.  The server doesn't know that
one of the files in the client's tree is somewhere else.  I honestly
don't see a reasonable way to do this right, even if we don't worry
about race conditions (and we do).

-rob

Re: [9fans] mv vs cp

[Lucio De Re]

On Mon, Oct 08, 2001 at 09:03:28AM -0400, rob pike wrote:
> 
> What should mv do to a `tree' that resides on multiple file servers?
> If you can't do something right, sometimes it's not worth doing at
> all.
> 
We accept that the "trees" (you're perfectly right about directed
graphs) are rooted at the same fileserver, I think.  Moving the root
to a new location on the fileserver isn't going to be affected by
portions below it that may stride other filesystems.

In my ignorance, I presume that only the root needs to move around and
the rest (hopefully acyclically) will remain unchanged.  Does the
possibility exist that we may be creating loops?

++L

Re: [9fans] mv vs cp

[rob pike]

What should mv do to a `tree' that resides on multiple file servers?
If you can't do something right, sometimes it's not worth doing at
all.

-rob

Re: [9fans] mv vs cp

[rob pike]

> writing correct tree-walker
> that would deal gracefully with cross-directory renames is _nasty_.

This is because the file system isn't a tree and hasn't been for a very
long time. It's a graph, but people still talk about it as thought it were
a tree and write simple recursive algorithms and get themselves in
trouble all the time.

-rob

Re: [9fans] mv vs cp

[Lucio De Re]

On Mon, Oct 08, 2001 at 09:50:27AM +0000, Douglas A. Gwyn wrote:
> 
> We should be careful to distinguish between a utility command,
> which usually is best if it works as widely as it can to do a
> simply described function (from the user point of view; it might
> be complicated to implement), and a fundamental operation in the
> protocol itself (I/O primitive).
> 
With the principle of least surprise applying here.  Mv copying only
one layer of files is in breach of it, in my opinion.

> I think it is reasonable for the 9P200x protocol to have a
> Trename request, which of course can fail if the operation is
> not possible or makes no sense for the particular object.
> Then it is up to the (file) server to make it happen; the
> important thing is that the file server need only use local
> locks while it shuffles things around (if precached, it could
> be serviced as an atomic operation).
> 
I think there's growing agreement on this score.  I certainly feel
it's justified.  But Jim's reservations about race conditions have to
be taken into account.  Alexander seems to think BSD did it wrong (did
Linux improve on that?) but didn't suggest it can't be done.

> There is the question of what permissions should be required
> for a rename to succeed: write access on the directory, and/or
> on the named object itself?  This may be the same question as
> whether the rename ought to be for a handle in the directory
> (i.e. Trename is an operation on a directory) or for a handle
> on the named object.
> 
These are rules, they need to be established and documented (and make
sense, of course).  But that's about it, really.

> I am inclined to think rename ought to work within a directory
> but not in general between distinct path prefixes.

My choice would be the point where copying the data becomes necessary.
Whatever operation doesn't need to relocate the actual file contents,
ought to succeed.  But I can't say I understand all the subtleties
involved.

++L

Re: [9fans] mv vs cp

[Boyd Roberts]

"Thomas Bushnell, BSG" wrote:
> I'm pretty sure you're incorrect.  Can you describe the case in more
> detail?

They spent quite a while in rename hell.

Re: [9fans] mv vs cp

[Alexander Viro]

On Mon, 8 Oct 2001, Thomas Bushnell, BSG wrote:

> viro@math.psu.edu (Alexander Viro) writes:
> 
> > So can anybody who had seen current 4.4BSD derivatives.  _None_ of them
> > got it right.  rename()/rename() race (creating a loop and detaching it
> > from the rest of tree) is still there.
> 
> I'm pretty sure you're incorrect.  Can you describe the case in more
> detail? 

I'm pretty sure in the opposite and unfortunately *BSD source happens to
agree with me.  Consider the scenario I've described upthread (s/ELOOP/EINVAL/
- sorry about the braino) and think what happens if both renames get to
ufs_checkpath() simultaneously.  See /src/ufs/ufs/ufs_vnode.c::ufs_rename()
for details.  Notice that none of the lookups is going to come anywhere near
the vnodes affected by another rename() and ufs_checkpath() is called with
fvp unlocked and it unlocks dp (parent of fvp).

Re: [9fans] mv vs cp

[Thomas Bushnell, BSG]

jmk@plan9.bell-labs.com writes:

> How often do you do a move of a whole tree, not just a rename?
> I can't remember the last time I did other than a rename or a copy
> of a few files. But I'm probably not the average user and I don't
> ever use a local disc.

I do it all the time.  

Re: [9fans] mv vs cp

[Thomas Bushnell, BSG]

viro@math.psu.edu (Alexander Viro) writes:

> So can anybody who had seen current 4.4BSD derivatives.  _None_ of them
> got it right.  rename()/rename() race (creating a loop and detaching it
> from the rest of tree) is still there.

I'm pretty sure you're incorrect.  Can you describe the case in more
detail? 

Re: [9fans] mv vs cp

[Douglas A. Gwyn]

jmk@plan9.bell-labs.com wrote:
> How often do you do a move of a whole tree, not just a rename?
> I can't remember the last time I did other than a rename or a copy
> of a few files. But I'm probably not the average user and I don't
> ever use a local disc.

I frequently move large trees around, but often they are not within
the same filesystem.

Re: [9fans] mv vs cp

[Douglas A. Gwyn]

Lucio De Re wrote:
> There are really only two social factors overriding here.  People
> like me have grown to expect it, unfortunate as it may be, and the
> alternative is inconsistent and open to error, besides being much
> more time and resource intensive.

We should be careful to distinguish between a utility command,
which usually is best if it works as widely as it can to do a
simply described function (from the user point of view; it might
be complicated to implement), and a fundamental operation in the
protocol itself (I/O primitive).

I think it is reasonable for the 9P200x protocol to have a
Trename request, which of course can fail if the operation is
not possible or makes no sense for the particular object.
Then it is up to the (file) server to make it happen; the
important thing is that the file server need only use local
locks while it shuffles things around (if precached, it could
be serviced as an atomic operation).

There is the question of what permissions should be required
for a rename to succeed: write access on the directory, and/or
on the named object itself?  This may be the same question as
whether the rename ought to be for a handle in the directory
(i.e. Trename is an operation on a directory) or for a handle
on the named object.

I am inclined to think rename ought to work within a directory
but not in general between distinct path prefixes.

Re: [9fans] mv vs cp

[Thomas Bushnell, BSG]

jmk@plan9.bell-labs.com writes:

> As Dave points out, there are a lot of balls in the air during
> an atomic 'rename'. 4.2BSD introduced the 'rename' system call,
> and as an alpha/beta tester for 4.1[abc] and 4.2BSD I can testify
> to how long it took to get it right and how much ugly code was
> involved.

Color me confused, but isn't it a jillion times harder to get it right
in user space?

Re: [9fans] mv vs cp

[Thomas Bushnell, BSG]

rsc@plan9.bell-labs.com (Russ Cox) writes:

> How would you determine whether it would work?
> How would you specify it to the underlying 9P server?

We had this problem in the Hurd, and the answer was give to one of the
servers for the two names (it doesn't matter which) your capability
for the other server.  

Then if both are handled by the same server (what Unix thinks of as
"same filesystem") it can be done directly.  If not, the server could
return EXDEV (as Unix does).  Or, two cooperating servers might have a
private protocol for arranging it between them.  

I don't know whether this would work in Plan 9, though I'm interested
in hearing details.  Can you hand your capability for one server off
to the other?  

Re: [9fans] mv vs cp

[Lucio De Re]

On Mon, Oct 08, 2001 at 05:10:27PM +1000, George Michaelson wrote:
> 
> Pointing out M$ has .LNK isn't going to help my cause much is it :-)
> 
Well, and a registry.  In Plan 9's idiom, they could do with a few
more iterations.

One valid point is that computers are able to mimic real life more
closely, but I'm of the school that was trained on deterministic
behaviour.

Non-deterministic bahaviour is a curse and a sufficiently vast
computer system is non-deterministic.  Dijkstra had trouble with
interrupts, but I believe they could be assimilated into a
deterministic framework, modern bloat is orders of magnitude too
complex for such treatment.

There's a strange commercial strength in non-determinism, basically
getting the luser to accept failure as if it were essential.  I
suppose that was the M$ genius: don't let the user expect perfection
in anything but the _next_ generation of software.

++L

Re: [9fans] mv vs cp

[Alexander Viro]

[I suggest taking the thread off-list - further details of rename() races
in *BSD may be interesting, but they are off-topic for 9fans]

On Mon, 8 Oct 2001, George Michaelson wrote:

> Are you saying that this problem demonstrably exploited the race condition
> between cp/mv and rename as implemented in FreeBSD?

	Yes.  ufs_rename() is racy and yes, that race is wide enough to be
exploitable.  BTDT.
 
> I really do mean the question as put:
> 
> when was the last time anybody saw a successful exploit of this race condition
> or an unstable filesystem they can show came from it, exploit or accident?
> 
> I have seen many problems with UFS/FFS, and Softupdates gave me the willeys
> but I have also not yet seen serious corruption of the on-disk state which
> lies directly with problems in the FS code itself. Side-effects of kernel
> crashes during meta-state updates, sure. But this sounds to me like FUD which
> in practice doesn't exist.

	process 1:  current directory in /tmp/a/a/a/a/a, does
rename("/tmp/b/b", "a");
	process 2:  current directory in /tmp/b/b/b/b/b, does
rename("/tmp/a/a", "b");

	Normal outcome: first process to do rename() succeeds, second - fails
with ELOOP.  With the right timing _both_ succeed, creating a loop and
detaching it from the rest of filesystem.

	Notice that use of relative pathnames is critical here - otherwise
lookup in the second rename() will block on the lock acquired by the first
one.  Code in /sys/ufs/ufs/ufs_vnops.c implicitly assumes that lookups
ending in descendants of directory will have to pass through that directory.
That assumption is obviously false - namei(9) can start in a descendant of
directory in question.

	And no, it's not too narrow - window includes quite a bit of disk
IO.  Figuring out details of turning that into full-blown attack (i.e.
what should be done to widen the window) are left as an exercise to anyone
who can RTFS - it's pretty straightforward.

RE: [9fans] mv vs cp

[Skip Tavakkolian]

Gator Grip, comes with a handy ratchet and a drill attachment too.

At 03:11 PM 10/7/2001 -0400, presotto@closedmind.org wrote:
>Sorry about that.  I didn't see that the return address was
>9fans.  But if anyone can come up with the name of the tool,
>I'ld be a happy camper...
>
>

Re: [9fans] mv vs cp

[Alexander Viro]

On Mon, 8 Oct 2001, Lucio De Re wrote:

> On Mon, Oct 08, 2001 at 04:49:43PM +1000, George Michaelson wrote:
> > 
> > > Any time when attacker feels like that.  System where nonprivileged users
> > > can cause filesystem corruption is broken.  Period.
> > 
> > Umm yes, but Alexander, when was the last time you *saw* one of these?
> > 
> It only needs to happen once.  Code Red/NIMDA anyone?

Now, now.  It's nowhere near the nastiness of remote root compromise, but
yes, "nobody will ever try to screw me" is exactly the attitude that made
them possible.

As for the original question - two weeks ago, when I had demonstrated the
effect to a guy who claimed the OpenBSD kernel was "bulletproof".  OTOH, in
case of OpenBSD it's one of the mildest problems - there being able to do
rfork(RFPROC) means being able to cause kernel panic (races between
fstat()/close(), dup2()/close(), write()/close() - you name it).

Re: [9fans] mv vs cp

[George Michaelson]

> On Mon, Oct 08, 2001 at 04:49:43PM +1000, George Michaelson wrote:
> > 
> > > Any time when attacker feels like that.  System where nonprivileged use
rs
> > > can cause filesystem corruption is broken.  Period.
> > > 
> > 
> > Umm yes, but Alexander, when was the last time you *saw* one of these?
> > 
> It only needs to happen once.  Code Red/NIMDA anyone?
> 
> ++L
> 

Are you saying that this problem demonstrably exploited the race condition
between cp/mv and rename as implemented in FreeBSD?

I really do mean the question as put:

when was the last time anybody saw a successful exploit of this race condition
or an unstable filesystem they can show came from it, exploit or accident?

I have seen many problems with UFS/FFS, and Softupdates gave me the willeys
but I have also not yet seen serious corruption of the on-disk state which
lies directly with problems in the FS code itself. Side-effects of kernel
crashes during meta-state updates, sure. But this sounds to me like FUD which
in practice doesn't exist.

You could probably argue half a million potential race conditions exist in
lots of systems. The frequency they occur is different.

-George
--
George Michaelson       |  APNIC
Email: ggm@apnic.net    |  PO Box 2131 Milton QLD 4064
Phone: +61 7 3367 0490  |  Australia
  Fax: +61 7 3367 0482  |  http://www.apnic.net

Re: [9fans] mv vs cp

[George Michaelson]

> I suppose the fact that some, like me, will have trouble understanding
> the discussion (took me a while just to figure the Directed Acyclical
> Graph reference) I suppose religious war is to be expected?
> 
> If I understand the above correctly, my take is that symbolic links
> are totally unchecked filesystem GOTOs and deserve a much worse fate
> than they get.  Specially as they can point to the void and often do.
> 

Riiight, but that was designed in. Its a goal. They wanted that behaviour.

> As to the rules (-L) for when they are dereferenced or not, that's a
> morrass of confusion.

No argument. That could have been done so much better.

But the idea of a filename, booked into the filename space which says

	"I may not resolve all the time, but if I do, I point to <x>"

Isn't so inherently evil to me. it seems to me it mimics real-world behaviours
rather well.

Pointing out M$ has .LNK isn't going to help my cause much is it :-)

cheers
	-George

Re: [9fans] mv vs cp

[Lucio De Re]

On Mon, Oct 08, 2001 at 04:49:43PM +1000, George Michaelson wrote:
> 
> > Any time when attacker feels like that.  System where nonprivileged users
> > can cause filesystem corruption is broken.  Period.
> > 
> 
> Umm yes, but Alexander, when was the last time you *saw* one of these?
> 
It only needs to happen once.  Code Red/NIMDA anyone?

++L

Re: [9fans] mv vs cp

[Lucio De Re]

On Mon, Oct 08, 2001 at 04:10:29PM +1000, George Michaelson wrote:
> 
> I understand its a deeply held religion, but what IS it with runtime/use-time
> reference by name which people find so offensive in the filesystem apart from
> implementation complexity? We have call by name/value/reference in code, why
> not in namespaces?
> 
I suppose the fact that some, like me, will have trouble understanding
the discussion (took me a while just to figure the Directed Acyclical
Graph reference) I suppose religious war is to be expected?

If I understand the above correctly, my take is that symbolic links
are totally unchecked filesystem GOTOs and deserve a much worse fate
than they get.  Specially as they can point to the void and often do.

As to the rules (-L) for when they are dereferenced or not, that's a
morrass of confusion.

++L

Re: [9fans] mv vs cp

[nigel]

[-- Attachment #1: Type: text/plain, Size: 131 bytes --]

Well, I've seen only one Windows machine infected with a virus my
whole life. Am I to conclude that viruses are not a problem?


[-- Attachment #2: Type: message/rfc822, Size: 2137 bytes --]

From: George Michaelson <ggm@apnic.net>
To: 9fans@cse.psu.edu
Subject: Re: [9fans] mv vs cp
Date: Mon, 08 Oct 2001 16:49:43 +1000
Message-ID: <4615.1002523783@apnic.net>


> Any time when attacker feels like that.  System where nonprivileged users
> can cause filesystem corruption is broken.  Period.
> 

Umm yes, but Alexander, when was the last time you *saw* one of these?

-George
--
George Michaelson       |  APNIC
Email: ggm@apnic.net    |  PO Box 2131 Milton QLD 4064
Phone: +61 7 3367 0490  |  Australia
  Fax: +61 7 3367 0482  |  http://www.apnic.net

Re: [9fans] mv vs cp

[George Michaelson]

> Any time when attacker feels like that.  System where nonprivileged users
> can cause filesystem corruption is broken.  Period.
> 

Umm yes, but Alexander, when was the last time you *saw* one of these?

-George
--
George Michaelson       |  APNIC
Email: ggm@apnic.net    |  PO Box 2131 Milton QLD 4064
Phone: +61 7 3367 0490  |  Australia
  Fax: +61 7 3367 0482  |  http://www.apnic.net

Re: [9fans] mv vs cp

[Alexander Viro]

On Mon, 8 Oct 2001, George Michaelson wrote:

> 
> > So can anybody who had seen current 4.4BSD derivatives.  _None_ of them
> > got it right.  rename()/rename() race (creating a loop and detaching it
> > from the rest of tree) is still there.
> >  
> 
> So can you say how often in practice that race condition is exercised? 

Any time when attacker feels like that.  System where nonprivileged users
can cause filesystem corruption is broken.  Period.

Re: [9fans] mv vs cp

[Lucio De Re]

On Sun, Oct 07, 2001 at 10:45:24PM -0700, Mike Haertel wrote:
> 
> Often.  This is one of the things that most irritates me about Plan 9.
> 
Talk about extreme responses :-)

++L

Re: [9fans] mv vs cp

[Lucio De Re]

On Mon, Oct 08, 2001 at 01:05:30AM -0400, jmk@plan9.bell-labs.com wrote:
> 
> How often do you do a move of a whole tree, not just a rename?
> I can't remember the last time I did other than a rename or a copy
> of a few files. But I'm probably not the average user and I don't
> ever use a local disc.

Because I'm steeped in SysV (which is where I cut my Unix teeth), I
wouldn't find it hard to use "find . -print | cpio -pvdm dest" as
taught in one of the man pages in my early days.  So, to answer your
question truthfully: "Never".

++L

Re: [9fans] mv vs cp

[George Michaelson]

> So can anybody who had seen current 4.4BSD derivatives.  _None_ of them
> got it right.  rename()/rename() race (creating a loop and detaching it
> from the rest of tree) is still there.
>  

So can you say how often in practice that race condition is exercised? 

> > I should imagine that BSD "rename" must have been made orders of
> > magnitude more complex by symbolic links.  After all, SysV 3.2 "mv"
> > could rename directories into other directories in the same
> > filesystem, couldn't it?  It's been a long time since my 3B2 was
> 
> Same race + metric buttload of other fun stuff.

turning a strict tree into a mesh/DAG is always fun. But, its also possibly
useful for some people. Me? I liked the newcastle connection but we don't
have /.../ so I guess I lost. Namespaces might be in the same spirit.

I understand its a deeply held religion, but what IS it with runtime/use-time
reference by name which people find so offensive in the filesystem apart from
implementation complexity? We have call by name/value/reference in code, why
not in namespaces?

-George
--
George Michaelson       |  APNIC
Email: ggm@apnic.net    |  PO Box 2131 Milton QLD 4064
Phone: +61 7 3367 0490  |  Australia
  Fax: +61 7 3367 0482  |  http://www.apnic.net

Re: [9fans] mv vs cp

[Mike Haertel]

>How often do you do a move of a whole tree, not just a rename?
>I can't remember the last time I did other than a rename or a copy
>of a few files. But I'm probably not the average user and I don't
>ever use a local disc.

Often.  This is one of the things that most irritates me about Plan 9.

In particular, the oft-cited idiom of using tar to copy the hierarchy
is especially undesirable under Plan 9 because it generates lots of
garbage that will fill up your dump filesystem much faster.

To what extent do you think your expectations and habits are shaped by
the limitations of your tools?

Re: [9fans] mv vs cp

[jmk]

How often do you do a move of a whole tree, not just a rename?
I can't remember the last time I did other than a rename or a copy
of a few files. But I'm probably not the average user and I don't
ever use a local disc.

Re: [9fans] mv vs cp

[Alexander Viro]

On Mon, 8 Oct 2001, Lucio De Re wrote:

> On Sun, Oct 07, 2001 at 12:23:59PM -0400, jmk@plan9.bell-labs.com wrote:
> > 
> > As Dave points out, there are a lot of balls in the air during
> > an atomic 'rename'. 4.2BSD introduced the 'rename' system call,
> > and as an alpha/beta tester for 4.1[abc] and 4.2BSD I can testify
> > to how long it took to get it right and how much ugly code was
> > involved.

So can anybody who had seen current 4.4BSD derivatives.  _None_ of them
got it right.  rename()/rename() race (creating a loop and detaching it
from the rest of tree) is still there.
 
> There are really only two social factors overriding here.  People
> like me have grown to expect it, unfortunate as it may be, and the
> alternative is inconsistent and open to error, besides being much
> more time and resource intensive.

> I should imagine that BSD "rename" must have been made orders of
> magnitude more complex by symbolic links.  After all, SysV 3.2 "mv"
> could rename directories into other directories in the same
> filesystem, couldn't it?  It's been a long time since my 3B2 was

Same race + metric buttload of other fun stuff.

And then there is _really_ fun stuff in userland.  Like find / mv and
rm -rf / mv races that allow any user to create a deep tree in /tmp,
wait for cron-ran script to start removing it and do something along
the lines of mv /tmp/foo/bar/baz /tmp/quux, tricking find(1) or rm(1)
into walking up into the root and proceeding from there.  Works like
charm on almost all Unices I've seen.

Sure, it's a broken userland and it needs to be fixed whenever such
bugs are found, but the point is that writing correct tree-walker
that would deal gracefully with cross-directory renames is _nasty_.

Re: [9fans] mv vs cp

[Lucio De Re]

On Sun, Oct 07, 2001 at 12:23:59PM -0400, jmk@plan9.bell-labs.com wrote:
> 
> As Dave points out, there are a lot of balls in the air during
> an atomic 'rename'. 4.2BSD introduced the 'rename' system call,
> and as an alpha/beta tester for 4.1[abc] and 4.2BSD I can testify
> to how long it took to get it right and how much ugly code was
> involved.
> 
There are really only two social factors overriding here.  People
like me have grown to expect it, unfortunate as it may be, and the
alternative is inconsistent and open to error, besides being much
more time and resource intensive.

In addition, it may be more consistent for mv not to go part way
and do a cp+rm on my behalf _on_condition_ that the request is not
recursive, rather not do it at all.  But I suppose my MSDOS background
shows here, the "mv" I'm thinking of is "REN".  That, I ought to
be able to fix by crippling "mv".

What wasn't clear from Dave's response, is whether we would need
a "Tmv" to do it all, including recursive copies that (shudder!)
may need to understand the entire namespace, or just "Tren" that
may still change the namespace, but at least won't have to make
decisions on how to follow mounted resources.

I should imagine that BSD "rename" must have been made orders of
magnitude more complex by symbolic links.  After all, SysV 3.2 "mv"
could rename directories into other directories in the same
filesystem, couldn't it?  It's been a long time since my 3B2 was
turned off, so it's hard for me to check.  I'm almost certain SCO
Xenix and Unix could do it without copying.

++L

RE: [9fans] mv vs cp

[nigel]

[-- Attachment #1: Type: text/plain, Size: 75 bytes --]

Don't know the name, but it's been available mail order here for years.


[-- Attachment #2: Type: message/rfc822, Size: 1219 bytes --]

From: presotto@closedmind.org
To: 9fans@cse.psu.edu
Subject: RE: [9fans] mv vs cp
Date: Sun, 7 Oct 2001 15:11:27 -0400
Message-ID: <20011007191129.4E4CB1998A@mail.cse.psu.edu>

Sorry about that.  I didn't see that the return address was
9fans.  But if anyone can come up with the name of the tool,
I'ld be a happy camper...

RE: [9fans] mv vs cp

[presotto]

Sorry about that.  I didn't see that the return address was
9fans.  But if anyone can come up with the name of the tool,
I'ld be a happy camper...

RE: [9fans] mv vs cp

[presotto]

It was time.  I got cable too.  Now I've seen a $19.95
tool I want, though I can't remember the name.  It's a socket
wrench that's essentially a cylinder filled with pins that can
contour around a nut.  Seen it?  Even cooler than the metriwrench.

RE: [9fans] mv vs cp

[philw]

[-- Attachment #1: Type: text/plain, Size: 35 bytes --]

like the new email address.
phil

[-- Attachment #2: Type: text/html, Size: 423 bytes --]

Re: [9fans] mv vs cp

[jmk]

As Dave points out, there are a lot of balls in the air during
an atomic 'rename'. 4.2BSD introduced the 'rename' system call,
and as an alpha/beta tester for 4.1[abc] and 4.2BSD I can testify
to how long it took to get it right and how much ugly code was
involved.

--jim

Re: [9fans] mv vs cp

[Lucio De Re]

On Sun, Oct 07, 2001 at 08:43:53AM -0400, presotto@closedmind.org wrote:
> 
> It could be done, it just depends on what you'ld like to pay for it,
> its just code after all.
> 
I guess I'll have to put my money where my mouth is.  Don't nobody
hold their breath!  :-)

Presotto certainly covered the important facets rather nicely, it
would be a pity to drop that particular ball now.

In passing, I can't easily find a reference to the 9P2000 preliminary
information, could someone post a URL?  I seem to remember some
sort of version information being designed in which would certainly be
helpful.

++L

PS: Dave, nice domain name.

Re: [9fans] mv vs cp

[presotto]

On Sun Oct  7 02:30:21 EDT 2001, lucio@proxima.alt.za wrote:
> Am I being utterly obtuse, or does it really make sense to duplicate
> then delete files when moving them to another directory?

It could be done, it just depends on what you'ld like to pay for it,
its just code after all.

You'ld need to create a new file system message (Tmv) and a new system
call (mv).  The kernel would have to start by walking both paths and
determine if they came from the same mount point, error if not.  It would
then send a Tmv message with 2 fids.  The file system would determine
if it could do it and return an error if not (might not be space, ...).
The file system structure would also have to be changed to accomodate it, we
don't have links, though the copy on write makes that not too terrible.

We consciously traded simplicity for it.  It hasn't bothered us enough to
regret it, perhaps because a real file server is faster than kfs, perhaps
because we don't do a lot of file moving.

If you think otherwise, that's why the source is open.  Have a look and
see what you'ld need to change and then convince others that the change
is worth it.  Implementations are no big deals.  New file system messages
are since you have to change lots of file servers to go with it.

> I had the distribution archives in /dist under "kfs" and decided to do
> the following:
> 
> 	% cd /dist
> 	% mkdir plan9
> 	% mv *.9gz plan9
> 
> the machine is still busy copying plan9.9gz :-(
> 
> In this case, a "rename" would have been appropriate.  Is it not
> possible to determine whether it would work and resort to cp+rm if
> it doesn't?

This is indeed what happened, mv determined it wouldn't work and did a
cp+rm when it found it wouldn't.

Re: [9fans] mv vs cp

[Lucio De Re]

On Sat, Oct 06, 2001 at 11:42:15PM -0700, Quinn Dunkan wrote:
> 
> mv does try to rename (uses dirwstat, I think), but you can only rename to the
> same directory, check mv.c.  There was a discussion a while back about how
> hard cross directory renames are.

Sorry to be pig-headed about this, but I guess I'm spoilt (Boyd,
none of your sarcasm, please) with NetBSD doing it all for me.

Thing is, I may be representative of a largish community of spoilt
users, but there are also other considerations, for example, there
may not be enough space for the copy, a situation made worse by
the presence of a large time window in which race conditions can
occur.

Russ asks a pertinent question, how does one tell?  I'm wondering,
not being too good at the innards of Plan 9 and/or the filesystem,
whether it would not be worth sacrificing the cleanliness of the
filesystem to the ability to know.  Yes, it is a slippery slope:
once you can tell, why not move directories around, and so forth.

All I can say in self-defence is that user convenience is being
sacrificed here (I ought to know, but I can't remember how the
various Windows flavours deal with the problem) and the sacrifice
may be greater than the gain in simplicity.

I'm sure I'm wrong, but I never quite understood the underlying
limitation, so perhaps I can be pointed in the right direction and
I'll shut up for good.

++L

Re: [9fans] mv vs cp

[forsyth]

[-- Attachment #1: Type: text/plain, Size: 66 bytes --]

wstat gives a directory entry a new name;
it doesn't move it.


[-- Attachment #2: Type: message/rfc822, Size: 2314 bytes --]

From: Lucio De Re <lucio@proxima.alt.za>
To: 9fans mailing list <9fans@cse.psu.edu>
Subject: [9fans] mv vs cp
Date: Sun, 7 Oct 2001 08:29:00 +0200
Message-ID: <20011007082859.I28720@cackle.proxima.alt.za>

Am I being utterly obtuse, or does it really make sense to duplicate
then delete files when moving them to another directory?

I had the distribution archives in /dist under "kfs" and decided to do
the following:

	% cd /dist
	% mkdir plan9
	% mv *.9gz plan9

the machine is still busy copying plan9.9gz :-(

In this case, a "rename" would have been appropriate.  Is it not
possible to determine whether it would work and resort to cp+rm if
it doesn't?

In fact, a rename command would be an adequate compromise.  I see
there's a "kfs" command called "rename" but the man page does not
specify its actual scope.  Should I have used that (one file at the
time) instead?

++L

Re: [9fans] mv vs cp

[Quinn Dunkan]

> Am I being utterly obtuse, or does it really make sense to duplicate
> then delete files when moving them to another directory?

mv does try to rename (uses dirwstat, I think), but you can only rename to the
same directory, check mv.c.  There was a discussion a while back about how
hard cross directory renames are.

Re: [9fans] mv vs cp

[Russ Cox]

> In this case, a "rename" would have been appropriate.  Is it not
> possible to determine whether it would work and resort to cp+rm if
> it doesn't?

How would you determine whether it would work?
How would you specify it to the underlying 9P server?

Don't use kfs rename; it doesn't move files 
between directories.

Russ

[9fans] mv vs cp

[Lucio De Re]

Am I being utterly obtuse, or does it really make sense to duplicate
then delete files when moving them to another directory?

I had the distribution archives in /dist under "kfs" and decided to do
the following:

	% cd /dist
	% mkdir plan9
	% mv *.9gz plan9

the machine is still busy copying plan9.9gz :-(

In this case, a "rename" would have been appropriate.  Is it not
possible to determine whether it would work and resort to cp+rm if
it doesn't?

In fact, a rename command would be an adequate compromise.  I see
there's a "kfs" command called "rename" but the man page does not
specify its actual scope.  Should I have used that (one file at the
time) instead?

++L