Re: [9fans] Playing with unwrap on Unix

Re: [9fans] Playing with unwrap on Unix

[Russ Cox]

> Doing the right thing for directory mtimes, when the directory itself
> is not in the wrap, is a bit more difficult; ideally any directory
> containing files would have an mtime that was the maximum of mtimes of
> those contained files (this should already be so for directories which

This is true, because we set every mtime in the whole
archive to be the same -- the date of distribution.
But in general it's not true of any file system I know of.

> are themselves in the wrap), but setting the correct mtime on a
> directory not in the wrap, that has only directories, that gets

These should not exist.  If a path is included, so is an entry
for each directory along the way.  (I think.)

> affected by the wrap, is *really* difficult.  Looking at 9e.c, it
> appears that ctime - and hence directory creation time? - is not stored
> in a wrap, making the necessary information unavailable.

Yes, ctime is omitted.  Please don't tell me you can set ctime.

Russ

Re: [9fans] Playing with unwrap on Unix

[Matthew C Weigel]

Russ Cox <9fans@cse.psu.edu> wrote:

>> Doing the right thing for directory mtimes, when the directory itself
>> is not in the wrap, is a bit more difficult; ideally any directory
>> containing files would have an mtime that was the maximum of mtimes of
>> those contained files (this should already be so for directories which
>
>This is true, because we set every mtime in the whole
>archive to be the same -- the date of distribution.

That is not what it looks like to me: in the 6/17/00 archfs.9gz, wrap/
has an mtime of 6/4/00, while everything else has an mtime of 6/17/00.

Would you mind unwrapping that on a Plan 9 system and telling me what
all the files are, and their mtimes?

>But in general it's not true of any file system I know of.

Sure - it's an approximation of 'when the last file in the directory
was created' - something that would be helped by storing ctime (with
the same caveat as below).

>> are themselves in the wrap), but setting the correct mtime on a
>> directory not in the wrap, that has only directories, that gets
>
>These should not exist.  If a path is included, so is an entry
>for each directory along the way.  (I think.)

I've been doing my testing with the 6/17 archfs.9gz. It contains
386/bin/archfs and sys/src/cmd/archfs.c but not 386, 386/bin, sys, and
so on.  At least according to the unwrap perl script (original).

The script would have to be a lot less clever if those didn't exist.

>> affected by the wrap, is *really* difficult.  Looking at 9e.c, it
>> appears that ctime - and hence directory creation time? - is not stored
>> in a wrap, making the necessary information unavailable.
>
>Yes, ctime is omitted.  Please don't tell me you can set ctime.

No: the point I was discussing was setting the mtime on a directory
without files (only directories) to the maximum of ctimes of contained
directories.  So as to accurately reflect when said directory was last
modified (since the LXR only displays mtimes, that's all I want to
set).
-- 
 Matthew Weigel
 Research Systems Programmer
 mcweigel+@cs.cmu.edu

Re: [9fans] Playing with unwrap on Unix

[rob pike, esq.]

> If I remember correctly, the source for this misconception is K&R. 

The first printing of the second edition calls ctime 'the time originally created'.

-rob

Re: [9fans] Playing with unwrap on Unix

[forsyth]

>> affected by the wrap, is *really* difficult.  Looking at 9e.c, it
>> appears that ctime - and hence directory creation time? - is not stored
>> in a wrap, making the necessary information unavailable.

>Yes, ctime is omitted.  Please don't tell me you can set ctime.

of course, Unix ctime isn't creation time either; that term was probably
encouraged by a comment in the old include file.
i think `inode change time' was a more accurate description: internally
the ICHG flag was updated in many places, causing ctime to change
on allocation of new blocks, modes, owner, ...
(i'm not sure why i bothered to point that out.)

Re: [9fans] Playing with unwrap on Unix

[Boyd Roberts]

forsyth@caldo.demon.co.uk wrote:
> of course, Unix ctime isn't creation time either; that term was probably
> encouraged by a comment in the old include file.
> i think `inode change time' was a more accurate description:

Yes, it's 'inode change time' which occurs when the inode data is changed
[chmod/chown/chgrp/...].  It was never the 'creation' time.  Amazing how
many people believe it to be the 'creation' time ...

Re: [9fans] Playing with unwrap on Unix

[Aharon Robbins]

In article <3C9B495A.5AFB8D2A@strakt.com>,
Boyd Roberts <9fans@cse.psu.edu> wrote:
>forsyth@caldo.demon.co.uk wrote:
>> of course, Unix ctime isn't creation time either; that term was probably
>> encouraged by a comment in the old include file.
>> i think `inode change time' was a more accurate description:
>
>Yes, it's 'inode change time' which occurs when the inode data is changed
>[chmod/chown/chgrp/...].  It was never the 'creation' time.  Amazing how
>many people believe it to be the 'creation' time ...

If I remember correctly, the source for this misconception is K&R.  Check out
the description of the stat structure in chapter 8.  In K&R-I and early
printings of K&R-II, the entry for ctime said file creation time.  It is
fixed in my eighth printing of K&R-II, and I don't have my first printing
of K&R-II, or my K&R-I handy to double check, but I'm sure there are others
on this list who do... :-)

(I'm pretty sure I mentioned this to BWK at some point, but whether that is
what caused the fix, I don't know.)

Arnold
-- 
Aharon (Arnold) Robbins --- Pioneer Consulting Ltd.	arnold@skeeve.com
P.O. Box 354		Home Phone: +972  8 979-0381	Fax: +1 928 569 9018
Nof Ayalon		Cell Phone: +972 51  297-545
D.N. Shimshon 99785	ISRAEL

Re: [9fans] Playing with unwrap on Unix

[Russ Cox]

> >There is a remove file included in the wrap
> >metadata directory containing a list of files
> >to be removed.  There are no remove directories though.
> 
> Hmmm. Unwrap seems to think that it is of length zero, which it assumes
> is a directory.  The source to the actual Plan 9 wrap programs is a bit
> more convoluted that 9e.c, so I haven't gotten to it yet.

Many of the remove files are of length zero, but that doesn't
mean they are directories.  An entry is a directory if and only if
$2 (which is octal) has the high bit set.  For example, in:

/wrap 20000000775 sys sys 1007472771 0
/sys/src/cmd/du.c 664 sys sys 1007472771 3297

wrap is a directory, du.c is not.

Wrap archives are in the same format as used by 
/sys/src/cmd/disk/mkext.c, which is a nice short reference.

Re: [9fans] Playing with unwrap on Unix

[Matthew C Weigel]

Russ Cox <9fans@cse.psu.edu> wrote:

>Many of the remove files are of length zero, but that doesn't
>mean they are directories.  An entry is a directory if and only if
>$2 (which is octal) has the high bit set.

Aha!

>Wrap archives are in the same format as used by 
>/sys/src/cmd/disk/mkext.c, which is a nice short reference.

This is exactly what I needed, thanks.
-- 
 Matthew Weigel
 Research Systems Programmer
 mcweigel+@cs.cmu.edu

Re: [9fans] Playing with unwrap on Unix

[Russ Cox]

> That is not what it looks like to me: in the 6/17/00 archfs.9gz, wrap/
> has an mtime of 6/4/00, while everything else has an mtime of 6/17/00.

Ah.  Yes, the partial updates sometimes have varying mtimes inside them.
For the full releases we timestamp all the files with the date of the release.

> I've been doing my testing with the 6/17 archfs.9gz. It contains
> 386/bin/archfs and sys/src/cmd/archfs.c but not 386, 386/bin, sys, and
> so on.  At least according to the unwrap perl script (original).

Sorry, you're right.  I went back and forth on this in the wrap code,
and it looks like I didn't include the directory tree for partial updates.

We have no ctime, so I'm afraid there's not much that can be
done on that front.  

Russ

Re: [9fans] Playing with unwrap on Unix

[Matthew C Weigel]

Russ Cox <9fans@cse.psu.edu> wrote:

>Ah.  Yes, the partial updates sometimes have varying mtimes inside them.
>For the full releases we timestamp all the files with the date of the release.

Hmmm.  Well, I'm planning on mostly using it for full updates, so I'll
leave the brokenness for someone who needs correct behavior for partial
updates (which may well be me, three weeks from now :).

If partial updates contained subdirectories as well, there wouldn't be
any broken behavior to fix.

>We have no ctime, so I'm afraid there's not much that can be
>done on that front.  

If all wraps contained all requisite directories, it wouldn't matter;
as several people have pointed out the ctime for directories is not
constant, so it wouldn't be a fix regardless.  And anyway I wouldn't
expect the wrap format to change for this problem when there is an
easier, more elegant, and backwards-compatible fix.
-- 
 Matthew Weigel
 Research Systems Programmer
 mcweigel+@cs.cmu.edu

Re: [9fans] Playing with unwrap on Unix

[Russ Cox]

> Also, does wrap support removing files?  I thought perhaps that was the
> purpose of the 'remove' directories I see in a few places... that seems
> like needed functionality in a version repository.

There is a remove file included in the wrap
metadata directory containing a list of files
to be removed.  There are no remove directories though.

The last entry isn't bogus.  You're probably misparsing it.
The archive ends with "end of archive\n".

Russ

Re: [9fans] Playing with unwrap on Unix

[Matthew C Weigel]

Russ Cox <9fans@cse.psu.edu> wrote:

>There is a remove file included in the wrap
>metadata directory containing a list of files
>to be removed.  There are no remove directories though.

Hmmm. Unwrap seems to think that it is of length zero, which it assumes
is a directory.  The source to the actual Plan 9 wrap programs is a bit
more convoluted that 9e.c, so I haven't gotten to it yet.

>The last entry isn't bogus.  You're probably misparsing it.
>The archive ends with "end of archive\n".

Heh, that would do some interesting things with the way it's currently
parsed.
-- 
 Matthew Weigel
 Research Systems Programmer
 mcweigel+@cs.cmu.edu

Re: [9fans] Playing with unwrap on Unix

[Russ Cox]

> It *will* still overwrite newer files, and it doesn't seem to set the
> mtime on directories correctly (from one simple test), but I think it
> still qualifies as an improvement.

That's okay; wrap doesn't set the mtime on directories
properly either.  The problem is that the directory gets
created before its children, and creating children changes
the mtime of the directory, so you'd have to set it on
the way out too.

Re: [9fans] Playing with unwrap on Unix

[Matthew C Weigel]

Russ Cox <9fans@cse.psu.edu> wrote:

>That's okay; wrap doesn't set the mtime on directories
>properly either.  The problem is that the directory gets
>created before its children, and creating children changes
>the mtime of the directory, so you'd have to set it on
>the way out too.

For my purposes, I think that setting the mtime on directories
correctly is worth the effort if I can do it. The LXR's main screen is
a list of the source directories and their mtimes, and - at least for
now - that's a much clearer indication of vintage than the version
numbers used. 3.0 is the initial release of 3e, (mtime 6/17), 3.2 is
the 7/29/00 update (mtime 7/31), and 3.3 is the 10/14/00 update (mtime
10/15), but 3.1 is something between 3.0 and 3.2 and your guess is as
good (better, if you look at the code) as mine as to which updates,
exactly, that includes.

There's also the last, bogus, entry, which doesn't show up in the
original unwrap because it only prints $filename (which is empty).

Also, does wrap support removing files?  I thought perhaps that was the
purpose of the 'remove' directories I see in a few places... that seems
like needed functionality in a version repository.

I've got some work to do on this script.
-- 
 Matthew Weigel
 Research Systems Programmer
 mcweigel+@cs.cmu.edu

Re: [9fans] Playing with unwrap on Unix

[Matthew C Weigel]

Russ Cox <9fans@cse.psu.edu> wrote:

>the mtime of the directory, so you'd have to set it on
>the way out too.

The 'joys' of perl - just build a stack of directories as you create
them, and a hash of mtimes.  Then run through that after you're done
creating files.

That works for files whose path is in also in the wrap; right now it
does something a bit wrong for directories that have a directory that
has a file in the wrap, but are not in the wrap nor contain files in
the wrap themselves (it creates the directories if they don't exist,
and sets its mtime to that of the first file in a subdirectory in the
wrap).

Doing the right thing for directory mtimes, when the directory itself
is not in the wrap, is a bit more difficult; ideally any directory
containing files would have an mtime that was the maximum of mtimes of
those contained files (this should already be so for directories which
are themselves in the wrap), but setting the correct mtime on a
directory not in the wrap, that has only directories, that gets
affected by the wrap, is *really* difficult.  Looking at 9e.c, it
appears that ctime - and hence directory creation time? - is not stored
in a wrap, making the necessary information unavailable.

Doing the *really* correct thing, including not destroying newer files,
would take a prepass to establish the mtime of each directory before
actually creating any files.  And of course, all this assumes that you
don't want the mtime to record when you made the update, but when the
files were actually edited (with a WORM, of course, finding the former
should be relatively simple).
-- 
 Matthew Weigel
 Research Systems Programmer
 mcweigel+@cs.cmu.edu

[9fans] Playing with unwrap on Unix

[Matthew C Weigel]

I noticed yesterday that the Plan 9 LXR
(http://offworld.fac.cs.cmu.edu/plan9) is actually being used a little,
despite being out of date, so I decided to try to update it.

In poking at doing so, I ran across the problem that the unwrap Perl
script doesn't set the mtime on the files it writes to the mtime of the
file in the wrap - they're set to the time unwrap is run.  Since I am
coming to this long after Steve Wynne left, the timestamps the LXR
reports would no longer even closely correspond to the actual times the
files were last edited.

Now I've got a slightly better unwrap: it both prints out and sets the
mtime and atime of the file to the mtime for the file according to the
wrap.  I figured it out with a little help from the 9e source.

It *will* still overwrite newer files, and it doesn't seem to set the
mtime on directories correctly (from one simple test), but I think it
still qualifies as an improvement.

I've put it up at http://offworld.fac.cs.cmu.edu/unwrap in case it will
help someone else.
-- 
 Matthew Weigel
 Research Systems Programmer
 mcweigel+@cs.cmu.edu
Followup-To: 
Distribution: 
Organization: University of Bath Computing Services, UK
Keywords: 
Cc: 


-- 
Dennis Davis, BUCS, University of Bath, Bath, BA2 7AY, UK
D.H.Davis@bath.ac.uk