Re: [9fans] du vs. ls: duplication or not?

Re: [9fans] du vs. ls: duplication or not?

[arnold]

Hi.

> What I meant was the size of the file is already given via ls(1). So
> a recursive output that make sense and fit a manipulation via join(1)
> (to combine a srv/qid) and sort and uniq etc. could do the trick.

Not quite. On Unix (don't remember 'bout Plan 9 but I assume it's the
same) files can have holes.  All you have to do is lseek pass the end of
the file and then write something.  Bingo - size (or better, length of
the byte stream) no longer has a direct relationship to the number of
disk blocks occupied.

On Unix systems, du looks in the stat structure at the count of
blocks occupied (st_blocks IIRC) and does the computation from that.
Similaly, many Unix du implementations will track hard links based
on device+inode to only count a file's blocks once.

> Since ls(1) gives the size of the file; since du(1) can not really or at
> least not always in an arbitrary context tells the "real" occupation of
> disk size, is not ls(1) enough?

For the two above reasons, I think not.  Again, at least on Unix. :-)

(I suppose ls could print device+inode, and sort+join on that to remove
duplicates, but the holes problems is still there.  Hmmm. I bet ls can
print the number of blocks instead of or in addition to the size. So
maybe I'm wrong after all. :-)

Arnold

Re: [9fans] du vs. ls: duplication or not?

[tlaronde]

On Sun, Jan 15, 2012 at 11:39:13PM -0800, arnold@skeeve.com wrote:
> Hi.
>
> > What I meant was the size of the file is already given via ls(1). So
> > a recursive output that make sense and fit a manipulation via join(1)
> > (to combine a srv/qid) and sort and uniq etc. could do the trick.
>
> Not quite. On Unix (don't remember 'bout Plan 9 but I assume it's the
> same) files can have holes.  All you have to do is lseek pass the end of
> the file and then write something.  Bingo - size (or better, length of
> the byte stream) no longer has a direct relationship to the number of
> disk blocks occupied.

That's indeed a reason, showing that ls(1) is more on the abstract level
(whatever the implementation, ls(1) shows ownership and permissions
---that can have no real relationship in the actual store) while du(1)
tries to show effective story and needs to know more about effective
storage.

And it can work only with some filesystems. (Think ftpfs etc. and think
sharing of blocks)

Unix du(1) seems more accurate simply because it is used in a more
limited context (of filesystems).

So the question remains for me: whether use the fscons, or ls(1) if no
fscons(1). But does a du(1) have to exist?
--
        Thierry Laronde <tlaronde +AT+ polynum +dot+ com>
                      http://www.kergis.com/
Key fingerprint = 0FF7 E906 FBAF FE95 FD89  250D 52B1 AE95 6006 F40C

Re: [9fans] du vs. ls: duplication or not?

[erik quanstrom]

> That's indeed a reason, showing that ls(1) is more on the abstract level
> (whatever the implementation, ls(1) shows ownership and permissions
> ---that can have no real relationship in the actual store) while du(1)
> tries to show effective story and needs to know more about effective
> storage.
>
> And it can work only with some filesystems.  (Think ftpfs etc. and think
> sharing of blocks)

du works the same with all file servers.

i think the problem here is applying semantics to the output.  du
performance a specified operation (adding up the d->size in the
stat output.)  as is plan 9 tradition, what that /means/ is up to the
file server you're talking to.

- erik

Re: [9fans] du vs. ls: duplication or not?

[erik quanstrom]

On Mon Jan 16 02:40:52 EST 2012, arnold@skeeve.com wrote:
> Hi.
>
> > What I meant was the size of the file is already given via ls(1). So
> > a recursive output that make sense and fit a manipulation via join(1)
> > (to combine a srv/qid) and sort and uniq etc. could do the trick.
>
> Not quite. On Unix (don't remember 'bout Plan 9 but I assume it's the
> same) files can have holes.  All you have to do is lseek pass the end of
> the file and then write something.  Bingo - size (or better, length of
> the byte stream) no longer has a direct relationship to the number of
> disk blocks occupied.

a plan 9 file server have holes as an implementation detail, but they
wouldn't exist at the protocol level, and therefore d->size is still good enough.

you can't seek to the end of a pipe on plan 9.

- erik

Re: [9fans] du vs. ls: duplication or not?

[erik quanstrom]

On Mon Jan 16 12:46:42 EST 2012, tlaronde@polynum.com wrote:
> On Mon, Jan 16, 2012 at 11:13:49AM -0500, Joel C. Salomon wrote:
> > 
> > My guess would be that this discussion illustrates exactly why: ls(1) is
> > a gadget that pretty-prints the directory entry.  Extending it with '-R'
> >  would require it to learn about possibly-circular mount points; yuck.
> > On the other hand, du(1) has this sort of feature as its raison d'être.
> 
> And as my blunders have shown, du(1) tries but does not succeed.

you got the correct answer to the "wrong" question.  that's not
du's fault.  and changing the name of the program to "ls" will not
change the behavior.

- erik

Re: [9fans] du vs. ls: duplication or not?

[tlaronde]

On Mon, Jan 16, 2012 at 11:13:49AM -0500, Joel C. Salomon wrote:
> 
> My guess would be that this discussion illustrates exactly why: ls(1) is
> a gadget that pretty-prints the directory entry.  Extending it with '-R'
>  would require it to learn about possibly-circular mount points; yuck.
> On the other hand, du(1) has this sort of feature as its raison d'être.

And as my blunders have shown, du(1) tries but does not succeed.

And Disk Usage has only a meaning on a file server for the file locally
served. 

But the thread will be going nowhere, since, if I'm convinced that du(1)
is special, I'm not convinced it makes really sense as a "general" tool
in a Plan9 environment.

But let this thread dies. I have some job to do on fdisk.
-- 
        Thierry Laronde <tlaronde +AT+ polynum +dot+ com>
                      http://www.kergis.com/
Key fingerprint = 0FF7 E906 FBAF FE95 FD89  250D 52B1 AE95 6006 F40C

Re: [9fans] du vs. ls: duplication or not?

[Joel C. Salomon]

On 01/16/2012 06:46 AM, tlaronde@polynum.com wrote:
> It seems what I'm trying to say is not clear. I know that shipping Plan9
> has no '-R'. What I mean is, since find(1) and others are not here
> because they are duplicating other utils, and can be recreated with
> other "primitives", why du(1) was kept and not simply ls(1) extended
> with a '-R'? Since ls(1) already displays the size of a file (in bytes).

My guess would be that this discussion illustrates exactly why: ls(1) is
a gadget that pretty-prints the directory entry.  Extending it with '-R'
 would require it to learn about possibly-circular mount points; yuck.
On the other hand, du(1) has this sort of feature as its raison d'être.

--Joel

Re: [9fans] du vs. ls: duplication or not?

[tlaronde]

On Sun, Jan 15, 2012 at 04:17:43PM -0500, erik quanstrom wrote:
> > Since ls(1) gives the size of the file; since du(1) can not really or at
> > least not always in an arbitrary context tells the "real" occupation of
> > disk size, is not ls(1) enough?
>
> plan 9 ls does not have a -R option.

It seems what I'm trying to say is not clear. I know that shipping Plan9
has no '-R'. What I mean is, since find(1) and others are not here
because they are duplicating other utils, and can be recreated with
other "primitives", why du(1) was kept and not simply ls(1) extended
with a '-R'? Since ls(1) already displays the size of a file (in bytes).

--
        Thierry Laronde <tlaronde +AT+ polynum +dot+ com>
                      http://www.kergis.com/
Key fingerprint = 0FF7 E906 FBAF FE95 FD89  250D 52B1 AE95 6006 F40C

Re: [9fans] du vs. ls: duplication or not?

[erik quanstrom]

> Since ls(1) gives the size of the file; since du(1) can not really or at
> least not always in an arbitrary context tells the "real" occupation of
> disk size, is not ls(1) enough?

plan 9 ls does not have a -R option.

- erik

Re: [9fans] du vs. ls: duplication or not?

[tlaronde]

On Sat, Jan 14, 2012 at 09:20:51AM +0000, Charles Forsyth wrote:
>[...] On Linux, I
> never use ls -R, partly because I'm
> running p9p's ls, but mainly because the default format of /bin/ls -R is
> amazingly useless (even worse than I remembered).

I wasn't refering to whatever implementation (I don't use ls -R either
since the output is simply not parsable). I was simply using -R, since
-r is already taken.

What I meant was the size of the file is already given via ls(1). So
a recursive output that make sense and fit a manipulation via join(1)
(to combine a srv/qid) and sort and uniq etc. could do the trick.

Since ls(1) gives the size of the file; since du(1) can not really or at
least not always in an arbitrary context tells the "real" occupation of
disk size, is not ls(1) enough?
--
        Thierry Laronde <tlaronde +AT+ polynum +dot+ com>
                      http://www.kergis.com/
Key fingerprint = 0FF7 E906 FBAF FE95 FD89  250D 52B1 AE95 6006 F40C

Re: [9fans] du vs. ls: duplication or not?

[erik quanstrom]

> du(1) stands for Disk Usage. But does this, with the "historical" use,
> has any real sense now, with a namespace that can be, not only built by
> presenting a same file via different paths, but a combination of local
> storage (including memory one...) and remote storage?

i agree with charles.  while one may need to take into account what
you're du'ing, du works as advertized.  -s adds up all the directly-used
storage under the given path.  now if one points it at something that
might include things one is not interested in, i would say the way to
correct that is by changing the path handed to du.

i think another way to state what you're getting at might be, why
doesn't du have some ai to figure out the exact on-disk footprint
under some path?  clearly du can't do that since it's a 9p program,
and won't know a thing about the file server(s) it's talking to.

sort of reminds me of the holy grail, there's no way for du to be
the 1st soldier:

1st soldier:	Who goes there?
King Arthur:	It is I, Arthur, son of Uther Pendragon, from the castle of Camelot.
		King of the Britons, defeater of the Saxons, Sovereign of all England!
1st soldier: 	Pull the other one!
King Arthur: 	I am, and this is my trusty servant Patsy. We have ridden the length and breadth
		of the land in search of knights who will join me in my court at Camelot.
		I must speak with your lord and master.
1st soldier: 	What? Ridden on a horse?
King Arthur: 	Yes!
1st soldier: 	You're using coconuts!

- erik

Re: [9fans] du vs. ls: duplication or not?

[David du Colombier]

Russ Cox did a recursive ls few years ago.

http://swtch.com/lsr.c

I use a slightly modified version which can display
most Dir attributes.

--
David du Colombier

Re: [9fans] du vs. ls: duplication or not?

[Charles Forsyth]

[-- Attachment #1: Type: text/plain, Size: 976 bytes --]

Du answers the question: (roughly) how big are these files (the files in
these directories)?
It still does that reasonably well in Plan 9 (I'm not ruling out possible
improvements, but "it works for me!").
It doesn't answer questions about physical storage. On my Linux machines, I
do use it as a guide to
"which of my or the system's directories is exhausting the space on my
SSD", with du -s * | sort +0nr
[in rc on Linux: with sh you'd need to account for the stupid dotfile
convention]. On Plan 9, I more often
use it to see roughly how much data I'm going to move to a remote machine,
or whether I've left temporary
objects and executables, or whether to tar | gzip something. I sometimes
use du -a to list the names in a
hierarchy, but then I do the same on Linux (or I use find). On Linux, I
never use ls -R, partly because I'm
running p9p's ls, but mainly because the default format of /bin/ls -R is
amazingly useless (even worse than I remembered).

[-- Attachment #2: Type: text/html, Size: 1125 bytes --]

[9fans] du vs. ls: duplication or not?

[tlaronde]

This is a slightly different point of view from the thread my blunders
have started.

du(1) stands for Disk Usage. But does this, with the "historical" use,
has any real sense now, with a namespace that can be, not only built by
presenting a same file via different paths, but a combination of local
storage (including memory one...) and remote storage?

Furthermore, even for an Unix du(1), if the underlying filesystem shares
common blocks, the count will be wrong. So Unix du(1) seems more
accurate only when it is used on "old" technology.

So if du(1) should stand for Disk Usage, it should tell what disk place
would be needed to _store_ "locally" the files. And this depends
on a lot of things: deduplication of blocks? If one uses a FAT, it
will need the total amount displayed etc.

So, in fact, since there is no real and immediate answer, and keeping in
mind the principle: don't duplicate fonctionnality, wouldn't it be
better to have a 'ls -R' (récursive listing) and no du(1) at all on
Plan9?

This is not a proposal or whatever. But, say, a semantical and
engineering question. And BTW, this is not a critic of what has been and
is done on Plan9. (I finally consider that du(1) does what it promises
in the man page; the "lack" is just a one line caveat that du(1) can not
answer a fuzzy question.)
-- 
        Thierry Laronde <tlaronde +AT+ polynum +dot+ com>
                      http://www.kergis.com/
Key fingerprint = 0FF7 E906 FBAF FE95 FD89  250D 52B1 AE95 6006 F40C