Bug Report: glob qualifier Lm-1 does not work

Bug Report: glob qualifier Lm-1 does not work

[Roy Zuo]

Hi,

Some one just tells me that glob qualifier Lm-1 in zsh does not work, and I
can confirm it is true. Whenever you do 

    ls *(Lm-1)

it complains "no matches found". Lm+1, Lk-1024, Lk-1, Lm-2 all work well. The
only thing that does not work is Lm-1.

Roy

-- 
 ___________________________________________________________ 
< Let's organize this thing and take all the fun out of it. >
 ----------------------------------------------------------- 
       \   ,__,
        \  (oo)____
           (__)    )\
              ||--|| *

Re: Bug Report: glob qualifier Lm-1 does not work

[Jérémie Roquet]

Hi,

2010/9/1 Roy Zuo <roylzuo@gmail.com>:
> Some one just tells me that glob qualifier Lm-1 in zsh does not work, and I
> can confirm it is true. Whenever you do
>
>    ls *(Lm-1)
>
> it complains "no matches found". Lm+1, Lk-1024, Lk-1, Lm-2 all work well. The
> only thing that does not work is Lm-1.

Can't it be because of the minimal file size allowed by your filesystem?

I get :

$ ls *(Lm-1)
zsh: no matches found: *(Lm-1)
$ touch foo
$ ls *(Lm-1)
foo
$ echo -n c >| foo
$ ls *(Lm-1)
zsh: no matches found: *(Lm-1)

-- 
Jérémie

Re: Bug Report: glob qualifier Lm-1 does not work

[Bart Schaefer]

On Sep 1, 11:54pm, Roy Zuo wrote:
}
} Some one just tells me that glob qualifier Lm-1 in zsh does not work, and I
} can confirm it is true. Whenever you do 
} 
}     ls *(Lm-1)
} 
} it complains "no matches found". Lm+1, Lk-1024, Lk-1, Lm-2 all work well. The
} only thing that does not work is Lm-1.

I think something related to this was discussed quite recently ... it's
a question of (a) rounding error and (b) using absolute less/greater
comparisons rather than <= or >=.  The comparison is made by scaling
the size of the file (read from the filesystem) *down* rather than by
multiplying the target units *up*, so any non-zero-sized file is always
treated as at having size at least one, no matter what the scale.  And
L-1 only matches files having size smaller than 1, which means that it
is semantically the same as L0, even with k or m inserted.

The same effect should occur for every other scaling factor, but the
smaller the scaling factor the fewer files are likely to be affected,
so probably you just don't notice.

That comparison has been that way ever since zsh existed; I don't know
why it was decided to do division on the file size rather than to do
multiplication on the units, other than perhaps to avoid integer
overflow when searching for very big files.

Re: Bug Report: glob qualifier Lm-1 does not work

[Roy Zuo]

Hi,

Sorry I made some mistake about Lk-1. It has the same issue as well.

If the problem is because of the minimal file size set by the file system, then
it could not explain why Lm-2 and Lk-2 work. Lm-1 and Lk-1 only capture zero
size files here. My zsh version is 4.3.10 .

Roy

$ ll *(Lm-2)
-rw-r--r-- 1 roylez roylez 4.0K [2010-09-01  8:39] abc
-rw-r--r-- 1 roylez roylez    0 [2010-09-02  0:24] bcd
-rw-r--r-- 1 roylez roylez    2 [2010-09-02  0:28] efg
$ ll *(Lm-1)
-rw-r--r-- 1 roylez roylez 0 [2010-09-02  0:24] bcd
$ ll *(Lk-1)
-rw-r--r-- 1 roylez roylez 0 [2010-09-02  0:24] bcd
$ ll *(Lk-2)
-rw-r--r-- 1 roylez roylez 0 [2010-09-02  0:24] bcd
-rw-r--r-- 1 roylez roylez 2 [2010-09-02  0:28] efg

On Wed, Sep 01, 2010 at 06:16:59PM +0200, Jérémie Roquet wrote:
> Hi,
> 
> 2010/9/1 Roy Zuo <roylzuo@gmail.com>:
> > Some one just tells me that glob qualifier Lm-1 in zsh does not work, and I
> > can confirm it is true. Whenever you do
> >
> >    ls *(Lm-1)
> >
> > it complains "no matches found". Lm+1, Lk-1024, Lk-1, Lm-2 all work well. The
> > only thing that does not work is Lm-1.
> 
> Can't it be because of the minimal file size allowed by your filesystem?
> 
> I get :
> 
> $ ls *(Lm-1)
> zsh: no matches found: *(Lm-1)
> $ touch foo
> $ ls *(Lm-1)
> foo
> $ echo -n c >| foo
> $ ls *(Lm-1)
> zsh: no matches found: *(Lm-1)
> 
> -- 
> Jérémie
> 

-- 
 _____________________________________________________________________ 
/ You are always doing something marginal when the boss drops by your \
\ desk.                                                               /
 --------------------------------------------------------------------- 
       \   ,__,
        \  (oo)____
           (__)    )\
              ||--|| *

Re: Bug Report: glob qualifier Lm-1 does not work

[Peter Stephenson]

On Wed, 1 Sep 2010 23:54:16 +0800
Roy Zuo <roylzuo@gmail.com> wrote:
> Some one just tells me that glob qualifier Lm-1 in zsh does not work,
> and I can confirm it is true. Whenever you do 
> 
>     ls *(Lm-1)
> 
> it complains "no matches found". Lm+1, Lk-1024, Lk-1, Lm-2 all work
> well. The only thing that does not work is Lm-1.

(Basically Bart's answer, but I added a bit more so I'll send it anyway.)

It does do what it's intended to, it just doesn't do what you expect (and
what it's intended to do is quite confusing if you don't know what's behind
it).  Something similar came up recently; as a minimum the doc could do
with updating.  This comment on times is also relevant to sizes:

  Any  fractional  part  of the difference between the access time
  and the current part in the appropriate units is ignored in  the
  comparison.   For  instance,  ‘echo  *(ah-5)’  would  echo files
  accessed within the last five hours, while ‘echo *(ah+5)’  would
  echo  files  accessed  at least six hours ago, as times strictly
  between five and six hours are treated as five hours.

In other words, files "less than 1 megabyte" have to be zero.  You'll see
that it does indeed pick up files of zero size.

The reason for this bizarre behaviour is that the same file size is used
when testing for less than, equal to, or greater than the requested
number.  Hence *(Lm1) picks up all those files you expect to see --- the
files that are 1 megabyte in size, ignoring the fractional part in the
difference.

The logic behind the reason is that this makes *(Lm-1), *(Lm1) and *(Lm+1)
disjunct sets, as you might naively expect (until you worked out the
implications).  In other words, if you sort files into the three sets
you're guaranteed to get all files exactly once.  That's quite a useful
property, although you might have thought it would make more sense to round
file sizes down rather than up.  If it did, I think it would be much more
natural and you could probably guess that files "greater than 1 MB" needed
to be 2 MB or more without having to read the instructions.

It wouldn't be hard to add the other behaviour (rounding up), but I'm not
sure how to control it: an option (on or off by default?), or alternative
syntax (for which unfortunately < and > aren't available which is probably
why it's currently - and +)?

Index: Doc/Zsh/expn.yo
===================================================================
RCS file: /cvsroot/zsh/zsh/Doc/Zsh/expn.yo,v
retrieving revision 1.118
diff -p -u -r1.118 expn.yo
--- Doc/Zsh/expn.yo	22 Aug 2010 20:08:57 -0000	1.118
+++ Doc/Zsh/expn.yo	1 Sep 2010 16:56:18 -0000
@@ -2302,10 +2302,17 @@ time.
 )
 item(tt(L)[tt(PLUS())|tt(-)]var(n))(
 files less than var(n) bytes (tt(-)), more than var(n) bytes (tt(PLUS())), or
-exactly var(n) bytes in length. If this flag is directly followed by a `tt(k)'
-(`tt(K)'), `tt(m)' (`tt(M)'), or `tt(p)' (`tt(P)') (e.g. `tt(Lk-50)')
-the check is performed with kilobytes, megabytes, or blocks (of 512 bytes)
-instead.
+exactly var(n) bytes in length.
+
+If this flag is directly followed by a `tt(k)' (`tt(K)'), `tt(m)'
+(`tt(M)'), or `tt(p)' (`tt(P)') (e.g. `tt(Lk-50)') the check is performed
+with kilobytes, megabytes, or blocks (of 512 bytes) instead.  In this
+case a file is regarded as "exactly" the size if the file size rounded up
+to the next unit is equal to the test size.  Hence `tt(*LPAR()Lm1+RPAR())'
+matches files from 1 byte up to 1 Megabyte inclusive.  Note also that
+the set of files "less than" the test size only includes files that would
+not match the equality test; hence `tt(*LPAR()Lm-1+RPAR())' only matches
+files of zero size.
 )
 item(tt(^))(
 negates all qualifiers following it

-- 
Peter Stephenson <pws@csr.com>            Software Engineer
Tel: +44 (0)1223 692070                   Cambridge Silicon Radio Limited
Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, UK


Member of the CSR plc group of companies. CSR plc registered in England and Wales, registered number 4187346, registered office Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, United Kingdom

Re: Bug Report: glob qualifier Lm-1 does not work

[Bart Schaefer]

On Sep 1,  6:01pm, Peter Stephenson wrote:
} Subject: Re: Bug Report: glob qualifier Lm-1 does not work
}
} In other words, files "less than 1 megabyte" have to be zero.  You'll see
} that it does indeed pick up files of zero size.
} 
} The logic behind the reason is that this makes *(Lm-1), *(Lm1) and
} *(Lm+1) disjunct sets, as you might naively expect (until you worked
} out the implications) [...] although you might have thought it would
} make more sense to round file sizes down rather than up.

I seem to recall that these semantics are intended to produce the same
results as the output of "find" with the same -/+ modifiers.  In my
zsh build tree,

    find . -size 2k -print
    print -l **/*(Lk2)

both ouptut the same 62 files, and with -2 both print the same 219
other files.  If not for that, it might make the most sense to round
to the *nearest* size (or whatever measure) rather than up or down.

E.g. right now if U is a multiplier expressed in bytes for a unit such
as "megabyte" and S is a file size in bytes as read from stat(), zsh
computes

    N = (S + (U - 1)) / U

using long integers so the result is truncated.  Looking for files less
(or greater or equal) X units then compares N to X.  But zsh could do

    N = (S + (U / 2)) / U

and preserve the property of distinct sets while treating files less
than half a megabyte in size as "less than one" megabytes.  However, I'm
not sure that would satisfy anyone either, because users couldd still
be confused as to why three-fourths of a megabyte is not less than one.

As I mentioned before, zsh instead could compare S to (X * U) without
rounding or truncating at all, which is probably what most people are
expecting; but then "files exactly 1 megabyte in size" becomes "too
precise."

Of course the ulitmate question here is, how does one express what
Roy actually intended when he wrote (Lm-1), other than by writing it
out longhand as (L-1048576)?  (And is even that what was intended?)
I think the answer is that you can't.  You must choose a multiplier
based on how precise a comparison you want.