Re: Inconsistent behavior with comparisons and recursive glob patterns

Re: Inconsistent behavior with comparisons and recursive glob patterns

[Bart Schaefer]

[-- Attachment #1: Type: text/plain, Size: 569 bytes --]

(Moved from -users)

On Tue, Apr 30, 2024 at 11:15 AM Bart Schaefer
<schaefer@brasslantern.com> wrote:
>
> I suspect there's no explicit reasoning in zmv.  **/ is specifically
> handled in the case of zmv -w/-W (workers/27247) but not for '$f'
> placeholders.
>
> It works if you do this:
>
> zmv -n '(**/)f?' '$f.txt'
>
> Whether it should also work without the parens and also without the -w
> option is unclear.

Attached patch makes it work without the parens.  Is this within spec?
 It's hard to tell from the solitary '$f' example in the doc.

[-- Attachment #2: zmv-recur.txt --]
[-- Type: text/plain, Size: 871 bytes --]

diff --git a/Functions/Misc/zmv b/Functions/Misc/zmv
index 5c03e9ea1..2002af5a6 100644
--- a/Functions/Misc/zmv
+++ b/Functions/Misc/zmv
@@ -249,13 +249,13 @@ errs=()
 (( ${#files} )) || errs=( "no files matched \`$fpat'" )
 
 for f in $files; do
-  if [[ $pat = (#b)(*)\(\*\*##/\)(*) ]]; then
+  if [[ $pat = (#b)(*)(\(\*\*##/\)|\*\*##/)(*) ]]; then
     # This looks like a recursive glob.  This isn't good enough,
-    # because we should really enforce that $match[1] and $match[2]
+    # because we should really enforce that $match[1] and $match[3]
     # don't match slashes unless they were explicitly given.  But
     # it's a start.  It's fine for the classic case where (**/) is
     # at the start of the pattern.
-    pat="$match[1](*/|)$match[2]"
+    pat="$match[1](*/|)$match[3]"
   fi
   [[ -e $f && $f = (#b)${~pat} ]] || continue
   set -- "$match[@]"

Re: Inconsistent behavior with comparisons and recursive glob patterns

[Alan Wagner-Krankel]

Thanks - I did miss that **/ wasn't supported in conditional expressions.

I wonder if there might be some value in making that explicit in the
documentation, possibly by changing the paragraph you referenced to
something like this:

    Pattern metacharacters are active for the pattern arguments. The patterns
    are the same as those used for filename generation, see zshexpn(1), but
    there is no special behaviour of `/' nor initial dots, shorthand operators
    such as **/ are not converted to their expanded forms, and glob qualifiers
    are only allowed for forcing filenames to be generated as described above.

There's a slight issue that "such as **/" implies there are many
shorthand operators; I have no idea if there are any others besides
**/ and ***/.

Thanks,
Awk

On Tue, Apr 30, 2024 at 9:11 AM Eric Cook <llua@gmx.com> wrote:
>
> On 4/30/24 04:14, Alan Wagner-Krankel wrote:
> > The results of these conditional expressions using recursive glob
> > operators were unexpectedly different:
> >
> >> [[ f0 = **/f? ]] && print true || print false
> > false
> >> setopt extendedglob
> >> [[ f0 = (*/)#f? ]] && print true || print false
> > true
>
> A couple things about [[:
>
> ```
> Pattern metacharacters are active for the pattern arguments; the patterns  are
> the  same  as those used for filename generation, see zshexpn(1), but there is
> no special behaviour of ‘/' nor initial dots, and no glob qualifiers  are  al‐
> lowed.
> ```
>
> That last bit about glob qualifiers is out of date since you can now use them to
> force filename generation on in [[. but the point is you aren't doing filename
> generation in [[ (by default), which is where **/ is shorthand for (*/)#.
> pattern matching/globbing is tightly coupled to filename generation in unix shells
> but there are difference.
>
> Even when you use a glob qualifier to force filename generation to happen, you
> still wasn't going to get the behavior you desired since the pattern would've
> underwent filename generation before the comparison that [['s =/== does.
>
> % (set -x; [[ f0 = **/f?(#qN) ]]; touch f2; [[ f0 = **/f?(#qN) ]])
> +zsh:3> [[ f0 =  ]]
> +zsh:3> touch f2
> +zsh:3> [[ f0 = f2 ]]
>
> >
> > Since **/ is a shorthand version of (*/)#, it seems like they both
> > should have returned 'true'. The comparisons behave the same when
> > there is at least one directory in the path:
> >
> >> [[ d1/f1 = **/f? ]] && print true || print false
> > true
> >> setopt extendedglob
> >> [[ d1/f1 = (*/)#f? ]] && print true || print false
> > true
> >
> > I came across this because it affects the behavior of zmv, via a test
> > that filters filenames (line 254 in zsh5.9). An example:
> >
> >> mkdir d1 d1/d2
> >> touch f0 d1/f1 d1/d2/f2
> >> print -rl -- **/f?
> > d1/d2/f2
> > d1/f1
> > f0
> >> autoload zmv
> >> zmv -n '**/f?' '$f.txt'
> > mv -- d1/d2/f2 d1/d2/f2.txt
> > mv -- d1/f1 d1/f1.txt
> >> zmv -n '(*/)#f?' '$f.txt'
> > mv -- d1/d2/f2 d1/d2/f2.txt
> > mv -- d1/f1 d1/f1.txt
> > mv -- f0 f0.txt
> >
> > I think both zmv calls should have attempted to update the 'f0' file
> > in the base directory. Am I missing something?
> >
> > Thanks,
> > Awk
> >
> I'll leave zmv's implementation for someone else to answer but i suspect the reasoning is similar.
>

Re: Inconsistent behavior with comparisons and recursive glob patterns

[Peter Stephenson]

> On 01/05/2024 08:55 BST Alan Wagner-Krankel <awk@awkwork.com> wrote:
> Thanks - I did miss that **/ wasn't supported in conditional expressions.
> 
> I wonder if there might be some value in making that explicit in the
> documentation, possibly by changing the paragraph you referenced to
> something like this:
> 
>     Pattern metacharacters are active for the pattern arguments. The patterns
>     are the same as those used for filename generation, see zshexpn(1), but
>     there is no special behaviour of `/' nor initial dots, shorthand operators
>     such as **/ are not converted to their expanded forms, and glob qualifiers
>     are only allowed for forcing filenames to be generated as described above.
> 
> There's a slight issue that "such as **/" implies there are many
> shorthand operators; I have no idea if there are any others besides
> **/ and ***/.

That's a sensible suggestion, how about this?  Please do comment further.

pws

diff --git a/Doc/Zsh/cond.yo b/Doc/Zsh/cond.yo
index 000e576..4216f89 100644
--- a/Doc/Zsh/cond.yo
+++ b/Doc/Zsh/cond.yo
@@ -241,7 +241,10 @@ ifnzman(\
 noderef(Filename Generation)\
 )\
 , but there is no special behaviour
-of `tt(/)' nor initial dots, and no glob qualifiers are allowed.
+of `tt(/)' nor initial dot, and the patterns `tt(**/)' and `tt(***/)' behave
+the same as (*/), in which the tt(*) may match one or more `tt(/)' characters.
+Also, no bare glob qualifiers are allowed, though the form `(#q)var(...)' is
+allowed as shown above.
 
 In each of the above expressions, if
 var(file) is of the form `tt(/dev/fd/)var(n)',

Re: Inconsistent behavior with comparisons and recursive glob patterns

[Alan Wagner-Krankel]

I dug through StackOverflow a bit to see if I could find how folks in
the wild were expecting zmv to work. Interestingly, most of the posts
that called zmv with `**/` and '$f' were by the same author. In one if
them (https://unix.stackexchange.com/a/271816), they explicitly stated
that "zmv '(**/)(*)' $1$2..." and "zmv '**/*' '$f...'" were
equivalent.

In the absence of a specific statement in the documentation about
'**/' and '$f', it seems safe to assume that the intended spec was
that '**/' would behave the same with the various zmv flavors as it
does in other situations.

Thanks,
Awk


On Tue, Apr 30, 2024 at 3:09 PM Bart Schaefer <schaefer@brasslantern.com> wrote:
>
> (Moved from -users)
>
> On Tue, Apr 30, 2024 at 11:15 AM Bart Schaefer
> <schaefer@brasslantern.com> wrote:
> >
> > I suspect there's no explicit reasoning in zmv.  **/ is specifically
> > handled in the case of zmv -w/-W (workers/27247) but not for '$f'
> > placeholders.
> >
> > It works if you do this:
> >
> > zmv -n '(**/)f?' '$f.txt'
> >
> > Whether it should also work without the parens and also without the -w
> > option is unclear.
>
> Attached patch makes it work without the parens.  Is this within spec?
>  It's hard to tell from the solitary '$f' example in the doc.

Re: Inconsistent behavior with comparisons and recursive glob patterns

[Alan Wagner-Krankel]

I think there's some risk that "... '*' may match one or more '/'
characters" could be misinterpreted as "... '*' may *only* match one
or more '/' characters".

Maybe:
the patterns `tt(**/)' and `tt(***/)' behave the same as `tt(*/)',
i.e. as a standard wildcard tt(*) that could match any set of zero or
more characters (including `tt(/)'), followed by a `tt(/)'.



Thanks,
Awk

On Wed, May 1, 2024 at 3:38 AM Peter Stephenson
<p.w.stephenson@ntlworld.com> wrote:
>
> > On 01/05/2024 08:55 BST Alan Wagner-Krankel <awk@awkwork.com> wrote:
> > Thanks - I did miss that **/ wasn't supported in conditional expressions.
> >
> > I wonder if there might be some value in making that explicit in the
> > documentation, possibly by changing the paragraph you referenced to
> > something like this:
> >
> >     Pattern metacharacters are active for the pattern arguments. The patterns
> >     are the same as those used for filename generation, see zshexpn(1), but
> >     there is no special behaviour of `/' nor initial dots, shorthand operators
> >     such as **/ are not converted to their expanded forms, and glob qualifiers
> >     are only allowed for forcing filenames to be generated as described above.
> >
> > There's a slight issue that "such as **/" implies there are many
> > shorthand operators; I have no idea if there are any others besides
> > **/ and ***/.
>
> That's a sensible suggestion, how about this?  Please do comment further.
>
> pws
>
> diff --git a/Doc/Zsh/cond.yo b/Doc/Zsh/cond.yo
> index 000e576..4216f89 100644
> --- a/Doc/Zsh/cond.yo
> +++ b/Doc/Zsh/cond.yo
> @@ -241,7 +241,10 @@ ifnzman(\
>  noderef(Filename Generation)\
>  )\
>  , but there is no special behaviour
> -of `tt(/)' nor initial dots, and no glob qualifiers are allowed.
> +of `tt(/)' nor initial dot, and the patterns `tt(**/)' and `tt(***/)' behave
> +the same as (*/), in which the tt(*) may match one or more `tt(/)' characters.
> +Also, no bare glob qualifiers are allowed, though the form `(#q)var(...)' is
> +allowed as shown above.
>
>  In each of the above expressions, if
>  var(file) is of the form `tt(/dev/fd/)var(n)',

Re: Inconsistent behavior with comparisons and recursive glob patterns

[Peter Stephenson]

> On 05/05/2024 07:33 BST Alan Wagner-Krankel <awk@awkwork.com> wrote:
> I think there's some risk that "... '*' may match one or more '/'
> characters" could be misinterpreted as "... '*' may *only* match one
> or more '/' characters".
> 
> Maybe:
> the patterns `tt(**/)' and `tt(***/)' behave the same as `tt(*/)',
> i.e. as a standard wildcard tt(*) that could match any set of zero or
> more characters (including `tt(/)'), followed by a `tt(/)'.

Hmm, I'd perhaps limit that to only the information we wish to add to
keep it simple...

pws

diff --git a/Doc/Zsh/cond.yo b/Doc/Zsh/cond.yo
index 000e576..c58aea6 100644
--- a/Doc/Zsh/cond.yo
+++ b/Doc/Zsh/cond.yo
@@ -241,7 +241,11 @@ ifnzman(\
 noderef(Filename Generation)\
 )\
 , but there is no special behaviour
-of `tt(/)' nor initial dots, and no glob qualifiers are allowed.
+of `tt(/)' nor initial dot, and the patterns `tt(**/)' and `tt(***/)' behave
+the same as `tt(*/)', in which the `tt(*)' has its standard behaviour
+but may also match further `tt(/)' characters.  Also, no bare glob
+qualifiers are allowed, though the form `((#q)var(...))' is allowed as
+shown above.
 
 In each of the above expressions, if
 var(file) is of the form `tt(/dev/fd/)var(n)',

Re: Inconsistent behavior with comparisons and recursive glob patterns

[Alan Wagner-Krankel]

Excellent. It's succinct and conveys the necessary information.

Thanks,
Awk

On Tue, May 7, 2024 at 10:08 AM Peter Stephenson
<p.w.stephenson@ntlworld.com> wrote:
>
> > On 05/05/2024 07:33 BST Alan Wagner-Krankel <awk@awkwork.com> wrote:
> > I think there's some risk that "... '*' may match one or more '/'
> > characters" could be misinterpreted as "... '*' may *only* match one
> > or more '/' characters".
> >
> > Maybe:
> > the patterns `tt(**/)' and `tt(***/)' behave the same as `tt(*/)',
> > i.e. as a standard wildcard tt(*) that could match any set of zero or
> > more characters (including `tt(/)'), followed by a `tt(/)'.
>
> Hmm, I'd perhaps limit that to only the information we wish to add to
> keep it simple...
>
> pws
>
> diff --git a/Doc/Zsh/cond.yo b/Doc/Zsh/cond.yo
> index 000e576..c58aea6 100644
> --- a/Doc/Zsh/cond.yo
> +++ b/Doc/Zsh/cond.yo
> @@ -241,7 +241,11 @@ ifnzman(\
>  noderef(Filename Generation)\
>  )\
>  , but there is no special behaviour
> -of `tt(/)' nor initial dots, and no glob qualifiers are allowed.
> +of `tt(/)' nor initial dot, and the patterns `tt(**/)' and `tt(***/)' behave
> +the same as `tt(*/)', in which the `tt(*)' has its standard behaviour
> +but may also match further `tt(/)' characters.  Also, no bare glob
> +qualifiers are allowed, though the form `((#q)var(...))' is allowed as
> +shown above.
>
>  In each of the above expressions, if
>  var(file) is of the form `tt(/dev/fd/)var(n)',