Allow slash in alternation patterns in limited cases?

Allow slash in alternation patterns in limited cases?

[Mikael Magnusson]

You can't do (/foo/bar|/baz)/bong, but would it be possible to allow
it when each alternatee is a full pattern and there's nothing outside
the alternation? To put it another way:
/path/(to/file|or/another/file) # nope, too hard
(/path/to/a/dir/*|/path/to/some/other/files/*) # can we allow this?

The reason one might want this is for example doing things like
echo ($file1|file2)(om[1]) #where $file1 and $file2 are full paths to
files in different directories
echo (${(~j:|:)fpath}/_stat)(om)
etc

I imagine this is at least a little bit more possible than the general
case? The scanner() function is scary, but since I think some of you
have looked at it recently, is your gut feeling that it's doable? If
so, I'll have a try.

If not, would it be possible to invent some new syntax to "paste" two
or more globs together so that a single set of glob
quals/sorts/flags/subscripts could apply to it?

-- 
Mikael Magnusson

Re: Allow slash in alternation patterns in limited cases?

[Bart Schaefer]

On Apr 10, 10:36pm, Mikael Magnusson wrote:
}
} /path/(to/file|or/another/file) # nope, too hard
} (/path/to/a/dir/*|/path/to/some/other/files/*) # can we allow this?

My gut feeling is that this is close to impossible.  In fact your first
example might actually be easier than your second, because there are no
pattern characters within the alternation.  Similarly I think
    (/path/to/a/dir|/path/to/some/other/files)/*
would be easier than the case where the wildcard is inside the parens,
because an opendir() might be forced on each fixed path.

} If not, would it be possible to invent some new syntax to "paste" two
} or more globs together so that a single set of glob
} quals/sorts/flags/subscripts could apply to it?

Syntactically, this would probably work best as a parameter expansion
flag, which accepts a glob qualifier and applies it to every value in
the (array) expansion.  I'm not sure if there are any letters left for
this; we've reserved (_:stuff:) for future use, so possibly something
like e.g. ${(_:#qom[1]:)array} would work.  I insert the #q so that
other stuff in (_:stuff:) might still be adopted in future.

(Also is the above equivalent to ${${(_:#qom:)array}[1]} and if so do
we prohibit the qualifier subscripting in this form?)

Internally, to make this fly, the glob.c:zglob() function must be
factored apart into the bit that parses glob flags, the bit that makes
the call to scanner() [which would get called in a loop over the array
elements], and finally the bit that handles gf_sortlist and makes the
calls to insert_glob_match().

This is probably a significant effort because of the use of C globals
for glob state [although that might be easier since workers/38188].

Of course readability takes a major dive here as we're mating together
our two most cryptic bits of syntax into a single monster.  The only
other idea I have is to use a command form like

    zglob -q 'om[1]' $array

which to substitute-in place like a normal glob would need to be written

    echo $(zglob -q 'om[1]' $array)

which for efficiency would mean we implement the ksh-style non-fork of
builtins appearing in $(...), and now we're down another rabbit hole.

Re: Allow slash in alternation patterns in limited cases?

[Peter Stephenson]

On Sun, 10 Apr 2016 15:11:05 -0700
Bart Schaefer <schaefer@brasslantern.com> wrote:

> On Apr 10, 10:36pm, Mikael Magnusson wrote:
> }
> } /path/(to/file|or/another/file) # nope, too hard
> } (/path/to/a/dir/*|/path/to/some/other/files/*) # can we allow this?
> 
> My gut feeling is that this is close to impossible.

The problem's not the parsing, it's the fact that you don't have
appropriate chunks corresponding to directories for the scanner to loop
over once pattern matching has parsed it.

The only way I can see is effectively to parse it first in the globbing
code to treat a complete set of parentheses specially.  But this is
going to be inconsistent with pattern matching one way or another.  So a
different syntax would be more sensible.

> } If not, would it be possible to invent some new syntax to "paste" two
> } or more globs together so that a single set of glob
> } quals/sorts/flags/subscripts could apply to it?

I think Bart's answered this.

pws

Re: Allow slash in alternation patterns in limited cases?

[Mikael Magnusson]

On Mon, Apr 11, 2016 at 10:37 AM, Peter Stephenson
<p.stephenson@samsung.com> wrote:
> On Sun, 10 Apr 2016 15:11:05 -0700
> Bart Schaefer <schaefer@brasslantern.com> wrote:
>
>> On Apr 10, 10:36pm, Mikael Magnusson wrote:
>> }
>> } /path/(to/file|or/another/file) # nope, too hard
>> } (/path/to/a/dir/*|/path/to/some/other/files/*) # can we allow this?
>>
>> My gut feeling is that this is close to impossible.
>
> The problem's not the parsing, it's the fact that you don't have
> appropriate chunks corresponding to directories for the scanner to loop
> over once pattern matching has parsed it.
>
> The only way I can see is effectively to parse it first in the globbing
> code to treat a complete set of parentheses specially.

Yeah, that's what I had in mind too as the possibility. This is
probably naive but I imagined we would see the ( and go "ah, the
matching ) is at the end [and there's only glob qualifiers after it],
so i'll loop over each alternatee and push them on the pile of glob
results, and then apply the qualifiers".

> But this is
> going to be inconsistent with pattern matching one way or another.  So a
> different syntax would be more sensible.

Doesn't this already work as is with pattern matching? Eg, I could do
/path/to/**/*~^(/path/to/a/dir/*|/path/to/some/other/files/*)
even though that would be insanely inefficient, especially when
/path/to is just /.

>> } If not, would it be possible to invent some new syntax to "paste" two
>> } or more globs together so that a single set of glob
>> } quals/sorts/flags/subscripts could apply to it?
>
> I think Bart's answered this.

This occurred to me just now,
a=( (/path/to/a/dir/*|/path/to/some/other/files/*) )
echo .(e*'reply=($a)'*om[1])

This works syntactically, the only problem seems to be that sort
qualifiers do nothing on matches inserted by e::, nor do ones like .
and / (they all just act on the original . which is probably
reasonable). It's also pretty ugly to have to act on a dummy match.

-- 
Mikael Magnusson

Re: Allow slash in alternation patterns in limited cases?

[Peter Stephenson]

On Mon, 11 Apr 2016 12:22:49 +0200
Mikael Magnusson <mikachu@gmail.com> wrote:
> > But this is
> > going to be inconsistent with pattern matching one way or another.  So a
> > different syntax would be more sensible.
> 
> Doesn't this already work as is with pattern matching? Eg, I could do
> /path/to/**/*~^(/path/to/a/dir/*|/path/to/some/other/files/*)
> even though that would be insanely inefficient, especially when
> /path/to is just /.

"~" is already handled specially: we pass in a flag to say we're at top
level so just keep going if you find a "/" after a "~".  This is much
easier as once we've seen the ~ we can relax --- no more handling of
individual directories is needed as we're going to apply the exclusion
in one go at the end (hence the suggestion of the new option to prune
directories).  This has always been the documented way in this case.

pws

Re: Allow slash in alternation patterns in limited cases?

[Mikael Magnusson]

On Mon, Apr 11, 2016 at 12:29 PM, Peter Stephenson
<p.stephenson@samsung.com> wrote:
> On Mon, 11 Apr 2016 12:22:49 +0200
> Mikael Magnusson <mikachu@gmail.com> wrote:
>> > But this is
>> > going to be inconsistent with pattern matching one way or another.  So a
>> > different syntax would be more sensible.
>>
>> Doesn't this already work as is with pattern matching? Eg, I could do
>> /path/to/**/*~^(/path/to/a/dir/*|/path/to/some/other/files/*)
>> even though that would be insanely inefficient, especially when
>> /path/to is just /.
>
> "~" is already handled specially: we pass in a flag to say we're at top
> level so just keep going if you find a "/" after a "~".  This is much
> easier as once we've seen the ~ we can relax --- no more handling of
> individual directories is needed as we're going to apply the exclusion
> in one go at the end (hence the suggestion of the new option to prune
> directories).  This has always been the documented way in this case.

Maybe I misunderstood your original point. I thought you meant making
(foo/bar|baz/bong) work in a glob would make it more incompatible with
pattern matching, but it already works there. So as far as I can see
if the change was possible, it would bring the two closer together,
not further apart. I only used the ~ example to mean that the part
after the ~ is a pattern match, not a glob, and uses the exact pattern
I wanted in the glob with the same results.

I guess the takeaway from the thread is that it's mostly not possible.

-- 
Mikael Magnusson

Re: Allow slash in alternation patterns in limited cases?

[Peter Stephenson]

On Mon, 11 Apr 2016 12:47:21 +0200
Mikael Magnusson <mikachu@gmail.com> wrote:
> Maybe I misunderstood your original point. I thought you meant making
> (foo/bar|baz/bong) work in a glob would make it more incompatible with
> pattern matching, but it already works there.

The point is that, unlike the ~ case which is just a flag passed in to
the pattern match parser, it would longer done by pattern matching at
all.  It would be done in the glob code.  The pattern match code would
see the foo, bar, baz, bong, and it would it be reassembled higher up.
So it's not so much incompatible as something utterly different.

pws

Re: Allow slash in alternation patterns in limited cases?

[Mikael Magnusson]

On Mon, Apr 11, 2016 at 1:07 PM, Peter Stephenson
<p.stephenson@samsung.com> wrote:
> On Mon, 11 Apr 2016 12:47:21 +0200
> Mikael Magnusson <mikachu@gmail.com> wrote:
>> Maybe I misunderstood your original point. I thought you meant making
>> (foo/bar|baz/bong) work in a glob would make it more incompatible with
>> pattern matching, but it already works there.
>
> The point is that, unlike the ~ case which is just a flag passed in to
> the pattern match parser, it would longer done by pattern matching at
> all.  It would be done in the glob code.  The pattern match code would
> see the foo, bar, baz, bong, and it would it be reassembled higher up.
> So it's not so much incompatible as something utterly different.

This is already the same difference we have between (foo|bar) in
globbing and pattern matching though. If it's a glob, it's handled
recursively by scanner() and if it's pattern, it's somewhere else (I
don't even know where the general pattern matching code is by heart).
Maybe this is just exactly what you're saying, and I read some
objection where there was just a statement about how things are. I
don't see how any additional inconsistensies with the pattern matching
code arises from the end-users's perspective though (see separate
patch). Well, things like (#i) don't carry across from one pattern to
the next... I suppose that's somewhat major.

-- 
Mikael Magnusson

PATCH: Allow / in full pattern alternations

[Mikael Magnusson]

Well, it was a bit easier than I thought.

% print -l (${(~j:|:)${:-$^fpath/rep*}})
/home/mikaelh/.zsh/functions/repoint
/home/mikaelh/.zsh/functions/repointmany
/usr/local/share/zsh/5.2-dev-1-mika/functions/replace-argument
/usr/local/share/zsh/5.2-dev-1-mika/functions/replace-string
/usr/local/share/zsh/5.2-dev-1-mika/functions/replace-string-again

This patch is obviously pretty ugly, and just intended as a proof of
concept / pile of crap, and it will leak stuff / do weird things if
there was an error in a sub-globpattern.

It is also obviously not intended for inclusion, I just wanted to see
what the code might look like and what would be needed. But if someone
is struck by some inspiration about how to implement this in a way that
is less horrible, feel free to say how :).

---
 Src/glob.c | 47 ++++++++++++++++++++++++++++++++++++-----------
 1 file changed, 36 insertions(+), 11 deletions(-)

diff --git a/Src/glob.c b/Src/glob.c
index 7848598..610396d 100644
--- a/Src/glob.c
+++ b/Src/glob.c
@@ -1829,17 +1829,27 @@ zglob(LinkList list, LinkNode np, int nountok)
 	    quals = newquals;
     }
     q = parsepat(str);
+    int specialhack = 0;
     if (!q || errflag) {	/* if parsing failed */
-	restore_globstate(saved);
-	if (unset(BADPATTERN)) {
-	    if (!nountok)
-		untokenize(ostr);
-	    insertlinknode(list, node, ostr);
+	char *par = str;
+	if (*str == zpc_special[ZPC_INPAR] &&
+	    !skipparens(Inpar, Outpar, (char **)&par) &&
+	    !*par && zpc_special[ZPC_BAR] && strchr(str, zpc_special[ZPC_BAR]))
+	{
+	    specialhack = 1;
+	    errflag &= ~ERRFLAG_ERROR;
+	} else {
+	    restore_globstate(saved);
+	    if (unset(BADPATTERN)) {
+		if (!nountok)
+		    untokenize(ostr);
+		insertlinknode(list, node, ostr);
+		return;
+	    }
+	    errflag &= ~ERRFLAG_ERROR;
+	    zerr("bad pattern: %s", ostr);
 	    return;
 	}
-	errflag &= ~ERRFLAG_ERROR;
-	zerr("bad pattern: %s", ostr);
-	return;
     }
     if (!gf_nsorts) {
 	gf_sortlist[0].tp = gf_sorts = (shortcircuit ? GS_NONE : GS_NAME);
@@ -1852,9 +1862,24 @@ zglob(LinkList list, LinkNode np, int nountok)
     matchct = 0;
     pattrystart();
 
-    /* The actual processing takes place here: matches go into  *
-     * matchbuf.  This is the only top-level call to scanner(). */
-    scanner(q, shortcircuit);
+    if (!specialhack) {
+	/* The actual processing takes place here: matches go into  *
+	 * matchbuf.  This is the only top-level call to scanner(). */
+	scanner(q, shortcircuit);
+    } else {
+	str++;
+	while (*str) {
+	    char *next = strchr(str, zpc_special[ZPC_BAR]);
+	    if (!next) next = strchr(str, Outpar);
+	    if (!next) break;
+	    *next = '\0';
+	    q = parsepat(str);
+	    // XXX handle errors magically
+	    scanner(q, shortcircuit);
+	    str = next+1;
+	}
+    }
+
 
     /* Deal with failures to match depending on options */
     if (matchct)
-- 
2.6.1

Re: Allow slash in alternation patterns in limited cases?

[Peter Stephenson]

On Mon, 11 Apr 2016 14:06:10 +0200
Mikael Magnusson <mikachu@gmail.com> wrote:

> On Mon, Apr 11, 2016 at 1:07 PM, Peter Stephenson
> <p.stephenson@samsung.com> wrote:
> > The point is that, unlike the ~ case which is just a flag passed in to
> > the pattern match parser, it would longer done by pattern matching at
> > all.  It would be done in the glob code.  The pattern match code would
> > see the foo, bar, baz, bong, and it would it be reassembled higher up.
> > So it's not so much incompatible as something utterly different.
> 
> This is already the same difference we have between (foo|bar) in
> globbing and pattern matching though. If it's a glob, it's handled
> recursively by scanner() and if it's pattern, it's somewhere else (I
> don't even know where the general pattern matching code is by heart).

No, expressions like (foo|bar) are *only* handled by the pattern
matcher.  The scanner's sole responsibility is to till the pattern
matcher whether or not it should stop if it sees a "/".

pws

Re: Allow slash in alternation patterns in limited cases?

[Mikael Magnusson]

On Mon, Apr 11, 2016 at 2:31 PM, Peter Stephenson
<p.stephenson@samsung.com> wrote:
> On Mon, 11 Apr 2016 14:06:10 +0200
> Mikael Magnusson <mikachu@gmail.com> wrote:
>
>> On Mon, Apr 11, 2016 at 1:07 PM, Peter Stephenson
>> <p.stephenson@samsung.com> wrote:
>> > The point is that, unlike the ~ case which is just a flag passed in to
>> > the pattern match parser, it would longer done by pattern matching at
>> > all.  It would be done in the glob code.  The pattern match code would
>> > see the foo, bar, baz, bong, and it would it be reassembled higher up.
>> > So it's not so much incompatible as something utterly different.
>>
>> This is already the same difference we have between (foo|bar) in
>> globbing and pattern matching though. If it's a glob, it's handled
>> recursively by scanner() and if it's pattern, it's somewhere else (I
>> don't even know where the general pattern matching code is by heart).
>
> No, expressions like (foo|bar) are *only* handled by the pattern
> matcher.  The scanner's sole responsibility is to tell the pattern
> matcher whether or not it should stop if it sees a "/".

Ah, that explains it :). Thanks.

-- 
Mikael Magnusson

Re: PATCH: Allow / in full pattern alternations

[Peter Stephenson]

On Mon, 11 Apr 2016 14:06:14 +0200
Mikael Magnusson <mikachu@gmail.com> wrote:
> % print -l (${(~j:|:)${:-$^fpath/rep*}})
> /home/mikaelh/.zsh/functions/repoint
> /home/mikaelh/.zsh/functions/repointmany
> /usr/local/share/zsh/5.2-dev-1-mika/functions/replace-argument
> /usr/local/share/zsh/5.2-dev-1-mika/functions/replace-string
> /usr/local/share/zsh/5.2-dev-1-mika/functions/replace-string-again

Are you simply trying to do this so you can add the same glob qualifiers
etc.?  What's wrong with

print -l ${^fpath}/rep*

etc.?

pws

Re: Allow slash in alternation patterns in limited cases?

[Peter Stephenson]

On Mon, 11 Apr 2016 15:45:02 +0200
Mikael Magnusson <mikachu@gmail.com> wrote:
> On Mon, Apr 11, 2016 at 2:31 PM, Peter Stephenson
> <p.stephenson@samsung.com> wrote:
> > No, expressions like (foo|bar) are *only* handled by the pattern
> > matcher.  The scanner's sole responsibility is to tell the pattern
> > matcher whether or not it should stop if it sees a "/".
> 
> Ah, that explains it :). Thanks.

To be unhelpfully pedantic... actually, there's one exception, which is
that (.../) is handled specially so you can stick # or ## after it.
That's a special case in parsecomplist(), which looks a little like
what you're trying to do.

pws

Re: PATCH: Allow / in full pattern alternations

[Bart Schaefer]

On Apr 11,  2:06pm, Mikael Magnusson wrote:
}
} Well, it was a bit easier than I thought.

What you've done is pretty close to what I was expecting/explaining
with respect to adding it as a parameter flag, except without actually
splitting up zglob() or looping over a parameter value.

This sort of thing would actually be easier in the ksh model where a
prefix character tells you what to expect from the parenthesized group
that follows.  Maybe we can do the same with an extendedglob marker
similar to (#m) [yes, I know (#m) itself doesn't work in globbing].

Re: PATCH: Allow / in full pattern alternations

[Bart Schaefer]

On Apr 11,  2:47pm, Peter Stephenson wrote:
}
} Are you simply trying to do this so you can add the same glob qualifiers
} etc.?  What's wrong with
} 
} print -l ${^fpath}/rep*

For qualifiers that sort or act as subscript ranges, Mikael wants the sort
and range to apply to the entire set of matches.  Using brace expansion
applies creates a list of otherwise independent globs.

Re: Allow slash in alternation patterns in limited cases?

[Bart Schaefer]

On Apr 11, 12:22pm, Mikael Magnusson wrote:
} Subject: Re: Allow slash in alternation patterns in limited cases?
}
} On Mon, Apr 11, 2016 at 10:37 AM, Peter Stephenson
} <p.stephenson@samsung.com> wrote:
} >
} > The only way I can see is effectively to parse it first in the globbing
} > code to treat a complete set of parentheses specially.
} 
} Doesn't this already work as is with pattern matching?

Yes, but so does somestuff(one/thing|another/things)morestuff.  So it's
inconsistent with pattern matching in that a set of parens enclosing the
entire pattern now takes on special semantics.