REMATCH_PCRE with zsh built without pcre support

REMATCH_PCRE with zsh built without pcre support

[Eric Cook]

Should setting this option be allowed when zsh was built without pcre support?

A user with this option enabled noticed this when using an completer that uses =~ but
his operating system built zsh without pcre support.

One could currently test if $^module_path/zsh/pcre.so(N) exist before trying to
enable it, but i don't see the use of the option if zsh can't actually use pcre.

Re: REMATCH_PCRE with zsh built without pcre support

[Bart Schaefer]

On Apr 1,  3:09pm, Eric Cook wrote:
}
} Should setting this option be allowed when zsh was built without pcre
} support?

That might make sense, but it would mean that the option-setting code
has to know about modules and add a special case for this one.  Presently
the only existing module-specific hack that deep in the core shell is the
bit in conditionals that implements the =~ operator.

I'd personally be more inclined to do away with REMATCH_PCRE and that
=~ special-case entirely, and require explicit use of the -pcre-match
operator (or a shorter equivalent) when you want those semantics.

It's already been pointed out in another thread that =~ doesn't work at
all if the shell is built with the default modules and without dynamic
linking support.

Re: REMATCH_PCRE with zsh built without pcre support

[Phil Pennock]

On 2017-04-01 at 14:53 -0700, Bart Schaefer wrote:
> I'd personally be more inclined to do away with REMATCH_PCRE and that
> =~ special-case entirely, and require explicit use of the -pcre-match
> operator (or a shorter equivalent) when you want those semantics.

That's a shame.  Perl introduced the syntax and I originally wrote the
Zsh =~ code to use the existing zsh/pcre module.  It was only
reluctantly that I added the zsh/regex module so that =~ could match
Bash's behavior by default, because folks felt that compatibility with
Bash was more important than being powerful.  Almost everything I do
with =~ in Zsh uses REMATCH_PCRE.

> It's already been pointed out in another thread that =~ doesn't work at
> all if the shell is built with the default modules and without dynamic
> linking support.

I'll have to hunt down that thread and see what I can do about that.

-Phil

Re: REMATCH_PCRE with zsh built without pcre support

[Bart Schaefer]

On Sun, Apr 2, 2017 at 6:12 PM, Phil Pennock
<zsh-workers+phil.pennock@spodhuis.org> wrote:
> On 2017-04-01 at 14:53 -0700, Bart Schaefer wrote:
>> I'd personally be more inclined to do away with REMATCH_PCRE and that
>> =~ special-case entirely, and require explicit use of the -pcre-match
>> operator (or a shorter equivalent) when you want those semantics.
>
> That's a shame.

To clarify, I'm not advocating any such demotion of PCRE.  However,
*IF* the presence of the option when the module is not available is
really a serious problem, then I think the right thing is to remove
the option, rather than to expand the special case.

>> It's already been pointed out in another thread that =~ doesn't work at
>> all if the shell is built with the default modules and without dynamic
>> linking support.
>
> I'll have to hunt down that thread and see what I can do about that.

It's the recent "zregexparse" thread where this came up.  I'm not sure
there is anything to be done about it; the zsh/regex module would have
to become part of the default static build, which might have licensing
consequences.

Re: REMATCH_PCRE with zsh built without pcre support

[Daniel Shahaf]

Bart Schaefer wrote on Mon, Apr 03, 2017 at 00:13:00 -0700:
> Phil Pennock wrote:
> > On 2017-04-01 at 14:53 -0700, Bart Schaefer wrote:
> >> I'd personally be more inclined to do away with REMATCH_PCRE and that
> >> =~ special-case entirely, and require explicit use of the -pcre-match
> >> operator (or a shorter equivalent) when you want those semantics.
> >
> > That's a shame.
> 
> To clarify, I'm not advocating any such demotion of PCRE.  However,
> *IF* the presence of the option when the module is not available is
> really a serious problem, then I think the right thing is to remove
> the option, rather than to expand the special case.

I agree that semantics of operators shouldn't depend on option, but I'm
concerned about an upgrade path for people who build with pcre and set
REMATCH_PCRE.

> > I'll have to hunt down that thread and see what I can do about that.
> 
> It's the recent "zregexparse" thread where this came up.  I'm not sure
> there is anything to be done about it; the zsh/regex module would have
> to become part of the default static build, which might have licensing
> consequences.

What would those be?  zsh/regex is BSD licensed.

>

While at it, the docs of REMATCH_PCRE made me think that =~ fell back to
zsh/regex semantics if zsh/pcre were unavailable, which isn't the case
(it just fails — which is good).  Hence:

diff --git a/Doc/Zsh/options.yo b/Doc/Zsh/options.yo
index 1f8d98b..cc6ae2a 100644
--- a/Doc/Zsh/options.yo
+++ b/Doc/Zsh/options.yo
@@ -725,7 +725,8 @@ cindex(regexp, PCRE)
 cindex(PCRE, regexp)
 item(tt(REMATCH_PCRE))(
 If set, regular expression matching with the tt(=~) operator will use
-Perl-Compatible Regular Expressions from the PCRE library, if available.
+Perl-Compatible Regular Expressions from the PCRE library.
+(The tt(zsh/pcre) module must be available.)
 If not set, regular expressions will use the extended regexp syntax
 provided by the system libraries.
 )

Re: REMATCH_PCRE with zsh built without pcre support

[Bart Schaefer]

On Apr 3, 11:26am, Daniel Shahaf wrote:
}
} I agree that semantics of operators shouldn't depend on option

I'm not even particularly worried about semantic dependency; the gripe
here is that "setopt re_match_pcre" does not succeed (or fail) based on
the presence (or absence) of the module.  Exactly how big a problem is
that?  If you are relying on =~ having perl semantics, you ought to be
calling "zmodload zsh/pcre" anyway, and not just expecting setopt to do
that for you.

On the other hand if others think it's a major shortcoming that setopt
does not hard fail when the option won't actually have a useful effect,
then we should use a different paradigm than setopt here; yes it would
be possible for setopt to barf with "can't change option: rematchpcre"
but that means a module-specific special case that no other setopt has.

And now I've repeated myself at least three times so I'll stop.

} > [...] the zsh/regex module would have to become part of the default
} > static build, which might have licensing consequences.
} 
} What would those be?  zsh/regex is BSD licensed.

I don't know, and don't have time to figure it out.

Re: REMATCH_PCRE with zsh built without pcre support

[Daniel Shahaf]

Bart Schaefer wrote on Mon, Apr 03, 2017 at 11:00:16 -0700:
> On Apr 3, 11:26am, Daniel Shahaf wrote:
> }
> } I agree that semantics of operators shouldn't depend on option
> 
> I'm not even particularly worried about semantic dependency; the gripe
> here is that "setopt re_match_pcre" does not succeed (or fail) based on
> the presence (or absence) of the module.  Exactly how big a problem is
> that?  If you are relying on =~ having perl semantics, you ought to be
> calling "zmodload zsh/pcre" anyway, and not just expecting setopt to do
> that for you.

Yes, you should load zsh/pcre before using =~ with Perl semantics;
setopt won't do that for you.  No one is disputing either of these
assertions.  The question is just what should the failure mode be when
zsh/pcre _should_ have been loaded, but [due to a bug in the user's
script] hadn't been.  Should errflag be set once REMATCH_PCRE is set, or
at the first affected use of =~ ?

That's not a trivial question, but the point I was trying to make in my
previous post is that, while both of us might prefer [for different
reasons] that REMATCH_PCRE had never been added, we can't just remove it
tomorrow morning; before removing that option, we should provide an
upgrade path to users who use it correctly (= with 'zmodload -e' checks)
and rely on it to continue to work as documented.

That's all I was trying to say.  I'll step out of this thread for a day
or two now.  Apologies if I've tested your patience, that wasn't my
intention.

Cheers,

Daniel

Re: REMATCH_PCRE with zsh built without pcre support

[Bart Schaefer]

On Apr 4, 12:36am, Daniel Shahaf wrote:
}
} The question is just what should the failure mode be when
} zsh/pcre _should_ have been loaded, but [due to a bug in the user's
} script] hadn't been.  Should errflag be set once REMATCH_PCRE is set,
} or at the first affected use of =~ ?

You're correct that this is the question posed at the start of this
thread, but it's not the question I want to answer.

The question I want to answer is:  Given that the only reasonable time
to set errflag is at the first affected use of =~ , does that make the
present behavior (i.e., no error) of setopt REMATCH_PCRE so egregiously
wrong that we should get rid of the option itself?

My own answer is "no," but I'm willing to entertain counter-assertions.

I'm also willing to entertain "oh, well, who cares if we muck up the
code a bit more with a module-specfic hack in options.c, let's have
an error at an unreasonable time." I don't think it's a good idea, but
there have been worse ones.

} That's not a trivial question, but the point I was trying to make in my
} previous post is that, while both of us might prefer [for different
} reasons] that REMATCH_PCRE had never been added, we can't just remove it

Certainly one approach would be to always use pcre when it is loaded, and
always use regex when it is not, and only throw an error when neither of
them is available.  That would only introduce a problem if someone is
expecting regex semantics out of a pattern that is valid in both cases
but means something else in pcre semantics.  I'm not going to give myself
a headache trying to construct an example.

} Apologies if I've tested your patience, that wasn't my intention.

*You* are not who/what's testing my patience, and it wasn't my intent to
give you that impression.

If we go with the module-specific hack in options.c, we should also have
an option-specific hack in Modules/pcre.c so "zmodload -u zsh/pcre" is
an error when REMATCH_PCRE is set.  Enforced dependence cuts both ways.

Re: REMATCH_PCRE with zsh built without pcre support

[Eric Cook]

On 04/03/2017 10:43 PM, Bart Schaefer wrote:
> If we go with the module-specific hack in options.c, we should also have
> an option-specific hack in Modules/pcre.c so "zmodload -u zsh/pcre" is
> an error when REMATCH_PCRE is set.  Enforced dependence cuts both ways.
> 

If it isn't trivial to do, we could just commit the man page patch daniel posted,
because i was also under the impression that zsh/regex would've been used when
pcre wasn't available based on the sentence that didn't really explicitly say
that it would.

This is the first time i've seen someone run into the problem, but it seemed
simple enough for other people to trip over if they do set the option and use
their .zshrc across different OS'.

Re: REMATCH_PCRE with zsh built without pcre support

[Ray Andrews]

On 03/04/17 07:43 PM, Bart Schaefer wrote:
> Certainly one approach would be to always use pcre when it is loaded, and
> always use regex when it is not, and only throw an error when neither of
> them is available.  That would only introduce a problem if someone is
> expecting regex semantics out of a pattern that is valid in both cases
> but means something else in pcre semantics.  I'm not going to give myself
> a headache trying to construct an example.

Pardon, I don't understand the details of any of this, but would not the 
above be setting a time-bomb?  Surely it should be explicit which method 
is being used and one should never crash into something like: "Oh!  I've 
been thinking I was using pcre all this time and now I see that it's 
really regex."  or: "I want regex but since that's not available I'm 
getting pcre whether I want it or not, it seems."  Friendly fall-backs 
might seem friendly, but if they are doing something without explicit 
approval or maybe explicit notification are they not gotchas waiting to 
sabotage you?   This is one of my philosophical musings, just ignore it 
if it's not worth answering.

Re: REMATCH_PCRE with zsh built without pcre support

[Bart Schaefer]

On Apr 4,  8:22am, Ray Andrews wrote:
} Subject: Re: REMATCH_PCRE with zsh built without pcre support
}
} Pardon, I don't understand the details of any of this, but would not the 
} above be setting a time-bomb? Surely it should be explicit which method 

There would have to be an explicit "zmodload zsh/pcre" somewhere in order
for the pcre semantics to apply.

This is a little worse than REMATCH_PCRE because the option could be
toggled with LOCAL_OPTIONS whereas other special handling would be
needed to use the module only within a certain function scope.  That's
why I asked whether there were any known cases that would break.

But if you use =~ at all, you're accepting that *some* regex module
is going to be loaded.  If PCRE is a strict superset, you don't need
to care *what* module -- unless you want to use superset features, in
which case you'd better "zmodload zsh/pcre" yourself anyway.

Re: REMATCH_PCRE with zsh built without pcre support

[Daniel Shahaf]

Eric Cook wrote on Tue, Apr 04, 2017 at 08:27:16 -0400:
> On 04/03/2017 10:43 PM, Bart Schaefer wrote:
> > If we go with the module-specific hack in options.c, we should also have
> > an option-specific hack in Modules/pcre.c so "zmodload -u zsh/pcre" is
> > an error when REMATCH_PCRE is set.  Enforced dependence cuts both ways.
> > 
> 
> If it isn't trivial to do, we could just commit the man page patch daniel posted,
> because i was also under the impression that zsh/regex would've been used when
> pcre wasn't available based on the sentence that didn't really explicitly say
> that it would.

I'll go ahead and push it, to better document the incumbent behaviour,
until we decide whether to change it.

Re: REMATCH_PCRE with zsh built without pcre support

[Peter Stephenson]

I think the right way to do this "properly" would be generically in
terms of module features.  Options could have lists of module / prefix /
name strings, in this case "zsh/pcre", "C:", "pcre-match", for passing
to ensurefeature() --- or possibly autoloading.  It would have to pass
reverse dependencies into module.c somehow to force an unload to fail.
I suppose you could special case a null prefix and name to say just load
the module, but module features look like the right level of, er,
feature, to associate with the backend to an option in general.

It seems highly unlikely this is worth the significant effort.

pws

Re: REMATCH_PCRE with zsh built without pcre support

[Bart Schaefer]

On Apr 5,  2:42pm, Peter Stephenson wrote:
}
} It seems highly unlikely this is worth the significant effort.

It also seems like it would make toggling options on and off a very
heavy-weight affair.  If we had to check this on every option at
every state change, localoptions and sticky contexts would be way
too slow (and would have to be completely rewritten, as at this time
they are just bit-copying).