Re: Test failures when Git is built with libpcre and grep is built without it

Re: Test failures when Git is built with libpcre and grep is built without it

[A. Wilcox]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

(Hi, musl developers!  Read on for some information regarding what
began as a test failure during packaging of Git 2.7.3 that appears to
be related to musl's regexp library.)


On 09/01/17 05:27, Jeff King wrote:
> Are you trying with a version of git older than v2.7.x?


We are using 2.7.3.  I will admit the version is a bit old, but it is
working on all other arches (and it took an inordinate amount of time
to spin up PowerPC - hooray for toolchain issues).


On 09/01/17 15:33, Jeff King wrote:
> On Mon, Jan 09, 2017 at 02:05:44PM +0100, Andreas Schwab wrote:
>> You need to quote the regexp argument, see the line starting
>> with "test_must_fail" above.
> 
> Oh, duh. I checked that the line in the test was quoted


Guilty of this as well.  Sorry about that.  With the proper
invocation, I receive a success:


elaine trash directory.t7810-grep # git grep -G -F -P -E
"a\x{2b}b\x{2a}c" ab
error: cannot run less: No such file or directory
ab:a+b*c
elaine trash directory.t7810-grep # echo $?
0


Haven't managed to build less(1) for PowerPC yet, but it does seem to
work when quoted.  Yikes!


> The problem is that we are expecting the regex "\x{2b}" to complain
> in regcomp() (as an ERE), but it doesn't. And that probably _is_
> related to musl, which is providing the libc regex (I know this
> looks like a pcre test, but it's checking that "-P -E" overrides
> the pcre option with "-E").
> 
> I'm not sure if musl is wrong for failing to complain about a
> bogus regex. Generally making something that would break into
> something that works is an OK way to extend the standard. So our
> test is at fault for assuming that the regex will fail. I guess
> we'd need to find some more exotic syntax that pcre supports, but
> that ERE doesn't. Maybe "(?:)" or something.
> 
> -Peff
> 


I am cc:ing in the musl development list to see what their thoughts are.

Thanks for the help so far; I really appreciate it.  Hopefully we can
resolve this in a way that makes musl and Git's test suite both more
correct, if necessary.

Happy new year!

- --arw


- -- 
A. Wilcox (awilfox)
Project Lead, Adélie Linux
http://adelielinux.org
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBCAAGBQJYdLk+AAoJEMspy1GSK50Uz9kP/2JUSzSSXkKLy3788aAjgBOq
aTPBqKXLCvnTeBpsymXoGueaJzgDnYP6Xi9tUb/j4JAXqaJKGHXTgz8ixsFQJ4SJ
p7NN1JZXsIVGKWQHcxvEXEIRmBK7T9BQ2Hq6qUuk3n4PM6lbD1Ur3G1rUqIM/z76
5cSkvwgvGuOVYXLwSTTprb6EbbZc6WTgcDECFIl4z7eJ3GihChg+EgH4cTDt3xLD
9VHx++hWoEZXued0g0myENjBMeCiFOpYw1bzxn7d2s8masx/If2TkK0CLdexy2ti
hFV/F7g8etgZSdmrmiaJIudTsY6p46/sm/VmGm+8yPIeR8/8EOBlyhKhz4EidNtA
2dpUUJW/RD4uz9yVNjmWhIHaL10weuNLQLd/Tt5O/0dG422vhJxOwwJL9Vhxw70y
1PprD11+ZUJeWsz91C/Bl9CqHqljDP00QX8w/dbHCDsjeKgfUdksy2iKlHBG0sx3
CbL0EcHeE9BLmqmJi9ED7w78cTmNryemK29WFHUGSwzR99Rv9QlMn+4GCF1s0O3N
UKPCueLpPcNvRK+1WHkAwUC0iV97Un3WI6kyExs9khygIHRd1y/yRhbpxibbz337
FJfOTcT7e/YqML8Nb8EdwECRhdGX5u1tKqaOfBwLC62SKGTPi5qXpjaSPPJcJsrZ
phxFt6a45PPECV3ZsEO1
=PMkh
-----END PGP SIGNATURE-----

Re: Re: Test failures when Git is built with libpcre and grep is built without it

[Szabolcs Nagy]

* A. Wilcox <awilfox@adelielinux.org> [2017-01-10 04:36:50 -0600]:
> On 09/01/17 15:33, Jeff King wrote:
> > The problem is that we are expecting the regex "\x{2b}" to complain
> > in regcomp() (as an ERE), but it doesn't. And that probably _is_
> > related to musl, which is providing the libc regex (I know this
> > looks like a pcre test, but it's checking that "-P -E" overrides
> > the pcre option with "-E").
> > 
> > I'm not sure if musl is wrong for failing to complain about a
> > bogus regex. Generally making something that would break into
> > something that works is an OK way to extend the standard. So our
> > test is at fault for assuming that the regex will fail. I guess

\x is undefined in posix and musl is based on tre which
supports \x{hexdigits} in ere.

(i think some bsd platforms use tre as libc regex so
i'm surprised musl is the first to run into this.)

> > we'd need to find some more exotic syntax that pcre supports, but
> > that ERE doesn't. Maybe "(?:)" or something.

i think you would have to use something that's invalid
in posix ere, ? after empty expression is undefined,
not an error so "(?:)" is a valid ere extension.

since most syntax is either defined or undefined in ere
instead of being invalid, distinguishing pcre using
syntax is not easy.

there are semantic differences in subexpression matching:
leftmost match has higher priority in pcre, longest match
has higher priority in ere.

$ echo ab | grep -o -E '(a|ab)'
ab
$ echo ab | grep -o -P '(a|ab)'
a

unfortunately grep -o is not portable.

Re: [musl] Re: Test failures when Git is built with libpcre and grep is built without it

[Jeff King]

On Tue, Jan 10, 2017 at 12:40:00PM +0100, Szabolcs Nagy wrote:

> > > I'm not sure if musl is wrong for failing to complain about a
> > > bogus regex. Generally making something that would break into
> > > something that works is an OK way to extend the standard. So our
> > > test is at fault for assuming that the regex will fail. I guess
> 
> \x is undefined in posix and musl is based on tre which
> supports \x{hexdigits} in ere.

Thanks for confirming; I figured it was something like that.

> > > we'd need to find some more exotic syntax that pcre supports, but
> > > that ERE doesn't. Maybe "(?:)" or something.
> 
> i think you would have to use something that's invalid
> in posix ere, ? after empty expression is undefined,
> not an error so "(?:)" is a valid ere extension.

Reading through POSIX[1], hardly anything is explicitly labeled as
"invalid". Most things are just "undefined", which leaves rooms for
implementations to do what they like.

That's a good thing for a standard to do, but a bad thing when you are
trying to find behavior that differs reliably between PCRE and ERE. :)
In most cases, PCRE constructs could be viable extensions to ERE.

> since most syntax is either defined or undefined in ere
> instead of being invalid, distinguishing pcre using
> syntax is not easy.
> 
> there are semantic differences in subexpression matching:
> leftmost match has higher priority in pcre, longest match
> has higher priority in ere.
> 
> $ echo ab | grep -o -E '(a|ab)'
> ab
> $ echo ab | grep -o -P '(a|ab)'
> a
> 
> unfortunately grep -o is not portable.

In this case we're testing whether Git has internally fed the regex to
pcre or to regcomp(), not a system grep. So we'd need something like
"-o" for "git grep", which I don't think exists.

Another difference I found is that "[\d]" matches a literal "\" or "d"
in ERE, but behaves like "[0-9]" in PCRE. I'll work up a patch based on
that.

Thanks for your answer. I'll drop the musl list from the cc when I
follow-up, as this is most definitely not a musl problem, but a git one.

-Peff