fnmatch and FNM_CASEFOLD

fnmatch and FNM_CASEFOLD

[orc]

Hi list,

This one really terrorized me well. Musl defines FNM_CASEFOLD in fnmatch.h but really does nothing when this flag is passed to function. This breaks busybox find -iname I believe, instead of file list matching "Test" among with "test" i get only those which contain "test" in their names with -iname "test". Missed lot of files because of it.
Wishing this fixed or such define being removed.

Re: fnmatch and FNM_CASEFOLD

[Rich Felker]

On Fri, Nov 29, 2013 at 05:59:12PM +0800, orc wrote:
> Hi list,
> 
> This one really terrorized me well. Musl defines FNM_CASEFOLD in
> fnmatch.h but really does nothing when this flag is passed to
> function. This breaks busybox find -iname I believe, instead of file
> list matching "Test" among with "test" i get only those which
> contain "test" in their names with -iname "test". Missed lot of
> files because of it.
> Wishing this fixed or such define being removed.

I think this is a reasonable request. FNM_CASEFOLD is problematic to
support and I don't see adding it in the near/forseeable future.
However I do wonder if we'd lose a lot of out-of-the-box compatibility
relative to what we have now; in many cases, compiling without
case-insensitive matching working might be preferable to not compiling
at all. Is anyone willing to take up a task of checking which packages
in pkgsrc or other 'supported' packages are using FNM_CASEFOLD
unconditionally and how much work it would be to get patches upstream
to make the use conditional on #ifdef FNM_CASEFOLD?

Rich

Re: fnmatch and FNM_CASEFOLD

[Rich Felker]

On Fri, Nov 29, 2013 at 12:42:49PM -0500, Rich Felker wrote:
> On Fri, Nov 29, 2013 at 05:59:12PM +0800, orc wrote:
> > Hi list,
> > 
> > This one really terrorized me well. Musl defines FNM_CASEFOLD in
> > fnmatch.h but really does nothing when this flag is passed to
> > function. This breaks busybox find -iname I believe, instead of file
> > list matching "Test" among with "test" i get only those which
> > contain "test" in their names with -iname "test". Missed lot of
> > files because of it.
> > Wishing this fixed or such define being removed.
> 
> I think this is a reasonable request. FNM_CASEFOLD is problematic to

I'd also like to remove FNM_LEADING_DIR, which is unsupported by musl
and documented in the man page as:

    "This flag is mainly for the internal use of glibc and is
    implemented only in certain cases."

Rich

Re: fnmatch and FNM_CASEFOLD

[Rich Felker]

On Fri, Nov 29, 2013 at 06:34:17PM -0500, Rich Felker wrote:
> On Fri, Nov 29, 2013 at 12:42:49PM -0500, Rich Felker wrote:
> > On Fri, Nov 29, 2013 at 05:59:12PM +0800, orc wrote:
> > > Hi list,
> > > 
> > > This one really terrorized me well. Musl defines FNM_CASEFOLD in
> > > fnmatch.h but really does nothing when this flag is passed to
> > > function. This breaks busybox find -iname I believe, instead of file
> > > list matching "Test" among with "test" i get only those which
> > > contain "test" in their names with -iname "test". Missed lot of
> > > files because of it.
> > > Wishing this fixed or such define being removed.
> > 
> > I think this is a reasonable request. FNM_CASEFOLD is problematic to
> 
> I'd also like to remove FNM_LEADING_DIR, which is unsupported by musl
> and documented in the man page as:
> 
>     "This flag is mainly for the internal use of glibc and is
>     implemented only in certain cases."

Unfortunately, Debian codesearch is showing a lot of hits for
FNM_LEADING_DIR; I haven't researched whether they're hard or soft
dependencies, so if anyone is willing to look into that, it'd be
helpful. See: http://codesearch.debian.net/search?q=FNM_LEADING_DIR

I'm not opposed to implementing it if it's non-intrusive. Same goes
for FNM_CASEFOLD though we need to consider what semantics it should
have; nsz has researched what the BSD version does and it's fairly
simple but possibly counter-intuitive or possibly "right" depending on
your perspective. :-)

Unfortunately I'm leaning towards punting on all of these issues until
after 1.0, since it's not clear how much build breakage would result
from (temporarily) removing the macros, whereas the functional
breakage from them being no-ops seems low..

Rich

Re: fnmatch and FNM_CASEFOLD

[Felix Janda]

Rich Felker wrote:
> On Fri, Nov 29, 2013 at 06:34:17PM -0500, Rich Felker wrote:
[...]
> > I'd also like to remove FNM_LEADING_DIR, which is unsupported by musl
> > and documented in the man page as:
> > 
> >     "This flag is mainly for the internal use of glibc and is
> >     implemented only in certain cases."
> 
> Unfortunately, Debian codesearch is showing a lot of hits for
> FNM_LEADING_DIR; I haven't researched whether they're hard or soft
> dependencies, so if anyone is willing to look into that, it'd be
> helpful. See: http://codesearch.debian.net/search?q=FNM_LEADING_DIR

The vast majority of the matches comes from code copied from gnulib,...
implementing fnmatch().

busybox's tar uses it if available for the -X and -T option for file
in/exclusion.

Jane Street's core library for OCaml provides libc's version of
fnmatch() to its users. Its implementation expects FNM_LEADING_DIR
(and also FNM_CASE_FOLD) to exists.

recoll will only use FNM_LEADING_DIR if it is available. However its
documentation says it is always used.

csync2 and dds2tar use the flag unconditionally.

sitecopy checks in its config script for the existance of FNM_LEADING_DIR
and uses it if declared. It is therefore currently broken for musl and
fixed if one removes the declaration.

Felix

Re: fnmatch and FNM_CASEFOLD

[Rich Felker]

On Sun, Dec 01, 2013 at 03:02:35PM +0100, Felix Janda wrote:
> Rich Felker wrote:
> > On Fri, Nov 29, 2013 at 06:34:17PM -0500, Rich Felker wrote:
> [...]
> > > I'd also like to remove FNM_LEADING_DIR, which is unsupported by musl
> > > and documented in the man page as:
> > > 
> > >     "This flag is mainly for the internal use of glibc and is
> > >     implemented only in certain cases."
> > 
> > Unfortunately, Debian codesearch is showing a lot of hits for
> > FNM_LEADING_DIR; I haven't researched whether they're hard or soft
> > dependencies, so if anyone is willing to look into that, it'd be
> > helpful. See: http://codesearch.debian.net/search?q=FNM_LEADING_DIR
> 
> The vast majority of the matches comes from code copied from gnulib,...
> implementing fnmatch().

That makes sense. However there seem to be gnulib tests checking calls
with FNM_LEADING_DIR even though they don't seem to depend on the
behavior. Would removing FNM_LEADING_DIR from musl's fnmatch.h cause
these tests to fail and cause programs using gnulib to pull in the
(highly inefficient and broken) GNU fnmatch source?

> busybox's tar uses it if available for the -X and -T option for file
> in/exclusion.

Sadly removing FNM_LEADING_DIR from musl's headers would not fix
busybox since busybox just defines it to 0 if it's missing and then
proceeds to use it as if it worked... This is documented in the
comments as being something to fix, but I haven't seen any progress on
fixing it.

> Jane Street's core library for OCaml provides libc's version of
> fnmatch() to its users. Its implementation expects FNM_LEADING_DIR
> (and also FNM_CASE_FOLD) to exists.

What do you mean by "expects"? What goes wrong if it's missing?

> recoll will only use FNM_LEADING_DIR if it is available. However its
> documentation says it is always used.

Does lack of FNM_LEADING_DIR impact functionality?

> csync2 and dds2tar use the flag unconditionally.

I would guess these mostly work with it as a no-op (like now), only
failing to support some user-provided patterns. Is this true?

> sitecopy checks in its config script for the existance of FNM_LEADING_DIR
> and uses it if declared. It is therefore currently broken for musl and
> fixed if one removes the declaration.

While writing this email, I actually looked into how FNM_LEADING_DIR
is supposed to work, and it's a trivial one-line change to make it
work in combination with FNM_PATHNAME, so I think we should just add
that, and document that it's a no-op without FNM_PATHNAME. Can you
tell me if this would cover most real-world usage? At some later point
we might be able to add it for non-FNM_PATHNAME. I have ideas how to
do this but I'd like to combine that with an overhaul to the whole
file I have in mind for post-1.0.

Rich

Re: fnmatch and FNM_CASEFOLD

[Rich Felker]

On Sun, Dec 01, 2013 at 05:06:10PM -0500, Rich Felker wrote:
> While writing this email, I actually looked into how FNM_LEADING_DIR
> is supposed to work, and it's a trivial one-line change to make it
> work in combination with FNM_PATHNAME, so I think we should just add
> that, and document that it's a no-op without FNM_PATHNAME. Can you
> tell me if this would cover most real-world usage? At some later point
> we might be able to add it for non-FNM_PATHNAME. I have ideas how to
> do this but I'd like to combine that with an overhaul to the whole
> file I have in mind for post-1.0.

I found a trivial but inefficient way to handle the non-FNM_PATHNAME
case too, so I've implemented both and committed. Now FNM_LEADING_DIR
should be fully supported/working.

FNM_CASEFOLD on the other hand remains a no-op and I'm unclear on what
to do with it. The intent of POSIX seems to be that fnmatch bracket
expressions behave like RE brackets, for which POSIX specifies
case-insensitive semanics (for REG_ICASE flag, etc.) that disagree
with what glibc is doing. So the choices we're stuck with are
implementing a "correct" version of a nonstandard interface that
differs from the real-world systems it's modeled after, an "incorrect"
case-insensitivity, or something else entirely. It's messy enough that
I see why I originally did not even want to go here at all, but I'm
not sure what we should do.

Rich