More GNU semantics for getopt_long?

More GNU semantics for getopt_long?

[Felix Janda]

I have had two problems with the fact that musl's getopt_long() does
not behave as applications expect.


No argument reordering:

musl's getopt_long() behaves similarly to POSIX getopt() and therefore
stops after the first non-option argument. This for example makes the
utilities widl and wrc from wine when compiled with musl libc behave
differently than for everyone else. In particular, this breaks the
(git) wine build itself, since its generated Makefiles call widl with
mixed option and non-option arguments.

Should wine use a (runtime) config test detecting whether
getopt_long() does argument reordering and use its internal copy of
getopt_long() if not? Should the Makefile generation be carefully
rewritten such that non-options are given last?

It seems difficult to upstream any of these. The config test
seems like a test specific to exclude musl and the second is
likely to break in the future and who knows what scripts might
depend on the argument reordering of widl.


No abbreviations:

musl's getopt_long() does not support abbreviation of long options.
Ironically this breaks util-linux' (or busybox') getopt(1) itself.
Most scripts use "getopt --long" instead of "getopt --longoptions"
(likely because getopt includes an example script using
"getopt --long"). I noticed this on my gentoo system when glib would
fail at configure because /etc/xml/catalog was empty because the
gentoo specific script build-docbook-catalog was broken because
it used "getopt --long".

I noticed that there is a patch from Michael Forney on the mailing
list implementing the abbreviated options but there were not any
comments on it.


These problems make me question the usefulness of musl having
getopt_long() at all in its current form. There is no way to know
which programs depend on the GNU getopt_long() behavior. Furthermore,
musl seems to have the only implementation behaving differently.

Felix

Re: More GNU semantics for getopt_long?

[Rich Felker]

On Sat, Jul 26, 2014 at 11:12:36AM +0200, Felix Janda wrote:
> I have had two problems with the fact that musl's getopt_long() does
> not behave as applications expect.
> 
> 
> No argument reordering:
> 
> musl's getopt_long() behaves similarly to POSIX getopt() and therefore
> stops after the first non-option argument. This for example makes the
> utilities widl and wrc from wine when compiled with musl libc behave

This has been discussed a few times and could possibly be changed.
Some consideration is needed for whether it leads to problematic
inconsistency between getopt and getopt_long, but I don't think so.
The worst part is that it makes it impossible to get conforming
behavior from GNU coreutils since there would be no way to override
the reordering (whereas, if they replace getopt_long with their own
version, it can be overridden by setting the POSIXLY_CORRECT
environment variable).

> differently than for everyone else. In particular, this breaks the
> (git) wine build itself, since its generated Makefiles call widl with
> mixed option and non-option arguments.
> 
> Should wine use a (runtime) config test detecting whether
> getopt_long() does argument reordering and use its internal copy of
> getopt_long() if not? Should the Makefile generation be carefully
> rewritten such that non-options are given last?

I'm not a fan of the misordered options/non-option-arguments, so I'd
recommend "fixing" that in the Makefile even if musl's behavior
changes. Note that the build process probably breaks if the user has
POSIXLY_CORRECT set in their environment since this turns off the GNU
reordering in glibc.

> It seems difficult to upstream any of these. The config test
> seems like a test specific to exclude musl and the second is
> likely to break in the future and who knows what scripts might
> depend on the argument reordering of widl.
> 
> 
> No abbreviations:

Oh, there was a patch submitted for this one, and I thought I
remembered applying it, but perhaps not. I'll go back and look. There
might have still been some open issues with it to be resolved before
applying.

> I noticed that there is a patch from Michael Forney on the mailing
> list implementing the abbreviated options but there were not any
> comments on it.

Yes that's it.

Rich

Re: More GNU semantics for getopt_long?

[Felix Janda]

Rich Felker wrote:
> On Sat, Jul 26, 2014 at 11:12:36AM +0200, Felix Janda wrote:
> > I have had two problems with the fact that musl's getopt_long() does
> > not behave as applications expect.
> > 
> > 
> > No argument reordering:
> > 
> > musl's getopt_long() behaves similarly to POSIX getopt() and therefore
> > stops after the first non-option argument. This for example makes the
> > utilities widl and wrc from wine when compiled with musl libc behave
> 
> This has been discussed a few times and could possibly be changed.
> Some consideration is needed for whether it leads to problematic
> inconsistency between getopt and getopt_long, but I don't think so.
> The worst part is that it makes it impossible to get conforming
> behavior from GNU coreutils since there would be no way to override
> the reordering (whereas, if they replace getopt_long with their own
> version, it can be overridden by setting the POSIXLY_CORRECT
> environment variable).

The BSDs seem to have this inconsistency between getopt and getopt_long
as well. Their getopt_long also respects POSIXLY_CORRECT, though. This
is IMO something musl should do as well when possibly introducing the
reordering of arguments in future.

> > differently than for everyone else. In particular, this breaks the
> > (git) wine build itself, since its generated Makefiles call widl with
> > mixed option and non-option arguments.
> > 
> > Should wine use a (runtime) config test detecting whether
> > getopt_long() does argument reordering and use its internal copy of
> > getopt_long() if not? Should the Makefile generation be carefully
> > rewritten such that non-options are given last?
> 
> I'm not a fan of the misordered options/non-option-arguments, so I'd
> recommend "fixing" that in the Makefile even if musl's behavior
> changes. Note that the build process probably breaks if the user has
> POSIXLY_CORRECT set in their environment since this turns off the GNU
> reordering in glibc.

Good, I have a patch for their Makefile generation. Thanks for
pointing out that the build should also break for others when
POSIXLY_CORRECT is set.

> > It seems difficult to upstream any of these. The config test
> > seems like a test specific to exclude musl and the second is
> > likely to break in the future and who knows what scripts might
> > depend on the argument reordering of widl.
> > 
> > 
> > No abbreviations:
> 
> Oh, there was a patch submitted for this one, and I thought I
> remembered applying it, but perhaps not. I'll go back and look. There
> might have still been some open issues with it to be resolved before
> applying.
> 
> > I noticed that there is a patch from Michael Forney on the mailing
> > list implementing the abbreviated options but there were not any
> > comments on it.
> 
> Yes that's it.

Thanks for revisiting the patch.

Felix

Re: More GNU semantics for getopt_long?

[Rich Felker]

On Sat, Jul 26, 2014 at 07:48:30PM +0200, Felix Janda wrote:
> Rich Felker wrote:
> > On Sat, Jul 26, 2014 at 11:12:36AM +0200, Felix Janda wrote:
> > > I have had two problems with the fact that musl's getopt_long() does
> > > not behave as applications expect.
> > > 
> > > 
> > > No argument reordering:
> > > 
> > > musl's getopt_long() behaves similarly to POSIX getopt() and therefore
> > > stops after the first non-option argument. This for example makes the
> > > utilities widl and wrc from wine when compiled with musl libc behave
> > 
> > This has been discussed a few times and could possibly be changed.
> > Some consideration is needed for whether it leads to problematic
> > inconsistency between getopt and getopt_long, but I don't think so.
> > The worst part is that it makes it impossible to get conforming
> > behavior from GNU coreutils since there would be no way to override
> > the reordering (whereas, if they replace getopt_long with their own
> > version, it can be overridden by setting the POSIXLY_CORRECT
> > environment variable).
> 
> The BSDs seem to have this inconsistency between getopt and getopt_long
> as well. Their getopt_long also respects POSIXLY_CORRECT, though. This
> is IMO something musl should do as well when possibly introducing the
> reordering of arguments in future.

Yes, that may be reasonable. Since getopt_long is not standard anyway
there's no reason it has to avoid reading the environment. For
standard functions, they can't read the environment unless they're
specified to because doing so introduces a usage limitation (they
can't be called when the environment could be being modified); glibc's
getopt doing so is a bug, I think.

Rich

Re: More GNU semantics for getopt_long?

[Rich Felker]

On Sat, Jul 26, 2014 at 07:48:30PM +0200, Felix Janda wrote:
> > > It seems difficult to upstream any of these. The config test
> > > seems like a test specific to exclude musl and the second is
> > > likely to break in the future and who knows what scripts might
> > > depend on the argument reordering of widl.
> > > 
> > > 
> > > No abbreviations:
> > 
> > Oh, there was a patch submitted for this one, and I thought I
> > remembered applying it, but perhaps not. I'll go back and look. There
> > might have still been some open issues with it to be resolved before
> > applying.
> > 
> > > I noticed that there is a patch from Michael Forney on the mailing
> > > list implementing the abbreviated options but there were not any
> > > comments on it.
> > 
> > Yes that's it.
> 
> Thanks for revisiting the patch.

Have you tried it and confirmed that it solves your problem and
doesn't introduce any bugs that you immediately notice? I'd like to
include it in the upcoming release but I don't want to introduce a
stupid regression from lack of testing...

Rich

Re: More GNU semantics for getopt_long?

[Felix Janda]

Rich Felker wrote:
[..]
> > > > I noticed that there is a patch from Michael Forney on the mailing
> > > > list implementing the abbreviated options but there were not any
> > > > comments on it.
> > > 
> > > Yes that's it.
> > 
> > Thanks for revisiting the patch.
> 
> Have you tried it and confirmed that it solves your problem and
> doesn't introduce any bugs that you immediately notice? I'd like to
> include it in the upcoming release but I don't want to introduce a
> stupid regression from lack of testing...

I did some tests right now. The abbreviated options seem to work.

If the long option abbreviation is ambiguous, patched musl's
getopt_long will return the last match in the longopts array instead
of '?'.

I also noticed that 66fcde4ae4a52ae3edb1cf237ce2c22d08d7a062 seems
to have broken getopt_long: Even if optstring does not begin with
':', getopt_long will return ':' if a long option is not supplied
by its required argument.

Felix

Re: More GNU semantics for getopt_long?

[Rich Felker]

On Mon, Jul 28, 2014 at 12:30:44AM +0200, Felix Janda wrote:
> Rich Felker wrote:
> [..]
> > > > > I noticed that there is a patch from Michael Forney on the mailing
> > > > > list implementing the abbreviated options but there were not any
> > > > > comments on it.
> > > > 
> > > > Yes that's it.
> > > 
> > > Thanks for revisiting the patch.
> > 
> > Have you tried it and confirmed that it solves your problem and
> > doesn't introduce any bugs that you immediately notice? I'd like to
> > include it in the upcoming release but I don't want to introduce a
> > stupid regression from lack of testing...

I'd like to resume working on getting this integrated, especially
since one of the main Roadmap goals for 1.1.6 is solving the remaining
issues Alpine Linux is matching musl for, of which getopt_long is one.

> I did some tests right now. The abbreviated options seem to work.
> 
> If the long option abbreviation is ambiguous, patched musl's
> getopt_long will return the last match in the longopts array instead
> of '?'.
> 
> I also noticed that 66fcde4ae4a52ae3edb1cf237ce2c22d08d7a062 seems
> to have broken getopt_long: Even if optstring does not begin with
> ':', getopt_long will return ':' if a long option is not supplied
> by its required argument.

Is this still broken, and if so, could you provide a short testcase I
could use for checking it and figuring out what's wrong?

Rich

Re: More GNU semantics for getopt_long?

[Felix Janda]

Rich Felker wrote:
> I'd like to resume working on getting this integrated, especially
> since one of the main Roadmap goals for 1.1.6 is solving the remaining
> issues Alpine Linux is matching musl for, of which getopt_long is one.

Thanks for coming back to this.

> > I also noticed that 66fcde4ae4a52ae3edb1cf237ce2c22d08d7a062 seems
> > to have broken getopt_long: Even if optstring does not begin with
> > ':', getopt_long will return ':' if a long option is not supplied
> > by its required argument.
> 
> Is this still broken, and if so, could you provide a short testcase I
> could use for checking it and figuring out what's wrong?

Nothing has changed. Below is a testcase.

--Felix

#include <stdio.h>
#include <getopt.h>

int main(void) {
	struct option opts[2] = {{"opt", 1, NULL, 'o'}, {0, 0, 0, 0}};
	int ret;

	ret = getopt_long(2, (char *[3]){"a", "--opt", 0}, "", opts, NULL);
	if (ret != '?') printf("'%d' != '%d'\n", ret, '?');

	return 0;
}

Re: More GNU semantics for getopt_long?

[Rich Felker]

On Wed, Nov 19, 2014 at 07:44:50PM +0100, Felix Janda wrote:
> Rich Felker wrote:
> > I'd like to resume working on getting this integrated, especially
> > since one of the main Roadmap goals for 1.1.6 is solving the remaining
> > issues Alpine Linux is matching musl for, of which getopt_long is one.
> 
> Thanks for coming back to this.
> 
> > > I also noticed that 66fcde4ae4a52ae3edb1cf237ce2c22d08d7a062 seems
> > > to have broken getopt_long: Even if optstring does not begin with
> > > ':', getopt_long will return ':' if a long option is not supplied
> > > by its required argument.
> > 
> > Is this still broken, and if so, could you provide a short testcase I
> > could use for checking it and figuring out what's wrong?
> 
> Nothing has changed. Below is a testcase.
> 
> --Felix
> 
> #include <stdio.h>
> #include <getopt.h>
> 
> int main(void) {
> 	struct option opts[2] = {{"opt", 1, NULL, 'o'}, {0, 0, 0, 0}};
> 	int ret;
> 
> 	ret = getopt_long(2, (char *[3]){"a", "--opt", 0}, "", opts, NULL);
> 	if (ret != '?') printf("'%d' != '%d'\n", ret, '?');
> 
> 	return 0;
> }

Are you sure it's related to the change in that commit? I think it was
always this way. getopt_long.c seems to return ':' unconditionally
when the argument is missing rather than returning '?'. We probably
also need to look into what the appropriate behavior should be for how
the parsing state is left when errors like this are encountered,
although I would guess optind should just end up pointing to the null
pointer at the end of argv so that subsequent calls report that
they're at the end.

Rich

Re: More GNU semantics for getopt_long?

[Felix Janda]

Rich Felker wrote:
> > > > I also noticed that 66fcde4ae4a52ae3edb1cf237ce2c22d08d7a062 seems
> > > > to have broken getopt_long: Even if optstring does not begin with
> > > > ':', getopt_long will return ':' if a long option is not supplied
> > > > by its required argument.
> > > 
> > > Is this still broken, and if so, could you provide a short testcase I
> > > could use for checking it and figuring out what's wrong?
> > 
> > Nothing has changed. Below is a testcase.
> > 
> > --Felix
> > 
> > #include <stdio.h>
> > #include <getopt.h>
> > 
> > int main(void) {
> > 	struct option opts[2] = {{"opt", 1, NULL, 'o'}, {0, 0, 0, 0}};
> > 	int ret;
> > 
> > 	ret = getopt_long(2, (char *[3]){"a", "--opt", 0}, "", opts, NULL);
> > 	if (ret != '?') printf("'%d' != '%d'\n", ret, '?');
> > 
> > 	return 0;
> > }
> 
> Are you sure it's related to the change in that commit? I think it was
> always this way. getopt_long.c seems to return ':' unconditionally
> when the argument is missing rather than returning '?'.

Right, it doesn't seem to be related to the commit.

> We probably
> also need to look into what the appropriate behavior should be for how
> the parsing state is left when errors like this are encountered,
> although I would guess optind should just end up pointing to the null
> pointer at the end of argv so that subsequent calls report that
> they're at the end.

The POSIX page on getopt confuses me a bit. It says:

  The getopt() function shall return the next option character (if one
  is found) from argv that matches a character in optstring, if there
  is one that matches. If the option takes an argument, getopt() shall
  set the variable optarg to point to the option-argument as follows:

   1. If the option was the last character in the string pointed to by
      an element of argv, then optarg shall contain the next element of
      argv, and optind shall be incremented by 2. If the resulting
      value of optind is greater than argc, this indicates a missing
      option-argument, and getopt() shall return an error indication.

So should optind be increased by 2 to point into nirvana when the
argument is missing? Considering the Example below on the page, I think
that it rather should behave as you have described.

Felix