printf POSIX compliance

printf POSIX compliance

[Reuben Thomas]

I notice that the musl FAQ says "the fundamentally broken freadahead
function in gnulib cannot be fixed by adding features to the C
library". I found this question because a user complained about my use
of gnulib in a package I maintain.

I contacted the gnulib maintainers, one of whom replied:

"IIRC, gnulib's freadahead use is caused by musl's printf not being
posix compliant, causing gnulib to pull in its printf replacement,
which doesn't work on musl."

It would be nice to sort this out: either musl's printf is not
POSIX-compliant, or gnulib's detection of POSIX-compliance is buggy.

Does any musl developer fancy looking at a gnulib-using package (e.g.
GNU grep, coreutils &c.) and finding out what gnulib thinks is wrong
with musl's printf? There are obvious benefits to sorting this out,
wherever the fault lies!

-- 
http://rrt.sc3d.org

Re: printf POSIX compliance

[Luca Barbato]

On 06/08/2012 12:34 PM, Reuben Thomas wrote:
> Does any musl developer fancy looking at a gnulib-using package (e.g.
> GNU grep, coreutils &c.) and finding out what gnulib thinks is wrong
> with musl's printf? There are obvious benefits to sorting this out,
> wherever the fault lies!

Could you please look at the config.log and report?

lu

-- 

Luca Barbato
Gentoo/linux
http://dev.gentoo.org/~lu_zero

Re: printf POSIX compliance

[Reuben Thomas]

On 8 June 2012 13:19, Luca Barbato <lu_zero@gentoo.org> wrote:
> On 06/08/2012 12:34 PM, Reuben Thomas wrote:
>> Does any musl developer fancy looking at a gnulib-using package (e.g.
>> GNU grep, coreutils &c.) and finding out what gnulib thinks is wrong
>> with musl's printf? There are obvious benefits to sorting this out,
>> wherever the fault lies!
>
> Could you please look at the config.log and report?

I don't have a config.log; I don't use musl. I meant to suggest that
perhaps someone who does use musl could try this.

-- 
http://rrt.sc3d.org

Re: printf POSIX compliance

[Szabolcs Nagy]

* Reuben Thomas <rrt@sc3d.org> [2012-06-08 11:34:04 +0100]:
> I contacted the gnulib maintainers, one of whom replied:
> 
> "IIRC, gnulib's freadahead use is caused by musl's printf not being
> posix compliant, causing gnulib to pull in its printf replacement,
> which doesn't work on musl."
> 

what does freadahead have to do with printf?
that sounds weird..

> It would be nice to sort this out: either musl's printf is not
> POSIX-compliant, or gnulib's detection of POSIX-compliance is buggy.
> 

so far i haven't seen much code from gnulib that was *not* buggy

but yes this should be sorted out..

Re: printf POSIX compliance

[Rich Felker]

On Fri, Jun 08, 2012 at 04:04:00PM +0200, Szabolcs Nagy wrote:
> * Reuben Thomas <rrt@sc3d.org> [2012-06-08 11:34:04 +0100]:
> > I contacted the gnulib maintainers, one of whom replied:
> > 
> > "IIRC, gnulib's freadahead use is caused by musl's printf not being
> > posix compliant, causing gnulib to pull in its printf replacement,
> > which doesn't work on musl."
> 
> what does freadahead have to do with printf?
> that sounds weird..

I agree. Even if there is a bug in musl causing this stuff to be
pulled in, there seems to be a bug in gnulib as well; why should a
function that pertains only to reading get pulled in by a replacement
for a function that only does writing?

> > It would be nice to sort this out: either musl's printf is not
> > POSIX-compliant, or gnulib's detection of POSIX-compliance is buggy.
> 
> so far i haven't seen much code from gnulib that was *not* buggy

Indeed. If nothing else, almost all of it is non-thread-safe...

Rich

Re: printf POSIX compliance

[Rich Felker]

On Fri, Jun 08, 2012 at 11:34:04AM +0100, Reuben Thomas wrote:
> I notice that the musl FAQ says "the fundamentally broken freadahead
> function in gnulib cannot be fixed by adding features to the C
> library". I found this question because a user complained about my use
> of gnulib in a package I maintain.
> 
> I contacted the gnulib maintainers, one of whom replied:
> 
> "IIRC, gnulib's freadahead use is caused by musl's printf not being
> posix compliant, causing gnulib to pull in its printf replacement,
> which doesn't work on musl."

The first failing test I found was invalid. It's a test for whether
printf supports printing non-finite long double values. gnulib is
constructing an illegal long double representation (which they call
"pseudo-denormal") that will never occur as a value and passing it to
printf:

|   { /* Pseudo-Denormal.  */
|     static union { unsigned int word[4]; long double value; } x =
|       { LDBL80_WORDS (0x0000, 0x83333333, 0x00000000) };
|     if (sprintf (buf, "%Lf", x.value) < 0
|         || !strisnan (buf, 0, strlen (buf)))
|       result |= 64;

This test should simply be removed; there is no reason printf should
handle invalid bit patterns that cannot occur as values.

The second failing test actually caught a slight bug in %ls with
precision modifiers. I've fixed it in musl git.

Still working on finding whether this long double issue is what's
causign the gnulib junk to get pulled in...

Rich

Re: printf POSIX compliance

[Rich Felker]

On Fri, Jun 08, 2012 at 10:44:23AM -0400, Rich Felker wrote:
> Still working on finding whether this long double issue is what's
> causign the gnulib junk to get pulled in...

And it's not, at least not in the worst package, GNU m4. I #if'd out
the broken long double test in configure, re-ran it, and even with no
printf problems reported, freadahead is still getting pulled in...
I'll see if I see anything else odd.

Rich

Re: printf POSIX compliance

[Szabolcs Nagy]

* Rich Felker <dalias@aerifal.cx> [2012-06-08 10:55:19 -0400]:
> On Fri, Jun 08, 2012 at 10:44:23AM -0400, Rich Felker wrote:
> > Still working on finding whether this long double issue is what's
> > causign the gnulib junk to get pulled in...
> 
> And it's not, at least not in the worst package, GNU m4. I #if'd out
> the broken long double test in configure, re-ran it, and even with no
> printf problems reported, freadahead is still getting pulled in...
> I'll see if I see anything else odd.
> 

i've just checked m4 and it uses freadseek and closein

both functions are from gnulib and depend on freadahead

so m4 will use freadahead independently of the printf issue

Re: printf POSIX compliance

[Rich Felker]

On Fri, Jun 08, 2012 at 05:06:18PM +0200, Szabolcs Nagy wrote:
> * Rich Felker <dalias@aerifal.cx> [2012-06-08 10:55:19 -0400]:
> > On Fri, Jun 08, 2012 at 10:44:23AM -0400, Rich Felker wrote:
> > > Still working on finding whether this long double issue is what's
> > > causign the gnulib junk to get pulled in...
> > 
> > And it's not, at least not in the worst package, GNU m4. I #if'd out
> > the broken long double test in configure, re-ran it, and even with no
> > printf problems reported, freadahead is still getting pulled in...
> > I'll see if I see anything else odd.
> > 
> 
> i've just checked m4 and it uses freadseek and closein
> 
> both functions are from gnulib and depend on freadahead
> 
> so m4 will use freadahead independently of the printf issue

Thanks; I think that settles it then. I wonder if they'd accept a
patch upstream to fix this bug..

I've just fixed a few minor issues found by the autoconf/gnulib tests:

- strtod("-0x",0) returning positive zero when it should return
  negative zero.
- stdint.h applying wrong signedness in some of the const macros.

I've also found a couple more invalid tests in gnulib/autoconf:

- The getopt test for "POSIX compatible" getopt is backwards; it
  actually tests for POSIX-incompatible GNU semantics.
- isnanl is checking behavior on invalid ld80 representations, just
  like the printf test.

And one valid test that fails:

- The strtod test is attempting to convert "nan()". This is required
  to be accepted and yield a NAN, even though the interpretation of
  the contents of the () is implementation-defined. I'll add support
  for this later.

Rich

Re: printf POSIX compliance

[Reuben Thomas]

On 8 June 2012 16:29, Rich Felker <dalias@aerifal.cx> wrote:
>
> Thanks; I think that settles it then. I wonder if they'd accept a
> patch upstream to fix this bug..

The answer is very much "yes", in my experience (I'm not a gnulib
developer, but I have submitted patches on several occasions).
bug-gnulib@gnu.org!

-- 
http://rrt.sc3d.org

Re: printf POSIX compliance

[Rich Felker]

On Fri, Jun 08, 2012 at 04:43:45PM +0100, Reuben Thomas wrote:
> On 8 June 2012 16:29, Rich Felker <dalias@aerifal.cx> wrote:
> >
> > Thanks; I think that settles it then. I wonder if they'd accept a
> > patch upstream to fix this bug..
> 
> The answer is very much "yes", in my experience (I'm not a gnulib
> developer, but I have submitted patches on several occasions).
> bug-gnulib@gnu.org!

I was referring to m4, which seems largely unmaintained, not gnulib.
My view is that it's a bug to be directly using gnulib functions that
are not (and cannot be) portable and that presumably only exist for
the sake of implementing replacement functions on a finite known set
of broken systems for which the existing code suffices.

Of course it would be nice if gnulib could do something to prevent
this sort of situation from arising in the future too (and fix the
bugs I found in the tests and reported elsewhere in this thread), but
since gnulib is integrated into packages that use it, resolving even
those issues also requires getting the new fixed version deployed into
the affected packages.

Rich

Re: printf POSIX compliance

[Reuben Thomas]

On 8 June 2012 16:53, Rich Felker <dalias@aerifal.cx> wrote:
> On Fri, Jun 08, 2012 at 04:43:45PM +0100, Reuben Thomas wrote:
>> On 8 June 2012 16:29, Rich Felker <dalias@aerifal.cx> wrote:
>> >
>> > Thanks; I think that settles it then. I wonder if they'd accept a
>> > patch upstream to fix this bug..
>>
>> The answer is very much "yes", in my experience (I'm not a gnulib
>> developer, but I have submitted patches on several occasions).
>> bug-gnulib@gnu.org!
>
> I was referring to m4, which seems largely unmaintained, not gnulib.

The maintainers (Gary Vaughan and Eric Blake) are both active, so I
think it's worth it; Eric works on gnulib too.

You're right that of course for practical purposes it requires new
releases using the fixed version of gnulib, but many of these issues
(this one included) affect multiple packages, and also it's possible
to "repair" a broken package by refreshing gnulib before building.

-- 
http://rrt.sc3d.org

Re: printf POSIX compliance

[John Spencer]

On 06/08/2012 06:16 PM, Reuben Thomas wrote:
>
> The maintainers (Gary Vaughan and Eric Blake) are both active, so I
> think it's worth it; Eric works on gnulib too.
>
> You're right that of course for practical purposes it requires new
> releases using the fixed version of gnulib, but many of these issues
> (this one included) affect multiple packages, and also it's possible
> to "repair" a broken package by refreshing gnulib before building.
>
i was recently following a conversation in #gdb where GNU people (!) 
were trying to update gnulib and failed...
it took them about 5 hours to get everything fixed.
i don't think most users of gnulib want to go through this mess...

Re: printf POSIX compliance

[Reuben Thomas]

On 8 June 2012 17:38, John Spencer <maillist-musl@barfooze.de> wrote:
> i was recently following a conversation in #gdb where GNU people (!) were
> trying to update gnulib and failed...

That's odd. I have gnulib as a submodule of my project (GNU Zile), and
to update it I cd into the gnulib directory and do "git up" followed
by committing the new revision of the submodule. Usually takes me
rather less than 5 hours!

> i don't think most users of gnulib want to go through this mess...

I heartily agree. If it takes 5 hours to update you're doing something
wrong. Ask on the gnulib mailing list, there's usually someone happy
to help there.

-- 
http://rrt.sc3d.org

Re: printf POSIX compliance

[John Spencer]

On 06/08/2012 06:37 PM, Reuben Thomas wrote:
> On 8 June 2012 17:38, John Spencer<maillist-musl@barfooze.de>  wrote:
>> i was recently following a conversation in #gdb where GNU people (!) were
>> trying to update gnulib and failed...
> That's odd. I have gnulib as a submodule of my project (GNU Zile), and
> to update it I cd into the gnulib directory and do "git up" followed
> by committing the new revision of the submodule. Usually takes me
> rather less than 5 hours!
>
apparently it gets hard when you dont update it frequently.

here's the relevant extract of the discussion

Apr 19 17:43:04 <brobecke>      wow, that gnulib thing is a lot more 
effort than I thought. Gotta send big thanks to palves for doing it.
Apr 19 17:44:17 <tromey>        one reason we should maybe update gnulib 
more frequently is that if we wait too long, it is harder to do
Apr 19 17:44:36 <palves>        I had no idea when I started either :-P

[... 5 hours later ...]

Apr 19 22:45:35 <palves>        yay, gnulib patch in, I can finally 
start my week's plan :-D

Re: printf POSIX compliance

[Reuben Thomas]

On 9 June 2012 08:43, John Spencer <maillist-musl@barfooze.de> wrote:
>>
> apparently it gets hard when you dont update it frequently.
>
> here's the relevant extract of the discussion

I wonder if it's a question of adjusting to changes in gnulib since
its last update. This is a disadvantage of a source-based unversioned
library; on the other hand, you can choose when to update, rather than
worrying about what users have on their machines.

-- 
http://rrt.sc3d.org

Re: printf POSIX compliance

[John Spencer]

On 06/08/2012 05:06 PM, Szabolcs Nagy wrote:
>
> i've just checked m4 and it uses freadseek and closein
>
> both functions are from gnulib and depend on freadahead
>
> so m4 will use freadahead independently of the printf issue
>
closein.c reads:

"
Most programs can get by with close_stdout.  close_stdin is only
needed when a program wants to guarantee that partially read input
from seekable stdin is not consumed, for any subsequent clients.
For example, POSIX requires that these two commands behave alike:
(sed -ne 1q; cat) < file
tail -n 1 file
"

this is bogus, according to Rich:
"all files are closed when a process terminates normally/calls exit.
  if you want to report write failures, just fflush(stdout) before exit 
and check the return value"

to make this stuff happening, they manipulate the libc-internal FILE 
struct, which is even more bogus.

gnulib is simply a huge pile of junk...

Re: printf POSIX compliance

[Reuben Thomas]

On 8 June 2012 17:46, John Spencer <maillist-musl@barfooze.de> wrote:
>
> this is bogus, according to Rich:
> "all files are closed when a process terminates normally/calls exit.
>  if you want to report write failures, just fflush(stdout) before exit and
> check the return value"

Jim Meyering has an analysis of the problem here:

http://www.gnu.org/ghm/2011/paris/#sec-2-1

> gnulib is simply a huge pile of junk...

Like the stdout problem, it's not that simple! gnulib has many
enthusiastic users, myself included, because it solves many
portability problems and adds much useful functionality. Thanks to
gnulib I was able to remove a total of about 1,000 lines of code from
GNU Zile. I have not heard a single complaint from users, so I guess
no-one tried to build it with musl. However, I have had success
stories from users building on far-out platforms like DOS, and I've
had far fewer bug reports on platforms I don't use since I started
using gnulib.

So, please file bug reports rather than insults! gnulib has receptive
and active maintainers, and we'll all benefit much more from fixed
software than from merely venting frustration.

As regards the particular problem with freadahead, looking at the code
suggests a workaround of -DSLOW_BUT_NO_HACKS to avoid trying to build
the FILE-fiddling code.

-- 
http://rrt.sc3d.org

Re: printf POSIX compliance

[Rich Felker]

On Fri, Jun 08, 2012 at 05:46:10PM +0100, Reuben Thomas wrote:
> On 8 June 2012 17:46, John Spencer <maillist-musl@barfooze.de> wrote:
> >
> > this is bogus, according to Rich:
> > "all files are closed when a process terminates normally/calls exit.
> >  if you want to report write failures, just fflush(stdout) before exit and
> > check the return value"
> 
> Jim Meyering has an analysis of the problem here:
> 
> http://www.gnu.org/ghm/2011/paris/#sec-2-1

Thanks; I'll take a look.

> > gnulib is simply a huge pile of junk...
> 
> Like the stdout problem, it's not that simple! gnulib has many
> enthusiastic users, myself included, because it solves many
> portability problems and adds much useful functionality. Thanks to
> gnulib I was able to remove a total of about 1,000 lines of code from
> GNU Zile. I have not heard a single complaint from users, so I guess
> no-one tried to build it with musl. However, I have had success
> stories from users building on far-out platforms like DOS, and I've
> had far fewer bug reports on platforms I don't use since I started
> using gnulib.

I agree gnulib is very useful and successful for porting to obscure
and broken platforms by making them outwardly resemble a POSIX-like
platform. What I object to is the methodology of how replacements are
selected and the pervasive use of undefined behavior and poking at
implementation internals.

To clarify, if there's a block of ugly not-possibly-portable code
that's conditioned to only ever be compiled on known broken systems X,
Y, and Z, it's unfortunate but probably non-problematic. But if this
code is enabled on a possibly-infinite set of systems based on tests
for certain behaviors, you end up invoking undefined behavior on
systems for which you have not researched the results of that
undefined behavior, and the result could range from just not building
to serious security compromises.

> So, please file bug reports rather than insults!

I agree with this sentiment. That's why the quotation cited above was
not filed; it was dug up from a past discussion of frustration with
these kind of issues...

> gnulib has receptive
> and active maintainers, and we'll all benefit much more from fixed
> software than from merely venting frustration.

Agreed.

> As regards the particular problem with freadahead, looking at the code
> suggests a workaround of -DSLOW_BUT_NO_HACKS to avoid trying to build
> the FILE-fiddling code.

Someone building the package should not have to do this. The whole
purpose of configure is to detect the needs of the system you're
building for and get the build config right for you. The #error cases
should be removed and replaced with code that works on ANY system,
unless the relevant code will never be compiled at all except on a
known finite set of old broken systems.

Rich

Re: printf POSIX compliance

[Reuben Thomas]

On 8 June 2012 17:51, Rich Felker <dalias@aerifal.cx> wrote:
>> As regards the particular problem with freadahead, looking at the code
>> suggests a workaround of -DSLOW_BUT_NO_HACKS to avoid trying to build
>> the FILE-fiddling code.
>
> Someone building the package should not have to do this.

I agree. Again, I suspect that pointing out the problems you go on to
cite may influence change in gnulib. I suspect that, as you said
earlier, the maintainers of gnulib have been primarily concerned in
the past with porting to old, broken systems, not new ones which don't
so much need their help.

-- 
http://rrt.sc3d.org

Re: printf POSIX compliance

[Rich Felker]

On Fri, Jun 08, 2012 at 05:46:10PM +0100, Reuben Thomas wrote:
> On 8 June 2012 17:46, John Spencer <maillist-musl@barfooze.de> wrote:
> >
> > this is bogus, according to Rich:
> > "all files are closed when a process terminates normally/calls exit.
> >  if you want to report write failures, just fflush(stdout) before exit and
> > check the return value"
> 
> Jim Meyering has an analysis of the problem here:
> 
> http://www.gnu.org/ghm/2011/paris/#sec-2-1

He makes it a lot more difficult than it has to be. My preferred way
of handling the situation is never to close stdout at all, only to
flush it with fflush(stdout), then check ferror(stdout). This will
report any new or past write errors that weren't already cleared by
clearerr; the only errors it cannot report are error returns from the
underlying close(1) operation. Some people want to detect such errors,
in which case the following expression will reliably determine if any
error occurred:

ferror(stdout) || fclose(stdout)

With that said, I question the value of checking for errors in
close(). The only historical case where they have any consequence are
with broken NFS setups (NFS is broken beyond usability for countless
reasons, but this is getting off-topic...), where even the lack of an
error from close() cannot ensure consistency.

There's certainly not a need to do any of this with atexit functions
or other ugly hacks. Whatever error check you do, it belongs at the
point of normal program termination (or if you're handling multiple
output files, perhaps the point where you finish handling stdout).

Rich

Re: printf POSIX compliance

[Reuben Thomas]

On 8 June 2012 18:00, Rich Felker <dalias@aerifal.cx> wrote:
> On Fri, Jun 08, 2012 at 05:46:10PM +0100, Reuben Thomas wrote:
>>
>> Jim Meyering has an analysis of the problem here:
>>
>> http://www.gnu.org/ghm/2011/paris/#sec-2-1
>
> He makes it a lot more difficult than it has to be...

Possibly a discussion worth having with Jim? Most obviously on
bug-gnulib@gnu.org, as he's an active gnulib maintainer.

-- 
http://rrt.sc3d.org

Re: printf POSIX compliance

[Rich Felker]

On Fri, Jun 08, 2012 at 06:07:37PM +0100, Reuben Thomas wrote:
> On 8 June 2012 18:00, Rich Felker <dalias@aerifal.cx> wrote:
> > On Fri, Jun 08, 2012 at 05:46:10PM +0100, Reuben Thomas wrote:
> >>
> >> Jim Meyering has an analysis of the problem here:
> >>
> >> http://www.gnu.org/ghm/2011/paris/#sec-2-1
> >
> > He makes it a lot more difficult than it has to be...
> 
> Possibly a discussion worth having with Jim? Most obviously on
> bug-gnulib@gnu.org, as he's an active gnulib maintainer.

I'm not sure it's worth starting a debate. His slides made for an
interesting presentation, but in my opinion, the reason it got to be a
mess is that his choice to "factor out" the close operation into an
exit handler registered with atexit was a fundamentally bad design. It
creates additional global state (not to mention the fact that exiting
from an atexit handler has some major issues in itself!) and
unnaturally breaks up the use and checking of stdout status into
unrelated parts of the program, which then required adding ugly and
non-portable hacks to determine if closing stdout needs to be checked.

There's also the issue that if fd 1 did not exist when the program
started and got assigned to another file the program opened,
fclose(stdout) could wrongly close that fd; in the worst case
(especially with multi-threaded programs) this could then lead to
another file getting reassigned to the same fd, and code that's still
in the process of writing to the original one could clobber the wrong
file.

If he'd stuck to closing stdout and checking for errors in the main
program flow after the program is done using stdout, everything would
remain incredibly simple. By the way, note that my test
ferror(stdout)||fclose(stdout) avoids calling fclose if ferror returns
nonzero, so you don't clobber the existing errno value, but that's
fragile anyway since it's likely that you already clobbered errno
elsewhere.

Rich

Re: printf POSIX compliance

[Isaac Dunham]

On Fri, 8 Jun 2012 17:46:10 +0100
Reuben Thomas <rrt@sc3d.org> wrote:

> As regards the particular problem with freadahead, looking at the code
> suggests a workaround of -DSLOW_BUT_NO_HACKS to avoid trying to build
> the FILE-fiddling code.
Having looked at that code myself, I think there's some idiotic tests
going on:
#ifdef __OSNAME
..
#else if __system2__
..
#else if SLOW_BUT_NO_HACKS
//return 1
#else 
//build error
#endif

1. If I define SLOW_BUT_NO_HACKS, it should be the first test.
(if I run MINT and define this, assume I mean it!)
2. If it works with a stub, why do we get an error?

I'd suggest more-or-less this approach:
-#else if SLOW_BUT_NO_HACKS
+#else
//return 1
-#else
-//build error
+//warn "falling back to stub, please port"
#endif

Of course, I know this isn't the right place to discuss such
things--that would be for gnulib. 
Isaac Dunham

Re: printf POSIX compliance

[Rich Felker]

On Fri, Jun 08, 2012 at 07:33:57PM -0700, Isaac Dunham wrote:
> On Fri, 8 Jun 2012 17:46:10 +0100
> Reuben Thomas <rrt@sc3d.org> wrote:
> 
> > As regards the particular problem with freadahead, looking at the code
> > suggests a workaround of -DSLOW_BUT_NO_HACKS to avoid trying to build
> > the FILE-fiddling code.
> Having looked at that code myself, I think there's some idiotic tests
> going on:
> #ifdef __OSNAME
> ...
> #else if __system2__
> ...
> #else if SLOW_BUT_NO_HACKS
> //return 1
> #else 
> //build error
> #endif
> 
> 1. If I define SLOW_BUT_NO_HACKS, it should be the first test.
> (if I run MINT and define this, assume I mean it!)
> 2. If it works with a stub, why do we get an error?
> 
> I'd suggest more-or-less this approach:
> -#else if SLOW_BUT_NO_HACKS
> +#else
> //return 1
> -#else
> -//build error
> +//warn "falling back to stub, please port"
> #endif
> 
> Of course, I know this isn't the right place to discuss such
> things--that would be for gnulib. 

Actually I don't think it's very off-topic here. This is certainly a
major affected community that has an interest in solving the problem,
and Reuben seems familiar with gnulib and the developer community and
interested in helping us find solutions and get them integrated. What
is the next step we should take? Posting to the gnulib mailing list or
bug tracker, or contacting somebody directly?

Rich

Re: printf POSIX compliance

[Szabolcs Nagy]

* Rich Felker <dalias@aerifal.cx> [2012-06-08 22:45:56 -0400]:
> On Fri, Jun 08, 2012 at 07:33:57PM -0700, Isaac Dunham wrote:
> > Of course, I know this isn't the right place to discuss such
> > things--that would be for gnulib. 
> 
> Actually I don't think it's very off-topic here. This is certainly a
> major affected community that has an interest in solving the problem,
> and Reuben seems familiar with gnulib and the developer community and
> interested in helping us find solutions and get them integrated. What
> is the next step we should take? Posting to the gnulib mailing list or
> bug tracker, or contacting somebody directly?
> 

i don't like the gnulib approach to portability

1) it implements broken functions which simply must not
be used in an application with portability requirements
(eg alloca is implemented using assumptions about the
stack mechanism and invokes undefined behaviour)

2) instead of sharing code between platforms they use ifdefs
and depend on internals of the implementation
(eg. broken gnu stdio functions: gnulib will never work on
a new platform out of the box, you have to port it whenever
there is a new libc or even a new version of a libc)

3) they duplicate a lot of the libc writing effort
and provide bad quality code
(eg. math functions: a mathematical software will only be
portable if libm is high quality so if portability matters
you need a libm written in portable c code (and maybe even
softfloat emulation)

for non-mathematical software probably a simple naive
implementation is ok, but again it could be a separate
'libmdummy' written in clean ansi c and it could be used
everywhere, so the same code gets tested on all platforms

what gnulib gives you is mostly dummy functions with
inconsistent quality, randomly pulled in depending on the
target platform which means math code with gnulib can easily
explode whenever high accuracy, correct nan handling or fenv
support is required)


in my opinion the more correct approaches to portability are

1) have a high quality libc (like musl) and port it
to all the required targets (eg by emulating syscalls)

2) write the application with portability in mind: instead
of using posix api use a 'libportability' api that covers
enough areas so the application can work and simple enough
so it can be implemented on any target (so eg. it has no
alloca and freadahead in it, and only has limited form of
networking, process or file management etc)

these approaches are scalable because the platform specific
code is minimized
(the difficult part of the code is the same on all platforms)

Re: printf POSIX compliance

[Reuben Thomas]

On 9 June 2012 13:58, Szabolcs Nagy <nsz@port70.net> wrote:
>
> i don't like the gnulib approach to portability

Again, this is a message that's merely interesting on this list,
whereas on bug-gnulib it might be useful!

-- 
http://rrt.sc3d.org

Re: printf POSIX compliance

[Rich Felker]

On Sat, Jun 09, 2012 at 02:58:55PM +0200, Szabolcs Nagy wrote:
> i don't like the gnulib approach to portability
> 
> 1) it implements broken functions which simply must not
> be used in an application with portability requirements
> (eg alloca is implemented using assumptions about the
> stack mechanism and invokes undefined behaviour)

This is definitely a problem, perhaps the biggest problem with gnulib.
Since most programs that use gnulib almost require GCC anyway, I'm not
sure alloca is the best example, but it surely applies to things like
freadahead. Anyway this sort of junk is at best orthogonal to goal of
writing multi-platform applications.

> 2) instead of sharing code between platforms they use ifdefs
> and depend on internals of the implementation
> (eg. broken gnu stdio functions: gnulib will never work on
> a new platform out of the box, you have to port it whenever
> there is a new libc or even a new version of a libc)

Actually the legacy unices shared a surprising amount of stdio
internals, largely because of broken code like gnulib that poked at
the internals...

> 3) they duplicate a lot of the libc writing effort

Are you sure? I think most of the libc-like code is copied from (and
shared with) glibc...

> and provide bad quality code
> (eg. math functions: a mathematical software will only be
> portable if libm is high quality so if portability matters
> you need a libm written in portable c code (and maybe even
> softfloat emulation)

I haven't looked at their math functions, but if they really are that
broken, it brings up a major point you missed:

The gnulib configure tests are not just extremely pedantic in judging
a system as non-conformant, but _selectively_ pedantic. That is, they
test every single corner case that glibc went to the trouble to get
right, but ignore the (equally many) cases glibc got horrible wrong.
And sometimes they even test for non-conformant, glibc-specific
behavior...

How does this relate to math? Well, it seems they check for all sorts
of obscure breakage in the host libm, but then replace it with their
own broken math code...

> in my opinion the more correct approaches to portability are
> 
> 1) have a high quality libc (like musl) and port it
> to all the required targets (eg by emulating syscalls)

I don't think enforcing implementation homogeneity is a viable
solution for applications.

> 2) write the application with portability in mind: instead
> of using posix api use a 'libportability' api that covers
> enough areas so the application can work and simple enough
> so it can be implemented on any target (so eg. it has no
> alloca and freadahead in it, and only has limited form of
> networking, process or file management etc)

This approach is not just wrong but extremely harmful. It's the
approach of APR, NSPR, and basically any library whose name ends in
"PR" (portable runtime). I've not read the Apache code, but I've read
SVN's code and its use of APR, and the level of idiocy it involves is
astounding. The equivalent of a simple printf call ends up as this
giant chain of allocations and concatenations building strings to send
to the output system. I seriously doubt there's any way a program that
uses APR (Apache itself included) could avoid crashing under OOM
conditions.

Surely there's no fundamental reason it _has_ to be this way. After
all, someone making a portable system interface library could end up
making it just like POSIX. The problem is that the task attracts
people with the wrong traits: particularly, obsessiveness with
designing something absolutely general and no intuition for the actual
necessary usage cases.

More fundamentally (i.e. even if it didn't attract the wrong people)
the approach is wrong for the following two reasons:

1. It penalizes correct systems by putting an added layer of bloat
   between the application and the underlying correct POSIX
   implementation.

2. It makes it impossible to reuse code that was written to C99/POSIX
   in your application without "porting" the code to your new
   portability library, and conversely, impossible to reuse code
   written to your portability in projects that want to be
   light-weight and pure-C-only or POSIX-only (or else forces them to
   pull in an ugly library they don't want and that might have
   portability issues by virtue of not being ported to some systems
   they want/need to support).

Ultimately, gnulib does actually have the right basic approach; they
just botched a lot of the details. The correct approach from an
application side (where you can't just fix the whole broken OS) is to
write your code to standard C/POSIX assuming it all works (possibly
avoiding making unnecessary assumptions for interfaces that are "hard
to fix"), and include patch-up code for a known finite set of broken
systems you want to support. Today, that set would probably be just
"Windows, Android, and OSX/iOS".

This minimizes (to zero) the cost of portability on clean, non-broken
systems and avoids doing anything that could cause your program to
break on such systems. It also keeps the cost reasonably low (much
lower than the *PR solutions) even on broken systems. And it works
well in practice. This is not something I just made up; it's a policy
I pushed during the entire time I was working on MPlayer, which
usually took the form of "Keep the hacks to fix broken stuff as close
to the broken component as possible, and avoid polluting code that has
nothing to do with the broken stuff."

The main differences from gnulib are:

- Don't try to "fix" any system except the known-broken ones.
- Avoid unnecessary fixes whose costs are disproportionately large
  compared to their benefits.
- Don't add all sorts of unnecessary functions that have nothing to do
  with portability.

I don't expect to "revolutionize" gnulib, but since the right (in my
opinion) approach is reasonably similar to gnulib's basic idea, I
think we could influence the evolution of gnulib in a more positive
direction. Some things that might be nice to see...

- Clear distinction in gnulib between "broken libc replacement" stuff
  and "convenience utility" stuff where the latter is implemented
  portably without any #ifdeffery.
- Automatically-generated option to configure that inhibits all gnulib
  tests and replacement functions and forcibly assumes the underlying
  system is C and POSIX conformant.
- And of course cleanup/fixing of existing known bugs and portability
  issues (and thread-safety issues).

Rich

Re: printf POSIX compliance

[Reuben Thomas]

On 9 June 2012 22:11, Rich Felker <dalias@aerifal.cx> wrote:
[Much interesting discussion]

I think all of this discussion raises perhaps the most important
point, which is that many of the underlying motivations for gnulib and
musl is the same.

Personally, the best thing about gnulib for me is that it essentially
provides a sort of "libposix": I can write to standards, and gnulib
picks up the slack on real systems.
That's similar to musl's approach of "provide a working libc you can
use anywhere".

While I suspect that fundamentally the two projects will continue,
with good reason, to go their own way, it'd be really good to see
co-operation to the extent that is practical, so that for example more
software can build with either approach, and in particular, that the
two projects can happily co-exist, so that hackers like myself can
spend more time writing software to standards and less time worrying
about bugs in the code on which we rely, whether system code, or
efforts like gnulib and musl to fill in the gaps without the pain and
slowness of involving vendors (who of necessity must be relatively
conservative).

-- 
http://rrt.sc3d.org

Re: printf POSIX compliance

[Reuben Thomas]

On 9 June 2012 03:45, Rich Felker <dalias@aerifal.cx> wrote:
>
> Actually I don't think it's very off-topic here. This is certainly a
> major affected community that has an interest in solving the problem,
> and Reuben seems familiar with gnulib and the developer community and
> interested in helping us find solutions and get them integrated. What
> is the next step we should take? Posting to the gnulib mailing list or
> bug tracker, or contacting somebody directly?

Write to the mailing list (I believe there's no separate bug tracker):
bug-gnulib@gnu.org. If you Cc: me that'd be helpful (if you'd like me
there!), as I'm subscribed but usually disable delivery.

-- 
http://rrt.sc3d.org

Re: printf POSIX compliance

[Pedro Alves]

(Escuse me for breaking the threading, but I can't find the message's
ids from the mailing list archive.)

> Apr 19 17:43:04 <brobecke>      wow, that gnulib thing is a lot more
> effort than I thought. Gotta send big thanks to palves for doing it.
> Apr 19 17:44:17 <tromey>        one reason we should maybe update gnulib
> more frequently is that if we wait too long, it is harder to do
> Apr 19 17:44:36 <palves>        I had no idea when I started either :-P
>
> [... 5 hours later ...]
>
> Apr 19 22:45:35 <palves>        yay, gnulib patch in, I can finally
> start my week's plan :-D

Wow, way to pull things out of context to spread some silly FUD.  First, you have
no idea what I had been up to those 5 hours, let alone if I was parked in
front of the computer the whole time...  Then, the "patch" was really a
multitude of patches, involving fixes to issues in GDB and GDBserver (GDBserver was
not using gnulib before, and that was that prompted the gnulib update), with the
most time consuming bit being that the way that GDB imports gnulib was changed
dramatically, because we now have _two_ programs in the same tree that share
the _same_ gnulib import in a way no other program does, afaik.

  http://sourceware.org/ml/gdb-patches/2012-04/msg00426.html
  http://sourceware.org/ml/gdb-patches/2012-04/msg00556.html
  http://sourceware.org/ml/gdb-patches/2012-04/msg00558.html
  http://sourceware.org/ml/gdb-patches/2012-04/msg00608.html
  http://sourceware.org/ml/gdb-patches/2012-04/msg00609.html
  http://sourceware.org/ml/gdb-patches/2012-04/msg00618.html
  http://sourceware.org/ml/gdb-patches/2012-04/msg00620.html
  http://sourceware.org/ml/gdb-patches/2012-04/msg00622.html
  http://sourceware.org/ml/gdb-patches/2012-04/msg00627.html
  http://sourceware.org/ml/gdb-patches/2012-04/msg00631.html
  http://sourceware.org/ml/gdb-patches/2012-04/msg00637.html
  http://sourceware.org/ml/gdb-patches/2012-04/msg00646.html
  http://sourceware.org/ml/gdb-patches/2012-04/msg00692.html

As Reuben says, just updating gnulib is a trivial case of git
pull + running gnulib-tool.

-- 
Pedro Alves