Why are stdin/stdout/stderr `FILE *const` in musl?

Why are stdin/stdout/stderr `FILE *const` in musl?

[CodingMarkus]

Hello everyone!

Just a quick question:
Why does musl define stdin, stdout, and stderr to be of type `FILE *const`? Neither the C standard, nor the POSIX standard require, recommend or even imply that it would be allowed that this is a `const` pointer. That’s why other C libraries define it as `FILE *` only because that matches the examples given by POSIX and that matches the description found in any ISO-C standard. Making them const only break compatibility with other C libraries, e.g. considered this code:

	void * getOutputPtr ( void ) {
		if (/* whatever */) {
			return &stdout;
		}
		return &stderr;
	}

This code is correct by C standard and it is correct by POSIX standard but it won’t compile with strict compiler settings when using musl as the compiler complains that the `const` is lost and there is no explicitly cast for that. 

	error: cannot initialize return object of type 'void *' with an rvalue of type 'FILE *const *'


The reason why there is no cast is because a C programmer doesn’t have to expect a const here and thus this code would usually not require a cast. I don’t know what the intention is behind making stdX const, it’s probably a good one, but if that makes your header files incompatible to both, C and POSIX standard, then it is a horrible change that only means trouble and will not improve anything for anyone.

Regards,
Markus

Re: Why are stdin/stdout/stderr `FILE *const` in musl?

[Jens Gustedt]

[-- Attachment #1: Type: text/plain, Size: 1994 bytes --]

Hello CodingMarkus,

On Fri, 2 Feb 2018 14:24:28 +0100 CodingMarkus
<CodingMarkus@hanauska.name> wrote:

> Hello everyone!
> 
> Just a quick question:
> Why does musl define stdin, stdout, and stderr to be of type `FILE
> *const`? Neither the C standard, nor the POSIX standard require,
> recommend or even imply that it would be allowed that this is a
> `const` pointer. That’s why other C libraries define it as `FILE *`
> only because that matches the examples given by POSIX and that
> matches the description found in any ISO-C standard.

I don't think so. The C standard says

 > stderr
 > stdin
 > stdout
 > which are expressions of type "pointer to FILE" that point to the FILE objects associated, respectively,

which clearly states that these are "expressions" not lvalues, and so
your code

> 	void * getOutputPtr ( void ) {
> 		if (/* whatever */) {
> 			return &stdout;
> 		}
> 		return &stderr;
> 	}

is not conforming. In is undefined (by omission of defining it) to
take the address of such an expression.

(This is different from errno, e.g, which is defined as a modifiable
lvalue.)

> and it is correct by POSIX
> standard but it won’t compile with strict compiler settings when
> using musl as the compiler complains that the `const` is lost and
> there is no explicitly cast for that. 

> 	error: cannot initialize return object of type 'void *' with
> an rvalue of type 'FILE *const *'

and this is a feature, because it points to where your program has
undefined behavior.

The C standard has freopen for the cases that you want to change the
association or the access mode of one of the standard streams.

Jens

-- 
:: INRIA Nancy Grand Est ::: Camus ::::::: ICube/ICPS :::
:: ::::::::::::::: office Strasbourg : +33 368854536   ::
:: :::::::::::::::::::::: gsm France : +33 651400183   ::
:: ::::::::::::::: gsm international : +49 15737185122 ::
:: http://icube-icps.unistra.fr/index.php/Jens_Gustedt ::

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Why are stdin/stdout/stderr `FILE *const` in musl?

[Alexander Monakov]

On Fri, 2 Feb 2018, Jens Gustedt wrote:
>  > stderr
>  > stdin
>  > stdout
>  > which are expressions of type "pointer to FILE" that point to the FILE objects associated, respectively,
> 
> which clearly states that these are "expressions" not lvalues, and so
> your code

Note that musl's stdio.h can prevent abuse a bit harder by doing

#define stdin (stdin+0)

(making the expansion an rvalue and &stdin diagnosed with an error)

Alexander

Re: Why are stdin/stdout/stderr `FILE *const` in musl?

[Markus Wichmann]

On Fri, Feb 02, 2018 at 02:24:28PM +0100, CodingMarkus wrote:
> 	void * getOutputPtr ( void ) {
> 		if (/* whatever */) {
> 			return &stdout;
> 		}
> 		return &stderr;
> 	}
> 

Why would you ever need a pointer to stdout or stderr? They are already
pointers to FILE. The right way to do this is

FILE *getOutputPtr (void) {
    return intofile? fopen("log.txt", "w") : whatever? stdout: stderr;
}

Ciao,
Markus

Re: Why are stdin/stdout/stderr `FILE *const` in musl?

[CodingMarkus]

> On 2018-02-02, at 16:01, Markus Wichmann <nullplan@gmx.net> wrote:
> 
> Why would you ever need a pointer to stdout or stderr?


"Why would you ever need" is no valid argument. It’s no argument for anything and especially never an argument against anything. 

If I buy a toolset to paint my wall and I cannot use it to paint it green, then “why would you ever want to have a green wall” is simply no argument because it doesn’t matter why the person who bought the toolset wants a green wall or whether it’s a good idea to paint walls in green. This is a meta discusion that misses the actual point which is that the toolset won’t allow me to paint my wall green.

Here is a real life code sample that breaks exactly because the reason I was pointing out, so apparently clang developers like green walls:

https://git.alpinelinux.org/cgit/aports/tree/main/llvm5/dynamiclibrary-fix-build-musl.patch?id=HEAD

And that the people who themselves makes the currently second most successful open source compiler on the market act outside the C standard doesn’t sound very convincing to me. If in doubt, these people know the C standard better than I probably ever will.

And, of course, this affects other projects based on LLVM infrastructure, e.g. this one, which simply cannot be built on Alpine and that’s because of musl:

https://github.com/avast-tl/retdec

Even with that patch it cannot be built but that has other reasons and this time the problem is not a question related to any specs or standards but to the fact that LLVM expects functions like fseeko64 to either not be present (they don’t have to, they are non-standard) or to be present as real functions. In musl they are present but they are defines (tfeeko64 is a define to fseek and so on) and this is unexpected but I see no reason why it would not be allowed so here I can blame LLVM. There is also a fix for that BTW and the latest LLVM versions contain that fix already (but retdec bases on an older one).

I’m only worried with how interchangeable musl is as a standard libc because the idea of a standard is that it guarantees compatibility. If there are ten libc libraries and they all conform to the same standard, then they should always be interchangeable and all code building with one of them should also build with the other nine. Every time this is not the case, there is a problem that needs to be fixed IMHO. In that case either nine of them are doing it right and one of them is doing it wrong or nine of them are doing it wrong and one of them is doing it right, well, guess what is more likely.

So in the end, the question is whether LLVM is using incorrect code here that simply doesn’t need to compile because it is broken (regardless if it’s syntactically correct) or whether musl should define stdX the same way all the other libraries are doing it.

Regards,
Markus

Re: Why are stdin/stdout/stderr `FILE *const` in musl?

[Jon Chesterfield]

[-- Attachment #1: Type: text/plain, Size: 3202 bytes --]

I think compiler engineers are among those most likely to ignore parts of
the standard they don't agree with. We're not a good choice for an appeal
to authority in this instance.

On 2 Feb 2018 16:30, "CodingMarkus" <CodingMarkus@hanauska.name> wrote:



> On 2018-02-02, at 16:01, Markus Wichmann <nullplan@gmx.net> wrote:
>
> Why would you ever need a pointer to stdout or stderr?


"Why would you ever need" is no valid argument. It’s no argument for
anything and especially never an argument against anything.

If I buy a toolset to paint my wall and I cannot use it to paint it green,
then “why would you ever want to have a green wall” is simply no argument
because it doesn’t matter why the person who bought the toolset wants a
green wall or whether it’s a good idea to paint walls in green. This is a
meta discusion that misses the actual point which is that the toolset won’t
allow me to paint my wall green.

Here is a real life code sample that breaks exactly because the reason I
was pointing out, so apparently clang developers like green walls:

https://git.alpinelinux.org/cgit/aports/tree/main/llvm5/
dynamiclibrary-fix-build-musl.patch?id=HEAD

And that the people who themselves makes the currently second most
successful open source compiler on the market act outside the C standard
doesn’t sound very convincing to me. If in doubt, these people know the C
standard better than I probably ever will.

And, of course, this affects other projects based on LLVM infrastructure,
e.g. this one, which simply cannot be built on Alpine and that’s because of
musl:

https://github.com/avast-tl/retdec

Even with that patch it cannot be built but that has other reasons and this
time the problem is not a question related to any specs or standards but to
the fact that LLVM expects functions like fseeko64 to either not be present
(they don’t have to, they are non-standard) or to be present as real
functions. In musl they are present but they are defines (tfeeko64 is a
define to fseek and so on) and this is unexpected but I see no reason why
it would not be allowed so here I can blame LLVM. There is also a fix for
that BTW and the latest LLVM versions contain that fix already (but retdec
bases on an older one).

I’m only worried with how interchangeable musl is as a standard libc
because the idea of a standard is that it guarantees compatibility. If
there are ten libc libraries and they all conform to the same standard,
then they should always be interchangeable and all code building with one
of them should also build with the other nine. Every time this is not the
case, there is a problem that needs to be fixed IMHO. In that case either
nine of them are doing it right and one of them is doing it wrong or nine
of them are doing it wrong and one of them is doing it right, well, guess
what is more likely.

So in the end, the question is whether LLVM is using incorrect code here
that simply doesn’t need to compile because it is broken (regardless if
it’s syntactically correct) or whether musl should define stdX the same way
all the other libraries are doing it.

Regards,
Markus

[-- Attachment #2: Type: text/html, Size: 3976 bytes --]

Re: Why are stdin/stdout/stderr `FILE *const` in musl?

[Jens Gustedt]

[-- Attachment #1: Type: text/plain, Size: 2508 bytes --]

Hello,

On Fri, 2 Feb 2018 16:30:03 +0100 CodingMarkus
<CodingMarkus@hanauska.name> wrote:

> > On 2018-02-02, at 16:01, Markus Wichmann <nullplan@gmx.net> wrote:
> > 
> > Why would you ever need a pointer to stdout or stderr?  
> 
> 
> "Why would you ever need" is no valid argument. It’s no argument for
> anything and especially never an argument against anything.

No, but the exact definition that I gave you from the standard, is
certainly an argument. Did you see this? I didn't see a reply from you
to this.

I checked the provision that this is an expression is there at least
since C99.

> https://git.alpinelinux.org/cgit/aports/tree/main/llvm5/dynamiclibrary-fix-build-musl.patch?id=HEAD

What this shows is that they are clearly wrong in doing

EXPLICIT_SYMBOL(stderr);

stderr is *not* an explicit symbol and should not be.

> And that the people who themselves makes the currently second most
> successful open source compiler on the market act outside the C
> standard doesn’t sound very convincing to me. If in doubt, these
> people know the C standard better than I probably ever will.

Some of them certainly do.

I hope that you are not suggesting that the people that replied to you
here on that list don't know the C standard at least as good.

> I’m only worried with how interchangeable musl is as a standard libc
> because the idea of a standard is that it guarantees compatibility.

exactly, so what they are doing is to be frowned upon

(In particular, because this is a compiler framework that is making
assumptions about the library framework that is not justified.)

> If there are ten libc libraries and they all conform to the same
> standard,

As said in my other mail, this is undefined behavior LaLaLand. So
implementors can do what they want, they don't even have to document
it. But it is just not portable.

Now, if you are claiming that this would be an important feature, one
could certainly at least discuss it. But what I heard so far from you
was only reference to authority (llvm must know what they are doing).
You were not arguing why they are doing this and why this would be
important.

Jens

-- 
:: INRIA Nancy Grand Est ::: Camus ::::::: ICube/ICPS :::
:: ::::::::::::::: office Strasbourg : +33 368854536   ::
:: :::::::::::::::::::::: gsm France : +33 651400183   ::
:: ::::::::::::::: gsm international : +49 15737185122 ::
:: http://icube-icps.unistra.fr/index.php/Jens_Gustedt ::

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Why are stdin/stdout/stderr `FILE *const` in musl?

[Jeff Hammond]

[-- Attachment #1: Type: text/plain, Size: 3293 bytes --]

On Fri, Feb 2, 2018 at 7:30 AM, CodingMarkus <CodingMarkus@hanauska.name>
wrote:


> Here is a real life code sample that breaks exactly because the reason I
> was pointing out, so apparently clang developers like green walls:
>
> https://git.alpinelinux.org/cgit/aports/tree/main/llvm5/dyna
> miclibrary-fix-build-musl.patch?id=HEAD
>
> And that the people who themselves makes the currently second most
> successful open source compiler on the market act outside the C standard
> doesn’t sound very convincing to me. If in doubt, these people know the C
> standard better than I probably ever will.
>

I seriously doubt that they would describe themselves as infallible.  In
this specific case, the patch was submitted by a JuliaLang developer who is
an expert in LLVM but probably not ISO C.

The appropriate action here is not to assume they know everything but to
file a bug report.


> And, of course, this affects other projects based on LLVM infrastructure,
> e.g. this one, which simply cannot be built on Alpine and that’s because of
> musl:
>
> https://github.com/avast-tl/retdec
>
> Even with that patch it cannot be built but that has other reasons and
> this time the problem is not a question related to any specs or standards
> but to the fact that LLVM expects functions like fseeko64 to either not be
> present (they don’t have to, they are non-standard) or to be present as
> real functions. In musl they are present but they are defines (tfeeko64 is
> a define to fseek and so on) and this is unexpected but I see no reason why
> it would not be allowed so here I can blame LLVM. There is also a fix for
> that BTW and the latest LLVM versions contain that fix already (but retdec
> bases on an older one).
>
> I’m only worried with how interchangeable musl is as a standard libc
> because the idea of a standard is that it guarantees compatibility. If
> there are ten libc libraries and they all conform to the same standard,
> then they should always be interchangeable and all code building with one
> of them should also build with the other nine. Every time this is not the
> case, there is a problem that needs to be fixed IMHO. In that case either
> nine of them are doing it right and one of them is doing it wrong or nine
> of them are doing it wrong and one of them is doing it right, well, guess
> what is more likely.
>

Programming standards and the associated expectation of compatibility are
aspirational.

Your argument that 9/10 must be right is a variant of "tyranny of the
majority".  This is not how standards compliance works.  A compliant
implementation is compliant, even if there are a million non-compliant
implementations.

So in the end, the question is whether LLVM is using incorrect code here
> that simply doesn’t need to compile because it is broken (regardless if
> it’s syntactically correct) or whether musl should define stdX the same way
> all the other libraries are doing it.
>

LLVM is using incorrect code that is broken and needs to be fixed if the
LLVM developers want their implementation to be portable to faithful
implementations of the C/POSIX library standard.

Jeff

-- 
Jeff Hammond
jeff.science@gmail.com
http://jeffhammond.github.io/

[-- Attachment #2: Type: text/html, Size: 4411 bytes --]

Re: Why are stdin/stdout/stderr `FILE *const` in musl?

[William Pitcock]

Hi,

On Fri, Feb 2, 2018 at 9:30 AM, CodingMarkus <CodingMarkus@hanauska.name> wrote:
>> On 2018-02-02, at 16:01, Markus Wichmann <nullplan@gmx.net> wrote:
>>
>> Why would you ever need a pointer to stdout or stderr?

What he wants is a pointer to the pointer itself.

> "Why would you ever need" is no valid argument. It’s no argument for anything and especially never an argument against anything.
>
> If I buy a toolset to paint my wall and I cannot use it to paint it green, then “why would you ever want to have a green wall” is simply no argument because it doesn’t matter why the person who bought the toolset wants a green wall or whether it’s a good idea to paint walls in green. This is a meta discusion that misses the actual point which is that the toolset won’t allow me to paint my wall green.
>
> Here is a real life code sample that breaks exactly because the reason I was pointing out, so apparently clang developers like green walls:
>
> https://git.alpinelinux.org/cgit/aports/tree/main/llvm5/dynamiclibrary-fix-build-musl.patch?id=HEAD
>
> And that the people who themselves makes the currently second most successful open source compiler on the market act outside the C standard doesn’t sound very convincing to me. If in doubt, these people know the C standard better than I probably ever will.

Actually, that patch should not be necessary soon, thankfully.
Also stdin/stdout/stderr and many other objects provided by the libc
should never ever be considered explicit symbols.  Only ISO C
functions should be considered explicit symbols.  But overall, the
EXPLICIT_SYMBOL() stuff in LLVM is a massive hackjob and needs to just
be completely removed.

> And, of course, this affects other projects based on LLVM infrastructure, e.g. this one, which simply cannot be built on Alpine and that’s because of musl:
>
> https://github.com/avast-tl/retdec
>
> Even with that patch it cannot be built but that has other reasons and this time the problem is not a question related to any specs or standards but to the fact that LLVM expects functions like fseeko64 to either not be present (they don’t have to, they are non-standard) or to be present as real functions. In musl they are present but they are defines (tfeeko64 is a define to fseek and so on) and this is unexpected but I see no reason why it would not be allowed so here I can blame LLVM. There is also a fix for that BTW and the latest LLVM versions contain that fix already (but retdec bases on an older one).
>
> I’m only worried with how interchangeable musl is as a standard libc because the idea of a standard is that it guarantees compatibility. If there are ten libc libraries and they all conform to the same standard, then they should always be interchangeable and all code building with one of them should also build with the other nine. Every time this is not the case, there is a problem that needs to be fixed IMHO. In that case either nine of them are doing it right and one of them is doing it wrong or nine of them are doing it wrong and one of them is doing it right, well, guess what is more likely.

A lot of libcs are derived in part from the historical BSD libc
(including the GNU one etc).  Thusly, it is possible that these libc
have a design that is informed by the state of the 1980s instead of
modern practice.  It does not mean that any of the implementations are
doing it wrong.

> So in the end, the question is whether LLVM is using incorrect code here that simply doesn’t need to compile because it is broken (regardless if it’s syntactically correct) or whether musl should define stdX the same way all the other libraries are doing it.

It shouldn't, there are advantages to the current way it is done.
Anything that catches technically incorrect code is a major win from
security and overall robustness points of view.

William

Re: Why are stdin/stdout/stderr `FILE *const` in musl?

[Rich Felker]

On Fri, Feb 02, 2018 at 02:24:28PM +0100, CodingMarkus wrote:
> Hello everyone!
> 
> Just a quick question:
> Why does musl define stdin, stdout, and stderr to be of type `FILE
> *const`? Neither the C standard, nor the POSIX standard require,

They don't require it, but they allow it. musl's implementation _does_
require it, because behind the macros (all the C standard requires to
exist) are const-qualified objects (living inside libc.so/libc.a), and
declaring them with a type that mismatches their definitions would
result in undefined behavior.

> recommend or even imply that it would be allowed that this is a
> `const` pointer. That’s why other C libraries define it as `FILE *`
> only because that matches the examples given by POSIX and that
> matches the description found in any ISO-C standard. Making them
> const only break compatibility with other C libraries, e.g.
> considered this code:
> 
> 	void * getOutputPtr ( void ) {
> 		if (/* whatever */) {
> 			return &stdout;
> 		}
> 		return &stderr;
> 	}
> 
> This code is correct by C standard and it is correct by POSIX

It's not. As others have noted, stderr is not specified as an object;
it's a macro that expands to an expression with type FILE *. You
cannot take the address of an expression in general.

Rich