MUSL Feature Detection

MUSL Feature Detection

[William Ahern]

I'm familiar with the policy that MUSL is not interested in adding version
macros, etc. I mostly agree with the policy, however there are some corner
cases where it's problematic.

I maintain a Lua bindings library to Unix APIs

	http://25thandclement.com/~william/projects/lunix.html

Unlike the most commonly used module, luaposix, my module tries to be as
thread-safe as possible. (It also supports "portable" but non-POSIX
interfaces.) That means using all the re-entrant versions of functions if
available. But some functions don't have re-entrant versions, such as
strsignal. In those cases I try to synthesize a safe version if possible and
practical, such as using sys_siglist if available.

Knowing whether an interface is thread-safe requires me inspecting the code
and basically documenting my findings inside the source code with
conditionals. I could _also_ do run-time checks, but that's more susceptible
to false negatives, and so I think it would only make sense to do that as a
back-stop. (Compile-time checks won't work for cross-compiling).

The vast majority of people don't pay attention to obscure thread-safety
issues, especially because the most popular platforms like Linux/glibc and
Solaris do a pretty decent job of implementing APIs in a thread-safe manner
(e.g. getenv, strsignal). This lack of interest is especially true in the
context of a library, and doubly so of a scripting module, where people just
assume the mode and the dependencies do the right thing, assuming they
bother worrying about it in the first place.

So you can see I'm stuck between a rock and a hard place. MUSL is probably
safe in most regards. It doesn't even do localization, so strsignal is safe.
But I have no easy way to _know_ that I'm building against MUSL without the
person compiling the module knowingly and explicitly changing the build
configuration.

So, is there any sort of sanctioned way to detect MUSL at all, version or no
version? Is there any interest in supporting any kind of feature detection,
such as an API that communicates implementation choices wrt unspecified and
undefined behavior.

Re: MUSL Feature Detection

[Rich Felker]

On Wed, Mar 04, 2015 at 12:54:58PM -0800, William Ahern wrote:
> I'm familiar with the policy that MUSL is not interested in adding version
> macros, etc. I mostly agree with the policy, however there are some corner
> cases where it's problematic.
> 
> I maintain a Lua bindings library to Unix APIs
> 
> 	http://25thandclement.com/~william/projects/lunix.html

Thanks for the background. I missed this in replying to your other
message thread.

> Unlike the most commonly used module, luaposix, my module tries to be as
> thread-safe as possible. (It also supports "portable" but non-POSIX
> interfaces.) That means using all the re-entrant versions of functions if
> available. But some functions don't have re-entrant versions, such as
> strsignal. In those cases I try to synthesize a safe version if possible and
> practical, such as using sys_siglist if available.

FYI, musl's strerror and strsignal are always thread-safe. They
return immutable strings which are valid for the entire lifetime of
the program.

> Knowing whether an interface is thread-safe requires me inspecting the code
> and basically documenting my findings inside the source code with
> conditionals.

Thread-safety for all standard functions is documented by the
standards. All functions defined in POSIX are thread-safe except those
in the table in XSH 2.9.1:

http://pubs.opengroup.org/onlinepubs/9699919799/functions/V2_chap02.html#tag_15_09_01

C11 also specifies which functions are exempt from the requirement not
to have data races, but there does not seem to be any official unified
table; you have to look locally at the documentation for each.

At some point in the future, musl's documentation will specify which
non-standard functions are guaranteed to be thread-safe and which
additional standard functions, beyond those required to be
thread-safe, are guaranteed to be thread-safe in musl. I'm hoping to
make progress on this documentation by the time of the 1.2.0 release.

> I could _also_ do run-time checks, but that's more susceptible
> to false negatives, and so I think it would only make sense to do that as a
> back-stop. (Compile-time checks won't work for cross-compiling).

I don't think run-time checks are practical to deduce thread-safety.
It could easily require months or years of hammering on an interface
to demonstrate it non-thread-safe, and similar time scales to achieve
reasonable certainty that a function is thread-safe.

> The vast majority of people don't pay attention to obscure thread-safety
> issues, especially because the most popular platforms like Linux/glibc and
> Solaris do a pretty decent job of implementing APIs in a thread-safe manner
> (e.g. getenv, strsignal). This lack of interest is especially true in the
> context of a library, and doubly so of a scripting module, where people just
> assume the mode and the dependencies do the right thing, assuming they
> bother worrying about it in the first place.
> 
> So you can see I'm stuck between a rock and a hard place. MUSL is probably
> safe in most regards. It doesn't even do localization, so strsignal is safe.

It does, and the localization is thread-safe. Both hard-coded English
message strings and the translated messages (it any) are immutable.

> But I have no easy way to _know_ that I'm building against MUSL without the
> person compiling the module knowingly and explicitly changing the build
> configuration.
> 
> So, is there any sort of sanctioned way to detect MUSL at all, version or no
> version? Is there any interest in supporting any kind of feature detection,
> such as an API that communicates implementation choices wrt unspecified and
> undefined behavior.

If we make any information available for programmatic use, I would
like to coordinate this with at least one other libc implementation
(which would probably be glibc, but BSDs are welcome to participate
too). That way it becomes something of a cross-platform convention
rather than our own local quirk.

Rich

Re: MUSL Feature Detection

[u-wsnj]

On Wed, Mar 04, 2015 at 12:54:58PM -0800, William Ahern wrote:
> So, is there any sort of sanctioned way to detect MUSL at all, version or no
> version? Is there any interest in supporting any kind of feature detection,
> such as an API that communicates implementation choices wrt unspecified and
> undefined behavior.

Any "feature-collection" flag like "this is a certain implementation of
libc" is extremely volatile, as the corresponding "feature collection"
varies with every minor release. There is no contract/promise of keeping
the implementation details, which is actually the supposed isolation
property of APIs.

You are not expected to rely on anything not specified in the API -
but this seems to be what you are asking for.

Thus there can be no reliable predefined macro. Nor can there be a proper
build time feature check (unless you assume that the build result is to
be run in the exact environment of the build). I can regrettably hardly
imagine a reliable runtime check for thread safety either.

If you really feel that a check for "musl" is meaningful, then let
the person building your library specify this explicitly. If the person
does not know, you can not do better than unreliably guess or otherwise
avoid such assumptions. The choice is yours.

Sorry if I sound negative, I have full respect for your practical needs
but I doubt that they can be met in the way you wish.

Rune

Re: MUSL Feature Detection

[u-wsnj]

On Thu, Mar 05, 2015 at 09:33:15AM +0100, u-wsnj@aetey.se wrote:
> On Wed, Mar 04, 2015 at 12:54:58PM -0800, William Ahern wrote:
> > So, is there any sort of sanctioned way to detect MUSL at all, version or no
> > version? Is there any interest in supporting any kind of feature detection,
> > such as an API that communicates implementation choices wrt unspecified and
> > undefined behavior.

Sorry for having made a too large citation.

To be clear, I commented only on the part:

> > So, is there any sort of sanctioned way to detect MUSL at all, version or no
> > version?

[skipping my former message]

As for your proposal

> > Is there any interest in supporting any kind of feature detection,
> > such as an API that communicates implementation choices wrt unspecified and
> > undefined behavior.

I did not mean to comment on this in the previous message.

It looks otherwise reasonable but amounts to a standardization effort
for a new API with exactly the details intentionally omitted by
the existing standards. This might be hard to accomplish.

Rune

Re: MUSL Feature Detection

[stephen Turner]

[-- Attachment #1: Type: text/plain, Size: 1818 bytes --]

On Mar 5, 2015 3:58 AM, <u-wsnj@aetey.se> wrote:
>
> On Thu, Mar 05, 2015 at 09:33:15AM +0100, u-wsnj@aetey.se wrote:
> > On Wed, Mar 04, 2015 at 12:54:58PM -0800, William Ahern wrote:
> > > So, is there any sort of sanctioned way to detect MUSL at all,
version or no
> > > version? Is there any interest in supporting any kind of feature
detection,
> > > such as an API that communicates implementation choices wrt
unspecified and
> > > undefined behavior.
>
> Sorry for having made a too large citation.
>
> To be clear, I commented only on the part:
>
> > > So, is there any sort of sanctioned way to detect MUSL at all,
version or no
> > > version?
>
> [skipping my former message]
>
> As for your proposal
>
> > > Is there any interest in supporting any kind of feature detection,
> > > such as an API that communicates implementation choices wrt
unspecified and
> > > undefined behavior.
>
> I did not mean to comment on this in the previous message.
>
> It looks otherwise reasonable but amounts to a standardization effort
> for a new API with exactly the details intentionally omitted by
> the existing standards. This might be hard to accomplish.
>
> Rune
>

Ok im going to show my programming ignorance here.

Isnt libc a set of functions built to use I guess they are abi calls? The
short almost command looking bits like segsrv? So then if someone wanted to
test libc they would just have to write a macro or something that calls the
functions individually and uses them in a small reversible if needed test
case to say " yea we got that feature " ? That would also be good for
verifying the libc isnt corrupted assuming there isnt already some test
case like it.

Wouldnt something like that be more helpful than a libc version and a
assumption its standard full featured and unmodified?

Thanks
Stephen

[-- Attachment #2: Type: text/html, Size: 2367 bytes --]

Re: MUSL Feature Detection

[William Ahern]

On Thu, Mar 05, 2015 at 10:34:22AM -0500, stephen Turner wrote:
> On Mar 5, 2015 3:58 AM, <u-wsnj@aetey.se> wrote:
> > On Thu, Mar 05, 2015 at 09:33:15AM +0100, u-wsnj@aetey.se wrote:
> > > On Wed, Mar 04, 2015 at 12:54:58PM -0800, William Ahern wrote:
<snip>
> > > > Is there any interest in supporting any kind of feature detection,
> > > > such as an API that communicates implementation choices wrt
> unspecified and
> > > > undefined behavior.
> >
> > I did not mean to comment on this in the previous message.
> >
> > It looks otherwise reasonable but amounts to a standardization effort
> > for a new API with exactly the details intentionally omitted by
> > the existing standards. This might be hard to accomplish.
> >
> > Rune
> >
> 
> Ok im going to show my programming ignorance here.
> 
> Isnt libc a set of functions built to use I guess they are abi calls? The
> short almost command looking bits like segsrv? So then if someone wanted to
> test libc they would just have to write a macro or something that calls the
> functions individually and uses them in a small reversible if needed test
> case to say " yea we got that feature " ? That would also be good for
> verifying the libc isnt corrupted assuming there isnt already some test
> case like it.
> 
> Wouldnt something like that be more helpful than a libc version and a
> assumption its standard full featured and unmodified?

I understand and basically agree with the arguments against defining a
version macro or symbol. I don't think anybody disagrees with the argument
that feature detection is the preferable route. It's just not sufficient.

The kind of feature detection I had in mind isn't simply whether or not a
specific API is provided. That can be accomplished pretty easily with
autoconf. I probably wasn't sufficiently clear on that in the beginning.

But some implementation-defined behaviors are impractical or impossible to
programmatically determine. Most obvious in this case is thread-safety.
Theoretically you could implement a test to determine whether shared buffers
are used, but knowing that doesn't 100% resolve the question, because there
could be other thread-safety issues. Or maybe the implementation conditions
shared buffer use on something else, like initializing locale support.

Implementations usually document the choices they make when a standard
interface provides them an option. But that is typically done in
human-readable documentation.

I've been trying to curate a small database (unstructured) of thread-safety
and other portability issues by analyzing libc source code and
documentation, but in the case of musl I can't query my database because I
can't know whether musl is being used, let alone the version. Except,
perhaps, by process of elimination, or inspecting the toolchain directly.

I suppose my next step should be to try to turn by collected information on
portability problems into a structured database of some sort. Then that
would be a starting point for discussing the possiblity of defining a libc
API that communicates features and semantics.

Re: MUSL Feature Detection

[Rich Felker]

On Thu, Mar 05, 2015 at 01:02:18PM -0800, William Ahern wrote:
> I understand and basically agree with the arguments against defining a
> version macro or symbol. I don't think anybody disagrees with the argument
> that feature detection is the preferable route. It's just not sufficient.
> 
> The kind of feature detection I had in mind isn't simply whether or not a
> specific API is provided. That can be accomplished pretty easily with
> autoconf. I probably wasn't sufficiently clear on that in the beginning.

My position is that detection is the right approach for presence of
interfaces but not for behaviors. For the latter it precludes
cross-compiling and also fails to deal with behavioral differences
that vary by run-time environment (e.g. kernel version, which affects
even static linking).

> But some implementation-defined behaviors are impractical or impossible to
> programmatically determine. Most obvious in this case is thread-safety.
> Theoretically you could implement a test to determine whether shared buffers
> are used, but knowing that doesn't 100% resolve the question, because there
> could be other thread-safety issues. Or maybe the implementation conditions
> shared buffer use on something else, like initializing locale support.

As an example where it doesn't work, you could always build a public
interface that _appears_ thread-safe on a non-thread-safe backend by
interning the results (thus never reusing buffers). This would be
fully thread-safe as long as ony the public interface is used
(assuming it has proper locking), but fails horribly if the backend is
externally callable without the lock being held.

As an aside, this exact issue is _why_ POSIX is so extreme in its
definition of non-thread-safe: non-thread-safe functions cannot be
called concurrently with ANY standard function, even unrelated
thread-safe ones, without invoking UB. This basically means their use
is completely forbidden in multi-threaded programs (since there is no
way to preclude concurrency short of spinning on an atomic).

As an example, as far as I can tell, POSIX allows implementing
getpwnam_r or localtime_r as a wrapper around getpwnam or localtime
that holds a lock while calling the underlying non-thread-safe
function and copying its result to a caller-provided buffer.

> Implementations usually document the choices they make when a standard
> interface provides them an option. But that is typically done in
> human-readable documentation.
> 
> I've been trying to curate a small database (unstructured) of thread-safety
> and other portability issues by analyzing libc source code and
> documentation, but in the case of musl I can't query my database because I
> can't know whether musl is being used, let alone the version. Except,
> perhaps, by process of elimination, or inspecting the toolchain directly.
> 
> I suppose my next step should be to try to turn by collected information on
> portability problems into a structured database of some sort. Then that
> would be a starting point for discussing the possiblity of defining a libc
> API that communicates features and semantics.

I think that sounds reasonable.

Rich