Detecting musl at compile and/or configure time

Detecting musl at compile and/or configure time

[Chris Hall]

Is there a supported way of detecting that one is building against the 
musl libc ?

At compile time I can, of course, detect glibc because it sets __GLIBC__ 
etc.  For macOS I can see _DARWIN_FEATURE_xxx macros.  The BSDs mostly 
give themselves away.  But for musl I have not found anything :-(

Chris

Re: Detecting musl at compile and/or configure time

[A. Wilcox]

[-- Attachment #1.1: Type: text/plain, Size: 1249 bytes --]

On 06/29/19 06:48, Chris Hall wrote:
> 
> Is there a supported way of detecting that one is building against the
> musl libc ?
> 
> At compile time I can, of course, detect glibc because it sets __GLIBC__
> etc.  For macOS I can see _DARWIN_FEATURE_xxx macros.  The BSDs mostly
> give themselves away.  But for musl I have not found anything :-(
> 
> Chris


The musl libc specifically does not have a FTM (feature test macro)
because it aims to be an exact implementation of C11 / POSIX and
therefore it has no "quirks" to detect.

If there are bugs in musl libc, the authors prefer if you report them
and have them fixed, instead of making your software look for __MUSL__
and then assume all musl versions will always have that bug.

See here for more information:

https://wiki.musl-libc.org/faq.html#Q:-Why-is-there-no-%3Ccode%3E__MUSL__%3C/code%3E-macro?

https://www.openwall.com/lists/musl/2013/03/29/13

Is there a reason you wish to detect musl at configure/compile time?
Perhaps if we knew the reason, we could help you make your code more
portable and/or not require such detection.

Best to you and yours,
--arw


-- 
A. Wilcox (awilfox)
Project Lead, Adélie Linux
https://www.adelielinux.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Detecting musl at compile and/or configure time

[Szabolcs Nagy]

* A. Wilcox <awilfox@adelielinux.org> [2019-06-29 08:27:50 -0500]:
> On 06/29/19 06:48, Chris Hall wrote:
> > 
> > Is there a supported way of detecting that one is building against the
> > musl libc ?
> > 
> > At compile time I can, of course, detect glibc because it sets __GLIBC__
> > etc.  For macOS I can see _DARWIN_FEATURE_xxx macros.  The BSDs mostly
> > give themselves away.  But for musl I have not found anything :-(
> > 
> > Chris
> 
> 
> The musl libc specifically does not have a FTM (feature test macro)
> because it aims to be an exact implementation of C11 / POSIX and
> therefore it has no "quirks" to detect.

that's not entirely true.

there are some standard macros in posix to test optional features.

and musl implements various extensions to the standard.

note that __GLIBC__ does not detect glibc: uclibc defines it too.

you really want to configure check for particular features
instead of detecting the libc (which is a moving target,
ppl can patch it or change it over time)

> 
> If there are bugs in musl libc, the authors prefer if you report them
> and have them fixed, instead of making your software look for __MUSL__
> and then assume all musl versions will always have that bug.
> 
> See here for more information:
> 
> https://wiki.musl-libc.org/faq.html#Q:-Why-is-there-no-%3Ccode%3E__MUSL__%3C/code%3E-macro?
> 
> https://www.openwall.com/lists/musl/2013/03/29/13
> 
> Is there a reason you wish to detect musl at configure/compile time?
> Perhaps if we knew the reason, we could help you make your code more
> portable and/or not require such detection.
> 
> Best to you and yours,
> --arw
> 
> 
> -- 
> A. Wilcox (awilfox)
> Project Lead, Adélie Linux
> https://www.adelielinux.org
> 

Re: Detecting musl at compile and/or configure time

[Chris Hall]

On 29/06/2019 14:58, Szabolcs Nagy wrote:
>> On 06/29/19 06:48, Chris Hall wrote:
>>>
>>> Is there a supported way of detecting that one is building against the
>>> musl libc ?
>>>
>>> At compile time I can, of course, detect glibc because it sets __GLIBC__
>>> etc.  For macOS I can see _DARWIN_FEATURE_xxx macros.  The BSDs mostly
>>> give themselves away.  But for musl I have not found anything :-(
...
> note that __GLIBC__ does not detect glibc: uclibc defines it too.

Happily it has the courtesy to also:

   #define __UCLIBC__ 1

> you really want to configure check for particular features
> instead of detecting the libc (which is a moving target,
> ppl can patch it or change it over time)

In the limit, yes.  I'm interested in how far one can get without the 
need for configure/cmake/etc. -- accepting that means being quite 
conservative in the choice of features to use (and assuming that the 
target system is reasonably up to date).

Of course, for major issues -- eg epoll vs kqueue or netlink vs routeing 
sockets -- it's the underlying OS that really matters.  Detecting the OS 
at compile time seems reasonably effective (also the compiler and the 
processor).  Detecting the libc feels like it should be a small step to 
complete the picture.

Chris

malloc() alignment and max_align_t

[Chris Hall]

The C11 Standard says that malloc() will return a pointer which "is 
suitably aligned so that it may be assigned to a pointer to any type of 
object with a fundamental alignment requirement", and it says that 
_Alignof(max_align_t) is equal to the "fundamental alignment".

I note that:

  * musl malloc() works to an alignment SIZE_ALIGN, which is
    #defined'd to 4*sizeof(size_t) == 32 for 64-bit processors.

  * musl defines max_align_t:

      typedef struct { long long __ll;
                       long double __ld; } max_align_t;

    (same like other libc, FWIW).

    _Alignof(max_align_t) == 16 for x86_64 (at least),
    because that's the ABI alignment for long double.

Clearly, C11 does not require malloc() to align exactly as max_align_t, 
and bigger is fine.

But I'm curious as to why SIZE_ALIGN is twice as big as it needs to be ?

Thanks,

Chris

Re: malloc() alignment and max_align_t

[Markus Wichmann]

On Sun, Jul 07, 2019 at 07:17:48PM +0100, Chris Hall wrote:
> Clearly, C11 does not require malloc() to align exactly as max_align_t, and
> bigger is fine.
>
> But I'm curious as to why SIZE_ALIGN is twice as big as it needs to be ?
>

Design decision. For the algorithm to work, every chunk needs to be able
to contain four machine words (one struct chunk). And with a bit of
maneuvering, this alignment can just be achieved on all chunks with
minimal waste (if a completely new region is allocated in expand_heap(),
then two machine words at the start of it are wasted, but if the new
memory happens to be directly behind the previous section, no waste
occurs at all).

This makes it easier to reason about chunk sizes. Right now, SIZE_ALIGN
and minimum chunk size (after adding OVERHEAD) are the same. If we
lowered the alignment to two machine words, that would change.

> Thanks,
>
> Chris

Ciao,
Markus

Re: malloc() alignment and max_align_t

[Rich Felker]

On Sun, Jul 07, 2019 at 09:22:00PM +0200, Markus Wichmann wrote:
> On Sun, Jul 07, 2019 at 07:17:48PM +0100, Chris Hall wrote:
> > Clearly, C11 does not require malloc() to align exactly as max_align_t, and
> > bigger is fine.
> >
> > But I'm curious as to why SIZE_ALIGN is twice as big as it needs to be ?
> >
> 
> Design decision. For the algorithm to work, every chunk needs to be able
> to contain four machine words (one struct chunk). And with a bit of
> maneuvering, this alignment can just be achieved on all chunks with
> minimal waste (if a completely new region is allocated in expand_heap(),
> then two machine words at the start of it are wasted, but if the new
> memory happens to be directly behind the previous section, no waste
> occurs at all).
> 
> This makes it easier to reason about chunk sizes. Right now, SIZE_ALIGN
> and minimum chunk size (after adding OVERHEAD) are the same. If we
> lowered the alignment to two machine words, that would change.

Indeed, we could do 16- instead of 32-byte alignment on 64-bit, but
then we'd have to preclude the smallest size (16) because it has 0
space after header/footer are accounted for. This is problematic
because splitting can lead to a smallest-size chunk without special
logic to preclude that, and possibly for other reasons.

Note that the header and footer could be reduced to uint32_t rather
than size_t; there's no need for huge sizes in this field for managed
heap chunks, and mmap-serviced allocations could use a different
encoding. And the linked list pointers could be an xor list, dropping
the minimum size to 4+4=8==16 on 64-bit and 4+4+4==12 on 32-bit.
However the malloc implementation is slated for replacement, so tuning
stuff like this is not a very productive activity right now.

Rich

Re: Detecting musl at compile and/or configure time

[Florian Weimer]

* Samuel Holland:

> The Linux UAPI is stable, so you can use whatever features you want as long as
> you gracefully handle ENOSYS/EINVAL (or document a minimum supported version).

Note that this only applies to the numbered syscall interface.  For
example, it is not true for vsyscall (which is now considered optional,
despite part of the userspace ABI at first, and really hard to detect
properly).  Netlink packet layout also keeps changing for some
subsystems.

Thanks,
Florian

Re: Detecting musl at compile and/or configure time

[Jeffrey Walton]

On Sun, Jun 30, 2019 at 8:29 AM Samuel Holland <samuel@sholland.org> wrote:
>
> On 6/30/19 7:03 AM, Chris Hall wrote:
> > I have a little build system which tries to detect the "usual suspects"
> > automatically, even without a full configure/cmake/etc. step.  If the detection
> > process fails, it generates a warning and the user must (at least) add a '-Dxxx'
> > to suppress that.
> >
> > I confess I have only recently stumbled across musl.  Perhaps systems which
> > default to musl are so rare that I can, as a practical matter, ignore them ?
> > The question then is whether to add a '-DqLIB_MUSL' gizmo to my build stuff --
> > so that "musl-gcc -DqLIB_MUSL" will do the trick.
>
> The fundamental problem with that is that you're not really checking for musl.
> You're checking for "the supported version (and options) of POSIX", "the
> supported version of the Linux UAPI", and "the set of extensions currently
> supported by musl", which all change over time. And some of them depend on who's
> distributing musl, as they may add their own customizations.

Or perhaps crafting workarounds for a buggy versions of musl. For
example, https://www.openwall.com/lists/musl/2019/06/26/10 .

Jeff

Re: Detecting musl at compile and/or configure time

[Chris Hall]

On 30/06/2019 13:28, Samuel Holland wrote:
> On 6/30/19 7:03 AM, Chris Hall wrote:
>> I have a little build system which tries to detect the "usual suspects"
>> automatically, even without a full configure/cmake/etc. step.  If the detection
>> process fails, it generates a warning and the user must (at least) add a '-Dxxx'
>> to suppress that.
...
> For POSIX, you should declare which version of the standard you follow with
> _POSIX_C_SOURCE, which you can verify is supported with _POSIX_VERSION, or for
> optional parts, the other macros in unistd.h[1].
> 
> The Linux UAPI is stable, so you can use whatever features you want as long as
> you gracefully handle ENOSYS/EINVAL (or document a minimum supported version).

My experience with Linux and BSD is that setting _POSIX_C_SOURCE and/or 
_XOPEN_SOURCE does not work very well.  For BSD it tends to rule out 
stuff I need -- notably in the area of sockets.  So, I have found it is 
more practical to declare _GNU_SOURCE for Linux and nothing at all for 
the BSDs and try to stick to the common areas.  The big differences 
between Linux and the BSDs you just have to deal with, in any case.

> Extensions (anything not specified by POSIX or in the UAPI) must be detected
> individually, because they may appear (or disappear! like sys/sysmacros.h) at
> any time. If you know which extensions you rely on, and have fallbacks as
> possible/needed, then it doesn't matter what the libc is named, or what it's
> version number is. And it will be much easier to port your application to a new
> environment.

I accept that the only way to be absolutely sure of any given extension 
(or local interpretation of some Standard) is to do the required tests 
at configure/cmake/etc time.

Nevertheless, I think it is possible to get a long way without a 
configure/cmake/etc step... much/most of the time.  And where something 
very specific needs to be configured, the more common stuff still does 
not (simplifying the configure step).

I agree that, in this context, the libc is of less interest than the 
underlying OS.

Nevertheless, I see no reason not to detect the libc, where I can.

Clearly musl falls into the (small) category of libc where I cannot.

Thanks,

Chris

Re: Detecting musl at compile and/or configure time

[Samuel Holland]

Hi,

On 6/30/19 7:03 AM, Chris Hall wrote:
> I have a little build system which tries to detect the "usual suspects"
> automatically, even without a full configure/cmake/etc. step.  If the detection
> process fails, it generates a warning and the user must (at least) add a '-Dxxx'
> to suppress that.
> 
> I confess I have only recently stumbled across musl.  Perhaps systems which
> default to musl are so rare that I can, as a practical matter, ignore them ? 
> The question then is whether to add a '-DqLIB_MUSL' gizmo to my build stuff --
> so that "musl-gcc -DqLIB_MUSL" will do the trick.

The fundamental problem with that is that you're not really checking for musl.
You're checking for "the supported version (and options) of POSIX", "the
supported version of the Linux UAPI", and "the set of extensions currently
supported by musl", which all change over time. And some of them depend on who's
distributing musl, as they may add their own customizations.

For POSIX, you should declare which version of the standard you follow with
_POSIX_C_SOURCE, which you can verify is supported with _POSIX_VERSION, or for
optional parts, the other macros in unistd.h[1].

The Linux UAPI is stable, so you can use whatever features you want as long as
you gracefully handle ENOSYS/EINVAL (or document a minimum supported version).

Extensions (anything not specified by POSIX or in the UAPI) must be detected
individually, because they may appear (or disappear! like sys/sysmacros.h) at
any time. If you know which extensions you rely on, and have fallbacks as
possible/needed, then it doesn't matter what the libc is named, or what it's
version number is. And it will be much easier to port your application to a new
environment.

[1]: https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/unistd.h.html

> Thanks,
> 
> Chris
> 

Cheers,
Samuel