Detecting musl at compile and/or configure time

Detecting musl at compile and/or configure time

[Chris Hall]

Is there a supported way of detecting that one is building against the 
musl libc ?

At compile time I can, of course, detect glibc because it sets __GLIBC__ 
etc.  For macOS I can see _DARWIN_FEATURE_xxx macros.  The BSDs mostly 
give themselves away.  But for musl I have not found anything :-(

Chris

Re: Detecting musl at compile and/or configure time

[A. Wilcox]

[-- Attachment #1.1: Type: text/plain, Size: 1249 bytes --]

On 06/29/19 06:48, Chris Hall wrote:
> 
> Is there a supported way of detecting that one is building against the
> musl libc ?
> 
> At compile time I can, of course, detect glibc because it sets __GLIBC__
> etc.  For macOS I can see _DARWIN_FEATURE_xxx macros.  The BSDs mostly
> give themselves away.  But for musl I have not found anything :-(
> 
> Chris


The musl libc specifically does not have a FTM (feature test macro)
because it aims to be an exact implementation of C11 / POSIX and
therefore it has no "quirks" to detect.

If there are bugs in musl libc, the authors prefer if you report them
and have them fixed, instead of making your software look for __MUSL__
and then assume all musl versions will always have that bug.

See here for more information:

https://wiki.musl-libc.org/faq.html#Q:-Why-is-there-no-%3Ccode%3E__MUSL__%3C/code%3E-macro?

https://www.openwall.com/lists/musl/2013/03/29/13

Is there a reason you wish to detect musl at configure/compile time?
Perhaps if we knew the reason, we could help you make your code more
portable and/or not require such detection.

Best to you and yours,
--arw


-- 
A. Wilcox (awilfox)
Project Lead, Adélie Linux
https://www.adelielinux.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Detecting musl at compile and/or configure time

[Szabolcs Nagy]

* A. Wilcox <awilfox@adelielinux.org> [2019-06-29 08:27:50 -0500]:
> On 06/29/19 06:48, Chris Hall wrote:
> > 
> > Is there a supported way of detecting that one is building against the
> > musl libc ?
> > 
> > At compile time I can, of course, detect glibc because it sets __GLIBC__
> > etc.  For macOS I can see _DARWIN_FEATURE_xxx macros.  The BSDs mostly
> > give themselves away.  But for musl I have not found anything :-(
> > 
> > Chris
> 
> 
> The musl libc specifically does not have a FTM (feature test macro)
> because it aims to be an exact implementation of C11 / POSIX and
> therefore it has no "quirks" to detect.

that's not entirely true.

there are some standard macros in posix to test optional features.

and musl implements various extensions to the standard.

note that __GLIBC__ does not detect glibc: uclibc defines it too.

you really want to configure check for particular features
instead of detecting the libc (which is a moving target,
ppl can patch it or change it over time)

> 
> If there are bugs in musl libc, the authors prefer if you report them
> and have them fixed, instead of making your software look for __MUSL__
> and then assume all musl versions will always have that bug.
> 
> See here for more information:
> 
> https://wiki.musl-libc.org/faq.html#Q:-Why-is-there-no-%3Ccode%3E__MUSL__%3C/code%3E-macro?
> 
> https://www.openwall.com/lists/musl/2013/03/29/13
> 
> Is there a reason you wish to detect musl at configure/compile time?
> Perhaps if we knew the reason, we could help you make your code more
> portable and/or not require such detection.
> 
> Best to you and yours,
> --arw
> 
> 
> -- 
> A. Wilcox (awilfox)
> Project Lead, Adélie Linux
> https://www.adelielinux.org
> 

Re: Detecting musl at compile and/or configure time

[Chris Hall]

On 29/06/2019 14:58, Szabolcs Nagy wrote:
>> On 06/29/19 06:48, Chris Hall wrote:
>>>
>>> Is there a supported way of detecting that one is building against the
>>> musl libc ?
>>>
>>> At compile time I can, of course, detect glibc because it sets __GLIBC__
>>> etc.  For macOS I can see _DARWIN_FEATURE_xxx macros.  The BSDs mostly
>>> give themselves away.  But for musl I have not found anything :-(
...
> note that __GLIBC__ does not detect glibc: uclibc defines it too.

Happily it has the courtesy to also:

   #define __UCLIBC__ 1

> you really want to configure check for particular features
> instead of detecting the libc (which is a moving target,
> ppl can patch it or change it over time)

In the limit, yes.  I'm interested in how far one can get without the 
need for configure/cmake/etc. -- accepting that means being quite 
conservative in the choice of features to use (and assuming that the 
target system is reasonably up to date).

Of course, for major issues -- eg epoll vs kqueue or netlink vs routeing 
sockets -- it's the underlying OS that really matters.  Detecting the OS 
at compile time seems reasonably effective (also the compiler and the 
processor).  Detecting the libc feels like it should be a small step to 
complete the picture.

Chris

malloc() alignment and max_align_t

[Chris Hall]

The C11 Standard says that malloc() will return a pointer which "is 
suitably aligned so that it may be assigned to a pointer to any type of 
object with a fundamental alignment requirement", and it says that 
_Alignof(max_align_t) is equal to the "fundamental alignment".

I note that:

  * musl malloc() works to an alignment SIZE_ALIGN, which is
    #defined'd to 4*sizeof(size_t) == 32 for 64-bit processors.

  * musl defines max_align_t:

      typedef struct { long long __ll;
                       long double __ld; } max_align_t;

    (same like other libc, FWIW).

    _Alignof(max_align_t) == 16 for x86_64 (at least),
    because that's the ABI alignment for long double.

Clearly, C11 does not require malloc() to align exactly as max_align_t, 
and bigger is fine.

But I'm curious as to why SIZE_ALIGN is twice as big as it needs to be ?

Thanks,

Chris

Re: malloc() alignment and max_align_t

[Markus Wichmann]

On Sun, Jul 07, 2019 at 07:17:48PM +0100, Chris Hall wrote:
> Clearly, C11 does not require malloc() to align exactly as max_align_t, and
> bigger is fine.
>
> But I'm curious as to why SIZE_ALIGN is twice as big as it needs to be ?
>

Design decision. For the algorithm to work, every chunk needs to be able
to contain four machine words (one struct chunk). And with a bit of
maneuvering, this alignment can just be achieved on all chunks with
minimal waste (if a completely new region is allocated in expand_heap(),
then two machine words at the start of it are wasted, but if the new
memory happens to be directly behind the previous section, no waste
occurs at all).

This makes it easier to reason about chunk sizes. Right now, SIZE_ALIGN
and minimum chunk size (after adding OVERHEAD) are the same. If we
lowered the alignment to two machine words, that would change.

> Thanks,
>
> Chris

Ciao,
Markus

Re: malloc() alignment and max_align_t

[Rich Felker]

On Sun, Jul 07, 2019 at 09:22:00PM +0200, Markus Wichmann wrote:
> On Sun, Jul 07, 2019 at 07:17:48PM +0100, Chris Hall wrote:
> > Clearly, C11 does not require malloc() to align exactly as max_align_t, and
> > bigger is fine.
> >
> > But I'm curious as to why SIZE_ALIGN is twice as big as it needs to be ?
> >
> 
> Design decision. For the algorithm to work, every chunk needs to be able
> to contain four machine words (one struct chunk). And with a bit of
> maneuvering, this alignment can just be achieved on all chunks with
> minimal waste (if a completely new region is allocated in expand_heap(),
> then two machine words at the start of it are wasted, but if the new
> memory happens to be directly behind the previous section, no waste
> occurs at all).
> 
> This makes it easier to reason about chunk sizes. Right now, SIZE_ALIGN
> and minimum chunk size (after adding OVERHEAD) are the same. If we
> lowered the alignment to two machine words, that would change.

Indeed, we could do 16- instead of 32-byte alignment on 64-bit, but
then we'd have to preclude the smallest size (16) because it has 0
space after header/footer are accounted for. This is problematic
because splitting can lead to a smallest-size chunk without special
logic to preclude that, and possibly for other reasons.

Note that the header and footer could be reduced to uint32_t rather
than size_t; there's no need for huge sizes in this field for managed
heap chunks, and mmap-serviced allocations could use a different
encoding. And the linked list pointers could be an xor list, dropping
the minimum size to 4+4=8==16 on 64-bit and 4+4+4==12 on 32-bit.
However the malloc implementation is slated for replacement, so tuning
stuff like this is not a very productive activity right now.

Rich