why is there no __MUSL__ macro?

why is there no __MUSL__ macro?

[Jörg Krause]

[-- Attachment #1: Type: text/plain, Size: 841 bytes --]

Hi,

I am trying to add support for the musl toolchain to FFmpeg.

FFmpeg needs support for library features defined in POSIX.1-2001 with 
XSI extension and the standards below. Currently configure probes the 
host and target libc by checking for defined macros like __GLIBC__ and 
__UCLIBC__. In case of glibc and uclibc it sets -D_XOPEN_SOURCE=600 
properly.

After this it checks for some combinations of hardware and the probed 
libc to set some more compile options, if necessary.

I know that musl does not have a macro __MUSL__ and I have read the 
explanation. However, I don't understand what's meant by "[..] it's a 
bug to assume a certain implementation has particular properties rather 
than testing." and how does it affect the way FFmpeg probes for the libc.

What could be a solution which supports musl?

Many thanks!
Jörg

[-- Attachment #2: Type: text/html, Size: 1434 bytes --]

Re: why is there no __MUSL__ macro?

[Laurent Bercot]

> FFmpeg needs support for library features defined in POSIX.1-2001 with XSI extension and the standards below. Currently configure probes the host and target libc by checking for defined macros like __GLIBC__ and __UCLIBC__. In case of glibc and uclibc it sets -D_XOPEN_SOURCE=600 properly.

  Why not set this macro unconditionally ?
  All standards-compliant libcs will make the correct symbols visible
if you define _XOPEN_SOURCE to a certain value. This include glibc,
uClibc, musl, and most other modern libcs. Some systems are
notoriously broken (I'm thinking of FreeBSD, which makes a few
standard symbols *not* visible when you define the correct macro),
but they should be the ones with specialcasing if you need to
support them.


> What could be a solution which supports musl?

  My answer as a musl user, which may be different from answers from
musl developers, is to use -D_XOPEN_SOURCE=600 without testing for
the libc name, and only use system-specific macros to work around
bugs in non-compliant systems.

-- 
  Laurent

Re: why is there no __MUSL__ macro?

[Szabolcs Nagy]

* Laurent Bercot <ska-dietlibc@skarnet.org> [2014-09-11 12:09:07 +0100]:
> >FFmpeg needs support for library features defined in POSIX.1-2001 with XSI extension and the standards below. Currently configure probes the host and target libc by checking for defined macros like __GLIBC__ and __UCLIBC__. In case of glibc and uclibc it sets -D_XOPEN_SOURCE=600 properly.
> 
>  Why not set this macro unconditionally ?
>  All standards-compliant libcs will make the correct symbols visible
> if you define _XOPEN_SOURCE to a certain value. This include glibc,

this has to be the most frequently asked question

http://wiki.musl-libc.org/wiki/FAQ#Q:_why_is_there_no_MUSL_macro_.3F

and yes, assuming standard conformance by default is the
sane thing to do

then _testing_ for conformance issues is the second try
if the default fails

and only if testing is somehow difficult/inappropriate
should one fall back to hardcoding behaviour for particular
non-conforming systems

in case of musl conformance bugs are fixed so you should
report them instead of trying to work them around

Re: why is there no __MUSL__ macro?

[Jens]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1860 bytes --]


On Thu, 11 Sep 2014, Jörg Krause wrote:
> Hi,
> 
> I am trying to add support for the musl toolchain to FFmpeg.
> 
> FFmpeg needs support for library features defined in POSIX.1-2001 with XSI extension and the standards
> below. Currently configure probes the host and target libc by checking for defined macros like __GLIBC__ and
> __UCLIBC__. In case of glibc and uclibc it sets -D_XOPEN_SOURCE=600 properly.
> 
> After this it checks for some combinations of hardware and the probed libc to set some more compile options,
> if necessary.
> 
> I know that musl does not have a macro __MUSL__ and I have read the explanation. However, I don't understand
> what's meant by "[..] it's a bug to assume a certain implementation has particular properties rather than
> testing." and how does it affect the way FFmpeg probes for the libc.

Maybe the explanation in the FAQ could be a bit more verbose.
But if you stop and think for a bit you'll realize that __MUSL__ would 
tell you next to nothing. musl and any other libc that is maintained will 
change with time. You cannot assume a specific functionality by checking 
for __MUSL__. Bugs will be fixed, functionality will be implemented etc.

Even if you could check for a specific version of musl you will do most 
people a disservice since their libc/version wont be handled.

If instead you check for a specific functionality, it will be independent 
of libc and version of libc.

A somewhat limited build of ffmpeg, atleast, works fine with musl:
OPTPREFIX=opt/av
configure --prefix=/$OPTPREFIX \
  --enable-gpl\
  --enable-small --disable-ffplay --disable-ffprobe --disable-ffserver\
  --enable-libmp3lame\
  --enable-libx264 \
  --disable-network --cc=musl-gcc

Thats my understanding, for what its worth.

Regards,
Jens

> 
> What could be a solution which supports musl?
> 
> Many thanks!
> Jörg
> 
>

Re: why is there no __MUSL__ macro?

[Jörg Krause]

On 09/11/2014 01:17 PM, Szabolcs Nagy wrote:
> * Laurent Bercot <ska-dietlibc@skarnet.org> [2014-09-11 12:09:07 +0100]:
>>> FFmpeg needs support for library features defined in POSIX.1-2001 with XSI extension and the standards below. Currently configure probes the host and target libc by checking for defined macros like __GLIBC__ and __UCLIBC__. In case of glibc and uclibc it sets -D_XOPEN_SOURCE=600 properly.
>>   Why not set this macro unconditionally ?
>>   All standards-compliant libcs will make the correct symbols visible
>> if you define _XOPEN_SOURCE to a certain value. This include glibc,
> this has to be the most frequently asked question
>
> http://wiki.musl-libc.org/wiki/FAQ#Q:_why_is_there_no_MUSL_macro_.3F
>
> and yes, assuming standard conformance by default is the
> sane thing to do

I see. So it should be safe to assume standard conformance of the libc 
and set _XOPEN_SOURCE properly.

>
> then _testing_ for conformance issues is the second try
> if the default fails

What do you mean with testing for concormance?

Re: why is there no __MUSL__ macro?

[Jörg Krause]

On 09/11/2014 01:17 PM, Jens wrote:
>
> On Thu, 11 Sep 2014, Jörg Krause wrote:
>> Hi,
>>
>> I am trying to add support for the musl toolchain to FFmpeg.
>>
>> FFmpeg needs support for library features defined in POSIX.1-2001 
>> with XSI extension and the standards
>> below. Currently configure probes the host and target libc by 
>> checking for defined macros like __GLIBC__ and
>> __UCLIBC__. In case of glibc and uclibc it sets -D_XOPEN_SOURCE=600 
>> properly.
>>
>> After this it checks for some combinations of hardware and the probed 
>> libc to set some more compile options,
>> if necessary.
>>
>> I know that musl does not have a macro __MUSL__ and I have read the 
>> explanation. However, I don't understand
>> what's meant by "[..] it's a bug to assume a certain implementation 
>> has particular properties rather than
>> testing." and how does it affect the way FFmpeg probes for the libc.
>
> Maybe the explanation in the FAQ could be a bit more verbose.

Maybe that would help :-)

> But if you stop and think for a bit you'll realize that __MUSL__ would 
> tell you next to nothing. musl and any other libc that is maintained 
> will change with time. You cannot assume a specific functionality by 
> checking for __MUSL__. Bugs will be fixed, functionality will be 
> implemented etc.

I see.

>
> Even if you could check for a specific version of musl you will do 
> most people a disservice since their libc/version wont be handled.
> If instead you check for a specific functionality, it will be 
> independent of libc and version of libc.

I am not sure what you mean with "check for a specific functionality"?

>
> A somewhat limited build of ffmpeg, atleast, works fine with musl:
> OPTPREFIX=opt/av
> configure --prefix=/$OPTPREFIX \
>  --enable-gpl\
>  --enable-small --disable-ffplay --disable-ffprobe --disable-ffserver\
>  --enable-libmp3lame\
>  --enable-libx264 \
>  --disable-network --cc=musl-gcc

I'm using a musl cross-compiling toolchain and the limited build even 
without mp3 and x264 fails for me:

./configure --enable-cross-compile\
  --cross-prefix=/home/joerg/x86_64-linux-musl/bin/x86_64-linux-musl-\
  --arch=x86_64 --target-os=linux\
  --prefix=/home/joerg/test/ffmpeg\
  --enable-gpl\
  --enable-small --disable-ffplay --disable-ffprobe --disable-ffserver\
  --disable-network
[...]
make: *** [libavformat/segment.o] Error 1

Re: why is there no __MUSL__ macro?

[Szabolcs Nagy]

* J?rg Krause <jkrause@posteo.de> [2014-09-11 14:02:59 +0200]:
> On 09/11/2014 01:17 PM, Szabolcs Nagy wrote:
> >then _testing_ for conformance issues is the second try
> >if the default fails
> 
> What do you mean with testing for concormance?

eg glibc scanf uses "%a" for its own extension by default
and c99 behaviour is only provided with appropriate cflags

if your project depends on %a scanf then you may need to
test for this conformance issue (instead of ifdef __GLIBC__
because they may change the behaviour later or the cflag
might not work on an older version etc)

of course there are cases when you depend on behaviour that
is not described by any standard in which case it is not
"conformance testing" but you still need some kind of testing
of the behaviour for portability

Re: why is there no __MUSL__ macro?

[Jörg Krause]

On 09/11/2014 02:38 PM, Szabolcs Nagy wrote:
> * J?rg Krause <jkrause@posteo.de> [2014-09-11 14:02:59 +0200]:
>> On 09/11/2014 01:17 PM, Szabolcs Nagy wrote:
>>> then _testing_ for conformance issues is the second try
>>> if the default fails
>> What do you mean with testing for concormance?
> eg glibc scanf uses "%a" for its own extension by default
> and c99 behaviour is only provided with appropriate cflags
>
> if your project depends on %a scanf then you may need to
> test for this conformance issue (instead of ifdef __GLIBC__
> because they may change the behaviour later or the cflag
> might not work on an older version etc)

I see. But I can avoid the GNU specific bevahiour by undefining 
_GNU_SOURCE if I want POSIX-conformance. In this case I do not need to 
test, but can rely on the libc of being conformal. Do I have this right?

eg, in FFmpeg/libavutils uses strerror_r which is implemented as a 
XSI-compliant and a GNU-specific version. If I want to be sure to get 
the XSI-compliant version, I unset _GNU_SOURCE and set 
_XOPEN_SOURCE=600. So I do not need any further testing here, right?

Re: why is there no __MUSL__ macro?

[Christian Neukirchen]

Laurent Bercot <ska-dietlibc@skarnet.org> writes:

>> FFmpeg needs support for library features defined in POSIX.1-2001
>> with XSI extension and the standards below. Currently configure
>> probes the host and target libc by checking for defined macros like
>> __GLIBC__ and __UCLIBC__. In case of glibc and uclibc it sets
>> -D_XOPEN_SOURCE=600 properly.
>
>  Why not set this macro unconditionally ?
>  All standards-compliant libcs will make the correct symbols visible
> if you define _XOPEN_SOURCE to a certain value. This include glibc,
> uClibc, musl, and most other modern libcs. Some systems are
> notoriously broken (I'm thinking of FreeBSD, which makes a few
> standard symbols *not* visible when you define the correct macro),
> but they should be the ones with specialcasing if you need to
> support them.

OT, but when you report these to freebsd-standards, they'll be resolved
quickly.  That is at least my experience.

-- 
Christian Neukirchen  <chneukirchen@gmail.com>  http://chneukirchen.org

Re: why is there no __MUSL__ macro?

[Natanael Copa]

On Thu, 11 Sep 2014 13:00:09 +0200
Jörg Krause <jkrause@posteo.de> wrote:

> Hi,
> 
> I am trying to add support for the musl toolchain to FFmpeg.
> 
> FFmpeg needs support for library features defined in POSIX.1-2001 with 
> XSI extension and the standards below. Currently configure probes the 
> host and target libc by checking for defined macros like __GLIBC__ and 
> __UCLIBC__. In case of glibc and uclibc it sets -D_XOPEN_SOURCE=600 
> properly.
> 
> After this it checks for some combinations of hardware and the probed 
> libc to set some more compile options, if necessary.
> 
> I know that musl does not have a macro __MUSL__ and I have read the 
> explanation. However, I don't understand what's meant by "[..] it's a 
> bug to assume a certain implementation has particular properties rather 
> than testing." and how does it affect the way FFmpeg probes for the libc.
> 
> What could be a solution which supports musl?
> 
> Many thanks!
> Jörg

This is what we do on alpine linux:
http://git.alpinelinux.org/cgit/aports/tree/main/ffmpeg/fix-defines.patch

--- ffmpeg-1.2.2.orig/libavutil/error.c
+++ ffmpeg-1.2.2/libavutil/error.c
@@ -17,6 +17,7 @@
  */
 
 #undef _GNU_SOURCE
+#define _XOPEN_SOURCE 600
 #include "avutil.h"
 #include "avstring.h"
 #include "common.h"

Re: why is there no __MUSL__ macro?

[Szabolcs Nagy]

* J?rg Krause <jkrause@posteo.de> [2014-09-11 15:33:16 +0200]:
> eg, in FFmpeg/libavutils uses strerror_r which is implemented as a
> XSI-compliant and a GNU-specific version. If I want to be sure to get the
> XSI-compliant version, I unset _GNU_SOURCE and set _XOPEN_SOURCE=600. So I
> do not need any further testing here, right?

yes, using feature test macros should work
(unless you want to support some very old version of the libc)

Re: why is there no __MUSL__ macro?

[Isaac Dunham]

On Thu, Sep 11, 2014 at 02:24:21PM +0200, Jörg Krause wrote:
> 
> On 09/11/2014 01:17 PM, Jens wrote:
> >
> >A somewhat limited build of ffmpeg, atleast, works fine with musl:
> >OPTPREFIX=opt/av
> >configure --prefix=/$OPTPREFIX \
> > --enable-gpl\
> > --enable-small --disable-ffplay --disable-ffprobe --disable-ffserver\
> > --enable-libmp3lame\
> > --enable-libx264 \
> > --disable-network --cc=musl-gcc
> 
> I'm using a musl cross-compiling toolchain and the limited build even
> without mp3 and x264 fails for me:
> 
> ./configure --enable-cross-compile\
>  --cross-prefix=/home/joerg/x86_64-linux-musl/bin/x86_64-linux-musl-\
>  --arch=x86_64 --target-os=linux\
>  --prefix=/home/joerg/test/ffmpeg\
>  --enable-gpl\
>  --enable-small --disable-ffplay --disable-ffprobe --disable-ffserver\
>  --disable-network
> [...]
> make: *** [libavformat/segment.o] Error 1
> 
It would be easier to see what's going on if you provided the command line
for compiling that file, plus the rest of the error.
If ffmpeg is showing the "CC ..." message rather than a massive command
line, then run "make V=1".

Thanks,
Iaac Dunham

Re: why is there no __MUSL__ macro?

[Jörg Krause]

On 09/11/2014 04:47 PM, Natanael Copa wrote:
> On Thu, 11 Sep 2014 13:00:09 +0200
> Jörg Krause <jkrause@posteo.de> wrote:
>
>> Hi,
>>
>> I am trying to add support for the musl toolchain to FFmpeg.
>>
>> FFmpeg needs support for library features defined in POSIX.1-2001 with
>> XSI extension and the standards below. Currently configure probes the
>> host and target libc by checking for defined macros like __GLIBC__ and
>> __UCLIBC__. In case of glibc and uclibc it sets -D_XOPEN_SOURCE=600
>> properly.
>>
>> After this it checks for some combinations of hardware and the probed
>> libc to set some more compile options, if necessary.
>>
>> I know that musl does not have a macro __MUSL__ and I have read the
>> explanation. However, I don't understand what's meant by "[..] it's a
>> bug to assume a certain implementation has particular properties rather
>> than testing." and how does it affect the way FFmpeg probes for the libc.
>>
>> What could be a solution which supports musl?
>>
>> Many thanks!
>> Jörg
> This is what we do on alpine linux:
> http://git.alpinelinux.org/cgit/aports/tree/main/ffmpeg/fix-defines.patch
>
> --- ffmpeg-1.2.2.orig/libavutil/error.c
> +++ ffmpeg-1.2.2/libavutil/error.c
> @@ -17,6 +17,7 @@
>    */
>   
>   #undef _GNU_SOURCE
> +#define _XOPEN_SOURCE 600
>   #include "avutil.h"
>   #include "avstring.h"
>   #include "common.h"
>

Hi Natanal,

I had a look to alpine already. I submitted this patch to FFmpeg, but 
building FFmpeg with my configuration libavutils/error.c is not the only 
file which needs a feature test macro. The people of FFmpeg did not like 
the idea to have a lot of test macros in there source so I stopped with 
this solution and looked for a way to adopt the musl toolchain to their 
configure file.

Re: why is there no __MUSL__ macro?

[Jörg Krause]

On 09/11/2014 04:53 PM, Isaac Dunham wrote:
> On Thu, Sep 11, 2014 at 02:24:21PM +0200, Jörg Krause wrote:
>> On 09/11/2014 01:17 PM, Jens wrote:
>>> A somewhat limited build of ffmpeg, atleast, works fine with musl:
>>> OPTPREFIX=opt/av
>>> configure --prefix=/$OPTPREFIX \
>>> --enable-gpl\
>>> --enable-small --disable-ffplay --disable-ffprobe --disable-ffserver\
>>> --enable-libmp3lame\
>>> --enable-libx264 \
>>> --disable-network --cc=musl-gcc
>> I'm using a musl cross-compiling toolchain and the limited build even
>> without mp3 and x264 fails for me:
>>
>> ./configure --enable-cross-compile\
>>   --cross-prefix=/home/joerg/x86_64-linux-musl/bin/x86_64-linux-musl-\
>>   --arch=x86_64 --target-os=linux\
>>   --prefix=/home/joerg/test/ffmpeg\
>>   --enable-gpl\
>>   --enable-small --disable-ffplay --disable-ffprobe --disable-ffserver\
>>   --disable-network
>> [...]
>> make: *** [libavformat/segment.o] Error 1
>>
> It would be easier to see what's going on if you provided the command line
> for compiling that file, plus the rest of the error.
> If ffmpeg is showing the "CC ..." message rather than a massive command
> line, then run "make V=1".
>
> Thanks,
> Iaac Dunham

Hi Isaac,

it was not my intention to report an error. I just wanted to reply that 
the configuration posted by Jens did not work for me.

Btw, to fix the compilation with musl it's necessary to add the feature 
test macro _XOPEN_SOURCE=600.

Re: why is there no __MUSL__ macro?

[Rich Felker]

On Fri, Sep 12, 2014 at 09:35:22AM +0200, Jörg Krause wrote:
> 
> On 09/11/2014 04:47 PM, Natanael Copa wrote:
> >On Thu, 11 Sep 2014 13:00:09 +0200
> >Jörg Krause <jkrause@posteo.de> wrote:
> >
> >>Hi,
> >>
> >>I am trying to add support for the musl toolchain to FFmpeg.
> >>
> >>FFmpeg needs support for library features defined in POSIX.1-2001 with
> >>XSI extension and the standards below. Currently configure probes the
> >>host and target libc by checking for defined macros like __GLIBC__ and
> >>__UCLIBC__. In case of glibc and uclibc it sets -D_XOPEN_SOURCE=600
> >>properly.
> >>
> >>After this it checks for some combinations of hardware and the probed
> >>libc to set some more compile options, if necessary.
> >>
> >>I know that musl does not have a macro __MUSL__ and I have read the
> >>explanation. However, I don't understand what's meant by "[..] it's a
> >>bug to assume a certain implementation has particular properties rather
> >>than testing." and how does it affect the way FFmpeg probes for the libc.
> >>
> >>What could be a solution which supports musl?
> >>
> >>Many thanks!
> >>Jörg
> >This is what we do on alpine linux:
> >http://git.alpinelinux.org/cgit/aports/tree/main/ffmpeg/fix-defines.patch
> >
> >--- ffmpeg-1.2.2.orig/libavutil/error.c
> >+++ ffmpeg-1.2.2/libavutil/error.c
> >@@ -17,6 +17,7 @@
> >   */
> >  #undef _GNU_SOURCE
> >+#define _XOPEN_SOURCE 600
> >  #include "avutil.h"
> >  #include "avstring.h"
> >  #include "common.h"
> >
> 
> Hi Natanal,
> 
> I had a look to alpine already. I submitted this patch to FFmpeg,
> but building FFmpeg with my configuration libavutils/error.c is not
> the only file which needs a feature test macro. The people of FFmpeg
> did not like the idea to have a lot of test macros in there source
> so I stopped with this solution and looked for a way to adopt the
> musl toolchain to their configure file.

It might work to add -D_DEFAULT_SOURCE (note: not available until
recent musl git) or -D_BSD_SOURCE to the global CFLAGS.

What's happening, I think, is that because -std=c99 or similar is in
the CFLAGS, default feature profile is getting suppressed, and
_GNU_SOURCE is the only thing bringing back the usual features (in
addition to lots of _GNU_SOURCE-only stuff, and strerror_r breakage).
Having at least one more feature test macro defined globally would
prevent this.

Note also that modern glibc supports _DEFAULT_SOURCE, with the same
semantics as musl: keeping the default stuff exposed even when a
-std=* option would otherwise suppress it.

Rich