[musl] __MUSL__ macro

[musl] __MUSL__ macro

[Alastair Houghton]

[-- Attachment #1: Type: text/plain, Size: 2272 bytes --]

Hi all,

Before I start, I’m aware of

  <https://www.openwall.com/lists/musl/2013/03/29/13>

but I *still* want to add __MUSL__ (see attached patch).

Let me explain what we’re doing, why we want it and why we think musl *should* have it.  We’re trying to add support for musl to Swift <https://swift.org <https://swiftlang.org/>> and its attendant core libraries, and there are a number of things about musl that presently differ from other platforms/C libraries we support.

Examples include the use of `union`s in `pthread_mutex_t` et al (which means that we can’t write a C++ `constexpr` function that returns one, even if all it does is return `PTHREAD_MUTEX_INITIALIZER`), the fact that it doesn’t have the `d_namlen` member of `struct dirent`, or the fact that `dladdr()` is a no-op when statically linked.

I’m aware that there are other solutions for some of these cases - but for instance the `dladdr()` issue is not something you can test at configuration time when cross-compiling, since there’s no way to know what the result would be without running a program on the target system (which you don’t necessarily have available at configuration time in that case).  Likewise, configuration time tests are slow and  aren’t even a thing in some of the projects we need to add support in, whereas we do generally have the ability to use the preprocessor.

You might say we should just bite the bullet and add configuration steps to all of the other projects, but that is honestly a non-starter; the owners of those projects are not likely to accept patches to add such a thing, even as a short-term fix, and the patches to do so would be non-trivial in a number of cases also.  In the longer term we don’t want higher level projects to need to do this kind of thing and the lower level ones could in many cases do some kind of configuration time testing, but that’s a longer term goal and it doesn’t help us to add musl support in the interim, which we would like to do.

I’ve attached a proposed patch that adds `__MUSL__` (set to the major version) and `__MUSL_MINOR__` (set to the minor version) as well as `__MUSL_PREREQ()` (which works the same way `__GLIBC_PREREQ()` does).

Kind regards,

Alastair Houghton


[-- Attachment #2: musl-version.patch --]
[-- Type: application/octet-stream, Size: 2772 bytes --]

diff --git a/Makefile b/Makefile
index e8cc4436..ea91b9be 100644
--- a/Makefile
+++ b/Makefile
@@ -34,7 +34,7 @@ CRT_OBJS = $(filter obj/crt/%,$(ALL_OBJS))
 
 AOBJS = $(LIBC_OBJS)
 LOBJS = $(LIBC_OBJS:.o=.lo)
-GENH = obj/include/bits/alltypes.h obj/include/bits/syscall.h
+GENH = obj/include/bits/alltypes.h obj/include/bits/syscall.h obj/include/features.h
 GENH_INT = obj/src/internal/version.h
 IMPH = $(addprefix $(srcdir)/, src/internal/stdio_impl.h src/internal/pthread_impl.h src/internal/locale_impl.h src/internal/libc.h)
 
@@ -95,6 +95,17 @@ $(ALL_LIBS) $(ALL_TOOLS) $(ALL_OBJS) $(ALL_OBJS:%.o=%.lo) $(GENH) $(GENH_INT): |
 $(OBJ_DIRS):
 	mkdir -p $@
 
+MUSL_VERSION:=$(shell cd $(srcdir) && sh tools/version.sh)
+empty:=
+space:=$(empty) $(empty)
+musl_ver_parts:=$(subst -git-,$(space),$(subst .,$(space),$(MUSL_VERSION)))
+MUSL_MAJOR:=$(word 1, $(musl_ver_parts))
+MUSL_MINOR:=$(word 2, $(musl_ver_parts))
+MUSL_PATCH:=$(word 3, $(musl_ver_parts))
+
+obj/include/features.h: $(srcdir)/include/features.h.in $(wildcard $(srcdir)/VERSION $(srcdir)/.git)
+	sed -e "s/@MUSL_MAJOR@/$(MUSL_MAJOR)/g;s/@MUSL_MINOR@/$(MUSL_MINOR)/g;s/@MUSL_PATCH@/$(MUSL_PATCH)/g" $< > $@
+
 obj/include/bits/alltypes.h: $(srcdir)/arch/$(ARCH)/bits/alltypes.h.in $(srcdir)/include/alltypes.h.in $(srcdir)/tools/mkalltypes.sed
 	sed -f $(srcdir)/tools/mkalltypes.sed $(srcdir)/arch/$(ARCH)/bits/alltypes.h.in $(srcdir)/include/alltypes.h.in > $@
 
@@ -103,7 +114,7 @@ obj/include/bits/syscall.h: $(srcdir)/arch/$(ARCH)/bits/syscall.h.in
 	sed -n -e s/__NR_/SYS_/p < $< >> $@
 
 obj/src/internal/version.h: $(wildcard $(srcdir)/VERSION $(srcdir)/.git)
-	printf '#define VERSION "%s"\n' "$$(cd $(srcdir); sh tools/version.sh)" > $@
+	printf '#define VERSION "%s"\n' "$(MUSL_VERSION)" > $@
 
 obj/src/internal/version.o obj/src/internal/version.lo: obj/src/internal/version.h
 
@@ -209,6 +220,9 @@ $(DESTDIR)$(includedir)/bits/%: obj/include/bits/%
 $(DESTDIR)$(includedir)/%: $(srcdir)/include/%
 	$(INSTALL) -D -m 644 $< $@
 
+$(DESTDIR)$(includedir)/%: obj/include/%
+	$(INSTALL) -D -m 644 $< $@
+
 $(DESTDIR)$(LDSO_PATHNAME): $(DESTDIR)$(libdir)/libc.so
 	$(INSTALL) -D -l $(libdir)/libc.so $@ || true
 
diff --git a/include/features.h b/include/features.h.in
similarity index 80%
rename from include/features.h
rename to include/features.h.in
index 85cfb72a..07662804 100644
--- a/include/features.h
+++ b/include/features.h.in
@@ -1,6 +1,13 @@
 #ifndef _FEATURES_H
 #define _FEATURES_H
 
+#define __MUSL__       @MUSL_MAJOR@
+#define __MUSL_MINOR__ @MUSL_MINOR@
+#define __MUSL_PATCH__ @MUSL_PATCH@
+
+#define __MUSL_PREREQ(maj, min) \
+  ((__MUSL__ << 16) + __MUSL_MINOR >= ((maj) << 16) + (min))
+
 #if defined(_ALL_SOURCE) && !defined(_GNU_SOURCE)
 #define _GNU_SOURCE 1
 #endif

Re: [musl] __MUSL__ macro

[Alex Xu]

Quoting Alastair Houghton (2023-07-06 06:48:04)
> Hi all,
> 
> Before I start, I’m aware of
> 
>   <https://www.openwall.com/lists/musl/2013/03/29/13>
> 
> but I *still* want to add __MUSL__ (see attached patch).
> 
> Let me explain what we’re doing, why we want it and why we think musl *should* have it.  We’re trying to add support for musl to Swift <https://swift.org <https://swiftlang.org/>> and its attendant core libraries, and there are a number of things about musl that presently differ from other platforms/C libraries we support.

All of these seem to fall at least suspiciously closely to "the usage
case was badly wrong"...

> Examples include the use of `union`s in `pthread_mutex_t` et al (which means that we can’t write a C++ `constexpr` function that returns one, even if all it does is return `PTHREAD_MUTEX_INITIALIZER`),

The issue is that some of the members are volatile, not the union
itself. As I understand, though, constexpr is useful to evaluate certain
expressions at compile time, such as if (x < 10) or int arr[getlen(x)].
You can't do any arithmetic with pthread_mutex_t, and the only valid
value at compile time is PTHREAD_MUTEX_INITIALIZER though, so what is
the use case?

> the fact that it doesn’t have the `d_namlen` member of `struct dirent`

You can use strlen(d->d_name) instead? This seems like exactly the
reason why __MUSL__ should absolutely not be added, because it
incentivizes individual checks for endless platforms instead of writing
portable code. If strlen(d->d_name) is too slow in some context, then an
argument should be made to add d_namlen. glibc even already has a macro
_DIRENT_HAVE_D_NAMLEN that could be used to signal support for this
rather than hardcoding specific musl versions.

> or the fact that `dladdr()` is a no-op when statically linked.

When statically linked, the dynamic linking interface doesn't work at
all though? And furthermore, even if it did, what would it return? If
you're (once again) trying to do stack traces manually, try libunwind?

> I’m aware that there are other solutions for some of these cases - but for instance the `dladdr()` issue is not something you can test at configuration time when cross-compiling, since there’s no way to know what the result would be without running a program on the target system (which you don’t necessarily have available at configuration time in that case).

I don't understand, don't you know at compile time whether you're
linking statically or dynamically?

> Likewise, configuration time tests are slow and  aren’t even a thing in some of the projects we need to add support in, whereas we do generally have the ability to use the preprocessor.

Configure-time checks for specific functionality are a standard part of
writing portable C code. Preprocessor checks for specific platforms lead
to ifdef swamps that are still not portable to any platforms not
initially considered.

> You might say we should just bite the bullet and add configuration steps to all of the other projects, but that is honestly a non-starter; the owners of those projects are not likely to accept patches to add such a thing, even as a short-term fix, and the patches to do so would be non-trivial in a number of cases also.  In the longer term we don’t want higher level projects to need to do this kind of thing and the lower level ones could in many cases do some kind of configuration time testing, but that’s a longer term goal and it doesn’t help us to add musl support in the interim, which we would like to do.
> 
> I’ve attached a proposed patch that adds `__MUSL__` (set to the major version) and `__MUSL_MINOR__` (set to the minor version) as well as `__MUSL_PREREQ()` (which works the same way `__GLIBC_PREREQ()` does).

In theory, there could possibly be some reason why __MUSL__ could be
useful in some cases. However, your examples appear to be exactly why
__MUSL__ should not be added because it leads to bad code.

Cheers,
Alex.

Re: [musl] __MUSL__ macro

[Szabolcs Nagy]

* Alex Xu <alex_y_xu@yahoo.ca> [2023-07-06 08:17:48 -0400]:

> Quoting Alastair Houghton (2023-07-06 06:48:04)
> > Hi all,
> > 
> > Before I start, I’m aware of
> > 
> >   <https://www.openwall.com/lists/musl/2013/03/29/13>
> > 
> > but I *still* want to add __MUSL__ (see attached patch).
> > 
> > Let me explain what we’re doing, why we want it and why we think musl *should* have it.  We’re trying to add support for musl to Swift <https://swift.org <https://swiftlang.org/>> and its attendant core libraries, and there are a number of things about musl that presently differ from other platforms/C libraries we support.
> 
> All of these seem to fall at least suspiciously closely to "the usage
> case was badly wrong"...

the problem is not that the usage is wrong, but that
these can change between different versions of musl
(within abi constraints) and header changes can be
backported by distros.

so __MUSL__ cannot solve these issues.

Re: [musl] __MUSL__ macro

[Alastair Houghton]

On 6 Jul 2023, at 13:17, Alex Xu <alex_y_xu@yahoo.ca> wrote:
> 
>> Examples include the use of `union`s in `pthread_mutex_t` et al (which means that we can’t write a C++ `constexpr` function that returns one, even if all it does is return `PTHREAD_MUTEX_INITIALIZER`),
> 
> The issue is that some of the members are volatile, not the union
> itself.

...

Arguing about the details of individual issues is an unnecessary distraction, honestly.  They were just OTOH examples of oddities that we’ve come across so far, and yes, I agree that in some cases there are potentially better fixes for them than preprocessor shenanigans - and where those are available I’m totally onboard with doing that instead, because I *entirely agree* that it’s better to avoid a huge mass of `#if` conditions in code.  There’s a very good chance that any I add in the various places I might be interested in adding them will end up being my responsibility to maintain, so that, if nothing else, is a good motivation for minimising them.

>> or the fact that `dladdr()` is a no-op when statically linked.
> 
> When statically linked, the dynamic linking interface doesn't work at
> all though? And furthermore, even if it did, what would it return? If
> you're (once again) trying to do stack traces manually, try libunwind?

:-D Oh, the irony.  One of the places where this behaviour turns out to be a problem is *in the tests for libunwind*.  (That can be solved in another way without resorting to macros, however.)

> I don't understand, don't you know at compile time whether you're
> linking statically or dynamically?

Not always, no (see situation 2 below).

> Configure-time checks for specific functionality are a standard part of
> writing portable C code. Preprocessor checks for specific platforms lead
> to ifdef swamps that are still not portable to any platforms not
> initially considered.

There are two situations where the lack of macros is absolutely a problem and there isn’t a sane workaround.

1. You’re cross compiling and the thing you’re interested in is *runtime* behaviour, so configure-style checks simply aren’t an option.  In a cross-compilation situation the only thing they can test is that a program will *build*; they can’t test its behaviour since you don’t necessarily have anywhere to run it.

2. You’re in a header-only library, particularly one where the final environment you’re running in wasn’t available at whatever configuration time you had, assuming you had one at all.  A really good example of this is the built-in headers in Clang or GCC, but the problem also applies to things like STL or Boost.

> On 6 Jul 2023, at 17:26, Szabolcs Nagy <nsz@port70.net> wrote:
> 
> the problem is not that the usage is wrong, but that
> these can change between different versions of musl
> (within abi constraints) and header changes can be
> backported by distros.
> 
> so __MUSL__ cannot solve these issues.


That’s why `__MUSL__` should evaluate to the major version number and `__MUSL_MINOR__` to the minor version number, and why we should have `__MUSL_PREREQ()`, just like Glibc, which, by the way, has the exact same problems you raise (things can change, and distros can backport things).

If some third-party package wishes to test `__MUSL__`, it’s up to the maintainers of that package to make sure that they perform appropriate version checks.  And yes, if they’re working around a bug or musl changes its behaviour to add a feature that wasn’t there, they may have to update their code from `#ifndef __MUSL__` to `#if __MUSL_PREREQ(x,y)` or similar.  Package maintainers that make these kinds of tests are used to having to do that and if they’re willing to accept macro-based tests, then they’ll be well aware of the cost of doing so.

The alternative to adding these isn’t “lots of non-buggy support for musl”.  It’s “less support for musl”, because there are awkward corner cases where it just isn’t possible to do things any other way.

Kind regards,

Alastair.

Re: [musl] __MUSL__ macro

[A. Wilcox]

[-- Attachment #1: Type: text/plain, Size: 2261 bytes --]

On Jul 7, 2023, at 2:14 AM, Alastair Houghton <ahoughton@apple.com> wrote:
> 
> There are two situations where the lack of macros is absolutely a problem and there isn’t a sane workaround.
> 
> 1. You’re cross compiling and the thing you’re interested in is *runtime* behaviour, so configure-style checks simply aren’t an option.  In a cross-compilation situation the only thing they can test is that a program will *build*; they can’t test its behaviour since you don’t necessarily have anywhere to run it.
> 
> 2. You’re in a header-only library, particularly one where the final environment you’re running in wasn’t available at whatever configuration time you had, assuming you had one at all.  A really good example of this is the built-in headers in Clang or GCC, but the problem also applies to things like STL or Boost.


These were two of the situations that I highlighted last year in my article on this ( https://catfox.life/2022/04/16/the-musl-preprocessor-debate/ ).

My proposal obviates the need for a PREREQ-style macro because user-facing behavioural differences would be tracked by the monotonically increasing number.  I suppose the next bikeshed would be determining when to increase and when not to, but I don’t think we should open that up here.

*As a distro maintainer* I think the backport argument is flimsy at best (I already cover that in my article) and insulting at worst.  It implies that distro maintainers aren’t capable of understanding what they are backporting.  If you do not trust distros to package musl properly, then you have two options: tell them to stop shipping it, or let them keep their broken mess.  Don’t handicap the developers and engineers of user-facing software because you don’t trust distros.

We’ve looked at bringing Swift over to Adélie before, and found that it would be too much to port by ourselves.  I’m quite excited to see upstream interest and hope that the community writ large can help out where possible.  I’m also hopeful that the maintainers at musl take the need for some sort of preprocessor macro more seriously.

Best regards,
-A.

--
A. Wilcox (they/them)
SW Engineering: C++/Rust, DevOps, POSIX, Py/Ruby
Wilcox Technologies Inc.

[-- Attachment #2: Type: text/html, Size: 2809 bytes --]

Re: [musl] __MUSL__ macro

[Alastair Houghton]

On 7 Jul 2023, at 08:30, A. Wilcox <AWilcox@Wilcox-Tech.com> wrote:
> 
> My proposal obviates the need for a PREREQ-style macro because user-facing behavioural differences would be tracked by the monotonically increasing number.  I suppose the next bikeshed would be determining when to increase and when not to, but I don’t think we should open that up here.

A single monotonically increasing number might be OK, though it does mean that there’s no way to signal a significant incompatibility by bumping the major number, depending on how the project chooses to manage versioning.  (Basically, your monotonically increasing number is my minor version number here :-))

I also don’t understand why people doing stuff like this

  https://stackoverflow.com/questions/58177815/how-to-actually-detect-musl-libc

is not seen as a problem.  To my mind, it's *far* worse than having musl’s maintainers in charge of a macro definition (perhaps alongside documentation for it that explains that you usually shouldn’t use it, which could come with copious examples of how to avoid doing the wrong thing... I didn’t provide that in my patch, but if writing that is what it takes to get it accepted, I’m game).

Kind regards,

Alastair.

Re: [musl] __MUSL__ macro

[Laurent Bercot]

  I am agnostic on the whole __MUSL__ thing (I can see the validity of
arguments on both sides) but, as a C developer often brawling with
the following issue, I have Opinions:


>1. You’re cross compiling and the thing you’re interested in is *runtime* behaviour, so configure-style checks simply aren’t an option.  In a cross-compilation situation the only thing they can test is that a program will *build*; they can’t test its behaviour since you don’t necessarily have anywhere to run it.

  This happens a lot, because there are a lot of subtly different
behaviours that can only be tested at run-time, and it's pretty annoying
indeed. The thing is, a __MUSL__ macro would not help in that case,
unless you're willing to go the extra mile and perform a herculean task
(you're probably not).

  I don't want to remember, store, and include in every package I 
publish,
a set of "right combination of feature test macros -> behaviour"
mappings. This is extremely cumbersome to use; best case, it ends up
in a giant header file with #ifdef forests, worst case, it gets
desynced across projects and becomes yet another source of headaches.

  Even GNU autoconf, the absolute reference in configuration bloatware,
does not do this. It includes an extensive set of build-time tests and
of run-time tests that work on native builds, but it does not store
predetermined run-time test values for cross-builds. Instead, on
cross-builds, autoconf *assumes* behaviours (and, obviously, gets them
wrong half the time).

  The only way to make it viable is if there were some organization that
collected, gathered and published such a dataset on a well-known site
in an easy-to-use format for every build system out there (and dear
reader, if you reflexively thought "JSON", you lost, close your mail
client and try again later). That is tremendous work, that can only
be achieved via benevolent funding and/or extensive community
cooperation. And then you'd have to convince all the build systems to
use it.
  Given the state of open source in 2023, I'm not holding my breath.

  The usual counter-argument to this proposition is "but having something
is better than nothing", and it is also wrong. We already have a lot of
somethings that should be better than nothing; but apparently, they all
suck, since people cannot stop inventing new somethings that will, too,
doubtlessly be better than nothing. This problem needs to be solved
for everyone once and for all; a new partial solution would only add to
the pile of reinvented square wheels.

  (The good thing about square wheels is that they stack nicely.)

  In short, __MUSL__ would not help with run-time tests in cross-builds,
because 1. nobody would use it, 2. the people who would use it would
use it wrong, 3. the people who use it right would be overwhelmed by
the amount of boilerplate and maintenance this requires, 4. solving
the boilerplate/maintenance thing basically starts with abolishing
capitalism and we're not going to discuss that here.

--
  Laurent

Re: [musl] __MUSL__ macro

[Jeffrey Walton]

On Fri, Jul 7, 2023 at 3:14 AM Alastair Houghton <ahoughton@apple.com> wrote:
>
> On 6 Jul 2023, at 13:17, Alex Xu <alex_y_xu@yahoo.ca> wrote:
> > [...]
> > Configure-time checks for specific functionality are a standard part of
> > writing portable C code. Preprocessor checks for specific platforms lead
> > to ifdef swamps that are still not portable to any platforms not
> > initially considered.
>
> There are two situations where the lack of macros is absolutely a problem and there isn’t a sane workaround.
>
> 1. You’re cross compiling and the thing you’re interested in is *runtime* behaviour, so configure-style checks simply aren’t an option.  In a cross-compilation situation the only thing they can test is that a program will *build*; they can’t test its behaviour since you don’t necessarily have anywhere to run it.
>
> 2. You’re in a header-only library, particularly one where the final environment you’re running in wasn’t available at whatever configuration time you had, assuming you had one at all.  A really good example of this is the built-in headers in Clang or GCC, but the problem also applies to things like STL or Boost.

A third problem is, Musl assumes a project is using Autotools or
similar. That's not always the case.

We use preprocessor signalling to enable or disable code paths at build time.

(In fact, we do not wish Autotools on our worst enemies.)

Jeff

Re: [musl] __MUSL__ macro

[Rich Felker]

On Thu, Jul 06, 2023 at 11:48:04AM +0100, Alastair Houghton wrote:
> Hi all,
> 
> Before I start, I’m aware of
> 
>   <https://www.openwall.com/lists/musl/2013/03/29/13>
> 
> but I *still* want to add __MUSL__ (see attached patch).
> 
> Let me explain what we’re doing, why we want it and why we think
> musl *should* have it. We’re trying to add support for musl to Swift
> <https://swift.org <https://swiftlang.org/>> and its attendant core
> libraries, and there are a number of things about musl that
> presently differ from other platforms/C libraries we support.
> 
> Examples include the use of `union`s in `pthread_mutex_t` et al
> (which means that we can’t write a C++ `constexpr` function that
> returns one, even if all it does is return
> `PTHREAD_MUTEX_INITIALIZER`), the fact that it doesn’t have the
> `d_namlen` member of `struct dirent`, or the fact that `dladdr()` is
> a no-op when statically linked.

This has nothing to do with being on musl. This has to do with writing
semantically incorrect code that does not meet the requirements of the
specification.

pthread_mutex_t is not a value type. It's an object. You cannot
initialize a pthread_mutex_t in one context and copy/assign/return it
to be used somewhere else as a value. Only the actual object
initialized may be used. For example, it may even contain references
to its own address. Whatever implementations you think this works on
on probably to not have a contract to make it work; you're writing to
something that you observed working on your system. And in that case,
once you've read the relevant source, you need to hard-code the unsafe
code to be used *only on that specific implementation and version of
it*, not assume musl is the one doing something odd/different because
you hack doesn't work here.

Rich

Re: [musl] __MUSL__ macro

[Alastair Houghton]

On 7 Jul 2023, at 13:47, Rich Felker <dalias@libc.org> wrote:
> 
> pthread_mutex_t is not a value type. It's an object. You cannot

[snip]

This is a somewhat irrelevant distraction and I rather wish I hadn’t mentioned that as an example of odd behaviour.  I’m well aware that you cannot copy or assign `pthread_mutex_t` values in general (and I understand the reasons why).

Please can we instead focus on the issue of whether or not musl should have `__MUSL__` and `__MUSL_MINOR__`.

Kind regards,

Alastair.

Re: [musl] __MUSL__ macro

[Rich Felker]

On Fri, Jul 07, 2023 at 08:14:11AM +0100, Alastair Houghton wrote:
> On 6 Jul 2023, at 13:17, Alex Xu <alex_y_xu@yahoo.ca> wrote:
> > 
> >> Examples include the use of `union`s in `pthread_mutex_t` et al
> >> (which means that we can’t write a C++ `constexpr` function that
> >> returns one, even if all it does is return
> >> `PTHREAD_MUTEX_INITIALIZER`),
> > 
> > The issue is that some of the members are volatile, not the union
> > itself.
> 
> ....
> 
> Arguing about the details of individual issues is an unnecessary
> distraction, honestly.

Except here it's really not, because the example you brought up where
you couldn't do detection was where your code was doing something very
wrong.

If you can fix your code not to do that wrong thing, it can be fixed
not to do it anywhere.

If you can't, knowing it's "not going to work because you're on
__MUSL__" does not help you. The compiler error is just as good.

> They were just OTOH examples of oddities that
> we’ve come across so far, and yes, I agree that in some cases there
> are potentially better fixes for them than preprocessor shenanigans
> - and where those are available I’m totally onboard with doing that
> instead, because I *entirely agree* that it’s better to avoid a huge
> mass of `#if` conditions in code. There’s a very good chance that
> any I add in the various places I might be interested in adding them
> will end up being my responsibility to maintain, so that, if nothing
> else, is a good motivation for minimising them.
> 
> >> or the fact that `dladdr()` is a no-op when statically linked.
> > 
> > When statically linked, the dynamic linking interface doesn't work at
> > all though? And furthermore, even if it did, what would it return? If
> > you're (once again) trying to do stack traces manually, try libunwind?
> 
> :-D Oh, the irony. One of the places where this behaviour turns out
> to be a problem is *in the tests for libunwind*. (That can be solved
> in another way without resorting to macros, however.)

That dladdr is a no-op rather than always returning the single
static-linked object is a bug that would be worth reporting. Having
someone leave it unreported and instead hard-coding "musl dladdr
doesn't work" is not in our interests.

> > I don't understand, don't you know at compile time whether you're
> > linking statically or dynamically?
> 
> Not always, no (see situation 2 below).
> 
> > Configure-time checks for specific functionality are a standard part of
> > writing portable C code. Preprocessor checks for specific platforms lead
> > to ifdef swamps that are still not portable to any platforms not
> > initially considered.
> 
> There are two situations where the lack of macros is absolutely a
> problem and there isn’t a sane workaround.
> 
> 1. You’re cross compiling and the thing you’re interested in is
> *runtime* behaviour, so configure-style checks simply aren’t an
> option. In a cross-compilation situation the only thing they can
> test is that a program will *build*; they can’t test its behaviour
> since you don’t necessarily have anywhere to run it.
> 
> 2. You’re in a header-only library, particularly one where the final
> environment you’re running in wasn’t available at whatever
> configuration time you had, assuming you had one at all. A really
> good example of this is the built-in headers in Clang or GCC, but
> the problem also applies to things like STL or Boost.

In neither of these cases do you want to know "this is musl", which
would at best help you make wrong platform-specific assumptions that
might have been true at the time you wrote the code, but which aren't
true later when someone is using it.

Almost every single time somebody has made a big deal about requesting
__MUSL__, whatever it was they wanted to do would have broken on a
later, if not the very next, version of, musl, because what they were
really getting at was that something they wanted to use didn't work
the way they hoped (often because of a bug or extended functionality
that wasn't yet implemented but was reasonable to implement) and they
were trying to hard-code their idea that "musl doesn't work" rather
than "find out if this thing I want to use is available".

I already have a proposal for a *granular* version of the latter, that
doesn't depend on hard-coding assumptions about particular
implementations/versions, initially posted in the libc-coord thread at
https://www.openwall.com/lists/libc-coord/2020/04/22/1. Unfortunately,
while everyone seems to agree "this is a good idea", nobody is rushing
to actually do it. Proposals for an initial list of what to include
here, that could be reviewed and pruned, would be a lot more useful
than re-proposing something that's been rejected as harmful multiple
times already.

Rich

Re: [musl] __MUSL__ macro

[Alastair Houghton]

On 7 Jul 2023, at 14:53, Rich Felker <dalias@libc.org> wrote:
> 
> On Fri, Jul 07, 2023 at 08:14:11AM +0100, Alastair Houghton wrote:
> 
>> Arguing about the details of individual issues is an unnecessary
>> distraction, honestly.
> 
> Except here it's really not, because the example you brought up where
> you couldn't do detection was where your code was doing something very
> wrong.

In point of fact, I didn’t even try to detect musl in that case and instead changed the code to work differently.  So it really is irrelevant and again, I wish I hadn’t mentioned it because now we’re spending time on that rather than talking about the actual issue.

> That dladdr is a no-op rather than always returning the single
> static-linked object is a bug that would be worth reporting. Having
> someone leave it unreported and instead hard-coding "musl dladdr
> doesn't work" is not in our interests.

The bug report instructions appear to say that an email to this list is sufficient.  Does this need to be a separate message?  I’m quite happy to report it as a bug however you please.

>> There are two situations where the lack of macros is absolutely a
>> problem and there isn’t a sane workaround.
>> 
>> 1. You’re cross compiling...

[snip]

>> 
>> 2. You’re in a header-only library, ...

[snip]

> In neither of these cases do you want to know "this is musl", which
> would at best help you make wrong platform-specific assumptions that
> might have been true at the time you wrote the code, but which aren't
> true later when someone is using it.

Nevertheless, “this is musl” (or better, “this is musl version x.y”) is a reasonable proxy in the absence of the availability of any finer grained information.  I entirely agree that it isn’t perfect, and adding such a test potentially necessitates future maintenance when musl is updated - but that future maintenance is on the owner of the project that added such a test, not on you.  Many projects have already made a conscious choice to accept those kinds of tests and the associated maintenance overhead in the context of other libraries and environments.  This is definitely a balancing act; explicitly testing for particular functionality is better, on that we agree, but it does have overheads and problems of its own and sometimes it makes sense to try to do things in the preprocessor or with compile-time tests instead of having a complex and often slow configuration step.

Without these kinds of macros being available, it’s difficult to make perfectly reasonable tests (like “don’t use this function when building for musl 1.1.7 because it has a known bug”, or “don’t use this function because it isn’t supported on musl (yet)”), and people will just do what various posts suggest on Stack Overflow, which is far worse because they’re relying on hacky methods of detecting musl that are totally outside of your control.

If it is your position that you are not going to add this, ever, then fine.  I’ve made my argument and I think that’s the wrong decision but it’s your choice to make.

Kind regards,

Alastair.

Re: [musl] __MUSL__ macro

[Markus Wichmann]

Am Fri, Jul 07, 2023 at 02:14:30PM +0100 schrieb Alastair Houghton:
> This is a somewhat irrelevant distraction and I rather wish I hadn’t
> mentioned that as an example of odd behaviour.  I’m well aware that
> you cannot copy or assign `pthread_mutex_t` values in general (and I
> understand the reasons why).
>
> Please can we instead focus on the issue of whether or not musl should
> have `__MUSL__` and `__MUSL_MINOR__`.
>

The counter-examples are not irrelevant. That is precisely the point.
Nobody advocating for implementation identification macros has so far
given a valid reason to do so. Every single one so far has turned out to
be spurious. Well, I tell a lie, there is one case with a shadow of
reason behind it: Header-only libraries. But maybe the problem with
those is trying to be a header-only library.

If you have different implementations with different runtime behavior,
check whether the behavior is acceptable from the specification. If it
is, write code that can accept the behavior. If it is not, write a bug
report.

Very often I am astonished that the problem presented is claimed to have
the __MUSL__ macro as a solution. I am reminded of a Stack Overflow
question where someone wanted to identify musl, because musl doesn't
have a trustworthy vfork(). Pressed on what exactly he meant, the poster
said that depending on version and architecture, calling vfork() with
musl actually results in it calling fork(). OK, that was his concern.
His solution? If he found he was running on Apple or musl, he wanted to
call fork() instead of vfork(). Why he would not just always call
vfork() if the claimed untrustworthy behavior was also his remedy is a
question the guy skillfully avoided an answer to.

In your case, apparently there is a way to deal with failing dladdr().
So why don't you put the code for that in the failure path for the call,
instead of into "#ifdef __MUSL__"? That way, even unknown
implementations would be supported.

For header-only libraries, the customer could configure them at build
time. I have so far not figured out why people that write programs for a
living cannot be expected to fill out a config.h template. There was a
person here before who wanted a macro to identify that qsort_r() is
available, and I told him much the same thing, and never got a
satisfactory answer to the above query. Also, he already had his own
fallback sorting algorithm, so the portable solution was just to call
that, and then the whole need evaporated.

I remain in staunch opposition to identification macros, because those
have so far always - no matter how benevolent they might have seemed at
the beginning - lead to #ifdef hell of levels Dante couldn't dream of.
They lead to people writing bad code and making bad assumptions.

None of this matters one bit, because Rich is God as far as musl is
concerned, and he has not weighed in yet. However, in the past he has
spoken out against these macros, and I doubt your arguments have
convinced him. They haven't convinced me, at any rate.

Ciao,
Markus

Re: [musl] __MUSL__ macro

[Markus Wichmann]

Am Fri, Jul 07, 2023 at 04:19:14PM +0200 schrieb Markus Wichmann:
> None of this matters one bit, because Rich is God as far as musl is
> concerned, and he has not weighed in yet. However, in the past he has
> spoken out against these macros, and I doubt your arguments have
> convinced him. They haven't convinced me, at any rate.
>

Goddangit, for some reason Rich's messages had only showed up the moment
I hit send. Not sure how exactly that worked. Seems like I guessed
right, though.

Ciao,
Markus

Re: [musl] __MUSL__ macro

[Alastair Houghton]

On 7 Jul 2023, at 15:19, Markus Wichmann <nullplan@gmx.net> wrote:
> 
> The counter-examples are not irrelevant. That is precisely the point.
> Nobody advocating for implementation identification macros has so far
> given a valid reason to do so. Every single one so far has turned out to
> be spurious. Well, I tell a lie, there is one case with a shadow of
> reason behind it: Header-only libraries.

And cross-compilation when you can’t detect runtime behaviour at configuration time.

Those two are actually what I care about, as it happens.

> I remain in staunch opposition to identification macros, ... [snip]

So I gather.

> None of this matters one bit, because Rich is God as far as musl is
> concerned, and he has not weighed in yet.

:-)  I think you crossed over with him there.

It’s fine.  There’s no point in arguing; we clearly aren’t going to agree here.  I think it’s a shame because I think it makes things unnecessarily worse for everyone than they have to be (people will still try to detect musl, as in that Stack Overflow post, but they’ll do so in a less robust manner, and it makes supporting cross-compilation for musl targets and header only libraries that much harder).

Kind regards,

Alastair.

Re: [musl] __MUSL__ macro

[Andrew Bell]

[-- Attachment #1: Type: text/plain, Size: 1214 bytes --]

On Fri, Jul 7, 2023 at 10:19 AM Markus Wichmann <nullplan@gmx.net> wrote:

> Am Fri, Jul 07, 2023 at 02:14:30PM +0100 schrieb Alastair Houghton:
> > This is a somewhat irrelevant distraction and I rather wish I hadn’t
> > mentioned that as an example of odd behaviour.  I’m well aware that
> > you cannot copy or assign `pthread_mutex_t` values in general (and I
> > understand the reasons why).
> >
> > Please can we instead focus on the issue of whether or not musl should
> > have `__MUSL__` and `__MUSL_MINOR__`.
> >
>
> The counter-examples are not irrelevant. That is precisely the point.
> Nobody advocating for implementation identification macros has so far
> given a valid reason to do so.
>

I think you can say this and still add the macros.

Perhaps it's true that nobody has valid reasons, but the fact that it comes
up repeatedly says that some people don't understand this or perhaps they
have valid reasons yet to be considered.

I guess I don't understand the opposition -- is there any downside to musl
to having the macros defined, necessary or not? (I'm saying this as a
minimalist, so I'm surprising myself here.)

-- 
Andrew Bell
andrew.bell.ia@gmail.com

[-- Attachment #2: Type: text/html, Size: 1814 bytes --]

Re: [musl] __MUSL__ macro

[i262jq]

On Fri, Jul 07, 2023 at 02:14:30PM +0100, Alastair Houghton wrote:
> issue of whether or not musl should have `__MUSL__` and `__MUSL_MINOR__`.

Introducing such macros would allow and encourage reliance on
implementation details instead of following the standards.

This means encouraging non-portable programming.

I think this already have been said on this list.

In other words, the question is not whether a change would make sense
for specific cases, but whether the effect on the _rest_ of the usage
of the library would be acceptable.

IMHO the effect would be to undermine portability and maintainability
in the long run, for yet unknown / unlimited set of softwares to be
maintained or written in the future.

Kudos to musl developers for a firm stance against misfeatures, also
ones which non-portable (possibly _right now and here_ attractive)
solutions would have a use for.

/i262jq

Re: [musl] __MUSL__ macro

[Markus Wichmann]

Am Fri, Jul 07, 2023 at 11:02:21AM -0400 schrieb Andrew Bell:
> I guess I don't understand the opposition -- is there any downside to musl
> to having the macros defined, necessary or not? (I'm saying this as a
> minimalist, so I'm surprising myself here.)
>

Yes, it makes people write worse code. Not making the macros available
makes people write more portable code, which is a good thing. Sometimes
people have to be made to think for a moment, and broken out of their
rut, to get them to do the right thing.

There is also the issue of what exactly the macros mean. Between
distribution patches and backports, a version number does not
necessarily map to a feature or bug set. And musl does not want to have
any quirks, it wants to just be a POSIX implementation. So what
specialties are supposed to be kept in mind when the musl macro is
defined? Keep in mind that __GNUC__ also does not just mean gcc anymore.
These things keep going off the rails all the time.

Ciao,
Markus

Re: [musl] __MUSL__ macro

[Andrew Bell]

[-- Attachment #1: Type: text/plain, Size: 1557 bytes --]

On Fri, Jul 7, 2023 at 11:20 AM Markus Wichmann <nullplan@gmx.net> wrote:

> Am Fri, Jul 07, 2023 at 11:02:21AM -0400 schrieb Andrew Bell:
> > I guess I don't understand the opposition -- is there any downside to
> musl
> > to having the macros defined, necessary or not? (I'm saying this as a
> > minimalist, so I'm surprising myself here.)
> >
>
> Yes, it makes people write worse code. Not making the macros available
> makes people write more portable code, which is a good thing. Sometimes
> people have to be made to think for a moment, and broken out of their
> rut, to get them to do the right thing.
>

I guess I just don't think this is the job of a library -- to be the
arbiter of other people's programming. I may have seen some code inside
musl that I personally don't like ;)


> There is also the issue of what exactly the macros mean. Between
> distribution patches and backports, a version number does not
> necessarily map to a feature or bug set. And musl does not want to have
> any quirks, it wants to just be a POSIX implementation. So what
> specialties are supposed to be kept in mind when the musl macro is
> defined?


I understand if it can't be done well, but I would hope that it's not a big
deal to do. I do appreciate the adherence to standards and it seems
perfectly fine to respond to questions/proposals with "that's not the way
the standard works." But if this is easy, I just don't see the downside
from the standpoint of maintaining musl.

-- 
Andrew Bell
andrew.bell.ia@gmail.com

[-- Attachment #2: Type: text/html, Size: 2266 bytes --]

Re: [musl] __MUSL__ macro

[Alastair Houghton]

On 7 Jul 2023, at 16:19, Markus Wichmann <nullplan@gmx.net> wrote:
> 
> Yes, it makes people write worse code. Not making the macros available
> makes people write more portable code, which is a good thing.

The people who would have misused the macros will simply find another way to detect musl that you’ll like even less (look at the Stack Overflow post we both mentioned; it’s literally the top answer).

Not making the macros available doesn’t stop people writing bad code, nor does making them available force anyone to write bad code.

Anyway, I’ve said enough on the subject.

Kind regards,

Alastair.

Re: [musl] __MUSL__ macro

[Rich Felker]

On Fri, Jul 07, 2023 at 04:34:37PM +0100, Alastair Houghton wrote:
> On 7 Jul 2023, at 16:19, Markus Wichmann <nullplan@gmx.net> wrote:
> > 
> > Yes, it makes people write worse code. Not making the macros available
> > makes people write more portable code, which is a good thing.
> 
> The people who would have misused the macros will simply find
> another way to detect musl that you’ll like even less (look at the
> Stack Overflow post we both mentioned; it’s literally the top
> answer).

The difference is whose fault it is, and who users blame, when they
do.

If we make a macro that says "you can now rely on implementation
details of musl version x.y.z" and people do that, then it ceases to
work in a future version, it's our fault, and people rightly blame us.

If people do stupid hacks explicitly against our warnings, and against
the specification for the interfaces they're using, and it breaks,
it's clearly their fault.

We have a really good history of being consistent on this, and as a
result, of users and projects taking us seriously and making efforts
to improve the overall ecosystem. Throwing that away would be really
counterproductive.

Rich

Re: [musl] __MUSL__ macro

[Alastair Houghton]

On 7 Jul 2023, at 16:45, Rich Felker <dalias@libc.org> wrote:
> 
> If we make a macro that says "you can now rely on implementation
> details of musl version x.y.z" and people do that, then it ceases to
> work in a future version, it's our fault, and people rightly blame us.

I don’t think that’s true at all - people tend to blame whoever is in front of them, which is probably not going to be musl but something higher up the stack.  In many cases projects can detect the musl version already, either at configuration time by examining the version string, or by specifying it as part of their build.  Both of those things would have *exactly* that same downside, and the only way to remove that would be to do away with a version number altogether... which nobody is seriously proposing, right?

The *only* thing you’re actually preventing here is detecting the musl version *in the preprocessor*.  And only in situations where the project itself can’t somehow supply a relevant macro definition.

Kind regards,

Alastair.