* [musl] __MUSL__ macro @ 2023-07-06 10:48 Alastair Houghton 2023-07-06 12:17 ` Alex Xu 2023-07-07 12:47 ` Rich Felker 0 siblings, 2 replies; 22+ messages in thread From: Alastair Houghton @ 2023-07-06 10:48 UTC (permalink / raw) To: musl [-- Attachment #1: Type: text/plain, Size: 2272 bytes --] Hi all, Before I start, I’m aware of <https://www.openwall.com/lists/musl/2013/03/29/13> but I *still* want to add __MUSL__ (see attached patch). Let me explain what we’re doing, why we want it and why we think musl *should* have it. We’re trying to add support for musl to Swift <https://swift.org <https://swiftlang.org/>> and its attendant core libraries, and there are a number of things about musl that presently differ from other platforms/C libraries we support. Examples include the use of `union`s in `pthread_mutex_t` et al (which means that we can’t write a C++ `constexpr` function that returns one, even if all it does is return `PTHREAD_MUTEX_INITIALIZER`), the fact that it doesn’t have the `d_namlen` member of `struct dirent`, or the fact that `dladdr()` is a no-op when statically linked. I’m aware that there are other solutions for some of these cases - but for instance the `dladdr()` issue is not something you can test at configuration time when cross-compiling, since there’s no way to know what the result would be without running a program on the target system (which you don’t necessarily have available at configuration time in that case). Likewise, configuration time tests are slow and aren’t even a thing in some of the projects we need to add support in, whereas we do generally have the ability to use the preprocessor. You might say we should just bite the bullet and add configuration steps to all of the other projects, but that is honestly a non-starter; the owners of those projects are not likely to accept patches to add such a thing, even as a short-term fix, and the patches to do so would be non-trivial in a number of cases also. In the longer term we don’t want higher level projects to need to do this kind of thing and the lower level ones could in many cases do some kind of configuration time testing, but that’s a longer term goal and it doesn’t help us to add musl support in the interim, which we would like to do. I’ve attached a proposed patch that adds `__MUSL__` (set to the major version) and `__MUSL_MINOR__` (set to the minor version) as well as `__MUSL_PREREQ()` (which works the same way `__GLIBC_PREREQ()` does). Kind regards, Alastair Houghton [-- Attachment #2: musl-version.patch --] [-- Type: application/octet-stream, Size: 2772 bytes --] diff --git a/Makefile b/Makefile index e8cc4436..ea91b9be 100644 --- a/Makefile +++ b/Makefile @@ -34,7 +34,7 @@ CRT_OBJS = $(filter obj/crt/%,$(ALL_OBJS)) AOBJS = $(LIBC_OBJS) LOBJS = $(LIBC_OBJS:.o=.lo) -GENH = obj/include/bits/alltypes.h obj/include/bits/syscall.h +GENH = obj/include/bits/alltypes.h obj/include/bits/syscall.h obj/include/features.h GENH_INT = obj/src/internal/version.h IMPH = $(addprefix $(srcdir)/, src/internal/stdio_impl.h src/internal/pthread_impl.h src/internal/locale_impl.h src/internal/libc.h) @@ -95,6 +95,17 @@ $(ALL_LIBS) $(ALL_TOOLS) $(ALL_OBJS) $(ALL_OBJS:%.o=%.lo) $(GENH) $(GENH_INT): | $(OBJ_DIRS): mkdir -p $@ +MUSL_VERSION:=$(shell cd $(srcdir) && sh tools/version.sh) +empty:= +space:=$(empty) $(empty) +musl_ver_parts:=$(subst -git-,$(space),$(subst .,$(space),$(MUSL_VERSION))) +MUSL_MAJOR:=$(word 1, $(musl_ver_parts)) +MUSL_MINOR:=$(word 2, $(musl_ver_parts)) +MUSL_PATCH:=$(word 3, $(musl_ver_parts)) + +obj/include/features.h: $(srcdir)/include/features.h.in $(wildcard $(srcdir)/VERSION $(srcdir)/.git) + sed -e "s/@MUSL_MAJOR@/$(MUSL_MAJOR)/g;s/@MUSL_MINOR@/$(MUSL_MINOR)/g;s/@MUSL_PATCH@/$(MUSL_PATCH)/g" $< > $@ + obj/include/bits/alltypes.h: $(srcdir)/arch/$(ARCH)/bits/alltypes.h.in $(srcdir)/include/alltypes.h.in $(srcdir)/tools/mkalltypes.sed sed -f $(srcdir)/tools/mkalltypes.sed $(srcdir)/arch/$(ARCH)/bits/alltypes.h.in $(srcdir)/include/alltypes.h.in > $@ @@ -103,7 +114,7 @@ obj/include/bits/syscall.h: $(srcdir)/arch/$(ARCH)/bits/syscall.h.in sed -n -e s/__NR_/SYS_/p < $< >> $@ obj/src/internal/version.h: $(wildcard $(srcdir)/VERSION $(srcdir)/.git) - printf '#define VERSION "%s"\n' "$$(cd $(srcdir); sh tools/version.sh)" > $@ + printf '#define VERSION "%s"\n' "$(MUSL_VERSION)" > $@ obj/src/internal/version.o obj/src/internal/version.lo: obj/src/internal/version.h @@ -209,6 +220,9 @@ $(DESTDIR)$(includedir)/bits/%: obj/include/bits/% $(DESTDIR)$(includedir)/%: $(srcdir)/include/% $(INSTALL) -D -m 644 $< $@ +$(DESTDIR)$(includedir)/%: obj/include/% + $(INSTALL) -D -m 644 $< $@ + $(DESTDIR)$(LDSO_PATHNAME): $(DESTDIR)$(libdir)/libc.so $(INSTALL) -D -l $(libdir)/libc.so $@ || true diff --git a/include/features.h b/include/features.h.in similarity index 80% rename from include/features.h rename to include/features.h.in index 85cfb72a..07662804 100644 --- a/include/features.h +++ b/include/features.h.in @@ -1,6 +1,13 @@ #ifndef _FEATURES_H #define _FEATURES_H +#define __MUSL__ @MUSL_MAJOR@ +#define __MUSL_MINOR__ @MUSL_MINOR@ +#define __MUSL_PATCH__ @MUSL_PATCH@ + +#define __MUSL_PREREQ(maj, min) \ + ((__MUSL__ << 16) + __MUSL_MINOR >= ((maj) << 16) + (min)) + #if defined(_ALL_SOURCE) && !defined(_GNU_SOURCE) #define _GNU_SOURCE 1 #endif ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [musl] __MUSL__ macro 2023-07-06 10:48 [musl] __MUSL__ macro Alastair Houghton @ 2023-07-06 12:17 ` Alex Xu 2023-07-06 16:26 ` Szabolcs Nagy 2023-07-07 7:14 ` Alastair Houghton 2023-07-07 12:47 ` Rich Felker 1 sibling, 2 replies; 22+ messages in thread From: Alex Xu @ 2023-07-06 12:17 UTC (permalink / raw) To: musl Quoting Alastair Houghton (2023-07-06 06:48:04) > Hi all, > > Before I start, I’m aware of > > <https://www.openwall.com/lists/musl/2013/03/29/13> > > but I *still* want to add __MUSL__ (see attached patch). > > Let me explain what we’re doing, why we want it and why we think musl *should* have it. We’re trying to add support for musl to Swift <https://swift.org <https://swiftlang.org/>> and its attendant core libraries, and there are a number of things about musl that presently differ from other platforms/C libraries we support. All of these seem to fall at least suspiciously closely to "the usage case was badly wrong"... > Examples include the use of `union`s in `pthread_mutex_t` et al (which means that we can’t write a C++ `constexpr` function that returns one, even if all it does is return `PTHREAD_MUTEX_INITIALIZER`), The issue is that some of the members are volatile, not the union itself. As I understand, though, constexpr is useful to evaluate certain expressions at compile time, such as if (x < 10) or int arr[getlen(x)]. You can't do any arithmetic with pthread_mutex_t, and the only valid value at compile time is PTHREAD_MUTEX_INITIALIZER though, so what is the use case? > the fact that it doesn’t have the `d_namlen` member of `struct dirent` You can use strlen(d->d_name) instead? This seems like exactly the reason why __MUSL__ should absolutely not be added, because it incentivizes individual checks for endless platforms instead of writing portable code. If strlen(d->d_name) is too slow in some context, then an argument should be made to add d_namlen. glibc even already has a macro _DIRENT_HAVE_D_NAMLEN that could be used to signal support for this rather than hardcoding specific musl versions. > or the fact that `dladdr()` is a no-op when statically linked. When statically linked, the dynamic linking interface doesn't work at all though? And furthermore, even if it did, what would it return? If you're (once again) trying to do stack traces manually, try libunwind? > I’m aware that there are other solutions for some of these cases - but for instance the `dladdr()` issue is not something you can test at configuration time when cross-compiling, since there’s no way to know what the result would be without running a program on the target system (which you don’t necessarily have available at configuration time in that case). I don't understand, don't you know at compile time whether you're linking statically or dynamically? > Likewise, configuration time tests are slow and aren’t even a thing in some of the projects we need to add support in, whereas we do generally have the ability to use the preprocessor. Configure-time checks for specific functionality are a standard part of writing portable C code. Preprocessor checks for specific platforms lead to ifdef swamps that are still not portable to any platforms not initially considered. > You might say we should just bite the bullet and add configuration steps to all of the other projects, but that is honestly a non-starter; the owners of those projects are not likely to accept patches to add such a thing, even as a short-term fix, and the patches to do so would be non-trivial in a number of cases also. In the longer term we don’t want higher level projects to need to do this kind of thing and the lower level ones could in many cases do some kind of configuration time testing, but that’s a longer term goal and it doesn’t help us to add musl support in the interim, which we would like to do. > > I’ve attached a proposed patch that adds `__MUSL__` (set to the major version) and `__MUSL_MINOR__` (set to the minor version) as well as `__MUSL_PREREQ()` (which works the same way `__GLIBC_PREREQ()` does). In theory, there could possibly be some reason why __MUSL__ could be useful in some cases. However, your examples appear to be exactly why __MUSL__ should not be added because it leads to bad code. Cheers, Alex. ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [musl] __MUSL__ macro 2023-07-06 12:17 ` Alex Xu @ 2023-07-06 16:26 ` Szabolcs Nagy 2023-07-07 7:14 ` Alastair Houghton 1 sibling, 0 replies; 22+ messages in thread From: Szabolcs Nagy @ 2023-07-06 16:26 UTC (permalink / raw) To: Alex Xu; +Cc: musl * Alex Xu <alex_y_xu@yahoo.ca> [2023-07-06 08:17:48 -0400]: > Quoting Alastair Houghton (2023-07-06 06:48:04) > > Hi all, > > > > Before I start, I’m aware of > > > > <https://www.openwall.com/lists/musl/2013/03/29/13> > > > > but I *still* want to add __MUSL__ (see attached patch). > > > > Let me explain what we’re doing, why we want it and why we think musl *should* have it. We’re trying to add support for musl to Swift <https://swift.org <https://swiftlang.org/>> and its attendant core libraries, and there are a number of things about musl that presently differ from other platforms/C libraries we support. > > All of these seem to fall at least suspiciously closely to "the usage > case was badly wrong"... the problem is not that the usage is wrong, but that these can change between different versions of musl (within abi constraints) and header changes can be backported by distros. so __MUSL__ cannot solve these issues. ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [musl] __MUSL__ macro 2023-07-06 12:17 ` Alex Xu 2023-07-06 16:26 ` Szabolcs Nagy @ 2023-07-07 7:14 ` Alastair Houghton 2023-07-07 7:30 ` A. Wilcox ` (3 more replies) 1 sibling, 4 replies; 22+ messages in thread From: Alastair Houghton @ 2023-07-07 7:14 UTC (permalink / raw) To: musl On 6 Jul 2023, at 13:17, Alex Xu <alex_y_xu@yahoo.ca> wrote: > >> Examples include the use of `union`s in `pthread_mutex_t` et al (which means that we can’t write a C++ `constexpr` function that returns one, even if all it does is return `PTHREAD_MUTEX_INITIALIZER`), > > The issue is that some of the members are volatile, not the union > itself. ... Arguing about the details of individual issues is an unnecessary distraction, honestly. They were just OTOH examples of oddities that we’ve come across so far, and yes, I agree that in some cases there are potentially better fixes for them than preprocessor shenanigans - and where those are available I’m totally onboard with doing that instead, because I *entirely agree* that it’s better to avoid a huge mass of `#if` conditions in code. There’s a very good chance that any I add in the various places I might be interested in adding them will end up being my responsibility to maintain, so that, if nothing else, is a good motivation for minimising them. >> or the fact that `dladdr()` is a no-op when statically linked. > > When statically linked, the dynamic linking interface doesn't work at > all though? And furthermore, even if it did, what would it return? If > you're (once again) trying to do stack traces manually, try libunwind? :-D Oh, the irony. One of the places where this behaviour turns out to be a problem is *in the tests for libunwind*. (That can be solved in another way without resorting to macros, however.) > I don't understand, don't you know at compile time whether you're > linking statically or dynamically? Not always, no (see situation 2 below). > Configure-time checks for specific functionality are a standard part of > writing portable C code. Preprocessor checks for specific platforms lead > to ifdef swamps that are still not portable to any platforms not > initially considered. There are two situations where the lack of macros is absolutely a problem and there isn’t a sane workaround. 1. You’re cross compiling and the thing you’re interested in is *runtime* behaviour, so configure-style checks simply aren’t an option. In a cross-compilation situation the only thing they can test is that a program will *build*; they can’t test its behaviour since you don’t necessarily have anywhere to run it. 2. You’re in a header-only library, particularly one where the final environment you’re running in wasn’t available at whatever configuration time you had, assuming you had one at all. A really good example of this is the built-in headers in Clang or GCC, but the problem also applies to things like STL or Boost. > On 6 Jul 2023, at 17:26, Szabolcs Nagy <nsz@port70.net> wrote: > > the problem is not that the usage is wrong, but that > these can change between different versions of musl > (within abi constraints) and header changes can be > backported by distros. > > so __MUSL__ cannot solve these issues. That’s why `__MUSL__` should evaluate to the major version number and `__MUSL_MINOR__` to the minor version number, and why we should have `__MUSL_PREREQ()`, just like Glibc, which, by the way, has the exact same problems you raise (things can change, and distros can backport things). If some third-party package wishes to test `__MUSL__`, it’s up to the maintainers of that package to make sure that they perform appropriate version checks. And yes, if they’re working around a bug or musl changes its behaviour to add a feature that wasn’t there, they may have to update their code from `#ifndef __MUSL__` to `#if __MUSL_PREREQ(x,y)` or similar. Package maintainers that make these kinds of tests are used to having to do that and if they’re willing to accept macro-based tests, then they’ll be well aware of the cost of doing so. The alternative to adding these isn’t “lots of non-buggy support for musl”. It’s “less support for musl”, because there are awkward corner cases where it just isn’t possible to do things any other way. Kind regards, Alastair. ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [musl] __MUSL__ macro 2023-07-07 7:14 ` Alastair Houghton @ 2023-07-07 7:30 ` A. Wilcox 2023-07-07 8:24 ` Alastair Houghton 2023-07-07 11:20 ` Laurent Bercot ` (2 subsequent siblings) 3 siblings, 1 reply; 22+ messages in thread From: A. Wilcox @ 2023-07-07 7:30 UTC (permalink / raw) To: musl [-- Attachment #1: Type: text/plain, Size: 2261 bytes --] On Jul 7, 2023, at 2:14 AM, Alastair Houghton <ahoughton@apple.com> wrote: > > There are two situations where the lack of macros is absolutely a problem and there isn’t a sane workaround. > > 1. You’re cross compiling and the thing you’re interested in is *runtime* behaviour, so configure-style checks simply aren’t an option. In a cross-compilation situation the only thing they can test is that a program will *build*; they can’t test its behaviour since you don’t necessarily have anywhere to run it. > > 2. You’re in a header-only library, particularly one where the final environment you’re running in wasn’t available at whatever configuration time you had, assuming you had one at all. A really good example of this is the built-in headers in Clang or GCC, but the problem also applies to things like STL or Boost. These were two of the situations that I highlighted last year in my article on this ( https://catfox.life/2022/04/16/the-musl-preprocessor-debate/ ). My proposal obviates the need for a PREREQ-style macro because user-facing behavioural differences would be tracked by the monotonically increasing number. I suppose the next bikeshed would be determining when to increase and when not to, but I don’t think we should open that up here. *As a distro maintainer* I think the backport argument is flimsy at best (I already cover that in my article) and insulting at worst. It implies that distro maintainers aren’t capable of understanding what they are backporting. If you do not trust distros to package musl properly, then you have two options: tell them to stop shipping it, or let them keep their broken mess. Don’t handicap the developers and engineers of user-facing software because you don’t trust distros. We’ve looked at bringing Swift over to Adélie before, and found that it would be too much to port by ourselves. I’m quite excited to see upstream interest and hope that the community writ large can help out where possible. I’m also hopeful that the maintainers at musl take the need for some sort of preprocessor macro more seriously. Best regards, -A. -- A. Wilcox (they/them) SW Engineering: C++/Rust, DevOps, POSIX, Py/Ruby Wilcox Technologies Inc. [-- Attachment #2: Type: text/html, Size: 2809 bytes --] ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [musl] __MUSL__ macro 2023-07-07 7:30 ` A. Wilcox @ 2023-07-07 8:24 ` Alastair Houghton 0 siblings, 0 replies; 22+ messages in thread From: Alastair Houghton @ 2023-07-07 8:24 UTC (permalink / raw) To: musl On 7 Jul 2023, at 08:30, A. Wilcox <AWilcox@Wilcox-Tech.com> wrote: > > My proposal obviates the need for a PREREQ-style macro because user-facing behavioural differences would be tracked by the monotonically increasing number. I suppose the next bikeshed would be determining when to increase and when not to, but I don’t think we should open that up here. A single monotonically increasing number might be OK, though it does mean that there’s no way to signal a significant incompatibility by bumping the major number, depending on how the project chooses to manage versioning. (Basically, your monotonically increasing number is my minor version number here :-)) I also don’t understand why people doing stuff like this https://stackoverflow.com/questions/58177815/how-to-actually-detect-musl-libc is not seen as a problem. To my mind, it's *far* worse than having musl’s maintainers in charge of a macro definition (perhaps alongside documentation for it that explains that you usually shouldn’t use it, which could come with copious examples of how to avoid doing the wrong thing... I didn’t provide that in my patch, but if writing that is what it takes to get it accepted, I’m game). Kind regards, Alastair. ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [musl] __MUSL__ macro 2023-07-07 7:14 ` Alastair Houghton 2023-07-07 7:30 ` A. Wilcox @ 2023-07-07 11:20 ` Laurent Bercot 2023-07-07 11:45 ` Jeffrey Walton 2023-07-07 13:53 ` Rich Felker 3 siblings, 0 replies; 22+ messages in thread From: Laurent Bercot @ 2023-07-07 11:20 UTC (permalink / raw) To: musl I am agnostic on the whole __MUSL__ thing (I can see the validity of arguments on both sides) but, as a C developer often brawling with the following issue, I have Opinions: >1. You’re cross compiling and the thing you’re interested in is *runtime* behaviour, so configure-style checks simply aren’t an option. In a cross-compilation situation the only thing they can test is that a program will *build*; they can’t test its behaviour since you don’t necessarily have anywhere to run it. This happens a lot, because there are a lot of subtly different behaviours that can only be tested at run-time, and it's pretty annoying indeed. The thing is, a __MUSL__ macro would not help in that case, unless you're willing to go the extra mile and perform a herculean task (you're probably not). I don't want to remember, store, and include in every package I publish, a set of "right combination of feature test macros -> behaviour" mappings. This is extremely cumbersome to use; best case, it ends up in a giant header file with #ifdef forests, worst case, it gets desynced across projects and becomes yet another source of headaches. Even GNU autoconf, the absolute reference in configuration bloatware, does not do this. It includes an extensive set of build-time tests and of run-time tests that work on native builds, but it does not store predetermined run-time test values for cross-builds. Instead, on cross-builds, autoconf *assumes* behaviours (and, obviously, gets them wrong half the time). The only way to make it viable is if there were some organization that collected, gathered and published such a dataset on a well-known site in an easy-to-use format for every build system out there (and dear reader, if you reflexively thought "JSON", you lost, close your mail client and try again later). That is tremendous work, that can only be achieved via benevolent funding and/or extensive community cooperation. And then you'd have to convince all the build systems to use it. Given the state of open source in 2023, I'm not holding my breath. The usual counter-argument to this proposition is "but having something is better than nothing", and it is also wrong. We already have a lot of somethings that should be better than nothing; but apparently, they all suck, since people cannot stop inventing new somethings that will, too, doubtlessly be better than nothing. This problem needs to be solved for everyone once and for all; a new partial solution would only add to the pile of reinvented square wheels. (The good thing about square wheels is that they stack nicely.) In short, __MUSL__ would not help with run-time tests in cross-builds, because 1. nobody would use it, 2. the people who would use it would use it wrong, 3. the people who use it right would be overwhelmed by the amount of boilerplate and maintenance this requires, 4. solving the boilerplate/maintenance thing basically starts with abolishing capitalism and we're not going to discuss that here. -- Laurent ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [musl] __MUSL__ macro 2023-07-07 7:14 ` Alastair Houghton 2023-07-07 7:30 ` A. Wilcox 2023-07-07 11:20 ` Laurent Bercot @ 2023-07-07 11:45 ` Jeffrey Walton 2023-07-07 13:53 ` Rich Felker 3 siblings, 0 replies; 22+ messages in thread From: Jeffrey Walton @ 2023-07-07 11:45 UTC (permalink / raw) To: musl On Fri, Jul 7, 2023 at 3:14 AM Alastair Houghton <ahoughton@apple.com> wrote: > > On 6 Jul 2023, at 13:17, Alex Xu <alex_y_xu@yahoo.ca> wrote: > > [...] > > Configure-time checks for specific functionality are a standard part of > > writing portable C code. Preprocessor checks for specific platforms lead > > to ifdef swamps that are still not portable to any platforms not > > initially considered. > > There are two situations where the lack of macros is absolutely a problem and there isn’t a sane workaround. > > 1. You’re cross compiling and the thing you’re interested in is *runtime* behaviour, so configure-style checks simply aren’t an option. In a cross-compilation situation the only thing they can test is that a program will *build*; they can’t test its behaviour since you don’t necessarily have anywhere to run it. > > 2. You’re in a header-only library, particularly one where the final environment you’re running in wasn’t available at whatever configuration time you had, assuming you had one at all. A really good example of this is the built-in headers in Clang or GCC, but the problem also applies to things like STL or Boost. A third problem is, Musl assumes a project is using Autotools or similar. That's not always the case. We use preprocessor signalling to enable or disable code paths at build time. (In fact, we do not wish Autotools on our worst enemies.) Jeff ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [musl] __MUSL__ macro 2023-07-07 7:14 ` Alastair Houghton ` (2 preceding siblings ...) 2023-07-07 11:45 ` Jeffrey Walton @ 2023-07-07 13:53 ` Rich Felker 2023-07-07 14:18 ` Alastair Houghton 3 siblings, 1 reply; 22+ messages in thread From: Rich Felker @ 2023-07-07 13:53 UTC (permalink / raw) To: Alastair Houghton; +Cc: musl On Fri, Jul 07, 2023 at 08:14:11AM +0100, Alastair Houghton wrote: > On 6 Jul 2023, at 13:17, Alex Xu <alex_y_xu@yahoo.ca> wrote: > > > >> Examples include the use of `union`s in `pthread_mutex_t` et al > >> (which means that we can’t write a C++ `constexpr` function that > >> returns one, even if all it does is return > >> `PTHREAD_MUTEX_INITIALIZER`), > > > > The issue is that some of the members are volatile, not the union > > itself. > > .... > > Arguing about the details of individual issues is an unnecessary > distraction, honestly. Except here it's really not, because the example you brought up where you couldn't do detection was where your code was doing something very wrong. If you can fix your code not to do that wrong thing, it can be fixed not to do it anywhere. If you can't, knowing it's "not going to work because you're on __MUSL__" does not help you. The compiler error is just as good. > They were just OTOH examples of oddities that > we’ve come across so far, and yes, I agree that in some cases there > are potentially better fixes for them than preprocessor shenanigans > - and where those are available I’m totally onboard with doing that > instead, because I *entirely agree* that it’s better to avoid a huge > mass of `#if` conditions in code. There’s a very good chance that > any I add in the various places I might be interested in adding them > will end up being my responsibility to maintain, so that, if nothing > else, is a good motivation for minimising them. > > >> or the fact that `dladdr()` is a no-op when statically linked. > > > > When statically linked, the dynamic linking interface doesn't work at > > all though? And furthermore, even if it did, what would it return? If > > you're (once again) trying to do stack traces manually, try libunwind? > > :-D Oh, the irony. One of the places where this behaviour turns out > to be a problem is *in the tests for libunwind*. (That can be solved > in another way without resorting to macros, however.) That dladdr is a no-op rather than always returning the single static-linked object is a bug that would be worth reporting. Having someone leave it unreported and instead hard-coding "musl dladdr doesn't work" is not in our interests. > > I don't understand, don't you know at compile time whether you're > > linking statically or dynamically? > > Not always, no (see situation 2 below). > > > Configure-time checks for specific functionality are a standard part of > > writing portable C code. Preprocessor checks for specific platforms lead > > to ifdef swamps that are still not portable to any platforms not > > initially considered. > > There are two situations where the lack of macros is absolutely a > problem and there isn’t a sane workaround. > > 1. You’re cross compiling and the thing you’re interested in is > *runtime* behaviour, so configure-style checks simply aren’t an > option. In a cross-compilation situation the only thing they can > test is that a program will *build*; they can’t test its behaviour > since you don’t necessarily have anywhere to run it. > > 2. You’re in a header-only library, particularly one where the final > environment you’re running in wasn’t available at whatever > configuration time you had, assuming you had one at all. A really > good example of this is the built-in headers in Clang or GCC, but > the problem also applies to things like STL or Boost. In neither of these cases do you want to know "this is musl", which would at best help you make wrong platform-specific assumptions that might have been true at the time you wrote the code, but which aren't true later when someone is using it. Almost every single time somebody has made a big deal about requesting __MUSL__, whatever it was they wanted to do would have broken on a later, if not the very next, version of, musl, because what they were really getting at was that something they wanted to use didn't work the way they hoped (often because of a bug or extended functionality that wasn't yet implemented but was reasonable to implement) and they were trying to hard-code their idea that "musl doesn't work" rather than "find out if this thing I want to use is available". I already have a proposal for a *granular* version of the latter, that doesn't depend on hard-coding assumptions about particular implementations/versions, initially posted in the libc-coord thread at https://www.openwall.com/lists/libc-coord/2020/04/22/1. Unfortunately, while everyone seems to agree "this is a good idea", nobody is rushing to actually do it. Proposals for an initial list of what to include here, that could be reviewed and pruned, would be a lot more useful than re-proposing something that's been rejected as harmful multiple times already. Rich ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [musl] __MUSL__ macro 2023-07-07 13:53 ` Rich Felker @ 2023-07-07 14:18 ` Alastair Houghton 0 siblings, 0 replies; 22+ messages in thread From: Alastair Houghton @ 2023-07-07 14:18 UTC (permalink / raw) To: Rich Felker; +Cc: musl On 7 Jul 2023, at 14:53, Rich Felker <dalias@libc.org> wrote: > > On Fri, Jul 07, 2023 at 08:14:11AM +0100, Alastair Houghton wrote: > >> Arguing about the details of individual issues is an unnecessary >> distraction, honestly. > > Except here it's really not, because the example you brought up where > you couldn't do detection was where your code was doing something very > wrong. In point of fact, I didn’t even try to detect musl in that case and instead changed the code to work differently. So it really is irrelevant and again, I wish I hadn’t mentioned it because now we’re spending time on that rather than talking about the actual issue. > That dladdr is a no-op rather than always returning the single > static-linked object is a bug that would be worth reporting. Having > someone leave it unreported and instead hard-coding "musl dladdr > doesn't work" is not in our interests. The bug report instructions appear to say that an email to this list is sufficient. Does this need to be a separate message? I’m quite happy to report it as a bug however you please. >> There are two situations where the lack of macros is absolutely a >> problem and there isn’t a sane workaround. >> >> 1. You’re cross compiling... [snip] >> >> 2. You’re in a header-only library, ... [snip] > In neither of these cases do you want to know "this is musl", which > would at best help you make wrong platform-specific assumptions that > might have been true at the time you wrote the code, but which aren't > true later when someone is using it. Nevertheless, “this is musl” (or better, “this is musl version x.y”) is a reasonable proxy in the absence of the availability of any finer grained information. I entirely agree that it isn’t perfect, and adding such a test potentially necessitates future maintenance when musl is updated - but that future maintenance is on the owner of the project that added such a test, not on you. Many projects have already made a conscious choice to accept those kinds of tests and the associated maintenance overhead in the context of other libraries and environments. This is definitely a balancing act; explicitly testing for particular functionality is better, on that we agree, but it does have overheads and problems of its own and sometimes it makes sense to try to do things in the preprocessor or with compile-time tests instead of having a complex and often slow configuration step. Without these kinds of macros being available, it’s difficult to make perfectly reasonable tests (like “don’t use this function when building for musl 1.1.7 because it has a known bug”, or “don’t use this function because it isn’t supported on musl (yet)”), and people will just do what various posts suggest on Stack Overflow, which is far worse because they’re relying on hacky methods of detecting musl that are totally outside of your control. If it is your position that you are not going to add this, ever, then fine. I’ve made my argument and I think that’s the wrong decision but it’s your choice to make. Kind regards, Alastair. ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [musl] __MUSL__ macro 2023-07-06 10:48 [musl] __MUSL__ macro Alastair Houghton 2023-07-06 12:17 ` Alex Xu @ 2023-07-07 12:47 ` Rich Felker 2023-07-07 13:14 ` Alastair Houghton 1 sibling, 1 reply; 22+ messages in thread From: Rich Felker @ 2023-07-07 12:47 UTC (permalink / raw) To: Alastair Houghton; +Cc: musl On Thu, Jul 06, 2023 at 11:48:04AM +0100, Alastair Houghton wrote: > Hi all, > > Before I start, I’m aware of > > <https://www.openwall.com/lists/musl/2013/03/29/13> > > but I *still* want to add __MUSL__ (see attached patch). > > Let me explain what we’re doing, why we want it and why we think > musl *should* have it. We’re trying to add support for musl to Swift > <https://swift.org <https://swiftlang.org/>> and its attendant core > libraries, and there are a number of things about musl that > presently differ from other platforms/C libraries we support. > > Examples include the use of `union`s in `pthread_mutex_t` et al > (which means that we can’t write a C++ `constexpr` function that > returns one, even if all it does is return > `PTHREAD_MUTEX_INITIALIZER`), the fact that it doesn’t have the > `d_namlen` member of `struct dirent`, or the fact that `dladdr()` is > a no-op when statically linked. This has nothing to do with being on musl. This has to do with writing semantically incorrect code that does not meet the requirements of the specification. pthread_mutex_t is not a value type. It's an object. You cannot initialize a pthread_mutex_t in one context and copy/assign/return it to be used somewhere else as a value. Only the actual object initialized may be used. For example, it may even contain references to its own address. Whatever implementations you think this works on on probably to not have a contract to make it work; you're writing to something that you observed working on your system. And in that case, once you've read the relevant source, you need to hard-code the unsafe code to be used *only on that specific implementation and version of it*, not assume musl is the one doing something odd/different because you hack doesn't work here. Rich ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [musl] __MUSL__ macro 2023-07-07 12:47 ` Rich Felker @ 2023-07-07 13:14 ` Alastair Houghton 2023-07-07 14:19 ` Markus Wichmann 2023-07-07 15:05 ` i262jq 0 siblings, 2 replies; 22+ messages in thread From: Alastair Houghton @ 2023-07-07 13:14 UTC (permalink / raw) To: Rich Felker; +Cc: musl On 7 Jul 2023, at 13:47, Rich Felker <dalias@libc.org> wrote: > > pthread_mutex_t is not a value type. It's an object. You cannot [snip] This is a somewhat irrelevant distraction and I rather wish I hadn’t mentioned that as an example of odd behaviour. I’m well aware that you cannot copy or assign `pthread_mutex_t` values in general (and I understand the reasons why). Please can we instead focus on the issue of whether or not musl should have `__MUSL__` and `__MUSL_MINOR__`. Kind regards, Alastair. ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [musl] __MUSL__ macro 2023-07-07 13:14 ` Alastair Houghton @ 2023-07-07 14:19 ` Markus Wichmann 2023-07-07 14:26 ` Markus Wichmann ` (2 more replies) 2023-07-07 15:05 ` i262jq 1 sibling, 3 replies; 22+ messages in thread From: Markus Wichmann @ 2023-07-07 14:19 UTC (permalink / raw) To: musl Am Fri, Jul 07, 2023 at 02:14:30PM +0100 schrieb Alastair Houghton: > This is a somewhat irrelevant distraction and I rather wish I hadn’t > mentioned that as an example of odd behaviour. I’m well aware that > you cannot copy or assign `pthread_mutex_t` values in general (and I > understand the reasons why). > > Please can we instead focus on the issue of whether or not musl should > have `__MUSL__` and `__MUSL_MINOR__`. > The counter-examples are not irrelevant. That is precisely the point. Nobody advocating for implementation identification macros has so far given a valid reason to do so. Every single one so far has turned out to be spurious. Well, I tell a lie, there is one case with a shadow of reason behind it: Header-only libraries. But maybe the problem with those is trying to be a header-only library. If you have different implementations with different runtime behavior, check whether the behavior is acceptable from the specification. If it is, write code that can accept the behavior. If it is not, write a bug report. Very often I am astonished that the problem presented is claimed to have the __MUSL__ macro as a solution. I am reminded of a Stack Overflow question where someone wanted to identify musl, because musl doesn't have a trustworthy vfork(). Pressed on what exactly he meant, the poster said that depending on version and architecture, calling vfork() with musl actually results in it calling fork(). OK, that was his concern. His solution? If he found he was running on Apple or musl, he wanted to call fork() instead of vfork(). Why he would not just always call vfork() if the claimed untrustworthy behavior was also his remedy is a question the guy skillfully avoided an answer to. In your case, apparently there is a way to deal with failing dladdr(). So why don't you put the code for that in the failure path for the call, instead of into "#ifdef __MUSL__"? That way, even unknown implementations would be supported. For header-only libraries, the customer could configure them at build time. I have so far not figured out why people that write programs for a living cannot be expected to fill out a config.h template. There was a person here before who wanted a macro to identify that qsort_r() is available, and I told him much the same thing, and never got a satisfactory answer to the above query. Also, he already had his own fallback sorting algorithm, so the portable solution was just to call that, and then the whole need evaporated. I remain in staunch opposition to identification macros, because those have so far always - no matter how benevolent they might have seemed at the beginning - lead to #ifdef hell of levels Dante couldn't dream of. They lead to people writing bad code and making bad assumptions. None of this matters one bit, because Rich is God as far as musl is concerned, and he has not weighed in yet. However, in the past he has spoken out against these macros, and I doubt your arguments have convinced him. They haven't convinced me, at any rate. Ciao, Markus ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [musl] __MUSL__ macro 2023-07-07 14:19 ` Markus Wichmann @ 2023-07-07 14:26 ` Markus Wichmann 2023-07-07 14:46 ` Alastair Houghton 2023-07-07 15:02 ` Andrew Bell 2 siblings, 0 replies; 22+ messages in thread From: Markus Wichmann @ 2023-07-07 14:26 UTC (permalink / raw) To: musl Am Fri, Jul 07, 2023 at 04:19:14PM +0200 schrieb Markus Wichmann: > None of this matters one bit, because Rich is God as far as musl is > concerned, and he has not weighed in yet. However, in the past he has > spoken out against these macros, and I doubt your arguments have > convinced him. They haven't convinced me, at any rate. > Goddangit, for some reason Rich's messages had only showed up the moment I hit send. Not sure how exactly that worked. Seems like I guessed right, though. Ciao, Markus ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [musl] __MUSL__ macro 2023-07-07 14:19 ` Markus Wichmann 2023-07-07 14:26 ` Markus Wichmann @ 2023-07-07 14:46 ` Alastair Houghton 2023-07-07 15:02 ` Andrew Bell 2 siblings, 0 replies; 22+ messages in thread From: Alastair Houghton @ 2023-07-07 14:46 UTC (permalink / raw) To: musl On 7 Jul 2023, at 15:19, Markus Wichmann <nullplan@gmx.net> wrote: > > The counter-examples are not irrelevant. That is precisely the point. > Nobody advocating for implementation identification macros has so far > given a valid reason to do so. Every single one so far has turned out to > be spurious. Well, I tell a lie, there is one case with a shadow of > reason behind it: Header-only libraries. And cross-compilation when you can’t detect runtime behaviour at configuration time. Those two are actually what I care about, as it happens. > I remain in staunch opposition to identification macros, ... [snip] So I gather. > None of this matters one bit, because Rich is God as far as musl is > concerned, and he has not weighed in yet. :-) I think you crossed over with him there. It’s fine. There’s no point in arguing; we clearly aren’t going to agree here. I think it’s a shame because I think it makes things unnecessarily worse for everyone than they have to be (people will still try to detect musl, as in that Stack Overflow post, but they’ll do so in a less robust manner, and it makes supporting cross-compilation for musl targets and header only libraries that much harder). Kind regards, Alastair. ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [musl] __MUSL__ macro 2023-07-07 14:19 ` Markus Wichmann 2023-07-07 14:26 ` Markus Wichmann 2023-07-07 14:46 ` Alastair Houghton @ 2023-07-07 15:02 ` Andrew Bell 2023-07-07 15:19 ` Markus Wichmann 2 siblings, 1 reply; 22+ messages in thread From: Andrew Bell @ 2023-07-07 15:02 UTC (permalink / raw) To: musl [-- Attachment #1: Type: text/plain, Size: 1214 bytes --] On Fri, Jul 7, 2023 at 10:19 AM Markus Wichmann <nullplan@gmx.net> wrote: > Am Fri, Jul 07, 2023 at 02:14:30PM +0100 schrieb Alastair Houghton: > > This is a somewhat irrelevant distraction and I rather wish I hadn’t > > mentioned that as an example of odd behaviour. I’m well aware that > > you cannot copy or assign `pthread_mutex_t` values in general (and I > > understand the reasons why). > > > > Please can we instead focus on the issue of whether or not musl should > > have `__MUSL__` and `__MUSL_MINOR__`. > > > > The counter-examples are not irrelevant. That is precisely the point. > Nobody advocating for implementation identification macros has so far > given a valid reason to do so. > I think you can say this and still add the macros. Perhaps it's true that nobody has valid reasons, but the fact that it comes up repeatedly says that some people don't understand this or perhaps they have valid reasons yet to be considered. I guess I don't understand the opposition -- is there any downside to musl to having the macros defined, necessary or not? (I'm saying this as a minimalist, so I'm surprising myself here.) -- Andrew Bell andrew.bell.ia@gmail.com [-- Attachment #2: Type: text/html, Size: 1814 bytes --] ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [musl] __MUSL__ macro 2023-07-07 15:02 ` Andrew Bell @ 2023-07-07 15:19 ` Markus Wichmann 2023-07-07 15:24 ` Andrew Bell 2023-07-07 15:34 ` Alastair Houghton 0 siblings, 2 replies; 22+ messages in thread From: Markus Wichmann @ 2023-07-07 15:19 UTC (permalink / raw) To: musl Am Fri, Jul 07, 2023 at 11:02:21AM -0400 schrieb Andrew Bell: > I guess I don't understand the opposition -- is there any downside to musl > to having the macros defined, necessary or not? (I'm saying this as a > minimalist, so I'm surprising myself here.) > Yes, it makes people write worse code. Not making the macros available makes people write more portable code, which is a good thing. Sometimes people have to be made to think for a moment, and broken out of their rut, to get them to do the right thing. There is also the issue of what exactly the macros mean. Between distribution patches and backports, a version number does not necessarily map to a feature or bug set. And musl does not want to have any quirks, it wants to just be a POSIX implementation. So what specialties are supposed to be kept in mind when the musl macro is defined? Keep in mind that __GNUC__ also does not just mean gcc anymore. These things keep going off the rails all the time. Ciao, Markus ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [musl] __MUSL__ macro 2023-07-07 15:19 ` Markus Wichmann @ 2023-07-07 15:24 ` Andrew Bell 2023-07-07 15:34 ` Alastair Houghton 1 sibling, 0 replies; 22+ messages in thread From: Andrew Bell @ 2023-07-07 15:24 UTC (permalink / raw) To: musl [-- Attachment #1: Type: text/plain, Size: 1557 bytes --] On Fri, Jul 7, 2023 at 11:20 AM Markus Wichmann <nullplan@gmx.net> wrote: > Am Fri, Jul 07, 2023 at 11:02:21AM -0400 schrieb Andrew Bell: > > I guess I don't understand the opposition -- is there any downside to > musl > > to having the macros defined, necessary or not? (I'm saying this as a > > minimalist, so I'm surprising myself here.) > > > > Yes, it makes people write worse code. Not making the macros available > makes people write more portable code, which is a good thing. Sometimes > people have to be made to think for a moment, and broken out of their > rut, to get them to do the right thing. > I guess I just don't think this is the job of a library -- to be the arbiter of other people's programming. I may have seen some code inside musl that I personally don't like ;) > There is also the issue of what exactly the macros mean. Between > distribution patches and backports, a version number does not > necessarily map to a feature or bug set. And musl does not want to have > any quirks, it wants to just be a POSIX implementation. So what > specialties are supposed to be kept in mind when the musl macro is > defined? I understand if it can't be done well, but I would hope that it's not a big deal to do. I do appreciate the adherence to standards and it seems perfectly fine to respond to questions/proposals with "that's not the way the standard works." But if this is easy, I just don't see the downside from the standpoint of maintaining musl. -- Andrew Bell andrew.bell.ia@gmail.com [-- Attachment #2: Type: text/html, Size: 2266 bytes --] ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [musl] __MUSL__ macro 2023-07-07 15:19 ` Markus Wichmann 2023-07-07 15:24 ` Andrew Bell @ 2023-07-07 15:34 ` Alastair Houghton 2023-07-07 15:45 ` Rich Felker 1 sibling, 1 reply; 22+ messages in thread From: Alastair Houghton @ 2023-07-07 15:34 UTC (permalink / raw) To: musl On 7 Jul 2023, at 16:19, Markus Wichmann <nullplan@gmx.net> wrote: > > Yes, it makes people write worse code. Not making the macros available > makes people write more portable code, which is a good thing. The people who would have misused the macros will simply find another way to detect musl that you’ll like even less (look at the Stack Overflow post we both mentioned; it’s literally the top answer). Not making the macros available doesn’t stop people writing bad code, nor does making them available force anyone to write bad code. Anyway, I’ve said enough on the subject. Kind regards, Alastair. ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [musl] __MUSL__ macro 2023-07-07 15:34 ` Alastair Houghton @ 2023-07-07 15:45 ` Rich Felker 2023-07-07 15:58 ` Alastair Houghton 0 siblings, 1 reply; 22+ messages in thread From: Rich Felker @ 2023-07-07 15:45 UTC (permalink / raw) To: Alastair Houghton; +Cc: musl On Fri, Jul 07, 2023 at 04:34:37PM +0100, Alastair Houghton wrote: > On 7 Jul 2023, at 16:19, Markus Wichmann <nullplan@gmx.net> wrote: > > > > Yes, it makes people write worse code. Not making the macros available > > makes people write more portable code, which is a good thing. > > The people who would have misused the macros will simply find > another way to detect musl that you’ll like even less (look at the > Stack Overflow post we both mentioned; it’s literally the top > answer). The difference is whose fault it is, and who users blame, when they do. If we make a macro that says "you can now rely on implementation details of musl version x.y.z" and people do that, then it ceases to work in a future version, it's our fault, and people rightly blame us. If people do stupid hacks explicitly against our warnings, and against the specification for the interfaces they're using, and it breaks, it's clearly their fault. We have a really good history of being consistent on this, and as a result, of users and projects taking us seriously and making efforts to improve the overall ecosystem. Throwing that away would be really counterproductive. Rich ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [musl] __MUSL__ macro 2023-07-07 15:45 ` Rich Felker @ 2023-07-07 15:58 ` Alastair Houghton 0 siblings, 0 replies; 22+ messages in thread From: Alastair Houghton @ 2023-07-07 15:58 UTC (permalink / raw) To: Rich Felker; +Cc: musl On 7 Jul 2023, at 16:45, Rich Felker <dalias@libc.org> wrote: > > If we make a macro that says "you can now rely on implementation > details of musl version x.y.z" and people do that, then it ceases to > work in a future version, it's our fault, and people rightly blame us. I don’t think that’s true at all - people tend to blame whoever is in front of them, which is probably not going to be musl but something higher up the stack. In many cases projects can detect the musl version already, either at configuration time by examining the version string, or by specifying it as part of their build. Both of those things would have *exactly* that same downside, and the only way to remove that would be to do away with a version number altogether... which nobody is seriously proposing, right? The *only* thing you’re actually preventing here is detecting the musl version *in the preprocessor*. And only in situations where the project itself can’t somehow supply a relevant macro definition. Kind regards, Alastair. ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [musl] __MUSL__ macro 2023-07-07 13:14 ` Alastair Houghton 2023-07-07 14:19 ` Markus Wichmann @ 2023-07-07 15:05 ` i262jq 1 sibling, 0 replies; 22+ messages in thread From: i262jq @ 2023-07-07 15:05 UTC (permalink / raw) To: musl On Fri, Jul 07, 2023 at 02:14:30PM +0100, Alastair Houghton wrote: > issue of whether or not musl should have `__MUSL__` and `__MUSL_MINOR__`. Introducing such macros would allow and encourage reliance on implementation details instead of following the standards. This means encouraging non-portable programming. I think this already have been said on this list. In other words, the question is not whether a change would make sense for specific cases, but whether the effect on the _rest_ of the usage of the library would be acceptable. IMHO the effect would be to undermine portability and maintainability in the long run, for yet unknown / unlimited set of softwares to be maintained or written in the future. Kudos to musl developers for a firm stance against misfeatures, also ones which non-portable (possibly _right now and here_ attractive) solutions would have a use for. /i262jq ^ permalink raw reply [flat|nested] 22+ messages in thread
end of thread, other threads:[~2023-07-07 15:59 UTC | newest] Thread overview: 22+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2023-07-06 10:48 [musl] __MUSL__ macro Alastair Houghton 2023-07-06 12:17 ` Alex Xu 2023-07-06 16:26 ` Szabolcs Nagy 2023-07-07 7:14 ` Alastair Houghton 2023-07-07 7:30 ` A. Wilcox 2023-07-07 8:24 ` Alastair Houghton 2023-07-07 11:20 ` Laurent Bercot 2023-07-07 11:45 ` Jeffrey Walton 2023-07-07 13:53 ` Rich Felker 2023-07-07 14:18 ` Alastair Houghton 2023-07-07 12:47 ` Rich Felker 2023-07-07 13:14 ` Alastair Houghton 2023-07-07 14:19 ` Markus Wichmann 2023-07-07 14:26 ` Markus Wichmann 2023-07-07 14:46 ` Alastair Houghton 2023-07-07 15:02 ` Andrew Bell 2023-07-07 15:19 ` Markus Wichmann 2023-07-07 15:24 ` Andrew Bell 2023-07-07 15:34 ` Alastair Houghton 2023-07-07 15:45 ` Rich Felker 2023-07-07 15:58 ` Alastair Houghton 2023-07-07 15:05 ` i262jq
Code repositories for project(s) associated with this public inbox https://git.vuxu.org/mirror/musl/ This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).