[musl] adding C23 support

[musl] adding C23 support

[Jₑₙₛ Gustedt]

Hello,
as you might have heard, C23 has been finally approved by the C
committee in January 2024 and is now in the loop with ISO to be
published mid-2024. There have been no surprises since the last time
we talked, so all patches that I already proposed some time ago should
still be valid. They can still be found at

      https://forge.icube.unistra.fr/icps/musl/-/branches

The latest is branch c23-v9 based on today's release of musl, but this
will hopefully be updated when we go along. The patches should be
stacked from the least debated to some which perhaps will never make
it into musl. So if they could be considered bottom-up for inclusion,
that would be great.

As far as I can see my patches cover most what is needed for C23
library support, with one noticeable exception which are functions for
<math.h>. These are really too much for me and my level of
expertise. But I have seen that Paul Zimmermann has some of them (the
π functions) in his project, so this could maybe be covered from
there.

Compilers are now basically there, gcc-14 and clang-18 have most what
is needed for C23 from POV of the language.

Thanks

Jₑₙₛ

-- 
:: ICube :::::::::::::::::::::::::::::: deputy director ::
:: Université de Strasbourg :::::::::::::::::::::: ICPS ::
:: INRIA Nancy Grand Est :::::::::::::::::::::::: Camus ::
:: :::::::::::::::::::::::::::::::::::: ☎ +33 368854536 ::
:: https://icube-icps.unistra.fr/index.php/Jens_Gustedt ::

Re: [musl] adding C23 support

[Rich Felker]

On Fri, Mar 01, 2024 at 04:46:21PM +0100, Jₑₙₛ Gustedt wrote:
> Hello,
> as you might have heard, C23 has been finally approved by the C
> committee in January 2024 and is now in the loop with ISO to be
> published mid-2024. There have been no surprises since the last time
> we talked, so all patches that I already proposed some time ago should
> still be valid. They can still be found at
> 
>       https://forge.icube.unistra.fr/icps/musl/-/branches
> 
> The latest is branch c23-v9 based on today's release of musl, but this
> will hopefully be updated when we go along. The patches should be
> stacked from the least debated to some which perhaps will never make
> it into musl. So if they could be considered bottom-up for inclusion,
> that would be great.
> 
> As far as I can see my patches cover most what is needed for C23
> library support, with one noticeable exception which are functions for
> <math.h>. These are really too much for me and my level of
> expertise. But I have seen that Paul Zimmermann has some of them (the
> π functions) in his project, so this could maybe be covered from
> there.
> 
> Compilers are now basically there, gcc-14 and clang-18 have most what
> is needed for C23 from POV of the language.

Great! Thanks for following up. I was just thinking about trying to
get C23 additions integrated as a main focus point for the newly
started release cycle. I'll take a look at your repo and follow up
with comments.

Rich

Re: [musl] adding C23 support

[Rich Felker]

On Fri, Mar 01, 2024 at 11:34:09AM -0500, Rich Felker wrote:
> On Fri, Mar 01, 2024 at 04:46:21PM +0100, Jₑₙₛ Gustedt wrote:
> > Hello,
> > as you might have heard, C23 has been finally approved by the C
> > committee in January 2024 and is now in the loop with ISO to be
> > published mid-2024. There have been no surprises since the last time
> > we talked, so all patches that I already proposed some time ago should
> > still be valid. They can still be found at
> > 
> >       https://forge.icube.unistra.fr/icps/musl/-/branches
> > 
> > The latest is branch c23-v9 based on today's release of musl, but this
> > will hopefully be updated when we go along. The patches should be
> > stacked from the least debated to some which perhaps will never make
> > it into musl. So if they could be considered bottom-up for inclusion,
> > that would be great.
> > 
> > As far as I can see my patches cover most what is needed for C23
> > library support, with one noticeable exception which are functions for
> > <math.h>. These are really too much for me and my level of
> > expertise. But I have seen that Paul Zimmermann has some of them (the
> > π functions) in his project, so this could maybe be covered from
> > there.
> > 
> > Compilers are now basically there, gcc-14 and clang-18 have most what
> > is needed for C23 from POV of the language.
> 
> Great! Thanks for following up. I was just thinking about trying to
> get C23 additions integrated as a main focus point for the newly
> started release cycle. I'll take a look at your repo and follow up
> with comments.

Summary of definitely-welcome functionality-changes:

- exposing existing functions that are now baseline C standard:
  timegm, gmtime_r, localtime_r, memccpy, strdup, strndup

- exposing call_once in stdlib.h

- WIDTH macros (for supported types)

- free_sized and free_aligned_sized (the latter should probably assert
  alignment and these could be wired up to check size, but that can be
  done later and is more delicate with how malloc replacements work)

- memalignment

- strfrom[dfl]

- c8 interfaces

- nullptr_t

- timespec_getres

- printf/scanf b/B, wN/wfN (for supported types)

- const-safety macros (but see below)


And things that can clearly be deferred or omitted:

- changes to make musl itself compile as C23. This is a separate task
  from making musl provide a C23 environment, and much lower priority,
  so I'd like to put it out-of-scope for now.

- everything [u]int128_t. This is also out-of-scope; it could at some
  point in the future be considered for whether it makes sense to
  support, but it's not a requirement for C23.



Some questions:

- Do the final stdbit.h interfaces require external functions, or is a
  header-only implementation acceptable? Has anything changed on these
  we need to be aware of?

- Your stdckdint.h is header-only and is basically just a wrapper for
  gcc/clang builtins. If this is the case, does it make sense for it
  to be supplied by libc at all rather than by the compiler?

- I don't understand the __STDC_VERSION_*_H__ macros. Are these a
  requirement of C23, and if so, what are the rules for the values?
  I'm not sure if it makes sense to use these as the inclusion-guards
  if it might make sense to define them with varying values depending
  on feature profile; in that case, it might break optimization
  against multiple-include. However we end up doing these I'd probably
  like to make one clean commit converting all the headers to the same
  approach at once, rather than mixing it with other changes.



Some proposed changes:

- strfroml admits a simpler approach without parsing/converting the
  precision, as described in the previous thread on it (after dropping
  leading zeros, just limit the number of digits).

- assert changes as written are C89-incompatible (no _Bool, and
  strictly speaking __VA_ARGS__, although I think compilers in
  practice have supported it going way back). New assert should
  probably be conditional on C23+ or tweaked to be more compatible
  with older compilers/language versions.

- I think all the const-safe macro things admit a solution that
  doesn't repeat the call or require const-casting hacks, and that may
  not even need _Generic, just using a cast on the return value with
  typeof, ala:

  #define memchr(p,c,n) ((typeof(1?(p):(void *)1))memchr(p,c,n))

  It looks like char/void issues make the str* ones a little bit
  tricker but I think they ultimately admit the same approach.

- As discussed before, the printf wN/wfN length prefixes fit within
  the existing state machine, without needing added code. I think I
  posted a draft patch for this.


Overall strategy:

Let's start with getting uncontroversial things that are mostly
standalone (no dependencies on other changes) mergeable/merged, so
that what's left can be narrowed down. I'll start going through the
first list above and see what looks mergeable without changes or with
only trivial changes.

Rich

Re: [musl] adding C23 support

[Jₑₙₛ Gustedt]

Rich,
that was quick!

on Fri, 1 Mar 2024 13:10:04 -0500 you (Rich Felker <dalias@libc.org>)
wrote:

> Summary of definitely-welcome functionality-changes:
> 
> - exposing existing functions that are now baseline C standard:
>   timegm, gmtime_r, localtime_r, memccpy, strdup, strndup
> 
> - exposing call_once in stdlib.h
> 
> - WIDTH macros (for supported types)
> 
> - free_sized and free_aligned_sized (the latter should probably assert
>   alignment and these could be wired up to check size, but that can be
>   done later and is more delicate with how malloc replacements work)
> 
> - memalignment
> 
> - strfrom[dfl]
> 
> - c8 interfaces
> 
> - nullptr_t
> 
> - timespec_getres
> 
> - printf/scanf b/B, wN/wfN (for supported types)
> 
> - const-safety macros (but see below)

That would be great!

> And things that can clearly be deferred or omitted:
> 
> - changes to make musl itself compile as C23. This is a separate task
>   from making musl provide a C23 environment, and much lower priority,
>   so I'd like to put it out-of-scope for now.

If you like. But AFAIR these are really no big deal, typing function
pointers correctly and stuff like that. These kind of things that
never get high priority but should be dealed with some day ;-)

> - everything [u]int128_t. This is also out-of-scope; it could at some
>   point in the future be considered for whether it makes sense to
>   support, but it's not a requirement for C23.

sure, after the last discussions I expected as much. I'll just keep
them on top, so nothing else depends on them.

> Some questions:
> 
> - Do the final stdbit.h interfaces require external functions, or is a
>   header-only implementation acceptable? Has anything changed on these
>   we need to be aware of?

Yes, these require all external functions, besides for the tg
interfaces.

I think I have changed things since we last discussed, because I (and
I was joined by many on the C committee ;-) had misunderstood some of
the interfaces, which direction the bit count went and stuff like
that. A thorough review would be necessary and appreciated.

> - Your stdckdint.h is header-only and is basically just a wrapper for
>   gcc/clang builtins. If this is the case, does it make sense for it
>   to be supplied by libc at all rather than by the compiler?

For modern compilers it definitively would make sense. But it is an
important step to make these interfaces portable, all this discussion
on making C safer on integer overflow.

So it would probably also good to have them independently from C23
even for old compilers that have the builtins. Then a simple check
with `__has_include` would make them available for everybody.

I have the impression that these interfaces reach far back in history.
A difficulty to make them testable at library-compilation time by
tooling is that they don't depend on the compiler with which the C
library is compiled but on the compiler that the application has in
hand when compiling their code.

(`__has_include` is also in C23 but has been around since ages, so
this provides cheap test for people to see if an interface is
provided.)

> - I don't understand the __STDC_VERSION_*_H__ macros. Are these a
>   requirement of C23,

yes

>  and if so, what are the rules for the values?

the final values are all `202311L`. They are meant to dissociate from
the C version that the compiler provides and provide means to test for
the presence of particular parts of C23 library support.  So C
libraries can improve step by step. For example you could upgrade the
macro for <stdio.h> as soon as its ready, and do <math.h> much later,
or people could even have libmath provided from somewhere else.

>   I'm not sure if it makes sense to use these as the inclusion-guards

That was your idea, really ;-)

>   if it might make sense to define them with varying values depending
>   on feature profile; in that case, it might break optimization
>   against multiple-include. However we end up doing these I'd probably
>   like to make one clean commit converting all the headers to the same
>   approach at once, rather than mixing it with other changes.

I'd really advise against that. It is really meant to be on a per
header base, that's why there are so many of them. In particular,
<math.h> is really a large piece of work, that could delay us for
months or even years.

> Some proposed changes:
> 
> - strfroml admits a simpler approach without parsing/converting the
>   precision, as described in the previous thread on it (after dropping
>   leading zeros, just limit the number of digits).
> 
> - assert changes as written are C89-incompatible (no _Bool, and
>   strictly speaking __VA_ARGS__, although I think compilers in
>   practice have supported it going way back). New assert should
>   probably be conditional on C23+ or tweaked to be more compatible
>   with older compilers/language versions.

I'd have to look into both of these, that's too long ago.

(And my knowledge basically only starts at C99.)

> - I think all the const-safe macro things admit a solution that
>   doesn't repeat the call or require const-casting hacks, and that may
>   not even need _Generic, just using a cast on the return value with
>   typeof, ala:
> 
>   #define memchr(p,c,n) ((typeof(1?(p):(void *)1))memchr(p,c,n))
> 
>   It looks like char/void issues make the str* ones a little bit
>   tricker but I think they ultimately admit the same approach.

Yes, maybe, I'll look into these, too.

(And so you think that `typeof` is more portable than generic? Or is
it just that this is simpler?)

> - As discussed before, the printf wN/wfN length prefixes fit within
>   the existing state machine, without needing added code. I think I
>   posted a draft patch for this.

yes, feel free to use your's and drop mine. But at the time I was not
completely convinced that you'd cover all the cases. In any case,
these specifiers probably need a lot of unit tests.

I'll adapt what I have on top (which are only the 128 things, anyhow,
I think)

> Overall strategy:
> 
> Let's start with getting uncontroversial things that are mostly
> standalone (no dependencies on other changes) mergeable/merged, so
> that what's left can be narrowed down. I'll start going through the
> first list above and see what looks mergeable without changes or with
> only trivial changes.

That sounds good!

Thanks
Jₑₙₛ

-- 
:: ICube :::::::::::::::::::::::::::::: deputy director ::
:: Université de Strasbourg :::::::::::::::::::::: ICPS ::
:: INRIA Nancy Grand Est :::::::::::::::::::::::: Camus ::
:: :::::::::::::::::::::::::::::::::::: ☎ +33 368854536 ::
:: https://icube-icps.unistra.fr/index.php/Jens_Gustedt ::

Re: [musl] adding C23 support

[Rich Felker]

On Fri, Mar 01, 2024 at 08:57:44PM +0100, Jₑₙₛ Gustedt wrote:
> > Some questions:
> > 
> > - Do the final stdbit.h interfaces require external functions, or is a
> >   header-only implementation acceptable? Has anything changed on these
> >   we need to be aware of?
> 
> Yes, these require all external functions, besides for the tg
> interfaces.

That's unfortunate. It's a really huge number of functions that it's
hard to imagine anyone wanting as external rather than inline.
Normally we aim to put everything in independent TUs, but I think here
that might end up overflowing the ar/link command lines or something
if we did it.

> I think I have changed things since we last discussed, because I (and
> I was joined by many on the C committee ;-) had misunderstood some of
> the interfaces, which direction the bit count went and stuff like
> that. A thorough review would be necessary and appreciated.

Ok, so this is going to be one of the slow parts...

> > - Your stdckdint.h is header-only and is basically just a wrapper for
> >   gcc/clang builtins. If this is the case, does it make sense for it
> >   to be supplied by libc at all rather than by the compiler?
> 
> For modern compilers it definitively would make sense. But it is an
> important step to make these interfaces portable, all this discussion
> on making C safer on integer overflow.
> 
> So it would probably also good to have them independently from C23
> even for old compilers that have the builtins. Then a simple check
> with `__has_include` would make them available for everybody.
> 
> I have the impression that these interfaces reach far back in history.
> A difficulty to make them testable at library-compilation time by
> tooling is that they don't depend on the compiler with which the C
> library is compiled but on the compiler that the application has in
> hand when compiling their code.
> 
> (`__has_include` is also in C23 but has been around since ages, so
> this provides cheap test for people to see if an interface is
> provided.)

Oh, it's standard now? That's very nice!

> > - I don't understand the __STDC_VERSION_*_H__ macros. Are these a
> >   requirement of C23,
> 
> yes
> 
> >  and if so, what are the rules for the values?
> 
> the final values are all `202311L`. They are meant to dissociate from
> the C version that the compiler provides and provide means to test for
> the presence of particular parts of C23 library support.  So C
> libraries can improve step by step. For example you could upgrade the
> macro for <stdio.h> as soon as its ready, and do <math.h> much later,
> or people could even have libmath provided from somewhere else.
> 
> >   I'm not sure if it makes sense to use these as the inclusion-guards
> 
> That was your idea, really ;-)
> 
> >   if it might make sense to define them with varying values depending
> >   on feature profile; in that case, it might break optimization
> >   against multiple-include. However we end up doing these I'd probably
> >   like to make one clean commit converting all the headers to the same
> >   approach at once, rather than mixing it with other changes.
> 
> I'd really advise against that. It is really meant to be on a per
> header base, that's why there are so many of them. In particular,
> <math.h> is really a large piece of work, that could delay us for
> months or even years.

OK, I seem to have forgotten past conversations on this, so I'll see
if I can dig something up.

> > - I think all the const-safe macro things admit a solution that
> >   doesn't repeat the call or require const-casting hacks, and that may
> >   not even need _Generic, just using a cast on the return value with
> >   typeof, ala:
> > 
> >   #define memchr(p,c,n) ((typeof(1?(p):(void *)1))memchr(p,c,n))
> > 
> >   It looks like char/void issues make the str* ones a little bit
> >   tricker but I think they ultimately admit the same approach.
> 
> Yes, maybe, I'll look into these, too.
> 
> (And so you think that `typeof` is more portable than generic? Or is
> it just that this is simpler?)

It's that it follows a DRY principle, and avoids the extra creative
machinery you were trying to do to get warnings/errors to appear in
the right places due to how multiple clauses of _Generic are handled.

I think the char version might still need _Generic, but it would be a
_Generic outside/in-front-of the call (as a subexpression of the
typeof for the cast) without anything complex inside it.

Re: "more portable", these are only used conditional on >= C23, in
which case either is available.

Rich

Re: [musl] adding C23 support

[Jₑₙₛ Gustedt]

Rich,

on Fri, 1 Mar 2024 16:17:32 -0500 you (Rich Felker <dalias@libc.org>)
wrote:

> On Fri, Mar 01, 2024 at 08:57:44PM +0100, Jₑₙₛ Gustedt wrote:
> > > Some questions:
> > > 
> > > - Do the final stdbit.h interfaces require external functions, or
> > > is a header-only implementation acceptable? Has anything changed
> > > on these we need to be aware of?  
> > 
> > Yes, these require all external functions, besides for the tg
> > interfaces.  
> 
> That's unfortunate. It's a really huge number of functions that it's
> hard to imagine anyone wanting as external rather than inline.

This is a lot, I completely agree.

> Normally we aim to put everything in independent TUs, but I think here
> that might end up overflowing the ar/link command lines or something
> if we did it.

Then wait for the <math.h> functions, this is even worse :-(

At least on the architecture that I use musl is compiled with function
sections these days, so this should not blow up executable size
unreasonably, I think.

> It's that it follows a DRY principle, and avoids the extra creative
> machinery you were trying to do to get warnings/errors to appear in
> the right places due to how multiple clauses of _Generic are handled.
> 
> I think the char version might still need _Generic, but it would be a
> _Generic outside/in-front-of the call (as a subexpression of the
> typeof for the cast) without anything complex inside it.

Ok, I'll look into that.

Thanks
Jₑₙₛ

-- 
:: ICube :::::::::::::::::::::::::::::: deputy director ::
:: Université de Strasbourg :::::::::::::::::::::: ICPS ::
:: INRIA Nancy Grand Est :::::::::::::::::::::::: Camus ::
:: :::::::::::::::::::::::::::::::::::: ☎ +33 368854536 ::
:: https://icube-icps.unistra.fr/index.php/Jens_Gustedt ::

Re: [musl] adding C23 support

[Gabriel Ravier]

On 3/1/24 21:17, Rich Felker wrote:
> On Fri, Mar 01, 2024 at 08:57:44PM +0100, Jₑₙₛ Gustedt wrote:
>>> Some questions:
>>>
>>> - Do the final stdbit.h interfaces require external functions, or is a
>>>    header-only implementation acceptable? Has anything changed on these
>>>    we need to be aware of?
>> Yes, these require all external functions, besides for the tg
>> interfaces.
> That's unfortunate. It's a really huge number of functions that it's
> hard to imagine anyone wanting as external rather than inline.
> Normally we aim to put everything in independent TUs, but I think here
> that might end up overflowing the ar/link command lines or something
> if we did it.

I agree it's quite a lot, but w.r.t. whether or not it was good to have 
this in the standard I think it makes sense to have them, for 
consistency - basically for the same reason why functions like `getc` 
which were historically macros were changed to be mandated to be visible 
as actual functions (though a macro definition was still allowed to be 
present) for the purposes of e.g. taking an address to them.


>
>> I think I have changed things since we last discussed, because I (and
>> I was joined by many on the C committee ;-) had misunderstood some of
>> the interfaces, which direction the bit count went and stuff like
>> that. A thorough review would be necessary and appreciated.
> Ok, so this is going to be one of the slow parts...
>
>>> - Your stdckdint.h is header-only and is basically just a wrapper for
>>>    gcc/clang builtins. If this is the case, does it make sense for it
>>>    to be supplied by libc at all rather than by the compiler?
>> For modern compilers it definitively would make sense. But it is an
>> important step to make these interfaces portable, all this discussion
>> on making C safer on integer overflow.
>>
>> So it would probably also good to have them independently from C23
>> even for old compilers that have the builtins. Then a simple check
>> with `__has_include` would make them available for everybody.
>>
>> I have the impression that these interfaces reach far back in history.
>> A difficulty to make them testable at library-compilation time by
>> tooling is that they don't depend on the compiler with which the C
>> library is compiled but on the compiler that the application has in
>> hand when compiling their code.
>>
>> (`__has_include` is also in C23 but has been around since ages, so
>> this provides cheap test for people to see if an interface is
>> provided.)
> Oh, it's standard now? That's very nice!
>
>>> - I don't understand the __STDC_VERSION_*_H__ macros. Are these a
>>>    requirement of C23,
>> yes
>>
>>>   and if so, what are the rules for the values?
>> the final values are all `202311L`. They are meant to dissociate from
>> the C version that the compiler provides and provide means to test for
>> the presence of particular parts of C23 library support.  So C
>> libraries can improve step by step. For example you could upgrade the
>> macro for <stdio.h> as soon as its ready, and do <math.h> much later,
>> or people could even have libmath provided from somewhere else.
>>
>>>    I'm not sure if it makes sense to use these as the inclusion-guards
>> That was your idea, really ;-)
>>
>>>    if it might make sense to define them with varying values depending
>>>    on feature profile; in that case, it might break optimization
>>>    against multiple-include. However we end up doing these I'd probably
>>>    like to make one clean commit converting all the headers to the same
>>>    approach at once, rather than mixing it with other changes.
>> I'd really advise against that. It is really meant to be on a per
>> header base, that's why there are so many of them. In particular,
>> <math.h> is really a large piece of work, that could delay us for
>> months or even years.
> OK, I seem to have forgotten past conversations on this, so I'll see
> if I can dig something up.
>
>>> - I think all the const-safe macro things admit a solution that
>>>    doesn't repeat the call or require const-casting hacks, and that may
>>>    not even need _Generic, just using a cast on the return value with
>>>    typeof, ala:
>>>
>>>    #define memchr(p,c,n) ((typeof(1?(p):(void *)1))memchr(p,c,n))
>>>
>>>    It looks like char/void issues make the str* ones a little bit
>>>    tricker but I think they ultimately admit the same approach.
>> Yes, maybe, I'll look into these, too.
>>
>> (And so you think that `typeof` is more portable than generic? Or is
>> it just that this is simpler?)
> It's that it follows a DRY principle, and avoids the extra creative
> machinery you were trying to do to get warnings/errors to appear in
> the right places due to how multiple clauses of _Generic are handled.
>
> I think the char version might still need _Generic, but it would be a
> _Generic outside/in-front-of the call (as a subexpression of the
> typeof for the cast) without anything complex inside it.
>
> Re: "more portable", these are only used conditional on >= C23, in
> which case either is available.
>
> Rich