Re: [musl] [PATCH] vfprintf: support C2x %b and %B conversion specifiers

[Rich Felker <dalias@libc.org>]

On Tue, Sep 20, 2022 at 03:29:29PM +0200, Jₑₙₛ Gustedt wrote:
> Rich,
> 
> on Tue, 20 Sep 2022 08:28:29 -0400 you (Rich Felker <dalias@libc.org>)
> wrote:
> 
> > On Tue, Sep 20, 2022 at 11:19:34AM +0200, Jₑₙₛ Gustedt wrote:
> > > Rich,
> > > 
> > > on Mon, 19 Sep 2022 14:10:39 -0400 you (Rich Felker
> > > <dalias@libc.org>) wrote:
> > >   
> > > > On Mon, Sep 19, 2022 at 07:59:52PM +0200, Szabolcs Nagy wrote:  
> > > So yes, sorry, for the separate library part I forgot formated IO
> > > and string functions. But the huge amount of functions that are
> > > added for these types are math functions (I guess something like
> > > 600 or so) stepping on user's identifier space all over.  
> > 
> > Yes, I think it's fine for now to have a separate math library for the
> > math functions. Otherwise the work of adding these interfaces becomes
> > rather prohibitive. I would assume they're all pure functions where
> > correct implementations are basically interchangable, so I don't see a
> > lot of value in insisting these "go with" libc.
> 
> Depends on your instantiation of "pure", but yes, these should be
> mostly interchangeable. The only thing to worry here are that there
> are two possible representations for these types, one where the
> mantissa is basically represented as an integer, and the other where
> decimal digits are packed into groups of bytes in a clever way.

Well the one in use is presumably defined by the psABI. Does it
actually vary by arch, or is there a common form that everyone
reasonable has agreed upon? Note that we're not considering the full
generality the C standard allows, just a single family of platforms.

> > > But for implementing the parts that are outside of math, things
> > > should indeed not be so difficult. gcc has support for the types
> > > since long, I think, and should also provide predefined macros that
> > > could be used to check for language support. Then, the types
> > > themselves have clear definition and prescribed representation, the
> > > ABI is de-facto sorted out, so there would be not much other
> > > implementation dependency to worry about.  
> > 
> > The thing is we don't have the option to "check for language support".
> > Doing that would mean you get a deficient musl build if your compiler
> > doesn't have the language features, so essentially we'd be requiring
> > bleeding-edge gcc or clang (dropping all other-compiler support at the
> > same time) to get a properly featured libc.so that's capable of
> > supporting arbitrary musl-linked binaries.
> 
> I don't think that this needs to be. If you add e.g support for
> decimal floating point to `printf`, the compiler support for that only
> has to be there on the platform where you compile musl. If a user
> platform that uses such a library does not support it, that part will
> simply never be called because users can't defined variables of that
> type. This increases the size of `printf` a bit, though, but my guess
> is that this would be marginal compared to the size that `printf` has,
> anyhow.

You're looking at it from a perspective of producing your own software
linked to musl. I'm looking at it from a perspective of building your
own build of musl to execute existing dynamic-linked binaries outside
your control. Users building musl need to be able to build a
feature-complete version regardless of their compiler. We don't have
different interface/feature profiles based on which compiler version
you built with or any build-time options, and not having that is very
intentional.

> > This is why we're going to need asm thunks for performing va_arg with
> > the new types and (programmatically generated, I assume) asm entry
> > thunks for accepting arguments to any non-variadic functions, which
> > can convert (ideally as a no-op) the decimal float type arguments to
> > integer-type or struct arguments the underlying implementation files
> > would then receive.
> 
> There are no C library functions other than in math.h, I think, that
> accept decimal floating types as prototyped arguments. So if we don't
> do math.h, only `printf` and similar remain with `va_arg` calling
> conventions.
> 
> The only functions that have decimal floating return types are the
> `strtodN` functions in 7.24.1.6, AFAICS.
> 
> So, yes, we'd have to extend `va_arg` with the necessary knowledge to
> obtain a decimal floating point, but hopefully that is just the same
> as obtaining access to other 32, 64 or 128 bit types.

Oh, nice -- I didn't realize they only appear as return values not
arguments. Unfortunately, return values are worse. In order to do the
thunk, you need to setup a call frame and possibly munge the return
value into the storage it belongs in to meet the ABI requirement. For
arguments, on the other hand, it's likely just a tail-call.

> > > Other types that come with C23, and these are mandatory, are
> > > bit-precise integers. There the support by compilers is probably not
> > > yet completely established. I know of an integration into llvm, but
> > > I am not sure about the state of affairs for gcc, nor if there is a
> > > de-facto agreement on ABI issues. In any case, these types need
> > > support in formatted IO, too.  
> > 
> > As far as I can tell, the draft standard makes printf support for all
> > but the ones defined as [u]intNN_t a choice for the implementation, so
> > the obvious choice is not to support any additional ones.
> 
> (There are also the "fast" version that have a different format
> specifier, but which hopefully are basically the same as for the exact
> width.)
> 
> I think for QoI it would be really good to support the bit-precise
> types. These are a quite good design that avoids a lot of the
> complications of the classical integer types. In particular we will
> see them pop up for bit-fields and stuff like that, where there have
> clearer semantics than the traditional ones and extend the
> possibilities beyond the width of `int` to at least 64 bit.

Surely there are going to be all sorts of ABI issues with passing them
as arguments. The right thing to do, as an application programmer
writing a portable program, is the same as when printing types without
a format specifier like time_t: just cast them up to a maximal-size
type that you know is supported and can represent the value. There is
no sense in optimizing the type you pass these things as when the
operation you're passing them for is as "big" as printf.

> > > Also, C23, provides the possibility for extended integer types that
> > > are wider than `[u]intmax_t` under some conditions. This is intended
> > > in particular to allow for implementations such as gcc on x86_64 to
> > > interface the existing 128 bit integer types properly as
> > > `[u]int128_t`. From a C library POV, these then also would need
> > > integration into formatted IO, but here again support in the
> > > compiler with usable feature test macros is there for ages and the
> > > ABI should already be sorted out.  
> > 
> > Yes. I haven't followed the latest on this but my leaning was to leave
> > them as "compiler extensions" that don't count as "extended integer
> > types". However presumably they could be handled the same way as
> > decimal floats if needed.
> 
> For once this allows to define extended integer types in the sense of
> the standard and to provide full support for them. But you are right
> the approach could be the same as for decimal floating point: compile
> them in if the compilation platform of the C library supports them.
> 
> > > So in summary that means that there is some work to do to make
> > > formatted IO of C libraries become compliant with C23. Let me know
> > > if and where I could help to make that happen for musl.  
> > 
> > The big issue is probably collating the list of what's actually needed
> > to meet requirements,
> 
> that I could do

Thanks!

> > and what the ABIs for them are.
> 
> that's were I am not an expert in :-((

OK, hopefully someone else is.

> > If there's cross-arch agreement on a general pattern ABIs follow for
> > them, that would be wonderful, and even if not entirely so, a
> > general pattern would advise how we structure the underlying
> > functions (to make thunks as minimal as possible on the largest
> > number of archs).
> 
> My guess for that is that the decimal floating point types are just
> handled by their respective width, and that the bit-precise integer

Unfortunately in most ABIs the convention for passing a type is not
just a function of its size. Floating point types are usually passed
in different registers, and aggregate types often have complex
conventions for whether they're passed in registers or on the stack
depending on their member types and sizes. So, for decimal float, it's
probably the same as *some* N-bit type, but which one? :)

> types of width N will be rounded up to the next power of two M and use
> representation and calling convention for `uintM_t`. But that would of
> course have to be verified. I can ask Aaron (who wrote this stuff and
> has provided the implementation in llvm) how that is actually done
> there.

If the width-N stuff is uniform like that, it might be possible to
support arbitrary in-range N, but again I'm skeptical of the value
especially when applications cannot portably use any Ns except the
ones with macros defined in stdint.h.

In general, offering non-portable functionality that applications
can't already generally expect to have on popular systems, with no way
to probe for availability, does not seem useful, and it's even less
useful when there's a trivial portable way to do the same thing.

Rich