pthread object sizes for new archs

pthread object sizes for new archs

[Rich Felker]

One issue that's arisen in merging the aarch64 port is that new glibc
ports keep making the pthread object sizes bigger and bigger, to the
point where aligning the size part of the ABI seems highly
detrimental. For example, the mutex sizes are:

glibc/musl 32-bit:      24 bytes
glibc/musl x86_64:      40 bytes
glibc aarch64:          48 bytes

Actually needed 32-bit: 20 bytes
Actually needed 64-bit: 32 bytes (28 + 4 padding for alignment)

I don't mind having extra space in the structure in case we want/need
to expand to meet new requirements or improve performance. But since
musl does not have or want arch-specific algorithm variants, there's
absolutely no use in one arch having more 'extra space' in its pthread
struct than another arch does; this space will never be used.

What I'd like to propose is that, for all new archs, we ignore the
glibc-provided size of the pthread types and stick with the baseline
32-bit sizes, enlarged only by the plausible number of fields that
could need to be pointers if the arch is 64-bit. For mutexes, I'd like
to stick to 32 bytes (33% increase going from 32-bit to 64-bit), but
40 (66% increase) wouldn't be horrible. The glibc aarch64 64-bit bloat
cost of 100% is ridiculous though.

If we go with 32 bytes, there's no slot available for use as a 64-bit
pointer without repurposing existing slots, while 32-bit archs can fit
an extra pointer. If we go with 40 bytes, the 64-bit archs have a lot
more space to expand (which they won't use).

One advantage of using a size of 40 is that we could just make the
x86_64 type sizes the universal 64-bit pthread type sizes and move
these types to an arch-independent file.

For what it's worth, if anyone is wondering why glibc keeps enlarging
these objects, it seems to be to work around bad core design that's
wasting large amounts of space. For example their mutex has separate
lock-word and owner-id fields, and I can't think of any explanation
for this except for accommodating legacy archs that lack atomic CAS
(which is needed for the more advanced types like robust mutexes
anyway). It also has a useless counter for the number of condition
variables currently using a mutex. As a result, despite being larger
than what musl needs, their baseline 32-bit mutex structure can't even
accommodate a doubly-linked list for robust mutexes, so robust mutex
unlock is O(n) in the number of robust locks held for removal from a
singly-linked list. So basically they seem to be pessimizing memory
usage on modern 64-bit archs, and pessimizing performance on modern
32-bit ones, to have baseline algorithms work on junk that's been
obsolete for decades and that store useless state.

I haven't done this much analysis of the other glibc types, but in any
case the same principle applies on musl's side: there's no sense in
making any of the pthread objects have more "extra space" than the
least amount of extra space they have on any existing arch. So I think
we should do the same thing and either go with the x86_64 sizes or
something smaller with "extra space" capacity comparable to the 32-bit
structs.

How does this affect "glibc ABI compatibility"? Very little. As long
as the sizes of the musl types are no larger than the sizes of the
glibc types on the same arch, compatibility with glibc binary code
(apps or libs) is not affected. All that is affected if the ABI of
third-party libraries that use pthread types as members of public
structs, and it only matters if you're calling such libraries using
the glibc-derived ABI from code compiled against musl and using the
affected structures. This is a very unusual usage case, not something
we've ever prioritized supporting (it's broken in several other ways
anyway, or at least it was historically), and IMO it's not worth
severely bloating new archs that people actually want to use.

Rich

Re: pthread object sizes for new archs

[Rich Felker]

On Mon, Mar 09, 2015 at 09:07:33PM -0400, Rich Felker wrote:
> What I'd like to propose is that, for all new archs, we ignore the
> glibc-provided size of the pthread types and stick with the baseline
> 32-bit sizes, enlarged only by the plausible number of fields that
> could need to be pointers if the arch is 64-bit. For mutexes, I'd like
> to stick to 32 bytes (33% increase going from 32-bit to 64-bit), but
> 40 (66% increase) wouldn't be horrible. The glibc aarch64 64-bit bloat
> cost of 100% is ridiculous though.

Note that for x32, this would mean discarding the size fix recently
made as commit e7b9887e8b65253087ab0b209dc8dd85c9f09614 and changing
all the pthread object sizes on x32 to match all the plain 32-bit
archs.

Rich

Re: pthread object sizes for new archs

[Jens Gustedt]

[-- Attachment #1: Type: text/plain, Size: 2180 bytes --]

Hello,

Am Montag, den 09.03.2015, 21:07 -0400 schrieb Rich Felker:
> How does this affect "glibc ABI compatibility"? Very little. As long
> as the sizes of the musl types are no larger than the sizes of the
> glibc types on the same arch, compatibility with glibc binary code
> (apps or libs) is not affected. All that is affected if the ABI of
> third-party libraries that use pthread types as members of public
> structs, and it only matters if you're calling such libraries using
> the glibc-derived ABI from code compiled against musl and using the
> affected structures. This is a very unusual usage case, not something
> we've ever prioritized supporting (it's broken in several other ways
> anyway, or at least it was historically), and IMO it's not worth
> severely bloating new archs that people actually want to use.

So to rephrase your arguments for that

 - For the case I compile my code with the glibc ABI and link it with
   musl: any struct that has pthread (or C11 thread) components in it
   will potentially only use a smaller part of that pthread (C11)
   component. In particular in that case when compiling with the glibc
   ABI the compiler always generates a consistent offset for the
   pthread components. Sounds ok.

   But this only works because the glibc ABI doesn't export any
   interface that has a visible combination of different pthread
   components. Are we sure about that?

 - For the case that I compile code with the musl ABI and link it with
   glibc, this may (and probably will) cause out-of-bounds errors on
   execution. I would very much prefer that big warnings are already
   issued when doing such a link, or even better if such a link would
   fail systematically.

   I hope that there is nobody out there, who would be currently using
   this model, perhaps unknowingly.

Jens


-- 
:: INRIA Nancy Grand Est ::: Camus ::::::: ICube/ICPS :::
:: ::::::::::::::: office Strasbourg : +33 368854536   ::
:: :::::::::::::::::::::: gsm France : +33 651400183   ::
:: ::::::::::::::: gsm international : +49 15737185122 ::
:: http://icube-icps.unistra.fr/index.php/Jens_Gustedt ::




[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: pthread object sizes for new archs

[Rich Felker]

On Tue, Mar 10, 2015 at 09:30:44AM +0100, Jens Gustedt wrote:
> Hello,
> 
> Am Montag, den 09.03.2015, 21:07 -0400 schrieb Rich Felker:
> > How does this affect "glibc ABI compatibility"? Very little. As long
> > as the sizes of the musl types are no larger than the sizes of the
> > glibc types on the same arch, compatibility with glibc binary code
> > (apps or libs) is not affected. All that is affected if the ABI of
> > third-party libraries that use pthread types as members of public
> > structs, and it only matters if you're calling such libraries using
> > the glibc-derived ABI from code compiled against musl and using the
> > affected structures. This is a very unusual usage case, not something
> > we've ever prioritized supporting (it's broken in several other ways
> > anyway, or at least it was historically), and IMO it's not worth
> > severely bloating new archs that people actually want to use.
> 
> So to rephrase your arguments for that
> 
>  - For the case I compile my code with the glibc ABI and link it with
>    musl: any struct that has pthread (or C11 thread) components in it
>    will potentially only use a smaller part of that pthread (C11)
>    component. In particular in that case when compiling with the glibc
>    ABI the compiler always generates a consistent offset for the
>    pthread components. Sounds ok.

Yes.

>    But this only works because the glibc ABI doesn't export any
>    interface that has a visible combination of different pthread
>    components. Are we sure about that?

Yes. There are no public structs containing these, and it's unlikely
there ever would be. If so they could contain their own internal
padding to match.

>  - For the case that I compile code with the musl ABI and link it with
>    glibc, this may (and probably will) cause out-of-bounds errors on
>    execution. I would very much prefer that big warnings are already
>    issued when doing such a link, or even better if such a link would
>    fail systematically.

This does not work now anyway, and it's not intended to work.

- On glibc certain symbols like strerror_r map to nonstandard
  functions with different signatures and the standard function has
  some alternate name.

- On 32-bit archs the 32-bit-off_t functions would get used and
  horribly misinterpret their argument lists.

There may be other reasons I'm not thinking of right now too.

>    I hope that there is nobody out there, who would be currently using
>    this model, perhaps unknowingly.

If so it's already broken, so breaking quicker and more spectacularly
would be nice. :-)

Rich