implement stdatomic.h library interface

implement stdatomic.h library interface

[Jens Gustedt]

[-- Attachment #1: Type: text/plain, Size: 2165 bytes --]

Hello,
about half a year ago well already discussed this, and at that time
Joakim had a proposal for an implementation.

Looking back on this, I think that we had a bit of a wrong focus,
then. The point for musl wouldn't necessarily be to provide the
<stdatomic.h> header itself, but the stub functions that are needed if
the compiler isn't able to inline all operations.

Since I needed it, now, I just went for it and implemented it. For the
moment this isn't integrated to musl but standalone, and for a proper
integration into musl I'd still have some work to do.

At the moment I see three principal possibilities for improvement with
atomics in musl:

(1) Provide the library interfaces that gcc >= 4.8 and clang > 3.6 (?)
    need. The interface is well documented, so let's just do it.

(2) Provide an interface to atomics for compilers that don't have it
    yet. This can be done more or less easily up to some limits for
    all compilers that implement the __sync builtins. gcc 4.8 is a bit
    special since it implements the atomic operations but not the type
    qualification.

    For compilers that don't have __sync or __atomic we could provide
    a minimal interface with lock-free _Atomic(int), _Atomic(void*)
    and atomic_flag, say.

    The limits for these platforms would be that code could only use
    the _Atomic(bla) specification for the types and macro calls to
    act upon them. But since for pre-C11 compilers this is just an
    extension, that would be a start.

(3) Review the internal use of atomics. AFAIR, Rich has already
    started to look into that, but in any case that part is probably
    independent of the first two points.

    In a later stage, once (1) and (2) are stable, we could then look
    into using _Atomic(int) and the corresponding macros.

Jens


-- 
:: INRIA Nancy Grand Est ::: Camus ::::::: ICube/ICPS :::
:: ::::::::::::::: office Strasbourg : +33 368854536   ::
:: :::::::::::::::::::::: gsm France : +33 651400183   ::
:: ::::::::::::::: gsm international : +49 15737185122 ::
:: http://icube-icps.unistra.fr/index.php/Jens_Gustedt ::





[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: implement stdatomic.h library interface

[Joakim Sindholt]

On Thu, 2015-07-23 at 11:58 +0200, Jens Gustedt wrote:
> Hello,
> about half a year ago well already discussed this, and at that time
> Joakim had a proposal for an implementation.
> 
> Looking back on this, I think that we had a bit of a wrong focus,
> then. The point for musl wouldn't necessarily be to provide the
> <stdatomic.h> header itself, but the stub functions that are needed if
> the compiler isn't able to inline all operations.

I was mainly motivated by the __c11_atomic primitives in clang being
added in 3.1 but the header not being there until 3.6. I still think
it's worthwhile to provide stdatomic.h in musl, just as it provides
stddef.h.

> Since I needed it, now, I just went for it and implemented it. For the
> moment this isn't integrated to musl but standalone, and for a proper
> integration into musl I'd still have some work to do.

You (re)implemented all the libgcc/compiler-rt atomic symbols?

> At the moment I see three principal possibilities for improvement with
> atomics in musl:
> 
> (1) Provide the library interfaces that gcc >= 4.8 and clang > 3.6 (?)
>     need. The interface is well documented, so let's just do it.

GCC has __atomic in 4.7 (_Atomic and header added in 4.9 IIRC)
Clang has __c11_atomic in 3.1 (header added in 3.6)

> (2) Provide an interface to atomics for compilers that don't have it
>     yet. This can be done more or less easily up to some limits for
>     all compilers that implement the __sync builtins. gcc 4.8 is a bit
>     special since it implements the atomic operations but not the type
>     qualification.
> 
>     For compilers that don't have __sync or __atomic we could provide
>     a minimal interface with lock-free _Atomic(int), _Atomic(void*)
>     and atomic_flag, say.
> 
>     The limits for these platforms would be that code could only use
>     the _Atomic(bla) specification for the types and macro calls to
>     act upon them. But since for pre-C11 compilers this is just an
>     extension, that would be a start.

I suppose the real question here is how you're going to implement it
macro-wise to map to actual C functions. I've never really managed to
come up with a proper solution. I would love to have it though.

What I don't understand is why it's seen as a necessity to re-implement
compiler-specific functionality. Musl doesn't provide umoddi3 or other
libgcc symbols, so why the stdatomic internals? Are you trying to
compile without a compiler runtime library?
I get that it could be useful to have instead of musl's current internal
atomics but IMO only if it actually saves you the work of implementing
the atomic primitives in the first place.

-- Joakim

Re: implement stdatomic.h library interface

[Jens Gustedt]

[-- Attachment #1: Type: text/plain, Size: 4967 bytes --]

Hello,

Am Donnerstag, den 23.07.2015, 14:14 +0200 schrieb Joakim Sindholt:
> > Looking back on this, I think that we had a bit of a wrong focus,
> > then. The point for musl wouldn't necessarily be to provide the
> > <stdatomic.h> header itself, but the stub functions that are needed if
> > the compiler isn't able to inline all operations.
> 
> I was mainly motivated by the __c11_atomic primitives in clang being
> added in 3.1 but the header not being there until 3.6. I still think
> it's worthwhile to provide stdatomic.h in musl, just as it provides
> stddef.h.

Ok, I see.

I think that this situation is relatively easy to detect (e.g testing
for __ATOMIC_RELAXED) and we can then provide a replacement. But my
idea is that generally, if the compiler provides a <stdatomic.h>, we
should use that. These will probably improve in time, we shouldn't cut
off from that.

> > Since I needed it, now, I just went for it and implemented it. For the
> > moment this isn't integrated to musl but standalone, and for a proper
> > integration into musl I'd still have some work to do.
> 
> You (re)implemented all the libgcc/compiler-rt atomic symbols?

We'll see if I have them all, but it is less difficult than it
sounds. There is a lot of repetition.

> > At the moment I see three principal possibilities for improvement with
> > atomics in musl:
> > 
> > (1) Provide the library interfaces that gcc >= 4.8 and clang > 3.6 (?)
> >     need. The interface is well documented, so let's just do it.
> 
> GCC has __atomic in 4.7 (_Atomic and header added in 4.9 IIRC)
> Clang has __c11_atomic in 3.1 (header added in 3.6)

I wasn't aware of 4.7 already implementing that stuff. My current
strategy is

if __ATOMIC_RELAXED is there,
  if there is __clang__ use for __c11_atomic
  else use __atomic
else use __sync

> > (2) Provide an interface to atomics for compilers that don't have it
> >     yet. This can be done more or less easily up to some limits for
> >     all compilers that implement the __sync builtins. gcc 4.8 is a bit
> >     special since it implements the atomic operations but not the type
> >     qualification.
> > 
> >     For compilers that don't have __sync or __atomic we could provide
> >     a minimal interface with lock-free _Atomic(int), _Atomic(void*)
> >     and atomic_flag, say.
> > 
> >     The limits for these platforms would be that code could only use
> >     the _Atomic(bla) specification for the types and macro calls to
> >     act upon them. But since for pre-C11 compilers this is just an
> >     extension, that would be a start.
> 
> I suppose the real question here is how you're going to implement it
> macro-wise to map to actual C functions. I've never really managed to
> come up with a proper solution. I would love to have it though.

For compilers that implement the __atomic.. or __c11_atomic..  this is
not very difficult. Basically the compilers are providing all the
logic to differentiate the cases.

For the __sync... interfaces I am currently doing that. This uses
__builtin_choose_expr as a poor-man's _Generic to separate the special
cases for the size. Its a bit ugly, but does the job.

(And it also forces the choice expression to be compile time constant,
so the effects of multiple evaluation in __typeof__ can be mastered.)

> What I don't understand is why it's seen as a necessity to re-implement
> compiler-specific functionality. Musl doesn't provide umoddi3 or other
> libgcc symbols, so why the stdatomic internals? Are you trying to
> compile without a compiler runtime library?

Same reason as your motivation for having a working <stdatomic.h> for
previous versions of clang. It would be equally nice to have such a
thing, even if the support would only partial (only the functions, not
operators) for older gcc's.

Many people will be stuck with older compilers for years. This would
offer an easy an portable way to use that feature already, the same
way as it is possible to use C11 threads without having a compiler
that is fully compliant to C11.

> I get that it could be useful to have instead of musl's current internal
> atomics but IMO only if it actually saves you the work of implementing
> the atomic primitives in the first place.

Implementing just a frontend to atomics by dispatching them to
__sync/__atomic/__c11 builtins is much, much simpler than doing that
on an per-arch base, I think. Also, it might enable us to use finer
memory models for certain operations (well, we'll see) and to profit
from improvements that are done on the compiler side.

Jens

-- 
:: INRIA Nancy Grand Est ::: Camus ::::::: ICube/ICPS :::
:: ::::::::::::::: office Strasbourg : +33 368854536   ::
:: :::::::::::::::::::::: gsm France : +33 651400183   ::
:: ::::::::::::::: gsm international : +49 15737185122 ::
:: http://icube-icps.unistra.fr/index.php/Jens_Gustedt ::





[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: implement stdatomic.h library interface

[Rich Felker]

On Thu, Jul 23, 2015 at 02:14:15PM +0200, Joakim Sindholt wrote:
> What I don't understand is why it's seen as a necessity to re-implement
> compiler-specific functionality. Musl doesn't provide umoddi3 or other
> libgcc symbols, so why the stdatomic internals? Are you trying to
> compile without a compiler runtime library?
> I get that it could be useful to have instead of musl's current internal
> atomics but IMO only if it actually saves you the work of implementing
> the atomic primitives in the first place.

I don't have a set-in-stone position yet on whether this should be
done, but I can give a little bit of background on the libgcc side.
Originally I had considered implementing all of the libgcc-type
compiler runtime functions in musl (and providing external definitions
for them) to eliminate the need for an additional library and possibly
allow for more-optimized implementations. However, it turned out that
there's quite a bit of diversity in the set of symbols, signatures,
calling conventions, etc. that are needed from one arch to another,
and trying to go down this path would have meant maintaining a lot
more arch-specific code that's a duplication of work already done in
the compiler's target libs.

I'm not sure whether these issues apply to the atomic fallback
functions. If not, then given that there seem to be serious issues
with the gcc libatomic implementation (the whole ifunc mess, compat
problems between code built with 'hard' and 'soft' atomics, etc.) it
_might_ make sense to provide them in musl. But I'd like to make sure
we consider all other options, including possibly a separate
replacement for libatomic that's independent of musl.

One question about gcc's libatomic: does it work correctly with musl
if you've told gcc at build time that ifunc is not supported?

Rich

Re: implement stdatomic.h library interface

[Jens Gustedt]

[-- Attachment #1: Type: text/plain, Size: 4074 bytes --]

Hello,

Am Freitag, den 24.07.2015, 19:18 -0400 schrieb Rich Felker:
> On Thu, Jul 23, 2015 at 02:14:15PM +0200, Joakim Sindholt wrote:
> > What I don't understand is why it's seen as a necessity to re-implement
> > compiler-specific functionality. Musl doesn't provide umoddi3 or other
> > libgcc symbols, so why the stdatomic internals? Are you trying to
> > compile without a compiler runtime library?
> > I get that it could be useful to have instead of musl's current internal
> > atomics but IMO only if it actually saves you the work of implementing
> > the atomic primitives in the first place.
> 
> I don't have a set-in-stone position yet on whether this should be
> done, but I can give a little bit of background on the libgcc side.
> Originally I had considered implementing all of the libgcc-type
> compiler runtime functions in musl (and providing external definitions
> for them) to eliminate the need for an additional library and possibly
> allow for more-optimized implementations. However, it turned out that
> there's quite a bit of diversity in the set of symbols, signatures,
> calling conventions, etc. that are needed from one arch to another,
> and trying to go down this path would have meant maintaining a lot
> more arch-specific code that's a duplication of work already done in
> the compiler's target libs.

yes, I imagine, this would have been quite tedious, I definitively
don't want to go for arch specific code, here

> I'm not sure whether these issues apply to the atomic fallback
> functions.

I don't think so. The assembler that the newer versions of gcc and
clang produce looks quite good. I don't have an architecture where
that is really relevant at hand, but AFAICS clang seems to take the
memory_order argument into account.

Generally, by using these interfaces we can profit of all advances
that the compiler and architecture people make for these kind of ops.

> If not, then given that there seem to be serious issues
> with the gcc libatomic implementation (the whole ifunc mess, compat
> problems between code built with 'hard' and 'soft' atomics, etc.) it
> _might_ make sense to provide them in musl. But I'd like to make sure
> we consider all other options, including possibly a separate
> replacement for libatomic that's independent of musl.

First, the gcc libatomic is what it is, gcc. So a complete
installation of musl then would always depend on the presence of
gcc. And if maybe this needs a gcc (or parts of its library) that is
compiled with special options, this sounds relatively prohibitive to
me.

clang relies on the C library (it seems on bsd) or a gcc installation
as a fallback.

Then the two options you mention are not exclusive. The code I have
now works as standalone library and can be injected to musl to be
compiled as part of it.

One argument for me to want it in musl is that it is relying on locks
for the lockfull versions of the calls. gcc libatomic does this with
pthread_mutex_t, which is there only reasonable choice at the moment
since C11 threads are not present in glibc, yet.

My implementation within musl just uses LOCK/UNLOCK. This seems to be
more appropriate if you have it. Lockfull atomics are not expected to
work in the context of signal handlers, so a minimal lock structure
with non-active wait does the trick.

When compiled standalone, my code uses a simple spinlock. This makes
it less efficient on high contention, but also makes it completely
independent of the OS or C library.

> One question about gcc's libatomic: does it work correctly with musl
> if you've told gcc at build time that ifunc is not supported?

I am not building gcc myself, so I can't answer that.

Jens

-- 
:: INRIA Nancy Grand Est ::: Camus ::::::: ICube/ICPS :::
:: ::::::::::::::: office Strasbourg : +33 368854536   ::
:: :::::::::::::::::::::: gsm France : +33 651400183   ::
:: ::::::::::::::: gsm international : +49 15737185122 ::
:: http://icube-icps.unistra.fr/index.php/Jens_Gustedt ::





[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: implement stdatomic.h library interface

[Joakim Sindholt]

On Sat, 2015-07-25 at 08:33 +0200, Jens Gustedt wrote:
> > If not, then given that there seem to be serious issues
> > with the gcc libatomic implementation (the whole ifunc mess, compat
> > problems between code built with 'hard' and 'soft' atomics, etc.) it
> > _might_ make sense to provide them in musl. But I'd like to make sure
> > we consider all other options, including possibly a separate
> > replacement for libatomic that's independent of musl.
> 
> First, the gcc libatomic is what it is, gcc. So a complete
> installation of musl then would always depend on the presence of
> gcc. And if maybe this needs a gcc (or parts of its library) that is
> compiled with special options, this sounds relatively prohibitive to
> me.

The ifunc mess means that it just plain wont work on musl, and musl
doesn't support ifuncs because they're effectively a completely
unspecified hack to turn existing ABI into function pointers. I'm not
sure if it's even possible to support in static binaries.
So regardless of how any of us feel about it, we would need to either
get ifunc support or replace gcc's libatomic.

> clang relies on the C library (it seems on bsd) or a gcc installation
> as a fallback.

I'm not sure what you mean by this. Can you clarify? Compiler-rt has all
of clang's own fallback symbols and it looks like they're incompatible
with libatomic.

As far as I can tell they just use regular spinlocks on linux without
any libc dependency what-so-ever and on FreeBSD it goes straight to
_umtx_op which looks like their version of futex.

; nm -g amd64/lib/libcompiler_rt.a | grep atomic
atomic.o:
0000000000000130 T __atomic_compare_exchange
0000000000000880 T __atomic_compare_exchange_1
00000000000008e0 T __atomic_compare_exchange_2
0000000000000940 T __atomic_compare_exchange_4
00000000000009a0 T __atomic_compare_exchange_8
0000000000000580 T __atomic_exchange
0000000000000790 T __atomic_exchange_1
00000000000007d0 T __atomic_exchange_2
0000000000000810 T __atomic_exchange_4
0000000000000840 T __atomic_exchange_8
0000000000000a00 T __atomic_fetch_add_1
0000000000000a40 T __atomic_fetch_add_2
0000000000000a80 T __atomic_fetch_add_4
0000000000000ac0 T __atomic_fetch_add_8
0000000000000c00 T __atomic_fetch_and_1
0000000000000c90 T __atomic_fetch_and_2
0000000000000d20 T __atomic_fetch_and_4
0000000000000da0 T __atomic_fetch_and_8
0000000000000e70 T __atomic_fetch_or_1
0000000000000f00 T __atomic_fetch_or_2
0000000000000f90 T __atomic_fetch_or_4
0000000000001010 T __atomic_fetch_or_8
0000000000000b00 T __atomic_fetch_sub_1
0000000000000b40 T __atomic_fetch_sub_2
0000000000000b80 T __atomic_fetch_sub_4
0000000000000bc0 T __atomic_fetch_sub_8
00000000000010e0 T __atomic_fetch_xor_1
0000000000001170 T __atomic_fetch_xor_2
0000000000001200 T __atomic_fetch_xor_4
0000000000001280 T __atomic_fetch_xor_8
0000000000000000 T __atomic_load
00000000000006c0 T __atomic_load_1
00000000000006e0 T __atomic_load_2
0000000000000700 T __atomic_load_4
0000000000000710 T __atomic_load_8
0000000000000090 T __atomic_store
0000000000000720 T __atomic_store_1
0000000000000740 T __atomic_store_2
0000000000000760 T __atomic_store_4
0000000000000770 T __atomic_store_8
atomic_flag_clear.o:
0000000000000000 T atomic_flag_clear
atomic_flag_clear_explicit.o:
0000000000000000 T atomic_flag_clear_explicit
atomic_flag_test_and_set.o:
0000000000000000 T atomic_flag_test_and_set
atomic_flag_test_and_set_explicit.o:
0000000000000000 T atomic_flag_test_and_set_explicit
atomic_signal_fence.o:
0000000000000000 T atomic_signal_fence
atomic_thread_fence.o:
0000000000000000 T atomic_thread_fence

https://github.com/llvm-mirror/compiler-rt/blob/master/lib/builtins/atomic.c

-- Joakim

Re: implement stdatomic.h library interface

[Jens Gustedt]

[-- Attachment #1: Type: text/plain, Size: 1368 bytes --]

Hello,

Am Samstag, den 25.07.2015, 11:17 +0200 schrieb Joakim Sindholt:
> On Sat, 2015-07-25 at 08:33 +0200, Jens Gustedt wrote:
> > clang relies on the C library (it seems on bsd) or a gcc installation
> > as a fallback.
> 
> I'm not sure what you mean by this. Can you clarify? Compiler-rt has all
> of clang's own fallback symbols and it looks like they're incompatible
> with libatomic.
> 
> As far as I can tell they just use regular spinlocks on linux without
> any libc dependency what-so-ever and on FreeBSD it goes straight to
> _umtx_op which looks like their version of futex.

so I must have read some obsolete documentation somewhere

In any case a lock with kernel support seems to be preferable. I have
a test application that wildly allocates and deallocates list elements
and inserts them atomically. When there is a 16 byte atomic all is
done with that, if not the lockfull replacement is used. For the
lockfull version I seem to get 3 times more work done when using
LOCK/UNLOCK over just using a spinlock.

Jens


-- 
:: INRIA Nancy Grand Est ::: Camus ::::::: ICube/ICPS :::
:: ::::::::::::::: office Strasbourg : +33 368854536   ::
:: :::::::::::::::::::::: gsm France : +33 651400183   ::
:: ::::::::::::::: gsm international : +49 15737185122 ::
:: http://icube-icps.unistra.fr/index.php/Jens_Gustedt ::




[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: implement stdatomic.h library interface

[Szabolcs Nagy]

* Jens Gustedt <jens.gustedt@inria.fr> [2015-07-23 11:58:28 +0200]:
> Looking back on this, I think that we had a bit of a wrong focus,
> then. The point for musl wouldn't necessarily be to provide the
> <stdatomic.h> header itself, but the stub functions that are needed if
> the compiler isn't able to inline all operations.
> 
> Since I needed it, now, I just went for it and implemented it. For the
> moment this isn't integrated to musl but standalone, and for a proper
> integration into musl I'd still have some work to do.
> 
> At the moment I see three principal possibilities for improvement with
> atomics in musl:
> 
> (1) Provide the library interfaces that gcc >= 4.8 and clang > 3.6 (?)
>     need. The interface is well documented, so let's just do it.
> 
> (2) Provide an interface to atomics for compilers that don't have it
>     yet. This can be done more or less easily up to some limits for
>     all compilers that implement the __sync builtins. gcc 4.8 is a bit
>     special since it implements the atomic operations but not the type
>     qualification.
> 

so you plan to implement gcc libatomic abi in musl?

it seems gcc does not pass -latomic automatically to the linker
which is ugly (i think atomics should work transparently), but
ppl using atomics will.

(there is another problem that gcc libatomic on x86 uses ifunc
dispatch for some __atomic_*_16 functions which does not work
with musl now, this can be worked around by rebuilding gcc with
--disable-gnu-indirect-function.)

>     For compilers that don't have __sync or __atomic we could provide
>     a minimal interface with lock-free _Atomic(int), _Atomic(void*)
>     and atomic_flag, say.
> 
>     The limits for these platforms would be that code could only use
>     the _Atomic(bla) specification for the types and macro calls to
>     act upon them. But since for pre-C11 compilers this is just an
>     extension, that would be a start.
> 
> (3) Review the internal use of atomics. AFAIR, Rich has already
>     started to look into that, but in any case that part is probably
>     independent of the first two points.
> 
>     In a later stage, once (1) and (2) are stable, we could then look
>     into using _Atomic(int) and the corresponding macros.
> 
> Jens
> 
> 
> -- 
> :: INRIA Nancy Grand Est ::: Camus ::::::: ICube/ICPS :::
> :: ::::::::::::::: office Strasbourg : +33 368854536   ::
> :: :::::::::::::::::::::: gsm France : +33 651400183   ::
> :: ::::::::::::::: gsm international : +49 15737185122 ::
> :: http://icube-icps.unistra.fr/index.php/Jens_Gustedt ::
> 
> 
> 
> 

Re: implement stdatomic.h library interface

[Jens Gustedt]

[-- Attachment #1: Type: text/plain, Size: 2481 bytes --]

Am Donnerstag, den 23.07.2015, 15:05 +0200 schrieb Szabolcs Nagy:
> so you plan to implement gcc libatomic abi in musl?

yes

> it seems gcc does not pass -latomic automatically to the linker
> which is ugly (i think atomics should work transparently), but
> ppl using atomics will.

I think the situation is somewhat comparable to -lm. Other libraries
may implement that as separate .a or .so, we should just integrate it
to the whole, and provide an empty libatomic.[a|so]

So with whatever arguments people call their linker, it should work
for us.

> (there is another problem that gcc libatomic on x86 uses ifunc
> dispatch for some __atomic_*_16 functions which does not work
> with musl now, this can be worked around by rebuilding gcc with
> --disable-gnu-indirect-function.)

Can you explain in more detail what is not working?

The libatomic abi consists in providing stubs for all
__atomic_*_[1|2|4|8|16] functions, for the case that these can't use
processor atomics.

I plan to provide all these, by pointing to the lock-full "memory"
variants if there is not processor CAS for the type. These "memory"
variants are also the fallback for all other sizes. Works sufficiently
well already, I think.

The memory version is important for atomics on structs that are used
for lock-free dynamic data types. Typically they would need some
atomic for a pair of pointers or so, to avoid the ABA problem. On some
archs such a pair would fit into a lock-free atomic, on others it'd
have to be done by a lockfull replacement.

My strategy is to provide versions according to the presence or not of
__GCC_HAVE_SYNC_COMPARE_AND_SWAP_XX

But you are right, the "16" case, that is 16byte atomics need some
careful inspection. The documentation says something that they provide
the primitives if __int128_t exists. (Or suppose that the stub
functions then are implemented?)

What also bugs me a bit, is that I'd like to prevent objects that are
compiled with different architecture version together. If one uses a
processor atomic for 16 byte and the other goes into the "memory"
variant, they wouldn't even detect it.

Jens

-- 
:: INRIA Nancy Grand Est ::: Camus ::::::: ICube/ICPS :::
:: ::::::::::::::: office Strasbourg : +33 368854536   ::
:: :::::::::::::::::::::: gsm France : +33 651400183   ::
:: ::::::::::::::: gsm international : +49 15737185122 ::
:: http://icube-icps.unistra.fr/index.php/Jens_Gustedt ::




[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: implement stdatomic.h library interface

[Szabolcs Nagy]

* Jens Gustedt <jens.gustedt@inria.fr> [2015-07-23 16:04:14 +0200]:
> Am Donnerstag, den 23.07.2015, 15:05 +0200 schrieb Szabolcs Nagy:
> > it seems gcc does not pass -latomic automatically to the linker
> > which is ugly (i think atomics should work transparently), but
> > ppl using atomics will.
> 
> I think the situation is somewhat comparable to -lm. Other libraries
> may implement that as separate .a or .so, we should just integrate it
> to the whole, and provide an empty libatomic.[a|so]

i think it's a bit different because -lm is libc and libatomic is
compiler runtime now. (and extern linkage behaviour for math functions
is more clear than for "generic" atomic functions).

> > (there is another problem that gcc libatomic on x86 uses ifunc
> > dispatch for some __atomic_*_16 functions which does not work
> > with musl now, this can be worked around by rebuilding gcc with
> > --disable-gnu-indirect-function.)
> 
> Can you explain in more detail what is not working?

i mean that on a normal build of gcc >=4.9 libatomic uses
the gnu IFUNC feature (runtime dispatch for function symbols
based on cpu features at early startup).

this IFUNC elf extension is not supported by musl now so
loading a shared object with such symbols will fail
(not sure what happens with static linking) so you have
to build a musl specific toolchain now to use libatomic.

(some x86_64 cpus have cmpxchg16b and there is different
__atomic_*_16 implementation for them).

Re: implement stdatomic.h library interface

[Jens Gustedt]

[-- Attachment #1: Type: text/plain, Size: 2651 bytes --]

Am Donnerstag, den 23.07.2015, 16:42 +0200 schrieb Szabolcs Nagy:
> * Jens Gustedt <jens.gustedt@inria.fr> [2015-07-23 16:04:14 +0200]:
> > Am Donnerstag, den 23.07.2015, 15:05 +0200 schrieb Szabolcs Nagy:
> > > it seems gcc does not pass -latomic automatically to the linker
> > > which is ugly (i think atomics should work transparently), but
> > > ppl using atomics will.
> > 
> > I think the situation is somewhat comparable to -lm. Other libraries
> > may implement that as separate .a or .so, we should just integrate it
> > to the whole, and provide an empty libatomic.[a|so]
> 
> i think it's a bit different because -lm is libc and libatomic is
> compiler runtime now. (and extern linkage behaviour for math functions
> is more clear than for "generic" atomic functions).

It is not clear if this is compiler runtime. gcc provides libatomic,
clang refers to the C library.

In any case, do you think that there would be a problem with the
approach that I propose. (provided that everybody sticks to the ABI
...)

> > > (there is another problem that gcc libatomic on x86 uses ifunc
> > > dispatch for some __atomic_*_16 functions which does not work
> > > with musl now, this can be worked around by rebuilding gcc with
> > > --disable-gnu-indirect-function.)
> > 
> > Can you explain in more detail what is not working?
> 
> i mean that on a normal build of gcc >=4.9 libatomic uses
> the gnu IFUNC feature (runtime dispatch for function symbols
> based on cpu features at early startup).

I see how that can cause problems for us.

But I don't really see how that helps. To be efficient instruction
based lock-free atomics should be inlined at compile time, no? Or do
they do lto on that kind of stuff?

> this IFUNC elf extension is not supported by musl now so
> loading a shared object with such symbols will fail
> (not sure what happens with static linking) so you have
> to build a musl specific toolchain now to use libatomic.

Static linking seems to work fine.

> (some x86_64 cpus have cmpxchg16b and there is different
> __atomic_*_16 implementation for them).

I think that it is even fair to say many if not most have them,
nowadays. The question here usually is just if people compile with
-march=native or not, which then can produce incompatible binaries.

Jens

-- 
:: INRIA Nancy Grand Est ::: Camus ::::::: ICube/ICPS :::
:: ::::::::::::::: office Strasbourg : +33 368854536   ::
:: :::::::::::::::::::::: gsm France : +33 651400183   ::
:: ::::::::::::::: gsm international : +49 15737185122 ::
:: http://icube-icps.unistra.fr/index.php/Jens_Gustedt ::




[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: implement stdatomic.h library interface

[Joakim Sindholt]

On Thu, 2015-07-23 at 17:04 +0200, Jens Gustedt wrote:
> Am Donnerstag, den 23.07.2015, 16:42 +0200 schrieb Szabolcs Nagy:
> > * Jens Gustedt <jens.gustedt@inria.fr> [2015-07-23 16:04:14 +0200]:
> > > Am Donnerstag, den 23.07.2015, 15:05 +0200 schrieb Szabolcs Nagy:
> > > > it seems gcc does not pass -latomic automatically to the linker
> > > > which is ugly (i think atomics should work transparently), but
> > > > ppl using atomics will.
> > > 
> > > I think the situation is somewhat comparable to -lm. Other libraries
> > > may implement that as separate .a or .so, we should just integrate it
> > > to the whole, and provide an empty libatomic.[a|so]
> > 
> > i think it's a bit different because -lm is libc and libatomic is
> > compiler runtime now. (and extern linkage behaviour for math functions
> > is more clear than for "generic" atomic functions).
> 
> It is not clear if this is compiler runtime. gcc provides libatomic,
> clang refers to the C library.

compiler-rt provides the fallback symbols:
https://github.com/llvm-mirror/compiler-rt/blob/master/lib/builtins/atomic.c

> In any case, do you think that there would be a problem with the
> approach that I propose. (provided that everybody sticks to the ABI
> ...)
> 
> > > > (there is another problem that gcc libatomic on x86 uses ifunc
> > > > dispatch for some __atomic_*_16 functions which does not work
> > > > with musl now, this can be worked around by rebuilding gcc with
> > > > --disable-gnu-indirect-function.)
> > > 
> > > Can you explain in more detail what is not working?
> > 
> > i mean that on a normal build of gcc >=4.9 libatomic uses
> > the gnu IFUNC feature (runtime dispatch for function symbols
> > based on cpu features at early startup).
> 
> I see how that can cause problems for us.
> 
> But I don't really see how that helps. To be efficient instruction
> based lock-free atomics should be inlined at compile time, no? Or do
> they do lto on that kind of stuff?

The symbols are only used as a fallback, just like with large division
and the like. At least with clang the instructions are directly
inserted. The reason they use ifuncs is to optimize the slow path (dumb
as it sounds).

> > this IFUNC elf extension is not supported by musl now so
> > loading a shared object with such symbols will fail
> > (not sure what happens with static linking) so you have
> > to build a musl specific toolchain now to use libatomic.
> 
> Static linking seems to work fine.

Then you're not hitting the fallback symbols. They will initially be
NULL and you should get SIGSEGV. At least this was the result I saw when
trying to use ifunc resolver in a static binary with musl.

I still don't know why I get caught in the spam filter. This email will
be as delayed as the one I sent hours ago as it needs manual approval. I
apologize if this is antiquated by the time it reaches the list.
-- Joakim

Re: implement stdatomic.h library interface

[Szabolcs Nagy]

* Joakim Sindholt <opensource@zhasha.com> [2015-07-23 18:32:26 +0200]:
> On Thu, 2015-07-23 at 17:04 +0200, Jens Gustedt wrote:
> > Am Donnerstag, den 23.07.2015, 16:42 +0200 schrieb Szabolcs Nagy:
> > > this IFUNC elf extension is not supported by musl now so
> > > loading a shared object with such symbols will fail
> > > (not sure what happens with static linking) so you have
> > > to build a musl specific toolchain now to use libatomic.
> > 
> > Static linking seems to work fine.
> 
> Then you're not hitting the fallback symbols. They will initially be
> NULL and you should get SIGSEGV. At least this was the result I saw when
> trying to use ifunc resolver in a static binary with musl.

these are the ifunc symbols in current gcc libatomic on x86_64:

$ nm libatomic.a |grep ' i '
0000000000000040 i __atomic_load_16
0000000000000030 i __atomic_store_16
0000000000000070 i __atomic_compare_exchange_16
0000000000000050 i __atomic_exchange_16
00000000000000c0 i __atomic_add_fetch_16
00000000000000a0 i __atomic_fetch_add_16
00000000000000b0 i __atomic_fetch_sub_16
00000000000000d0 i __atomic_sub_fetch_16
00000000000000b0 i __atomic_and_fetch_16
0000000000000090 i __atomic_fetch_and_16
0000000000000090 i __atomic_fetch_or_16
00000000000000b0 i __atomic_or_fetch_16
0000000000000090 i __atomic_fetch_xor_16
00000000000000b0 i __atomic_xor_fetch_16
0000000000000090 i __atomic_fetch_nand_16
00000000000000b0 i __atomic_nand_fetch_16
0000000000000010 i __atomic_test_and_set_16

Re: implement stdatomic.h library interface

[Jens Gustedt]

[-- Attachment #1: Type: text/plain, Size: 2751 bytes --]

Am Donnerstag, den 23.07.2015, 18:32 +0200 schrieb Joakim Sindholt:
> > But I don't really see how that helps. To be efficient instruction
> > based lock-free atomics should be inlined at compile time, no? Or do
> > they do lto on that kind of stuff?
> 
> The symbols are only used as a fallback, just like with large division
> and the like. At least with clang the instructions are directly
> inserted. The reason they use ifuncs is to optimize the slow path (dumb
> as it sounds).

sure, so my understanding is, that these IFUNC symbols are used as
fallbacks. The IFUNC mechanism is then used to put arch specific code
in place as such a replacement.

> > > this IFUNC elf extension is not supported by musl now so
> > > loading a shared object with such symbols will fail
> > > (not sure what happens with static linking) so you have
> > > to build a musl specific toolchain now to use libatomic.
> > 
> > Static linking seems to work fine.
> 
> Then you're not hitting the fallback symbols. They will initially be
> NULL and you should get SIGSEGV. At least this was the result I saw when
> trying to use ifunc resolver in a static binary with musl.

I probably explained wrong what I was doing. Sure if we would use the
existing libatomic, we would hit that wall. Thus the need to implement
these calls in musl directly, such that they don't use or need IFUNC.

What I was refering to as "working" is such a different approach:

at musl compile time, decide which atomics are supported on
instruction level (or not). This can be done by testing for
__GCC_HAVE_SYNC_COMPARE_AND_SWAP_XX

For those where it is supported, compile the stubs by using the
builtins. These stubs will probably never be used, but they are there
if a user compiles with an oldish compiler, conservative arch
options or switches off the builtins.

Those where it is not supported are implemented with the generic
lockfull memcpy-memcmp version. On my platform this is usually the 16
byte version, or not depending on the arch options.

This is a bit less optimal than the IFUNC version, because it fixes
the instruction set that can be used at musl-compile time.

> I still don't know why I get caught in the spam filter. This email will
> be as delayed as the one I sent hours ago as it needs manual approval. I
> apologize if this is antiquated by the time it reaches the list.

No problem for me.

Jens


-- 
:: INRIA Nancy Grand Est ::: Camus ::::::: ICube/ICPS :::
:: ::::::::::::::: office Strasbourg : +33 368854536   ::
:: :::::::::::::::::::::: gsm France : +33 651400183   ::
:: ::::::::::::::: gsm international : +49 15737185122 ::
:: http://icube-icps.unistra.fr/index.php/Jens_Gustedt ::





[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]