* Re: FreeBSD's Google Summer of Code 2016
2016-03-06 0:25 ` Rich Felker
@ 2016-03-06 0:31 ` Rich Felker
2016-03-06 1:11 ` Pedro Giffuni
2016-03-06 0:59 ` Pedro Giffuni
2016-03-07 8:32 ` Felix Fietkau
2 siblings, 1 reply; 14+ messages in thread
From: Rich Felker @ 2016-03-06 0:31 UTC (permalink / raw)
To: Pedro Giffuni; +Cc: musl
On Sat, Mar 05, 2016 at 07:25:47PM -0500, Rich Felker wrote:
> On Sat, Mar 05, 2016 at 07:14:34PM -0500, Pedro Giffuni wrote:
> >
> >
> > On 03/05/16 18:32, Rich Felker wrote:
> > >On Sat, Mar 05, 2016 at 05:41:25PM -0500, Pedro Giffuni wrote:
> > >>First of all, great to hear there is interest on the musl side too.
> > >>
> > >>I think the biggest precedent of porting linux-oriented C libraries
> > >>came from Debian's kFreeBSD. We accomodated a little by for them
> > >>by defining __FreeBSD_kernel__ in sys/param.h.
> > >>
> > >>While using the optional linux-abi futex in FreeBSD could be an option,
> > >>it is not really the cleanest option. The Debian guys did a port of
> > >>NPTL using regular pthreads:
> > >>
> >
> > Of course I ahould have meant "based on regular FreeBSD kernel services".
> >
> > >>http://thread.gmane.org/gmane.linux.debian.ports.bsd/11702
> > >>
> > >>I am certain this will require more research but it would be useful
> > >>for other ports as well.
> > >
> >
> > We could ask Petr Salinger for the details, but I am pretty sure
> > FreeBSD has the required functionality natively.
> >
> > >Glibc/NPTL has a lot of what I'd call "gratuitous abstraction" (like
> > >the lll stuff) in their pthread primitives which makes this
> > >"possible". I call it gratuitous because it's really really hard to
> > >achieve correct implementations of the pthread sync primitives that
> > >don't have serious corner-case bugs, and it's unlikely that their
> > >abstractions actually suffice to make correct alternate
> > >implementations.
> > >
> > >musl does not have any such abstraction. We require a compare-and-swap
> > >operation or equivalent on which arbitrary atomic operations can be
> > >constructed, a futex or equivalent operation that's roughly
> > >while(*addr==expected) sleep(), and implement all the sync primitives
> > >just once on top of these.
> > >
> >
> > I am not a threading expert (or even a CS guy), but it sounds like
> > mutex(9) with condvar(9) would do [1]:
>
> No, they don't satisfy the needs of musl; they have their own
> additional storage requirements and are probably not AS-safe. It might
> be possible to use them to implement a userspace-emulated futex queue
> (only if they are AS-safe), but I don't see a way to extend that to
> the process-shared case.
Actually these look like kernelspace APIs, not userspace, unless I'm
mistaken. In that case I don't see how they're relevant/usable. Is tht
correct?
Rich
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: FreeBSD's Google Summer of Code 2016
2016-03-06 0:31 ` Rich Felker
@ 2016-03-06 1:11 ` Pedro Giffuni
0 siblings, 0 replies; 14+ messages in thread
From: Pedro Giffuni @ 2016-03-06 1:11 UTC (permalink / raw)
To: Rich Felker; +Cc: musl
On 03/05/16 19:31, Rich Felker wrote:
> On Sat, Mar 05, 2016 at 07:25:47PM -0500, Rich Felker wrote:
>> On Sat, Mar 05, 2016 at 07:14:34PM -0500, Pedro Giffuni wrote:
>>>
>>>
>>> On 03/05/16 18:32, Rich Felker wrote:
>>>> On Sat, Mar 05, 2016 at 05:41:25PM -0500, Pedro Giffuni wrote:
>>>>> First of all, great to hear there is interest on the musl side too.
>>>>>
>>>>> I think the biggest precedent of porting linux-oriented C libraries
>>>>> came from Debian's kFreeBSD. We accomodated a little by for them
>>>>> by defining __FreeBSD_kernel__ in sys/param.h.
>>>>>
>>>>> While using the optional linux-abi futex in FreeBSD could be an option,
>>>>> it is not really the cleanest option. The Debian guys did a port of
>>>>> NPTL using regular pthreads:
>>>>>
>>>
>>> Of course I ahould have meant "based on regular FreeBSD kernel services".
>>>
>>>>> http://thread.gmane.org/gmane.linux.debian.ports.bsd/11702
>>>>>
>>>>> I am certain this will require more research but it would be useful
>>>>> for other ports as well.
>>>>
>>>
>>> We could ask Petr Salinger for the details, but I am pretty sure
>>> FreeBSD has the required functionality natively.
>>>
>>>> Glibc/NPTL has a lot of what I'd call "gratuitous abstraction" (like
>>>> the lll stuff) in their pthread primitives which makes this
>>>> "possible". I call it gratuitous because it's really really hard to
>>>> achieve correct implementations of the pthread sync primitives that
>>>> don't have serious corner-case bugs, and it's unlikely that their
>>>> abstractions actually suffice to make correct alternate
>>>> implementations.
>>>>
>>>> musl does not have any such abstraction. We require a compare-and-swap
>>>> operation or equivalent on which arbitrary atomic operations can be
>>>> constructed, a futex or equivalent operation that's roughly
>>>> while(*addr==expected) sleep(), and implement all the sync primitives
>>>> just once on top of these.
>>>>
>>>
>>> I am not a threading expert (or even a CS guy), but it sounds like
>>> mutex(9) with condvar(9) would do [1]:
>>
>> No, they don't satisfy the needs of musl; they have their own
>> additional storage requirements and are probably not AS-safe. It might
>> be possible to use them to implement a userspace-emulated futex queue
>> (only if they are AS-safe), but I don't see a way to extend that to
>> the process-shared case.
>
> Actually these look like kernelspace APIs, not userspace, unless I'm
> mistaken. In that case I don't see how they're relevant/usable. Is tht
> correct?
>
Ugh .. yeah ... sem(9) is kernel only API too.
The best is to look at the native thread library, not precisely
the easiest to read code in the world, I'm afraid.
OpenGrok may help:
http://src.illumos.org/source/xref/freebsd-head/lib/libthr/
Pedro.
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: FreeBSD's Google Summer of Code 2016
2016-03-06 0:25 ` Rich Felker
2016-03-06 0:31 ` Rich Felker
@ 2016-03-06 0:59 ` Pedro Giffuni
2016-03-06 1:08 ` Rich Felker
2016-03-07 8:32 ` Felix Fietkau
2 siblings, 1 reply; 14+ messages in thread
From: Pedro Giffuni @ 2016-03-06 0:59 UTC (permalink / raw)
To: Rich Felker; +Cc: musl
On 03/05/16 19:25, Rich Felker wrote:
> On Sat, Mar 05, 2016 at 07:14:34PM -0500, Pedro Giffuni wrote:
>>
>>
>> On 03/05/16 18:32, Rich Felker wrote:
>>> On Sat, Mar 05, 2016 at 05:41:25PM -0500, Pedro Giffuni wrote:
>>>> First of all, great to hear there is interest on the musl side too.
>>>>
>>>> I think the biggest precedent of porting linux-oriented C libraries
>>>> came from Debian's kFreeBSD. We accomodated a little by for them
>>>> by defining __FreeBSD_kernel__ in sys/param.h.
>>>>
>>>> While using the optional linux-abi futex in FreeBSD could be an option,
>>>> it is not really the cleanest option. The Debian guys did a port of
>>>> NPTL using regular pthreads:
>>>>
>>
>> Of course I ahould have meant "based on regular FreeBSD kernel services".
>>
>>>> http://thread.gmane.org/gmane.linux.debian.ports.bsd/11702
>>>>
>>>> I am certain this will require more research but it would be useful
>>>> for other ports as well.
>>>
>>
>> We could ask Petr Salinger for the details, but I am pretty sure
>> FreeBSD has the required functionality natively.
>>
>>> Glibc/NPTL has a lot of what I'd call "gratuitous abstraction" (like
>>> the lll stuff) in their pthread primitives which makes this
>>> "possible". I call it gratuitous because it's really really hard to
>>> achieve correct implementations of the pthread sync primitives that
>>> don't have serious corner-case bugs, and it's unlikely that their
>>> abstractions actually suffice to make correct alternate
>>> implementations.
>>>
>>> musl does not have any such abstraction. We require a compare-and-swap
>>> operation or equivalent on which arbitrary atomic operations can be
>>> constructed, a futex or equivalent operation that's roughly
>>> while(*addr==expected) sleep(), and implement all the sync primitives
>>> just once on top of these.
>>>
>>
>> I am not a threading expert (or even a CS guy), but it sounds like
>> mutex(9) with condvar(9) would do [1]:
>
> No, they don't satisfy the needs of musl; they have their own
> additional storage requirements and are probably not AS-safe. It might
> be possible to use them to implement a userspace-emulated futex queue
> (only if they are AS-safe), but I don't see a way to extend that to
> the process-shared case.
>
OK, it looks like sema(9) may be nearer (and also simpler but slower).
For the process-shared case libthr(2) uses the stuff in sys/utmx.h,
shich should be looked at but it is not documented[1].
Luckily Ed, the developer that would be mentoring the project, knows
this stuff better than I do.
Pedro.
[1] http://fxr.watson.org/fxr/source/sys/umtx.h?v=FREEBSD8
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: FreeBSD's Google Summer of Code 2016
2016-03-06 0:59 ` Pedro Giffuni
@ 2016-03-06 1:08 ` Rich Felker
0 siblings, 0 replies; 14+ messages in thread
From: Rich Felker @ 2016-03-06 1:08 UTC (permalink / raw)
To: Pedro Giffuni; +Cc: musl
On Sat, Mar 05, 2016 at 07:59:35PM -0500, Pedro Giffuni wrote:
>
>
> On 03/05/16 19:25, Rich Felker wrote:
> >On Sat, Mar 05, 2016 at 07:14:34PM -0500, Pedro Giffuni wrote:
> >>
> >>
> >>On 03/05/16 18:32, Rich Felker wrote:
> >>>On Sat, Mar 05, 2016 at 05:41:25PM -0500, Pedro Giffuni wrote:
> >>>>First of all, great to hear there is interest on the musl side too.
> >>>>
> >>>>I think the biggest precedent of porting linux-oriented C libraries
> >>>>came from Debian's kFreeBSD. We accomodated a little by for them
> >>>>by defining __FreeBSD_kernel__ in sys/param.h.
> >>>>
> >>>>While using the optional linux-abi futex in FreeBSD could be an option,
> >>>>it is not really the cleanest option. The Debian guys did a port of
> >>>>NPTL using regular pthreads:
> >>>>
> >>
> >>Of course I ahould have meant "based on regular FreeBSD kernel services".
> >>
> >>>>http://thread.gmane.org/gmane.linux.debian.ports.bsd/11702
> >>>>
> >>>>I am certain this will require more research but it would be useful
> >>>>for other ports as well.
> >>>
> >>
> >>We could ask Petr Salinger for the details, but I am pretty sure
> >>FreeBSD has the required functionality natively.
> >>
> >>>Glibc/NPTL has a lot of what I'd call "gratuitous abstraction" (like
> >>>the lll stuff) in their pthread primitives which makes this
> >>>"possible". I call it gratuitous because it's really really hard to
> >>>achieve correct implementations of the pthread sync primitives that
> >>>don't have serious corner-case bugs, and it's unlikely that their
> >>>abstractions actually suffice to make correct alternate
> >>>implementations.
> >>>
> >>>musl does not have any such abstraction. We require a compare-and-swap
> >>>operation or equivalent on which arbitrary atomic operations can be
> >>>constructed, a futex or equivalent operation that's roughly
> >>>while(*addr==expected) sleep(), and implement all the sync primitives
> >>>just once on top of these.
> >>>
> >>
> >>I am not a threading expert (or even a CS guy), but it sounds like
> >>mutex(9) with condvar(9) would do [1]:
> >
> >No, they don't satisfy the needs of musl; they have their own
> >additional storage requirements and are probably not AS-safe. It might
> >be possible to use them to implement a userspace-emulated futex queue
> >(only if they are AS-safe), but I don't see a way to extend that to
> >the process-shared case.
> >
>
> OK, it looks like sema(9) may be nearer (and also simpler but slower).
> For the process-shared case libthr(2) uses the stuff in sys/utmx.h,
> shich should be looked at but it is not documented[1].
>
> Luckily Ed, the developer that would be mentoring the project, knows
> this stuff better than I do.
OK, well let's see if there are any proposals and what ideas people
come up with.
Rich
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: FreeBSD's Google Summer of Code 2016
2016-03-06 0:25 ` Rich Felker
2016-03-06 0:31 ` Rich Felker
2016-03-06 0:59 ` Pedro Giffuni
@ 2016-03-07 8:32 ` Felix Fietkau
2016-03-07 13:06 ` Szabolcs Nagy
2016-03-07 13:10 ` Pedro Giffuni
2 siblings, 2 replies; 14+ messages in thread
From: Felix Fietkau @ 2016-03-07 8:32 UTC (permalink / raw)
To: musl, Pedro Giffuni
On 2016-03-06 01:25, Rich Felker wrote:
> On Sat, Mar 05, 2016 at 07:14:34PM -0500, Pedro Giffuni wrote:
>>
>>
>> On 03/05/16 18:32, Rich Felker wrote:
>> >On Sat, Mar 05, 2016 at 05:41:25PM -0500, Pedro Giffuni wrote:
>> >>First of all, great to hear there is interest on the musl side too.
>> >>
>> >>I think the biggest precedent of porting linux-oriented C libraries
>> >>came from Debian's kFreeBSD. We accomodated a little by for them
>> >>by defining __FreeBSD_kernel__ in sys/param.h.
>> >>
>> >>While using the optional linux-abi futex in FreeBSD could be an option,
>> >>it is not really the cleanest option. The Debian guys did a port of
>> >>NPTL using regular pthreads:
>> >>
>>
>> Of course I ahould have meant "based on regular FreeBSD kernel services".
>>
>> >>http://thread.gmane.org/gmane.linux.debian.ports.bsd/11702
>> >>
>> >>I am certain this will require more research but it would be useful
>> >>for other ports as well.
>> >
>>
>> We could ask Petr Salinger for the details, but I am pretty sure
>> FreeBSD has the required functionality natively.
>>
>> >Glibc/NPTL has a lot of what I'd call "gratuitous abstraction" (like
>> >the lll stuff) in their pthread primitives which makes this
>> >"possible". I call it gratuitous because it's really really hard to
>> >achieve correct implementations of the pthread sync primitives that
>> >don't have serious corner-case bugs, and it's unlikely that their
>> >abstractions actually suffice to make correct alternate
>> >implementations.
>> >
>> >musl does not have any such abstraction. We require a compare-and-swap
>> >operation or equivalent on which arbitrary atomic operations can be
>> >constructed, a futex or equivalent operation that's roughly
>> >while(*addr==expected) sleep(), and implement all the sync primitives
>> >just once on top of these.
>> >
>>
>> I am not a threading expert (or even a CS guy), but it sounds like
>> mutex(9) with condvar(9) would do [1]:
>
> No, they don't satisfy the needs of musl; they have their own
> additional storage requirements and are probably not AS-safe. It might
> be possible to use them to implement a userspace-emulated futex queue
> (only if they are AS-safe), but I don't see a way to extend that to
> the process-shared case.
What about umtx? It's supposed to be just like linux futex.
- Felix
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: FreeBSD's Google Summer of Code 2016
2016-03-07 8:32 ` Felix Fietkau
@ 2016-03-07 13:06 ` Szabolcs Nagy
2016-03-07 14:22 ` Pedro Giffuni
2016-03-07 13:10 ` Pedro Giffuni
1 sibling, 1 reply; 14+ messages in thread
From: Szabolcs Nagy @ 2016-03-07 13:06 UTC (permalink / raw)
To: musl; +Cc: Pedro Giffuni
* Felix Fietkau <nbd@openwrt.org> [2016-03-07 09:32:17 +0100]:
> What about umtx? It's supposed to be just like linux futex.
>
i see
https://svnweb.FreeBSD.org/base/head/sys/sys/umtx.h?view=markup
but there seems to be no documentation for it in
https://www.freebsd.org/cgi/man.cgi
if _umtx_op does what it seems to do, then i think musl can be
ported to native freebsd syscalls, but i don't know how much
syscall abi stability freebsd is willing to guarantee.
i think historically that was assumed to be a private interface
between the libc and the kernel on bsd (in which case a musl port
cannot be easily maintained outside of the bsd source repo).
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: FreeBSD's Google Summer of Code 2016
2016-03-07 13:06 ` Szabolcs Nagy
@ 2016-03-07 14:22 ` Pedro Giffuni
0 siblings, 0 replies; 14+ messages in thread
From: Pedro Giffuni @ 2016-03-07 14:22 UTC (permalink / raw)
To: musl
Hello;
On 03/07/16 08:06, Szabolcs Nagy wrote:
> * Felix Fietkau <nbd@openwrt.org> [2016-03-07 09:32:17 +0100]:
>> What about umtx? It's supposed to be just like linux futex.
>>
>
> i see
> https://svnweb.FreeBSD.org/base/head/sys/sys/umtx.h?view=markup
>
> but there seems to be no documentation for it in
> https://www.freebsd.org/cgi/man.cgi
>
> if _umtx_op does what it seems to do, then i think musl can be
> ported to native freebsd syscalls, but i don't know how much
> syscall abi stability freebsd is willing to guarantee.
> i think historically that was assumed to be a private interface
> between the libc and the kernel on bsd (in which case a musl port
> cannot be easily maintained outside of the bsd source repo).
>
I wouldn't worry about ABI stability. We are not allowed to break
the ABI per branch and with so many users depending on being able
to run previous versions of FreeBSD within jails, and with the
native threads and now linux binaries depending on it, there is
no chance we can break it,
We may introduce a new interface in the future (who knows) but we
can't just drop an old one.
Pedro.
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: FreeBSD's Google Summer of Code 2016
2016-03-07 8:32 ` Felix Fietkau
2016-03-07 13:06 ` Szabolcs Nagy
@ 2016-03-07 13:10 ` Pedro Giffuni
1 sibling, 0 replies; 14+ messages in thread
From: Pedro Giffuni @ 2016-03-07 13:10 UTC (permalink / raw)
To: Felix Fietkau, musl
On 03/07/16 03:32, Felix Fietkau wrote:
> On 2016-03-06 01:25, Rich Felker wrote:
>> On Sat, Mar 05, 2016 at 07:14:34PM -0500, Pedro Giffuni wrote:
... <snip>
>>> Of course I ahould have meant "based on regular FreeBSD kernel services".
>>>
>>>>> http://thread.gmane.org/gmane.linux.debian.ports.bsd/11702
>>>>>
>>>>> I am certain this will require more research but it would be useful
>>>>> for other ports as well.
>>>>
>>>
>>> We could ask Petr Salinger for the details, but I am pretty sure
>>> FreeBSD has the required functionality natively.
>>>
>>>> Glibc/NPTL has a lot of what I'd call "gratuitous abstraction" (like
>>>> the lll stuff) in their pthread primitives which makes this
>>>> "possible". I call it gratuitous because it's really really hard to
>>>> achieve correct implementations of the pthread sync primitives that
>>>> don't have serious corner-case bugs, and it's unlikely that their
>>>> abstractions actually suffice to make correct alternate
>>>> implementations.
>>>>
>>>> musl does not have any such abstraction. We require a compare-and-swap
>>>> operation or equivalent on which arbitrary atomic operations can be
>>>> constructed, a futex or equivalent operation that's roughly
>>>> while(*addr==expected) sleep(), and implement all the sync primitives
>>>> just once on top of these.
>>>>
>>>
>>> I am not a threading expert (or even a CS guy), but it sounds like
>>> mutex(9) with condvar(9) would do [1]:
>>
>> No, they don't satisfy the needs of musl; they have their own
>> additional storage requirements and are probably not AS-safe. It might
>> be possible to use them to implement a userspace-emulated futex queue
>> (only if they are AS-safe), but I don't see a way to extend that to
>> the process-shared case.
> What about umtx? It's supposed to be just like linux futex.
>
Yes, that's it.
It is not well documented beyond the <sys/umtx.h> header but it
is indeed used to implement the linux abi futex and the native
libthr on FreeBSD.
Pedro.
^ permalink raw reply [flat|nested] 14+ messages in thread