Pending issues for next release

Pending issues for next release

[Rich Felker]

Here's a quick list of open issues that we might want to take care of
before another release:

- ARM-optimized memcpy, etc.
- put*ent functions in pwd/shadow/etc.
- program_invocation_name
- Posssible misaligned access bugs in mbsrtowcs (?)
- [u]intmax_t underlying type mismatches ABI on 64-bit systems (?)
- dn_skipname
- switching libc.so/ld-musl.so symlink direction

And a few things that have been on the agenda but neglected for a long
time, which may stay that way unless anybody's excited about moving
them forward quickly:

- zoneinfo
- inet_makeaddr
- scanf %m modifier
- getifaddrs
- cpuset/affinity
- ether.h functions

We're well overdue for a release, so I'm hoping we can get one out
sometime this week. I'm going to be pretty busy next week so if not
this week it'll probably be two weeks out.

Rich

Re: Pending issues for next release

[Szabolcs Nagy]

* Rich Felker <dalias@aerifal.cx> [2013-04-01 19:31:44 -0400]:
> Here's a quick list of open issues that we might want to take care of
> before another release:
> 
> - ARM-optimized memcpy, etc.
> - put*ent functions in pwd/shadow/etc.
> - program_invocation_name
> - Posssible misaligned access bugs in mbsrtowcs (?)
> - [u]intmax_t underlying type mismatches ABI on 64-bit systems (?)

and fix [u]intmax_t and wchar_t limits in stdint.h etc. (U,L,LL suffix)
and fix printf fmt in inttypes.h (otherwise gcc will warn)

> - dn_skipname
> - switching libc.so/ld-musl.so symlink direction
> 

and fix the download page
(and fix install link in the faq)

> And a few things that have been on the agenda but neglected for a long
> time, which may stay that way unless anybody's excited about moving
> them forward quickly:
> 
> - zoneinfo
> - inet_makeaddr
> - scanf %m modifier
> - getifaddrs
> - cpuset/affinity
> - ether.h functions

other pending things from my todo:
arpa/tftp.h (inetutils http://www.openwall.com/lists/musl/2012/11/26/2 )
fgetspent, putpwent (pam_unix http://www.openwall.com/lists/musl/2013/02/27/4 )
inb, outb,.. (xorg, grub, lmsensors,..?)
mkostemp (better rand and stack address hiding)
struct ntptimeval (glibc vs musl abi http://www.openwall.com/lists/musl/2013/03/04/3 )
random_r (used by mesa?)

> We're well overdue for a release, so I'm hoping we can get one out
> sometime this week. I'm going to be pretty busy next week so if not
> this week it'll probably be two weeks out.
> 
> Rich

Re: Pending issues for next release

[Rich Felker]

On Tue, Apr 02, 2013 at 01:11:16PM +0200, Szabolcs Nagy wrote:
> * Rich Felker <dalias@aerifal.cx> [2013-04-01 19:31:44 -0400]:
> > Here's a quick list of open issues that we might want to take care of
> > before another release:
> > 
> > - ARM-optimized memcpy, etc.
> > - put*ent functions in pwd/shadow/etc.
> > - program_invocation_name
> > - Posssible misaligned access bugs in mbsrtowcs (?)
> > - [u]intmax_t underlying type mismatches ABI on 64-bit systems (?)
> 
> and fix [u]intmax_t and wchar_t limits in stdint.h etc. (U,L,LL suffix)

For [u]intmax_t, assuming all archs use the lowest-rank type that can
hold the value, wouldn't it suffice to just omit the suffix entirely?
Then the compiler would give it the right type. Or will this break in
c89 mode or something obscure like that?

> and fix printf fmt in inttypes.h (otherwise gcc will warn)

Yes..

> other pending things from my todo:
> arpa/tftp.h (inetutils http://www.openwall.com/lists/musl/2012/11/26/2 )

OK I'll look at this.

> fgetspent, putpwent (pam_unix http://www.openwall.com/lists/musl/2013/02/27/4 )

Just added, but the shadow stuff is stubbed still.

> inb, outb,.. (xorg, grub, lmsensors,..?)

These have been present on x86 for a long time. Are there other archs
that need versions of them?

> mkostemp (better rand and stack address hiding)

Any specific recommendations? As temp filename generation is not
really security-critical (at worst you can slow down or cause
tempfile-creation errors in a target program), I think we should
probably avoid using the stack address at all. Leaking the stack
address to an attacker is too great a risk.

> struct ntptimeval (glibc vs musl abi http://www.openwall.com/lists/musl/2013/03/04/3 )

Uhg this issue again.

> random_r (used by mesa?)

Bleh, the API is pretty ugly. Thoughts on what we should do? Do they
actually care about having a thread-local PRNG state, or are they only
using it on the misguided idea that random might not be thread-safe?

Rich

Re: Pending issues for next release

[Rich Felker]

On Thu, Apr 04, 2013 at 07:37:42PM -0400, Rich Felker wrote:
> On Tue, Apr 02, 2013 at 01:11:16PM +0200, Szabolcs Nagy wrote:
> > * Rich Felker <dalias@aerifal.cx> [2013-04-01 19:31:44 -0400]:
> > > Here's a quick list of open issues that we might want to take care of
> > > before another release:
> > > 
> > > - ARM-optimized memcpy, etc.
> > > - put*ent functions in pwd/shadow/etc.
> > > - program_invocation_name
> > > - Posssible misaligned access bugs in mbsrtowcs (?)
> > > - [u]intmax_t underlying type mismatches ABI on 64-bit systems (?)
> > 
> > and fix [u]intmax_t and wchar_t limits in stdint.h etc. (U,L,LL suffix)
> 
> For [u]intmax_t, assuming all archs use the lowest-rank type that can
> hold the value, wouldn't it suffice to just omit the suffix entirely?
> Then the compiler would give it the right type. Or will this break in
> c89 mode or something obscure like that?

Should be fixed, but I'd appreciate a second review of it.

> > other pending things from my todo:
> > arpa/tftp.h (inetutils http://www.openwall.com/lists/musl/2012/11/26/2 )
> 
> OK I'll look at this.

That email also mentions some extra macros for utmp/wtmp pathname. I
looked at the issue, and utmp.h is actually defining _PATH_UTMP and
_PATH_WTMP in ways that conflict with paths.h... We should address
this. Any ideas how? I'd like to keep the /dev/null definitions, but
I'm a little bit scared some broken program might see them and end up
unlinking /dev/null and replacing it with an empty utmp file at
startup.

> > mkostemp (better rand and stack address hiding)
> 
> Any specific recommendations? As temp filename generation is not
> really security-critical (at worst you can slow down or cause
> tempfile-creation errors in a target program), I think we should
> probably avoid using the stack address at all. Leaking the stack
> address to an attacker is too great a risk.

A single call to clock_gettime lacks sufficient entropy to fill the
30-bit space we have. Should we just accept that? I don't see a lot of
value to adding any PRNG since the attack vector would simply be
predicting the time clock_gettime will return (within a range) then
creating filenames to clash with every possible output for that range
of inputs. The PRNG does not increase the size of the filename space
that must be filled. One option would be calling clock_gettime twice,
but the second call will have significantly less entropy than the
first since the interval of time between them is somewhat predictable.

Rich

Re: Pending issues for next release

[Isaac Dunham]

On Thu, 4 Apr 2013 19:37:42 -0400
Rich Felker <dalias@aerifal.cx> wrote:

> > random_r (used by mesa?)
> 
> Bleh, the API is pretty ugly. Thoughts on what we should do? Do they
> actually care about having a thread-local PRNG state, or are they only
> using it on the misguided idea that random might not be thread-safe?
The only place it happens is in src/glx/glxhash.c, which says this:
#ifndef __GLIBC__
#define HASH_RANDOM_DECL	char *ps, rs[256]
#define HASH_RANDOM_INIT(seed)	ps = initstate(seed, rs, sizeof(rs))
#define HASH_RANDOM		random()
#define HASH_RANDOM_DESTROY	setstate(ps)
#else
#define HASH_RANDOM_DECL	struct random_data rd; int32_t rv; char rs[256]
#define HASH_RANDOM_INIT(seed)					\
   do {								\
      (void) memset(&rd, 0, sizeof(rd));			\
      (void) initstate_r(seed, rs, sizeof(rs), &rd);		\
   } while(0)
#define HASH_RANDOM             ((void) random_r(&rd, &rv), rv)
#define HASH_RANDOM_DESTROY
#endif

So mesa won't need it unless built for glibc.

This is the commit that made random_r conditional:
commit d09941c8cc2d4620eb774744c8878921b9dc3bcc
Author: Robert Noland <rnoland@2hip.net>
Date:   Tue Sep 22 11:49:57 2009 -0700

And the previous commit that made it an issue:

commit 9666529b5a5be1fcde82caadc2fe2efa5ea81e49
Author: Ian Romanick <ian.d.romanick@intel.com>
Date:   Wed Sep 16 16:43:50 2009 -0700

    glx: Use initstate_r / random_r instead of corrupting global random number state
    
    Previously srandom and random were used.  This cause the global random
    number generator state to be modified.  This caused problems for
    applications that called srandom before calling into GLX.  By using
    local state the global state is left unmodified.
    
    This should fix bug #23774.

So that's the rationale: nothing to do with threads, just leave global state alone.
-- 
Isaac Dunham <idunham@lavabit.com>

Re: Pending issues for next release

[Jens Gustedt]

[-- Attachment #1: Type: text/plain, Size: 1269 bytes --]

Hi,

is there a schedule for C11 thread interfaces? I have a shallow
wrapping of C11 threads via pthread in P99, but because of the
different calling conventions for the user's start function (return of
int instead of void* ) this is sort of suboptimal (read crude).

Having scanned through the musl sources, I don't think that supporting
C11 threads would be a big deal. Some additional wrappers should
suffice, I think.

I have seen Rich's mail to glibc asking for the size of some data
structures, but I have the impression that this has stalled and that
there is no reply to be expected soon. I understand that musl wants to
be ABI compatible, but I don't see anything that can't be fixed
immediately, once they ship a version of C11 threads. I would expect
any sane implementation that supports pthread and C11 thread
simultaneously not to change the ABI for the native C interface just
for the fun of it.

Jens

-- 
:: INRIA Nancy Grand Est :: http://www.loria.fr/~gustedt/   ::
:: AlGorille ::::::::::::::: office Nancy : +33 383593090   ::
:: ICube :::::::::::::: office Strasbourg : +33 368854536   ::
:: ::::::::::::::::::::::::::: gsm France : +33 651400183   ::
:: :::::::::::::::::::: gsm international : +49 15737185122 ::



[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Pending issues for next release

[Rich Felker]

On Tue, Apr 02, 2013 at 01:42:54PM +0200, Jens Gustedt wrote:
> Hi,
> 
> is there a schedule for C11 thread interfaces? I have a shallow
> wrapping of C11 threads via pthread in P99, but because of the
> different calling conventions for the user's start function (return of
> int instead of void* ) this is sort of suboptimal (read crude).

The easiest way to hack it on is to use a fixed start-function and
malloc a block for the C11 thread identifier that contains the
caller's start function and argument and space for the return value,
then pass that to the start-function. But I agree this is ugly and
bloated.

> Having scanned through the musl sources, I don't think that supporting
> C11 threads would be a big deal. Some additional wrappers should
> suffice, I think.

The main difficulty is respecting the namespace. Some functions would
need to be renamed to be in the reserved namespace with aliases for
their POSIX names.

> I have seen Rich's mail to glibc asking for the size of some data
> structures, but I have the impression that this has stalled and that
> there is no reply to be expected soon.

Yes...

> I understand that musl wants to
> be ABI compatible, but I don't see anything that can't be fixed
> immediately, once they ship a version of C11 threads. I would expect

This should probably be discussed in a bit more detail. Obviously
there's the C++ ABI issue (underlying struct tags) but we haven't
addressed that at all anyway. The ABI issues in terms of struct size
do not actually affect the app-to-libc ABI at all; they just affect
the ABI between object files compiled using musl's headers.

> any sane implementation that supports pthread and C11 thread
> simultaneously not to change the ABI for the native C interface just
> for the fun of it.

I'm not sure what you mean by "change". The ABI isn't defined yet.
There's clearly a choice to be made between making the C11 objects
lighter, since they don't have to provide as much functionality, and
making them just duplicates of the POSIX objects (so that code can be
shared for both), and it's not clear which is better.

Even if the objects are shared, we may choose not to share code if it
allows the C11 synchronization primitives to have better performance
than the POSIX ones.

Rich

Re: Pending issues for next release

[Jens Gustedt]

[-- Attachment #1: Type: text/plain, Size: 4795 bytes --]

Hello,

Am Dienstag, den 02.04.2013, 14:09 -0400 schrieb Rich Felker:
> On Tue, Apr 02, 2013 at 01:42:54PM +0200, Jens Gustedt wrote:
> > is there a schedule for C11 thread interfaces? I have a shallow
> > wrapping of C11 threads via pthread in P99, but because of the
> > different calling conventions for the user's start function (return of
> > int instead of void* ) this is sort of suboptimal (read crude).
> 
> The easiest way to hack it on is to use a fixed start-function and
> malloc a block for the C11 thread identifier that contains the
> caller's start function and argument and space for the return value,
> then pass that to the start-function. But I agree this is ugly and
> bloated.

Basically this is what I am doing, but then there is this little
detail of pre-existing threads, in particular the one for main, that
makes that even uglier.

> > Having scanned through the musl sources, I don't think that supporting
> > C11 threads would be a big deal. Some additional wrappers should
> > suffice, I think.
> 
> The main difficulty is respecting the namespace. Some functions would
> need to be renamed to be in the reserved namespace with aliases for
> their POSIX names.

I we just talk of the 'thrd_' functions, first, I think that just
aliases wouldn't do the trick. They have different calling conventions
and interpretation of the return values (thrd_yield e.g), so a
decision would have to be made where to split things. My first idea
would be to have the shared code in static functions and have a
wrapper for each interface specification.

> > I have seen Rich's mail to glibc asking for the size of some data
> > structures, but I have the impression that this has stalled and that
> > there is no reply to be expected soon.
> 
> Yes...
> 
> > I understand that musl wants to
> > be ABI compatible, but I don't see anything that can't be fixed
> > immediately, once they ship a version of C11 threads. I would expect
> 
> This should probably be discussed in a bit more detail. Obviously
> there's the C++ ABI issue (underlying struct tags) but we haven't
> addressed that at all anyway. The ABI issues in terms of struct size
> do not actually affect the app-to-libc ABI at all; they just affect
> the ABI between object files compiled using musl's headers.
> 
> > any sane implementation that supports pthread and C11 thread
> > simultaneously not to change the ABI for the native C interface just
> > for the fun of it.
> 
> I'm not sure what you mean by "change".

By change I meant to use a different implementation for mtx_t than
pthread_mutex_t, e.g.

> The ABI isn't defined yet.
> There's clearly a choice to be made between making the C11 objects
> lighter, since they don't have to provide as much functionality, and
> making them just duplicates of the POSIX objects (so that code can be
> shared for both), and it's not clear which is better.

I was thinking that it would clearly be preferable to re-use the POSIX
types for the control structures (mtx_t and cnd_t) as much as
possible. If it'd just be for the fact that that code is already well
tested. But as you pose the challenge, you are right that there could
be advantages in re-implementing parts.

In particular a mtx_t being just an int and using futexes directly on
top has certainly some charm. But if you'd want to support
mtx_recursive things might get more complicated.

> Even if the objects are shared, we may choose not to share code if it
> allows the C11 synchronization primitives to have better performance
> than the POSIX ones.

performance figures that clearly distinguish two such implementations
would probably be quite difficult to obtain.

Both implementations for control structures should basically do some
instructions and one or two atomics for the fast path. For the slow
path, this then would mainly be calls to futex. So the atomics should
dominate the fast path, futex the slow path.

For thread creation we could even have the effect that C11 is
slower. C11 can't be created detached, so any thread creation for
"detached" threads has first to create a thread state somewhere that
then would be freed when detaching.

In any case we should then (shared objects, different primitives) come
up with a linker scheme that inhibits a successful linking of an
application that tries to use POSIX an C11 interfaces simultaneously.

Jens

-- 
:: INRIA Nancy Grand Est :: http://www.loria.fr/~gustedt/   ::
:: AlGorille ::::::::::::::: office Nancy : +33 383593090   ::
:: ICube :::::::::::::: office Strasbourg : +33 368854536   ::
:: ::::::::::::::::::::::::::: gsm France : +33 651400183   ::
:: :::::::::::::::::::: gsm international : +49 15737185122 ::






[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Pending issues for next release

[Rich Felker]

On Tue, Apr 02, 2013 at 09:40:47PM +0200, Jens Gustedt wrote:
> > > Having scanned through the musl sources, I don't think that supporting
> > > C11 threads would be a big deal. Some additional wrappers should
> > > suffice, I think.
> > 
> > The main difficulty is respecting the namespace. Some functions would
> > need to be renamed to be in the reserved namespace with aliases for
> > their POSIX names.
> 
> I we just talk of the 'thrd_' functions, first, I think that just
> aliases wouldn't do the trick. They have different calling conventions
> and interpretation of the return values (thrd_yield e.g), so a
> decision would have to be made where to split things. My first idea
> would be to have the shared code in static functions and have a
> wrapper for each interface specification.

I didn't mean to make the C11 functions aliases for the POSIX
functions. I meant the POSIX functions would be renamed to have names
like __pthread_mutex_lock, with pthread_mutex_lock as a weak alias,
and mtx_lock calling __pthread_mutex_lock.

> > The ABI isn't defined yet.
> > There's clearly a choice to be made between making the C11 objects
> > lighter, since they don't have to provide as much functionality, and
> > making them just duplicates of the POSIX objects (so that code can be
> > shared for both), and it's not clear which is better.
> 
> I was thinking that it would clearly be preferable to re-use the POSIX
> types for the control structures (mtx_t and cnd_t) as much as
> possible. If it'd just be for the fact that that code is already well
> tested. But as you pose the challenge, you are right that there could
> be advantages in re-implementing parts.
> 
> In particular a mtx_t being just an int and using futexes directly on
> top has certainly some charm. But if you'd want to support
> mtx_recursive things might get more complicated.

Single int is not enough for an efficient implementation; you need a
second one as a waiter-count, and a third one as a reference count if
you want to do recursive locks. This is still somewhat smaller than
the pthread_mutex_t we have now (6 ints on 32-bit systems and
considerably larger on 64-bit ones if I remember right). The size
doesn't matter so much when you're locking code, but if you're locking
data, the size of the mutex inside the structure it's protecting can
be a big deal if you have lots of them.

> > Even if the objects are shared, we may choose not to share code if it
> > allows the C11 synchronization primitives to have better performance
> > than the POSIX ones.
> 
> performance figures that clearly distinguish two such implementations
> would probably be quite difficult to obtain.
> 
> Both implementations for control structures should basically do some
> instructions and one or two atomics for the fast path. For the slow
> path, this then would mainly be calls to futex. So the atomics should
> dominate the fast path, futex the slow path.

The issue is that for POSIX mutexes, the fast path is much longer due
to having to support lots of mutex types and having different
conditional paths for all them. I've optimized it as much as possible
for regular mutexes, but I think the runtime in the uncontended case
is still about 50% longer than the ideal regular-mutex-only
implementation.

> For thread creation we could even have the effect that C11 is
> slower. C11 can't be created detached, so any thread creation for
> "detached" threads has first to create a thread state somewhere that
> then would be freed when detaching.

I don't follow...

> In any case we should then (shared objects, different primitives) come
> up with a linker scheme that inhibits a successful linking of an
> application that tries to use POSIX an C11 interfaces simultaneously.

Why would you want use of both to fail? The next issue of POSIX will
almost surely be aligned with C11, in which case, I would expect it to
include all the C11 thread interfaces, at least optionally, and
certainly not to forbid their use in programs that use the POSIX
thread interfaces. Even if not, I see no reason why using one set of
interfaces should preclude using the other. It's certainly reasonable
for a program to use one library written in pure C using C11 threads,
and another library written to POSIX using POSIX threads.

Rich

Re: Pending issues for next release

[Isaac Dunham]

[-- Attachment #1: Type: text/plain, Size: 400 bytes --]

On Mon, 1 Apr 2013 19:31:44 -0400
Rich Felker <dalias@aerifal.cx> wrote:

> Here's a quick list of open issues that we might want to take care of
> before another release:
>
(snipping several interesting improvements) 
> - switching libc.so/ld-musl.so symlink direction

Does this patch look suitable?
I know it does work, including in the DESTDIR=./.../ case.
-- 
Isaac Dunham <idunham@lavabit.com>

[-- Attachment #2: buildsys.diff --]
[-- Type: text/x-diff, Size: 871 bytes --]

diff --git a/Makefile b/Makefile
index 997c5bb..478e698 100644
--- a/Makefile
+++ b/Makefile
@@ -118,7 +118,10 @@ tools/musl-gcc: config.mak
 $(DESTDIR)$(bindir)/%: tools/%
 	install -D $< $@
 
-$(DESTDIR)$(libdir)/%.so: lib/%.so
+$(DESTDIR)$(libdir)/libc.so: $(DESTDIR)$(LDSO_PATHNAME)
+	ln -sf $(LDSO_PATHNAME) $@
+
+$(DESTDIR)$(LDSO_PATHNAME): lib/libc.so
 	install -D -m 755 $< $@
 
 $(DESTDIR)$(libdir)/%: lib/%
@@ -127,12 +130,6 @@ $(DESTDIR)$(libdir)/%: lib/%
 $(DESTDIR)$(includedir)/%: include/%
 	install -D -m 644 $< $@
 
-$(DESTDIR)$(LDSO_PATHNAME): $(DESTDIR)$(syslibdir)
-	ln -sf $(libdir)/libc.so $@ || true
-
-$(DESTDIR)$(syslibdir):
-	install -d -m 755 $(DESTDIR)$(syslibdir)
-
 install-libs: $(ALL_LIBS:lib/%=$(DESTDIR)$(libdir)/%) $(if $(SHARED_LIBS),$(DESTDIR)$(LDSO_PATHNAME),)
 
 install-headers: $(ALL_INCLUDES:include/%=$(DESTDIR)$(includedir)/%)

Re: Pending issues for next release

[Rich Felker]

On Wed, Apr 03, 2013 at 09:00:00PM -0700, Isaac Dunham wrote:
> On Mon, 1 Apr 2013 19:31:44 -0400
> Rich Felker <dalias@aerifal.cx> wrote:
> 
> > Here's a quick list of open issues that we might want to take care of
> > before another release:
> >
> (snipping several interesting improvements) 
> > - switching libc.so/ld-musl.so symlink direction
> 
> Does this patch look suitable?
> I know it does work, including in the DESTDIR=./.../ case.

I can't say it's wrong, but seeing this patch reminds me why I didn't
do it before...

> diff --git a/Makefile b/Makefile
> index 997c5bb..478e698 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -118,7 +118,10 @@ tools/musl-gcc: config.mak
>  $(DESTDIR)$(bindir)/%: tools/%
>  	install -D $< $@
>  
> -$(DESTDIR)$(libdir)/%.so: lib/%.so
> +$(DESTDIR)$(libdir)/libc.so: $(DESTDIR)$(LDSO_PATHNAME)
> +	ln -sf $(LDSO_PATHNAME) $@
> +
> +$(DESTDIR)$(LDSO_PATHNAME): lib/libc.so
>  	install -D -m 755 $< $@
>  
>  $(DESTDIR)$(libdir)/%: lib/%
> @@ -127,12 +130,6 @@ $(DESTDIR)$(libdir)/%: lib/%
>  $(DESTDIR)$(includedir)/%: include/%
>  	install -D -m 644 $< $@
>  
> -$(DESTDIR)$(LDSO_PATHNAME): $(DESTDIR)$(syslibdir)
> -	ln -sf $(libdir)/libc.so $@ || true
> -
> -$(DESTDIR)$(syslibdir):
> -	install -d -m 755 $(DESTDIR)$(syslibdir)
> -

Unfortunately, this makes it impossible to run "make install" without
a DESTDIR unless you're root. Before, the ||true in the dynamic-linker
installation allowed you to "make install" without root as long as you
have permissions to the prefix, and the resulting install could be
used for everything except _running_ dynamic-linked programs. (Also,
if root added a single symlink for you, it would be fully functional.)

I don't want to reject the patch, because I think this change is
mostly correct, but I would like to discuss and see if there's any way
we can get back the old level of functionality while making this
change. In particular, users should be able to install musl with
--prefix=$HOME/musl and build binaries which will run on any system
with the musl loader in /lib.

Rich

Re: Pending issues for next release

[Rich Felker]

On Thu, Apr 04, 2013 at 07:04:04PM -0400, Rich Felker wrote:
> On Wed, Apr 03, 2013 at 09:00:00PM -0700, Isaac Dunham wrote:
> > On Mon, 1 Apr 2013 19:31:44 -0400
> > Rich Felker <dalias@aerifal.cx> wrote:
> > 
> > > Here's a quick list of open issues that we might want to take care of
> > > before another release:
> > >
> > (snipping several interesting improvements) 
> > > - switching libc.so/ld-musl.so symlink direction
> > 
> > Does this patch look suitable?
> > I know it does work, including in the DESTDIR=./.../ case.
> 
> I can't say it's wrong, but seeing this patch reminds me why I didn't
> do it before...
> 
> > diff --git a/Makefile b/Makefile
> > index 997c5bb..478e698 100644
> > --- a/Makefile
> > +++ b/Makefile
> > @@ -118,7 +118,10 @@ tools/musl-gcc: config.mak
> >  $(DESTDIR)$(bindir)/%: tools/%
> >  	install -D $< $@
> >  
> > -$(DESTDIR)$(libdir)/%.so: lib/%.so
> > +$(DESTDIR)$(libdir)/libc.so: $(DESTDIR)$(LDSO_PATHNAME)
> > +	ln -sf $(LDSO_PATHNAME) $@
> > +
> > +$(DESTDIR)$(LDSO_PATHNAME): lib/libc.so
> >  	install -D -m 755 $< $@
> >  
> >  $(DESTDIR)$(libdir)/%: lib/%
> > @@ -127,12 +130,6 @@ $(DESTDIR)$(libdir)/%: lib/%
> >  $(DESTDIR)$(includedir)/%: include/%
> >  	install -D -m 644 $< $@
> >  
> > -$(DESTDIR)$(LDSO_PATHNAME): $(DESTDIR)$(syslibdir)
> > -	ln -sf $(libdir)/libc.so $@ || true
> > -
> > -$(DESTDIR)$(syslibdir):
> > -	install -d -m 755 $(DESTDIR)$(syslibdir)
> > -
> 
> Unfortunately, this makes it impossible to run "make install" without
> a DESTDIR unless you're root. Before, the ||true in the dynamic-linker
> installation allowed you to "make install" without root as long as you
> have permissions to the prefix, and the resulting install could be
> used for everything except _running_ dynamic-linked programs. (Also,
> if root added a single symlink for you, it would be fully functional.)
> 
> I don't want to reject the patch, because I think this change is
> mostly correct, but I would like to discuss and see if there's any way
> we can get back the old level of functionality while making this
> change. In particular, users should be able to install musl with
> --prefix=$HOME/musl and build binaries which will run on any system
> with the musl loader in /lib.

Do you have any good ideas for what we should do here? Some folks on
IRC have also been asking about the motivations for the change. The
ones I recall are:

1. (not a valid reason) Debian policy. As long as a Debian package for
musl is making its own lib dir (e.g. /usr/musl/lib or whatever) I
think it's exempt from policy on library versioning, just like firefox
or something that has package-local .so files.

2. Versioning. It's impossible to have multiple versions installed
simultaneously. However, I don't see how the proposed change would
help. /lib/ld-musl-$(ARCH).so.1 is not versioned either. The ".1" is
not a version that will change (unless we had a major new ABI) but
rather a silly convention. So I don't think this is a valid reason for
the change anyway, though it could be a valid reason for _alternate_
changes.

3. Performance. Having the PT_INTERP be a symbolic link adds a
symbolic link dereference in each exec() syscall for a dynamic linked
program. In theory, avoiding that should shave some time off exec().

4. Allowing $(libdir) on non-root partition.

The alternate change I came up with that would address issue 2, but
not 3 or 4, is making the real file be $(libdir)/musl-$(VERSION).so or
similar, and having both libc.so and ld-musl be symlinks to it.
However this also has the negative aspect of requiring the symlink in
$(syslibdir) to change with each upgrade, which is frustrating if
you're trying to allow a single user to upgrade musl without write
permission to /lib.

If there are no good solutions right now, I think it may be best to
leave this issue alone for the time being. But if a change is going to
be made, I want it to happen as far away from 1.0 as possible, so
please start thinking about good ideas for how this should work! :)

Rich

Re: Pending issues for next release

[Rich Felker]

On Mon, Apr 01, 2013 at 07:31:44PM -0400, Rich Felker wrote:
> Here's a quick list of open issues that we might want to take care of
> before another release:
> 
> - ARM-optimized memcpy, etc.
> - put*ent functions in pwd/shadow/etc.
> - program_invocation_name
> - Posssible misaligned access bugs in mbsrtowcs (?)
> - [u]intmax_t underlying type mismatches ABI on 64-bit systems (?)
> - dn_skipname
> - switching libc.so/ld-musl.so symlink direction

Of these, the bug in mbsrtowcs has been fixed (and several related
issues addressed), and the passwd/group/shadow stuff was just added.
The rest remain open. I'll see if I can finish up one or two more...

Rich

Update (Re: [musl] Pending issues for next release)

[Rich Felker]

On Mon, Apr 01, 2013 at 07:31:44PM -0400, Rich Felker wrote:
> Here's a quick list of open issues that we might want to take care of
> before another release:
> 
> - ARM-optimized memcpy, etc.
> - put*ent functions in pwd/shadow/etc.
> - program_invocation_name
> - Posssible misaligned access bugs in mbsrtowcs (?)
> - [u]intmax_t underlying type mismatches ABI on 64-bit systems (?)
> - dn_skipname
> - switching libc.so/ld-musl.so symlink direction

All of these except the ARM asm and symlink direction are finished. I
still want to do the symlink direction, but I'm looking for a clean
solution to the problem that it breaks build-only installs where the
user does not have permissions to install the dynamic loader.

But basically, this means the goals for release have largely been met.

> And a few things that have been on the agenda but neglected for a long
> time, which may stay that way unless anybody's excited about moving
> them forward quickly:
> 
> - zoneinfo
> - inet_makeaddr
> - scanf %m modifier
> - getifaddrs
> - cpuset/affinity
> - ether.h functions

Of these, getifaddrs is mostly done and presently in the repository.
Haven't seen any action on the rest. As mentioned in the recent scanf
thread, I discovered some conformance issues in scanf related to
implementation of the %m modifier, and these will need to wait until
after the next release. Haven't seen any interest in addressing the
others, so they'll need to wait too, I think.

Rich