Agenda for next release?

Agenda for next release?

[Rich Felker]

Hi,

I don't have any immediate plan for the next release, but I'd like to
start thinking about what we'd like to go into it. A few things I know
are still pending that should be possible to include are:

- strverscmp
- zoneinfo
- inet_makeaddr
- scanf %m modifier
- getifaddrs
- cpuset/affinity interfaces
- ether.h interfaces

I know there's also interest in exposing sigreturn and setcontext for
libunwind use, but it's difficult to do right and I'm really skeptical
of the whole thing. Their use of sigreturn seems completely incorrect
(and thus probably untested); sigreturn expects a struct sigcontext on
the stack, not a pointer to a sigcontext in the first function
argument slot. I would rather have a conversation with their
developers first to determine if that code is even working/valid, and
defer the corresponding additions to musl until we eventually add all
the ucontext.h interfaces in working form.

Another port (perhaps x32, like originally proposed) is also
definitely an option if anybody's up for doing it. I had considered
doing it myself in this release cycle, but I'm a bit busy with some
other projects for at least a couple weeks so I'd rather not get
bogged down in a big time-eater right away, and instead spend my time
with musl on small isolated feature additions and bug fixes.

Anything else I missed?

Rich

Re: Agenda for next release?

[Brad Conroy]

[-- Attachment #1: Type: text/plain, Size: 3257 bytes --]

Not functionally necessary for the release, but nice for future compatibility reasons, a few paths are hard-coded into the c files rather than defining them in the (non-)"standard" location
Ex.src/network/getaddrinfo.c has "/etc/hosts", "/etc/resolve.conf" and "/etc/services" hard-coded rather than defining them in netdb.h
#define _PATH_HOSTS "/etc/hosts"#define _PATH_NSSWITCH_CONF "/etc/nsswitch.conf"#define _PATH_SERVICES "/etc/services"
also defined in netdb.h for other uses

#define _PATH_HEQUIV "/etc/hosts.equiv"#define _PATH_NETWORKS "/etc/networks"#define _PATH_PROTOCOLS "/etc/protocols"

resolv.h does have #define _PATH_RESCONF "/etc/resolve.conf" however src/network/__dns.c still has the hard-coded path also
I'm sure there are others that grep could find (fstab.h,paths.h,ttyent.h xtables.h,???)
also android has etc in /system, so the actual path is /system/etc/hosts (though /etc is typically a symlink to /system/etc), but other strange layouts exist and it would be a lot easier/cleaner for work arounds to have the ifdefs confined to one location in the header files and use those definitions in the .c files so that port patches only hit a few headers without polluting the .c files with a lot of ifdef nests
I ran into this while building a static wget-like downloader that would use an alternate hosts file to complement a custom local server that locally hosts some problematic CDNs (to work around problems such as: waiting for ajax.googleapis.com, platform.twitter.com etc...) but this is an obvious hack, not the typical use case for a standard library... but then again part of the charm of musl is its "hackability"

--- On Mon, 12/31/12, Rich Felker <dalias@aerifal.cx> wrote:

From: Rich Felker <dalias@aerifal.cx>
Subject: [musl] Agenda for next release?
To: musl@lists.openwall.com
Date: Monday, December 31, 2012, 2:11 PM

Hi,

I don't have any immediate plan for the next release, but I'd like to
start thinking about what we'd like to go into it. A few things I know
are still pending that should be possible to include are:

- strverscmp
- zoneinfo
- inet_makeaddr
- scanf %m modifier
- getifaddrs
- cpuset/affinity interfaces
- ether.h interfaces

I know there's also interest in exposing sigreturn and setcontext for
libunwind use, but it's difficult to do right and I'm really skeptical
of the whole thing. Their use of sigreturn seems completely incorrect
(and thus probably untested); sigreturn expects a struct sigcontext on
the stack, not a pointer to a sigcontext in the first function
argument slot. I would rather have a conversation with their
developers first to determine if that code is even working/valid, and
defer the corresponding additions to musl until we eventually add all
the ucontext.h interfaces in working form.

Another port (perhaps x32, like originally proposed) is also
definitely an option if anybody's up for doing it. I had considered
doing it myself in this release cycle, but I'm a bit busy with some
other projects for at least a couple weeks so I'd rather not get
bogged down in a big time-eater right away, and instead spend my time
with musl on small isolated feature additions and bug fixes.

Anything else I missed?

Rich

[-- Attachment #2: Type: text/html, Size: 5532 bytes --]

Re: Agenda for next release?

[Rich Felker]

On Mon, Dec 31, 2012 at 01:37:39PM -0800, Brad Conroy wrote:
> Not functionally necessary for the release, but nice for future
> compatibility reasons, a few paths are hard-coded into the c files
> rather than defining them in the (non-)"standard" location
> Ex.src/network/getaddrinfo.c has "/etc/hosts", "/etc/resolve.conf"
> and "/etc/services" hard-coded rather than defining them in netdb.h

The following should be read with the understanding that I have been
convinced to change my mind on issues of similar scope in the past.
However, basically the way it is right now is intentional:

Unlike glibc, musl is intended to be useful for static linking, for
making binaries you can just drop onto an arbitrary system and have
them work. As such, I've aimed to eliminate most usages of any
external files in the filesystem; for example, even iconv uses builtin
tables rather than loading character table files. But in some cases
where there's a need for configurability or reading system-specific
information (like the user/group databases or dns servers), some
minimum amount of filesystem policy needs to be encoded into libc (and
thus, with static linking, into the application as well). To my
knowledge, so far all such hard-coded paths correspond to universal
(at least on Linux, and for the most part on ALL historical unices) so
that only intentional gratuitous breakage could break them.

Anyone who gets musl has access to the source, and can of course go
changing all the hard-coded paths to whatever they like. There's
nothing you or I can do to stop that. But there's also no reason to
encourage it or make it easier than it needs to be.

As for paths.h, it's purely there to make certain legacy programs
which expect it to exist easier to build. Many of the macros defined
in it are utter nonsense, or at least not policies I wish to impose on
anybody. Perhaps this should be better-documented.

Speaking of documentation, the actual documentation for musl, which I
hope to have ready as part of the 1.0 release, will document all
assumptions it makes about the layout of the filesystem, including
which functionality depends on the existence of certain
files/devices/mountpoints in the filesystem (for example, shm_open and
sem_open depending on /dev/shm, or fexecve depending on /proc, or
system and popen depending on /bin/sh).

> also android has etc in /system, so the actual path
> is /system/etc/hosts (though /etc is typically a symlink to
> /system/etc), but other strange layouts exist and it would be a lot
> easier/cleaner for work arounds to have the ifdefs confined to one
> location in the header files and use those definitions in the .c
> files so that port patches only hit a few headers without polluting
> the .c files with a lot of ifdef nests

The problem with making your binaries for alternate layouts like this
is that then they won't work on any system that lacks the alternate
layout. Now, let N people come up with their own N brilliant ideas of
how the FS layout should be, and every system needs N different
symlinks and fake directories all over the place in order to be able
to run arbitrary binaries... This approach does not scale.

On the other hand, if you want to use an alternate layout locally with
symlinks in the standard locations, then your nonstandard layout just
needs 1 set of symlinks (not N sets), and systems with standard
layouts don't need any hackery at all.

> I ran into this while building a static wget-like downloader that
> would use an alternate hosts file to complement a custom local
> server that locally hosts some problematic CDNs (to work around
> problems such as: waiting for ajax.googleapis.com,
> platform.twitter.com etc...) but this is an obvious hack, not the
> typical use case for a standard library... but then again part of
> the charm of musl is its "hackability"

Using a custom hosts file or resolv.conf file via environment vars is
actually an interesting idea we could explore at some point, but it
would have to be secured so as not to affect suid programs or programs
with other sorts of elevated privileges.

Rich

Re: Agenda for next release?

[Rob Landley]

On 12/31/2012 01:11:31 PM, Rich Felker wrote:
> Another port (perhaps x32, like originally proposed) is also
> definitely an option if anybody's up for doing it. I had considered
> doing it myself in this release cycle, but I'm a bit busy with some
> other projects for at least a couple weeks so I'd rather not get
> bogged down in a big time-eater right away, and instead spend my time
> with musl on small isolated feature additions and bug fixes.
> 
> Anything else I missed?

 From my point of view sh4, sparc32 support are interesting because  
switching aboriginal to musl would either have to drop support for  
those two platforms or support two libcs in parallel. (m68k and s390x  
are of secondary interest becauase those are the platforms I've been  
trying to add. In theory mips64 is currently supported but in practice  
it's deeply flaky at the kernel and possibly qemu level; not widely  
used.)

Not priorities, just thought I'd chip in...

Rob

Re: Agenda for next release?

[Rich Felker]

On Mon, Dec 31, 2012 at 10:06:12PM -0600, Rob Landley wrote:
> On 12/31/2012 01:11:31 PM, Rich Felker wrote:
> >Another port (perhaps x32, like originally proposed) is also
> >definitely an option if anybody's up for doing it. I had considered
> >doing it myself in this release cycle, but I'm a bit busy with some
> >other projects for at least a couple weeks so I'd rather not get
> >bogged down in a big time-eater right away, and instead spend my time
> >with musl on small isolated feature additions and bug fixes.
> >
> >Anything else I missed?
> 
> From my point of view sh4, sparc32 support are interesting because
> switching aboriginal to musl would either have to drop support for
> those two platforms or support two libcs in parallel. (m68k and
> s390x are of secondary interest becauase those are the platforms
> I've been trying to add. In theory mips64 is currently supported but
> in practice it's deeply flaky at the kernel and possibly qemu level;
> not widely used.)
> 
> Not priorities, just thought I'd chip in...

Thanks! And good to see you again! I'd be interested in seeing these
ports too, but like I said above now is not the best time for me to
start doing a new port myself. So unless somebody else wants to try a
shot at it, these will probably have to wait a bit.

Rich

Re: Agenda for next release?

[Rob Landley]

On 12/31/2012 10:51:07 PM, Rich Felker wrote:
> > Not priorities, just thought I'd chip in...
> 
> Thanks! And good to see you again! I'd be interested in seeing these
> ports too, but like I said above now is not the best time for me to
> start doing a new port myself. So unless somebody else wants to try a
> shot at it, these will probably have to wait a bit.

Yeah, I know. Part of the reason I was pestering you for port docs is  
that I might do one myself if it's not too much work, although there  
are still 15 other things competing for my attention.

Speaking of which, today is the last day to submit a presentation  
proposal for CELF:

http://events.linuxfoundation.org/events/embedded-linux-conference/cfp

(Which used to be a good conference. I haven't been since the Linux  
Foundation annexed it, but in theory they sometimes provide travel  
expenses and there's still a checkbox to apply. Tim Bird's at least  
still involved, if not running the thing...)

I submitted a couple off the top of my head and may submit more. No  
idea if they'll take 'em, in your case "writing a new Linux C library  
from scratch" would be a pretty AWESOME panel for the audience...

Rob