musl 0.9.9 released

musl 0.9.9 released

[Rich Felker]

Hi all,

I'm pleased to announce the release of musl 0.9.9. In some ways it's
less of a "big release" than I wanted while in other ways it's more.
The biggest aspect of this release which isn't reflected in the blurb
below is the record number of commits by persons other than myself,
particularly nsz and John Spencer. I didn't realize how much was going
to be new in this release until I sat down to write the change log and
looked back at how much they'd done. Thanks!

I know we also have some pending patches from other contributors that
didn't make it into this release, and some work of my own that's not
ready to commit yet. I'm looking forward to taking care of these after
the release, now that we have a bit more freedom to make bigger
changes without worrying about breaking an impending release.

With that said, musl 0.9.9 features:

    Major compatibility improvements, including Linux-specific
    features, GNU extension functions, grsec kernel quirks
    compatibility, and workarounds for subtle but dangerous
    portability errors in some programs. Size, performance, and
    correctness improvements to the math library. Numerous low-impact
    bugs fixed.

    http://www.musl-libc.org/releases/musl-0.9.9.tar.gz

There's a bunch of post-release agenda spread across several other
threads on the mailing list which I won't try to summarize at the
moment; let's just say there's lots of good stuff left to be done on
musl in the next few release cycles leading towards a 1.0...

Cheers,

Rich

Re: musl 0.9.9 released

[Isaac Dunham]

On Fri, 1 Feb 2013 02:10:53 -0500
Rich Felker <dalias@aerifal.cx> wrote:

> Hi all,
> 
> I'm pleased to announce the release of musl 0.9.9. In some ways it's
> less of a "big release" than I wanted while in other ways it's more.
> The biggest aspect of this release which isn't reflected in the blurb
> below is the record number of commits by persons other than myself,
> particularly nsz and John Spencer. I didn't realize how much was going
> to be new in this release until I sat down to write the change log and
> looked back at how much they'd done. Thanks!
> 

Glad to hear this. As usual, I've mirrored it at 
github.com/idunham/musl

On a semi-related topic: my development repository is currently broken due to 5 corrupted objects (I think I have backups, though), so I tried salvaging it and ended up going through all the working branches making diffs.
The patches implement these changes:

ABI: (much of this due to GregorR's patch)
Add stub for logwtmp, sigsetmask
Add getpt
Some aliases, including __xpg_strerror_r and e(uid)access
Add GNU error()
Add some BSD/GNU old math functions

Functionality:
Add vm86/vm86old
Add fgetgrent (for heirloom-pkgtools) (patch same as the last time I sent it)
Fix strverscmp (patch same as the last time I sent it)
WIP:
getifaddrs (I don't think this version will be useful, it's not netlink/IPv6 compatible)
res_querydomain (not tested)-for lshw

I'm expecting that much of the ABI stuff will not be desired, though it would be nice if the part for glibc-linked programs using correct C99/current POSIX functionality got merged.
It should probably move to #ifdef SHARED, though.
The vm86 stuff is something I'd like to see, for the sake of dosemu and some hardware utilities-this is something we'd need in order to replace klibc.
fgetgrent is low priority and small, but I'll be carrying it if it doesn't get merged. It will not go in a separate library, since it pokes at libc's internals.
I could switch some of the ABI stuff to a glibc compatability LD_PRELOAD library (libglibc.so.6 ?).

Rich, what would you like to see?

> I know we also have some pending patches from other contributors that
> didn't make it into this release, and some work of my own that's not
> ready to commit yet. I'm looking forward to taking care of these after
> the release, now that we have a bit more freedom to make bigger
> changes without worrying about breaking an impending release.
> 
> With that said, musl 0.9.9 features:
> 
>     Major compatibility improvements, including Linux-specific
>     features, GNU extension functions, grsec kernel quirks
>     compatibility, and workarounds for subtle but dangerous
>     portability errors in some programs. Size, performance, and
>     correctness improvements to the math library. Numerous low-impact
>     bugs fixed.
> 
>     http://www.musl-libc.org/releases/musl-0.9.9.tar.gz
> 
> There's a bunch of post-release agenda spread across several other
> threads on the mailing list which I won't try to summarize at the
> moment; let's just say there's lots of good stuff left to be done on
> musl in the next few release cycles leading towards a 1.0...

-- 
Isaac Dunham <idunham@lavabit.com>

Re: musl 0.9.9 released

[Rich Felker]

On Fri, Feb 01, 2013 at 06:23:40PM -0800, Isaac Dunham wrote:
> On Fri, 1 Feb 2013 02:10:53 -0500
> Rich Felker <dalias@aerifal.cx> wrote:
> 
> > Hi all,
> > 
> > I'm pleased to announce the release of musl 0.9.9. In some ways it's
> > less of a "big release" than I wanted while in other ways it's more.
> > The biggest aspect of this release which isn't reflected in the blurb
> > below is the record number of commits by persons other than myself,
> > particularly nsz and John Spencer. I didn't realize how much was going
> > to be new in this release until I sat down to write the change log and
> > looked back at how much they'd done. Thanks!
> > 
> 
> Glad to hear this. As usual, I've mirrored it at 
> github.com/idunham/musl
> 
> On a semi-related topic: my development repository is currently
> broken due to 5 corrupted objects (I think I have backups, though),
> so I tried salvaging it and ended up going through all the working
> branches making diffs.

Sorry to hear that -- hope restoring it goes well.

> The patches implement these changes:
> 
> ABI: (much of this due to GregorR's patch)
> Add stub for logwtmp, sigsetmask
> Add getpt
> Some aliases, including __xpg_strerror_r and e(uid)access
> Add GNU error()

If error() is added, I don't think this should be considered ABI only.
It's a documented public interface in glibc and might as well be
visible in musl too if it's implemented...

> Add some BSD/GNU old math functions

Like..?

> Functionality:
> Add vm86/vm86old

Indeed, these should be added.

> Add fgetgrent (for heirloom-pkgtools) (patch same as the last time I sent it)

I'll look at it again.

> Fix strverscmp (patch same as the last time I sent it)

I'm not sure whether we got it into a fully-working state or not; the
conversation kinda died out last time. I'll review it again too. I
remember it didn't look quite like the algorithm I described/proposed,
but that doesn't mean it's wrong. It looked like it could at least use
some streamlining though.

> WIP:
> getifaddrs (I don't think this version will be useful, it's not netlink/IPv6 compatible)

Actually it might. I discovered the simplest way to add IPv6 support
is parsing /proc/net/if_inet6; it should be easy to tack that on to
your implementation. Easier than the netlink hell.

BTW I discovered this via "strace ifconfig" on busybox. :-)

> res_querydomain (not tested)-for lshw

Oh yeah, I need to look at that one again too. If I remember right,
the trouble I had reviewing it was that the documentation was utter
crap and didn't specify how it's supposed to behave except very
loosely, so it was hard for me to tell if it's handling the corner
cases the way it should.

> I'm expecting that much of the ABI stuff will not be desired, though
> it would be nice if the part for glibc-linked programs using correct
> C99/current POSIX functionality got merged.

Yes, I agree.

> It should probably move to #ifdef SHARED, though.

I'm a bit undecided on this. If it would also help linking .a
libraries compiled against glibc headers, it might be nice to have
these ugly aliases in the static libc.a too. On the other hand, static
libs built against glibc are not all that common.. :-)

> The vm86 stuff is something I'd like to see, for the sake of dosemu
> and some hardware utilities-this is something we'd need in order to
> replace klibc.

Agreed.

> fgetgrent is low priority and small, but I'll be carrying it if it
> doesn't get merged. It will not go in a separate library, since it
> pokes at libc's internals.

OK, this can probably be merged. I just need to look at it again..

> I could switch some of the ABI stuff to a glibc compatability
> LD_PRELOAD library (libglibc.so.6 ?).

I think the ugliness in the dynamic linker of detecting the need and
loading it is going to be worse than just putting the mess in libc.so.

> Rich, what would you like to see?

Keeping up this discussion, and nagging me again if too much time goes
by without progress on the above issues. If you'd like most of them
(except possibly the ABI mess) to be in the agenda for 0.9.10, I think
that would be a very reasonable goal.

Rich

strverscmp

[Isaac Dunham]

[-- Attachment #1: Type: text/plain, Size: 676 bytes --]

On Fri, 1 Feb 2013 21:38:33 -0500
Rich Felker <dalias@aerifal.cx> wrote:

> > Fix strverscmp (patch same as the last time I sent it)
> 
> I'm not sure whether we got it into a fully-working state or not; the
> conversation kinda died out last time. I'll review it again too. I
> remember it didn't look quite like the algorithm I described/proposed,
> but that doesn't mean it's wrong. It looked like it could at least use
> some streamlining though.

The last review was just before I got it working. Here's the final version.

Probably somewhere it could be optimized, though...
As long as it doesn't end up looking like the GNU one.

-- 
Isaac Dunham <idunham@lavabit.com>

[-- Attachment #2: strverscmp.diff --]
[-- Type: text/x-diff, Size: 960 bytes --]

diff --git a/src/string/strverscmp.c b/src/string/strverscmp.c
index 7054967..8f3f11f 100644
--- a/src/string/strverscmp.c
+++ b/src/string/strverscmp.c
@@ -1,7 +1,41 @@
+#define _GNU_SOURCE
+#include <ctype.h>
 #include <string.h>
 
 int strverscmp(const char *l, const char *r)
 {
-	/* FIXME */
-	return strcmp(l, r);
+	int haszero=1;
+	while (*l && *r && l[0]==r[0]){
+		if (l[0]=='0'){
+			if (haszero==1) {
+				haszero=0;
+			}
+		} else if (isdigit(l[0])) {
+			if (haszero==1) {
+				haszero=2;
+			}
+		} else {
+			haszero=1;
+		}
+		l++; r++;
+	}
+	if (haszero==1 && (l[0]=='0' || r[0]=='0')) {
+		haszero=0;
+	}
+	if ((isdigit(l[0]) && isdigit(r[0]) ) && haszero) {
+		int lenl=0, lenr=0, firstl=l[0], firstr=r[0];
+		while (isdigit(l++[0]) ) {
+			lenl++;
+		}
+		while (isdigit(r++[0]) ) {
+			lenr++;
+		}
+		if (lenl==lenr) {
+			return (firstl -  firstr);
+		} else {
+			return (lenl - lenr);
+		}
+	} else {
+		return (l[0] -  r[0]);
+	}
 }

Re: strverscmp

[Rich Felker]

Hey, sorry it's taken me so long to get to reading and reviewing this
well. Here are some comments, mostly stylistic, but one's about a
corner-case bug. See below. With that said, the algorithm looks right
to me, and I think we can commit soon.

Rich


On Fri, Feb 01, 2013 at 08:04:52PM -0800, Isaac Dunham wrote:
> On Fri, 1 Feb 2013 21:38:33 -0500
> Rich Felker <dalias@aerifal.cx> wrote:
> 
> > > Fix strverscmp (patch same as the last time I sent it)
> > 
> > I'm not sure whether we got it into a fully-working state or not; the
> > conversation kinda died out last time. I'll review it again too. I
> > remember it didn't look quite like the algorithm I described/proposed,
> > but that doesn't mean it's wrong. It looked like it could at least use
> > some streamlining though.
> 
> The last review was just before I got it working. Here's the final version.
> 
> Probably somewhere it could be optimized, though...
> As long as it doesn't end up looking like the GNU one.
> 
> -- 
> Isaac Dunham <idunham@lavabit.com>

> diff --git a/src/string/strverscmp.c b/src/string/strverscmp.c
> index 7054967..8f3f11f 100644
> --- a/src/string/strverscmp.c
> +++ b/src/string/strverscmp.c
> @@ -1,7 +1,41 @@
> +#define _GNU_SOURCE
> +#include <ctype.h>
>  #include <string.h>
>  
>  int strverscmp(const char *l, const char *r)
>  {
> -	/* FIXME */
> -	return strcmp(l, r);
> +	int haszero=1;
> +	while (*l && *r && l[0]==r[0]){

I used to do a lot of while loops like this, but now I feel like it
makes it hard to follow the end-of-string logic -- a lot more code
runs after the null terminator is hit, and it's nontrivial to follow
through it all and make sure the right thing happens, as the
subsequent code after the loop is now handling two very-distinct
loop-exit conditions:

1. Loop exited due to mismatch.
2. Loop exited due to hitting end-of-string with no mismatch.

What about instead doing something like this:

	while (*l==*r) {
		if (!*l) return 0;
		/* ... */

That way, the "equal strings" case is trivially correct. It also might
generate better code.

Note that I also made a couple stylistic changes here; unless you
object, I prefer *p to p[0] when p is just being used as a moving
pointer within a string, always accessed with offset 0. It's just more
concise/less cluttered and avoids suggesting it's used as a string in
itself. Also added a space before the open brace, for consistency with
style elsewhere.


> +		if (l[0]=='0'){
> +			if (haszero==1) {
> +				haszero=0;
> +			}
> +		} else if (isdigit(l[0])) {
> +			if (haszero==1) {
> +				haszero=2;
> +			}
> +		} else {
> +			haszero=1;
> +		}
> +		l++; r++;
> +	}
> +	if (haszero==1 && (l[0]=='0' || r[0]=='0')) {
> +		haszero=0;
> +	}
> +	if ((isdigit(l[0]) && isdigit(r[0]) ) && haszero) {
> +		int lenl=0, lenr=0, firstl=l[0], firstr=r[0];

These variables have the wrong types. int is not sufficient to store a
character count in a string. You need size_t.

> +		while (isdigit(l++[0]) ) {
> +			lenl++;
> +		}
> +		while (isdigit(r++[0]) ) {
> +			lenr++;
> +		}
> +		if (lenl==lenr) {
> +			return (firstl -  firstr);
> +		} else {
> +			return (lenl - lenr);
> +		}

I see what's going on here, but I find saving the first mismatching
characters (btw, you could have just saved the difference) then
continuing to advance the l/r pointers a bit confusing at first. Why
not instead do:

		while (isdigit(l[lenl])) lenl++;

Re: strverscmp

[Isaac Dunham]

[-- Attachment #1: Type: text/plain, Size: 2347 bytes --]

On Sun, 17 Feb 2013 13:39:44 -0500
Rich Felker <dalias@aerifal.cx> wrote:

> 
> Hey, sorry it's taken me so long to get to reading and reviewing this
> well. Here are some comments, mostly stylistic, but one's about a
> corner-case bug. See below. With that said, the algorithm looks right
> to me, and I think we can commit soon.
> 
> Rich

All the comments seem like improvements to me.
Attached is a revised version.
 
> > +	while (*l && *r && l[0]==r[0]){
> 
> I used to do a lot of while loops like this, but now I feel like it
> makes it hard to follow the end-of-string logic -- a lot more code
> runs after the null terminator is hit, and it's nontrivial to follow
> through it all and make sure the right thing happens, as the
> subsequent code after the loop is now handling two very-distinct
> loop-exit conditions:
> 
> 1. Loop exited due to mismatch.
> 2. Loop exited due to hitting end-of-string with no mismatch.
> 
> What about instead doing something like this:
> 
> 	while (*l==*r) {
> 		if (!*l) return 0;
> 		/* ... */
> 
> That way, the "equal strings" case is trivially correct. It also might
> generate better code.

OK.

> Note that I also made a couple stylistic changes here; unless you
> object, I prefer *p to p[0] when p is just being used as a moving
> pointer within a string, always accessed with offset 0. It's just more
> concise/less cluttered and avoids suggesting it's used as a string in
> itself. Also added a space before the open brace, for consistency with
> style elsewhere.
<snip>
> > +	if ((isdigit(l[0]) && isdigit(r[0]) ) && haszero) {
> > +		int lenl=0, lenr=0, firstl=l[0], firstr=r[0];
> 
> These variables have the wrong types. int is not sufficient to store a
> character count in a string. You need size_t.
> 
> > +		while (isdigit(l++[0]) ) {
> > +			lenl++;
> > +		}
> > +		while (isdigit(r++[0]) ) {
> > +			lenr++;
> > +		}
> > +		if (lenl==lenr) {
> > +			return (firstl -  firstr);
> > +		} else {
> > +			return (lenl - lenr);
> > +		}
> 
> I see what's going on here, but I find saving the first mismatching
> characters (btw, you could have just saved the difference) then
> continuing to advance the l/r pointers a bit confusing at first. Why
> not instead do:
> 
> 		while (isdigit(l[lenl])) lenl++;

That would be clearer and more efficient.

-- 
Isaac Dunham <idunham@lavabit.com>

[-- Attachment #2: strverscmp.diff --]
[-- Type: text/x-diff, Size: 927 bytes --]

diff --git a/src/string/strverscmp.c b/src/string/strverscmp.c
index 7054967..33a42ee 100644
--- a/src/string/strverscmp.c
+++ b/src/string/strverscmp.c
@@ -1,7 +1,40 @@
+#define _GNU_SOURCE
+#include <ctype.h>
 #include <string.h>
+#include <sys/types.h>
 
 int strverscmp(const char *l, const char *r)
 {
-	/* FIXME */
-	return strcmp(l, r);
+	int haszero=1;
+	while (*l==*r) {
+		if (!*l) return 0;
+
+		if (*l=='0') {
+			if (haszero==1) {
+				haszero=0;
+			}
+		} else if (isdigit(*l)) {
+			if (haszero==1) {
+				haszero=2;
+			}
+		} else {
+			haszero=1;
+		}
+		l++; r++;
+	}
+	if (haszero==1 && (*l=='0' || *r=='0')) {
+		haszero=0;
+	}
+	if ((isdigit(*l) && isdigit(*r) ) && haszero) {
+		size_t lenl=0, lenr=0;
+		while (isdigit(l[lenl]) ) lenl++;
+		while (isdigit(r[lenr]) ) lenr++;
+		if (lenl==lenr) {
+			return (*l -  *r);
+		} else {
+			return (lenl - lenr);
+		}
+	} else {
+		return (*l -  *r);
+	}
 }

Re: strverscmp

[Rich Felker]

On Mon, Feb 18, 2013 at 05:15:58PM -0800, Isaac Dunham wrote:
> On Sun, 17 Feb 2013 13:39:44 -0500
> Rich Felker <dalias@aerifal.cx> wrote:
> 
> > 
> > Hey, sorry it's taken me so long to get to reading and reviewing this
> > well. Here are some comments, mostly stylistic, but one's about a
> > corner-case bug. See below. With that said, the algorithm looks right
> > to me, and I think we can commit soon.
> > 
> > Rich
> 
> All the comments seem like improvements to me.
> Attached is a revised version.

Committed. If any further improvements need to be made they can be
done as separate commits. Actually I just found one bug which I'll
fix..

Rich

fgetgrent

[Isaac Dunham]

[-- Attachment #1: Type: text/plain, Size: 350 bytes --]

On Fri, 1 Feb 2013 21:38:33 -0500
Rich Felker <dalias@aerifal.cx> wrote:

> > Add fgetgrent (for heirloom-pkgtools) (patch same as the last time I sent it)
> 
> I'll look at it again.

It would be possible to implement getgrent as a wrapper for __fgetgrent, if that shrinks code size for a worthwhile amount.


-- 
Isaac Dunham <idunham@lavabit.com>

[-- Attachment #2: fgetgrent.diff --]
[-- Type: text/x-diff, Size: 958 bytes --]

diff --git a/include/grp.h b/include/grp.h
index 030d7f8..cb40e07 100644
--- a/include/grp.h
+++ b/include/grp.h
@@ -30,6 +30,10 @@ struct group  *getgrent(void);
 void           endgrent(void);
 void           setgrent(void);
 
+#ifdef _GNU_SOURCE
+struct group  *fgetgrent(FILE *stream);
+#endif
+
 #if defined(_GNU_SOURCE) || defined(_BSD_SOURCE)
 int getgrouplist(const char *, gid_t, gid_t *, int *);
 int setgroups(size_t, const gid_t *);
diff --git a/src/passwd/getgrent.c b/src/passwd/getgrent.c
index 429a3e5..d59bcb6 100644
--- a/src/passwd/getgrent.c
+++ b/src/passwd/getgrent.c
@@ -10,6 +10,17 @@ void setgrent()
 
 weak_alias(setgrent, endgrent);
 
+struct group *__fgetgrent(FILE *f)
+{
+	static char *line, **mem;
+	static struct group gr;
+	size_t size=0, nmem=0;
+	if (!f) return 0;
+	return __getgrent_a(f, &gr, &line, &size, &mem, &nmem);
+}
+
+weak_alias(__fgetgrent,fgetgrent);
+
 struct group *getgrent()
 {
 	static char *line, **mem;

Re: fgetgrent

[Rich Felker]

On Fri, Feb 01, 2013 at 09:14:04PM -0800, Isaac Dunham wrote:
> On Fri, 1 Feb 2013 21:38:33 -0500
> Rich Felker <dalias@aerifal.cx> wrote:
> 
> > > Add fgetgrent (for heirloom-pkgtools) (patch same as the last time I sent it)
> > 
> > I'll look at it again.
> 
> It would be possible to implement getgrent as a wrapper for
> __fgetgrent, if that shrinks code size for a worthwhile amount.

The bigger advantage would be that it avoids adding more crap in bss.
The flip-side is that fgetgrent would clobber the results of getgrent,
and vice versa. I'm not sure whether this is reasonable.

Rich

Re: fgetgrent

[Rich Felker]

On Fri, Feb 01, 2013 at 09:14:04PM -0800, Isaac Dunham wrote:
> On Fri, 1 Feb 2013 21:38:33 -0500
> Rich Felker <dalias@aerifal.cx> wrote:
> 
> > > Add fgetgrent (for heirloom-pkgtools) (patch same as the last
> > > time I sent it)
> > 
> > I'll look at it again.
> 
> It would be possible to implement getgrent as a wrapper for
> __fgetgrent, if that shrinks code size for a worthwhile amount.

Committed, with minor changes.
I didn't make the wrapper changes we talked about since I was slightly
worried it might break programs that expect getgrent and fgetgrent not
to clobber each other; I have no idea if such programs exist. I did
move it to a separate file though to prevent pulling in more text/bss
than necessary in static programs.

Rich