What would make musl 1.2?

What would make musl 1.2?

[Rich Felker]

We're far enough along in the 1.1.x series now that I'd like to start
thinking about what milestones might justify calling a release 1.2.0.

Looking at the Open Issues and Roadmap on the wiki, the big things
musl could gain in the near future look to be:

- Finishing up all the loose ends on locale and multilingual support:
  IDN, message translations, iconv improvements, collation, and
  possibly the byte-based C locale.

- Hardening/security features.

- C++11 non-POD TLS.

- Alternate user/group db backends (hopefully in upcoming 1.1.7).

Any or all of these could become part of the wishlist for 1.2.

Aside from those big functionality areas though, I think archs/porting
might be one of the most important things to think about. Supporting
aarch64 is definitely important in the near future, and it could be a
big publicity boost. So could getting coverage for the remaining archs
uClibc has that musl doesn't, or at least the ones of modern interest.

Other ideas?

Rich

Re: What would make musl 1.2?

[Raphael Cohn]

[-- Attachment #1: Type: text/plain, Size: 1410 bytes --]

Is there any possibility of adding in the ucontext.h functions? I know
they're deprecated, but they're still widely used - particularly by go for
goroutines, IIRC. I realise there are problems correctly implementing the
*_r variants, but a well-written, modern and efficient implementation would
be a big win.

On 13 February 2015 at 13:16, Rich Felker <dalias@libc.org> wrote:

> We're far enough along in the 1.1.x series now that I'd like to start
> thinking about what milestones might justify calling a release 1.2.0.
>
> Looking at the Open Issues and Roadmap on the wiki, the big things
> musl could gain in the near future look to be:
>
> - Finishing up all the loose ends on locale and multilingual support:
>   IDN, message translations, iconv improvements, collation, and
>   possibly the byte-based C locale.
>
> - Hardening/security features.
>
> - C++11 non-POD TLS.
>
> - Alternate user/group db backends (hopefully in upcoming 1.1.7).
>
> Any or all of these could become part of the wishlist for 1.2.
>
> Aside from those big functionality areas though, I think archs/porting
> might be one of the most important things to think about. Supporting
> aarch64 is definitely important in the near future, and it could be a
> big publicity boost. So could getting coverage for the remaining archs
> uClibc has that musl doesn't, or at least the ones of modern interest.
>
> Other ideas?
>
> Rich
>

[-- Attachment #2: Type: text/html, Size: 1902 bytes --]

Re: What would make musl 1.2?

[Anthony J. Bentley]

Raphael Cohn writes:
> Is there any possibility of adding in the ucontext.h functions? I know
> they're deprecated, but they're still widely used - particularly by go for
> goroutines, IIRC.

It's worth mentioning that OpenBSD doesn't have ucontext, so given the
size of its package repository (which also contains Go), ucontext can't
be *that* widely used.

-- 
Anthony J. Bentley

Re: What would make musl 1.2?

[Raphael Cohn]

[-- Attachment #1: Type: text/plain, Size: 1147 bytes --]

Do they use a third party support lib?

If it's possible to support use cases with a third party lib, then I'm less
concerned - provided that that lib also works with musl. Given the nature
of ucontext, that may not be so. A musl native solution would be optimal
for performance - and performance is a common reason for going down this
route. It allows for far greater scale in certain server designs then
either thread-per-connection or a thread-pool can do.

Out of interest, how many packages are in the OpenBSD repository? How does
it compare to Debian's, say? For me, Debian's repo contents is a yardstick
of what Linux + Musl could be expected to work with.

On 13 February 2015 at 15:08, Anthony J. Bentley <anthony@cathet.us> wrote:

> Raphael Cohn writes:
> > Is there any possibility of adding in the ucontext.h functions? I know
> > they're deprecated, but they're still widely used - particularly by go
> for
> > goroutines, IIRC.
>
> It's worth mentioning that OpenBSD doesn't have ucontext, so given the
> size of its package repository (which also contains Go), ucontext can't
> be *that* widely used.
>
> --
> Anthony J. Bentley
>

[-- Attachment #2: Type: text/html, Size: 1698 bytes --]

Re: What would make musl 1.2?

[Raphael Cohn]

[-- Attachment #1: Type: text/plain, Size: 1584 bytes --]

Actually, answering my own question - a quick back of the packet
calculation suggests about 8,500 packages in OpenBSD, vs 65,000+ for Debian
- although it does depend how you count. And there are a lot more
'unofficial' Debian repos and packages, too... There's no doubt that
OpenBSD includes the most popular.

On 13 February 2015 at 15:20, Raphael Cohn <raphael.cohn@stormmq.com> wrote:

> Do they use a third party support lib?
>
> If it's possible to support use cases with a third party lib, then I'm
> less concerned - provided that that lib also works with musl. Given the
> nature of ucontext, that may not be so. A musl native solution would be
> optimal for performance - and performance is a common reason for going down
> this route. It allows for far greater scale in certain server designs then
> either thread-per-connection or a thread-pool can do.
>
> Out of interest, how many packages are in the OpenBSD repository? How does
> it compare to Debian's, say? For me, Debian's repo contents is a yardstick
> of what Linux + Musl could be expected to work with.
>
>
> On 13 February 2015 at 15:08, Anthony J. Bentley <anthony@cathet.us>
> wrote:
>
>> Raphael Cohn writes:
>> > Is there any possibility of adding in the ucontext.h functions? I know
>> > they're deprecated, but they're still widely used - particularly by go
>> for
>> > goroutines, IIRC.
>>
>> It's worth mentioning that OpenBSD doesn't have ucontext, so given the
>> size of its package repository (which also contains Go), ucontext can't
>> be *that* widely used.
>>
>> --
>> Anthony J. Bentley
>>
>
>

[-- Attachment #2: Type: text/html, Size: 2423 bytes --]

Re: What would make musl 1.2?

[Justin Cormack]

On 13 February 2015 at 09:38, Anthony J. Bentley <anthony@cathet.us> wrote:
> Raphael Cohn writes:
>> Is there any possibility of adding in the ucontext.h functions? I know
>> they're deprecated, but they're still widely used - particularly by go for
>> goroutines, IIRC.
>
> It's worth mentioning that OpenBSD doesn't have ucontext, so given the
> size of its package repository (which also contains Go), ucontext can't
> be *that* widely used.

The plan9-derived Go compilers do not need the ucontext functions, it
is gccgo that does, which is needed for less common architectures but
generally not used much elsewhere as it seems to have compatibility
issues.

I do not think it is used much elsewhere but there are those of us who
use them and there is no nice substitute.

I had been planning to implement it for Musl but I am a bit unsure.
The users of it really do not want any syscalls (because the point of
green threads is speed) but there seemed to be a view that the Posix
version should preserve signal masks, although this was not clear to
me from the spec if it was really required. Most/all implementations
do in fact make syscalls, so it is in fact unclear if they are that
useful, so was coming to the conclusion that a standalone library
would be better. I have not found one, so I may write one, if only so
there is a reference assembly implementation that people can just
reuse.

Musl should remove the prototypes while it is missing the functions
though, and we should decide whether to support them (I think the
previous decision was a weak yes).

Justin


Justin

Re: What would make musl 1.2?

[u-wsnj]

On Fri, Feb 13, 2015 at 10:45:39AM +0000, Justin Cormack wrote:
> > Raphael Cohn writes:
> >> Is there any possibility of adding in the ucontext.h functions? I know

> I do not think it is used much elsewhere but there are those of us who
> use them and there is no nice substitute.

+1

Rune

Re: What would make musl 1.2?

[Justin Cormack]

On 13 February 2015 at 07:46, Rich Felker <dalias@libc.org> wrote:
> Aside from those big functionality areas though, I think archs/porting
> might be one of the most important things to think about. Supporting
> aarch64 is definitely important in the near future, and it could be a
> big publicity boost. So could getting coverage for the remaining archs
> uClibc has that musl doesn't, or at least the ones of modern interest.

Not sure that any of the uClibc archs that Musl doesnt yet support are
that interesting. Here is my take on the interesting ones.

aarch64 would be nice, I think there will be lots of use once they
ship in volume.
mips n32 - this seems to have mostly bypassed linux and android alas,
with only gentoo supporting it, despite the o32 ABI being horrible.
But Musl users should be encouraged to use n32 if building form
scratch...
mips n64 - there is a bit of cheap hardware around (Cavium stuff), but
fairly niche at present, although recent Android supports it and
Imagination seems to be promoting it more. Maybe they could be
persuaded to sponsor mips ports?
ppc64, ppc64le - there has been a spate of ppc64le distros eg Ubuntu,
which is the new elfv2 ppc ABI which is a bit cleaned up and little
endian to make porting easier for dumb people. You can get access to
free hardware from IBM. Maybe they could sponsor a port?
risc-v64 - I know one of the people at lowrisc.org who are going to
ship completely open fpga then silicon designs for risc-v. There are
full qemu and llvm/gcc toolchains already, and NetBSD support, would
be nice to have a Musl distro when it ships (fpga this year, silicon
next).

Justin

Re: What would make musl 1.2?

[Justin Cormack]

Oh I forgot to check the openwrt platform support list which is relevant

https://dev.openwrt.org/wiki/platforms

The missing ones there are avr32, mips64, ppc64.

Justin

Re: What would make musl 1.2?

[Felix Fietkau]

On 2015-02-14 00:02, Justin Cormack wrote:
> Oh I forgot to check the openwrt platform support list which is relevant
> 
> https://dev.openwrt.org/wiki/platforms
> 
> The missing ones there are avr32, mips64, ppc64.
avr32 doesn't really seem to be widely used, It's probably worth
skipping that one. For OpenWrt, mips64 would probably be the most
important one among the missing platforms.

- Felix

Re: What would make musl 1.2?

[Rich Felker]

On Sun, Feb 15, 2015 at 10:23:57AM +1100, Felix Fietkau wrote:
> On 2015-02-14 00:02, Justin Cormack wrote:
> > Oh I forgot to check the openwrt platform support list which is relevant
> > 
> > https://dev.openwrt.org/wiki/platforms
> > 
> > The missing ones there are avr32, mips64, ppc64.
> avr32 doesn't really seem to be widely used, It's probably worth
> skipping that one. For OpenWrt, mips64 would probably be the most
> important one among the missing platforms.

You can always run 32bit code on a mips64 machine though. I can't
imagine any OpenWRT targets having large enough resources that you'd
actually want LP64. IMO mips64 is only interesting for desktop/server
systems.

Rich

Re: What would make musl 1.2?

[Isaac Dunham]

On Sat, Feb 14, 2015 at 09:44:19PM -0500, Rich Felker wrote:
> On Sun, Feb 15, 2015 at 10:23:57AM +1100, Felix Fietkau wrote:
> > On 2015-02-14 00:02, Justin Cormack wrote:
> > > Oh I forgot to check the openwrt platform support list which is relevant
> > > 
> > > https://dev.openwrt.org/wiki/platforms
> > > 
> > > The missing ones there are avr32, mips64, ppc64.
> > avr32 doesn't really seem to be widely used, It's probably worth
> > skipping that one. For OpenWrt, mips64 would probably be the most
> > important one among the missing platforms.
> 
> You can always run 32bit code on a mips64 machine though. I can't
> imagine any OpenWRT targets having large enough resources that you'd
> actually want LP64. IMO mips64 is only interesting for desktop/server
> systems.

I went looking for MIPS64 systems recently and noticed a surprising
number of networking devices with large amounts (8GB or more) of RAM,
barely any flash/rom, and a Linux/Busybox based OS to process network
requests. MIPS64 seems to be mainly used in higher-end networking 
systems (commercial routers/switches)--at least outside China, where
it's used for HPC, some desktops, and just about everything else.

HTH,
Isaac Dunham

Re: What would make musl 1.2?

[Sebastian Gottschall]

Am 15.02.2015 um 00:23 schrieb Felix Fietkau:
> On 2015-02-14 00:02, Justin Cormack wrote:
>> Oh I forgot to check the openwrt platform support list which is relevant
>>
>> https://dev.openwrt.org/wiki/platforms
>>
>> The missing ones there are avr32, mips64, ppc64.
> avr32 doesn't really seem to be widely used, It's probably worth
> skipping that one. For OpenWrt, mips64 would probably be the most
> important one among the missing platforms.
>
> - Felix
>
and for dd-wrt as well :-)

Re: What would make musl 1.2?

[Isaac Dunham]

On Fri, Feb 13, 2015 at 11:09:55AM +0000, Justin Cormack wrote:
> mips n32 - this seems to have mostly bypassed linux and android alas,
> with only gentoo supporting it, despite the o32 ABI being horrible.
> But Musl users should be encouraged to use n32 if building form
> scratch...

I've looked repeatedly, but everything I found has seemed to imply
that n32 is like x32: a "32-bit" ABI that requires 64-bit hardware.
Is this incorrect?

> mips n64 - there is a bit of cheap hardware around (Cavium stuff), but
> fairly niche at present, although recent Android supports it and
> Imagination seems to be promoting it more. Maybe they could be
> persuaded to sponsor mips ports?

They came out with a new MIPS64 processor design recently (no hardware
is available yet), and the latest version of Android features MIPS64
support. IIRC, Alexandru Voica commented that it usually takes a little
under 2 years from design release to hardware on the market.

I'd like to use musl on mips64, though I'm just a hobbyist and I've
not yet found the cheap hardware to which you refer.

Thanks,
Isaac Dunham

Re: What would make musl 1.2?

[Rich Felker]

On Sat, Feb 14, 2015 at 12:06:56AM +0000, Isaac Dunham wrote:
> On Fri, Feb 13, 2015 at 11:09:55AM +0000, Justin Cormack wrote:
> > mips n32 - this seems to have mostly bypassed linux and android alas,
> > with only gentoo supporting it, despite the o32 ABI being horrible.
> > But Musl users should be encouraged to use n32 if building form
> > scratch...
> 
> I've looked repeatedly, but everything I found has seemed to imply
> that n32 is like x32: a "32-bit" ABI that requires 64-bit hardware.
> Is this incorrect?

Yes, that's what it is. Interestingly they avoided some of the huge
mistakes of x32, but I think that was largely a consequence of having
most or all of the kernel n32 types match the plain 32-bit types
(sadly including, if I'm not mistaken, 32-bit time_t) rather than
being a crazy mix. Someone who knows n32 better could probably clarify
this.

If I remember correctly, the Loongson mips64-based laptops use n32,
probably because mips is painfully underpowered for a laptop/desktop
even without doubling the memory bus pressure with 64-bit pointers.

Rich

Re: What would make musl 1.2?

[Justin Cormack]

On 14 February 2015 at 00:06, Isaac Dunham <ibid.ag@gmail.com> wrote:
> On Fri, Feb 13, 2015 at 11:09:55AM +0000, Justin Cormack wrote:
>> mips n32 - this seems to have mostly bypassed linux and android alas,
>> with only gentoo supporting it, despite the o32 ABI being horrible.
>> But Musl users should be encouraged to use n32 if building form
>> scratch...
>
> I've looked repeatedly, but everything I found has seemed to imply
> that n32 is like x32: a "32-bit" ABI that requires 64-bit hardware.
> Is this incorrect?

That is correct but as mips has been 64 bit since the 1990s a lot of
low end hardware has actually been 64 bit even in configurations that
are fairly low end. Also there are some floating point limitations in
o32. I guess as hardware specs rise that n64 support is probably more
useful than n32 for musl.

There is a rant about how terrible they all are here
http://www.linux-mips.org/wiki/WhatsWrongWithO32N32N64 suggesting
replacing the whole lot... the all code is PIC thing is certainly very
odd. Have found toolchain quality low too, but now at least
Imagination has people working on llvm and gcc etc.

>> mips n64 - there is a bit of cheap hardware around (Cavium stuff), but
>> fairly niche at present, although recent Android supports it and
>> Imagination seems to be promoting it more. Maybe they could be
>> persuaded to sponsor mips ports?
>
> They came out with a new MIPS64 processor design recently (no hardware
> is available yet), and the latest version of Android features MIPS64
> support. IIRC, Alexandru Voica commented that it usually takes a little
> under 2 years from design release to hardware on the market.
>
> I'd like to use musl on mips64, though I'm just a hobbyist and I've
> not yet found the cheap hardware to which you refer.

The cheap hardware easily available is:
Edgerouter lite http://www.ubnt.com/edgemax/edgerouter-lite/
(note the full ethernet performance requires binary blobs, and the USB
is fussy about what storage it will accept, so it is a bit annoying).
they make higher end hardware too.

All the Chinese mips chips (Loongson) are 64 bit. I dont recommend the
old 2F models (really odd instruction set) but the Loongson 3 ones
sound better. Apparently you can get them as systems/boards from China
for not too much and the newer ones are decent specs. I havent yet
tried this yet but may do. See the loongson-dev list for details eg
https://groups.google.com/forum/#!searchin/loongson-dev/loongson-3a$20mainboard/loongson-dev/twzd5TBz_1k/fZnJs5Xf_6QJ

I am hoping that now Imagination have realised that devs need hardware
there will be a 64 bit dev board in the next year or so, especially as
there was huge demand for the 32 bit one
http://blog.imgtec.com/powervr/mips-creator-ci20-development-board-now-available
- mine just arrived a couple of days ago.

Justin

Re: What would make musl 1.2?

[Isaac Dunham]

On Sat, Feb 14, 2015 at 10:23:53AM +0000, Justin Cormack wrote:
> On 14 February 2015 at 00:06, Isaac Dunham <ibid.ag@gmail.com> wrote:
> > On Fri, Feb 13, 2015 at 11:09:55AM +0000, Justin Cormack wrote:
> >> mips n64 - there is a bit of cheap hardware around (Cavium stuff), but
> >> fairly niche at present, although recent Android supports it and
> >> Imagination seems to be promoting it more. Maybe they could be
> >> persuaded to sponsor mips ports?
> >
> > They came out with a new MIPS64 processor design recently (no hardware
> > is available yet), and the latest version of Android features MIPS64
> > support. IIRC, Alexandru Voica commented that it usually takes a little
> > under 2 years from design release to hardware on the market.
> >
> > I'd like to use musl on mips64, though I'm just a hobbyist and I've
> > not yet found the cheap hardware to which you refer.
> 
> The cheap hardware easily available is:
> Edgerouter lite http://www.ubnt.com/edgemax/edgerouter-lite/
> (note the full ethernet performance requires binary blobs, and the USB
> is fussy about what storage it will accept, so it is a bit annoying).
> they make higher end hardware too.

Thanks! I suppose that it's probably not really viable to hook up a
powered USB hub and some...other hardware.

> All the Chinese mips chips (Loongson) are 64 bit. I dont recommend the
> old 2F models (really odd instruction set) but the Loongson 3 ones
> sound better. Apparently you can get them as systems/boards from China
> for not too much and the newer ones are decent specs. I havent yet
> tried this yet but may do. See the loongson-dev list for details eg
> https://groups.google.com/forum/#!searchin/loongson-dev/loongson-3a$20mainboard/loongson-dev/twzd5TBz_1k/fZnJs5Xf_6QJ

Ah yes.
Loongson 2F is the one that had some bug where one instruction could cause
random segfaults (use -mfix-loongson-2f-nop to avoid it - I suppose this
should be in the flags for mips64el).
Even the new ones are far from competing with a new high-end PC, but they look
to be in a similar league with my Thinkpad X100e in performance
(though some have far more cores).
They actually use the same southbridge and integrated graphics.


> I am hoping that now Imagination have realised that devs need hardware
> there will be a 64 bit dev board in the next year or so, especially as
> there was huge demand for the 32 bit one
> http://blog.imgtec.com/powervr/mips-creator-ci20-development-board-now-available
> - mine just arrived a couple of days ago.

Neat! There's been a bit of interest expressed in a 64-bit board, but
the comment I referred to seemed to imply it might be a little over a
year to get a Warrior-based 64-bit system out.

Thanks,
Isaac Dunham

Re: What would make musl 1.2?

[Rich Felker]

On Fri, Feb 13, 2015 at 02:46:03AM -0500, Rich Felker wrote:
> We're far enough along in the 1.1.x series now that I'd like to start
> thinking about what milestones might justify calling a release 1.2.0.
> 
> [...]
> 
> Other ideas?

One more idea I'd like to add is possibly overhauling the build system
for out-of-tree builds, shorter/simpler makefile, less duplication
between archs' bits, ...

Rich