Support for musl libc in OpenWrt

Support for musl libc in OpenWrt

[Florian Fainelli]

Hi All,

Support for musl libc in OpenWrt has been added in revision r34314, this 
was tested against:

- X86 in VirtualBox
- X86_64 with UML
- ARM with the realview target

Note that the following issues are known:

- unable to complete libstdc++ build due to some missing declarations (WIP)
- MIPS and MIPSel ports are not functionnal (details below)
- PowerPC has not been tested at all due to lack of hardware/emulation 
target

I did not try yet to compile our entire set of packages besides the one 
pre-selected in trunk, so you will certainly encounter some breakage due 
to musl being more strict about header inclusions and declarations than 
uClibc and (e)glibc.

Have fun!
--
MIPS and MIPSel specific issues:

using qemu's binary emulation (qemu-mips[el]) against a statically 
linked binary works, a dynamic binary makes my host qemu segv. Using 
qemu-system-mips[el], the binary segfaults while being executed by the 
kernel. So far I have not had the time to debug this further.
--
Florian

Re: Support for musl libc in OpenWrt

[Szabolcs Nagy]

[-- Attachment #1: Type: text/plain, Size: 280 bytes --]

* Florian Fainelli <florian@openwrt.org> [2012-11-23 21:10:33 +0100]:
> - MIPS and MIPSel ports are not functionnal (details below)

attached a mips termios.h patch based on
linux/arch/mips/include/asm/termbits.h

i added some of the missing bits, i dont know if those are needed

[-- Attachment #2: mips_termio.diff --]
[-- Type: text/x-diff, Size: 1983 bytes --]

diff --git a/arch/mips/bits/termios.h b/arch/mips/bits/termios.h
index 9f6abd8..179af8e 100644
--- a/arch/mips/bits/termios.h
+++ b/arch/mips/bits/termios.h
@@ -14,19 +14,20 @@ struct termios
 #define VQUIT     1
 #define VERASE    2
 #define VKILL     3
-#define VEOF      4
+#define VMIN      4
 #define VTIME     5
-#define VMIN      6
+#define VEOL2     6
 #define VSWTC     7
+#define VSWTCH    7
 #define VSTART    8
 #define VSTOP     9
 #define VSUSP    10
-#define VEOL     11
 #define VREPRINT 12
 #define VDISCARD 13
 #define VWERASE  14
 #define VLNEXT   15
-#define VEOL2    16
+#define VEOF     16
+#define VEOL     17
 
 #define IGNBRK  0000001
 #define BRKINT  0000002
@@ -42,6 +43,7 @@ struct termios
 #define IXANY   0004000
 #define IXOFF   0010000
 #define IMAXBEL 0020000
+#define IUTF8   0040000
 
 #define OPOST  0000001
 #define OLCUC  0000002
@@ -94,7 +96,10 @@ struct termios
 #define B9600    0000015
 #define B19200   0000016
 #define B38400   0000017
+#define EXTA     0000016
+#define EXTB     0000017
 
+#define BOTHER   0010000
 #define B57600   0010001
 #define B115200  0010002
 #define B230400  0010003
@@ -124,26 +129,32 @@ struct termios
 #define PARODD 0001000
 #define HUPCL  0002000
 #define CLOCAL 0004000
+#define CBAUDEX 0010000
 
+#define CIBAUD   002003600000
+#define CMSPAR   010000000000
 #define CRTSCTS  020000000000
+#define IBSHIFT 16
 
 #define ISIG   0000001
 #define ICANON 0000002
+#define XCASE  0000004
 #define ECHO   0000010
 #define ECHOE  0000020
 #define ECHOK  0000040
 #define ECHONL 0000100
 #define NOFLSH 0000200
-#define TOSTOP 0000400
-#define IEXTEN 0100000
-
-/* Extensions? */
-#define CBAUDEX 0010000
+#define IEXTEN 0000400
 #define ECHOCTL 0001000
 #define ECHOPRT 0002000
 #define ECHOKE 0004000
-#define FLUSHO 0010000
+#define FLUSHO 0020000
 #define PENDIN 0040000
+#define TOSTOP 0100000
+#define ITOSTOP 0100000
+#define EXTPROC 0200000
+
+#define TIOCSER_TEMT 1
 
 #define TCOOFF 0
 #define TCOON  1

Re: Support for musl libc in OpenWrt

[Florian Fainelli]

Le 23/11/2012 23:19, Szabolcs Nagy a écrit :
> * Florian Fainelli <florian@openwrt.org> [2012-11-23 21:10:33 +0100]:
>> - MIPS and MIPSel ports are not functionnal (details below)
>
> attached a mips termios.h patch based on
> linux/arch/mips/include/asm/termbits.h
>
> i added some of the missing bits, i dont know if those are needed
>

As we discussed over IRC, this fixes the weird console behavior that I 
was seeing, thanks!
--
Florian

Re: Support for musl libc in OpenWrt

[Isaac Dunham]

On Fri, 23 Nov 2012 21:10:33 +0100
Florian Fainelli <florian@openwrt.org> wrote:

> Hi All,
> 
> Support for musl libc in OpenWrt has been added in revision r34314, 
Glad to hear it!

> Note that the following issues are known:
> 
> - unable to complete libstdc++ build due to some missing declarations (WIP)
Are you using the default "linux-gnu" libstdc++, or the "generic" target?
Only the latter is known to work at this point.

> - MIPS and MIPSel ports are not functionnal (details below)

> --
> MIPS and MIPSel specific issues:
> 
> using qemu's binary emulation (qemu-mips[el]) against a statically 
> linked binary works, a dynamic binary makes my host qemu segv. Using 
> qemu-system-mips[el], the binary segfaults while being executed by the 
> kernel. So far I have not had the time to debug this further.

There are likely other issues, but are you using a hardfloat-compatible kernel (FPU emulation enabled)?
At present, musl/mips relies on certain instructions that the kernel  ordinarily emulates when they aren't supported.


-- 
Isaac Dunham <idunham@lavabit.com>

Re: Support for musl libc in OpenWrt

[Szabolcs Nagy]

* Isaac Dunham <idunham@lavabit.com> [2012-11-23 22:15:33 -0800]:
> On Fri, 23 Nov 2012 21:10:33 +0100
> Florian Fainelli <florian@openwrt.org> wrote:
> > MIPS and MIPSel specific issues:
> > 
> > using qemu's binary emulation (qemu-mips[el]) against a statically 
> > linked binary works, a dynamic binary makes my host qemu segv. Using 
> > qemu-system-mips[el], the binary segfaults while being executed by the 
> > kernel. So far I have not had the time to debug this further.
> 
> There are likely other issues, but are you using a hardfloat-compatible kernel (FPU emulation enabled)?
> At present, musl/mips relies on certain instructions that the kernel  ordinarily emulates when they aren't supported.
> 

the fpu issue should not cause segfault,
iirc it infinite loops at the fp instruction

it's a dynamic linking related issue

i've already ran cross compiled, dynlinked
mips binaries on an openwrt router using
musl based toolchain[1] (although that was
about a month ago)

i would guess it's some build issue (wrong
thing gets linked, or -fPIC fails,..)
or something in the runtime environment is
different from what musl expects

maybe try to run the loader in 'ldd' mode:
musl's libc.so is the loader itself and
can be invoked as

 pathto/libc.so executable

which runs the dynlinked executable (so the
musl dynamic loader can be straced) or

 ln -s pathto/libc.so pathto/ldd
 pathto/ldd executable

which prints the opened dso paths during
loading and then quits

[1] https://bitbucket.org/GregorR/musl-cross/downloads

Re: [musl] Support for musl libc in OpenWrt

[Florian Fainelli]

Le samedi 24 novembre 2012 07:15:33, Isaac Dunham a écrit :
> On Fri, 23 Nov 2012 21:10:33 +0100
> 
> Florian Fainelli <florian@openwrt.org> wrote:
> > Hi All,
> > 
> > Support for musl libc in OpenWrt has been added in revision r34314,
> 
> Glad to hear it!
> 
> > Note that the following issues are known:
> > 
> > - unable to complete libstdc++ build due to some missing declarations
> > (WIP)
> 
> Are you using the default "linux-gnu" libstdc++, or the "generic" target?
> Only the latter is known to work at this point.

It is the linux-gnu libstdc++, we do not support building the "generic" one 
yet, might be worth supporting it now.

> 
> > - MIPS and MIPSel ports are not functionnal (details below)
> > 
> > --
> > MIPS and MIPSel specific issues:
> > 
> > using qemu's binary emulation (qemu-mips[el]) against a statically
> > linked binary works, a dynamic binary makes my host qemu segv. Using
> > qemu-system-mips[el], the binary segfaults while being executed by the
> > kernel. So far I have not had the time to debug this further.
> 
> There are likely other issues, but are you using a hardfloat-compatible
> kernel (FPU emulation enabled)? 

No, we have patched the kernel do disable the hardware FPU emulation, which 
was actually the issue, because musl setjmp MIPS-specific code tried to access 
Cop1 registers. After patching setjmp not to access these it works flawlessly 
on MIPS (big-endian) but MIPS (little-endian) still segfaults while being 
executed either from the kernel or qemu-mipsel.

> At present, musl/mips relies on certain
> instructions that the kernel  ordinarily emulates when they aren't
> supported.

You mean ll/sc right? I build for MIPS32r2 so this should not be an issue.
-- 
Florian

Re: [musl] Support for musl libc in OpenWrt

[Szabolcs Nagy]

* Florian Fainelli <florian@openwrt.org> [2012-11-25 12:46:26 +0100]:
> Cop1 registers. After patching setjmp not to access these it works flawlessly 
> on MIPS (big-endian) but MIPS (little-endian) still segfaults while being 
> executed either from the kernel or qemu-mipsel.
> 

so there is endian issue with dynamic linking

musl c code should not depend on endianness
(eg on x86 it works fine) so i guess the issue
is mips specific

maybe something in mips asm

or maybe there is some difference between
mips and mipsel other than the endianness?
(elf header or auxv something, or the build
toolchain is different in some way)

can you run libc.so itself?

/lib/libc.so

should print a usage message

if it works then try to run it as ldd on
a dynlinked binary

Re: Re: [OpenWrt-Devel] [musl] Support for musl libc in OpenWrt

[John Spencer]

On 11/25/2012 02:37 PM, Szabolcs Nagy wrote:
> * Florian Fainelli<florian@openwrt.org>  [2012-11-25 12:46:26 +0100]:
>> Cop1 registers. After patching setjmp not to access these it works flawlessly
>> on MIPS (big-endian) but MIPS (little-endian) still segfaults while being
>> executed either from the kernel or qemu-mipsel.
>>
> so there is endian issue with dynamic linking

nope, mipsel works as fine as mipseb using sabotage.
there must be something wrong on the openwrt side.

Re: Re: [OpenWrt-Devel] [musl] Support for musl libc in OpenWrt

[Florian Fainelli]

Le 25/11/2012 19:41, John Spencer a écrit :
> On 11/25/2012 02:37 PM, Szabolcs Nagy wrote:
>> * Florian Fainelli<florian@openwrt.org>  [2012-11-25 12:46:26 +0100]:
>>> Cop1 registers. After patching setjmp not to access these it works
>>> flawlessly
>>> on MIPS (big-endian) but MIPS (little-endian) still segfaults while
>>> being
>>> executed either from the kernel or qemu-mipsel.
>>>
>> so there is endian issue with dynamic linking
>
> nope, mipsel works as fine as mipseb using sabotage.
> there must be something wrong on the openwrt side.

Ok, then I will just start with more vanilla toolchains and kernel and 
see what could be going wrong. Thanks!
--
Florian

Re: Re: [OpenWrt-Devel] [musl] Support for musl libc in OpenWrt

[Rich Felker]

On Sun, Nov 25, 2012 at 08:29:34PM +0100, Florian Fainelli wrote:
> Le 25/11/2012 19:41, John Spencer a écrit :
> >On 11/25/2012 02:37 PM, Szabolcs Nagy wrote:
> >>* Florian Fainelli<florian@openwrt.org>  [2012-11-25 12:46:26 +0100]:
> >>>Cop1 registers. After patching setjmp not to access these it works
> >>>flawlessly
> >>>on MIPS (big-endian) but MIPS (little-endian) still segfaults while
> >>>being
> >>>executed either from the kernel or qemu-mipsel.
> >>>
> >>so there is endian issue with dynamic linking
> >
> >nope, mipsel works as fine as mipseb using sabotage.
> >there must be something wrong on the openwrt side.
> 
> Ok, then I will just start with more vanilla toolchains and kernel
> and see what could be going wrong. Thanks!

Any progress on this?

Rich

Re: Re: [OpenWrt-Devel] [musl] Support for musl libc in OpenWrt

[Florian Fainelli]

Le 01/12/2012 05:15, Rich Felker a écrit :
> On Sun, Nov 25, 2012 at 08:29:34PM +0100, Florian Fainelli wrote:
>> Le 25/11/2012 19:41, John Spencer a écrit :
>>> On 11/25/2012 02:37 PM, Szabolcs Nagy wrote:
>>>> * Florian Fainelli<florian@openwrt.org>  [2012-11-25 12:46:26 +0100]:
>>>>> Cop1 registers. After patching setjmp not to access these it works
>>>>> flawlessly
>>>>> on MIPS (big-endian) but MIPS (little-endian) still segfaults while
>>>>> being
>>>>> executed either from the kernel or qemu-mipsel.
>>>>>
>>>> so there is endian issue with dynamic linking
>>>
>>> nope, mipsel works as fine as mipseb using sabotage.
>>> there must be something wrong on the openwrt side.
>>
>> Ok, then I will just start with more vanilla toolchains and kernel
>> and see what could be going wrong. Thanks!
>
> Any progress on this?

Yes, the issue is not related to the endianness, the problem seems to be 
either in how we integrate musl into the OpenWrt build system (unlikely 
since arm and x86* work ok using the same integration), but rather the 
musl build system and/or compiler environment. I got different results 
depending on how I rebuilt musl and the rootfs, this still needs 
investigation.

In any case, don't consider this a blocker of any kind from the musl 
point of view.
--
Florian