Failed to execute /init (error -2) for system-wide musl libc

Failed to execute /init (error -2) for system-wide musl libc

[piranna]

On NodeOS we are having some problems creating an i686 cross-compiler
using musl (https://github.com/NodeOS/NodeOS/pull/89).

The point is that by generating a fully statically linked Node.js
executable it boot correctly, but by using a dynamically one we get a
"Failed to execute /init (error -2)" error. We have crafted the
cross-compiler following the instructions at
http://clfs.org/view/clfs-embedded with no lack. The fact is that
executables compiled with the cross-toolchain works on our desktops
but they don't on QEmu, and we've added all the dynamic libraries that
readelf request (only libc.so, libstdc++.so & libgcc_s.so). As a
curious side note, when installing the musl-linked gcc libraries, the
/lib/ld-musl-i386.so.1 link dissapear from the cross-toochain folder,
but also adding it by hand all the dynamic linked executables refuse
to load on QEmu, also a simple "Hello World" that only depends on
libc.so :-/

We are getting out of ideas, any clue about what could be happening?

Greetings, and Happy New Year :-)

Jesús Leganés Combarro "piranna"


-- 
"Si quieres viajar alrededor del mundo y ser invitado a hablar en un
monton de sitios diferentes, simplemente escribe un sistema operativo
Unix."
– Linus Tordvals, creador del sistema operativo Linux

Re: Failed to execute /init (error -2) for system-wide musl libc

[Szabolcs Nagy]

* piranna@gmail.com <piranna@gmail.com> [2014-12-31 19:04:55 +0100]:
> On NodeOS we are having some problems creating an i686 cross-compiler
> using musl (https://github.com/NodeOS/NodeOS/pull/89).
> 
> The point is that by generating a fully statically linked Node.js
> executable it boot correctly, but by using a dynamically one we get a
> "Failed to execute /init (error -2)" error. We have crafted the
> cross-compiler following the instructions at
> http://clfs.org/view/clfs-embedded with no lack. The fact is that

i think if static init works then try to use
that to boot up the system and then debug the
dynamic case..

you can run libc.so with a dynamic executable
as argument then you can debug the loader

if you have no debug tools on the target then
i think you can add dprintf's to early loader
code in src/ldso/dynlink.c which may help in
figuring out what fails


> executables compiled with the cross-toolchain works on our desktops
> but they don't on QEmu, and we've added all the dynamic libraries that
> readelf request (only libc.so, libstdc++.so & libgcc_s.so). As a

libgcc_s.so shouldnt be needed

> curious side note, when installing the musl-linked gcc libraries, the
> /lib/ld-musl-i386.so.1 link dissapear from the cross-toochain folder,

sounds like a clfs buildsystem issue

i've seen the symlink disappearing on openwrt buildsystem once
(that one was not deterministic) i'm not sure what caused it

> but also adding it by hand all the dynamic linked executables refuse
> to load on QEmu, also a simple "Hello World" that only depends on
> libc.so :-/
> 
> We are getting out of ideas, any clue about what could be happening?
> 
> Greetings, and Happy New Year :-)

Re: Failed to execute /init (error -2) for system-wide musl libc

[Isaac Dunham]

On Wed, Dec 31, 2014 at 07:04:55PM +0100, piranna@gmail.com wrote:
> On NodeOS we are having some problems creating an i686 cross-compiler
> using musl (https://github.com/NodeOS/NodeOS/pull/89).
> 
> The point is that by generating a fully statically linked Node.js
> executable it boot correctly, but by using a dynamically one we get a
> "Failed to execute /init (error -2)" error. We have crafted the
> cross-compiler following the instructions at
> http://clfs.org/view/clfs-embedded with no lack. The fact is that
> executables compiled with the cross-toolchain works on our desktops
> but they don't on QEmu, and we've added all the dynamic libraries that
> readelf request (only libc.so, libstdc++.so & libgcc_s.so). As a
> curious side note, when installing the musl-linked gcc libraries, the
> /lib/ld-musl-i386.so.1 link dissapear from the cross-toochain folder,
> but also adding it by hand all the dynamic linked executables refuse
> to load on QEmu, also a simple "Hello World" that only depends on
> libc.so :-/
> 
> We are getting out of ideas, any clue about what could be happening?

I'm not clear what your root filesystem for qemu looks like 
(the contents of /lib are the interesting part).
An ls -l of that directory would be useful.

It *should* look like this:
/lib
/lib/ld-musl-i386.so.1 # either file or  -> /lib/libc.so
/lib/libc.so # only if ld-musl-i386.so.1 is symlink
/lib/libstdc++.so.6
/lib/libgcc_s.so.1

Note the version numbers.
All of these should be chmod 0755.
The libraries are ordinarily symlinks to the real ones:
$ ls -l /usr/lib/libstdc++.so*
lrwxrwxrwx    1 root     root            19 Dec 30 15:27 /usr/lib/libstdc++.so -> libstdc++.so.6.0.20
lrwxrwxrwx    1 root     root            19 Dec 30 15:24 /usr/lib/libstdc++.so.6 -> libstdc++.so.6.0.20
-rwxr-xr-x    1 root     root        845592 Dec 12 01:38 /usr/lib/libstdc++.so.6.0.20


If they are symlinks, check that the symlink target is also present.
 
> Greetings, and Happy New Year :-)
> 
> Jesús Leganés Combarro "piranna"

And Happy New Year to you, also!

Isaac Dunham

Re: Failed to execute /init (error -2) for system-wide musl libc

[piranna]

> i think if static init works then try to use
> that to boot up the system and then debug the
> dynamic case..

It's a good idea, didn't though about it :-)


> you can run libc.so with a dynamic executable
> as argument then you can debug the loader

Definitely I can be able to use Node.js child_process module to launch
libc.so, I'll try it :-)


> if you have no debug tools on the target then
> i think you can add dprintf's to early loader
> code in src/ldso/dynlink.c which may help in
> figuring out what fails

It's just a Node.js REPL, so no debug tools here (at least, no C ones,
like GDB), but knowing that src/ldso/dynlink.c has the code of the
dynamic linker, add some debug printf's seems a good idea, I believe
although it's in the boot process it should write some output to the
console...


>> executables compiled with the cross-toolchain works on our desktops
>> but they don't on QEmu, and we've added all the dynamic libraries that
>> readelf request (only libc.so, libstdc++.so & libgcc_s.so). As a
>
> libgcc_s.so shouldnt be needed

It's shown as a dependency of libstdc++.so both by ldd and readelf -d.
Maybe could this be the issue? Both libstdc++.so & libgcc_s.so are
compiled against musl...


>> curious side note, when installing the musl-linked gcc libraries, the
>> /lib/ld-musl-i386.so.1 link dissapear from the cross-toochain folder,
>
> sounds like a clfs buildsystem issue

Or CLFS, or gcc, but it's something curious because before the
compilation of libstdc++.so & libgcc_s.so against musl the symbolic
link is there, and after compile them the symbolic link has
dissapear... It doesn't makes sense... :-/


> i've seen the symlink disappearing on openwrt buildsystem once
> (that one was not deterministic) i'm not sure what caused it

In my case is deterministic, allways happens on the libstdc++.so &
libgcc_s.so generation step (more exactly, the make install on the
cross-toolchain directory), checked by doing them by hand :-/


-- 
"Si quieres viajar alrededor del mundo y ser invitado a hablar en un
monton de sitios diferentes, simplemente escribe un sistema operativo
Unix."
– Linus Tordvals, creador del sistema operativo Linux

Re: Failed to execute /init (error -2) for system-wide musl libc

[Matias A. Fonzo]

Hola Jesús,
El Wed, 31 Dec 2014 20:49:06 +0100
"piranna@gmail.com" <piranna@gmail.com> escribió:
> > i think if static init works then try to use
> > that to boot up the system and then debug the
> > dynamic case..
> 
> It's a good idea, didn't though about it :-)
> 
> 
> > you can run libc.so with a dynamic executable
> > as argument then you can debug the loader
> 
> Definitely I can be able to use Node.js child_process module to launch
> libc.so, I'll try it :-)
> 
> 
> > if you have no debug tools on the target then
> > i think you can add dprintf's to early loader
> > code in src/ldso/dynlink.c which may help in
> > figuring out what fails
> 
> It's just a Node.js REPL, so no debug tools here (at least, no C ones,
> like GDB), but knowing that src/ldso/dynlink.c has the code of the
> dynamic linker, add some debug printf's seems a good idea, I believe
> although it's in the boot process it should write some output to the
> console...
> 
> 
> >> executables compiled with the cross-toolchain works on our desktops
> >> but they don't on QEmu, and we've added all the dynamic libraries
> >> that readelf request (only libc.so, libstdc++.so & libgcc_s.so).
> >> As a
> >
> > libgcc_s.so shouldnt be needed
> 
> It's shown as a dependency of libstdc++.so both by ldd and readelf -d.
> Maybe could this be the issue? Both libstdc++.so & libgcc_s.so are
> compiled against musl...
> 
> 
> >> curious side note, when installing the musl-linked gcc libraries,
> >> the /lib/ld-musl-i386.so.1 link dissapear from the cross-toochain
> >> folder,
> >
> > sounds like a clfs buildsystem issue
> 
> Or CLFS, or gcc, but it's something curious because before the
> compilation of libstdc++.so & libgcc_s.so against musl the symbolic
> link is there, and after compile them the symbolic link has
> dissapear... It doesn't makes sense... :-/
> 

You've used CLFS-embedded, right?.  Probably you are getting
dynamic-link issues because the steps used for musl in CLFS-embedded
are not linked with the cross binutils (or the binutils from the
clfs-toolchain).  You need to add CROSS_COMPILE="${CLFS_TARGET}-" to the
environment.

> 
> > i've seen the symlink disappearing on openwrt buildsystem once
> > (that one was not deterministic) i'm not sure what caused it
> 
> In my case is deterministic, allways happens on the libstdc++.so &
> libgcc_s.so generation step (more exactly, the make install on the
> cross-toolchain directory), checked by doing them by hand :-/
> 
> 

Re: Failed to execute /init (error -2) for system-wide musl libc

[piranna]

> I'm not clear what your root filesystem for qemu looks like
> (the contents of /lib are the interesting part).

It's being generated by a cpio.txt file (it's being bundle into Linux
kernel to be used as initramfs). You can check it at
https://github.com/NodeOS/NodeOS/pull/89/files#diff-2


> An ls -l of that directory would be useful.
>
> It *should* look like this:
> /lib
> /lib/ld-musl-i386.so.1 # either file or  -> /lib/libc.so
> /lib/libc.so # only if ld-musl-i386.so.1 is symlink
> /lib/libstdc++.so.6
> /lib/libgcc_s.so.1
>
> Note the version numbers.


> All of these should be chmod 0755.

Does this to be mandatory? As you can see, I'm using a 0555 permission
on libc.so (because it's an executable) and 0444 for the other libs,
and 0111 for the directories. This is on purposse, both for security
and to keep users eyes out of ugly things and internal implementation
details (they should only worry about Node.js APIs).

Anyway, this gave me and idea and reviewing the cpio.txt file I've
found this line:

#dir /lib 0111 0 0

I uncommented it and the Hello World init has written to the console,
so I've just recompile using Node.js as /init, and...

> [    2.255765] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input3
[    2.863453] Switched to clocksource tsc
1 +1
2
> Date()
'Wed Dec 31 2014 20:08:31 GMT+0000 (GMT)'
>

/me head banging in the wall X-D I'll do some more tests, but seems
that was the issue X-D If so, probably it's a bug on the Linux kernel
initramfs builder (gen_init_cpio), because the .cpio.gz file is being
correctly generated and include the libraries without raising an
error... :-/


> The libraries are ordinarily symlinks to the real ones:
> $ ls -l /usr/lib/libstdc++.so*
> lrwxrwxrwx    1 root     root            19 Dec 30 15:27 /usr/lib/libstdc++.so -> libstdc++.so.6.0.20
> lrwxrwxrwx    1 root     root            19 Dec 30 15:24 /usr/lib/libstdc++.so.6 -> libstdc++.so.6.0.20
> -rwxr-xr-x    1 root     root        845592 Dec 12 01:38 /usr/lib/libstdc++.so.6.0.20
>
>
> If they are symlinks, check that the symlink target is also present.
>
>> Greetings, and Happy New Year :-)
>>
>> Jesús Leganés Combarro "piranna"
>
> And Happy New Year to you, also!

Thank you! ^^

-- 
"Si quieres viajar alrededor del mundo y ser invitado a hablar en un
monton de sitios diferentes, simplemente escribe un sistema operativo
Unix."
– Linus Tordvals, creador del sistema operativo Linux

Re: Failed to execute /init (error -2) for system-wide musl libc

[piranna]

> Anyway, this gave me and idea and reviewing the cpio.txt file I've
> found this line:
>
> #dir /lib 0111 0 0

https://github.com/NodeOS/NodeOS/pull/89#issuecomment-68467239

Well, not exactly a musl-related issue, but without your help this
couldn't be fixed... :-P Now officially NodeOS is a musl-powered and
cross-compiled Operating System! :-D

Re: Failed to execute /init (error -2) for system-wide musl libc

[Matias A. Fonzo]

El Wed, 31 Dec 2014 21:46:48 +0100
"piranna@gmail.com" <piranna@gmail.com> escribió:
> > You've used CLFS-embedded, right?.  Probably you are getting
> > dynamic-link issues because the steps used for musl in CLFS-embedded
> > are not linked with the cross binutils (or the binutils from the
> > clfs-toolchain).  You need to add CROSS_COMPILE="${CLFS_TARGET}-"
> > to the environment.
> 
> Yes, cross-compiler is based on CLFS-embedded. I think what you say
> it's not an issue because I'm explicitly using $TARGET-gcc, but
> definitely is a good idea to also define the environment variable just
> to be sure :-)
> 

With CC="${CLFS_TARGET}-gcc" you won't get musl linked against
cross-binutils. CROSS_COMPILE will use $TARGET-ld.

Re: Failed to execute /init (error -2) for system-wide musl libc

[piranna]

> With CC="${CLFS_TARGET}-gcc" you won't get musl linked against
> cross-binutils. CROSS_COMPILE will use $TARGET-ld.

Oh, so they would be linked with the host binutils? Then you are
right, this would be an issue when compiling for another platform like
ARM... Thanks for the advice! :-)


-- 
"Si quieres viajar alrededor del mundo y ser invitado a hablar en un
monton de sitios diferentes, simplemente escribe un sistema operativo
Unix."
– Linus Tordvals, creador del sistema operativo Linux

Re: Failed to execute /init (error -2) for system-wide musl libc

[Isaac Dunham]

On Wed, Dec 31, 2014 at 09:17:06PM +0100, piranna@gmail.com wrote:
> > I'm not clear what your root filesystem for qemu looks like
> > (the contents of /lib are the interesting part).
> 
> It's being generated by a cpio.txt file (it's being bundle into Linux
> kernel to be used as initramfs). You can check it at
> https://github.com/NodeOS/NodeOS/pull/89/files#diff-2
> 
> 
> > An ls -l of that directory would be useful.
> >
> > It *should* look like this:
> > /lib
> > /lib/ld-musl-i386.so.1 # either file or  -> /lib/libc.so
> > /lib/libc.so # only if ld-musl-i386.so.1 is symlink
> > /lib/libstdc++.so.6
> > /lib/libgcc_s.so.1
> >
> > Note the version numbers.
> 
> 
> > All of these should be chmod 0755.
> 
> Does this to be mandatory? As you can see, I'm using a 0555 permission
> on libc.so (because it's an executable) and 0444 for the other libs,
> and 0111 for the directories. This is on purposse, both for security
> and to keep users eyes out of ugly things and internal implementation
> details (they should only worry about Node.js APIs).
> 
> Anyway, this gave me and idea and reviewing the cpio.txt file I've
> found this line:
> 
> #dir /lib 0111 0 0
> 
> I uncommented it and the Hello World init has written to the console,
> so I've just recompile using Node.js as /init, and...
> 
> > [    2.255765] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input3
> [    2.863453] Switched to clocksource tsc
> 1 +1
> 2
> > Date()
> 'Wed Dec 31 2014 20:08:31 GMT+0000 (GMT)'
> >
> 
> /me head banging in the wall X-D I'll do some more tests, but seems
> that was the issue X-D If so, probably it's a bug on the Linux kernel
> initramfs builder (gen_init_cpio), because the .cpio.gz file is being
> correctly generated and include the libraries without raising an
> error... :-/

Glad you figured it out.

This is one of the arcane details of cpio:
as originally designed, every input that you specify corresponds to
exactly one record, and every record creates at most one file/directory.
The Linux kernel follows this approach; if you create /lib/file but not
/lib in your initrd, the non-existence of /lib will result in /lib/file
not being created.


Thanks,
Isaac Dunham

Re: Failed to execute /init (error -2) for system-wide musl libc

[piranna]

> Glad you figured it out.

Thank you! :-)


> This is one of the arcane details of cpio:
> as originally designed, every input that you specify corresponds to
> exactly one record, and every record creates at most one file/directory.
> The Linux kernel follows this approach; if you create /lib/file but not
> /lib in your initrd, the non-existence of /lib will result in /lib/file
> not being created.

Yeah, it's a simple algorythm, but if you de-compress the generated
.cpio.gz file the /lib/file are already there, so it doesn't make
sense. I've opened an issue on Linux bugtracker so gen_init_cpio
command exit with an error if trying to add a file on a directory that
doesn't exists:

https://bugzilla.kernel.org/show_bug.cgi?id=90561


-- 
"Si quieres viajar alrededor del mundo y ser invitado a hablar en un
monton de sitios diferentes, simplemente escribe un sistema operativo
Unix."
– Linus Tordvals, creador del sistema operativo Linux

Re: Failed to execute /init (error -2) for system-wide musl libc

[Andy Lutomirski]

On 12/31/2014 10:04 AM, piranna@gmail.com wrote:
> On NodeOS we are having some problems creating an i686 cross-compiler
> using musl (https://github.com/NodeOS/NodeOS/pull/89).
> 
> The point is that by generating a fully statically linked Node.js
> executable it boot correctly, but by using a dynamically one we get a
> "Failed to execute /init (error -2)" error. We have crafted the
> cross-compiler following the instructions at
> http://clfs.org/view/clfs-embedded with no lack. The fact is that
> executables compiled with the cross-toolchain works on our desktops
> but they don't on QEmu, and we've added all the dynamic libraries that
> readelf request (only libc.so, libstdc++.so & libgcc_s.so). As a
> curious side note, when installing the musl-linked gcc libraries, the
> /lib/ld-musl-i386.so.1 link dissapear from the cross-toochain folder,
> but also adding it by hand all the dynamic linked executables refuse
> to load on QEmu, also a simple "Hello World" that only depends on
> libc.so :-/
> 
> We are getting out of ideas, any clue about what could be happening?
> 
> Greetings, and Happy New Year :-)
> 
> Jesús Leganés Combarro "piranna"
> 
> 

To debug init problems, it might help a lot to run your init in a
container under strace.

For example:

$ unshare -r -p -f
# strace chroot /my_rootfs /sbin/init

--Andy