[musl] building statically linked DSOs with musl

[musl] building statically linked DSOs with musl

[Sebastien Bourdeauducq]

Dear musl libc developers,

I'm trying to build a shared library that is itself statically linked.
The reason for doing that is building a Python plug-in, which contains 
complex dependencies such as LLVM, and which should use a custom malloc 
everywhere. Dynamic linking doesn't work in my case as the malloc & co 
symbols are already resolved by Python when my plugin is loaded, and 
replacing all malloc & co calls in LLVM, libstdc++, etc. with a 
differently named function seems pretty difficult.

Naturally I've come to musl libc, since glibc comes with a bunch of 
nonsense when attempting to link against it statically.

For simple code this goes reasonably well:
$ nix-shell -p pkgsMusl.clang_13
$ cat staticlib.c
#include <stdio.h>

void test_func() {
	puts("hello world");
}

$ clang -o libstaticlib.so -shared staticlib.c -fPIC -Wl,-Bstatic -lc
$ ldd libstaticlib.so
	statically linked

The "test" program below is built with and linked against glibc, as 
would be the case in my Python plugin situation.

$ nix-shell -p clang_13
$ cat test.c
#include <stdio.h>

void test_func();

int main() {
	test_func();
	return 0;
}
$ clang -o test -L. -lstaticlib test.c
$ ldd test
	linux-vdso.so.1 (0x00007ffd4db5a000)
	libstaticlib.so => not found
	libc.so.6 => 
/nix/store/vjq3q7dq8vmc13c3py97v27qwizvq7fd-glibc-2.33-59/lib/libc.so.6 
(0x00007fbb29746000)
	/nix/store/vjq3q7dq8vmc13c3py97v27qwizvq7fd-glibc-2.33-59/lib/ld-linux-x86-64.so.2 => /nix/store/vjq3q7dq8vmc13c3py97v27qwizvq7fd-glibc-2.33-59/lib64/ld-linux-x86-64.so.2 (0x00007fbb2990d000)
$ LD_LIBRARY_PATH=. ./test
hello world

So far, so good. But trouble begins when calling certain musl functions 
from the library:
$ cat staticlib.c
#include <stdio.h>
#include <sys/time.h>

void test_func() {
	struct timeval tv;
	gettimeofday(&tv, NULL);
	printf("%ld", tv.tv_sec);
}
$ clang -o libstaticlib.so -shared staticlib.c -fPIC -Wl,-Bstatic -lc
$ LD_LIBRARY_PATH=. ./test
Segmentation fault (core dumped)

(gdb) backtrace
#0  0x00007ffff7fbf527 in __vdsosym () from ./libstaticlib.so
#1  0x00007ffff7fbf439 in cgt_init () from ./libstaticlib.so
#2  0x00007ffff7fbf47d in clock_gettime () from ./libstaticlib.so
#3  0x00007ffff7fbedba in gettimeofday () from ./libstaticlib.so
#4  0x00007ffff7fbd2c5 in test_func () from ./libstaticlib.so
#5  0x0000000000401138 in main ()

Has anyone encountered this and would have an explanation or fix?

Thanks,
Sébastien

Re: [musl] building statically linked DSOs with musl

[Joakim Sindholt]

On Sat, Jan 01, 2022 at 05:32:46PM +0800, Sebastien Bourdeauducq wrote:
> Dear musl libc developers,
> 
> I'm trying to build a shared library that is itself statically linked.
> The reason for doing that is building a Python plug-in, which contains 
> complex dependencies such as LLVM, and which should use a custom malloc 
> everywhere. Dynamic linking doesn't work in my case as the malloc & co 
> symbols are already resolved by Python when my plugin is loaded, and 
> replacing all malloc & co calls in LLVM, libstdc++, etc. with a 
> differently named function seems pretty difficult.
> 
> Naturally I've come to musl libc, since glibc comes with a bunch of 
> nonsense when attempting to link against it statically.

While musl is terribly good at static linking I don't think what you're
trying to achieve is technically possible. Libc, both the GNU and musl
ones, require exclusive access to a number of global resources for some
of their features and as such simply assume that they have it.

This is what's going to be your problem below:

> For simple code this goes reasonably well:
> $ nix-shell -p pkgsMusl.clang_13
> $ cat staticlib.c
> #include <stdio.h>
> 
> void test_func() {
> 	puts("hello world");
> }
> 
> $ clang -o libstaticlib.so -shared staticlib.c -fPIC -Wl,-Bstatic -lc
> $ ldd libstaticlib.so
> 	statically linked
> 
> The "test" program below is built with and linked against glibc, as 
> would be the case in my Python plugin situation.
> 
> $ nix-shell -p clang_13
> $ cat test.c
> #include <stdio.h>
> 
> void test_func();
> 
> int main() {
> 	test_func();
> 	return 0;
> }
> $ clang -o test -L. -lstaticlib test.c
> $ ldd test
> 	linux-vdso.so.1 (0x00007ffd4db5a000)
> 	libstaticlib.so => not found
> 	libc.so.6 => 
> /nix/store/vjq3q7dq8vmc13c3py97v27qwizvq7fd-glibc-2.33-59/lib/libc.so.6 
> (0x00007fbb29746000)
> 	/nix/store/vjq3q7dq8vmc13c3py97v27qwizvq7fd-glibc-2.33-59/lib/ld-linux-x86-64.so.2 => /nix/store/vjq3q7dq8vmc13c3py97v27qwizvq7fd-glibc-2.33-59/lib64/ld-linux-x86-64.so.2 (0x00007fbb2990d000)
> $ LD_LIBRARY_PATH=. ./test
> hello world
> 
> So far, so good. But trouble begins when calling certain musl functions 
> from the library:
> $ cat staticlib.c
> #include <stdio.h>
> #include <sys/time.h>
> 
> void test_func() {
> 	struct timeval tv;
> 	gettimeofday(&tv, NULL);
> 	printf("%ld", tv.tv_sec);
> }
> $ clang -o libstaticlib.so -shared staticlib.c -fPIC -Wl,-Bstatic -lc
> $ LD_LIBRARY_PATH=. ./test
> Segmentation fault (core dumped)
> 
> (gdb) backtrace
> #0  0x00007ffff7fbf527 in __vdsosym () from ./libstaticlib.so
> #1  0x00007ffff7fbf439 in cgt_init () from ./libstaticlib.so
> #2  0x00007ffff7fbf47d in clock_gettime () from ./libstaticlib.so
> #3  0x00007ffff7fbedba in gettimeofday () from ./libstaticlib.so
> #4  0x00007ffff7fbd2c5 in test_func () from ./libstaticlib.so
> #5  0x0000000000401138 in main ()
> 
> Has anyone encountered this and would have an explanation or fix?

clock_gettime(), which gettimeofday() uses internally now, wants to
check if linux provides a user space fast path through the
injected-into-every-process VDSO.
When the dynamic linker, in this case glibc's ld.so, loads your binary
it will do a bunch of internal setup in addition to symbol stitchup and
whatever else needs to be done. Here you've run into the first of many
brick walls. Musl will set up its own internal global libc structure
with a bunch of values during the initial dynamic loading phase; among
the members is libc.auxv, which is what __vdsosym will look through when
trying to find the VDSO. Since you never ran musl's dynamic linker (and
even if your host binary was musl-based, not the one that would have
initialized the libc.auxv baked in to your statically linked DSO) it
won't have set up this and a whole host of other things.

It's not just a question of "just run this setup code and it will work"
either, as a lot of libc functionality by necessity depends on exclusive
access and implementation details that can't cross the barrier you've
set up.

Best I can figure your only solution, if you want to ship your own
ecosystem, is to start another process and talk to it over a
socket/pipe.
I'm sure this wasn't a very helpful post. Maybe someone else on this
list has a solution that better fits your needs.

Joakim

Re: [musl] building statically linked DSOs with musl

[Alex Xu (Hello71)]

Excerpts from Joakim Sindholt's message of January 1, 2022 7:56 am:
> clock_gettime(), which gettimeofday() uses internally now, wants to
> check if linux provides a user space fast path through the
> injected-into-every-process VDSO.
> When the dynamic linker, in this case glibc's ld.so, loads your binary
> it will do a bunch of internal setup in addition to symbol stitchup and
> whatever else needs to be done. Here you've run into the first of many
> brick walls. Musl will set up its own internal global libc structure
> with a bunch of values during the initial dynamic loading phase; among
> the members is libc.auxv, which is what __vdsosym will look through when
> trying to find the VDSO. Since you never ran musl's dynamic linker (and
> even if your host binary was musl-based, not the one that would have
> initialized the libc.auxv baked in to your statically linked DSO) it
> won't have set up this and a whole host of other things.
> 
> It's not just a question of "just run this setup code and it will work"
> either, as a lot of libc functionality by necessity depends on exclusive
> access and implementation details that can't cross the barrier you've
> set up.
> 
> Best I can figure your only solution, if you want to ship your own
> ecosystem, is to start another process and talk to it over a
> socket/pipe.
> I'm sure this wasn't a very helpful post. Maybe someone else on this
> list has a solution that better fits your needs.
> 
> Joakim
> 

Yes, this is fundamentally impossible. You could avoid much of this 
difficulty by attempting to statically link or otherwise forcibly bind 
your library's dependencies to your alternative malloc; however, this 
merely decreases the size of the issue rather than removing it. First, 
malloc implementations (may) use a process-global resource: (s)brk. It 
will cause havoc if multiple implementations attempt to simultaneously 
manipulate it. Second, even if no more than one implementation uses 
(s)brk, they will still have incompatible metadata. Consider the case 
where your plugin allocates some memory with malloc, then passes the 
pointer to Python which then frees it. Since an incompatible 
implementation is processing it, it will most likely destroy the heap or 
crash.

In conclusion, without extremely careful preparation of the entire 
program and all its DSOs, it is mandatory to use a single libc and 
malloc implementation in all DSOs.

Cheers,
Alex.

Re: [musl] building statically linked DSOs with musl

[Sebastien Bourdeauducq]

On 1/1/22 21:58, Alex Xu (Hello71) wrote:
> First,
> malloc implementations (may) use a process-global resource: (s)brk. It
> will cause havoc if multiple implementations attempt to simultaneously
> manipulate it. Second, even if no more than one implementation uses
> (s)brk, they will still have incompatible metadata.

Can't there be two different heaps in different memory regions?

> Consider the case
> where your plugin allocates some memory with malloc, then passes the
> pointer to Python which then frees it. Since an incompatible
> implementation is processing it, it will most likely destroy the heap or
> crash.

Well yes, that's a fairly obvious issue and we are not doing that; the 
side that allocates the memory is always the one that frees it.

Sébastien

Re: [musl] building statically linked DSOs with musl

[Markus Wichmann]

On Sat, Jan 01, 2022 at 10:31:58PM +0800, Sebastien Bourdeauducq wrote:
> On 1/1/22 21:58, Alex Xu (Hello71) wrote:
> > First,
> > malloc implementations (may) use a process-global resource: (s)brk. It
> > will cause havoc if multiple implementations attempt to simultaneously
> > manipulate it. Second, even if no more than one implementation uses
> > (s)brk, they will still have incompatible metadata.
>
> Can't there be two different heaps in different memory regions?
>

There can be different heaps, but not different brk heaps. Basically,
brk() manages a global variable generated by the kernel. You can work
around that problem by installing a seccomp filter that makes brk()
always fail, forcing the allocators to fall back to mmap().

However, your approach has numerous other problems, especially with the
mismatch between glibc and musl. The thread pointer does not match up,
and so no functions accessing the thread pointers will work. That
includes functions that read the current thread pointer, so you cannot
know in advance which those are. All the implementation-defined
structures mismatch, so none of the locking functions work, and neither
do the semaphore functions. The thread-management functions also will
not work, and you are saying you have complex dependencies that might
use those.

No, the best you can probably do is put all the work into a separate
program. Define some sort of IPC interface, and have the plugin merely
serialize requests and read responses. Then your process can be as
complicated as you want and use whatever libc you fancy. It can even use
a custom malloc. And your plugin should be rather lean.

HTH,
Markus

Re: [musl] building statically linked DSOs with musl

[Sebastien Bourdeauducq]

On 1/1/22 20:56, Joakim Sindholt wrote:
> Musl will set up its own internal global libc structure
> with a bunch of values during the initial dynamic loading phase; among
> the members is libc.auxv, which is what __vdsosym will look through when
> trying to find the VDSO. Since you never ran musl's dynamic linker (and
> even if your host binary was musl-based, not the one that would have
> initialized the libc.auxv baked in to your statically linked DSO) it
> won't have set up this and a whole host of other things.

Thanks for the hint.

libc.auxv seems to be set up by __dls2b, which itself is called by 
__dls2 via find_sym(&ldso, "__dls2b", 0).
How does this code work when a program is statically linked against musl?

On 1/3/22 05:41, Markus Wichmann wrote:
 > There can be different heaps, but not different brk heaps. Basically,
 > brk() manages a global variable generated by the kernel. You can work
 > around that problem by installing a seccomp filter that makes brk()
 > always fail, forcing the allocators to fall back to mmap().

mimalloc exclusively uses mmap() already, so that will be OK and will 
not require a workaround. It shouldn't matter if the other allocator 
uses brk().

 > The thread pointer does not match up,

Seems it does (okay, probably by accident):
https://git.musl-libc.org/cgit/musl/tree/arch/x86_64/pthread_arch.h?id=cfdfd5ea3ce14c6abf7fb22a531f3d99518b5a1b#n4
https://github.com/bminor/glibc/blob/b92a49359f33a461db080a33940d73f47c756126/sysdeps/x86/nptl/thread_pointer.h#L30

 > and so no functions accessing the thread pointers will work. That
 > includes functions that read the current thread pointer, so you cannot
 > know in advance which those are. All the implementation-defined
 > structures mismatch, so none of the locking functions work, and neither
 > do the semaphore functions.

AFAIK I would not need to share locks or semaphores across the plugin. I 
need to see if the rest of this thread business has other consequences.

Sébastien

Re: [musl] building statically linked DSOs with musl

[Markus Wichmann]

On Mon, Jan 03, 2022 at 09:37:00AM +0800, Sebastien Bourdeauducq wrote:
> On 1/1/22 20:56, Joakim Sindholt wrote:
> > Musl will set up its own internal global libc structure
> > with a bunch of values during the initial dynamic loading phase; among
> > the members is libc.auxv, which is what __vdsosym will look through when
> > trying to find the VDSO. Since you never ran musl's dynamic linker (and
> > even if your host binary was musl-based, not the one that would have
> > initialized the libc.auxv baked in to your statically linked DSO) it
> > won't have set up this and a whole host of other things.
>
> Thanks for the hint.
>
> libc.auxv seems to be set up by __dls2b, which itself is called by __dls2
> via find_sym(&ldso, "__dls2b", 0).
> How does this code work when a program is statically linked against musl?
>

It doesn't. This code only runs in the dynamic linking case. In the
static linking one, the program is linked against crt0.c, which contains
_start and _start_c, which will call __libc_start_main, which itself
will call __init_libc. Note that you cannot call __init_libc, because
for that you need the original stack pointer, which you don't have.

Ciao,
Markus