Re: [musl] building statically linked DSOs with musl

[Joakim Sindholt <opensource@zhasha.com>]

On Sat, Jan 01, 2022 at 05:32:46PM +0800, Sebastien Bourdeauducq wrote:
> Dear musl libc developers,
> 
> I'm trying to build a shared library that is itself statically linked.
> The reason for doing that is building a Python plug-in, which contains 
> complex dependencies such as LLVM, and which should use a custom malloc 
> everywhere. Dynamic linking doesn't work in my case as the malloc & co 
> symbols are already resolved by Python when my plugin is loaded, and 
> replacing all malloc & co calls in LLVM, libstdc++, etc. with a 
> differently named function seems pretty difficult.
> 
> Naturally I've come to musl libc, since glibc comes with a bunch of 
> nonsense when attempting to link against it statically.

While musl is terribly good at static linking I don't think what you're
trying to achieve is technically possible. Libc, both the GNU and musl
ones, require exclusive access to a number of global resources for some
of their features and as such simply assume that they have it.

This is what's going to be your problem below:

> For simple code this goes reasonably well:
> $ nix-shell -p pkgsMusl.clang_13
> $ cat staticlib.c
> #include <stdio.h>
> 
> void test_func() {
> 	puts("hello world");
> }
> 
> $ clang -o libstaticlib.so -shared staticlib.c -fPIC -Wl,-Bstatic -lc
> $ ldd libstaticlib.so
> 	statically linked
> 
> The "test" program below is built with and linked against glibc, as 
> would be the case in my Python plugin situation.
> 
> $ nix-shell -p clang_13
> $ cat test.c
> #include <stdio.h>
> 
> void test_func();
> 
> int main() {
> 	test_func();
> 	return 0;
> }
> $ clang -o test -L. -lstaticlib test.c
> $ ldd test
> 	linux-vdso.so.1 (0x00007ffd4db5a000)
> 	libstaticlib.so => not found
> 	libc.so.6 => 
> /nix/store/vjq3q7dq8vmc13c3py97v27qwizvq7fd-glibc-2.33-59/lib/libc.so.6 
> (0x00007fbb29746000)
> 	/nix/store/vjq3q7dq8vmc13c3py97v27qwizvq7fd-glibc-2.33-59/lib/ld-linux-x86-64.so.2 => /nix/store/vjq3q7dq8vmc13c3py97v27qwizvq7fd-glibc-2.33-59/lib64/ld-linux-x86-64.so.2 (0x00007fbb2990d000)
> $ LD_LIBRARY_PATH=. ./test
> hello world
> 
> So far, so good. But trouble begins when calling certain musl functions 
> from the library:
> $ cat staticlib.c
> #include <stdio.h>
> #include <sys/time.h>
> 
> void test_func() {
> 	struct timeval tv;
> 	gettimeofday(&tv, NULL);
> 	printf("%ld", tv.tv_sec);
> }
> $ clang -o libstaticlib.so -shared staticlib.c -fPIC -Wl,-Bstatic -lc
> $ LD_LIBRARY_PATH=. ./test
> Segmentation fault (core dumped)
> 
> (gdb) backtrace
> #0  0x00007ffff7fbf527 in __vdsosym () from ./libstaticlib.so
> #1  0x00007ffff7fbf439 in cgt_init () from ./libstaticlib.so
> #2  0x00007ffff7fbf47d in clock_gettime () from ./libstaticlib.so
> #3  0x00007ffff7fbedba in gettimeofday () from ./libstaticlib.so
> #4  0x00007ffff7fbd2c5 in test_func () from ./libstaticlib.so
> #5  0x0000000000401138 in main ()
> 
> Has anyone encountered this and would have an explanation or fix?

clock_gettime(), which gettimeofday() uses internally now, wants to
check if linux provides a user space fast path through the
injected-into-every-process VDSO.
When the dynamic linker, in this case glibc's ld.so, loads your binary
it will do a bunch of internal setup in addition to symbol stitchup and
whatever else needs to be done. Here you've run into the first of many
brick walls. Musl will set up its own internal global libc structure
with a bunch of values during the initial dynamic loading phase; among
the members is libc.auxv, which is what __vdsosym will look through when
trying to find the VDSO. Since you never ran musl's dynamic linker (and
even if your host binary was musl-based, not the one that would have
initialized the libc.auxv baked in to your statically linked DSO) it
won't have set up this and a whole host of other things.

It's not just a question of "just run this setup code and it will work"
either, as a lot of libc functionality by necessity depends on exclusive
access and implementation details that can't cross the barrier you've
set up.

Best I can figure your only solution, if you want to ship your own
ecosystem, is to start another process and talk to it over a
socket/pipe.
I'm sure this wasn't a very helpful post. Maybe someone else on this
list has a solution that better fits your needs.

Joakim