Re: [musl] building statically linked DSOs with musl

[Markus Wichmann <nullplan@gmx.net>]

On Sat, Jan 01, 2022 at 10:31:58PM +0800, Sebastien Bourdeauducq wrote:
> On 1/1/22 21:58, Alex Xu (Hello71) wrote:
> > First,
> > malloc implementations (may) use a process-global resource: (s)brk. It
> > will cause havoc if multiple implementations attempt to simultaneously
> > manipulate it. Second, even if no more than one implementation uses
> > (s)brk, they will still have incompatible metadata.
>
> Can't there be two different heaps in different memory regions?
>

There can be different heaps, but not different brk heaps. Basically,
brk() manages a global variable generated by the kernel. You can work
around that problem by installing a seccomp filter that makes brk()
always fail, forcing the allocators to fall back to mmap().

However, your approach has numerous other problems, especially with the
mismatch between glibc and musl. The thread pointer does not match up,
and so no functions accessing the thread pointers will work. That
includes functions that read the current thread pointer, so you cannot
know in advance which those are. All the implementation-defined
structures mismatch, so none of the locking functions work, and neither
do the semaphore functions. The thread-management functions also will
not work, and you are saying you have complex dependencies that might
use those.

No, the best you can probably do is put all the work into a separate
program. Define some sort of IPC interface, and have the plugin merely
serialize requests and read responses. Then your process can be as
complicated as you want and use whatever libc you fancy. It can even use
a custom malloc. And your plugin should be rather lean.

HTH,
Markus