Re: Problems with pthreads from a shared object?

[Rich Felker <dalias@libc.org>]

On Thu, Nov 08, 2018 at 12:14:39PM -0500, Barry Flartus wrote:
> >
> > I think it would be helpful to understand what you're trying to do.
> >
> 
> Thanks for your help so far. Let me try and explain what I'm trying to
> accomplish. I have a program that runs as an executable and uses pthreads.
> I've compiled this program with musl statically with the end goal of it
> being portable across older and newer systems. I want to also be able to
> compile this program as a shared object so it may be loaded via dlopen()
> inside of a glibc program. As mentioned previously, if I compile my shared
> object with glibc, it loads via dlopen().
> 
> My lack of understanding is this: if I directly compile in musl's libc.a
> (which contains its implementation of pthreads) into my shared object,
> shouldn't it have the relevant pthreads functions compiled in, without
> having runtime issues? That's what seems to work for my musl-compiled

It has the right code for being able to run in a process where the
(part of) libc inside your .so is the only libc in the process. It
does not have the right code for interoperating with a different
arbitrary libc in the same process where both of them rightly believe
themselves the sole owner of various process-global or thread-global
state.

> static executable, so I'm trying to wrap my head around why it wouldn't
> work for a shared object. In my case, I'm considering the shared object
> just a different form-factor for the same program.

Conceptually, the reasons it won't work are unbounded; any particular
list could become incomplete with newer versions of musl and/or glibc.

In practice, issues you would definitely face include but are not
limited to:

- Anything that depends on initialization at startup time will fail,
  as the musl __libc_start_main was never called.

- Aside from a few fields which are compiler ABI, the content/layout
  of the thread structure is an implementation detail subject to
  change even between versions, and will not match between musl and
  glibc. This matters even if the glibc code is not starting threads
  since the main thread will have a glibc-format thread structure
  pointed to by its thread pointer (%fs:0 on x86_64) and any glibc
  functions called from your library will expect glibc-format thread
  structures (even if the thread was started by the musl
  pthread_create).

- Assuming you didn't link with -Bsymbolic[-functions], calls to libc
  functions that were linked in your library may (will) get interposed
  by glibc functions already present, and the musl functions won't
  actually get called, except for the ones that don't exist in glibc
  (because they're in its librt or libpthread or something). And when
  some functions from musl which do end up getting called execute,
  they'll call a mix of glibc functions (when they reference something
  bu its public symbol) and musl functions (when they call
  musl-internal interfaces). These will be expecting/operaring on
  different and incompatible data structures.

- If you did link with -Bsymbolic[-functions], you'd end up calling
  musl's malloc in the same process as glibc's malloc and they'd be
  fighting over ownership of the brk. Things would also blow up badly
  if a pointer returned by one's malloc was passed to the other's
  realloc/free at some point.

- Many more things...

Ultimately, it's *always* unsafe to have two libraries with
conflicting interfaces linked into the same program, and even moreso
when they're something as central (with ownership of central global
state) as libc.

Rich