Problems with pthreads from a shared object?

Problems with pthreads from a shared object?

[Barry Flartus]

[-- Attachment #1: Type: text/plain, Size: 2158 bytes --]

Hello,

I am encountering some issues that I believe are musl related when trying to
use pthreads from a shared object. Below is a minimal example (2 files) of
the problem.

==================== test.c ====================

// Compile with
// musl-gcc -Wall -lpthread -shared -fPIC test.c -o test.so

#include <pthread.h>
#include <stdio.h>

static void *launch(void *arg);
int start_thread(void);
void entry_point(void) __attribute__((constructor));

static pthread_t gthread;

static void *launch(void *arg)
{
    for (int i = 0; i < 4; i++){
        printf("[%d] Hello from the thread\n", i);
    }
    return NULL;
}

int start_thread(void)
{
    printf("Starting the thread\n");
    return pthread_create(&gthread, NULL, launch, NULL);
}

void entry_point(void){
    puts("Starting Execution");
    int res = start_thread();
    printf("start_thread returned %d\n", res);
    if (res == 0){
        pthread_join(gthread, NULL);
    } else {
        printf("pthread_create() returned an error. Aborting.\n");
    }
}

==================== launch.c ====================

// Compile with:
// gcc launch.c -ldl -o launch

#include <dlfcn.h>
#include <stdio.h>
#include <stdlib.h>

int main(){
    void *handle;

    puts("\nLaunching test...");
    handle = dlopen("./test.so", RTLD_NOW);
    if (!handle) {
        fprintf(stderr, "%s\n", dlerror());
        exit(EXIT_FAILURE);
    }
    dlerror();

    return EXIT_SUCCESS;
}

When the above examples are compiled with gcc, everything works correctly
and
produces the following output:

Launching test...
Starting Execution
Starting the threadstart_thread returned 0
[0] Hello from the thread
[1] Hello from the thread
[2] Hello from the thread
[3] Hello from the thread

When I switch to musl pthread_create() returns the integer '38', which is an
undocumented return value. I assume this corresponds with ENOSYS from
errno.h
but I can't seem to figure out why this is happening.

Output:

Launching test...
Starting Execution
Starting the thread
start_thread returned 38
pthread_create() returned an error. Aborting.

Is this even supported by musl? If so, where have I gone wrong?

Thanks in advance!

[-- Attachment #2: Type: text/html, Size: 7316 bytes --]

Re: Problems with pthreads from a shared object?

[Rich Felker]

On Wed, Nov 07, 2018 at 06:00:14PM -0500, Barry Flartus wrote:
> Hello,
> 
> I am encountering some issues that I believe are musl related when trying to
> use pthreads from a shared object. Below is a minimal example (2 files) of
> the problem.
> 
> ==================== test.c ====================
> 
> // Compile with
> // musl-gcc -Wall -lpthread -shared -fPIC test.c -o test.so
> 
> #include <pthread.h>
> #include <stdio.h>
> 
> static void *launch(void *arg);
> int start_thread(void);
> void entry_point(void) __attribute__((constructor));
> 
> static pthread_t gthread;
> 
> static void *launch(void *arg)
> {
>     for (int i = 0; i < 4; i++){
>         printf("[%d] Hello from the thread\n", i);
>     }
>     return NULL;
> }
> 
> int start_thread(void)
> {
>     printf("Starting the thread\n");
>     return pthread_create(&gthread, NULL, launch, NULL);
> }
> 
> void entry_point(void){
>     puts("Starting Execution");
>     int res = start_thread();
>     printf("start_thread returned %d\n", res);
>     if (res == 0){
>         pthread_join(gthread, NULL);
>     } else {
>         printf("pthread_create() returned an error. Aborting.\n");
>     }
> }
> 
> ==================== launch.c ====================
> 
> // Compile with:
> // gcc launch.c -ldl -o launch
> 
> #include <dlfcn.h>
> #include <stdio.h>
> #include <stdlib.h>
> 
> int main(){
>     void *handle;
> 
>     puts("\nLaunching test...");
>     handle = dlopen("./test.so", RTLD_NOW);
>     if (!handle) {
>         fprintf(stderr, "%s\n", dlerror());
>         exit(EXIT_FAILURE);
>     }
>     dlerror();
> 
>     return EXIT_SUCCESS;
> }
> 
> When the above examples are compiled with gcc, everything works correctly
> and
> produces the following output:
> 
> Launching test...
> Starting Execution
> Starting the threadstart_thread returned 0
> [0] Hello from the thread
> [1] Hello from the thread
> [2] Hello from the thread
> [3] Hello from the thread
> 
> When I switch to musl pthread_create() returns the integer '38', which is an
> undocumented return value. I assume this corresponds with ENOSYS from
> errno.h
> but I can't seem to figure out why this is happening.
> 
> Output:
> 
> Launching test...
> Starting Execution
> Starting the thread
> start_thread returned 38
> pthread_create() returned an error. Aborting.
> 
> Is this even supported by musl? If so, where have I gone wrong?
> 
> Thanks in advance!

Are you running on an ancient kernel or a kernel built with a lot of
default functionality configured out? These should be the only
conditions under which pthread_create can return ENOSYS.

Rich

Re: Problems with pthreads from a shared object?

[Barry Flartus]

[-- Attachment #1: Type: text/plain, Size: 2974 bytes --]

Nope! Linux ubuntu-x64 4.4.0-116-generic -  this is just the kernel from a
default Ubuntu 16.04 install.

On Wed, Nov 7, 2018 at 6:31 PM Rich Felker <dalias@libc.org> wrote:

> On Wed, Nov 07, 2018 at 06:00:14PM -0500, Barry Flartus wrote:
> > Hello,
> >
> > I am encountering some issues that I believe are musl related when
> trying to
> > use pthreads from a shared object. Below is a minimal example (2 files)
> of
> > the problem.
> >
> > ==================== test.c ====================
> >
> > // Compile with
> > // musl-gcc -Wall -lpthread -shared -fPIC test.c -o test.so
> >
> > #include <pthread.h>
> > #include <stdio.h>
> >
> > static void *launch(void *arg);
> > int start_thread(void);
> > void entry_point(void) __attribute__((constructor));
> >
> > static pthread_t gthread;
> >
> > static void *launch(void *arg)
> > {
> >     for (int i = 0; i < 4; i++){
> >         printf("[%d] Hello from the thread\n", i);
> >     }
> >     return NULL;
> > }
> >
> > int start_thread(void)
> > {
> >     printf("Starting the thread\n");
> >     return pthread_create(&gthread, NULL, launch, NULL);
> > }
> >
> > void entry_point(void){
> >     puts("Starting Execution");
> >     int res = start_thread();
> >     printf("start_thread returned %d\n", res);
> >     if (res == 0){
> >         pthread_join(gthread, NULL);
> >     } else {
> >         printf("pthread_create() returned an error. Aborting.\n");
> >     }
> > }
> >
> > ==================== launch.c ====================
> >
> > // Compile with:
> > // gcc launch.c -ldl -o launch
> >
> > #include <dlfcn.h>
> > #include <stdio.h>
> > #include <stdlib.h>
> >
> > int main(){
> >     void *handle;
> >
> >     puts("\nLaunching test...");
> >     handle = dlopen("./test.so", RTLD_NOW);
> >     if (!handle) {
> >         fprintf(stderr, "%s\n", dlerror());
> >         exit(EXIT_FAILURE);
> >     }
> >     dlerror();
> >
> >     return EXIT_SUCCESS;
> > }
> >
> > When the above examples are compiled with gcc, everything works correctly
> > and
> > produces the following output:
> >
> > Launching test...
> > Starting Execution
> > Starting the threadstart_thread returned 0
> > [0] Hello from the thread
> > [1] Hello from the thread
> > [2] Hello from the thread
> > [3] Hello from the thread
> >
> > When I switch to musl pthread_create() returns the integer '38', which
> is an
> > undocumented return value. I assume this corresponds with ENOSYS from
> > errno.h
> > but I can't seem to figure out why this is happening.
> >
> > Output:
> >
> > Launching test...
> > Starting Execution
> > Starting the thread
> > start_thread returned 38
> > pthread_create() returned an error. Aborting.
> >
> > Is this even supported by musl? If so, where have I gone wrong?
> >
> > Thanks in advance!
>
> Are you running on an ancient kernel or a kernel built with a lot of
> default functionality configured out? These should be the only
> conditions under which pthread_create can return ENOSYS.
>
> Rich
>

[-- Attachment #2: Type: text/html, Size: 4055 bytes --]

Re: Problems with pthreads from a shared object?

[Rich Felker]

On Wed, Nov 07, 2018 at 06:40:35PM -0500, Barry Flartus wrote:
> Nope! Linux ubuntu-x64 4.4.0-116-generic -  this is just the kernel from a
> default Ubuntu 16.04 install.

OK. Can you run strace -f ./launch (or whatever the launch program
executable is called) and post the output?

Rich


> On Wed, Nov 7, 2018 at 6:31 PM Rich Felker <dalias@libc.org> wrote:
> 
> > On Wed, Nov 07, 2018 at 06:00:14PM -0500, Barry Flartus wrote:
> > > Hello,
> > >
> > > I am encountering some issues that I believe are musl related when
> > trying to
> > > use pthreads from a shared object. Below is a minimal example (2 files)
> > of
> > > the problem.
> > >
> > > ==================== test.c ====================
> > >
> > > // Compile with
> > > // musl-gcc -Wall -lpthread -shared -fPIC test.c -o test.so
> > >
> > > #include <pthread.h>
> > > #include <stdio.h>
> > >
> > > static void *launch(void *arg);
> > > int start_thread(void);
> > > void entry_point(void) __attribute__((constructor));
> > >
> > > static pthread_t gthread;
> > >
> > > static void *launch(void *arg)
> > > {
> > >     for (int i = 0; i < 4; i++){
> > >         printf("[%d] Hello from the thread\n", i);
> > >     }
> > >     return NULL;
> > > }
> > >
> > > int start_thread(void)
> > > {
> > >     printf("Starting the thread\n");
> > >     return pthread_create(&gthread, NULL, launch, NULL);
> > > }
> > >
> > > void entry_point(void){
> > >     puts("Starting Execution");
> > >     int res = start_thread();
> > >     printf("start_thread returned %d\n", res);
> > >     if (res == 0){
> > >         pthread_join(gthread, NULL);
> > >     } else {
> > >         printf("pthread_create() returned an error. Aborting.\n");
> > >     }
> > > }
> > >
> > > ==================== launch.c ====================
> > >
> > > // Compile with:
> > > // gcc launch.c -ldl -o launch
> > >
> > > #include <dlfcn.h>
> > > #include <stdio.h>
> > > #include <stdlib.h>
> > >
> > > int main(){
> > >     void *handle;
> > >
> > >     puts("\nLaunching test...");
> > >     handle = dlopen("./test.so", RTLD_NOW);
> > >     if (!handle) {
> > >         fprintf(stderr, "%s\n", dlerror());
> > >         exit(EXIT_FAILURE);
> > >     }
> > >     dlerror();
> > >
> > >     return EXIT_SUCCESS;
> > > }
> > >
> > > When the above examples are compiled with gcc, everything works correctly
> > > and
> > > produces the following output:
> > >
> > > Launching test...
> > > Starting Execution
> > > Starting the threadstart_thread returned 0
> > > [0] Hello from the thread
> > > [1] Hello from the thread
> > > [2] Hello from the thread
> > > [3] Hello from the thread
> > >
> > > When I switch to musl pthread_create() returns the integer '38', which
> > is an
> > > undocumented return value. I assume this corresponds with ENOSYS from
> > > errno.h
> > > but I can't seem to figure out why this is happening.
> > >
> > > Output:
> > >
> > > Launching test...
> > > Starting Execution
> > > Starting the thread
> > > start_thread returned 38
> > > pthread_create() returned an error. Aborting.
> > >
> > > Is this even supported by musl? If so, where have I gone wrong?
> > >
> > > Thanks in advance!
> >
> > Are you running on an ancient kernel or a kernel built with a lot of
> > default functionality configured out? These should be the only
> > conditions under which pthread_create can return ENOSYS.
> >
> > Rich
> >

Re: Problems with pthreads from a shared object?

[Barry Flartus]

[-- Attachment #1: Type: text/plain, Size: 4268 bytes --]

>
> OK. Can you run strace -f ./launch (or whatever the launch program
> executable is called) and post the output?
>
> Rich
>

Here you go!

$ strace -f ./launch
execve("./launch", ["./launch"], [/* 22 vars */]) = 0
brk(NULL)                               = 0xe44000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or
directory)
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or
directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=42286, ...}) = 0
mmap(NULL, 42286, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fcda3aa9000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or
directory)
open("/lib/x86_64-linux-gnu/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3,
"\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240\r\0\0\0\0\0\0"..., 832)
= 832
fstat(3, {st_mode=S_IFREG|0644, st_size=14608, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7fcda3aa8000
mmap(NULL, 2109680, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) =
0x7fcda368b000
mprotect(0x7fcda368e000, 2093056, PROT_NONE) = 0
mmap(0x7fcda388d000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7fcda388d000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or
directory)
open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P\t\2\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1868984, ...}) = 0
mmap(NULL, 3971488, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) =
0x7fcda32c1000
mprotect(0x7fcda3481000, 2097152, PROT_NONE) = 0
mmap(0x7fcda3681000, 24576, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1c0000) = 0x7fcda3681000
mmap(0x7fcda3687000, 14752, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fcda3687000
close(3)                                = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7fcda3aa7000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7fcda3aa6000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7fcda3aa5000
arch_prctl(ARCH_SET_FS, 0x7fcda3aa6700) = 0
mprotect(0x7fcda3681000, 16384, PROT_READ) = 0
mprotect(0x7fcda388d000, 4096, PROT_READ) = 0
mprotect(0x600000, 4096, PROT_READ)     = 0
mprotect(0x7fcda3ab4000, 4096, PROT_READ) = 0
munmap(0x7fcda3aa9000, 42286)           = 0
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 2), ...}) = 0
brk(NULL)                               = 0xe44000
brk(0xe65000)                           = 0xe65000
write(1, "\n", 1
)                       = 1
write(1, "Launching test...\n", 18Launching test...
)     = 18
open("./test.so", O_RDONLY|O_CLOEXEC)   = 3
read(3,
"\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240!\0\0\0\0\0\0"..., 832)
= 832
fstat(3, {st_mode=S_IFREG|0755, st_size=40152, ...}) = 0
getcwd("/vagrant/test/wtf/public", 128) = 25
mmap(NULL, 2128504, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) =
0x7fcda30b9000
mprotect(0x7fcda30c0000, 2093056, PROT_NONE) = 0
mmap(0x7fcda32bf000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6000) = 0x7fcda32bf000
close(3)                                = 0
mprotect(0x7fcda32bf000, 4096, PROT_READ) = 0
write(1, "Starting Execution\n", 19Starting Execution
)    = 19
write(1, "Starting the thread\n", 20Starting the thread
)   = 20
write(1, "start_thread returned 38\n", 25start_thread returned 38
) = 25
write(1, "pthread_create() returned an err"..., 46pthread_create() returned
an error. Aborting.
) = 46
exit_group(0)                           = ?
+++ exited with 0 +++

$ ltrace -f ./launch
[pid 13508] __libc_start_main(0x400746, 1, 0x7ffff030bbc8, 0x4007b0
<unfinished ...>
[pid 13508] puts("\nLaunching test..."
Launching test...
)
                                     = 19
[pid 13508] dlopen("./test.so", 2Starting Execution
Starting the thread
start_thread returned 38
pthread_create() returned an error. Aborting.
)
                                          = 0x12d6440
[pid 13508] dlerror()

 = nil
[pid 13508] +++ exited (status 0) +++

[-- Attachment #2: Type: text/html, Size: 6098 bytes --]

Re: Problems with pthreads from a shared object?

[Rich Felker]

On Wed, Nov 07, 2018 at 06:53:33PM -0500, Barry Flartus wrote:
> >
> > OK. Can you run strace -f ./launch (or whatever the launch program
> > executable is called) and post the output?
> >
> > Rich
> >
> 
> Here you go!
> 
> $ strace -f ./launch
> execve("./launch", ["./launch"], [/* 22 vars */]) = 0
> brk(NULL)                               = 0xe44000
> access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or

OK, this is a glibc program, not a musl one. That's your problem. How
did you compile/link it?

Rich

Re: Problems with pthreads from a shared object?

[Barry Flartus]

[-- Attachment #1: Type: text/plain, Size: 750 bytes --]

On Wed, Nov 7, 2018 at 7:43 PM Rich Felker <dalias@libc.org> wrote:

> On Wed, Nov 07, 2018 at 06:53:33PM -0500, Barry Flartus wrote:
> > >
> > > OK. Can you run strace -f ./launch (or whatever the launch program
> > > executable is called) and post the output?
> > >
> > > Rich
> > >
>
>
The "launching" program was compiled with gcc:
gcc launch.c -ldl -o launch

When I attempt to compile the "launching" program with musl-gcc I get the
following output:
vagrant@ubuntu-x64:~$ /usr/local/musl/x86_64/bin/musl-gcc launch.c -ldl -o
launch
vagrant@ubuntu-x64:~$ ./launch

Launching test...
Dynamic loading not supported

The shared object was compiled with musl-gcc:
/usr/local/musl/x86_64/bin/musl-gcc -Wall -lpthread -shared -fPIC test.c -o
test.so

[-- Attachment #2: Type: text/html, Size: 1266 bytes --]

Re: Problems with pthreads from a shared object?

[Rich Felker]

On Wed, Nov 07, 2018 at 08:02:01PM -0500, Barry Flartus wrote:
> On Wed, Nov 7, 2018 at 7:43 PM Rich Felker <dalias@libc.org> wrote:
> 
> > On Wed, Nov 07, 2018 at 06:53:33PM -0500, Barry Flartus wrote:
> > > >
> > > > OK. Can you run strace -f ./launch (or whatever the launch program
> > > > executable is called) and post the output?
> > > >
> > > > Rich
> > > >
> >
> >
> The "launching" program was compiled with gcc:
> gcc launch.c -ldl -o launch

You can't use shared libraries built for musl with glibc, so it's
expected that this doesn't work.

> When I attempt to compile the "launching" program with musl-gcc I get the
> following output:
> vagrant@ubuntu-x64:~$ /usr/local/musl/x86_64/bin/musl-gcc launch.c -ldl -o
> launch
> vagrant@ubuntu-x64:~$ ./launch
> 
> Launching test...
> Dynamic loading not supported

Apparently it's static-linked. Did you omit shared libraries when
building musl?

> The shared object was compiled with musl-gcc:
> /usr/local/musl/x86_64/bin/musl-gcc -Wall -lpthread -shared -fPIC test.c -o
> test.so

If so (if there's no libc.so, only libc.a, present) then this command
will also be producing a broken .so file. You can't make shared
libraries without a shared libc existing.

Rich

Re: Problems with pthreads from a shared object?

[Jeffrey Walton]

On Wed, Nov 7, 2018 at 7:43 PM Rich Felker <dalias@libc.org> wrote:
>
> On Wed, Nov 07, 2018 at 06:53:33PM -0500, Barry Flartus wrote:
> > >
> > > OK. Can you run strace -f ./launch (or whatever the launch program
> > > executable is called) and post the output?
> > > ...
> >
> > $ strace -f ./launch
> > execve("./launch", ["./launch"], [/* 22 vars */]) = 0
> > brk(NULL)                               = 0xe44000
> > access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or
>
> OK, this is a glibc program, not a musl one. That's your problem. How
> did you compile/link it?

Sorry to butt-in.

Alpine Linux uses musl. BF won't need anything special to use or
exercise musl (and certainly won't need to worry about
cross-pollination).

Jeff

Re: Problems with pthreads from a shared object?

[Barry Flartus]

[-- Attachment #1: Type: text/plain, Size: 316 bytes --]

>
> Sorry to butt-in.
>
> Alpine Linux uses musl. BF won't need anything special to use or
> exercise musl (and certainly won't need to worry about
> cross-pollination).
>
> Jeff
>

Jeff, no worries about butting in. Could you clarify your statement?
Im not quite sure I understand your cross-pollination analogy :)

[-- Attachment #2: Type: text/html, Size: 539 bytes --]

Re: Problems with pthreads from a shared object?

[Rich Felker]

On Thu, Nov 08, 2018 at 09:53:38AM -0500, Barry Flartus wrote:
> >
> > Sorry to butt-in.
> >
> > Alpine Linux uses musl. BF won't need anything special to use or
> > exercise musl (and certainly won't need to worry about
> > cross-pollination).
> >
> 
> Jeff, no worries about butting in. Could you clarify your statement?
> Im not quite sure I understand your cross-pollination analogy :)

I think it would be helpful to understand what you're trying to do.

If your goal is just ensuring your application is compatible with musl
so that it can be built and used on musl-based systems/distros, using
one (such as Alpine, or Adelie, or Void with musl, or Sabotage, etc.),
possibly in a container, is going to be the easiest way to achieve
that.

If your goal is building an application with musl and dlopen'd modules
that you can deploy to systems that might or might not be musl-based,
this can be done, but requires using dynamic linking and some glue for
explicitly running the dynamic linker, since it can't be found via a
fixed absolute path the way 'native' dynamic linking works.

If your goal is to get around the glibc limitation that loading
libpthread dynamically doesn't work, trying to use musl to produce .so
files that use threads that glibc-linked programs can load, this
simply does not, and cannot, work. A much better simpler solution is
just ensuring the main program always links with -lpthread when using
glibc.

Rich

Re: Problems with pthreads from a shared object?

[Barry Flartus]

[-- Attachment #1: Type: text/plain, Size: 1037 bytes --]

>
> I think it would be helpful to understand what you're trying to do.
>

Thanks for your help so far. Let me try and explain what I'm trying to
accomplish. I have a program that runs as an executable and uses pthreads.
I've compiled this program with musl statically with the end goal of it
being portable across older and newer systems. I want to also be able to
compile this program as a shared object so it may be loaded via dlopen()
inside of a glibc program. As mentioned previously, if I compile my shared
object with glibc, it loads via dlopen().

My lack of understanding is this: if I directly compile in musl's libc.a
(which contains its implementation of pthreads) into my shared object,
shouldn't it have the relevant pthreads functions compiled in, without
having runtime issues? That's what seems to work for my musl-compiled
static executable, so I'm trying to wrap my head around why it wouldn't
work for a shared object. In my case, I'm considering the shared object
just a different form-factor for the same program.

[-- Attachment #2: Type: text/html, Size: 1447 bytes --]

Re: Problems with pthreads from a shared object?

[Rich Felker]

On Thu, Nov 08, 2018 at 12:14:39PM -0500, Barry Flartus wrote:
> >
> > I think it would be helpful to understand what you're trying to do.
> >
> 
> Thanks for your help so far. Let me try and explain what I'm trying to
> accomplish. I have a program that runs as an executable and uses pthreads.
> I've compiled this program with musl statically with the end goal of it
> being portable across older and newer systems. I want to also be able to
> compile this program as a shared object so it may be loaded via dlopen()
> inside of a glibc program. As mentioned previously, if I compile my shared
> object with glibc, it loads via dlopen().
> 
> My lack of understanding is this: if I directly compile in musl's libc.a
> (which contains its implementation of pthreads) into my shared object,
> shouldn't it have the relevant pthreads functions compiled in, without
> having runtime issues? That's what seems to work for my musl-compiled

It has the right code for being able to run in a process where the
(part of) libc inside your .so is the only libc in the process. It
does not have the right code for interoperating with a different
arbitrary libc in the same process where both of them rightly believe
themselves the sole owner of various process-global or thread-global
state.

> static executable, so I'm trying to wrap my head around why it wouldn't
> work for a shared object. In my case, I'm considering the shared object
> just a different form-factor for the same program.

Conceptually, the reasons it won't work are unbounded; any particular
list could become incomplete with newer versions of musl and/or glibc.

In practice, issues you would definitely face include but are not
limited to:

- Anything that depends on initialization at startup time will fail,
  as the musl __libc_start_main was never called.

- Aside from a few fields which are compiler ABI, the content/layout
  of the thread structure is an implementation detail subject to
  change even between versions, and will not match between musl and
  glibc. This matters even if the glibc code is not starting threads
  since the main thread will have a glibc-format thread structure
  pointed to by its thread pointer (%fs:0 on x86_64) and any glibc
  functions called from your library will expect glibc-format thread
  structures (even if the thread was started by the musl
  pthread_create).

- Assuming you didn't link with -Bsymbolic[-functions], calls to libc
  functions that were linked in your library may (will) get interposed
  by glibc functions already present, and the musl functions won't
  actually get called, except for the ones that don't exist in glibc
  (because they're in its librt or libpthread or something). And when
  some functions from musl which do end up getting called execute,
  they'll call a mix of glibc functions (when they reference something
  bu its public symbol) and musl functions (when they call
  musl-internal interfaces). These will be expecting/operaring on
  different and incompatible data structures.

- If you did link with -Bsymbolic[-functions], you'd end up calling
  musl's malloc in the same process as glibc's malloc and they'd be
  fighting over ownership of the brk. Things would also blow up badly
  if a pointer returned by one's malloc was passed to the other's
  realloc/free at some point.

- Many more things...

Ultimately, it's *always* unsafe to have two libraries with
conflicting interfaces linked into the same program, and even moreso
when they're something as central (with ownership of central global
state) as libc.

Rich

Re: Problems with pthreads from a shared object?

[Barry Flartus]

[-- Attachment #1: Type: text/plain, Size: 4128 bytes --]

On Thu, Nov 8, 2018 at 12:37 PM Rich Felker <dalias@libc.org> wrote:

> On Thu, Nov 08, 2018 at 12:14:39PM -0500, Barry Flartus wrote:
> > >
> > > I think it would be helpful to understand what you're trying to do.
> > >
> >
> > Thanks for your help so far. Let me try and explain what I'm trying to
> > accomplish. I have a program that runs as an executable and uses
> pthreads.
> > I've compiled this program with musl statically with the end goal of it
> > being portable across older and newer systems. I want to also be able to
> > compile this program as a shared object so it may be loaded via dlopen()
> > inside of a glibc program. As mentioned previously, if I compile my
> shared
> > object with glibc, it loads via dlopen().
> >
> > My lack of understanding is this: if I directly compile in musl's libc.a
> > (which contains its implementation of pthreads) into my shared object,
> > shouldn't it have the relevant pthreads functions compiled in, without
> > having runtime issues? That's what seems to work for my musl-compiled
>
> It has the right code for being able to run in a process where the
> (part of) libc inside your .so is the only libc in the process. It
> does not have the right code for interoperating with a different
> arbitrary libc in the same process where both of them rightly believe
> themselves the sole owner of various process-global or thread-global
> state.
>
> > static executable, so I'm trying to wrap my head around why it wouldn't
> > work for a shared object. In my case, I'm considering the shared object
> > just a different form-factor for the same program.
>
> Conceptually, the reasons it won't work are unbounded; any particular
> list could become incomplete with newer versions of musl and/or glibc.
>
> In practice, issues you would definitely face include but are not
> limited to:
>
> - Anything that depends on initialization at startup time will fail,
>   as the musl __libc_start_main was never called.
>
> - Aside from a few fields which are compiler ABI, the content/layout
>   of the thread structure is an implementation detail subject to
>   change even between versions, and will not match between musl and
>   glibc. This matters even if the glibc code is not starting threads
>   since the main thread will have a glibc-format thread structure
>   pointed to by its thread pointer (%fs:0 on x86_64) and any glibc
>   functions called from your library will expect glibc-format thread
>   structures (even if the thread was started by the musl
>   pthread_create).
>
> - Assuming you didn't link with -Bsymbolic[-functions], calls to libc
>   functions that were linked in your library may (will) get interposed
>   by glibc functions already present, and the musl functions won't
>   actually get called, except for the ones that don't exist in glibc
>   (because they're in its librt or libpthread or something). And when
>   some functions from musl which do end up getting called execute,
>   they'll call a mix of glibc functions (when they reference something
>   bu its public symbol) and musl functions (when they call
>   musl-internal interfaces). These will be expecting/operaring on
>   different and incompatible data structures.
>
> - If you did link with -Bsymbolic[-functions], you'd end up calling
>   musl's malloc in the same process as glibc's malloc and they'd be
>   fighting over ownership of the brk. Things would also blow up badly
>   if a pointer returned by one's malloc was passed to the other's
>   realloc/free at some point.
>
> - Many more things...
>
> Ultimately, it's *always* unsafe to have two libraries with
> conflicting interfaces linked into the same program, and even moreso
> when they're something as central (with ownership of central global
> state) as libc.
>
> Rich
>

Rich, thank you so much for taking the time to explain this to me!
Ultimately
my problems stem from my lack knowledge in this area, so you really have
taught me so much and I am very appreciative of that.

Additionally, thank you for your work on musl, it really is fantastic! I
will
continue to use it in the correct manner.

Best,
Barry

[-- Attachment #2: Type: text/html, Size: 4842 bytes --]