dlsym returning unresolved symbol address instead of dependency library symbol address

[Luiz Angelo Daros de Luca <luizluca@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 2129 bytes --]

Hello,

I'm ruby maintainer in OpenWrt 18.06 (musl 1.1.19). I got a bug report (
https://github.com/openwrt/packages/issues/9297) related to musl in mipsel
32bit.

When ruby loads a module (.so), it checks if that module was built for the
same ruby that is loading it. Ruby loads libruby at startup, which exports
ruby_xmalloc sym. So, the check consists on loading the module, searching
for ruby_xmalloc in the module context and comparing with global
ruby_xmalloc address. If they do not match, the module is using a different
libruby. Something like this:

handle = (void*)dlopen(file, RTLD_LAZY|RTLD_GLOBAL)
void *ex = dlsym(handle, EXTERNAL_PREFIX"ruby_xmalloc");
if (ex && ex != ruby_xmalloc) {
   // module is incompatible!
}

The first time a module is loaded, it simply works as expected.
I debugged and musl is working nicely. At do_dlsym(struct dso *p, const
char *s, void *ra), it correctly fails to find the symbol with:

sym = sysv_lookup(s, h, p)

and correctly find it with:

sysv_lookup(s, h, p->deps[0])

Now, when the second module is loaded, it find "ruby_xmalloc" already with:

sym = sysv_lookup(s, h, p)

However, sym now points to the address of the undefined symbol in the
second library (sym->st_shndx is NULL) instead of searching for it in
dependencies. It seems that do_dlsym() only checks for undefined symbol
(sym->shndx==NULL) when DL_FDPIC is 1 and DL_FDPIC is 0 in my case.

Does it make any sense to return an undefined symbol from dlsym()?
Or does it make sense to return an undefined symbol from sysv_lookup()?
Or is there any other arch specific issue that happened before, when
library was loaded?

I created a simple patch that skips a symbol if it is undefined.
https://raw.githubusercontent.com/luizluca/openwrt/b9674d528513c7c93205fa000fed7c0d3c6bb2e7/toolchain/musl/patches/020-dlsym_donot_return_address_from_undef_sym.patch

It fixes the issue and it did not break my system(it still boots). However,
I didn't test it with multiples archs nor I runned an extensive test.

I'm not subscribed. Please, CC me.

Regards,
---
     Luiz Angelo Daros de Luca
            luizluca@gmail.com

[-- Attachment #2: Type: text/html, Size: 3110 bytes --]