dlopen'ing glibc linked libraries

dlopen'ing glibc linked libraries

[Gabriel Jacobo]

[-- Attachment #1: Type: text/plain, Size: 855 bytes --]

Hey everyone! First message here...

I recently found about musl when googling for GLIBC alternatives. I read in
the FAQ that binary compatibility with GLIBC is not there yet, so my
question is if this is a priority for the library.

As a bit of background to my question, let me say that I'm an SDL2
contributor, and I've tried linking SDL against musl, which seemed to work
great. I'm linking dynamically BTW (statically linking to musl didn't quite
work, I didn't research the issue much but it's probably a problem in SDL's
build system).

However, SDL uses dlopen/dlsym heavily to load most of its functionality
(for example, to load the libGL.so.1 binary in order to provide OpenGL
support), and these binaries depend on GLIBC...so (expectedly?) almost
nothing actually works with the SDL/musl binary.

So, will it ever work?

Thanks!

-- 
Gabriel.

[-- Attachment #2: Type: text/html, Size: 1067 bytes --]

Re: dlopen'ing glibc linked libraries

[John Spencer]

Gabriel Jacobo wrote:
> Hey everyone! First message here...
> 
> I recently found about musl when googling for GLIBC alternatives. I read in
> the FAQ that binary compatibility with GLIBC is not there yet, so my
> question is if this is a priority for the library.

GLIBC compat is already pretty good, given that the glibc stuff was 
compiled in a sane manner (i.e. with the flags that enable 64bit off_t).
for example, it is possible to run Dwarf Fortress:
http://openwall.com/lists/sabotage/2013/11/01/1

it's also possible to run opera with a preloaded .so that provides a few
symbols that musl doesn't provide.

> 
> As a bit of background to my question, let me say that I'm an SDL2
> contributor, and I've tried linking SDL against musl, which seemed to work
> great. I'm linking dynamically BTW (statically linking to musl didn't quite
> work, I didn't research the issue much but it's probably a problem in SDL's
> build system).

statically linked musl does not support dlopen(), so you will have a 
hard time loading plugins (explanation for why it is not supported:
http://www.openwall.com/lists/musl/2012/12/08/4 )

> 
> However, SDL uses dlopen/dlsym heavily to load most of its functionality

i guess this kind of plug-in loading is considered a bad idea by most 
people here...

> (for example, to load the libGL.so.1 binary in order to provide OpenGL
> support), and these binaries depend on GLIBC...so (expectedly?) almost

that's not quite true, sabotage linux builds mesa fine (with 2 minor 
patches).
recipe:
https://github.com/sabotage-linux/sabotage/blob/master/pkg/mesalib#L19
patches:
https://github.com/sabotage-linux/sabotage/blob/master/KEEP/mesalib-fpclassify.patch
https://github.com/sabotage-linux/sabotage/blob/master/KEEP/mesalib-strtod.patch
https://github.com/sabotage-linux/sabotage/blob/master/KEEP/mesalib-strtof.patch

> nothing actually works with the SDL/musl binary.

basically what you should try to do is build all dependencies against musl.

> So, will it ever work?

even if it would work, mixing glibc and musl linked things is far from 
optimal.

> 
> Thanks!

btw, you might be interested in this SDL bug:
https://bugzilla.libsdl.org/show_bug.cgi?id=2019

Re: dlopen'ing glibc linked libraries

[Gabriel Jacobo]

[-- Attachment #1: Type: text/plain, Size: 1277 bytes --]

>
> that's not quite true, sabotage linux builds mesa fine (with 2 minor
> patches).
> recipe:
> https://github.com/sabotage-linux/sabotage/blob/master/pkg/mesalib#L19
> patches:
> https://github.com/sabotage-linux/sabotage/blob/master/
> KEEP/mesalib-fpclassify.patch
> https://github.com/sabotage-linux/sabotage/blob/master/
> KEEP/mesalib-strtod.patch
> https://github.com/sabotage-linux/sabotage/blob/master/
> KEEP/mesalib-strtof.patch
>
>  nothing actually works with the SDL/musl binary.
>>
>
> basically what you should try to do is build all dependencies against musl.
>
>  So, will it ever work?
>>
>
> even if it would work, mixing glibc and musl linked things is far from
> optimal.
>
>
>>
Thanks for the response. Let me express again that my experiment assumes
there's binaries that you just can't rebuild against musl (nVidia binaries
for example) or that's not practical to do so (like every binary provided
by Ubuntu ;) ), and that's the fringe case that interests me the most right
now.
I would assume that if you rebuild all libraries against musl (or use SDL
in a distro that's based on musl such as Sabotage), things would just work.
But my question was oriented towards what's the goal for providing "full"
binary compatibility with glibc.


-- 
Gabriel.

[-- Attachment #2: Type: text/html, Size: 2473 bytes --]

Re: dlopen'ing glibc linked libraries

[John Spencer]

Gabriel Jacobo wrote:
>> that's not quite true, sabotage linux builds mesa fine (with 2 minor
>> patches).
>> recipe:
>> https://github.com/sabotage-linux/sabotage/blob/master/pkg/mesalib#L19
>> patches:
>> https://github.com/sabotage-linux/sabotage/blob/master/
>> KEEP/mesalib-fpclassify.patch
>> https://github.com/sabotage-linux/sabotage/blob/master/
>> KEEP/mesalib-strtod.patch
>> https://github.com/sabotage-linux/sabotage/blob/master/
>> KEEP/mesalib-strtof.patch
>>
>>  nothing actually works with the SDL/musl binary.
>> basically what you should try to do is build all dependencies against musl.
>>
>>  So, will it ever work?
>> even if it would work, mixing glibc and musl linked things is far from
>> optimal.
>>
>>
> Thanks for the response. Let me express again that my experiment assumes
> there's binaries that you just can't rebuild against musl (nVidia binaries
> for example) or that's not practical to do so (like every binary provided
> by Ubuntu ;) ), and that's the fringe case that interests me the most right
> now.

OK, but in that case it's much saner if everything else is compiled 
against musl, and the nvidia binary the only impurity.
more binaries == more crazy glibc apis that could be willingly or 
unwillingy getting pulled in

> I would assume that if you rebuild all libraries against musl (or use SDL
> in a distro that's based on musl such as Sabotage), things would just work.

correct.

> But my question was oriented towards what's the goal for providing "full"
> binary compatibility with glibc.

i dont think we'll ever have 100% compatibility, since there are a lot 
of situations where the interfaces implemented by GLIBC are, to a 
varying degree, non-conformant, so things could fail at runtime:

if software uses crazy glibc extensions in printf format specifiers:
http://git.alsa-project.org/?p=alsa-lib.git;a=commitdiff;h=1d3f7975f920f47e6a8a324f547da2180e64171a
or the pthread stack size issue (SDL bug mentioned earlier).

there's also strerror_r, for which the default symbol uses the broken 
glibc semantics, whereas the POSIX-conformant version is called 
__xpg_strerror_r in glibc (users of strerror_r get redirected there, 
unless they #define _GNU_SOURCE)...
additionally there are some interfaces like qsort_r where different 
implementations exist (the earlier BSD version and the GNU version have 
the argument order swapped), so implementing one of the 2 versions in 
musl could lead to silent breakage when a configure check tests for 
qsort_r, finds and uses it, but assumes different behaviour.

lastly, glibc tends to pull in a lot of implementation-internal __ 
prefixed functions, for example when some macros like _FORTIFY_SOURCE 
are defined.

so in order to be 100% compatible, we'd have to turn musl into a full 
glibc clone, with all of its bugs, bloat, (mis)features, and throw POSIX 
conformance over board.

however we're constantly adding legacy APIs that we see in usage, if 
they're not extremely ugly or intrusive, so the compatibility improves 
with every release.
so it's well possible that the compatibility is good enough for a single 
piece of sw, like nvidia drivers, but not for some other libraries it 
wants to link against. in that case it's safe to hand it musl-linked 
versions of those libraries instead.

Re: dlopen'ing glibc linked libraries

[Szabolcs Nagy]

* Gabriel Jacobo <gabomdq@gmail.com> [2014-01-21 12:44:14 -0200]:
> But my question was oriented towards what's the goal for providing "full"
> binary compatibility with glibc.

no, full bug compatibility is not a goal (that would defeat the
purpose of a new libc)

that said, most apis are binary compatible with glibc (even for c++)

if you see a missing symbol or other breakage then please report it
here is a list of known issues:

http://wiki.musl-libc.org/wiki/Functional_differences_from_glibc

it may be possible to solve a lot of compatibility issues with
preloading an extra lib next to musl, i experimented with running
applications with

export LD_LIBRARY_PATH=/usr/lib:/usr/lib/i386-linux-gnu:/lib/i386-linux-gnu
export LD_PRELOAD=./libcompat.so
/lib/ld-musl-i386.so /usr/bin/something

on a debian system and a lot of things work (except the loader
crashes occasionally due to a kernel bug..)

(for instance glibc linked libstdc++ works easily or i can compile
with a glibc linked gcc loaded by musl)

Re: dlopen'ing glibc linked libraries

[Szabolcs Nagy]

* Szabolcs Nagy <nsz@port70.net> [2014-01-21 19:17:27 +0100]:
> it may be possible to solve a lot of compatibility issues with
> preloading an extra lib next to musl, i experimented with running
> applications with
> 
> export LD_LIBRARY_PATH=/usr/lib:/usr/lib/i386-linux-gnu:/lib/i386-linux-gnu
> export LD_PRELOAD=./libcompat.so
> /lib/ld-musl-i386.so /usr/bin/something

i did further experiments, the issues i found so far:

- brk pointer is sometimes at the wrong place at start up
so malloc fails (kernel bug, rarely happens here)

- some applications read /proc/self/exe to determine the
path to their binary (to start other instances with
different arguments) but that will point to the loader

- some applications use gcc's closure extension so they
need executable stack (the kernel maps the loader with
non-executable stack)

these are all rare and can be worked around by modifying
the interpreter in the binary using patchelf (or setting
up a namespace where /lib/ld-linux.so.2 points to musl)

- binaries compiled without _FILE_OFFSET_BITS=64 are not
compatible with musl on i386 (on my debian stable system
there are more than 200 binaries and libraries which use
readdir, __[fl]xstat instead of readdir64, __[fl]xstat64)
(this is probably the biggest obstacle for compatibility)

- programs using sbrk fail (affects programs that try to
implement their own malloc (eg bash) or programs using the
boehm gc) (this will go away when musl's alloc is updated)

- i had to add many symbols (>300) most of them are trivial
wrappers, but some of them would be hard to do correctly
(the implementation internals like __pthread_register_cancel
and the messy nss/nscd/rpc apis are the most painful ones)

Re: dlopen'ing glibc linked libraries

[Justin Cormack]

On Thu, Jan 23, 2014 at 10:07 AM, Szabolcs Nagy <nsz@port70.net> wrote:
> - binaries compiled without _FILE_OFFSET_BITS=64 are not
> compatible with musl on i386 (on my debian stable system
> there are more than 200 binaries and libraries which use
> readdir, __[fl]xstat instead of readdir64, __[fl]xstat64)
> (this is probably the biggest obstacle for compatibility)

It might be worth filing bugs for all these and see if we can get them
fixed. Will take a look.

Justin

Re: dlopen'ing glibc linked libraries

[Szabolcs Nagy]

* Justin Cormack <justin@specialbusservice.com> [2014-01-24 10:07:47 +0000]:
> On Thu, Jan 23, 2014 at 10:07 AM, Szabolcs Nagy <nsz@port70.net> wrote:
> > - binaries compiled without _FILE_OFFSET_BITS=64 are not
> > compatible with musl on i386 (on my debian stable system
> > there are more than 200 binaries and libraries which use
> > readdir, __[fl]xstat instead of readdir64, __[fl]xstat64)
> > (this is probably the biggest obstacle for compatibility)
> 
> It might be worth filing bugs for all these and see if we can get them
> fixed. Will take a look.
> 

the default toolchain on i386 does not use _FILE_OFFSET_BITS=64

i guess debian builds packages with that

i don't know if changing the default can break things
(i assume the debian developers had a reason not changing it)

here is a list of packages affected on my system
(most of these are probably harmless in practice: they only use
stat to see if a file is there or only work with tiny files)

acpi
android-tools-adb
android-tools-fastboot
aptitude
aspell
at
audacity
bluez
bsdmainutils
ccache
cron
cvsps
dbus
dc
debianutils
dwarves
e2fslibs
exiftran
exiv2
fakeroot
fontconfig
foomatic-db-engine
gconf2
gdbserver
geeqie
gnuplot-x11
groff
hplip
iceweasel
iproute
isc-dhcp-common
ispell
kbd
lesstif2
libaa1
libaacs0
libao4
libasound2
libaspell15
libaudit0
libbrlapi0.5
libc-bin
libc-dev-bin
libcap-ng0
libcdaudio1
libcdparanoia0
libcwidget3
libdbus-1-3
libdc1394-22
libdirectfb-1.2-9
libdirectfb-dev
libdiscover2
libdrm2
libedit2
libenchant1c2a
libept1.4.12
libevent-2.0-5
libexiv2-12
libflite1
libfltk1.1
libfontconfig1
libfreetype6
libgcrypt11
libgdbm3
libgeoip1
libgphoto2-port0
libgpm2
libgs9
libgssapi-krb5-2
libgutenprint2
libibverbs1
libice6
libicu48
libieee1284-3
libimlib2
libjack-jackd2-0
libk5crypto3
libkrb5-3
libkrb5support0
liblockfile-bin
libltdl7
libmozjs17d
libnetpbm10
libnewt0.52
libnfsidmap2
libnuma1
libopenal1
libopenmpi1.3
libp11-kit0
libpaper1
libpci3
libpciaccess0
libpcsclite1
libprocps0
libproxy0
libpth20
libraw1394-11
libreadline5
libsane
libsasl2-2
libsdl1.2debian
libsemanage1
libsensors4
libslv2-9
libssl1.0.0
libsvga1
libsvn1
libtorque2
libusb-0.1-4
libusb-1.0-0
libv4lconvert0
libva1
libvamp-hostsdk3
libvde0
libvdeplug2
libwildmidi1
libwmf0.2-7
libwrap0
libx11-6
libxau6
libxfont1
libxmlrpc-core-c3
libxpm4
libxslt1.1
libxt6
libzvbi0
mscompress
netpbm
openjdk-6-jre
openssl
optipng
pkg-config
plumb
pngcrush
pngtools
printer-driver-foo2zjs
printer-driver-min12xxw
printer-driver-pnm2ppa
procps
recode
sane-utils
screen
shared-mime-info
sxiv
tin
ufraw-batch
usbutils
valgrind
vde2
w3m
whiptail
x11-utils
x11-xkb-utils
xfonts-utils
xscreensaver
xscreensaver-data
xsel
xserver-xorg
xserver-xorg-core
xterm
xulrunner-17.0
zgv