MUSL_LIBRARY_PATH ?

MUSL_LIBRARY_PATH ?

[John Mudd]

[-- Attachment #1: Type: text/plain, Size: 1626 bytes --]

Possible dumb question...

I built Python using musl. Not easy but it works.

I also build libraries for Ncurses, Readline, Zlib, OpenSSL, BZip2 so that
all of that so the corresponding Python modules are working. Then I
installed setuptools and pip in Python. Then I used pip to download and
install several modules: Requests, ConcurrentLogHandlerand Psutil. All
using musl.

I experimented with dynamic and static binding for the musl lib. I lean
toward dynamic because I may have a need for the "shared" version of Python.

So now I can run this on older machines. That helps me because I need to
deploy on old boxes. Upgrading the O/S is not an option.

But I run into trouble when I start setting LD_LIBRARY_PATH so that Python
can locate the Readline and other libs. The musl built Python works but
these libs start causing native program to fail. e.g.  "vim: error while
loading shared libraries: /usr/lib/i386-linux-gnu/libc.so: invalid ELF
header".

And there's the ld-musl-i386.so.1 file in dynamic mode. I
specified --syslibdir=/tmp when I build musl so that's where I place the
lib. It works but I'd like more flexibility.

I'm naive so my question is... how about a separate MUSL_LIBRARY_PATH shell
variable. Just like LD_LIBRARY_PATH but specific to programs built using
musl. That way I assume I could mix my musl Python with native apps.

As long as I'm asking, can MUSL_LIBRARY_PATH also specify where to
find ld-musl-i386.so.1? That might be crazy because a dynamic musl program
can't start without the lib so it can't interrogate a shell variable? I'm
still asking even though it might require magic.

John

[-- Attachment #2: Type: text/html, Size: 1973 bytes --]

Re: MUSL_LIBRARY_PATH ?

[u-igbb]

Hi John,

On Tue, Apr 01, 2014 at 10:11:57AM -0400, John Mudd wrote:
> But I run into trouble when I start setting LD_LIBRARY_PATH so that Python
> can locate the Readline and other libs. The musl built Python works but
> these libs start causing native program to fail. e.g.  "vim: error while
> loading shared libraries: /usr/lib/i386-linux-gnu/libc.so: invalid ELF
> header".

You seem to encounter the very problem which the newly discussed
 --library-path solves. You may need to run your musl-linked binaries
via wrappers, starting them like
 /..../libc.so --library-path /...yourseparatelibplace /the...binary [args]

Let you look at the recent posts mentioning LD_LIBRARY_PATH.

(Note that hacks like "MUSL_LIBRARY_PATH" do not allow using
in the same process tree different versions/instances of libraries,
iow being about as much broken as LD_LIBRARY_PATH)

Regards,
Rune

Re: MUSL_LIBRARY_PATH ?

[Justin Cormack]

[-- Attachment #1: Type: text/plain, Size: 1061 bytes --]

On Apr 1, 2014 5:19 PM, <u-igbb@aetey.se> wrote:
>
> Hi John,
>
> On Tue, Apr 01, 2014 at 10:11:57AM -0400, John Mudd wrote:
> > But I run into trouble when I start setting LD_LIBRARY_PATH so that
Python
> > can locate the Readline and other libs. The musl built Python works but
> > these libs start causing native program to fail. e.g.  "vim: error while
> > loading shared libraries: /usr/lib/i386-linux-gnu/libc.so: invalid ELF
> > header".
>
> You seem to encounter the very problem which the newly discussed
>  --library-path solves. You may need to run your musl-linked binaries
> via wrappers, starting them like
>  /..../libc.so --library-path /...yourseparatelibplace /the...binary
[args]
>
> Let you look at the recent posts mentioning LD_LIBRARY_PATH.
>
> (Note that hacks like "MUSL_LIBRARY_PATH" do not allow using
> in the same process tree different versions/instances of libraries,
> iow being about as much broken as LD_LIBRARY_PATH)
>

Or use a Musl distro like Sabotage, perhaps in a chroot. That helps a lot
with dynamic libraries.

Justin

[-- Attachment #2: Type: text/html, Size: 1368 bytes --]

Re: MUSL_LIBRARY_PATH ?

[writeonce]

On 04/01/2014 10:11 AM, John Mudd wrote:
> Possible dumb question...
>
> I built Python using musl. Not easy but it works.
>
> I also build libraries for Ncurses, Readline, Zlib, OpenSSL, BZip2 so 
> that all of that so the corresponding Python modules are working. Then 
> I installed setuptools and pip in Python. Then I used pip to download 
> and install several modules: Requests, ConcurrentLogHandlerand Psutil. 
> All using musl.
>
> I experimented with dynamic and static binding for the musl lib. I 
> lean toward dynamic because I may have a need for the "shared" version 
> of Python.
>
> So now I can run this on older machines. That helps me because I need 
> to deploy on old boxes. Upgrading the O/S is not an option.
>
> But I run into trouble when I start setting LD_LIBRARY_PATH so that 
> Python can locate the Readline and other libs. The musl built Python 
> works but these libs start causing native program to fail. e.g.  "vim: 
> error while loading shared libraries: /usr/lib/i386-linux-gnu/libc.so: 
> invalid ELF header".

In addition to setting LD_LIBRARY_PATH, you also need to set 
LIBRARY_PATH at build time, as well as verify that the executable 
contains the correct interpreter (loader) information.  Two simple tests 
you could run are:

1) readelf -e /path/to/executable
--> verify that the program interpreter listed under INTERP is the one 
provided by musl.

2) ldd /path/to/executable
--> if LD_LIBRARY_PATH is improperly set, then ldd will either list the 
non-musl libraries, or completely fail.

my guess is that LIBRARY_PATH was not properly set when you built your 
libraries, and that (1) will thus list the glibc loader (rather than 
musl's) as the "requested program interpreter."

I hope that helps,
zg

>
> And there's the ld-musl-i386.so.1 file in dynamic mode. I 
> specified --syslibdir=/tmp when I build musl so that's where I place 
> the lib. It works but I'd like more flexibility.
>
> I'm naive so my question is... how about a separate MUSL_LIBRARY_PATH 
> shell variable. Just like LD_LIBRARY_PATH but specific to programs 
> built using musl. That way I assume I could mix my musl Python with 
> native apps.
>
> As long as I'm asking, can MUSL_LIBRARY_PATH also specify where to 
> find ld-musl-i386.so.1? That might be crazy because a dynamic musl 
> program can't start without the lib so it can't interrogate a shell 
> variable? I'm still asking even though it might require magic.
>
> John
>
>

Re: MUSL_LIBRARY_PATH ?

[Rich Felker]

On Tue, Apr 01, 2014 at 10:11:57AM -0400, John Mudd wrote:
> Possible dumb question...
> 
> I built Python using musl. Not easy but it works.
> 
> I also build libraries for Ncurses, Readline, Zlib, OpenSSL, BZip2 so that
> all of that so the corresponding Python modules are working. Then I
> installed setuptools and pip in Python. Then I used pip to download and
> install several modules: Requests, ConcurrentLogHandlerand Psutil. All
> using musl.
> 
> I experimented with dynamic and static binding for the musl lib. I lean
> toward dynamic because I may have a need for the "shared" version of Python.

Yes, Python is rather useless without dlopen...

> So now I can run this on older machines. That helps me because I need to
> deploy on old boxes. Upgrading the O/S is not an option.
> 
> But I run into trouble when I start setting LD_LIBRARY_PATH so that Python
> can locate the Readline and other libs. The musl built Python works but
> these libs start causing native program to fail. e.g.  "vim: error while
> loading shared libraries: /usr/lib/i386-linux-gnu/libc.so: invalid ELF
> header".
> 
> And there's the ld-musl-i386.so.1 file in dynamic mode. I
> specified --syslibdir=/tmp when I build musl so that's where I place the
> lib. It works but I'd like more flexibility.

/tmp is a highly unsafe location. If between boot time and your
installation of this file, another user (possibly a compromised nobody
account, even) creates a file by the name name, you'll be tricked into
executing it. If you have root permissions on the box you should just
use the default --syslibdir; if not, use something under your own home
directory.

Note to self: Choosing a proper --syslibdir when you can't use the
default is a topic that should be covered in the manual, since it has
implications for security, moving binaries between systems, etc.

> I'm naive so my question is... how about a separate MUSL_LIBRARY_PATH shell
> variable. Just like LD_LIBRARY_PATH but specific to programs built using
> musl. That way I assume I could mix my musl Python with native apps.
> 
> As long as I'm asking, can MUSL_LIBRARY_PATH also specify where to
> find ld-musl-i386.so.1? That might be crazy because a dynamic musl program
> can't start without the lib so it can't interrogate a shell variable? I'm
> still asking even though it might require magic.

I think you just need to have a path file, as described in the
documentation. For example if --syslibdir is /something/lib/, and your
dynamic linker is /something/lib/ld-musl-i386.so.1, then it will
search for its path file in /something/etc/ld-musl-i386.path.
LD_LIBRARY_PATH is really intended mainly for testing programs in the
build tree before installing them (note: libtool does this
automatically) rather than for configuring deployed programs.

As for adding new env vars, I'd prefer not to add any, and if they are
added, they MUST be in a well-known documented namespace. Adding
MUSL_LIBRARY_PATH would be a security problem because, even if it's
ignored for suid apps, such apps would have no way to knowing they
need to unset it to safely run an external program. If vars are to be
added, the need must be well-documented and alternative approaches
should have already been tried and exhausted.

Rich

Re: MUSL_LIBRARY_PATH ?

[John Mudd]

[-- Attachment #1: Type: text/plain, Size: 787 bytes --]

Thanks for the replies! I now have a proper ld-musl-i386.path file
containing the paths for my libraries. I've unset LD_LIBRARY_PATH and my
musl built Python works well.

I'm still configuring musl with --syslibdir=/tmp/musl/lib/. I agree /tmp is
a poor choice. But I use three different logins on my dev, test and prod
machines and it's not practical to get root access on the test or prod PCs.
So /tmp is a good place to at least run experiments.

Any chance I can specify multiple (colon separated?) paths with
the --syslibdir option? Or can I make it relative to the current user's
home path by starting with "~"?


On Tue, Apr 1, 2014 at 10:11 AM, John Mudd <johnbmudd@gmail.com> wrote:

> Possible dumb question...
>
> I built Python using musl. Not easy but it works.
>
> ...
>
>

[-- Attachment #2: Type: text/html, Size: 1360 bytes --]

Re: Re: MUSL_LIBRARY_PATH ?

[Rich Felker]

On Sun, Apr 06, 2014 at 10:38:53AM -0400, John Mudd wrote:
> Thanks for the replies! I now have a proper ld-musl-i386.path file
> containing the paths for my libraries. I've unset LD_LIBRARY_PATH and my
> musl built Python works well.
> 
> I'm still configuring musl with --syslibdir=/tmp/musl/lib/. I agree /tmp is
> a poor choice. But I use three different logins on my dev, test and prod
> machines and it's not practical to get root access on the test or prod PCs.
> So /tmp is a good place to at least run experiments.
> 
> Any chance I can specify multiple (colon separated?) paths with
> the --syslibdir option? Or can I make it relative to the current user's
> home path by starting with "~"?

No, this is a kernel limitation. Processing the PT_INTERP header and
loading the dynamic linker from the pathname stored there is performed
by the kernel, and only absolute pathnames are supported. It would be
really nice if $ORIGIN expansion were supported by the kernel, but
it's not.

The obvious workaround is to put a shell script in place of your
actual binary, and have it do:

  exec "$ldso" -- "foo.bin" "$@"

or similar. Alternatively a minimal static-linked binary could be used
instead of shell script to make it perform better and eliminate some
of the risks of shell script processing.

Another alternative (much more advanced) would be removing the
PT_INTERP header entirely from dynamic programs and instead static
linking into them some minimal loader code that would find and mmap
the dynamic linker and transfer control to it. IIRC this is actually
how a.out dynamic linking worked long, long ago.

Rich

Re: Re: MUSL_LIBRARY_PATH ?

[Laurent Bercot]

On 06/04/2014 17:18, Rich Felker wrote:
> The obvious workaround is to put a shell script in place of your
> actual binary, and have it do:
>
>    exec "$ldso" -- "foo.bin" "$@"
>
> or similar. Alternatively a minimal static-linked binary could be used
> instead of shell script to make it perform better and eliminate some
> of the risks of shell script processing.

  Shameless plug: this is the perfect use case for execline.
  http://skarnet.org/software/execline/
  An execline script will perform better here than a shell script, without
the risks, and still offer the editability of a script.

-- 
  Laurent

Re: Re: MUSL_LIBRARY_PATH ?

[Rich Felker]

On Sun, Apr 06, 2014 at 06:17:47PM +0100, Laurent Bercot wrote:
> On 06/04/2014 17:18, Rich Felker wrote:
> >The obvious workaround is to put a shell script in place of your
> >actual binary, and have it do:
> >
> >   exec "$ldso" -- "foo.bin" "$@"
> >
> >or similar. Alternatively a minimal static-linked binary could be used
> >instead of shell script to make it perform better and eliminate some
> >of the risks of shell script processing.
> 
>  Shameless plug: this is the perfect use case for execline.
>  http://skarnet.org/software/execline/
>  An execline script will perform better here than a shell script, without
> the risks, and still offer the editability of a script.

Actually execline is not a solution for this, since the #! line that
invokes execline has to contain an absolute pathname to execline,
which is exactly the same problem we were trying to solve to begin
with: the need to install the program interpreter at a fixed absolute
pathname on a system you might not have root on.

This can be avoided by using /usr/bin/env if you're willing to trust
that the env command is at that location and and the user adds
execline to their PATH, but that's fragile and adds yet another level
of indirection at program exec time...

The idea of the minimal static-linked binary as a solution is that it
would not need a program interpreter (dynamic linker or #! type) but
instead could do its own search (e.g. relative to its own binary) for
the dynamic linker to invoke.

Rich

Re: Re: MUSL_LIBRARY_PATH ?

[Laurent Bercot]

On 06/04/2014 18:22, Rich Felker wrote:
> Actually execline is not a solution for this, since the #! line that
> invokes execline has to contain an absolute pathname to execline,
> which is exactly the same problem we were trying to solve to begin
> with: the need to install the program interpreter at a fixed absolute
> pathname on a system you might not have root on.

  That will of course change when execline has conquered the world and
is installed by every administrator at a standardized location. You're
not helping me much with that goal here. :)


> This can be avoided by using /usr/bin/env if you're willing to trust
> that the env command is at that location and and the user adds
> execline to their PATH, but that's fragile and adds yet another level
> of indirection at program exec time...

  Yes, even /bin/sh is probably better than /usr/bin/env + another
indirection.


> The idea of the minimal static-linked binary as a solution is that it
> would not need a program interpreter (dynamic linker or #! type) but
> instead could do its own search (e.g. relative to its own binary) for
> the dynamic linker to invoke.

  It's certainly the most efficient solution for any given binary, but
it's less scalable - as in easy to maintain - than a script-based solution.

-- 
  Laurent