static linking and dlopen

static linking and dlopen

[Paul Schutte]

[-- Attachment #1: Type: text/plain, Size: 472 bytes --]

Hi,

I have a strong preference towards static linking these days because the
running program use so much less memory.

When I link a binary statically and that binary then use dlopen, would that
work 100% ?

What would open if the shared object that was dlopened want's to call
functions in other shared libraries ?
I understand that when using dynamic linking those libraries would just get
loaded, but I am not sure what would happen with static linking.

Regards
Paul

[-- Attachment #2: Type: text/html, Size: 520 bytes --]

Re: static linking and dlopen

[Szabolcs Nagy]

* Paul Schutte <sjpschutte@gmail.com> [2012-12-08 20:09:35 +0200]:
> When I link a binary statically and that binary then use dlopen, would that
> work 100% ?

not required to work

and musl does not implement it (always returns 0)

> What would open if the shared object that was dlopened want's to call
> functions in other shared libraries ?

see the first paragraph in the description of dlopen:

http://pubs.opengroup.org/onlinepubs/9699919799/functions/dlopen.html

Re: static linking and dlopen

[Rob Landley]

On 12/08/2012 02:47:09 PM, Szabolcs Nagy wrote:
> * Paul Schutte <sjpschutte@gmail.com> [2012-12-08 20:09:35 +0200]:
> > When I link a binary statically and that binary then use dlopen,  
> would that
> > work 100% ?
> 
> not required to work
> 
> and musl does not implement it (always returns 0)
> 
> > What would open if the shared object that was dlopened want's to  
> call
> > functions in other shared libraries ?
> 
> see the first paragraph in the description of dlopen:
> 
> http://pubs.opengroup.org/onlinepubs/9699919799/functions/dlopen.html

Please add this to the FAQ, or whatever "how shared libraries work"  
documentation you wind up doing. There were also earlier discussions of  
the horrors of dlclose that should be noted somewhere.

(Unless this is a "belling the cat" situation where suggesting it makes  
it my responsibility, but I don't actually know this stuff...)

Rob

Re: static linking and dlopen

[Rich Felker]

On Sat, Dec 08, 2012 at 08:09:35PM +0200, Paul Schutte wrote:
> Hi,
> 
> I have a strong preference towards static linking these days because the
> running program use so much less memory.
> 
> When I link a binary statically and that binary then use dlopen, would that
> work 100% ?

Presently, it does not work at all. At best, it loses all the
advantages of static linking.

> What would open if the shared object that was dlopened want's to call
> functions in other shared libraries ?

Dependencies of any loaded library also get loaded.

> I understand that when using dynamic linking those libraries would just get
> loaded, but I am not sure what would happen with static linking.

With static linking, they would have to be loaded too. This means a
static-linked program using dlopen would have to contain the entire
dynamic linker logic. What's worse, it would also have to contain at
least the entire libc, and if you were using static versions of any
other library in the main program, and a loaded module referenced a
dynamic version of the same library, you'd probably run into
unpredictable crashing when the versions do not match exactly.

The source of all these problems is basically the same as the benefit
of static linking: the fact that the linker resolves, statically at
link time, which object files are needed to satisfy the needs of the
program. With dlopen, however, there is no static answer; *any* object
is potentially-needed, not directly by the main program, but possibly
by loaded modules. Consider what happens now if you only link part of
libc into the main program statically: additional modules loaded at
runtime won't necessarily have all the stuff they need, so dlopen
would also have to load libc.so. But now you're potentially using two
different versions of libc in the same program; if
implementation-internal data structures like FILE or the pthread
structure are not identical between the 2 versions, you'll hit an ABI
incompatibility, despite the fact that these data structures were
intended to be implementation-internal and never affect ABI. Even
without that issue, you have issues like potentially 2 copies of
malloc trying to manage the heap without being aware of one another,
and thus clobbering it.

For libc, the issues are all fixable by making sure that a static
version of dlopen depends on every single function in libc, so that
another copy never needs to get loaded. However, for other static
libraries pulled into the main program, there is really no fix without
help from the linker (it would have to pull in the entire library, and
somehow leave a note for dlopen to see that library is already loaded
and avoid loading it dynamically too).

Note that, even if we could get this working with a reasonable level
of robustness, almost all the advantages of static linking would be
gone. Static-linked programs using dlopen would be huge and ugly.

If you really want to make single-file binaries with no dependencies
and dlopen support, I think the solution is to first build them
dynamically linked, then merge the main program and all .so files into
a single ELF file. I don't know of any tools capable of doing this,
but in principle it's possible to write one. There are at least 2
different approaches to this. One is to process the ELF files and
merge their list of LOAD segments, symbol and relocation tables, etc.
all into a single ELF file, leaving the relocations in place for the
dynamic linker to perform at startup. This would require some
modification to the dynamic linker still. The other approach is the
equivalent of emacs' unexec dumper: place some kind of hook to run
after the dynamic linker loads everything, but before any other
application code runs, which dumps the entire memory space to an ELF
file which, when run, will reconstruct itself.

Rich

Re: static linking and dlopen

[Charlie Kester]

I wonder if most of what people want to do with dlopen couldn't be done 
just as well (or better) with good old fork() and exec(), along with 
some suitable interprocess communication.

Re: static linking and dlopen

[Rich Felker]

On Sat, Dec 08, 2012 at 03:17:09PM -0800, Charlie Kester wrote:
> I wonder if most of what people want to do with dlopen couldn't be
> done just as well (or better) with good old fork() and exec(), along
> with some suitable interprocess communication.

I think it depends a lot on what you're using dlopen for. A lot of
programs these days use it as a ridiculous part of the development
model rather than for any real purpose; in my book, this is among the
highest levels of Considered Harmful. There's really no reason for any
code that's internal to an application (developed in the same tree, by
the same authors, and built at the same time) to be dynamically linked
at all, much less dynamically loaded. All this accomplishes is making
the program a lot slower and more bloated.

On the flip side, the main legitimate uses for dynamic linking and
loading are (1) sharing code that's used by a wide range of
applications and allowing it to be upgraded system-wide all at once,
and (2) facilitating the extension of an application with third-party
code. Usage 1 applies mostly to dynamic linking; 2 mostly to dynamic
loading (dlopen).

As for your suggestion that dlopen could largely be replaced by
running an external program, that's definitely an alternate way to
accomplish extensions/plug-ins (see GIMP, for example). How well it
works probably depends on the performance requirements (it's hard to
get good performance if the plugin is dealing with high volumes of
data unless you develop complex IPC methods using shared memory, so
there's a complexity trade-off too) and whether your extensions need
to survive across fork (which does not duplicate a process's child
process trees; this could very well matter for extension modules used
by language interpreters/runtimes like Python, Perl, etc.).

Rich

Re: static linking and dlopen

[Paul Schutte]

[-- Attachment #1: Type: text/plain, Size: 2333 bytes --]

On Sun, Dec 9, 2012 at 1:23 AM, Rich Felker <dalias@aerifal.cx> wrote:

> I think it depends a lot on what you're using dlopen for. A lot of
> programs these days use it as a ridiculous part of the development
> model rather than for any real purpose; in my book, this is among the
> highest levels of Considered Harmful. There's really no reason for any
> code that's internal to an application (developed in the same tree, by
> the same authors, and built at the same time) to be dynamically linked
> at all, much less dynamically loaded. All this accomplishes is making
> the program a lot slower and more bloated.
>
> I can not agree with you more on this.


> On the flip side, the main legitimate uses for dynamic linking and
> loading are (1) sharing code that's used by a wide range of
> applications and allowing it to be upgraded system-wide all at once,
> and (2) facilitating the extension of an application with third-party
> code. Usage 1 applies mostly to dynamic linking; 2 mostly to dynamic
> loading (dlopen).
>

Point 1 is probably the reason why most libraries end up as dynamic
libraries.

I was wondering about distributing all libraries as static libraries and
then have the package manager link the application statically as the final
step of the installation. This way the package manager can keep track
of dependencies and re-link them if a library change.

Distributions like Gentoo who install from source is actually in a very
good position to take advantage of static linking.

But I can see a lot of compiling/linking happening with this approach.

Another idea would be to just install a stub where the binary would be.
First time you run this stub, it will link the binary and store it on the
disk in some sort of cache. Then just do an exec of that binary. Second
time that you run this stub, it will check in this cache, link it again if
it is not there or just exec it if found. This way only the stuff that gets
used will be re-linked. You can force a re-link by clearing the cache. This
what made me wonder about programs that use dlopen.

I also wonder if the gain would be worth the trouble. I have seen a
reduction of up to 50% RSS usage on programs that has a lot of shared
libraries. It should improve responsiveness as there will be less paging.

Thanks for all the answers.

Regards
Paul

[-- Attachment #2: Type: text/html, Size: 3020 bytes --]

Re: static linking and dlopen

[Rich Felker]

On Sun, Dec 09, 2012 at 02:04:43AM +0200, Paul Schutte wrote:
> > On the flip side, the main legitimate uses for dynamic linking and
> > loading are (1) sharing code that's used by a wide range of
> > applications and allowing it to be upgraded system-wide all at once,
> > and (2) facilitating the extension of an application with third-party
> > code. Usage 1 applies mostly to dynamic linking; 2 mostly to dynamic
> > loading (dlopen).
> >
> 
> Point 1 is probably the reason why most libraries end up as dynamic
> libraries.
> 
> I was wondering about distributing all libraries as static libraries and
> then have the package manager link the application statically as the final
> step of the installation. This way the package manager can keep track
> of dependencies and re-link them if a library change.

This is a very reasonable design. There is _some_ risk of breakage if
the static libraries depend on the application being built using the
exact same headers as the library, but most such dependencies would
also correspond to ABI breakage for the shared library, so I think the
risk is low. The main difficulty is getting applications' build
processes to stop before the final linking and give you output that
you can relink when needed.

> Distributions like Gentoo who install from source is actually in a very
> good position to take advantage of static linking.
> 
> But I can see a lot of compiling/linking happening with this approach.
> 
> Another idea would be to just install a stub where the binary would be.
> First time you run this stub, it will link the binary and store it on the
> disk in some sort of cache. Then just do an exec of that binary. Second
> time that you run this stub, it will check in this cache, link it again if
> it is not there or just exec it if found. This way only the stuff that gets
> used will be re-linked. You can force a re-link by clearing the cache. This

This approach is a bit more difficult, because you need to manage
things like who has privileges to update the binaries. Surely you can
do it with suid and/or a daemon, but it's not entirely trivial.

> what made me wonder about programs that use dlopen.

Actually, I know one more solution for the dlopen issue, but it
requires some application-level hackery. You just link all the modules
you'll need into the main binary with a table of strings identifying
them, and make a dummy dlopen/dlsym implementation that gives you
access to stuff already linked into the application. The level of
"evil hackery" is pretty comparable to most of the stuff gnulib
does...

> I also wonder if the gain would be worth the trouble. I have seen a
> reduction of up to 50% RSS usage on programs that has a lot of shared
> libraries. It should improve responsiveness as there will be less paging.

I think the solution that achieves the best balance between reducing
bloat/slowness/paging and not spending huge amounts of effort is to
abandon the requirement of static linking everything, and instead go
with shared libraries for things that are used by a huge portion of
applications. For shared library "stacks" that have a chain of 10+ .so
files each depending on the rest, you could replace them with a single
.so file containing the whole library stack, as long as none of them
pollute the namespace horribly. This would cut most of the cost of
dynamic linking right there. For libs that aren't used by many apps,
or that are written in C++ (which results in huge dynamic-linking
bloat), I'd just use static versions.

Rich

Re: static linking and dlopen

[Paul Schutte]

[-- Attachment #1: Type: text/plain, Size: 4295 bytes --]

Hi,

On Sun, Dec 9, 2012 at 2:16 AM, Rich Felker <dalias@aerifal.cx> wrote:

> On Sun, Dec 09, 2012 at 02:04:43AM +0200, Paul Schutte wrote:
> > > On the flip side, the main legitimate uses for dynamic linking and
> > > loading are (1) sharing code that's used by a wide range of
> > > applications and allowing it to be upgraded system-wide all at once,
> > > and (2) facilitating the extension of an application with third-party
> > > code. Usage 1 applies mostly to dynamic linking; 2 mostly to dynamic
> > > loading (dlopen).
> > >
> >
> > Point 1 is probably the reason why most libraries end up as dynamic
> > libraries.
> >
> > I was wondering about distributing all libraries as static libraries and
> > then have the package manager link the application statically as the
> final
> > step of the installation. This way the package manager can keep track
> > of dependencies and re-link them if a library change.
>
> This is a very reasonable design. There is _some_ risk of breakage if
> the static libraries depend on the application being built using the
> exact same headers as the library, but most such dependencies would
> also correspond to ABI breakage for the shared library, so I think the
> risk is low. The main difficulty is getting applications' build
> processes to stop before the final linking and give you output that
> you can relink when needed.
>
>
I was thinking that one can butcher libtool. Software that use libtool
should then in theory work.


> > Distributions like Gentoo who install from source is actually in a very
> > good position to take advantage of static linking.
> >
> > But I can see a lot of compiling/linking happening with this approach.
> >
> > Another idea would be to just install a stub where the binary would be.
> > First time you run this stub, it will link the binary and store it on the
> > disk in some sort of cache. Then just do an exec of that binary. Second
> > time that you run this stub, it will check in this cache, link it again
> if
> > it is not there or just exec it if found. This way only the stuff that
> gets
> > used will be re-linked. You can force a re-link by clearing the cache.
> This
>
> This approach is a bit more difficult, because you need to manage
> things like who has privileges to update the binaries. Surely you can
> do it with suid and/or a daemon, but it's not entirely trivial.
>
>
You are right. That started me thinking along the lines of a "magic"
filesystem based on fuse.
The filesystem code can then do all these things "behind the scenes".


> > what made me wonder about programs that use dlopen.
>
> Actually, I know one more solution for the dlopen issue, but it
> requires some application-level hackery. You just link all the modules
> you'll need into the main binary with a table of strings identifying
> them, and make a dummy dlopen/dlsym implementation that gives you
> access to stuff already linked into the application. The level of
> "evil hackery" is pretty comparable to most of the stuff gnulib
> does...
>
> > I also wonder if the gain would be worth the trouble. I have seen a
> > reduction of up to 50% RSS usage on programs that has a lot of shared
> > libraries. It should improve responsiveness as there will be less paging.
>
> I think the solution that achieves the best balance between reducing
> bloat/slowness/paging and not spending huge amounts of effort is to
> abandon the requirement of static linking everything, and instead go
> with shared libraries for things that are used by a huge portion of
> applications. For shared library "stacks" that have a chain of 10+ .so
> files each depending on the rest, you could replace them with a single
> .so file containing the whole library stack, as long as none of them
> pollute the namespace horribly. This would cut most of the cost of
> dynamic linking right there. For libs that aren't used by many apps,
> or that are written in C++ (which results in huge dynamic-linking
> bloat), I'd just use static versions.
>
>
This makes sense.
 Wonder if there is a spesific reason why the browser folks does'nt produce
a statically linked browser anymore.
The browsers would be good candidates for what you mentioned about C++ and
dynamic bloat.

> Rich
>

Thanks. I understand it a lot better now.

Regards
Paul

[-- Attachment #2: Type: text/html, Size: 5549 bytes --]

Re: static linking and dlopen

[Rich Felker]

On Sun, Dec 09, 2012 at 05:24:29PM +0200, Paul Schutte wrote:
> This makes sense.
>  Wonder if there is a spesific reason why the browser folks does'nt produce
> a statically linked browser anymore.
> The browsers would be good candidates for what you mentioned about C++ and
> dynamic bloat.

It's because dynamic linking is part of their _development_ model. My
understanding is that they lack proper functional makefiles that would
facilitate clean incremental compiling and linking, so they instead
break the project up into a number of separate library components, and
they can then rebuild just a single component to test (since it gets
dynamically loaded anyway) rather than having to rebuild the whole
program from scratch.

While I think this is a stupid development model, as long as they're
just doing it for development, it doesn't really harm end users that
much. The problem is that they don't have a "release" build mode that
just links everything together the right way. It's not clear to me
whether this would be easy to change; it's possible that, due to
always using dynamic linking internally, a number of dependencies on
dynamic-linking-related behavior (symbol interposition, accessing
things via dlsym, etc.) crept into the code, and would be painful to
exorcise (especially under the constraint of not breaking their
dynamic builds).

Rich

Re: static linking and dlopen

[Szabolcs Nagy]

* Rich Felker <dalias@aerifal.cx> [2012-12-09 12:54:52 -0500]:
> just links everything together the right way. It's not clear to me
> whether this would be easy to change; it's possible that, due to
> always using dynamic linking internally, a number of dependencies on
> dynamic-linking-related behavior (symbol interposition, accessing
> things via dlsym, etc.) crept into the code, and would be painful to
> exorcise (especially under the constraint of not breaking their
> dynamic builds).

i guess it's hard to fix (legacy from netscape 2.0)

https://en.wikipedia.org/wiki/NPAPI
https://developer.mozilla.org/en-US/docs/Gecko_Plugin_API_Reference/Plug-in_Basics

this plugin system is a core part of dom
and uses the nspr (netscape portable runtime)
to do the dlopen/dlclose/dlsym

https://www.mozilla.org/projects/nspr/reference/html/prlink.html

and most likely there are other dynamic linking
dependencies, this is just an obvious one

Re: static linking and dlopen

[Rich Felker]

On Sun, Dec 09, 2012 at 08:07:53PM +0100, Szabolcs Nagy wrote:
> * Rich Felker <dalias@aerifal.cx> [2012-12-09 12:54:52 -0500]:
> > just links everything together the right way. It's not clear to me
> > whether this would be easy to change; it's possible that, due to
> > always using dynamic linking internally, a number of dependencies on
> > dynamic-linking-related behavior (symbol interposition, accessing
> > things via dlsym, etc.) crept into the code, and would be painful to
> > exorcise (especially under the constraint of not breaking their
> > dynamic builds).
> 
> i guess it's hard to fix (legacy from netscape 2.0)
> 
> https://en.wikipedia.org/wiki/NPAPI
> https://developer.mozilla.org/en-US/docs/Gecko_Plugin_API_Reference/Plug-in_Basics
> 
> this plugin system is a core part of dom
> and uses the nspr (netscape portable runtime)
> to do the dlopen/dlclose/dlsym

Yes, but being that it's abstracted by nspr, it could be replaced with
a system to use internal 'plugins'. Anyway I was less concerned with
features that are based on dynamic loading (plugins) and more
interested in the problem that the whole core browser is made up of
tons of shared libraries that are always-loaded (whether by dlopen or
at startup-time, I'm not sure..)

Rich

Re: static linking and dlopen

[Szabolcs Nagy]

* Paul Schutte <sjpschutte@gmail.com> [2012-12-09 02:04:43 +0200]:
> I was wondering about distributing all libraries as static libraries and
> then have the package manager link the application statically as the final
> step of the installation. This way the package manager can keep track
> of dependencies and re-link them if a library change.

note that binaries built by the user outside of the
package manager wont be relinked which may or may
not be what you want

static linking takes a lot of time and memory
(gold is better at this than gnu ld, but a libc
update would take some time either way)

you cannot easily collapse dependency chains:
eg. with x,y binaries, a,b,c objects and
 x->a->b->c
 y->b->c
dependency chains, the b->c linking is done twice
(so not much work can be shared between the linking
of various binaries and the linking of a huge binary
takes a lot of time even if an insignificant small
part changes)

other than dlopen the LD_PRELOAD and LD_LIBRARY_PATH
hacks would not work, nor system components that depend
on dynamic linking (vdso, proprietary libraries
(graphics drivers etc), nsswitch, pam)

i'm not saying your idea is bad, but it's non-trivial

> Another idea would be to just install a stub where the binary would be.
> First time you run this stub, it will link the binary and store it on the
> disk in some sort of cache. Then just do an exec of that binary. Second

i guess the stub would need setuid

Re: static linking and dlopen

[croco]

Hi folks,

On Sat, Dec 08, 2012 at 03:17:09PM -0800, Charlie Kester wrote:
> I wonder if most of what people want to do with dlopen couldn't be done  
> just as well (or better) with good old fork() and exec(), along with  
> some suitable interprocess communication.

The thing I recalled right off, is that, e.g., interpreters of Tcl are able
to load C-written functions to extend the functionality of the interpreter
(e.g. with a domain-specific set of functions), and this is done with
dlopen.  I think Tcl is not the only interpreter doing so.  I even used the
same technique in one of my works (it was an interpreter of (small subset
of) Lisp, extendable in the same way).  Well, may be this (in theory) can
be done with fork/exec, with some protocol on the pipes, etc., but as for
me, I would never use such a solution, preferring to keep the interpreter
non-extendable: having library functions' implementations in an external
process conflicts with my sense of aesthetics :-)

IMHO, for some (monstrous) programs such as browsers, or Gimp, or whatever
alike, having all the dynamic linking logic inside is not a catastrophe,
because the catastrophe is their size as such.  For interpreters, the
catastrophe is the fact of interpreted execution, and dlopen will add
next to nothing to it.

Anyway, for a lot of existing programs, the absense of dlopen will mean a
necessity for heavy rewritings in order to compile them with musl - perhaps
too heavy to be done at all.



P.S. thank you for the great work you do!

--
Cheers,
Croco

Re: static linking and dlopen

[Isaac Dunham]

On Sun, 9 Dec 2012 10:36:58 +0400
croco@openwall.com wrote:

> Anyway, for a lot of existing programs, the absense of dlopen will mean a
> necessity for heavy rewritings in order to compile them with musl - perhaps
> too heavy to be done at all.
> 
I think there's a misunderstanding here:

musl includes a dlopen function.
When a binary is statically linked, it is a stub.
When a binary is dynamically linked, it loads the shared library requested.

> P.S. thank you for the great work you do!
> 
> --
> Cheers,
> Croco

-- 
Isaac Dunham <idunham@lavabit.com>

Re: static linking and dlopen

[Charlie Kester]

On 12/08/2012 11:25 PM, Isaac Dunham wrote:
> On Sun, 9 Dec 2012 10:36:58 +0400
> croco@openwall.com wrote:
>
>> Anyway, for a lot of existing programs, the absense of dlopen will mean a
>> necessity for heavy rewritings in order to compile them with musl - perhaps
>> too heavy to be done at all.
>>
> I think there's a misunderstanding here:

Yes, and I apologize if I contributed to that misunderstanding with my 
musings on alternative designs.

> musl includes a dlopen function.
> When a binary is statically linked, it is a stub.
> When a binary is dynamically linked, it loads the shared library requested.

Re: static linking and dlopen

[croco]

On Sat, Dec 08, 2012 at 11:25:29PM -0800, Isaac Dunham wrote:
> I think there's a misunderstanding here:
> 
> musl includes a dlopen function.
> When a binary is statically linked, it is a stub.
> When a binary is dynamically linked, it loads the shared library requested.

Oh, well, I didn't catch the situation; if this is the case, the things are
definitely better.  However, to my mind, static linkage is good for
creating portable binaries (besides all the other advantages), and I can
easily imagine a situation in which I dislike the idea of a
dynamically-linked main binary (e.g. I ship some unusual software to
endusers, and they have different Linux distros but are not going to build
the soft from sources - yes, there are such Linux users who panic when they
hear the word "compiler" - and I've got no hope someone else will package
my soft for different distros, because it is too unusual, so either I spend
my time installing 20+ different versions of various distros and prepare
packages for them all, or I opt for -static).  So, I'd like to have all the
libs inside the binary of, e.g., my interpreter (actually, this can be a
program which does its job being controlled by embedded interpreter).  But,
at the same time, it is very possible I need these loadable modules, which
extend the functionality of the interpreter.  Surely it is not a
catastrophe, as I can link all libs but musl statically, and provide
libmusl.so along with the binary, also having a script which sets
LD_LIBRARY_PATH and then runs the binary; but it is a bit, errr...
/strange/ :)

Actually, when it comes to -static, the linker only picks the modules that
contain unresolved symbols, so it should (am I right?) be easy to break the
things down to modules so that all the dynamic linkage mechanics is linked
into the binary only in case it calls dlopen.  And, okay, to mention in the
man 3 dlopen that using it from within a statically-linked binary will
increase the size of the binary by another megabyte, and that the .so to be
loaded must itself use statically-linked version of libraries so that some
functions will be loaded to the code segment twice.  Such practice should
be discouraged but I don't think it should be made impossible at all.

I realize, however, that it is possible I simply miss something.


--
Croco

Re: static linking and dlopen

[Szabolcs Nagy]

* croco@openwall.com <croco@openwall.com> [2012-12-09 14:08:46 +0400]:
> packages for them all, or I opt for -static).  So, I'd like to have all the
> libs inside the binary of, e.g., my interpreter (actually, this can be a
> program which does its job being controlled by embedded interpreter).  But,
> at the same time, it is very possible I need these loadable modules, which

dalias just described why static linking with dlopen is not possible

(he also specifically mentioned interpreters using dlopen..)

> loaded must itself use statically-linked version of libraries so that some
> functions will be loaded to the code segment twice.  Such practice should
> be discouraged but I don't think it should be made impossible at all.

it is impossible

two version of the same function cannot go together:

they may work correctly only if some global state exists
only once or modified from a single place
(malloc with brk pointer and exit with file stream list
were such examples)

so you can use dlopen only if you can ensure you don't use
the same libraries as the dlopened code or all shared code
is pure (no writeable global state, no sideeffects), but if
you use dlopen you already depend on libc and if the dlopened
code also uses libc you have a problem

(different library version was an additional issue)

so you can have dlopen in your static interpreter if
1) you know all the dlopened code and verified that they don't
depend on libc or anything impure on which the interpreter depends
(but in this case you should link those statically)
2) include all the dependent libraries entirely into the
statically linked binary and have dlopen use the symbols from
these (for which you may need to modify the linker and make sure
the binary is not stripped or implement dlopen so it somehow has
a list of all the extern symbols)
(but this does not seem practical and still does not solve abi
breaking library versioning issues)

Re: static linking and dlopen

[Rich Felker]

On Sun, Dec 09, 2012 at 12:46:08PM +0100, Szabolcs Nagy wrote:
> * croco@openwall.com <croco@openwall.com> [2012-12-09 14:08:46 +0400]:
> > packages for them all, or I opt for -static).  So, I'd like to have all the
> > libs inside the binary of, e.g., my interpreter (actually, this can be a
> > program which does its job being controlled by embedded interpreter).  But,
> > at the same time, it is very possible I need these loadable modules, which
> 
> dalias just described why static linking with dlopen is not possible

To clarify, it's not possible right now, and difficult or impossible
to do "right". There are various partly-working approaches though.

> so you can use dlopen only if you can ensure you don't use
> the same libraries as the dlopened code or all shared code
> is pure (no writeable global state, no sideeffects), but if
> you use dlopen you already depend on libc and if the dlopened
> code also uses libc you have a problem

The dynamic linker/libc already refuses to load itself, so in musl
there's not really the danger of libc being loaded twice. The issue is
that we would have to ensure that the whole libc gets linked into the
main program so it's available to loaded modules -- and that the
symbols are all kept so that they can be used. Both of these, while
possible, are not entirely trivial to do.

I'm not against consideration of support for this, but I would like to
first explore the alternative design I suggested: initially linking as
a dynamic-linked program, then using a special utility to combine all
of the shared libraries into a single file. While this approach does
have some disadvantages (PIC code, startup time cost, etc. much like
dynamic linking), it has some of the advantages of static linking
(not searching/loading multiple files all over the filesystem,
single-file distribution, etc.) and an additional benefit is that,
with support from the dynamic linker, even the .so files intended for
loading with dlopen could be packed into the main file.

Rich

Re: static linking and dlopen

[Rob Landley]

On 12/09/2012 04:08:46 AM, croco@openwall.com wrote:
> On Sat, Dec 08, 2012 at 11:25:29PM -0800, Isaac Dunham wrote:
> > I think there's a misunderstanding here:
> >
> > musl includes a dlopen function.
> > When a binary is statically linked, it is a stub.
> > When a binary is dynamically linked, it loads the shared library  
> requested.
> 
> Oh, well, I didn't catch the situation; if this is the case, the  
> things are
> definitely better.  However, to my mind, static linkage is good for
> creating portable binaries (besides all the other advantages),

And musl has good support for static linking. Musl has good support for  
dynamic linking. Mixing the two is a bad idea at the design level,  
which posix does not guarantee to work.

> and I can
> easily imagine a situation in which I dislike the idea of a
> dynamically-linked main binary (e.g. I ship some unusual software to
> endusers, and they have different Linux distros but are not going to  
> build
> the soft from sources - yes, there are such Linux users who panic  
> when they
> hear the word "compiler" - and I've got no hope someone else will  
> package
> my soft for different distros, because it is too unusual, so either I  
> spend
> my time installing 20+ different versions of various distros and  
> prepare
> packages for them all, or I opt for -static).

Yay static linking.

> So, I'd like to have all the
> libs inside the binary of, e.g., my interpreter (actually, this can  
> be a
> program which does its job being controlled by embedded  
> interpreter).  But,
> at the same time, it is very possible I need these loadable modules,  
> which
> extend the functionality of the interpreter.

If you need dynamic linking, then you need dynamic linking.

> Surely it is not a
> catastrophe, as I can link all libs but musl statically, and provide
> libmusl.so along with the binary, also having a script which sets
> LD_LIBRARY_PATH and then runs the binary; but it is a bit, errr...
> /strange/ :)

You want a statically, dynamically linked program. Presumably combining  
PIC and non-PIC code.

Good luck.

> Actually, when it comes to -static, the linker only picks the modules  
> that
> contain unresolved symbols, so it should (am I right?) be easy to  
> break the
> things down to modules so that all the dynamic linkage mechanics is  
> linked
> into the binary only in case it calls dlopen.  And, okay, to mention  
> in the
> man 3 dlopen that using it from within a statically-linked binary will
> increase the size of the binary by another megabyte, and that the .so  
> to be
> loaded must itself use statically-linked version of libraries so that  
> some
> functions will be loaded to the code segment twice.  Such practice  
> should
> be discouraged but I don't think it should be made impossible at all.
> 
> I realize, however, that it is possible I simply miss something.

You're confusing "possible" with "a good idea". Many truly horrible  
ideas aren't actually impossible, as Windows extensively demonstrates.

Possibly what you want is large chunks of the musl dynamic loader  
factored out so you can easily suck them into your program and keep the  
pieces when it breaks.

Rob

Re: static linking and dlopen

[Paul Schutte]

[-- Attachment #1: Type: text/plain, Size: 4253 bytes --]

Thanks for the comprehensive answer, this is the answer I was looking for.

There is currently not a situation that require me to do this. I was just
thinking about it and decided to ask the experts.

On Sun, Dec 9, 2012 at 12:52 AM, Rich Felker <dalias@aerifal.cx> wrote:

> On Sat, Dec 08, 2012 at 08:09:35PM +0200, Paul Schutte wrote:
> > Hi,
> >
> > I have a strong preference towards static linking these days because the
> > running program use so much less memory.
> >
> > When I link a binary statically and that binary then use dlopen, would
> that
> > work 100% ?
>
> Presently, it does not work at all. At best, it loses all the
> advantages of static linking.
>
> > What would open if the shared object that was dlopened want's to call
> > functions in other shared libraries ?
>
> Dependencies of any loaded library also get loaded.
>
> > I understand that when using dynamic linking those libraries would just
> get
> > loaded, but I am not sure what would happen with static linking.
>
> With static linking, they would have to be loaded too. This means a
> static-linked program using dlopen would have to contain the entire
> dynamic linker logic. What's worse, it would also have to contain at
> least the entire libc, and if you were using static versions of any
> other library in the main program, and a loaded module referenced a
> dynamic version of the same library, you'd probably run into
> unpredictable crashing when the versions do not match exactly.
>
> The source of all these problems is basically the same as the benefit
> of static linking: the fact that the linker resolves, statically at
> link time, which object files are needed to satisfy the needs of the
> program. With dlopen, however, there is no static answer; *any* object
> is potentially-needed, not directly by the main program, but possibly
> by loaded modules. Consider what happens now if you only link part of
> libc into the main program statically: additional modules loaded at
> runtime won't necessarily have all the stuff they need, so dlopen
> would also have to load libc.so. But now you're potentially using two
> different versions of libc in the same program; if
> implementation-internal data structures like FILE or the pthread
> structure are not identical between the 2 versions, you'll hit an ABI
> incompatibility, despite the fact that these data structures were
> intended to be implementation-internal and never affect ABI. Even
> without that issue, you have issues like potentially 2 copies of
> malloc trying to manage the heap without being aware of one another,
> and thus clobbering it.
>
> For libc, the issues are all fixable by making sure that a static
> version of dlopen depends on every single function in libc, so that
> another copy never needs to get loaded. However, for other static
> libraries pulled into the main program, there is really no fix without
> help from the linker (it would have to pull in the entire library, and
> somehow leave a note for dlopen to see that library is already loaded
> and avoid loading it dynamically too).
>
> Note that, even if we could get this working with a reasonable level
> of robustness, almost all the advantages of static linking would be
> gone. Static-linked programs using dlopen would be huge and ugly.
>
> If you really want to make single-file binaries with no dependencies
> and dlopen support, I think the solution is to first build them
> dynamically linked, then merge the main program and all .so files into
> a single ELF file. I don't know of any tools capable of doing this,
> but in principle it's possible to write one. There are at least 2
> different approaches to this. One is to process the ELF files and
> merge their list of LOAD segments, symbol and relocation tables, etc.
> all into a single ELF file, leaving the relocations in place for the
> dynamic linker to perform at startup. This would require some
> modification to the dynamic linker still. The other approach is the
> equivalent of emacs' unexec dumper: place some kind of hook to run
> after the dynamic linker loads everything, but before any other
> application code runs, which dumps the entire memory space to an ELF
> file which, when run, will reconstruct itself.
>
> Rich
>

[-- Attachment #2: Type: text/html, Size: 4895 bytes --]

Re: static linking and dlopen

[Szabolcs Nagy]

* Rich Felker <dalias@aerifal.cx> [2012-12-08 17:52:37 -0500]:
> Presently, it does not work at all. At best, it loses all the
> advantages of static linking.

i think it can work

> With static linking, they would have to be loaded too. This means a
> static-linked program using dlopen would have to contain the entire
> dynamic linker logic. What's worse, it would also have to contain at
> least the entire libc, and if you were using static versions of any
> other library in the main program, and a loaded module referenced a
> dynamic version of the same library, you'd probably run into
> unpredictable crashing when the versions do not match exactly.

i think one could make a dso that has no dependency
(all dependencies are linked into it including libc),
and no pointers are passed (directly or indirectly)
to types which are not entirely part of the abi
between the dso and the main binary
(ie no FILE* or anything that might have different
internals on the two sides)

in that case dlopen() would work and even dlclose()
can be safe (as the programmer has control over the
entire code)

of course care should be taken and it might not be
very useful but one could implement loadable/unloadable
plugins with static linking

Re: static linking and dlopen

[Rich Felker]

On Sun, Dec 09, 2012 at 03:55:30AM +0100, Szabolcs Nagy wrote:
> i think one could make a dso that has no dependency
> (all dependencies are linked into it including libc),
> and no pointers are passed (directly or indirectly)
> to types which are not entirely part of the abi
> between the dso and the main binary
> (ie no FILE* or anything that might have different
> internals on the two sides)

It's not that simple. There's always at least one piece of state
that's shared: the brk. 2 versions of malloc running will always cause
horrible problems unless they're aware of each other or avoid using
brk at all.

There's also the issue of stdio flushing at exit. The main program's
exit() can't flush the stdio streams belonging to the dso.

Thread-local storage in the dso would also be another problematic
area..

Rich

Re: static linking and dlopen

[Isaac Dunham]

On Sun, 9 Dec 2012 03:55:30 +0100
Szabolcs Nagy <nsz@port70.net> wrote:

> i think one could make a dso that has no dependency
> (all dependencies are linked into it including libc),
> and no pointers are passed (directly or indirectly)
> to types which are not entirely part of the abi
> between the dso and the main binary
> (ie no FILE* or anything that might have different
> internals on the two sides)

Under at least glibc, this is supported.
This is what gcc/ld does with "-static -shared".

> in that case dlopen() would work and even dlclose()
> can be safe (as the programmer has control over the
> entire code)
> 
> of course care should be taken and it might not be
> very useful but one could implement loadable/unloadable
> plugins with static linking


-- 
Isaac Dunham <idunham@lavabit.com>

Re: static linking and dlopen

[Rob Landley]

On 12/08/2012 04:52:37 PM, Rich Felker wrote:
> The source of all these problems is basically the same as the benefit
> of static linking: the fact that the linker resolves, statically at
> link time, which object files are needed to satisfy the needs of the
> program. With dlopen, however, there is no static answer; *any* object
> is potentially-needed, not directly by the main program, but possibly
> by loaded modules.

Is there some way you could put a dlopen() symbol in libc.a that forces  
a build break? Perhaps a redirect to  
dlopen_is_not_compatible_with_static_linking_go_read_posix?

Rob

Re: static linking and dlopen

[Rich Felker]

On Sun, Dec 09, 2012 at 02:09:23PM -0600, Rob Landley wrote:
> On 12/08/2012 04:52:37 PM, Rich Felker wrote:
> >The source of all these problems is basically the same as the benefit
> >of static linking: the fact that the linker resolves, statically at
> >link time, which object files are needed to satisfy the needs of the
> >program. With dlopen, however, there is no static answer; *any* object
> >is potentially-needed, not directly by the main program, but possibly
> >by loaded modules.
> 
> Is there some way you could put a dlopen() symbol in libc.a that
> forces a build break? Perhaps a redirect to
> dlopen_is_not_compatible_with_static_linking_go_read_posix?

The dlopen in libc.a simply always-fails. For programs where loading
modules is optional, this is probably the ideal behavior.

Rich