Re: static linking and dlopen

[Paul Schutte <sjpschutte@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 4295 bytes --]

Hi,

On Sun, Dec 9, 2012 at 2:16 AM, Rich Felker <dalias@aerifal.cx> wrote:

> On Sun, Dec 09, 2012 at 02:04:43AM +0200, Paul Schutte wrote:
> > > On the flip side, the main legitimate uses for dynamic linking and
> > > loading are (1) sharing code that's used by a wide range of
> > > applications and allowing it to be upgraded system-wide all at once,
> > > and (2) facilitating the extension of an application with third-party
> > > code. Usage 1 applies mostly to dynamic linking; 2 mostly to dynamic
> > > loading (dlopen).
> > >
> >
> > Point 1 is probably the reason why most libraries end up as dynamic
> > libraries.
> >
> > I was wondering about distributing all libraries as static libraries and
> > then have the package manager link the application statically as the
> final
> > step of the installation. This way the package manager can keep track
> > of dependencies and re-link them if a library change.
>
> This is a very reasonable design. There is _some_ risk of breakage if
> the static libraries depend on the application being built using the
> exact same headers as the library, but most such dependencies would
> also correspond to ABI breakage for the shared library, so I think the
> risk is low. The main difficulty is getting applications' build
> processes to stop before the final linking and give you output that
> you can relink when needed.
>
>
I was thinking that one can butcher libtool. Software that use libtool
should then in theory work.


> > Distributions like Gentoo who install from source is actually in a very
> > good position to take advantage of static linking.
> >
> > But I can see a lot of compiling/linking happening with this approach.
> >
> > Another idea would be to just install a stub where the binary would be.
> > First time you run this stub, it will link the binary and store it on the
> > disk in some sort of cache. Then just do an exec of that binary. Second
> > time that you run this stub, it will check in this cache, link it again
> if
> > it is not there or just exec it if found. This way only the stuff that
> gets
> > used will be re-linked. You can force a re-link by clearing the cache.
> This
>
> This approach is a bit more difficult, because you need to manage
> things like who has privileges to update the binaries. Surely you can
> do it with suid and/or a daemon, but it's not entirely trivial.
>
>
You are right. That started me thinking along the lines of a "magic"
filesystem based on fuse.
The filesystem code can then do all these things "behind the scenes".


> > what made me wonder about programs that use dlopen.
>
> Actually, I know one more solution for the dlopen issue, but it
> requires some application-level hackery. You just link all the modules
> you'll need into the main binary with a table of strings identifying
> them, and make a dummy dlopen/dlsym implementation that gives you
> access to stuff already linked into the application. The level of
> "evil hackery" is pretty comparable to most of the stuff gnulib
> does...
>
> > I also wonder if the gain would be worth the trouble. I have seen a
> > reduction of up to 50% RSS usage on programs that has a lot of shared
> > libraries. It should improve responsiveness as there will be less paging.
>
> I think the solution that achieves the best balance between reducing
> bloat/slowness/paging and not spending huge amounts of effort is to
> abandon the requirement of static linking everything, and instead go
> with shared libraries for things that are used by a huge portion of
> applications. For shared library "stacks" that have a chain of 10+ .so
> files each depending on the rest, you could replace them with a single
> .so file containing the whole library stack, as long as none of them
> pollute the namespace horribly. This would cut most of the cost of
> dynamic linking right there. For libs that aren't used by many apps,
> or that are written in C++ (which results in huge dynamic-linking
> bloat), I'd just use static versions.
>
>
This makes sense.
 Wonder if there is a spesific reason why the browser folks does'nt produce
a statically linked browser anymore.
The browsers would be good candidates for what you mentioned about C++ and
dynamic bloat.

> Rich
>

Thanks. I understand it a lot better now.

Regards
Paul

[-- Attachment #2: Type: text/html, Size: 5549 bytes --]