From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.org/gmane.linux.lib.musl.general/2431 Path: news.gmane.org!not-for-mail From: Rich Felker Newsgroups: gmane.linux.lib.musl.general Subject: Re: static linking and dlopen Date: Sat, 8 Dec 2012 19:16:16 -0500 Message-ID: <20121209001616.GX20323@brightrain.aerifal.cx> References: <20121208225237.GV20323@brightrain.aerifal.cx> <50C3CA75.8000504@comcast.net> <20121208232301.GW20323@brightrain.aerifal.cx> Reply-To: musl@lists.openwall.com NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Trace: ger.gmane.org 1355012199 15198 80.91.229.3 (9 Dec 2012 00:16:39 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Sun, 9 Dec 2012 00:16:39 +0000 (UTC) To: musl@lists.openwall.com Original-X-From: musl-return-2432-gllmg-musl=m.gmane.org@lists.openwall.com Sun Dec 09 01:16:53 2012 Return-path: Envelope-to: gllmg-musl@plane.gmane.org Original-Received: from mother.openwall.net ([195.42.179.200]) by plane.gmane.org with smtp (Exim 4.69) (envelope-from ) id 1ThUZf-0003A2-Bx for gllmg-musl@plane.gmane.org; Sun, 09 Dec 2012 01:16:51 +0100 Original-Received: (qmail 1672 invoked by uid 550); 9 Dec 2012 00:16:38 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: Original-Received: (qmail 1663 invoked from network); 9 Dec 2012 00:16:38 -0000 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Xref: news.gmane.org gmane.linux.lib.musl.general:2431 Archived-At: On Sun, Dec 09, 2012 at 02:04:43AM +0200, Paul Schutte wrote: > > On the flip side, the main legitimate uses for dynamic linking and > > loading are (1) sharing code that's used by a wide range of > > applications and allowing it to be upgraded system-wide all at once, > > and (2) facilitating the extension of an application with third-party > > code. Usage 1 applies mostly to dynamic linking; 2 mostly to dynamic > > loading (dlopen). > > > > Point 1 is probably the reason why most libraries end up as dynamic > libraries. > > I was wondering about distributing all libraries as static libraries and > then have the package manager link the application statically as the final > step of the installation. This way the package manager can keep track > of dependencies and re-link them if a library change. This is a very reasonable design. There is _some_ risk of breakage if the static libraries depend on the application being built using the exact same headers as the library, but most such dependencies would also correspond to ABI breakage for the shared library, so I think the risk is low. The main difficulty is getting applications' build processes to stop before the final linking and give you output that you can relink when needed. > Distributions like Gentoo who install from source is actually in a very > good position to take advantage of static linking. > > But I can see a lot of compiling/linking happening with this approach. > > Another idea would be to just install a stub where the binary would be. > First time you run this stub, it will link the binary and store it on the > disk in some sort of cache. Then just do an exec of that binary. Second > time that you run this stub, it will check in this cache, link it again if > it is not there or just exec it if found. This way only the stuff that gets > used will be re-linked. You can force a re-link by clearing the cache. This This approach is a bit more difficult, because you need to manage things like who has privileges to update the binaries. Surely you can do it with suid and/or a daemon, but it's not entirely trivial. > what made me wonder about programs that use dlopen. Actually, I know one more solution for the dlopen issue, but it requires some application-level hackery. You just link all the modules you'll need into the main binary with a table of strings identifying them, and make a dummy dlopen/dlsym implementation that gives you access to stuff already linked into the application. The level of "evil hackery" is pretty comparable to most of the stuff gnulib does... > I also wonder if the gain would be worth the trouble. I have seen a > reduction of up to 50% RSS usage on programs that has a lot of shared > libraries. It should improve responsiveness as there will be less paging. I think the solution that achieves the best balance between reducing bloat/slowness/paging and not spending huge amounts of effort is to abandon the requirement of static linking everything, and instead go with shared libraries for things that are used by a huge portion of applications. For shared library "stacks" that have a chain of 10+ .so files each depending on the rest, you could replace them with a single .so file containing the whole library stack, as long as none of them pollute the namespace horribly. This would cut most of the cost of dynamic linking right there. For libs that aren't used by many apps, or that are written in C++ (which results in huge dynamic-linking bloat), I'd just use static versions. Rich