interesting discussion about static linking on luajit ML

interesting discussion about static linking on luajit ML

[John Spencer]

i thought some people here might be interested in a discussion that 
currently happens on luajit@freelists.org

as usual, a strong bias against static linking can be seen, based on 
myths or bad experience with glibc.

especially the statement here about memory usage seems to be a common 
misconception:

http://www.freelists.org/post/luajit/Creating-a-statically-linked-executable-for-a-LuaJITC-program,22

rich, you recently said something about this topic on irc, iirc it was 
along the lines of:
you'd need to link substantially huge parts of libc.a to use nearly as 
much memory,
as is typically wasted with relocation overhead in dynamically linked 
apps, do i remember correctly ?

Re: interesting discussion about static linking on luajit ML

[Szabolcs Nagy]

* John Spencer <maillist-musl@barfooze.de> [2012-10-29 07:14:35 +0100]:
> especially the statement here about memory usage seems to be a
> common misconception:
> 
> http://www.freelists.org/post/luajit/Creating-a-statically-linked-executable-for-a-LuaJITC-program,22
> 

well obviously if the statically linked binary is smaller
then the dynamically linked one, then he cannot save
space the way he wants.. i'm not sure where he got that
size only matters on disk

but he is right about that usually dynamically linked programs
are smaller than statically linked ones
(you should not compare to glibc binaries as they are huge no matter what)

> rich, you recently said something about this topic on irc, iirc it
> was along the lines of:
> you'd need to link substantially huge parts of libc.a to use nearly
> as much memory,
> as is typically wasted with relocation overhead in dynamically
> linked apps, do i remember correctly ?
> 

the point is that with dynamically linked binary you must
have the *entire* lib available in memory
(so even if you have that shared between different processes
you need multiple processes using the lib to gain anything)
(and of course not everything is shared: there are writable
data segments (.data, .bss) which cannot be shared between
processes once they are modified (although they are small in
case of musl), and pic code is a bit bigger as well)

with static linking you only link those object files that
are actually used by the binary
(eg. in case of musl the static lib is about 340k and about
80k is just the math+complex part, if you don't use any of
that then it won't get linked in so you don't have to keep
that around in memory)

(this does not help when you link with xlib because the
simplest x client pulls in a very large part of xlib
and when you run x you surely run several x clients)

(btw even static linked programs can have in memory sharing:
when you run the same process image several times.

this is a more limited sharing than what you get with dynamic
linking, but happens: ppl often have >10 terminals open
or >50 tabs open in a browser.. or a web server have >10 copy
of the same cgi script running (maybe written in lua), the
readonly segments should be shared between these even when
they are separate processes)

other issues with dynamic linking is that at exec you
need to do the magic relocations and symbol lookups etc
which slows down spawning new processes a bit, and
calling functions through plt or accessing data through
got is a bit slower as well

but it's true that security fixes in the libc can get
into the deployed binaries faster with dynamic linking
(and of course bugs can do as well, or when there is an
abi breaking change..)

i think a serious script language should have a static
version optionally available, there are situations where
it makes sense and the size difference they are complaining
about rarely matters

Re: interesting discussion about static linking on luajit ML

[Rich Felker]

On Mon, Oct 29, 2012 at 10:50:03AM +0100, Szabolcs Nagy wrote:
> * John Spencer <maillist-musl@barfooze.de> [2012-10-29 07:14:35 +0100]:
> > especially the statement here about memory usage seems to be a
> > common misconception:
> > 
> > http://www.freelists.org/post/luajit/Creating-a-statically-linked-executable-for-a-LuaJITC-program,22
> > 
> 
> well obviously if the statically linked binary is smaller
> then the dynamically linked one, then he cannot save
> space the way he wants.. i'm not sure where he got that
> size only matters on disk
> 
> but he is right about that usually dynamically linked programs
> are smaller than statically linked ones
> (you should not compare to glibc binaries as they are huge no matter what)
> 
> > rich, you recently said something about this topic on irc, iirc it
> > was along the lines of:
> > you'd need to link substantially huge parts of libc.a to use nearly
> > as much memory,
> > as is typically wasted with relocation overhead in dynamically
> > linked apps, do i remember correctly ?

If you use less than the "dynamic linking overhead" on the libc
comparison page worth of code from the library, then static linking
will use less memory at runtime than dynamic linking. Right now that
threshold is at 20k but I hope to get it back down to 12k soon.

> the point is that with dynamically linked binary you must
> have the *entire* lib available in memory

This is actually not true. Only the pages which are accessed must be
resident in memory. If commonly-used and rarely-used functions are
ordered randomly in the library, then this is likely to cause nearly
the whole library to be paged in, except in cases where individual
unused functions are larger than the page size. If the order of
objects linked into the library is optimized to put all the
rarely-used stuff together, then you can get reasonable memory usage
with shared libraries. This is actually a direction of optimization we
should explore in musl -- I think it should be fairly easy to make the
build scripts group either a known set of commonly-used object files
first, or a known set of rarely-used object files last.

> (so even if you have that shared between different processes
> you need multiple processes using the lib to gain anything)
> (and of course not everything is shared: there are writable
> data segments (.data, .bss) which cannot be shared between
> processes once they are modified (although they are small in
> case of musl), and pic code is a bit bigger as well)

The sice increase of PIC is nontrivial; I'd say it's around 10-15% on
average for i386. On x86_64 and ARM it should be less, but I have not
measured. The worst case is that small functions can double or even
triple in size.

> with static linking you only link those object files that
> are actually used by the binary
> (eg. in case of musl the static lib is about 340k and about
> 80k is just the math+complex part, if you don't use any of
> that then it won't get linked in so you don't have to keep
> that around in memory)

Don't forget that iconv.o is 71k. So that's already a total of 150k
that's unlikely to be used in most programs. Once you add in all the
legacy cruft, I'd say easily over 50% of the library is unlikely to be
used.

> (this does not help when you link with xlib because the
> simplest x client pulls in a very large part of xlib
> and when you run x you surely run several x clients)

Yes, that's a design flaw.

> (btw even static linked programs can have in memory sharing:
> when you run the same process image several times.

Yes. This is especially worth noting for shells.

> i think a serious script language should have a static
> version optionally available, there are situations where
> it makes sense and the size difference they are complaining
> about rarely matters

This is even more true for languages like lua whose primary use case
is embedded in other programs. The people responsible for the
interpreter/jit implementation have no idea what usage cases the
embedding application will be dealing with and whether it might need
static linking.

Rich