[9fans] impact of dynamic libraries on the speed of fork()

[9fans] impact of dynamic libraries on the speed of fork()

[Chris Brannon]

I wrote a really simple program, forktest.c.
Next, I performed some experiments using this program.  Fork is faster
for statically linked executables.  It becomes slower as more libraries
are added to a dynamically linked executable.
These tests were done on an x86 machine running Linux.
Here is a transcript of my experiments, followed by the source for forktest.

-- Chris

--- begin transcript ---
Script started on Fri 20 Feb 2009 01:23:10 PM CST
% dietcc forktest.c # link statically against dietlibc
% ./a.out
30864.1 forks per second
% gcc forktest.c # compile with gcc, dynamic linking
% ./a.out
15723.3 forks per second
% gcc forktest.c -lm # pull in the math library
% ./a.out
14792.9 forks per second
% gcc forktest.c -lm -lcurses
% ./a.out
13888.9 forks per second
% gcc forktest.c -lm -lcurses -lpthread
% ./a.out
11961.7 forks per second
% . gcc forktest.c -lm -lcurses -lpthread -lresolv
% ./a.out
11013.2 forks per second
% gcc forktest.c -lm -lcurses -lpthread -lresolv -lssl
% ./a.out
8250.83 forks per second
% gcc forktest.c -lm -lcurses -lpthread -lresolv -lssl -lreadline
% ./a.out
7961.78 forks per second
% exit

Script done on Fri 20 Feb 2009 01:27:20 PM CST
--- end transcript ---

---begin forktest.c---
#include <stdio.h>
#include <errno.h>
#include <stdlib.h>
#include <time.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/wait.h>

int
main(void)
{
    double elapsed_secs;
    int forks = 0;
    clock_t start = clock(), elapsed;
    int i;
    for(i = 0; i < 25000; i++) {
	int waitstatus;
	switch (fork()) {
	case -1:
	    perror("fork");
	    break;
	case 0:
	    exit(42);
	default:
	    wait(&waitstatus);
	    ++forks;
	    break;
	}
    }
    elapsed = clock() - start;
    elapsed_secs = (double)elapsed / CLOCKS_PER_SEC;
    printf("%g forks per second\n", (double)forks / elapsed_secs);
    exit(0);
}
---end forktest.c---

Re: [9fans] impact of dynamic libraries on the speed of fork()

[Brian L. Stuart]

> I wrote a really simple program, forktest.c.
> Next, I performed some experiments using this program.  Fork is faster
> for statically linked executables.  It becomes slower as more libraries
> are added to a dynamically linked executable.

What fascinates me here is that forktest doesn't even
use anything from those other libraries.  In the statically
linked case, listing an unneeded library is basically a
noop.  It appears to be rather more involved in the
world of GNU shared libraries.

BLS

Re: [9fans] impact of dynamic libraries on the speed of fork()

[erik quanstrom]

On Fri Feb 20 14:35:45 EST 2009, cmbrannon@cox.net wrote:
> I wrote a really simple program, forktest.c.
> Next, I performed some experiments using this program.  Fork is faster
> for statically linked executables.  It becomes slower as more libraries
> are added to a dynamically linked executable.
> These tests were done on an x86 machine running Linux.
> Here is a transcript of my experiments, followed by the source for forktest.
>
> -- Chris

very nice.

i wonder if you may be measuring the performance
of memory management more than the performance of dynamic
linking.  the reason i suspect this is because on my p3 machine i see

gcc static	13440 f/s
gcc dyn	12953 f/s
gcc static 6963 fork + exec/s

or the tipoff is the size of the executable:

; ls -l a.out
-rwxr-xr-x 1 quanstro users 609298 Feb 20 14:36 a.out

for snarky comparison, here are the programs on my system that
are that large or larger:
--rwxrwxr-x M 48625 sys      sys    13275174 Jan 16  2006 /bin/gs
--rwxrwxr-x M 48625 sys      sys      758520 Dec  9  2005 /bin/spin

the hard bit would be keeping the memory footprint the
same while increasing the number of shared libraries.

even in bad cases, this is like 0.2ms/fork + exec.
i wonder if the reason that there can be such big
differences is that linux fork+exec may have been
massaged for such syntetic benchmarks.  thus small
amounts of extra work might look big.

- erik

Re: [9fans] impact of dynamic libraries on the speed of fork()

[Micah Stetson]

[-- Attachment #1: Type: text/plain, Size: 1768 bytes --]

> I wrote a really simple program, forktest.c.

Chris beat me to the punch, but I'm posting anyway because I went a
different direction.  I wrote some rc scripts that make static and
dynamic libraries of various sizes and programs that use those
libraries (trivially).  For each number of functions 1, 10, 100, 1000,
10000, 100000 I timed static and dynamic execution of a program that
conditionally calls that many functions in 1 or 10 libraries.  The
scripts are attached, run mklibs, then mkprogs, then runtests.  Below
are the results of a single run on my laptop (fixed with font looks
better).  I can't spend any more time on this, but it was a fun
morning goof-off.

Static (functions libraries binary-size user system elapsed)
     1  1  556898 0.24 0.50 0.78
    10  1  557324 0.28 0.44 0.82
    10 10  557913 0.32 0.42 0.81
   100  1  561737 0.24 0.50 0.84
   100 10  562196 0.28 0.46 0.79
  1000  1  609496 0.29 0.47 0.83
  1000 10  606381 0.26 0.48 0.84
 10000  1 1105475 0.30 0.44 0.87
 10000 10 1083834 0.28 0.47 0.82
100000  1 6245494 0.27 0.48 0.88
100000 10 6043871 0.28 0.48 0.81

Dynamic (functions libraries binary-size user system elapsed)
     1  1     6489 0.49 0.86 1.39
    10  1     7322 0.52 0.86 1.45
    10 10     7464 0.83 1.14 2.03
   100  1    16366 0.59 0.78 1.42
   100 10    16177 1.14 1.11 2.35
  1000  1   108268 0.55 0.87 1.47
  1000 10   104496 0.88 1.12 2.07
 10000  1  1077758 0.81 0.98 1.89
 10000 10  1037387 1.12 1.36 2.63
100000  1 10915272 2.79 2.50 6.31
100000 10 10517862 3.13 3.68 7.13

I think dynamic 100-10 is a fluke, I also think it's interesting that
the dynamic binaries are bigger above 10000 function calls.  Don't
know why, don't have time to figure it out.

Micah

[-- Attachment #2: mklibs --]
[-- Type: application/octet-stream, Size: 384 bytes --]

#!/usr/local/plan9/bin/rc

letters=(a b c d e f g h i j)
sizes=(1 10 100 1000 10000 100000)

for(l in $letters){
	for(n in $sizes){
		seq $n | sed 's/.*/int '^$l$n^'_&(int);/' >tests/lib$l$n^.h
		seq $n | sed 's/.*/int '^$l$n^'_&(int i) { return i+&; }/' > tests/lib$l$n^.c
		gcc -c -o lib$l$n^.o lib$l$n^.c
		ar rcs lib$l$n^.a lib$l$n^.o
		ld -shared -o lib$l$n^.so lib$l$n^.o
	}
}


[-- Attachment #3: mkprog --]
[-- Type: application/octet-stream, Size: 291 bytes --]

#!/usr/local/plan9/bin/rc

for(lib)
	echo '#include "lib'^$lib^'.h"'

cat <<EOF

int
main(int argc, char* argv)
{
	int i = 0;
	if(argc == 1)
		return 0;
EOF

for(lib){
	n = `{echo $lib | 9 sed 's/[a-z]+([0-9]+)/\1/'}
	seq $n | sed 's/.*/	i += '^$lib^'_&(i);/'
}

cat <<EOF

	return i;
}
EOF

[-- Attachment #4: mkprogs --]
[-- Type: application/octet-stream, Size: 646 bytes --]

#!/usr/local/plan9/bin/rc

fn build {
	name=$1; shift
	./mkprog $* > test$name^.c
	gcc -c -o test$name^.o test$name^.c
	gcc -static -o test$name^-s test$name^.o -L. -l^$*
	gcc -o test$name^-d test$name^.o -L. -l^$*
}

build 1-1 a1
build 10-1 a10
build 10-10 a1 b1 c1 d1 e1 f1 g1 h1 i1 j1
build 100-1 a100
build 100-10 a10 b10 c10 d10 e10 f10 g10 h10 i10 j10
build 1000-1 a1000
build 1000-10 a100 b100 c100 d100 e100 f100 g100 h100 i100 j100
build 10000-1 a10000
build 10000-10 a1000 b1000 c1000 d1000 e1000 f1000 g1000 h1000 i1000 j1000
build 100000-1 a100000
build 100000-10 a10000 b10000 c10000 d10000 e10000 f10000 g10000 h10000 i10000 j10000

[-- Attachment #5: runtests --]
[-- Type: application/octet-stream, Size: 364 bytes --]

#!/usr/local/plan9/bin/rc

LD_LIBRARY_PATH=`{pwd}

fn t {
	echo Testing $1^-s...
	time rc -c 'seq 1000 | sed ''s/.*/.\/'^$1^'-s/'' | rc'
	echo Testing $1^-d...
	time rc -c 'seq 1000 | sed ''s/.*/.\/'^$1^'-d/'' | rc'
}

t test1-1
t test10-1
t test10-10
t test100-1
t test100-10
t test1000-1
t test1000-10
t test10000-1
t test10000-10
t test100000-1
t test100000-10