Incremental linking

Incremental linking

[Dawid Toton]

I have lot of modules and they are compiled to native code.
So I have .cmx and .o files and want to link them faster.

Is is possible to make linking an associative operation acting on modules?

I would like to do something like the following:
Knowing the correct partial order of modules (that compiler requires) I 
can create a tree that preserves that order. Leafs are modules. Other 
nodes of the tree correspond to a result of linking all descendant 
modules. Modules that are frequently recompiled are placed closer to the 
root. This way I expect to execute less linking operations during 
development.

Documentation of ld says that files produced with --relocatable can be 
used as intermediate partially linked files. Can something like this be 
done with object code produced by ocamlopt?
I don't know ocaml-specific details of linking, so maybe I overlook some 
obvoius obstacle?

Dawid

Re: [Caml-list] Incremental linking

[Xavier Leroy]

Dawid Toton wrote:
> I have lot of modules and they are compiled to native code.
> So I have .cmx and .o files and want to link them faster.
> Is is possible to make linking an associative operation acting on modules?
> [...]
> Documentation of ld says that files produced with --relocatable can be 
> used as intermediate partially linked files. Can something like this be 
> done with object code produced by ocamlopt?

Yes.  "ocamlopt -pack" actually calls "ld -r" underneath to
consolidate several compilation units in a single .cmx/.o file.
"ld -r" will resolve references between these compilation units.

Gerd Stolpmann wrote:
> Well, you can link several .cmx files (and their accompanying .o files)
> to a .cmxa file (and an accompanying .a file): ocamlopt -a

 From a linking standpoint, "ocamlopt -a" is equivalent to "ar": it
does not resolve any references, just concatenates individual
.cmx/.o files in a single .cmxa/.a file.   That can still speed up
linking a bit, since reading one big .a file is faster than reading a
zillion small .o files.

Generally speaking, I'm somewhat surprised that linking time is an
issue for Dawid.  Modern Unix linkers are quite fast, and the
additional link-time work that OCaml does is small.  Let us know if
you manage to narrow the problem.

- Xavier Leroy

Re: Incremental linking

[Dawid Toton]

> 
> Yes.  "ocamlopt -pack" actually calls "ld -r" underneath to
> consolidate several compilation units in a single .cmx/.o file.
> "ld -r" will resolve references between these compilation units.

I tried with this switch and indeed I can link a toy program by
arbitrary intermediate steps. It works OK and is good news to me.

Since today I thought that I have no way to generate the startup code
quickly, but I came to the following solution:
ocamlopt -o prog -dstartup --cc echo A.cmx B.cx C.cmx D.cmx
as -o prog.startup.o prog.startup.s

This way I have all the technical pieces working. Now I have to think
how to automate this. :)

> 
> Generally speaking, I'm somewhat surprised that linking time is an
> issue for Dawid.  Modern Unix linkers are quite fast, and the
> additional link-time work that OCaml does is small.  Let us know if
> you manage to narrow the problem.

My interest in this comes from combination of few facts: all my
calculations are done on a not very fresh machine, my code is a big tree
of small modules and I normally work this way:
1. modify some module (usually at leaf position)
2. recompile everything to native code
3. run, look at the results and go to 1.

I need therefore fast recompilation and fast linking (to stay focused on
the work). I'm about to solve the problem of ocamlbuild spending all its
time looking at unmodified modules. Then I wanted to make sure that it
will be possible to mitigate the second bottleneck.

This is why I'm happy to be able to do incremental linking.

It's helpful that the ocamlopt itself is very fast.

Dawid

Re: [Caml-list] Incremental linking

[Gerd Stolpmann]

Am Dienstag, den 29.09.2009, 20:39 +0200 schrieb Dawid Toton:
> I have lot of modules and they are compiled to native code.
> So I have .cmx and .o files and want to link them faster.

Well, you can link several .cmx files (and their accompanying .o files)
to a .cmxa file (and an accompanying .a file): ocamlopt -a

You cannot do the same again on the next level, i.e. link
several .cmxa/.a together to get another .cmxa/.a. (I don't remember why
this restriction exists.)

Gerd

> 
> Is is possible to make linking an associative operation acting on modules?
> 
> I would like to do something like the following:
> Knowing the correct partial order of modules (that compiler requires) I
> can create a tree that preserves that order. Leafs are modules. Other
> nodes of the tree correspond to a result of linking all descendant
> modules. Modules that are frequently recompiled are placed closer to the
> root. This way I expect to execute less linking operations during
> development.
> 
> Documentation of ld says that files produced with --relocatable can be
> used as intermediate partially linked files. Can something like this be
> done with object code produced by ocamlopt?
> I don't know ocaml-specific details of linking, so maybe I overlook some
> obvoius obstacle?
> 
> Dawid
> 
> _______________________________________________
> Caml-list mailing list. Subscription management:
> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
> Archives: http://caml.inria.fr
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
> 
-- 
------------------------------------------------------------
Gerd Stolpmann, Bad Nauheimer Str.3, 64289 Darmstadt,Germany 
gerd@gerd-stolpmann.de          http://www.gerd-stolpmann.de
Phone: +49-6151-153855                  Fax: +49-6151-997714
------------------------------------------------------------