Re: [Caml-list] Accelerating compilation

[Romain Bardou <romain.bardou@inria.fr>]

Le 06/09/2013 20:45, Martin Jambon a écrit :
> On 09/06/2013 06:56 AM, Romain Bardou wrote:
>> Hello list,
>>
>> As my project grows bigger, it is becoming much less efficient to use
>> the type-checker to quickly find places in my code which must be updated
>> (e.g. pattern-matching). Here is a wishlist for improvements.
>>
>> 1) Separate typing and code generation, in ocamlc and in ocamlopt
>>
>> For instance, provide an option -typing-only which would mean "only
>> produce the .cmi but do not produce the .cmo or the .cmx". The compiler
>> would only need the .cmi of the dependencies, not their .cmo or .cmx.
>> This would make it possible to have a Makefile target, or an Ocamlbuild
>> option, to just type.
>>
>> Also, provide an option -do-not-retype which would mean "if the .cmi
>> exists, load it instead of type-checking again". This would allow the
>> build process to first type-check (using -typing-only) and then generate
>> the code without type-checking again (using -do-not-retype). Of course
>> the build system should be very careful to ensure the .cmi is
>> up-to-date. This option could also help when compiling both in bytecode
>> and in native code. This option is not necessary to just find errors
>> quickly, though.
>>
>> 2) Be able to disable Ocamlbuild's digest mechanism and use dates and
>> file sizes instead
>>
>> If I am not mistaken, this is one of the main reasons why Ocamlbuild is
>> slower that make.
> 
> It would be interesting to know the size of your project and the time it
> takes to build it.

There are about 30K lines of code in the core of the project, which is
well-split in a rather large number of files (Ocamlbuild reports about
400 targets). It takes about 30s to compile in native mode using the
native compilers.

It might seem very small compared to say, Coq :) But still, having to
wait ~10s just to find the next pattern-matching to fix is already
annoying and it will only get worse.

> Things that have resulted for in significant speed improvements in the
> recent past (< 2 years):
> 
> 1. using omake instead of ocamlbuild
> 2. bytecode camlp4 preprocessing < native camlp4 < camlp5 < nothing
> 3. building only one executable

There are indeed several executables and linking them is one bottleneck.
Some of them I can't merge, but some of them I maybe could… Thanks for
your input!

> 
> I don't have much information on ocamlbuild other than I couldn't make
> it perform as expected (and I don't like the idea of having to write
> build instruction in OCaml anyway). I know that omake does what one
> would expect in terms on parallelism and caching.
> 
> Point (2) is easily checked with 'top'.
> 
> On my machine using an external SSD, what takes the most time is IOs at
> linking time when using a good number of libraries. I don't know if it's
> preventable, if it's a bad interaction between ocamlfind and the
> compilers, or something else. What I do now is build everything into a
> single executable with subcommands, which is handy anyway so why not.
> 
> 
>> It does help to prevent useless recompilation, but
>> what good does it make to prevent a useless recompilation once in a
>> while if it is at the cost of losing a lot of time in all other cases?
>> I'm sure it is project-dependent though so it should only be an option,
>> say, -do-not-hash, or -comparison-mode dateandsize.
>>
>> 3) Parallel compilation in Ocamlbuild
>>
>> Of course it would help but it is not easy to implement so I'm just
>> putting it there to be exhaustive.
>>
>> I think the most important point is the first one, as the other two
>> depend on the build system, and they have been debated already. I'm not
>> aware of any discussion about separing typing and code generation
>> though. What do you think about that?
>>
>> Cheers,
>>
> 
> 

-- 
Romain Bardou