[Caml-list] Accelerating compilation

[Caml-list] Accelerating compilation

[Romain Bardou]

Hello list,

As my project grows bigger, it is becoming much less efficient to use
the type-checker to quickly find places in my code which must be updated
(e.g. pattern-matching). Here is a wishlist for improvements.

1) Separate typing and code generation, in ocamlc and in ocamlopt

For instance, provide an option -typing-only which would mean "only
produce the .cmi but do not produce the .cmo or the .cmx". The compiler
would only need the .cmi of the dependencies, not their .cmo or .cmx.
This would make it possible to have a Makefile target, or an Ocamlbuild
option, to just type.

Also, provide an option -do-not-retype which would mean "if the .cmi
exists, load it instead of type-checking again". This would allow the
build process to first type-check (using -typing-only) and then generate
the code without type-checking again (using -do-not-retype). Of course
the build system should be very careful to ensure the .cmi is
up-to-date. This option could also help when compiling both in bytecode
and in native code. This option is not necessary to just find errors
quickly, though.

2) Be able to disable Ocamlbuild's digest mechanism and use dates and
file sizes instead

If I am not mistaken, this is one of the main reasons why Ocamlbuild is
slower that make. It does help to prevent useless recompilation, but
what good does it make to prevent a useless recompilation once in a
while if it is at the cost of losing a lot of time in all other cases?
I'm sure it is project-dependent though so it should only be an option,
say, -do-not-hash, or -comparison-mode dateandsize.

3) Parallel compilation in Ocamlbuild

Of course it would help but it is not easy to implement so I'm just
putting it there to be exhaustive.

I think the most important point is the first one, as the other two
depend on the build system, and they have been debated already. I'm not
aware of any discussion about separing typing and code generation
though. What do you think about that?

Cheers,

-- 
Romain Bardou

Re: [Caml-list] Accelerating compilation

[Markus Mottl]

On Fri, Sep 6, 2013 at 9:56 AM, Romain Bardou <romain.bardou@inria.fr> wrote:
> 1) Separate typing and code generation, in ocamlc and in ocamlopt
>
> For instance, provide an option -typing-only which would mean "only
> produce the .cmi but do not produce the .cmo or the .cmx". The compiler
> would only need the .cmi of the dependencies, not their .cmo or .cmx.
> This would make it possible to have a Makefile target, or an Ocamlbuild
> option, to just type.

This seems like an interesting suggestion.  Code generation,
especially for native code, can be quite expensive.  I do use byte
code compilation during development for that reason, which is somewhat
faster.

Note, however, that there is one problem with the approach as
suggested: if you have both an .mli and an .ml file, the build system
would have to know when to "type" the latter.  In the suggested
approach there would be no trace of this action, because the .cmi
would come from the .mli file.  We would need to generate a separate
dummy file in that case to visibly record the fact that the .ml file
unifies with the .cmi file.  Or (see below) write out the typed
abstract syntax tree of the .ml file, which would, of course, also
have to unify with the .cmi file.

> Also, provide an option -do-not-retype which would mean "if the .cmi
> exists, load it instead of type-checking again". This would allow the
> build process to first type-check (using -typing-only) and then generate
> the code without type-checking again (using -do-not-retype). Of course
> the build system should be very careful to ensure the .cmi is
> up-to-date. This option could also help when compiling both in bytecode
> and in native code. This option is not necessary to just find errors
> quickly, though.

The .cmi file does not contain enough information, because it only
contains the signature.  You need the typed AST for proper code
generation, which might be quite big.  I haven't looked into this, but
I wouldn't be surprised if the size of that thing could be so large
that you might prefer type checking again over writing to + reloading
it from a file.

> 2) Be able to disable Ocamlbuild's digest mechanism and use dates and
> file sizes instead
>
> If I am not mistaken, this is one of the main reasons why Ocamlbuild is
> slower that make. It does help to prevent useless recompilation, but
> what good does it make to prevent a useless recompilation once in a
> while if it is at the cost of losing a lot of time in all other cases?
> I'm sure it is project-dependent though so it should only be an option,
> say, -do-not-hash, or -comparison-mode dateandsize.

The problem here is that not all Unix-filesystems support sub-second
resolution in timestamps.  So if you build a file and change a
character in it within one second, the change may go unnoticed, i.e.
required recompilation doesn't happen.  That's why hashing provides
much stronger guarantees.  But I think this is not a big problem in
practice so I'd support such a flag in ocamlbuild.

> 3) Parallel compilation in Ocamlbuild
>
> Of course it would help but it is not easy to implement so I'm just
> putting it there to be exhaustive.

I'm not sure what you are referring to, OCamlBuild does already
support parallel builds.

Regards,
Markus

-- 
Markus Mottl        http://www.ocaml.info        markus.mottl@gmail.com

Re: [Caml-list] Accelerating compilation

[Gabriel Scherer]

Those are all good points, but they would be just as appropriate as
feature requests in the OCaml bugtracker (
http://caml.inria.fr/mantis/ ).

I think it should not be too difficult to add a flag to the compiler
to only produce .cmi files out of its source inputs, and not also a
.cmo (or also a .cmx) as is currently done. If you (or anyone
interested in contributing) are interested in providing a patch for
this, I'm more than ready to review the patch to help upstream
integration. I'd consider this "junior job" difficulty.

Integrating this into build systems may be trickier because, as Markus
noted, just asking for the .cmi of the main module of your project
will recursively build .cmi from the .mli of its dependencies, without
checking that the .ml matches them. You could add a phony rule/stamp
.impl.cmi that is handled differently by the build system (basically
like .cmo, except they would be empty).

Your points (2) and (3) are well noted. I've personally been thinking
about an optional -use-timestamp option for ocamlbuild, but devoted my
own contribution time to other things this summer.


Markus Mottl wrote:
> I'm not sure what you are referring to, OCamlBuild does already
> support parallel builds.

While there is support for parallel compilation in ocamlbuild (...
except on Windows, I'm afraid), I've often found out that it doesn't
parallelize much in practice because of current implementation
limitations -- but if you have success stories about this I'd be glad
to hear about them. Improving on this is on the medium-term
ocamlbuild-development TODO list.

On Fri, Sep 6, 2013 at 3:56 PM, Romain Bardou <romain.bardou@inria.fr> wrote:
> Hello list,
>
> As my project grows bigger, it is becoming much less efficient to use
> the type-checker to quickly find places in my code which must be updated
> (e.g. pattern-matching). Here is a wishlist for improvements.
>
> 1) Separate typing and code generation, in ocamlc and in ocamlopt
>
> For instance, provide an option -typing-only which would mean "only
> produce the .cmi but do not produce the .cmo or the .cmx". The compiler
> would only need the .cmi of the dependencies, not their .cmo or .cmx.
> This would make it possible to have a Makefile target, or an Ocamlbuild
> option, to just type.
>
> Also, provide an option -do-not-retype which would mean "if the .cmi
> exists, load it instead of type-checking again". This would allow the
> build process to first type-check (using -typing-only) and then generate
> the code without type-checking again (using -do-not-retype). Of course
> the build system should be very careful to ensure the .cmi is
> up-to-date. This option could also help when compiling both in bytecode
> and in native code. This option is not necessary to just find errors
> quickly, though.
>
> 2) Be able to disable Ocamlbuild's digest mechanism and use dates and
> file sizes instead
>
> If I am not mistaken, this is one of the main reasons why Ocamlbuild is
> slower that make. It does help to prevent useless recompilation, but
> what good does it make to prevent a useless recompilation once in a
> while if it is at the cost of losing a lot of time in all other cases?
> I'm sure it is project-dependent though so it should only be an option,
> say, -do-not-hash, or -comparison-mode dateandsize.
>
> 3) Parallel compilation in Ocamlbuild
>
> Of course it would help but it is not easy to implement so I'm just
> putting it there to be exhaustive.
>
> I think the most important point is the first one, as the other two
> depend on the build system, and they have been debated already. I'm not
> aware of any discussion about separing typing and code generation
> though. What do you think about that?
>
> Cheers,
>
> --
> Romain Bardou
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs

Re: [Caml-list] Accelerating compilation

[Romain Bardou]

Le 06/09/2013 16:55, Markus Mottl a écrit :
> On Fri, Sep 6, 2013 at 9:56 AM, Romain Bardou <romain.bardou@inria.fr> wrote:
>> 1) Separate typing and code generation, in ocamlc and in ocamlopt
>>
>> For instance, provide an option -typing-only which would mean "only
>> produce the .cmi but do not produce the .cmo or the .cmx". The compiler
>> would only need the .cmi of the dependencies, not their .cmo or .cmx.
>> This would make it possible to have a Makefile target, or an Ocamlbuild
>> option, to just type.
> 
> This seems like an interesting suggestion.  Code generation,
> especially for native code, can be quite expensive.  I do use byte
> code compilation during development for that reason, which is somewhat
> faster.

I considered doing that, but my project has some stubs which do not work
in bytecode for some reason (probably fixable). Also, it would mean that
I would have to compile both versions, as the program is too slow to be
used in bytecode, so the bytecode would only be used for quick
type-checking.

> Note, however, that there is one problem with the approach as
> suggested: if you have both an .mli and an .ml file, the build system
> would have to know when to "type" the latter.  In the suggested
> approach there would be no trace of this action, because the .cmi
> would come from the .mli file.  We would need to generate a separate
> dummy file in that case to visibly record the fact that the .ml file
> unifies with the .cmi file.  Or (see below) write out the typed
> abstract syntax tree of the .ml file, which would, of course, also
> have to unify with the .cmi file.

Indeed the build system would need to be tweaked. Another approach would
be to consider that .cmi files depend on .ml files as well, maybe only
if an option -just-type is passed. I'm not sure of the implications.

>> Also, provide an option -do-not-retype which would mean "if the .cmi
>> exists, load it instead of type-checking again". This would allow the
>> build process to first type-check (using -typing-only) and then generate
>> the code without type-checking again (using -do-not-retype). Of course
>> the build system should be very careful to ensure the .cmi is
>> up-to-date. This option could also help when compiling both in bytecode
>> and in native code. This option is not necessary to just find errors
>> quickly, though.
> 
> The .cmi file does not contain enough information, because it only
> contains the signature.  You need the typed AST for proper code
> generation, which might be quite big.  I haven't looked into this, but
> I wouldn't be surprised if the size of that thing could be so large
> that you might prefer type checking again over writing to + reloading
> it from a file.

Ah right I was thinking it contained the whole typed tree for some
reason, which is indeed not the case.

>> 2) Be able to disable Ocamlbuild's digest mechanism and use dates and
>> file sizes instead
>>
>> If I am not mistaken, this is one of the main reasons why Ocamlbuild is
>> slower that make. It does help to prevent useless recompilation, but
>> what good does it make to prevent a useless recompilation once in a
>> while if it is at the cost of losing a lot of time in all other cases?
>> I'm sure it is project-dependent though so it should only be an option,
>> say, -do-not-hash, or -comparison-mode dateandsize.
> 
> The problem here is that not all Unix-filesystems support sub-second
> resolution in timestamps.  So if you build a file and change a
> character in it within one second, the change may go unnoticed, i.e.
> required recompilation doesn't happen.  That's why hashing provides
> much stronger guarantees.  But I think this is not a big problem in
> practice so I'd support such a flag in ocamlbuild.
> 
>> 3) Parallel compilation in Ocamlbuild
>>
>> Of course it would help but it is not easy to implement so I'm just
>> putting it there to be exhaustive.
> 
> I'm not sure what you are referring to, OCamlBuild does already
> support parallel builds.

Does it? I actually thought the -j option was ignored.

I just did a quick test and I gain about 5 seconds with -j on a 1min15
build (I had cleaned, recompiled and recleaned before so that caching by
the file system would not impact the result too much), so it does seem
to be a *little* faster :)

Cheers,

-- 
Romain Bardou

Re: [Caml-list] Accelerating compilation

[Gabriel Scherer]

One option that I think would be really help is a way to preserve
separate compilation with ocamlopt, by not relying on .cmx of
dependencies for optimization. That could preserve a good incremental
development workflow and make most of those issues, I think, moot.

(I also hear a suggestion that editors with incremental type feedback,
such as Merlin and Typerex2, do help.)

On Fri, Sep 6, 2013 at 5:19 PM, Romain Bardou <romain.bardou@inria.fr> wrote:
> Le 06/09/2013 16:55, Markus Mottl a écrit :
>> On Fri, Sep 6, 2013 at 9:56 AM, Romain Bardou <romain.bardou@inria.fr> wrote:
>>> 1) Separate typing and code generation, in ocamlc and in ocamlopt
>>>
>>> For instance, provide an option -typing-only which would mean "only
>>> produce the .cmi but do not produce the .cmo or the .cmx". The compiler
>>> would only need the .cmi of the dependencies, not their .cmo or .cmx.
>>> This would make it possible to have a Makefile target, or an Ocamlbuild
>>> option, to just type.
>>
>> This seems like an interesting suggestion.  Code generation,
>> especially for native code, can be quite expensive.  I do use byte
>> code compilation during development for that reason, which is somewhat
>> faster.
>
> I considered doing that, but my project has some stubs which do not work
> in bytecode for some reason (probably fixable). Also, it would mean that
> I would have to compile both versions, as the program is too slow to be
> used in bytecode, so the bytecode would only be used for quick
> type-checking.
>
>> Note, however, that there is one problem with the approach as
>> suggested: if you have both an .mli and an .ml file, the build system
>> would have to know when to "type" the latter.  In the suggested
>> approach there would be no trace of this action, because the .cmi
>> would come from the .mli file.  We would need to generate a separate
>> dummy file in that case to visibly record the fact that the .ml file
>> unifies with the .cmi file.  Or (see below) write out the typed
>> abstract syntax tree of the .ml file, which would, of course, also
>> have to unify with the .cmi file.
>
> Indeed the build system would need to be tweaked. Another approach would
> be to consider that .cmi files depend on .ml files as well, maybe only
> if an option -just-type is passed. I'm not sure of the implications.
>
>>> Also, provide an option -do-not-retype which would mean "if the .cmi
>>> exists, load it instead of type-checking again". This would allow the
>>> build process to first type-check (using -typing-only) and then generate
>>> the code without type-checking again (using -do-not-retype). Of course
>>> the build system should be very careful to ensure the .cmi is
>>> up-to-date. This option could also help when compiling both in bytecode
>>> and in native code. This option is not necessary to just find errors
>>> quickly, though.
>>
>> The .cmi file does not contain enough information, because it only
>> contains the signature.  You need the typed AST for proper code
>> generation, which might be quite big.  I haven't looked into this, but
>> I wouldn't be surprised if the size of that thing could be so large
>> that you might prefer type checking again over writing to + reloading
>> it from a file.
>
> Ah right I was thinking it contained the whole typed tree for some
> reason, which is indeed not the case.
>
>>> 2) Be able to disable Ocamlbuild's digest mechanism and use dates and
>>> file sizes instead
>>>
>>> If I am not mistaken, this is one of the main reasons why Ocamlbuild is
>>> slower that make. It does help to prevent useless recompilation, but
>>> what good does it make to prevent a useless recompilation once in a
>>> while if it is at the cost of losing a lot of time in all other cases?
>>> I'm sure it is project-dependent though so it should only be an option,
>>> say, -do-not-hash, or -comparison-mode dateandsize.
>>
>> The problem here is that not all Unix-filesystems support sub-second
>> resolution in timestamps.  So if you build a file and change a
>> character in it within one second, the change may go unnoticed, i.e.
>> required recompilation doesn't happen.  That's why hashing provides
>> much stronger guarantees.  But I think this is not a big problem in
>> practice so I'd support such a flag in ocamlbuild.
>>
>>> 3) Parallel compilation in Ocamlbuild
>>>
>>> Of course it would help but it is not easy to implement so I'm just
>>> putting it there to be exhaustive.
>>
>> I'm not sure what you are referring to, OCamlBuild does already
>> support parallel builds.
>
> Does it? I actually thought the -j option was ignored.
>
> I just did a quick test and I gain about 5 seconds with -j on a 1min15
> build (I had cleaned, recompiled and recleaned before so that caching by
> the file system would not impact the result too much), so it does seem
> to be a *little* faster :)
>
> Cheers,
>
> --
> Romain Bardou
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs

Re: [Caml-list] Accelerating compilation

[Romain Bardou]

Le 06/09/2013 17:18, Gabriel Scherer a écrit :
> Those are all good points, but they would be just as appropriate as
> feature requests in the OCaml bugtracker (
> http://caml.inria.fr/mantis/ ).

Actually Xavier Clerc noticed me that there is already a related feature
request:
http://caml.inria.fr/mantis/view.php?id=6102

> I think it should not be too difficult to add a flag to the compiler
> to only produce .cmi files out of its source inputs, and not also a
> .cmo (or also a .cmx) as is currently done. If you (or anyone
> interested in contributing) are interested in providing a patch for
> this, I'm more than ready to review the patch to help upstream
> integration. I'd consider this "junior job" difficulty.

Thanks for your review proposal. I might actually consider doing this
the day it becomes too much of a pain. FR#6102 actually have a patch
attached which looks like a nice starting point.

> 
> Integrating this into build systems may be trickier because, as Markus
> noted, just asking for the .cmi of the main module of your project
> will recursively build .cmi from the .mli of its dependencies, without
> checking that the .ml matches them. You could add a phony rule/stamp
> .impl.cmi that is handled differently by the build system (basically
> like .cmo, except they would be empty).
> 
> Your points (2) and (3) are well noted. I've personally been thinking
> about an optional -use-timestamp option for ocamlbuild, but devoted my
> own contribution time to other things this summer.
> 
> 
> Markus Mottl wrote:
>> I'm not sure what you are referring to, OCamlBuild does already
>> support parallel builds.
> 
> While there is support for parallel compilation in ocamlbuild (...
> except on Windows, I'm afraid), I've often found out that it doesn't
> parallelize much in practice because of current implementation
> limitations -- but if you have success stories about this I'd be glad
> to hear about them. Improving on this is on the medium-term
> ocamlbuild-development TODO list.
> 
> On Fri, Sep 6, 2013 at 3:56 PM, Romain Bardou <romain.bardou@inria.fr> wrote:
>> Hello list,
>>
>> As my project grows bigger, it is becoming much less efficient to use
>> the type-checker to quickly find places in my code which must be updated
>> (e.g. pattern-matching). Here is a wishlist for improvements.
>>
>> 1) Separate typing and code generation, in ocamlc and in ocamlopt
>>
>> For instance, provide an option -typing-only which would mean "only
>> produce the .cmi but do not produce the .cmo or the .cmx". The compiler
>> would only need the .cmi of the dependencies, not their .cmo or .cmx.
>> This would make it possible to have a Makefile target, or an Ocamlbuild
>> option, to just type.
>>
>> Also, provide an option -do-not-retype which would mean "if the .cmi
>> exists, load it instead of type-checking again". This would allow the
>> build process to first type-check (using -typing-only) and then generate
>> the code without type-checking again (using -do-not-retype). Of course
>> the build system should be very careful to ensure the .cmi is
>> up-to-date. This option could also help when compiling both in bytecode
>> and in native code. This option is not necessary to just find errors
>> quickly, though.
>>
>> 2) Be able to disable Ocamlbuild's digest mechanism and use dates and
>> file sizes instead
>>
>> If I am not mistaken, this is one of the main reasons why Ocamlbuild is
>> slower that make. It does help to prevent useless recompilation, but
>> what good does it make to prevent a useless recompilation once in a
>> while if it is at the cost of losing a lot of time in all other cases?
>> I'm sure it is project-dependent though so it should only be an option,
>> say, -do-not-hash, or -comparison-mode dateandsize.
>>
>> 3) Parallel compilation in Ocamlbuild
>>
>> Of course it would help but it is not easy to implement so I'm just
>> putting it there to be exhaustive.
>>
>> I think the most important point is the first one, as the other two
>> depend on the build system, and they have been debated already. I'm not
>> aware of any discussion about separing typing and code generation
>> though. What do you think about that?
>>
>> Cheers,
>>
>> --
>> Romain Bardou
>>
>> --
>> Caml-list mailing list.  Subscription management and archives:
>> https://sympa.inria.fr/sympa/arc/caml-list
>> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>> Bug reports: http://caml.inria.fr/bin/caml-bugs
> 

Re: [Caml-list] Accelerating compilation

[Alain Frisch]

On 09/06/2013 05:27 PM, Gabriel Scherer wrote:
> One option that I think would be really help is a way to preserve
> separate compilation with ocamlopt, by not relying on .cmx of
> dependencies for optimization. That could preserve a good incremental
> development workflow and make most of those issues, I think, moot.

Please note, however, that .cmx files are required not only for 
cross-module optimizations, but also to support -pack.

-- Alain

Re: [Caml-list] Accelerating compilation

[Markus Mottl]

On Fri, Sep 6, 2013 at 11:18 AM, Gabriel Scherer
<gabriel.scherer@gmail.com> wrote:
> While there is support for parallel compilation in ocamlbuild (...
> except on Windows, I'm afraid), I've often found out that it doesn't
> parallelize much in practice because of current implementation
> limitations -- but if you have success stories about this I'd be glad
> to hear about them. Improving on this is on the medium-term
> ocamlbuild-development TODO list.

I definitely do see significant speedups on some projects, e.g.
building Lacaml.  But I don't see that on other projects even though
the dependency graph would allow for it.  I guess it's still work in
progress...

Regards,
Markus

-- 
Markus Mottl        http://www.ocaml.info        markus.mottl@gmail.com

Re: [Caml-list] Accelerating compilation

[Xavier Leroy]

On 06/09/13 15:56, Romain Bardou wrote:

> 1) Separate typing and code generation, in ocamlc and in ocamlopt
> For instance, provide an option -typing-only which would mean "only
> produce the .cmi but do not produce the .cmo or the .cmx". 

I strongly suspect that bytecode generation takes much less time than
parsing and typechecking, so you wouldn't gain much compared to simply
compiling to bytecode.

Some obvious recommendations:

- Make sure the .opt compilers (ocamlc.opt, ocamlopt.opt) are
  installed.  They are significantly faster than their non-.opt
  counterparts, and some Linux distros package the .opt compilers
  separately and don't install them by default.

- Put your project on a fast disk, ideally a SSD.  ocamlc can be I/O
  bound sometimes.  The only times where I observe somewhat slow
  compilations is when I work in an NFS partition mounted from our
  super-reliable-but-rather-slow NAS.

> 2) Be able to disable Ocamlbuild's digest mechanism and use dates and
> file sizes instead
> 
> If I am not mistaken, this is one of the main reasons why Ocamlbuild is
> slower that make.

It's one possible reason among others, e.g. that ocamlbuild runs
ocamldep very often.  Again, some profiling would help understanding
the bottleneck(s).  But this is a reasonable suggestion.

> 3) Parallel compilation in Ocamlbuild
> Of course it would help but it is not easy to implement so I'm just
> putting it there to be exhaustive.

As others said, ocamlbuild has some support for parallelism, but
it's much less effective than "make -j".  One reason could be that
ocamlbuild discovers dependencies as it goes, so it's doing online
scheduling, in contrast with make's offline scheduling.  But within
the online scheduling approach, there is certainly room for
improvement.  Suggestions and patches welcome.

- Xavier Leroy

Re: [Caml-list] Accelerating compilation

[Martin Jambon]

On 09/06/2013 06:56 AM, Romain Bardou wrote:
> Hello list,
>
> As my project grows bigger, it is becoming much less efficient to use
> the type-checker to quickly find places in my code which must be updated
> (e.g. pattern-matching). Here is a wishlist for improvements.
>
> 1) Separate typing and code generation, in ocamlc and in ocamlopt
>
> For instance, provide an option -typing-only which would mean "only
> produce the .cmi but do not produce the .cmo or the .cmx". The compiler
> would only need the .cmi of the dependencies, not their .cmo or .cmx.
> This would make it possible to have a Makefile target, or an Ocamlbuild
> option, to just type.
>
> Also, provide an option -do-not-retype which would mean "if the .cmi
> exists, load it instead of type-checking again". This would allow the
> build process to first type-check (using -typing-only) and then generate
> the code without type-checking again (using -do-not-retype). Of course
> the build system should be very careful to ensure the .cmi is
> up-to-date. This option could also help when compiling both in bytecode
> and in native code. This option is not necessary to just find errors
> quickly, though.
>
> 2) Be able to disable Ocamlbuild's digest mechanism and use dates and
> file sizes instead
>
> If I am not mistaken, this is one of the main reasons why Ocamlbuild is
> slower that make.

It would be interesting to know the size of your project and the time it 
takes to build it.

Things that have resulted for in significant speed improvements in the 
recent past (< 2 years):

1. using omake instead of ocamlbuild
2. bytecode camlp4 preprocessing < native camlp4 < camlp5 < nothing
3. building only one executable

I don't have much information on ocamlbuild other than I couldn't make 
it perform as expected (and I don't like the idea of having to write 
build instruction in OCaml anyway). I know that omake does what one 
would expect in terms on parallelism and caching.

Point (2) is easily checked with 'top'.

On my machine using an external SSD, what takes the most time is IOs at 
linking time when using a good number of libraries. I don't know if it's 
preventable, if it's a bad interaction between ocamlfind and the 
compilers, or something else. What I do now is build everything into a 
single executable with subcommands, which is handy anyway so why not.


 > It does help to prevent useless recompilation, but
> what good does it make to prevent a useless recompilation once in a
> while if it is at the cost of losing a lot of time in all other cases?
> I'm sure it is project-dependent though so it should only be an option,
> say, -do-not-hash, or -comparison-mode dateandsize.
>
> 3) Parallel compilation in Ocamlbuild
>
> Of course it would help but it is not easy to implement so I'm just
> putting it there to be exhaustive.
>
> I think the most important point is the first one, as the other two
> depend on the build system, and they have been debated already. I'm not
> aware of any discussion about separing typing and code generation
> though. What do you think about that?
>
> Cheers,
>

Re: [Caml-list] Accelerating compilation

[Fabrice Le Fessant]

On Fri, Sep 6, 2013 at 5:20 PM, Romain Bardou <romain.bardou@inria.fr> wrote:
>>> 3) Parallel compilation in Ocamlbuild
>>>
>>> Of course it would help but it is not easy to implement so I'm just
>>> putting it there to be exhaustive.
>>
>> I'm not sure what you are referring to, OCamlBuild does already
>> support parallel builds.
>
> Does it? I actually thought the -j option was ignored.
>
> I just did a quick test and I gain about 5 seconds with -j on a 1min15
> build (I had cleaned, recompiled and recleaned before so that caching by
> the file system would not impact the result too much), so it does seem
> to be a *little* faster :)

FWIW, I recently compiled Merlin with both ocamlbuild and ocp-build,
on a quad-core with "-j 10" for both (the link to the ocp-build
description file is in the latest OCamlPro's report), ocamlbuild needs
13s where ocp-build only needs 4s to compile everything.

As Xavier suggested, I think the main problem is that ocamlbuild must
discover the files to compile dynamically, starting from the last
module, so it cannot parallelize much. On the contrary, ocp-build has
the list of files in the description of the project, so for example,
it can call ocamldep on all of them almost at the same time.

ocp-build has an option to choose between digest and (timestamp +
inode), the default. When I benchmarked the two modes, I could notice
a small difference, but not more then 5%, so the digests are unlikely
to be responsible for the problem.

Finally, another optimization that could be considered for ocamlbuild
is to call camlp4 only once per file, and then call ocamldep, ocamlc
and ocamlopt on the preprocessed file. I could notice a huge
difference on Core, for example, when this optimization was
implemented first in ocp-build (about a 50% speed-up, if I remember
well).

--Fabrice

Re: [Caml-list] Accelerating compilation

[Matej Kosik]

On 06/09/13 16:19, Romain Bardou wrote:
> 
> I considered doing that, but my project has some stubs which do not work
> in bytecode for some reason (probably fixable). Also, it would mean that
> I would have to compile both versions, as the program is too slow to be
> used in bytecode, so the bytecode would only be used for quick
> type-checking.

There is a crucial difference between "ocamlc" and "ocamlopt".

"ocamlc" is fine because the recompilation time typically depends only on the number of changed files
(these simply have to be recompiled and then the whole thing can be linked together).

"ocamlopt" is a terrible choice for iterative recompilation time
typically *depends on the size of the project*
(it is necessary to recompile all the changed files as well as all unmodified files that simply happen to be after the changed files in the linking list)

The difference is huge.

Re: [Caml-list] Accelerating compilation

[Wojciech Meyer]

Xavier Leroy <Xavier.Leroy@inria.fr> writes:

>> 3) Parallel compilation in Ocamlbuild
>> Of course it would help but it is not easy to implement so I'm just
>> putting it there to be exhaustive.
>
> As others said, ocamlbuild has some support for parallelism, but
> it's much less effective than "make -j".  One reason could be that
> ocamlbuild discovers dependencies as it goes, so it's doing online
> scheduling, in contrast with make's offline scheduling.  But within
> the online scheduling approach, there is certainly room for
> improvement.  Suggestions and patches welcome.

That's exactly what happens and is described here:

  http://caml.inria.fr/mantis/view.php?id=5754

Unfortunately it's not that easy to fix. Possible solutions:

- do the scheduling after the project was build for the first time and
  then:
  * for each of the targets note the products and store it in the
    database
  * when the target changes, re-build the database for this target and
    it's products

- provide simpler and less dynamic rules for ocamlbuild:
  * that actually don't run the Caml code, but do only invoke shell
    commands with some substitution
  * try to convert existing rules to the static ones

Both solutions guarantee perfect scheduling.

Wojciech

Re: [Caml-list] Accelerating compilation

[Jacques-Pascal Deplaix]

Personally, I'd prefer the second solution because the former will not 
do anything during the first build (if I'm not wrong) and this is quite 
problematic for end user (for installation).

On 09/07/2013 09:13 PM, Wojciech Meyer wrote:
> Xavier Leroy <Xavier.Leroy@inria.fr> writes:
>
>>> 3) Parallel compilation in Ocamlbuild
>>> Of course it would help but it is not easy to implement so I'm just
>>> putting it there to be exhaustive.
>> As others said, ocamlbuild has some support for parallelism, but
>> it's much less effective than "make -j".  One reason could be that
>> ocamlbuild discovers dependencies as it goes, so it's doing online
>> scheduling, in contrast with make's offline scheduling.  But within
>> the online scheduling approach, there is certainly room for
>> improvement.  Suggestions and patches welcome.
> That's exactly what happens and is described here:
>
>    http://caml.inria.fr/mantis/view.php?id=5754
>
> Unfortunately it's not that easy to fix. Possible solutions:
>
> - do the scheduling after the project was build for the first time and
>    then:
>    * for each of the targets note the products and store it in the
>      database
>    * when the target changes, re-build the database for this target and
>      it's products
>
> - provide simpler and less dynamic rules for ocamlbuild:
>    * that actually don't run the Caml code, but do only invoke shell
>      commands with some substitution
>    * try to convert existing rules to the static ones
>
> Both solutions guarantee perfect scheduling.
>
> Wojciech
>

Re: [Caml-list] Accelerating compilation

[Markus Mottl]

On Sat, Sep 7, 2013 at 5:42 PM, Jacques-Pascal Deplaix
<jp.deplaix@gmail.com> wrote:
> Personally, I'd prefer the second solution because the former will not do
> anything during the first build (if I'm not wrong) and this is quite
> problematic for end user (for installation).

I actually prefer the first solution.  You can always ship
distributions including the file storing the dependency cache.

-- 
Markus Mottl        http://www.ocaml.info        markus.mottl@gmail.com

Re: [Caml-list] Accelerating compilation

[Francois Berenger]

On 09/07/2013 12:19 AM, Romain Bardou wrote:
> Le 06/09/2013 16:55, Markus Mottl a écrit :
>> On Fri, Sep 6, 2013 at 9:56 AM, Romain Bardou <romain.bardou@inria.fr> wrote:
>>> 1) Separate typing and code generation, in ocamlc and in ocamlopt
>>>
>>> For instance, provide an option -typing-only which would mean "only
>>> produce the .cmi but do not produce the .cmo or the .cmx". The compiler
>>> would only need the .cmi of the dependencies, not their .cmo or .cmx.
>>> This would make it possible to have a Makefile target, or an Ocamlbuild
>>> option, to just type.
>>
>> This seems like an interesting suggestion.  Code generation,
>> especially for native code, can be quite expensive.  I do use byte
>> code compilation during development for that reason, which is somewhat
>> faster.
>
> I considered doing that, but my project has some stubs which do not work
> in bytecode for some reason (probably fixable). Also, it would mean that
> I would have to compile both versions, as the program is too slow to be
> used in bytecode, so the bytecode would only be used for quick
> type-checking.
>
>> Note, however, that there is one problem with the approach as
>> suggested: if you have both an .mli and an .ml file, the build system
>> would have to know when to "type" the latter.  In the suggested
>> approach there would be no trace of this action, because the .cmi
>> would come from the .mli file.  We would need to generate a separate
>> dummy file in that case to visibly record the fact that the .ml file
>> unifies with the .cmi file.  Or (see below) write out the typed
>> abstract syntax tree of the .ml file, which would, of course, also
>> have to unify with the .cmi file.
>
> Indeed the build system would need to be tweaked. Another approach would
> be to consider that .cmi files depend on .ml files as well, maybe only
> if an option -just-type is passed. I'm not sure of the implications.
>
>>> Also, provide an option -do-not-retype which would mean "if the .cmi
>>> exists, load it instead of type-checking again". This would allow the
>>> build process to first type-check (using -typing-only) and then generate
>>> the code without type-checking again (using -do-not-retype). Of course
>>> the build system should be very careful to ensure the .cmi is
>>> up-to-date. This option could also help when compiling both in bytecode
>>> and in native code. This option is not necessary to just find errors
>>> quickly, though.
>>
>> The .cmi file does not contain enough information, because it only
>> contains the signature.  You need the typed AST for proper code
>> generation, which might be quite big.  I haven't looked into this, but
>> I wouldn't be surprised if the size of that thing could be so large
>> that you might prefer type checking again over writing to + reloading
>> it from a file.
>
> Ah right I was thinking it contained the whole typed tree for some
> reason, which is indeed not the case.
>
>>> 2) Be able to disable Ocamlbuild's digest mechanism and use dates and
>>> file sizes instead
>>>
>>> If I am not mistaken, this is one of the main reasons why Ocamlbuild is
>>> slower that make. It does help to prevent useless recompilation, but
>>> what good does it make to prevent a useless recompilation once in a
>>> while if it is at the cost of losing a lot of time in all other cases?
>>> I'm sure it is project-dependent though so it should only be an option,
>>> say, -do-not-hash, or -comparison-mode dateandsize.
>>
>> The problem here is that not all Unix-filesystems support sub-second
>> resolution in timestamps.  So if you build a file and change a
>> character in it within one second, the change may go unnoticed, i.e.
>> required recompilation doesn't happen.  That's why hashing provides
>> much stronger guarantees.  But I think this is not a big problem in
>> practice so I'd support such a flag in ocamlbuild.
>>
>>> 3) Parallel compilation in Ocamlbuild
>>>
>>> Of course it would help but it is not easy to implement so I'm just
>>> putting it there to be exhaustive.
>>
>> I'm not sure what you are referring to, OCamlBuild does already
>> support parallel builds.
>
> Does it? I actually thought the -j option was ignored.

No no, it is not ignored. It is just a joke option: it is just
here to make you laugh.

> I just did a quick test and I gain about 5 seconds with -j on a 1min15
> build (I had cleaned, recompiled and recleaned before so that caching by
> the file system would not impact the result too much), so it does seem
> to be a *little* faster :)
>
> Cheers,
>

Re: [Caml-list] Accelerating compilation

[Romain Bardou]

Le 06/09/2013 22:51, Fabrice Le Fessant a écrit :
> On Fri, Sep 6, 2013 at 5:20 PM, Romain Bardou <romain.bardou@inria.fr> wrote:
>>>> 3) Parallel compilation in Ocamlbuild
>>>>
>>>> Of course it would help but it is not easy to implement so I'm just
>>>> putting it there to be exhaustive.
>>>
>>> I'm not sure what you are referring to, OCamlBuild does already
>>> support parallel builds.
>>
>> Does it? I actually thought the -j option was ignored.
>>
>> I just did a quick test and I gain about 5 seconds with -j on a 1min15
>> build (I had cleaned, recompiled and recleaned before so that caching by
>> the file system would not impact the result too much), so it does seem
>> to be a *little* faster :)
> 
> FWIW, I recently compiled Merlin with both ocamlbuild and ocp-build,
> on a quad-core with "-j 10" for both (the link to the ocp-build
> description file is in the latest OCamlPro's report), ocamlbuild needs
> 13s where ocp-build only needs 4s to compile everything.
> 
> As Xavier suggested, I think the main problem is that ocamlbuild must
> discover the files to compile dynamically, starting from the last
> module, so it cannot parallelize much. On the contrary, ocp-build has
> the list of files in the description of the project, so for example,
> it can call ocamldep on all of them almost at the same time.
> 
> ocp-build has an option to choose between digest and (timestamp +
> inode), the default. When I benchmarked the two modes, I could notice
> a small difference, but not more then 5%, so the digests are unlikely
> to be responsible for the problem.
> 
> Finally, another optimization that could be considered for ocamlbuild
> is to call camlp4 only once per file, and then call ocamldep, ocamlc
> and ocamlopt on the preprocessed file. I could notice a huge
> difference on Core, for example, when this optimization was
> implemented first in ocp-build (about a 50% speed-up, if I remember
> well).

I was hoping to hear from ocp-build :) Interesting observations here. I
would have thought the digests would make more of a difference. Good to
know!

Cheers,

-- 
Romain Bardou

Re: [Caml-list] Accelerating compilation

[Romain Bardou]

Le 06/09/2013 18:30, Xavier Leroy a écrit :
> On 06/09/13 15:56, Romain Bardou wrote:
> 
>> 1) Separate typing and code generation, in ocamlc and in ocamlopt
>> For instance, provide an option -typing-only which would mean "only
>> produce the .cmi but do not produce the .cmo or the .cmx". 
> 
> I strongly suspect that bytecode generation takes much less time than
> parsing and typechecking, so you wouldn't gain much compared to simply
> compiling to bytecode.

Bytecode compilation is indeed much faster, I learned that with Melt,
which compiles an OCaml file which itself generates a .tex file. I used
to compile the OCaml file with ocamlopt. I gained a lot by using ocamlc
instead. The executable takes more time to execute but it is still much
faster overall.

However, compiling to bytecode is not always an option. In my current
case, it fails at link-time because of some C stubs (I could probably
fix this). Moreover, I compile a model-checker and speed is obviously
needed in that case. Maybe I could isolate the model-checker part…

> Some obvious recommendations:
> 
> - Make sure the .opt compilers (ocamlc.opt, ocamlopt.opt) are
>   installed.  They are significantly faster than their non-.opt
>   counterparts, and some Linux distros package the .opt compilers
>   separately and don't install them by default.

Indeed, I benchmarked this out of curiosity and it does accelerate the
compilation time by a factor of about 3.

> - Put your project on a fast disk, ideally a SSD.  ocamlc can be I/O
>   bound sometimes.  The only times where I observe somewhat slow
>   compilations is when I work in an NFS partition mounted from our
>   super-reliable-but-rather-slow NAS.
> 
>> 2) Be able to disable Ocamlbuild's digest mechanism and use dates and
>> file sizes instead
>>
>> If I am not mistaken, this is one of the main reasons why Ocamlbuild is
>> slower that make.
> 
> It's one possible reason among others, e.g. that ocamlbuild runs
> ocamldep very often.  Again, some profiling would help understanding
> the bottleneck(s).  But this is a reasonable suggestion.
> 
>> 3) Parallel compilation in Ocamlbuild
>> Of course it would help but it is not easy to implement so I'm just
>> putting it there to be exhaustive.
> 
> As others said, ocamlbuild has some support for parallelism, but
> it's much less effective than "make -j".  One reason could be that
> ocamlbuild discovers dependencies as it goes, so it's doing online
> scheduling, in contrast with make's offline scheduling.  But within
> the online scheduling approach, there is certainly room for
> improvement.  Suggestions and patches welcome.

-- 
Romain Bardou

Re: [Caml-list] Accelerating compilation

[Romain Bardou]

Le 06/09/2013 20:45, Martin Jambon a écrit :
> On 09/06/2013 06:56 AM, Romain Bardou wrote:
>> Hello list,
>>
>> As my project grows bigger, it is becoming much less efficient to use
>> the type-checker to quickly find places in my code which must be updated
>> (e.g. pattern-matching). Here is a wishlist for improvements.
>>
>> 1) Separate typing and code generation, in ocamlc and in ocamlopt
>>
>> For instance, provide an option -typing-only which would mean "only
>> produce the .cmi but do not produce the .cmo or the .cmx". The compiler
>> would only need the .cmi of the dependencies, not their .cmo or .cmx.
>> This would make it possible to have a Makefile target, or an Ocamlbuild
>> option, to just type.
>>
>> Also, provide an option -do-not-retype which would mean "if the .cmi
>> exists, load it instead of type-checking again". This would allow the
>> build process to first type-check (using -typing-only) and then generate
>> the code without type-checking again (using -do-not-retype). Of course
>> the build system should be very careful to ensure the .cmi is
>> up-to-date. This option could also help when compiling both in bytecode
>> and in native code. This option is not necessary to just find errors
>> quickly, though.
>>
>> 2) Be able to disable Ocamlbuild's digest mechanism and use dates and
>> file sizes instead
>>
>> If I am not mistaken, this is one of the main reasons why Ocamlbuild is
>> slower that make.
> 
> It would be interesting to know the size of your project and the time it
> takes to build it.

There are about 30K lines of code in the core of the project, which is
well-split in a rather large number of files (Ocamlbuild reports about
400 targets). It takes about 30s to compile in native mode using the
native compilers.

It might seem very small compared to say, Coq :) But still, having to
wait ~10s just to find the next pattern-matching to fix is already
annoying and it will only get worse.

> Things that have resulted for in significant speed improvements in the
> recent past (< 2 years):
> 
> 1. using omake instead of ocamlbuild
> 2. bytecode camlp4 preprocessing < native camlp4 < camlp5 < nothing
> 3. building only one executable

There are indeed several executables and linking them is one bottleneck.
Some of them I can't merge, but some of them I maybe could… Thanks for
your input!

> 
> I don't have much information on ocamlbuild other than I couldn't make
> it perform as expected (and I don't like the idea of having to write
> build instruction in OCaml anyway). I know that omake does what one
> would expect in terms on parallelism and caching.
> 
> Point (2) is easily checked with 'top'.
> 
> On my machine using an external SSD, what takes the most time is IOs at
> linking time when using a good number of libraries. I don't know if it's
> preventable, if it's a bad interaction between ocamlfind and the
> compilers, or something else. What I do now is build everything into a
> single executable with subcommands, which is handy anyway so why not.
> 
> 
>> It does help to prevent useless recompilation, but
>> what good does it make to prevent a useless recompilation once in a
>> while if it is at the cost of losing a lot of time in all other cases?
>> I'm sure it is project-dependent though so it should only be an option,
>> say, -do-not-hash, or -comparison-mode dateandsize.
>>
>> 3) Parallel compilation in Ocamlbuild
>>
>> Of course it would help but it is not easy to implement so I'm just
>> putting it there to be exhaustive.
>>
>> I think the most important point is the first one, as the other two
>> depend on the build system, and they have been debated already. I'm not
>> aware of any discussion about separing typing and code generation
>> though. What do you think about that?
>>
>> Cheers,
>>
> 
> 

-- 
Romain Bardou

Re: [Caml-list] Accelerating compilation

[Alain Frisch]

On 09/06/2013 06:30 PM, Xavier Leroy wrote:
> - Make sure the .opt compilers (ocamlc.opt, ocamlopt.opt) are
>    installed.  They are significantly faster than their non-.opt
>    counterparts, and some Linux distros package the .opt compilers
>    separately and don't install them by default.

Shouldn't we change the official installation procedure (from sources), 
so that the .opt compilers are installed as "ocamlc" and "ocamlopt" (not 
"ocamlc.opt", "ocamlopt.opt") when they are available?  And suggest to 
do the same for distribution packages?   It would make the life of other 
projects easier if they could just call "ocamlc" without having to check 
if "ocamlc.opt" is available or not.

Or are there compelling arguments in favor of using the bytecode version 
of ocamlc/ocamlopt?


Alain

Re: [Caml-list] Accelerating compilation

[Francois Berenger]

On 09/09/2013 05:25 PM, Alain Frisch wrote:
> On 09/06/2013 06:30 PM, Xavier Leroy wrote:
>> - Make sure the .opt compilers (ocamlc.opt, ocamlopt.opt) are
>>    installed.  They are significantly faster than their non-.opt
>>    counterparts, and some Linux distros package the .opt compilers
>>    separately and don't install them by default.
>
> Shouldn't we change the official installation procedure (from sources),
> so that the .opt compilers are installed as "ocamlc" and "ocamlopt" (not
> "ocamlc.opt", "ocamlopt.opt") when they are available?  And suggest to
> do the same for distribution packages?   It would make the life of other
> projects easier if they could just call "ocamlc" without having to check
> if "ocamlc.opt" is available or not.

I vote for, especially if ocamldoc is not forgotten in the process
(its .opt version is 6 times faster on batteries' ocamldoc).

Re: [Caml-list] Accelerating compilation

[Francois Berenger]

On 09/09/2013 05:15 PM, Romain Bardou wrote:
> Le 06/09/2013 20:45, Martin Jambon a écrit :
>> On 09/06/2013 06:56 AM, Romain Bardou wrote:
>>> Hello list,
>>>
>>> As my project grows bigger, it is becoming much less efficient to use
>>> the type-checker to quickly find places in my code which must be updated
>>> (e.g. pattern-matching). Here is a wishlist for improvements.
>>>
>>> 1) Separate typing and code generation, in ocamlc and in ocamlopt
>>>
>>> For instance, provide an option -typing-only which would mean "only
>>> produce the .cmi but do not produce the .cmo or the .cmx". The compiler
>>> would only need the .cmi of the dependencies, not their .cmo or .cmx.
>>> This would make it possible to have a Makefile target, or an Ocamlbuild
>>> option, to just type.
>>>
>>> Also, provide an option -do-not-retype which would mean "if the .cmi
>>> exists, load it instead of type-checking again". This would allow the
>>> build process to first type-check (using -typing-only) and then generate
>>> the code without type-checking again (using -do-not-retype). Of course
>>> the build system should be very careful to ensure the .cmi is
>>> up-to-date. This option could also help when compiling both in bytecode
>>> and in native code. This option is not necessary to just find errors
>>> quickly, though.
>>>
>>> 2) Be able to disable Ocamlbuild's digest mechanism and use dates and
>>> file sizes instead
>>>
>>> If I am not mistaken, this is one of the main reasons why Ocamlbuild is
>>> slower that make.
>>
>> It would be interesting to know the size of your project and the time it
>> takes to build it.
>
> There are about 30K lines of code in the core of the project, which is
> well-split in a rather large number of files (Ocamlbuild reports about
> 400 targets). It takes about 30s to compile in native mode using the
> native compilers.
>
> It might seem very small compared to say, Coq :) But still, having to
> wait ~10s just to find the next pattern-matching to fix is already
> annoying and it will only get worse.

Isn't the merlin tool able to do that: show compilation errors as you type?

I don't use merlin that much but thought it was.

Re: [Caml-list] Accelerating compilation

[Thomas Refis]

2013/9/9 Francois Berenger <berenger@riken.jp>:
> On 09/09/2013 05:15 PM, Romain Bardou wrote:
>> It might seem very small compared to say, Coq :) But still, having to
>> wait ~10s just to find the next pattern-matching to fix is already
>> annoying and it will only get worse.
>
>
> Isn't the merlin tool able to do that: show compilation errors as you type?

It is. :)

Re: [Caml-list] Accelerating compilation

[Anil Madhavapeddy]

On 9 Sep 2013, at 09:25, Alain Frisch <alain@frisch.fr> wrote:

> On 09/06/2013 06:30 PM, Xavier Leroy wrote:
>> - Make sure the .opt compilers (ocamlc.opt, ocamlopt.opt) are
>>   installed.  They are significantly faster than their non-.opt
>>   counterparts, and some Linux distros package the .opt compilers
>>   separately and don't install them by default.
> 
> Shouldn't we change the official installation procedure (from sources), so that the .opt compilers are installed as "ocamlc" and "ocamlopt" (not "ocamlc.opt", "ocamlopt.opt") when they are available?  And suggest to do the same for distribution packages?   It would make the life of other projects easier if they could just call "ocamlc" without having to check if "ocamlc.opt" is available or not.
> 
> Or are there compelling arguments in favor of using the bytecode version of ocamlc/ocamlopt?

The only reason I'm aware of for not exclusively using the native code version of tools is for camlp4 -- most extensions don't package up .cmxs files for Natdynlink.  It should be safe to swap just the compiler binaries.

I'll run a regression test on OPAM with this divert in place to see if anything new breaks  (after the current 4.01.0RC2 runs are done and triaged, that is).

-anil

Re: [Caml-list] Accelerating compilation

[Adrien Nader]

On Mon, Sep 09, 2013, Alain Frisch wrote:
> On 09/06/2013 06:30 PM, Xavier Leroy wrote:
> >- Make sure the .opt compilers (ocamlc.opt, ocamlopt.opt) are
> >   installed.  They are significantly faster than their non-.opt
> >   counterparts, and some Linux distros package the .opt compilers
> >   separately and don't install them by default.
> 
> Shouldn't we change the official installation procedure (from
> sources), so that the .opt compilers are installed as "ocamlc" and
> "ocamlopt" (not "ocamlc.opt", "ocamlopt.opt") when they are
> available?  And suggest to do the same for distribution packages?
> It would make the life of other projects easier if they could just
> call "ocamlc" without having to check if "ocamlc.opt" is available
> or not.
> 
> Or are there compelling arguments in favor of using the bytecode
> version of ocamlc/ocamlopt?

That sounds like the wrong place to fix this and a possible
incompatibility with quite a lot of code. Ocamlfind already handles it
nicely; the only thing missing is configuration.

Such things depend distribution policies. Bytecode is often smaller or
maybe that the native code generation for a given platform fails in a
subtle way.

If you make the .opt version unavailable, you'll break several build
scripts which don't use ocamlfind. You'll also break ocamlfind installs
which are set to use the .opt versions. You could make a symlink but
that won't work on Windows and I'm sure there will be many more issues.

Moreover, after having spent pretty much all my free time during the
past 3 months in the build system of the compiler, I strongly believe
that the fewer changes happen there, the better (when talking about my
sanity at least :) ).

Btw, considering that cross-compilation support will be in trunk very
soon, it's better not to assume anything about what's installed and rely
blindly on ocamlfind calling the right binaries (that or autotools).

-- 
Adrien Nader

Re: [Caml-list] Accelerating compilation

[Gabriel Kerneis]

On Mon, Sep 09, 2013 at 07:08:30PM +0200, Adrien Nader wrote:
> Btw, considering that cross-compilation support will be in trunk very
> soon, it's better not to assume anything about what's installed and rely
> blindly on ocamlfind calling the right binaries (that or autotools).

On a related note: a new version of ocaml-autoconf [1] should be
released soon [2] with a number of bug fixes, including support for
cross-compilation.

Gabriel

[1] http://forge.ocamlcore.org/projects/ocaml-autoconf/
[2] as soon as I have time to review the cross-compilation patch series
    and merge it.

Re: [Caml-list] Accelerating compilation

[Aleksey Nogin]

On 09.09.2013 01:15, Romain Bardou wrote:

> It might seem very small compared to say, Coq :) But still, having to
> wait ~10s just to find the next pattern-matching to fix is already
> annoying and it will only get worse.

"omake -p" is your friend ;-)

Aleksey

Re: [Caml-list] Accelerating compilation

[oleg]

> Or are there compelling arguments in favor of using the bytecode version 
> of ocamlc/ocamlopt?

I have come across one case of the meaningful difference between
ocamlc and ocamlc.opt: a particular project can only be linked on
MacOS using ocamlc; ocamlc.opt fails to link. Admittedly, the real
problem is in what I think a hard-to-justify design decision in
OCaml linker.

The problem occurs when linking byte-compiled executables that use the
delimcc library. I will only talk about byte-compiled version of
delimcc in the message (there are no problems for the native version).

The following message
        https://sympa.inria.fr/sympa/arc/caml-list/2013-04/msg00072.html
gives more detail.

Similar to the nums library, delimcc includes some C code, which is
arranged in a shared library libdelimcc.so. That shared library refers
to a particular symbol caml_realloc_stack, which is provided by the
byte-code run-time system (that is, ocamlrun). When we invoke the
byte-code executable that uses delimcc, ocamlrun loads libdelimcc.so,
the library looks for the symbol caml_realloc_stack. Since ocamlrun is
the running executable and its provides the symbol, the resolution
succeeds and everyone is happy.

The problem occurs when linking the executable that uses delimcc. When
OCaml linker processes delimcc.cma, it notices that a shared library
is required. The OCaml linker then loads the library *and forces the
resolution of all undefined references there*. Therefore, the symbol
caml_realloc_stack required by libdelimcc.so has to be found. When the
linking is done by ocamlc, then the running executable is ocamlrun,
which provides the symbol. However, if the linking is done by
ocamlc.opt, it has a different run-time that does not provide the
symbol. Linking fails. Therefore, although ocamlc.opt can be used to
compile projects with delimcc, the final linking step must be done
with ocamlc rather than ocamlc.opt

I managed to get around the problem using a weak reference. Alas, the
subterfuge does not seem to work on Mac OS.

The real problem in my view is the strange decision to load shared
libraries at link time and force the resolution of their undefined
references. This decision certainly makes linking slower, without
providing much benefit, it seems. After all, if the resolution
succeeded at link time, it may still fail at run time since the
linked executable can be run in a different location or even a
different computer.

So, the real problem to me is ocamlc using RTLD_NOW flag when loading
shared library. Removing the flag would make linking faster, and less
painful.

A few technical details: dlopen with the problematic flag occurs in
the function caml_dlopen in the file byterun/unix.c

void * caml_dlopen(char * libname, int for_execution, int global)
{
  return dlopen(libname, RTLD_NOW | (global ? RTLD_GLOBAL : RTLD_LOCAL) | RTLD_NODELETE);
  /* Could use RTLD_LAZY if for_execution == 0, but needs testing */
}

That function caml_dlopen is used within caml_dynlink_open_lib,
which, under alias dll_open, is called in Dll.open_dll. The latter
function is invoked in Bytelink.link_bytecode.

Re: [Caml-list] Accelerating compilation

[Gerd Stolpmann]

[-- Attachment #1: Type: text/plain, Size: 4249 bytes --]

Good point. However, I think without RTLD_NOW there is not enough
protection against errors. The better solution would be if ocamlc.opt
spawned a child process that is a real byterun executable, and that does
the loading check for it. For simplicity, this executable should be the
just created executable, and an environment variable requests to do the
load check only (no real execution beyond that). NB. The "ldd" utility
on Linux uses exactly the same trick.

Gerd

Am Dienstag, den 10.09.2013, 02:01 +0000 schrieb oleg@okmij.org:
> > Or are there compelling arguments in favor of using the bytecode version 
> > of ocamlc/ocamlopt?
> 
> I have come across one case of the meaningful difference between
> ocamlc and ocamlc.opt: a particular project can only be linked on
> MacOS using ocamlc; ocamlc.opt fails to link. Admittedly, the real
> problem is in what I think a hard-to-justify design decision in
> OCaml linker.
> 
> The problem occurs when linking byte-compiled executables that use the
> delimcc library. I will only talk about byte-compiled version of
> delimcc in the message (there are no problems for the native version).
> 
> The following message
>         https://sympa.inria.fr/sympa/arc/caml-list/2013-04/msg00072.html
> gives more detail.
> 
> Similar to the nums library, delimcc includes some C code, which is
> arranged in a shared library libdelimcc.so. That shared library refers
> to a particular symbol caml_realloc_stack, which is provided by the
> byte-code run-time system (that is, ocamlrun). When we invoke the
> byte-code executable that uses delimcc, ocamlrun loads libdelimcc.so,
> the library looks for the symbol caml_realloc_stack. Since ocamlrun is
> the running executable and its provides the symbol, the resolution
> succeeds and everyone is happy.
> 
> The problem occurs when linking the executable that uses delimcc. When
> OCaml linker processes delimcc.cma, it notices that a shared library
> is required. The OCaml linker then loads the library *and forces the
> resolution of all undefined references there*. Therefore, the symbol
> caml_realloc_stack required by libdelimcc.so has to be found. When the
> linking is done by ocamlc, then the running executable is ocamlrun,
> which provides the symbol. However, if the linking is done by
> ocamlc.opt, it has a different run-time that does not provide the
> symbol. Linking fails. Therefore, although ocamlc.opt can be used to
> compile projects with delimcc, the final linking step must be done
> with ocamlc rather than ocamlc.opt
> 
> I managed to get around the problem using a weak reference. Alas, the
> subterfuge does not seem to work on Mac OS.
> 
> The real problem in my view is the strange decision to load shared
> libraries at link time and force the resolution of their undefined
> references. This decision certainly makes linking slower, without
> providing much benefit, it seems. After all, if the resolution
> succeeded at link time, it may still fail at run time since the
> linked executable can be run in a different location or even a
> different computer.
> 
> So, the real problem to me is ocamlc using RTLD_NOW flag when loading
> shared library. Removing the flag would make linking faster, and less
> painful.
> 
> A few technical details: dlopen with the problematic flag occurs in
> the function caml_dlopen in the file byterun/unix.c
> 
> void * caml_dlopen(char * libname, int for_execution, int global)
> {
>   return dlopen(libname, RTLD_NOW | (global ? RTLD_GLOBAL : RTLD_LOCAL) | RTLD_NODELETE);
>   /* Could use RTLD_LAZY if for_execution == 0, but needs testing */
> }
> 
> That function caml_dlopen is used within caml_dynlink_open_lib,
> which, under alias dll_open, is called in Dll.open_dll. The latter
> function is invoked in Bytelink.link_bytecode.
> 
> 
> 
> 

-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
My OCaml site:          http://www.camlcity.org
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [Caml-list] Accelerating compilation

[Adrien Nader]

On Tue, Sep 10, 2013, oleg@okmij.org wrote:
> The real problem in my view is the strange decision to load shared
> libraries at link time and force the resolution of their undefined
> references. This decision certainly makes linking slower, without
> providing much benefit, it seems. After all, if the resolution
> succeeded at link time, it may still fail at run time since the
> linked executable can be run in a different location or even a
> different computer.
> 
> So, the real problem to me is ocamlc using RTLD_NOW flag when loading
> shared library. Removing the flag would make linking faster, and less
> painful.
> 
> A few technical details: dlopen with the problematic flag occurs in
> the function caml_dlopen in the file byterun/unix.c
> 
> void * caml_dlopen(char * libname, int for_execution, int global)
> {
>   return dlopen(libname, RTLD_NOW | (global ? RTLD_GLOBAL : RTLD_LOCAL) | RTLD_NODELETE);
>   /* Could use RTLD_LAZY if for_execution == 0, but needs testing */
> }
> 
> That function caml_dlopen is used within caml_dynlink_open_lib,
> which, under alias dll_open, is called in Dll.open_dll. The latter
> function is invoked in Bytelink.link_bytecode.

That completely breaks for cross-compilation. No matter how hard you
try, you're going to have troubles loading a PE32 shared library from
your ELF executable (and even more if you try to load AArch64 on MIPS).
 :) 

And of course, right now I can't find what I've done for that. I guess
I'll have to find again soon though.

-- 
Adrien Nader

Re: [Caml-list] Accelerating compilation

[Xavier Leroy]

On 10/09/13 04:01, oleg@okmij.org wrote:

> So, the real problem to me is ocamlc using RTLD_NOW flag when loading
> shared library. Removing the flag would make linking faster, and less
> painful.

Sounds reasonable.  I can give it a try later, but if anyone feels
like experimenting, please share your findings.

For the record, the intent of this dynamic loading is to check *at
link-time* that dynamically-loaded stubs provide all the external C
functions needed by the OCaml bytecode program.  (As opposed to
failing when the bytecode program is started.)  In other words, to
detect this problem as early as if one was doing "ocamlc -custom".

Adrian Nader adds:

> That completely breaks for cross-compilation. No matter how hard you
> try, you're going to have troubles loading a PE32 shared library from
> your ELF executable (and even more if you try to load AArch64 on
> MIPS).

Don't worry, it's only for bytecode.  Bytecode executables are already
portable across platforms, so no special cross-compilation support is
needed.  For native code, ocamlopt links C stub libraries statically,
delegating the job to the (cross-)linker.

- Xavier Leroy

Re: [Caml-list] Accelerating compilation

[Adrien Nader]

On Tue, Sep 10, 2013, Xavier Leroy wrote:
> On 10/09/13 04:01, oleg@okmij.org wrote:
> 
> > So, the real problem to me is ocamlc using RTLD_NOW flag when loading
> > shared library. Removing the flag would make linking faster, and less
> > painful.
> 
> Sounds reasonable.  I can give it a try later, but if anyone feels
> like experimenting, please share your findings.
> 
> For the record, the intent of this dynamic loading is to check *at
> link-time* that dynamically-loaded stubs provide all the external C
> functions needed by the OCaml bytecode program.  (As opposed to
> failing when the bytecode program is started.)  In other words, to
> detect this problem as early as if one was doing "ocamlc -custom".
> 
> Adrian Nader adds:
> 
> > That completely breaks for cross-compilation. No matter how hard you
> > try, you're going to have troubles loading a PE32 shared library from
> > your ELF executable (and even more if you try to load AArch64 on
> > MIPS).
> 
> Don't worry, it's only for bytecode.  Bytecode executables are already
> portable across platforms, so no special cross-compilation support is
> needed.  For native code, ocamlopt links C stub libraries statically,
> delegating the job to the (cross-)linker.

IIRC the name for unix-related functions are different for Win32, or
maybe that some are missing (non-implemented functions). There was
something that broke building the compiler (I'm really having a hard
time remembering the details though).

It's been months since I last touched that part but in the end it was
workable and anyway, I doubt many people will try to use bytecode for
Windows anyway.

-- 
Adrien Nader

Re: [Caml-list] Accelerating compilation

[ygrek]

On Tue, 10 Sep 2013 18:53:55 +0200
Adrien Nader <adrien@notk.org> wrote:

> IIRC the name for unix-related functions are different for Win32, or
> maybe that some are missing (non-implemented functions). There was
> something that broke building the compiler (I'm really having a hard
> time remembering the details though).
> 
> It's been months since I last touched that part but in the end it was
> workable and anyway, I doubt many people will try to use bytecode for
> Windows anyway.

They will, and they are brave :
http://roscidus.com/blog/blog/2013/07/07/ocaml-binary-compatibility/#windows--linux-compatibility

-- 

Re: [Caml-list] Accelerating compilation

[Francois Berenger]

On 9/7/13 5:51 AM, Fabrice Le Fessant wrote:
> On Fri, Sep 6, 2013 at 5:20 PM, Romain Bardou <romain.bardou@inria.fr> wrote:
>>>> 3) Parallel compilation in Ocamlbuild
>>>>
>>>> Of course it would help but it is not easy to implement so I'm just
>>>> putting it there to be exhaustive.
>>>
>>> I'm not sure what you are referring to, OCamlBuild does already
>>> support parallel builds.
>>
>> Does it? I actually thought the -j option was ignored.
>>
>> I just did a quick test and I gain about 5 seconds with -j on a 1min15
>> build (I had cleaned, recompiled and recleaned before so that caching by
>> the file system would not impact the result too much), so it does seem
>> to be a *little* faster :)
>
> FWIW, I recently compiled Merlin with both ocamlbuild and ocp-build,
> on a quad-core with "-j 10" for both (the link to the ocp-build
> description file is in the latest OCamlPro's report), ocamlbuild needs
> 13s where ocp-build only needs 4s to compile everything.

This morning, I played with a Makefile and ocamlbuild
to build batteries-included from a clean source tree.

The Makefile was provided by Cedric Cellier.

Here follows the wallclock times to build on my eight cores machine.

#You can plot them like this:
cat <<EOF > make_Vs_ocamlbuild.data
#nprocs Makefile ocamlbuild
1 14.52 21.43
2 6.84 17.57
3 4.95 16.44
4 4.18 15.95
5 3.70 15.67
6 3.36 15.62
7 3.12 15.54
8 3.03 15.292
EOF
gnuplot -persist <<EOF
set title 'make Vs ocamlbuild'
set xlabel 'nprocs'
set ylabel 'wallclock time(s)'
plot 'make_Vs_ocamlbuild.data' u 1:2 w lines title 'time -p make fast -j 
np', \
                             '' u 1:3 w lines title 'ocamlbuild -j np'
EOF

Re: [Caml-list] Accelerating compilation

[Wojciech Meyer]

[-- Attachment #1: Type: text/plain, Size: 2638 bytes --]

What are you trying to prove here? Makefiles will be always faster as
mentioned before, however there is still space for vast improvement, the
biggest elephant here being slow scanning. Patches welcome.

Also, it does not show the convenience of using ocamlbuild, in particular
dynamic dependency resolver, nice language for specyfing complex rules,
tags, and VPATH build that just works (does your makefile VPATH at all?)



On Wed, Sep 11, 2013 at 2:00 PM, Francois Berenger <berenger@riken.jp>wrote:

> On 9/7/13 5:51 AM, Fabrice Le Fessant wrote:
>
>> On Fri, Sep 6, 2013 at 5:20 PM, Romain Bardou <romain.bardou@inria.fr>
>> wrote:
>>
>>> 3) Parallel compilation in Ocamlbuild
>>>>>
>>>>> Of course it would help but it is not easy to implement so I'm just
>>>>> putting it there to be exhaustive.
>>>>>
>>>>
>>>> I'm not sure what you are referring to, OCamlBuild does already
>>>> support parallel builds.
>>>>
>>>
>>> Does it? I actually thought the -j option was ignored.
>>>
>>> I just did a quick test and I gain about 5 seconds with -j on a 1min15
>>> build (I had cleaned, recompiled and recleaned before so that caching by
>>> the file system would not impact the result too much), so it does seem
>>> to be a *little* faster :)
>>>
>>
>> FWIW, I recently compiled Merlin with both ocamlbuild and ocp-build,
>> on a quad-core with "-j 10" for both (the link to the ocp-build
>> description file is in the latest OCamlPro's report), ocamlbuild needs
>> 13s where ocp-build only needs 4s to compile everything.
>>
>
> This morning, I played with a Makefile and ocamlbuild
> to build batteries-included from a clean source tree.
>
> The Makefile was provided by Cedric Cellier.
>
> Here follows the wallclock times to build on my eight cores machine.
>
> #You can plot them like this:
> cat <<EOF > make_Vs_ocamlbuild.data
> #nprocs Makefile ocamlbuild
> 1 14.52 21.43
> 2 6.84 17.57
> 3 4.95 16.44
> 4 4.18 15.95
> 5 3.70 15.67
> 6 3.36 15.62
> 7 3.12 15.54
> 8 3.03 15.292
> EOF
> gnuplot -persist <<EOF
> set title 'make Vs ocamlbuild'
> set xlabel 'nprocs'
> set ylabel 'wallclock time(s)'
> plot 'make_Vs_ocamlbuild.data' u 1:2 w lines title 'time -p make fast -j
> np', \
>                             '' u 1:3 w lines title 'ocamlbuild -j np'
> EOF
>
>
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/**arc/caml-list<https://sympa.inria.fr/sympa/arc/caml-list>
> Beginner's list: http://groups.yahoo.com/group/**ocaml_beginners<http://groups.yahoo.com/group/ocaml_beginners>
> Bug reports: http://caml.inria.fr/bin/caml-**bugs<http://caml.inria.fr/bin/caml-bugs>
>

[-- Attachment #2: Type: text/html, Size: 3889 bytes --]

Re: [Caml-list] Accelerating compilation

[Francois Berenger]

On 09/11/2013 10:46 PM, Wojciech Meyer wrote:
> What are you trying to prove here?

I am not trying to prove anything.
I just manage to show something, which is the whole point of a good plot.

 > Makefiles will be always faster as
> mentioned before, however there is still space for vast improvement, the
> biggest elephant here being slow scanning. Patches welcome.
>
> Also, it does not show the convenience of using ocamlbuild, in
> particular dynamic dependency resolver, nice language for specyfing
> complex rules, tags, and VPATH build that just works (does your makefile
> VPATH at all?)
>
>
>
> On Wed, Sep 11, 2013 at 2:00 PM, Francois Berenger <berenger@riken.jp
> <mailto:berenger@riken.jp>> wrote:
>
>     On 9/7/13 5:51 AM, Fabrice Le Fessant wrote:
>
>         On Fri, Sep 6, 2013 at 5:20 PM, Romain Bardou
>         <romain.bardou@inria.fr <mailto:romain.bardou@inria.fr>> wrote:
>
>                     3) Parallel compilation in Ocamlbuild
>
>                     Of course it would help but it is not easy to
>                     implement so I'm just
>                     putting it there to be exhaustive.
>
>
>                 I'm not sure what you are referring to, OCamlBuild does
>                 already
>                 support parallel builds.
>
>
>             Does it? I actually thought the -j option was ignored.
>
>             I just did a quick test and I gain about 5 seconds with -j
>             on a 1min15
>             build (I had cleaned, recompiled and recleaned before so
>             that caching by
>             the file system would not impact the result too much), so it
>             does seem
>             to be a *little* faster :)
>
>
>         FWIW, I recently compiled Merlin with both ocamlbuild and ocp-build,
>         on a quad-core with "-j 10" for both (the link to the ocp-build
>         description file is in the latest OCamlPro's report), ocamlbuild
>         needs
>         13s where ocp-build only needs 4s to compile everything.
>
>
>     This morning, I played with a Makefile and ocamlbuild
>     to build batteries-included from a clean source tree.
>
>     The Makefile was provided by Cedric Cellier.
>
>     Here follows the wallclock times to build on my eight cores machine.
>
>     #You can plot them like this:
>     cat <<EOF > make_Vs_ocamlbuild.data
>     #nprocs Makefile ocamlbuild
>     1 14.52 21.43
>     2 6.84 17.57
>     3 4.95 16.44
>     4 4.18 15.95
>     5 3.70 15.67
>     6 3.36 15.62
>     7 3.12 15.54
>     8 3.03 15.292
>     EOF
>     gnuplot -persist <<EOF
>     set title 'make Vs ocamlbuild'
>     set xlabel 'nprocs'
>     set ylabel 'wallclock time(s)'
>     plot 'make_Vs_ocamlbuild.data' u 1:2 w lines title 'time -p make
>     fast -j np', \
>                                  '' u 1:3 w lines title 'ocamlbuild -j np'
>     EOF
>
>
>
>     --
>     Caml-list mailing list.  Subscription management and archives:
>     https://sympa.inria.fr/sympa/__arc/caml-list
>     <https://sympa.inria.fr/sympa/arc/caml-list>
>     Beginner's list: http://groups.yahoo.com/group/__ocaml_beginners
>     <http://groups.yahoo.com/group/ocaml_beginners>
>     Bug reports: http://caml.inria.fr/bin/caml-__bugs
>     <http://caml.inria.fr/bin/caml-bugs>
>
>

Re: [Caml-list] Accelerating compilation

[Jacques Le Normand]

[-- Attachment #1: Type: text/plain, Size: 4212 bytes --]

So is anyone working on accelerating compilation? Has any of these
proposals been deemed superior to the others?
On Sep 11, 2013 9:24 PM, "Francois Berenger" <berenger@riken.jp> wrote:

> On 09/11/2013 10:46 PM, Wojciech Meyer wrote:
>
>> What are you trying to prove here?
>>
>
> I am not trying to prove anything.
> I just manage to show something, which is the whole point of a good plot.
>
> > Makefiles will be always faster as
>
>> mentioned before, however there is still space for vast improvement, the
>> biggest elephant here being slow scanning. Patches welcome.
>>
>> Also, it does not show the convenience of using ocamlbuild, in
>> particular dynamic dependency resolver, nice language for specyfing
>> complex rules, tags, and VPATH build that just works (does your makefile
>> VPATH at all?)
>>
>>
>>
>> On Wed, Sep 11, 2013 at 2:00 PM, Francois Berenger <berenger@riken.jp
>> <mailto:berenger@riken.jp>> wrote:
>>
>>     On 9/7/13 5:51 AM, Fabrice Le Fessant wrote:
>>
>>         On Fri, Sep 6, 2013 at 5:20 PM, Romain Bardou
>>         <romain.bardou@inria.fr <mailto:romain.bardou@inria.fr**>> wrote:
>>
>>                     3) Parallel compilation in Ocamlbuild
>>
>>                     Of course it would help but it is not easy to
>>                     implement so I'm just
>>                     putting it there to be exhaustive.
>>
>>
>>                 I'm not sure what you are referring to, OCamlBuild does
>>                 already
>>                 support parallel builds.
>>
>>
>>             Does it? I actually thought the -j option was ignored.
>>
>>             I just did a quick test and I gain about 5 seconds with -j
>>             on a 1min15
>>             build (I had cleaned, recompiled and recleaned before so
>>             that caching by
>>             the file system would not impact the result too much), so it
>>             does seem
>>             to be a *little* faster :)
>>
>>
>>         FWIW, I recently compiled Merlin with both ocamlbuild and
>> ocp-build,
>>         on a quad-core with "-j 10" for both (the link to the ocp-build
>>         description file is in the latest OCamlPro's report), ocamlbuild
>>         needs
>>         13s where ocp-build only needs 4s to compile everything.
>>
>>
>>     This morning, I played with a Makefile and ocamlbuild
>>     to build batteries-included from a clean source tree.
>>
>>     The Makefile was provided by Cedric Cellier.
>>
>>     Here follows the wallclock times to build on my eight cores machine.
>>
>>     #You can plot them like this:
>>     cat <<EOF > make_Vs_ocamlbuild.data
>>     #nprocs Makefile ocamlbuild
>>     1 14.52 21.43
>>     2 6.84 17.57
>>     3 4.95 16.44
>>     4 4.18 15.95
>>     5 3.70 15.67
>>     6 3.36 15.62
>>     7 3.12 15.54
>>     8 3.03 15.292
>>     EOF
>>     gnuplot -persist <<EOF
>>     set title 'make Vs ocamlbuild'
>>     set xlabel 'nprocs'
>>     set ylabel 'wallclock time(s)'
>>     plot 'make_Vs_ocamlbuild.data' u 1:2 w lines title 'time -p make
>>     fast -j np', \
>>                                  '' u 1:3 w lines title 'ocamlbuild -j np'
>>     EOF
>>
>>
>>
>>     --
>>     Caml-list mailing list.  Subscription management and archives:
>>     https://sympa.inria.fr/sympa/_**_arc/caml-list<https://sympa.inria.fr/sympa/__arc/caml-list>
>>     <https://sympa.inria.fr/sympa/**arc/caml-list<https://sympa.inria.fr/sympa/arc/caml-list>
>> >
>>     Beginner's list: http://groups.yahoo.com/group/**__ocaml_beginners<http://groups.yahoo.com/group/__ocaml_beginners>
>>     <http://groups.yahoo.com/**group/ocaml_beginners<http://groups.yahoo.com/group/ocaml_beginners>
>> >
>>     Bug reports: http://caml.inria.fr/bin/caml-**__bugs<http://caml.inria.fr/bin/caml-__bugs>
>>     <http://caml.inria.fr/bin/**caml-bugs<http://caml.inria.fr/bin/caml-bugs>
>> >
>>
>>
>>
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/**arc/caml-list<https://sympa.inria.fr/sympa/arc/caml-list>
> Beginner's list: http://groups.yahoo.com/group/**ocaml_beginners<http://groups.yahoo.com/group/ocaml_beginners>
> Bug reports: http://caml.inria.fr/bin/caml-**bugs<http://caml.inria.fr/bin/caml-bugs>
>

[-- Attachment #2: Type: text/html, Size: 5603 bytes --]

[Caml-list] from oasis to obuild (original subject was Re: Accelerating compilation)

[Francois Berenger]

On 09/07/2013 05:51 AM, Fabrice Le Fessant wrote:
 > [...]
> FWIW, I recently compiled Merlin with both ocamlbuild and ocp-build,
> on a quad-core with "-j 10" for both (the link to the ocp-build
> description file is in the latest OCamlPro's report), ocamlbuild needs
> 13s where ocp-build only needs 4s to compile everything.

I recently switched from oasis to obuild.

The main reason I adopted oasis in the past is because the project
description file (_oasis) is very short.

With ocp-build, (as far as I know) I would have to list all the files
my program/libraries need to be built, and I am too lazy for that.

But now, I have a project.obuild file instead, which
is as terse as an _oasis file.

Here is my project.obuild file:
---
Name:         something
Version:      0.1
Authors:      me
License:      GPL
Synopsis:     some_more_things
obuild-ver:   1

Executable ac
   Path:           .
   BuildDepends:   batteries, dolog, parmap, vector3, biniou, atd, atdgen
   MainIs:         ac.ml

[...] some other executables follow
---

Compile time before, with oasis:

# cat build.sh
#!/bin/bash

#set -x

oasis setup
ocaml setup.ml -configure
ocaml setup.ml -build

# time ./build.sh
real    0m9.397s

Compile time after, with obuild:
# obuild clean && time (obuild configure && obuild build -j 1)
real    0m3.624s

-- 
Best regards,
Francois Berenger.

Re: [Caml-list] from oasis to obuild (original subject was Re: Accelerating compilation)

[Török Edwin]

On 09/30/2013 11:06 AM, Francois Berenger wrote:
> On 09/07/2013 05:51 AM, Fabrice Le Fessant wrote:
>> [...]
>> FWIW, I recently compiled Merlin with both ocamlbuild and ocp-build,
>> on a quad-core with "-j 10" for both (the link to the ocp-build
>> description file is in the latest OCamlPro's report), ocamlbuild needs
>> 13s where ocp-build only needs 4s to compile everything.
> 
> I recently switched from oasis to obuild.
> 
> The main reason I adopted oasis in the past is because the project
> description file (_oasis) is very short.
> 
> With ocp-build, (as far as I know) I would have to list all the files
> my program/libraries need to be built, and I am too lazy for that.
> 
> But now, I have a project.obuild file instead, which
> is as terse as an _oasis file.

obuild is more like CMake than automake, right? (i.e. it requires itself to be present at build time,
and you can't pregenerate build files)

Best regards,
--Edwin

Re: [Caml-list] from oasis to obuild (original subject was Re: Accelerating compilation)

[Fabrice Le Fessant]

On Mon, Sep 30, 2013 at 10:07 AM, Francois Berenger <berenger@riken.jp> wrote:
> # time ./build.sh
> real    0m9.397s
>
> Compile time after, with obuild:
> # obuild clean && time (obuild configure && obuild build -j 1)
> real    0m3.624s

If you compile with "-j 1", where does the speed-up come from ? Is
Oasis using bytecode commands instead of native-code commands ? Could
you ask the two tools to display the commands they are calling, to
understand the difference ?
--Fabrice
-- 
Fabrice LE FESSANT
Chercheur en Informatique
INRIA Paris Rocquencourt -- OCamlPro
Programming Languages and Distributed Systems

Re: [Caml-list] from oasis to obuild (original subject was Re: Accelerating compilation)

[Anil Madhavapeddy]

On 30 Sep 2013, at 10:00, Fabrice Le Fessant <Fabrice.Le_fessant@inria.fr> wrote:

> On Mon, Sep 30, 2013 at 10:07 AM, Francois Berenger <berenger@riken.jp> wrote:
>> # time ./build.sh
>> real    0m9.397s
>> 
>> Compile time after, with obuild:
>> # obuild clean && time (obuild configure && obuild build -j 1)
>> real    0m3.624s
> 
> If you compile with "-j 1", where does the speed-up come from ? Is
> Oasis using bytecode commands instead of native-code commands ? Could
> you ask the two tools to display the commands they are calling, to
> understand the difference ?

OASIS runs every single build command through ocamlfind.  Moving that
to the configure phase gives a big speedup for almost any project
with a non-trivial number of source files.

-anil

Re: [Caml-list] from oasis to obuild (original subject was Re: Accelerating compilation)

[Francois Berenger]

On 09/30/2013 06:00 PM, Fabrice Le Fessant wrote:
> On Mon, Sep 30, 2013 at 10:07 AM, Francois Berenger <berenger@riken.jp> wrote:
>> # time ./build.sh
>> real    0m9.397s
>>
>> Compile time after, with obuild:
>> # obuild clean && time (obuild configure && obuild build -j 1)
>> real    0m3.624s
>
> If you compile with "-j 1", where does the speed-up come from ?

Good question that I will leave the experts to answer.

 > Is
> Oasis using bytecode commands instead of native-code commands ? Could
> you ask the two tools to display the commands they are calling, to
> understand the difference ?

My main goal was to get rid of ocamlbuild.
So, case solved, I won't invest more time in that, sorry.

-- 
Best regards,
Francois Berenger.

Re: [Caml-list] from oasis to obuild (original subject was Re: Accelerating compilation)

[Alain Frisch]

On 9/30/2013 11:13 AM, Anil Madhavapeddy wrote:
> OASIS runs every single build command through ocamlfind.  Moving that
> to the configure phase gives a big speedup for almost any project
> with a non-trivial number of source files.

Has someone investigated the overhead induced by ocamlfind?  Is it about 
the extra process, parsing the META files, or something else?

Since both ocamlfind and the compiler come in the form of a library 
(findlib / compilerlib),  it should not be too difficult to merge them 
into a single executable.  If parsing the META files takes time, a cache 
mechanism in ocamlfind/findlib could help (maybe).

But anyway, doing the library resolution at configure time sounds like a 
good idea!

-- Alain

Re: [Caml-list] from oasis to obuild (original subject was Re: Accelerating compilation)

[Anil Madhavapeddy]

On 30 Sep 2013, at 12:13, Alain Frisch <alain@frisch.fr> wrote:

> On 9/30/2013 11:13 AM, Anil Madhavapeddy wrote:
>> OASIS runs every single build command through ocamlfind.  Moving that
>> to the configure phase gives a big speedup for almost any project
>> with a non-trivial number of source files.
> 
> Has someone investigated the overhead induced by ocamlfind?  Is it about the extra process, parsing the META files, or something else?
> 
> Since both ocamlfind and the compiler come in the form of a library (findlib / compilerlib),  it should not be too difficult to merge them into a single executable.  If parsing the META files takes time, a cache mechanism in ocamlfind/findlib could help (maybe).

Leo put together a compiler frontend that replaces the standard parsetree with camlp4 that's statically linked: https://github.com/lpw25/ocaml-with-pp

I'm currently untangling the precise camlp4 dependencies needed for a Core build, so I can link a Core js_of_ocaml interactive toplevel for the Real World OCaml website examples.  Once that's done, I think linking in findlib as a library and looking at performance is the next step, and then an inotify/kqueue backend, and see how this all looks performance-wise...

-anil

(incidentally, figuring out the precise camlp4 modules and their loading order is a bit staggering, since camlp4 helpfully rewrites the command line flags to load further modules internally depending on the revised/original syntax dance.  I'm looking forward to extension_points...)

Re: [Caml-list] from oasis to obuild (original subject was Re: Accelerating compilation)

[Alain Frisch]

On 9/30/2013 1:19 PM, Anil Madhavapeddy wrote:
> Leo put together a compiler frontend that replaces the standard parsetree with camlp4 that's statically linked: https://github.com/lpw25/ocaml-with-pp

That's interesting.  Most of the code is copied directly from the 
compiler.  I guess that adding a few hooks directly in the compiler 
would greatly simplify such projects.

-- Alain

Re: [Caml-list] from oasis to obuild (original subject was Re: Accelerating compilation)

[Anil Madhavapeddy]

On 30 Sep 2013, at 12:27, Alain Frisch <alain@frisch.fr> wrote:

> On 9/30/2013 1:19 PM, Anil Madhavapeddy wrote:
>> Leo put together a compiler frontend that replaces the standard parsetree with camlp4 that's statically linked: https://github.com/lpw25/ocaml-with-pp
> 
> That's interesting.  Most of the code is copied directly from the compiler.  I guess that adding a few hooks directly in the compiler would greatly simplify such projects.

Yep. We figured it's best to build a full prototype using nothing but OCaml 4.1 compiler-libs, and then figure out what the implications for the 4.2 compiler drivers would be when the dust has settled around a 4.1-based prototype.

I'd also like to start using this with 4.1 and not have to wait another year for fast builds, so some code duplication in the short term isn't too bad.

-anil

Re: [Caml-list] from oasis to obuild (original subject was Re: Accelerating compilation)

[Sylvain Le Gall]

Hello,

On 30-09-2013, Francois Berenger <berenger@riken.jp> wrote:
> On 09/07/2013 05:51 AM, Fabrice Le Fessant wrote:
> > [...]
>> FWIW, I recently compiled Merlin with both ocamlbuild and ocp-build,
>> on a quad-core with "-j 10" for both (the link to the ocp-build
>> description file is in the latest OCamlPro's report), ocamlbuild needs
>> 13s where ocp-build only needs 4s to compile everything.
>
> I recently switched from oasis to obuild.
>
> The main reason I adopted oasis in the past is because the project
> description file (_oasis) is very short.
>
> With ocp-build, (as far as I know) I would have to list all the files
> my program/libraries need to be built, and I am too lazy for that.
>
> But now, I have a project.obuild file instead, which
> is as terse as an _oasis file.
>
> Here is my project.obuild file:
> ---
> Name:         something
> Version:      0.1
> Authors:      me
> License:      GPL
> Synopsis:     some_more_things
> obuild-ver:   1
>
> Executable ac
>    Path:           .
>    BuildDepends:   batteries, dolog, parmap, vector3, biniou, atd, atdgen
>    MainIs:         ac.ml
>
> [...] some other executables follow
> ---
>
> Compile time before, with oasis:
>
> # cat build.sh
> #!/bin/bash
>
> #set -x
>
> oasis setup
> ocaml setup.ml -configure
> ocaml setup.ml -build
>
> # time ./build.sh
> real    0m9.397s
>
> Compile time after, with obuild:
> # obuild clean && time (obuild configure && obuild build -j 1)
> real    0m3.624s
>

Note that given the proximity of obuild and _oasis files it should be
super easy to just convert 1 time the _oasis file to foo.obuild and keep
using _oasis while changing the build backend from ocamlbuild to obuild.

OASIS is not building anything by itself, the time you report is
probably 90% due to ocamlbuild (the build backend).

BTW, this kind of _oasis -> obuild conversion is a topic I have
discussed with the author of obuild and I am 100% willing to do the
conversion plugin if it makes sense.


Regards
Sylvain

> -- 
> Best regards,
> Francois Berenger.
>
> -- 
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>

Cheers,
Sylvain Le Gall
-- 
Website:     http://sylvain.le-gall.net/
OCaml forge: http://forge.ocamlcore.org
OCaml blogs: http://planet.ocaml.org

Re: [Caml-list] from oasis to obuild (original subject was Re: Accelerating compilation)

[Francois Berenger]

On 09/30/2013 11:11 PM, Sylvain Le Gall wrote:
 >> [...]
>> Compile time before, with oasis:
>>
>> # cat build.sh
>> #!/bin/bash
>>
>> #set -x
>>
>> oasis setup
>> ocaml setup.ml -configure
>> ocaml setup.ml -build
>>
>> # time ./build.sh
>> real    0m9.397s
>>
>> Compile time after, with obuild:
>> # obuild clean && time (obuild configure && obuild build -j 1)
>> real    0m3.624s
>>
>
> Note that given the proximity of obuild and _oasis files it should be
> super easy to just convert 1 time the _oasis file to foo.obuild and keep
> using _oasis while changing the build backend from ocamlbuild to obuild.
>
> OASIS is not building anything by itself, the time you report is
> probably 90% due to ocamlbuild (the build backend).

Sylvain, I fully trust you on this one.

And since I dislike ocamlbuild, I am getting rid of it or tools
that use it as their default compilation backend.
obuild seems like the perfect choice to fill my needs.

I do like oasis, but you also know that I started quite early
to reclaim some other compilation engine to be supported by oasis.

Best regards,
Francois.

> BTW, this kind of _oasis -> obuild conversion is a topic I have
> discussed with the author of obuild and I am 100% willing to do the
> conversion plugin if it makes sense.
>
>
> Regards
> Sylvain
>
>> --
>> Best regards,
>> Francois Berenger.
>>
>> --
>> Caml-list mailing list.  Subscription management and archives:
>> https://sympa.inria.fr/sympa/arc/caml-list
>> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>> Bug reports: http://caml.inria.fr/bin/caml-bugs
>>
>
> Cheers,
> Sylvain Le Gall

Re: [Caml-list] from oasis to obuild (original subject was Re: Accelerating compilation)

[Sylvain Le Gall]

On 01-10-2013, Francois Berenger <berenger@riken.jp> wrote:
> On 09/30/2013 11:11 PM, Sylvain Le Gall wrote:
>>> oasis setup
>>> ocaml setup.ml -configure
>>> ocaml setup.ml -build
>>>
>>> # time ./build.sh
>>> real    0m9.397s
>>>
>>> Compile time after, with obuild:
>>> # obuild clean && time (obuild configure && obuild build -j 1)
>>> real    0m3.624s
>>>
>>
>> Note that given the proximity of obuild and _oasis files it should be
>> super easy to just convert 1 time the _oasis file to foo.obuild and keep
>> using _oasis while changing the build backend from ocamlbuild to obuild.
>>
>> OASIS is not building anything by itself, the time you report is
>> probably 90% due to ocamlbuild (the build backend).
>
> Sylvain, I fully trust you on this one.
>
> And since I dislike ocamlbuild, I am getting rid of it or tools
> that use it as their default compilation backend.
> obuild seems like the perfect choice to fill my needs.
>
> I do like oasis, but you also know that I started quite early
> to reclaim some other compilation engine to be supported by oasis.

Fair enough, so let make this happen! Are you willing to be a tester or
helping me on the topic. I have seen that there are some tools to handle
_oasis file in obuild, but given there extreme simpicity, I think this
was just a try and probably not working:
https://github.com/vincenthz/obuild/blob/master/tools/assimilate_oasis.ml

Let's grow this into something that work really with _oasis file and
have the possibility to choose a different backend for OASIS.

I have contacted obuild author to see how we can cooperate on the topic
and I would like to do something to have at least another build engine
possible.

Cheers
Sylvain

>
>> BTW, this kind of _oasis -> obuild conversion is a topic I have
>> discussed with the author of obuild and I am 100% willing to do the
>> conversion plugin if it makes sense.
>>
>>
>> Regards
>> Sylvain
>>
>>> --
>>> Best regards,
>>> Francois Berenger.
>>>
>>> --
>>> Caml-list mailing list.  Subscription management and archives:
>>> https://sympa.inria.fr/sympa/arc/caml-list
>>> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>>> Bug reports: http://caml.inria.fr/bin/caml-bugs
>>>
>>
>> Cheers,
>> Sylvain Le Gall
>
> -- 
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>

Cheers,
Sylvain Le Gall
-- 
Website:     http://sylvain.le-gall.net/
OCaml forge: http://forge.ocamlcore.org
OCaml blogs: http://planet.ocaml.org