[Caml-list] Speeding up compilation

[Caml-list] Speeding up compilation

[Anders Peter Fugmann]

Hi,

While trying to speed up compilation of a project of around ~150 source 
files, i found that ocamlfind invokes a bytecode version of CamlP4.

Our homemade Makefile uses ocamlfind to compile bytecode and optimized 
code as well as dependency tracking using ocamldep and hand written 
tools. Almost every source file makes use of p4 extensions.

When compiling though ocamlfind using the -syntax flag, I can see that 
ocamlfind starts ocamlp4 which is ocaml bytecode, and I cannot find any 
flags that allows me to use any optimized version. Also it seems that an 
optimized version of camlp4 is compiled (ocaml compiler is compiled via 
opam).

Is there any way to use a optimized version of camlp4. Also any other 
ideas on how to speed up compilation would be appreciated.

Regards
Anders Fugmann

Re: [Caml-list] Speeding up compilation

[Daniel Bünzli]

Le mercredi, 2 avril 2014 à 22:14, Anders Peter Fugmann a écrit :

> Is there any way to use a optimized version of camlp4.

Not sure that this will work as it it not mentioned in the documentation [1] but did you try to add

camlp4="camlp4.opt"  

in .opam/$SWITCH/lib/findlib.conf (or wherever it is on your machine).  

Best,

Daniel

[1] http://projects.camlcity.org/projects/dl/findlib-1.4.1/doc/ref-html/r775.html

Re: [Caml-list] Speeding up compilation

[Fabrice Le Fessant]

[-- Attachment #1: Type: text/plain, Size: 1198 bytes --]

ocp-build uses a trick to speed-up compilations with camlp4: it
preprocesses every file only once, and then uses the generated file (a
binary version of the file AST) with ocamldep, ocamlc and ocamlopt. It
usually means a x3 speedup.

--Fabrice


On Wed, Apr 2, 2014 at 10:34 PM, Daniel Bünzli
<daniel.buenzli@erratique.ch>wrote:

>
>
> Le mercredi, 2 avril 2014 à 22:14, Anders Peter Fugmann a écrit :
>
> > Is there any way to use a optimized version of camlp4.
>
> Not sure that this will work as it it not mentioned in the documentation
> [1] but did you try to add
>
> camlp4="camlp4.opt"
>
> in .opam/$SWITCH/lib/findlib.conf (or wherever it is on your machine).
>
> Best,
>
> Daniel
>
> [1]
> http://projects.camlcity.org/projects/dl/findlib-1.4.1/doc/ref-html/r775.html
>
>
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>



-- 
Fabrice LE FESSANT
Chercheur en Informatique
INRIA Paris Rocquencourt -- OCamlPro
Programming Languages and Distributed Systems

[-- Attachment #2: Type: text/html, Size: 2010 bytes --]

Re: [Caml-list] Speeding up compilation

[Anders Peter Fugmann]

On 02/04/14 22:34, Daniel Bünzli wrote:
>
>
> Le mercredi, 2 avril 2014 à 22:14, Anders Peter Fugmann a écrit :
>
>> Is there any way to use a optimized version of camlp4.
>
> Not sure that this will work as it it not mentioned in the documentation [1] but did you try to add
>
> camlp4="camlp4.opt"
Actually yes. That was my first attempt :-)

But it seems ocamlfind does does not allow overriding camlp4 binary 
(from reading the source code). Also I don't seem to have a camlp4.opt.

Regards
Anders

Re: [Caml-list] Speeding up compilation

[Yotam Barnoy]

[-- Attachment #1: Type: text/plain, Size: 1593 bytes --]

I've had good experience shifting from a custom makefile to using
ocamlbuild on a project with around 120 files and moderate preprocessing.
The custom makefile was a pain to maintain and seemed to be doing too much
work.

-Yotam


On Wed, Apr 2, 2014 at 4:43 PM, Fabrice Le Fessant <
Fabrice.Le_fessant@inria.fr> wrote:

> ocp-build uses a trick to speed-up compilations with camlp4: it
> preprocesses every file only once, and then uses the generated file (a
> binary version of the file AST) with ocamldep, ocamlc and ocamlopt. It
> usually means a x3 speedup.
>
> --Fabrice
>
>
> On Wed, Apr 2, 2014 at 10:34 PM, Daniel Bünzli <
> daniel.buenzli@erratique.ch> wrote:
>
>>
>>
>> Le mercredi, 2 avril 2014 à 22:14, Anders Peter Fugmann a écrit :
>>
>> > Is there any way to use a optimized version of camlp4.
>>
>> Not sure that this will work as it it not mentioned in the documentation
>> [1] but did you try to add
>>
>> camlp4="camlp4.opt"
>>
>> in .opam/$SWITCH/lib/findlib.conf (or wherever it is on your machine).
>>
>> Best,
>>
>> Daniel
>>
>> [1]
>> http://projects.camlcity.org/projects/dl/findlib-1.4.1/doc/ref-html/r775.html
>>
>>
>>
>> --
>> Caml-list mailing list.  Subscription management and archives:
>> https://sympa.inria.fr/sympa/arc/caml-list
>> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>> Bug reports: http://caml.inria.fr/bin/caml-bugs
>>
>
>
>
> --
> Fabrice LE FESSANT
> Chercheur en Informatique
> INRIA Paris Rocquencourt -- OCamlPro
> Programming Languages and Distributed Systems
>

[-- Attachment #2: Type: text/html, Size: 2735 bytes --]

Re: [Caml-list] Speeding up compilation

[Anders Peter Fugmann]

On 02/04/14 22:43, Fabrice Le Fessant wrote:
> ocp-build uses a trick to speed-up compilations with camlp4: it
> preprocesses every file only once, and then uses the generated file (a
> binary version of the file AST) with ocamldep, ocamlc and ocamlopt. It
> usually means a x3 speedup.
>
Good idea. What would indeed speed up dependency generation.
Do you know if its possible to have ocamlfind generate a binary version 
of the AST as a separate step?

Regards
Anders

Re: [Caml-list] Speeding up compilation

[Daniel Bünzli]

Le mercredi, 2 avril 2014 à 22:49, Anders Peter Fugmann a écrit :
> > camlp4="camlp4.opt"
>  
> Actually yes. That was my first attempt :-)

Damned. You should report it to Gerd. That was also the case for ocamldoc a few versions ago, I asked him and he added the variable.  
Meanwhile a first non-invasive step for your build system would be:  

    cd .opam/$SWITCH/bin
    rm camlp4
    ln -s camlp4.opt camlp4

Daniel

Re: [Caml-list] Speeding up compilation

[Gerd Stolpmann]

[-- Attachment #1: Type: text/plain, Size: 1331 bytes --]

The camlp4 executable is specified in the META file of camlp4, namely by
the "preprocessor" setting. So you could change that to camlp4.opt.

This is not configured by default because not every syntax extension can
be loaded into camlp4.opt, as it needs to be an cmxs. We would need here
some wrapper that first checks whether the cmxs files are available, and
switch to camlp4.opt only if this is the case.

Gerd

Am Mittwoch, den 02.04.2014, 22:34 +0200 schrieb Daniel Bünzli:
> 
> Le mercredi, 2 avril 2014 à 22:14, Anders Peter Fugmann a écrit :
> 
> > Is there any way to use a optimized version of camlp4.
> 
> Not sure that this will work as it it not mentioned in the documentation [1] but did you try to add
> 
> camlp4="camlp4.opt"  
> 
> in .opam/$SWITCH/lib/findlib.conf (or wherever it is on your machine).  
> 
> Best,
> 
> Daniel
> 
> [1] http://projects.camlcity.org/projects/dl/findlib-1.4.1/doc/ref-html/r775.html
> 
> 
> 

-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
My OCaml site:          http://www.camlcity.org
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [Caml-list] Speeding up compilation

[Francois Berenger]

I've had good experience shifting all my projects from whatever they
were using to obuild[1]. ;)

More seriously (yeah, shameless plug...), I have heard that people
handling a lot of OCaml source files and interested in fast compilation
use omake[2].

I also heard that there is an incremental build feature in merlin[3].

I did not try omake because I am too lazy to learn a new
syntax to describe my build.
I tried merlin, but it made my Emacs lag, so I removed it
quickly after a few days of trial.

[1] https://github.com/ocaml-obuild/obuild
[2] http://omake.metaprl.org/index.html
[3] https://github.com/the-lambda-church/merlin

On 04/03/2014 05:54 AM, Yotam Barnoy wrote:
> I've had good experience shifting from a custom makefile to using
> ocamlbuild on a project with around 120 files and moderate preprocessing.
> The custom makefile was a pain to maintain and seemed to be doing too much
> work.
>
> -Yotam
>
>
> On Wed, Apr 2, 2014 at 4:43 PM, Fabrice Le Fessant <
> Fabrice.Le_fessant@inria.fr> wrote:
>
>> ocp-build uses a trick to speed-up compilations with camlp4: it
>> preprocesses every file only once, and then uses the generated file (a
>> binary version of the file AST) with ocamldep, ocamlc and ocamlopt. It
>> usually means a x3 speedup.
>>
>> --Fabrice
>>
>>
>> On Wed, Apr 2, 2014 at 10:34 PM, Daniel Bünzli <
>> daniel.buenzli@erratique.ch> wrote:
>>
>>>
>>>
>>> Le mercredi, 2 avril 2014 à 22:14, Anders Peter Fugmann a écrit :
>>>
>>>> Is there any way to use a optimized version of camlp4.
>>>
>>> Not sure that this will work as it it not mentioned in the documentation
>>> [1] but did you try to add
>>>
>>> camlp4="camlp4.opt"
>>>
>>> in .opam/$SWITCH/lib/findlib.conf (or wherever it is on your machine).
>>>
>>> Best,
>>>
>>> Daniel
>>>
>>> [1]
>>> http://projects.camlcity.org/projects/dl/findlib-1.4.1/doc/ref-html/r775.html
>>>
>>>
>>>
>>> --
>>> Caml-list mailing list.  Subscription management and archives:
>>> https://sympa.inria.fr/sympa/arc/caml-list
>>> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>>> Bug reports: http://caml.inria.fr/bin/caml-bugs
>>>
>>
>>
>>
>> --
>> Fabrice LE FESSANT
>> Chercheur en Informatique
>> INRIA Paris Rocquencourt -- OCamlPro
>> Programming Languages and Distributed Systems
>>
>


-- 
Best regards,
Francois Berenger.

Re: [Caml-list] Speeding up compilation

[Anders Peter Fugmann]

On 02/04/14 23:42, Gerd Stolpmann wrote:
> The camlp4 executable is specified in the META file of camlp4, namely by
> the "preprocessor" setting. So you could change that to camlp4.opt.
I dont seem to have a camlp4.opt, but other variants (camlp4[orf]+.opt). 
Do I need to change how opam compiles the ocaml toolchain in order to 
get a camlp4.opt?

>
> This is not configured by default because not every syntax extension can
> be loaded into camlp4.opt, as it needs to be an cmxs. We would need here
> some wrapper that first checks whether the cmxs files are available, and
> switch to camlp4.opt only if this is the case.
That makes sense.
Thanks

Regards
Anders

Re: [Caml-list] Speeding up compilation

[Gerd Stolpmann]

[-- Attachment #1: Type: text/plain, Size: 1291 bytes --]

Am Donnerstag, den 03.04.2014, 23:00 +0200 schrieb Anders Peter Fugmann:
> On 02/04/14 23:42, Gerd Stolpmann wrote:
> > The camlp4 executable is specified in the META file of camlp4, namely by
> > the "preprocessor" setting. So you could change that to camlp4.opt.
> I dont seem to have a camlp4.opt, but other variants (camlp4[orf]+.opt). 
> Do I need to change how opam compiles the ocaml toolchain in order to 
> get a camlp4.opt?

Right, that makes it even harder. There's only camlp4<something>.opt, a
scheme that doesn't fit nicely into ocamlfind.

Gerd


> >
> > This is not configured by default because not every syntax extension can
> > be loaded into camlp4.opt, as it needs to be an cmxs. We would need here
> > some wrapper that first checks whether the cmxs files are available, and
> > switch to camlp4.opt only if this is the case.
> That makes sense.
> Thanks
> 
> Regards
> Anders
> 
> 

-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
My OCaml site:          http://www.camlcity.org
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [Caml-list] Speeding up compilation

[Pietro Abate]

For quite a while I've simply used 
pp(camlp4o.opt -I_build Camlp4MacroParser.cmxs)
in my _tag file and make sure that the plugin
Camlp4MacroParser.cmxs was either available on that I made
available in the _build directory.

In my makefile I have a rule I use to compile the plugin
before calling ocamlbuild. If you use a configure script
you can change accordingly these elements to use camlp4o instead
of camlp4o.opt .

camlp4cmxs:
  mkdir -p _build
  ocamlopt -shared $(shell ocamlc -where)/camlp4/Camlp4Parsers/Camlp4MacroParser.cmx -o _build/Camlp4MacroParser.cmxs

p

On 02/04/14 22:14, Anders Peter Fugmann wrote:
> Hi,
> 
> While trying to speed up compilation of a project of around ~150 source
> files, i found that ocamlfind invokes a bytecode version of CamlP4.
> 
> Our homemade Makefile uses ocamlfind to compile bytecode and optimized code
> as well as dependency tracking using ocamldep and hand written tools. Almost
> every source file makes use of p4 extensions.
> 
> When compiling though ocamlfind using the -syntax flag, I can see that
> ocamlfind starts ocamlp4 which is ocaml bytecode, and I cannot find any
> flags that allows me to use any optimized version. Also it seems that an
> optimized version of camlp4 is compiled (ocaml compiler is compiled via
> opam).
> 
> Is there any way to use a optimized version of camlp4. Also any other ideas
> on how to speed up compilation would be appreciated.
> 
> Regards
> Anders Fugmann
> 
> 
> -- 
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>