[Caml-list] factor 5 speed increase for natively compiled camlp4 parsers

[Caml-list] factor 5 speed increase for natively compiled camlp4 parsers

[Hendrik Tews]

Dear all,

yesterday I managed to natively compile a custom camlp4
application. The speed increase was dramatic: from 10 seconds
down to 1.9 seconds for 1.4 MB of Ocaml code in 121 files. Below
I describe howto compile your camlp4 parser to native-code.
Further below there are some questions for the camlp4
maintainers.


HOW TO GET ``camlp4 pa_o.cmo pa_op.cmo foo.cmo'' DOWN TO NATIVE-CODE?

1. Prerequisites: 

   Some of the files you need are not installed in the standard
   ocaml distribution. Therefore you need a fully configured
   ocaml source tree. Say its located at ${ocamlsource}. Do a
   ``make world.opt'' there. (But don't clean!)

2. Compile the camlp4 extensible grammars for ocaml to native-code.

   cp ${ocamlsource}/camlp4/etc/pa_o.ml .
   ocamlopt -c -I `camlp4 -where` -pp "camlp4r pa_extend.cmo q_MLast.cmo" \
	-o pa_o.cmx pa_o.ml
   cp ${ocamlsource}/camlp4/etc/pa_op.ml .
   ocamlopt -c -I `camlp4 -where` -pp "camlp4r pa_extend.cmo q_MLast.cmo" \
	-o pa_op.cmx pa_op.ml

   You might want to copy the two cmx files to some ${lib}
   directory. If this differs from ``ocamlc -where'' you have to
   add ``-I ${lib}'' in step 4 below.

3. foo.cmx: Compile all the modules you want to load to camlp4 to
   native-code.

4. Do

   ocamlopt -linkall -I `camlp4 -where` \
      odyl.cmxa camlp4.cmxa \
      pa_o.cmx pa_op.cmx \
      pr_dump.cmx \
      foo.cmx \
      ${ocamlsource}/camlp4/odyl/odyl.cmx

   Note the -linkall! The second line is for the camlp4 runtime
   system. The third line is for standard ocaml syntax (with
   optimized parsers). The fourth line contains the standard
   printer (leave it out if you use your own printer). The fivth
   line is your camlp4 extension. The last line is for the main
   program.

   If you get wired errors about non-matching interfaces: Make
   sure your ocaml source tree is in sync with our ocaml
   installation!


4. Try it! I tried it on Otags, the TAGS generator from
   Jean-Francois Monin. Otags just uses camlp4 parsers and some
   customized printers (_not_ the usual camlp4 preprocessor to
   ocamlc approach). My biggest ocaml repository contains 51000
   lines of code in 121 files (1.4 MB). TAGS generation time
   drops from 10.4 seconds to 1.9 seconds. (I'll post an Otags
   patch in a separate email in the near future.)


WHAT IF YOU DON'T NEED EXTENSIBLE GRAMMARS?

   Substitute ``${ocamlsource}/camlp4/compile/pa_o_fast.cmx'' for
   ``pa_o.cmx pa_op.cmx'' in line 3 in step 4 above. 

   Apparently there exists some way to compile camlp4 EXTEND
   expressions to plain stream parser source code. This is used
   during the ocaml build to generate pa_o_fast.ml from pa_o.ml
   and pa_op.ml. camlp4o.opt contains pa_o_fast.cmx.

   Applied to Otags this increases its speed by another factor 
   of 2. Now it takes mearly 1.2 second to generate the TAGS.
   This is almost 10 times faster than the original!


WHAT ABOUT THE REVISED SYNTAX?

   Leave out step two and substiture ``pa_r.cmx pa_rp.cmx'' in
   line 3 in step 4 above. 

   There is no pa_r_fast.ml. camlp4r.opt contains extensible
   grammars. 


QUESTIONS TO EXPERTS AND CAMLP4 MAINTAINERS:

1. Why is the -linkall neccessary? Its clear to me that one needs
   the -linkall when building camlp4o. However, I don't
   understand why camlp4o.opt needs the -linkall.

2. Is the process that generates pa_o_fast.ml applicable to all
   camlp4 grammars? Why is it not used for the revised syntax?
   What is the option -meta_action of pa_extend.cmo doing?

3. Would it be possible to include a ``mknativecamlp4'' script
   that automates the described procedure in the next
   distribution? 

   Would it be possible to build and install pa_o.cmx, pa_op.cmx,
   pa_o_fast.cmx, and odyl.cmx? 


Bye,

Hendrik

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] factor 5 speed increase for natively compiled camlp4 parsers

[Michel Mauny]

Dear Henrik,

Hendrik Tews wrote/écrivait (Tue, Jun 08, 2004 at 01:17:27AM +0200):

> QUESTIONS TO EXPERTS AND CAMLP4 MAINTAINERS:

> 1. Why is the -linkall neccessary? Its clear to me that one needs
>    the -linkall when building camlp4o. However, I don't
>    understand why camlp4o.opt needs the -linkall.

It is because of the programming style used in the modules to be
linked: some of them are not referenced (and therefore not linked in
without -linkall), but have useful actions in their initialisation
code, such as fill in an external reference with a function they've
just defined. The -linkall option makes sure that all initialisation
codes are executed.

Now I agree with you that it looks strange. What I did (in the CVS)
is to use -linkall for building camlp4.cmxa: this way your recipe
works even without -linkall.

> 2. Is the process that generates pa_o_fast.ml applicable to all
>    camlp4 grammars?

No: it relies on the particular structure of files pa_o.ml and
pa_op.ml.

>    Why is it not used for the revised syntax?

I suspect the revised syntax is meant to stay extensible, and this
extensibility would be lost if the parser was compiled `as much as'
the one obtained in pa_o_fast. This is just a guess: I don't master all
Camlp4 secrets (yet) ;-)

>    What is the option -meta_action of pa_extend.cmo doing?

It decides wether actions are to be interpreted as expressions or
meta-expressions (expressions at the meta-level). In short, it decides
wether "2+3" is parsed as the expression 2+3 or the expression
<:expr<2+3>>.

> 3. Would it be possible to include a ``mknativecamlp4'' script
>    that automates the described procedure in the next
>    distribution?

I'll see what I can do, but note that this not a top priority right now.

>    Would it be possible to build and install pa_o.cmx, pa_op.cmx,
>    pa_o_fast.cmx, and odyl.cmx? 

Done in the CVS.

Thanks,

-- Michel

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] factor 5 speed increase for natively compiled camlp4 parsers

[Hendrik Tews]

Michel Mauny <Michel.Mauny@inria.fr> writes:

   Hendrik Tews wrote/écrivait (Tue, Jun 08, 2004 at 01:17:27AM +0200):
   
   >    Why is it not used for the revised syntax?
   
   I suspect the revised syntax is meant to stay extensible, and this
   extensibility would be lost if the parser was compiled `as much as'
   the one obtained in pa_o_fast. 

No, this cannot be the reason. You can never extend the natively
compiled version because you cannot load any modules.
   
Thanks for your answers,

Hendrik

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] factor 5 speed increase for natively compiled camlp4 parsers

[Michel Mauny]

Yes, you're right. Then, I don't know. It might be possible, if
pa_r.ml and pa_rp.ml have the right structure (to be processed in the
same way, producing a pa_r_fast.ml).

-- Michel

Hendrik Tews wrote/écrivait (Thu, Jun 24, 2004 at 09:22:59AM +0200):
> Michel Mauny <Michel.Mauny@inria.fr> writes:
> 
>    Hendrik Tews wrote/écrivait (Tue, Jun 08, 2004 at 01:17:27AM +0200):
>    
>    >    Why is it not used for the revised syntax?
>    
>    I suspect the revised syntax is meant to stay extensible, and this
>    extensibility would be lost if the parser was compiled `as much as'
>    the one obtained in pa_o_fast. 
> 
> No, this cannot be the reason. You can never extend the natively
> compiled version because you cannot load any modules.
>    
> Thanks for your answers,
> 
> Hendrik
> 
> -------------------
> To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
> Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] factor 5 speed increase for natively compiled camlp4 parsers

[Michel Mauny]

Dear Henrik,

Michel Mauny wrote/écrivait (Wed, Jun 23, 2004 at 05:07:25PM +0200):

> Now I agree with you that it looks strange. What I did (in the CVS)
> is to use -linkall for building camlp4.cmxa: this way your recipe
> works even without -linkall.

I'm sorry, but I have to backtrack on this: building with -linkall
forbids using only part of this lib (say pr_o and its deps) without
parts such as odyl.

Regards,

-- Michel

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners