Native code compile time question

Native code compile time question

[Raj Bandyopadhyay]

Hi all

I am implementing a language by generating OCaml code for input source 
and using the OCaml native code compiler. However, I've run into a 
strange problem.

On using my code generator on a large program, I generate a file 
containing about 75K lines of OCaml. The native code compiler takes more 
than 30 *minutes* to compile this file!  The bytecode compiler takes 
about 5 secs.

I ran the native code compiler without any -inline or other 
optimizations and its still the same.

For smaller OCaml files, the compilation time is ok for me. For example, 
it takes about 12 seconds to compile a 15K line OCaml file.

Any ideas why this might be happening and how I can deal with this?

Thanks
Raj

Re: [Caml-list] Native code compile time question

[Basile STARYNKEVITCH]

Raj Bandyopadhyay wrote:
> Hi all
> 
> I am implementing a language by generating OCaml code for input source 
> and using the OCaml native code compiler. However, I've run into a 
> strange problem.

> 
> On using my code generator on a large program, I generate a file containing about 75K lines of OCaml.
> The native code compiler takes more than 30 *minutes* to compile this file!  The bytecode compiler takes about 5 secs. 



It depends of your langage and of the code you are generating, but maybe 
Ocaml is not the right tool for that. Did you consider stuff like LLVM 
http://llvm.org/ or C-- http://cminusminus.org/ both designed to be 
languages to be generated by a program (not to be coded in by a human)?

It could also be that you are generating huge functions. Avoid 
generating huge functions (and you'll get the same kind of troubles if 
you generate huge C functions).

All the low level code generation issues and algorithms (e.g. 
instruction scheduling, register allocation) are violently non linear.

-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***

Re: [Caml-list] Native code compile time question

[Jon Harrop]

On Friday 05 September 2008 17:26:10 Raj Bandyopadhyay wrote:
> Hi all
>
> I am implementing a language by generating OCaml code for input source
> and using the OCaml native code compiler. However, I've run into a
> strange problem.
>
> On using my code generator on a large program, I generate a file
> containing about 75K lines of OCaml. The native code compiler takes more
> than 30 *minutes* to compile this file!  The bytecode compiler takes
> about 5 secs.
>
> I ran the native code compiler without any -inline or other
> optimizations and its still the same.
>
> For smaller OCaml files, the compilation time is ok for me. For example,
> it takes about 12 seconds to compile a 15K line OCaml file.

I have had similar problems and, as you have almost certainly already guessed, 
the problem is that the OCaml compiler is designed to optimize hand-written 
code and its optimizer can easily run away when presented with machine 
generated code. In my case, optimization was irrelevant so I ended up 
encoding the data in strings and parsing them at run-time.

> Any ideas why this might be happening and how I can deal with this?

You might consider rebuilding ocamlopt with profiling to see where the time is 
being spent in the OCaml compiler itself and then make sure your compiler 
does not generate pathological cases.

Without seeing some of your code it is impossible for me to give a more 
productive answer but I found that some inlining-related optimizations can 
bog ocamlopt down without affecting ocamlc. So you might also try -inline 0.

Can you post the generated code?

-- 
Dr Jon Harrop, Flying Frog Consultancy Ltd.
http://www.ffconsultancy.com/?e

Re: [Caml-list] Native code compile time question

[Jon Harrop]

On Friday 05 September 2008 22:32:28 Basile STARYNKEVITCH wrote:
> It depends of your langage and of the code you are generating, but maybe
> Ocaml is not the right tool for that. Did you consider stuff like LLVM
> http://llvm.org...

FWIW: LLVM and its OCaml bindings are absolutely superb.

-- 
Dr Jon Harrop, Flying Frog Consultancy Ltd.
http://www.ffconsultancy.com/?e

Re: [Caml-list] Native code compile time question

[Richard Jones]

On Fri, Sep 05, 2008 at 11:26:10AM -0500, Raj Bandyopadhyay wrote:
> On using my code generator on a large program, I generate a file 
> containing about 75K lines of OCaml. The native code compiler takes more 
> than 30 *minutes* to compile this file!  The bytecode compiler takes 
> about 5 secs.

Yeah, bitstring suffers from this a little bit too.  Not to this
extent, but certainly processing our generated file of kernel parsers
(two levels of generated code) can take 45 seconds whereas a normal
invocation of ocamlopt is sub-second.

Perhaps it's worth just checking that your machine isn't running out
of physical RAM and thrashing.  The OCaml garbage collector performs
very badly when this happens.

Rich.

-- 
Richard Jones
Red Hat

Re: [Caml-list] Native code compile time question

[Christophe TROESTLER]

Hi,

On Fri, 05 Sep 2008 11:26:10 -0500, Raj Bandyopadhyay wrote:
> 
> On using my code generator on a large program, I generate a file
> containing about 75K lines of OCaml. The native code compiler takes
> more than 30 *minutes* to compile this file!  The bytecode compiler
> takes about 5 secs.

Are you speaking of ocamlopt (a bytecode program) or ocamlopt.opt (a
native code program) ?

C.