[Caml-list] Size of .cmo / .cmi workload of the compiler

[Caml-list] Size of .cmo / .cmi workload of the compiler

[Christoph Höger]

Dear all,

I autogenerate a rather large (> 12k) set of ocaml modules containing
classes which are parameterized over their final representation to allow
for hierarchic classes with polymorphic open recursion.

My compilation scheme seems to work well in principle, but I am reaching
a frustrating limit in practice: The compilation of the generated ml
files seems to run superlinear (in fact it seems to depend on the
hierarchical location of a class). As it turns out, the generated .cmo
and .cmi files are quite large (up to several hundreds of kb). When I
generate the .mli files or dump the .cmo files however, the output is
quite small (several hundred instructions in the bytecode, the .mli file
contains quite complex objects but still human readable).

Is there any known issue that leads to:

1. non-linear runtime when compiling inter-module classes
2. huge .cmo files outside of the actual bytecode

The generated source code can be found here:

https://www.dropbox.com/s/cllc0xikv9zwu1k/doesnotscale.tar.bz2?dl=0

To test the behavior, unpack and run

ocamlbuild ModelicaXX5FElectricalXX5FMachines.cmo

(there might be type-errors in the generated files somewhere, though).

Any advice or comments are deeply appreciated.

Christoph

-- 
Christoph Höger

Technische Universität Berlin
Fakultät IV - Elektrotechnik und Informatik
Übersetzerbau und Programmiersprachen

Sekr. TEL12-2, Ernst-Reuter-Platz 7, 10587 Berlin

Tel.: +49 (30) 314-24890
E-Mail: christoph.hoeger@tu-berlin.de

Re: [Caml-list] Size of .cmo / .cmi workload of the compiler

[Fabrice Le Fessant]

One possible explanation: object types are almost always expansed in
the .cmi files (and probably in the debug section of .cmo files), and
all the more if all the classes are defined in different .cmi files
(since each type is loaded from a different file, there is no
in-memory sharing when the same type appears in two different
modules).

When we created the Try-OCaml site for Js-of-OCaml
(https://try.ocamlpro.com/js_of_ocaml/), we ran in the same problem,
because js-of-ocaml uses object types for most values, and so the .cmi
files that were loaded were much bigger than for the original
Try-OCaml site (something like 12 MB instead of 1 MB to load the
toplevel). We ended up writting a compressor of .cmi files.

If you are using 4.01.0, you can download a bytecode image of that
compressor in the former repository of try-ocaml:

https://github.com/OCamlPro/tryocaml/tree/master/toplevellib/toplevellib-4.01.0

and then, you can test it on one of your .cmi files to see if it can
compress these files (for Try-OCaml, the image decreased from 12 MB to
2 MB if I remember correctly). Then, you could run it automatically on
each .cmi files after it has been generated by ocamlc, if it is indeed
the source of the problem.

--Fabrice


On Wed, Aug 19, 2015 at 5:38 PM, Christoph Höger
<christoph.hoeger@tu-berlin.de> wrote:
> Dear all,
>
> I autogenerate a rather large (> 12k) set of ocaml modules containing
> classes which are parameterized over their final representation to allow
> for hierarchic classes with polymorphic open recursion.
>
> My compilation scheme seems to work well in principle, but I am reaching
> a frustrating limit in practice: The compilation of the generated ml
> files seems to run superlinear (in fact it seems to depend on the
> hierarchical location of a class). As it turns out, the generated .cmo
> and .cmi files are quite large (up to several hundreds of kb). When I
> generate the .mli files or dump the .cmo files however, the output is
> quite small (several hundred instructions in the bytecode, the .mli file
> contains quite complex objects but still human readable).
>
> Is there any known issue that leads to:
>
> 1. non-linear runtime when compiling inter-module classes
> 2. huge .cmo files outside of the actual bytecode
>
> The generated source code can be found here:
>
> https://www.dropbox.com/s/cllc0xikv9zwu1k/doesnotscale.tar.bz2?dl=0
>
> To test the behavior, unpack and run
>
> ocamlbuild ModelicaXX5FElectricalXX5FMachines.cmo
>
> (there might be type-errors in the generated files somewhere, though).
>
> Any advice or comments are deeply appreciated.
>
> Christoph
>
> --
> Christoph Höger
>
> Technische Universität Berlin
> Fakultät IV - Elektrotechnik und Informatik
> Übersetzerbau und Programmiersprachen
>
> Sekr. TEL12-2, Ernst-Reuter-Platz 7, 10587 Berlin
>
> Tel.: +49 (30) 314-24890
> E-Mail: christoph.hoeger@tu-berlin.de
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs



-- 
Fabrice LE FESSANT
Chercheur en Informatique
INRIA Paris Rocquencourt -- OCamlPro
Programming Languages and Distributed Systems

Re: [Caml-list] Size of .cmo / .cmi workload of the compiler

[Goswin von Brederlow]

On Wed, Aug 19, 2015 at 05:38:12PM +0200, Christoph Höger wrote:
> Dear all,
> 
> I autogenerate a rather large (> 12k) set of ocaml modules containing
> classes which are parameterized over their final representation to allow
> for hierarchic classes with polymorphic open recursion.
> 
> My compilation scheme seems to work well in principle, but I am reaching
> a frustrating limit in practice: The compilation of the generated ml
> files seems to run superlinear (in fact it seems to depend on the
> hierarchical location of a class). As it turns out, the generated .cmo
> and .cmi files are quite large (up to several hundreds of kb). When I
> generate the .mli files or dump the .cmo files however, the output is
> quite small (several hundred instructions in the bytecode, the .mli file
> contains quite complex objects but still human readable).
> 
> Is there any known issue that leads to:
> 
> 1. non-linear runtime when compiling inter-module classes
> 2. huge .cmo files outside of the actual bytecode
> 
> The generated source code can be found here:
> 
> https://www.dropbox.com/s/cllc0xikv9zwu1k/doesnotscale.tar.bz2?dl=0
> 
> To test the behavior, unpack and run
> 
> ocamlbuild ModelicaXX5FElectricalXX5FMachines.cmo
> 
> (there might be type-errors in the generated files somewhere, though).
> 
> Any advice or comments are deeply appreciated.
> 
> Christoph

When you inherit a class the compiler has to parse that class and
everythig it inherits. So it is no surprise to get slower the longer
your inheritance chain gets. And I assume each step in the chain adds
some method so your interface gets bigger and bigger. Sounds like the
cmi file contains the full interface and not just a reference to the
inherited class(es) and the additional code while dumping the cmo file
only shows the additional code.

MfG
	Goswin