[Caml-list] a question about "ocamlopt" and "ocamldep"

[Caml-list] a question about "ocamlopt" and "ocamldep"

[Matej Košík]

Hi,

The "ocamldep" tool generates Makefile dependencies for both situations:
- when we use "ocamlc"
- as well as when we use "ocamlopt"

Dependencies, generated for "*.cmo" files,
are corresponding "*.cmi" files.

This is not surprising.

However, dependencies, generated for "*.cmx" files,
are always other "*.cmx" files.

This is surprising.

Why "*.cmx" files do not depend on "*.cmi" files?

I have noticed this in a bigger project but this phenomenon appear to
happen for arbitrarily small projects.

Consider the following ocamldep-generated couple of rules:

src/ml2c/typing/printtyp.cmo: src/ml2c/typing/types.cmi \
    src/ml2c/typing/primitive.cmi src/ml2c/typing/predef.cmi \
    src/ml2c/typing/path.cmi src/ml2c/typing/outcometree.cmi \
    src/ml2c/typing/oprint.cmi src/ml2c/utils/misc.cmi \
    src/ml2c/parsing/longident.cmi src/ml2c/typing/ident.cmi \
    src/ml2c/typing/env.cmi src/ml2c/typing/ctype.cmi \
    src/ml2c/utils/clflags.cmi src/ml2c/typing/btype.cmi \
    src/ml2c/parsing/asttypes.cmi src/ml2c/typing/printtyp.cmi
src/ml2c/typing/printtyp.cmx: src/ml2c/typing/types.cmx \
    src/ml2c/typing/primitive.cmx src/ml2c/typing/predef.cmx \
    src/ml2c/typing/path.cmx src/ml2c/typing/outcometree.cmx \
    src/ml2c/typing/oprint.cmx src/ml2c/utils/misc.cmx \
    src/ml2c/parsing/longident.cmx src/ml2c/typing/ident.cmx \
    src/ml2c/typing/env.cmx src/ml2c/typing/ctype.cmx \
    src/ml2c/utils/clflags.cmx src/ml2c/typing/btype.cmx \
    src/ml2c/parsing/asttypes.cmx src/ml2c/typing/printtyp.cmi

The second rule seems to be unnecessarily demanding (unless it makes no
sense to compile *.cmi files if we use ocamlopt). It should read:

src/ml2c/typing/printtyp.cmx: src/ml2c/typing/types.cmi \
    src/ml2c/typing/primitive.cmi src/ml2c/typing/predef.cmi \
    src/ml2c/typing/path.cmi src/ml2c/typing/outcometree.cmi \
    src/ml2c/typing/oprint.cmi src/ml2c/utils/misc.cmi \
    src/ml2c/parsing/longident.cmi src/ml2c/typing/ident.cmi \
    src/ml2c/typing/env.cmi src/ml2c/typing/ctype.cmi \
    src/ml2c/utils/clflags.cmi src/ml2c/typing/btype.cmi \
    src/ml2c/parsing/asttypes.cmi src/ml2c/typing/printtyp.cmi

Shouldn't it?

Re: [Caml-list] a question about "ocamlopt" and "ocamldep"

[Matthias Puech]

Hello,

This is consistent with how ocamlc/ocamlopt work: separate compilation 
is ensured the way you think by bytecode .cmo compilation: to build a 
module, you only need the *interfaces* of its dependencies, but it is 
unfortunately not ensured when compiling to native code, because of the 
global (inter-modules) optimizations performed (inlining AFAIK). Thus, 
to build a .cmx module, you need to be aware of the actual *code* of its 
dependencies.

I wonder now if it would be theoretically possible to do these 
optimization, not at compile-time, but delay them until link-time, when 
the code is fully known...

Cheers,
     -m

Le 03/13/2012 07:19 PM, Matej Košík a écrit :
> Hi,
>
> The "ocamldep" tool generates Makefile dependencies for both situations:
> - when we use "ocamlc"
> - as well as when we use "ocamlopt"
>
> Dependencies, generated for "*.cmo" files,
> are corresponding "*.cmi" files.
>
> This is not surprising.
>
> However, dependencies, generated for "*.cmx" files,
> are always other "*.cmx" files.
>
> This is surprising.
>
> Why "*.cmx" files do not depend on "*.cmi" files?
>
> I have noticed this in a bigger project but this phenomenon appear to
> happen for arbitrarily small projects.
>
> Consider the following ocamldep-generated couple of rules:
>
> src/ml2c/typing/printtyp.cmo: src/ml2c/typing/types.cmi \
>      src/ml2c/typing/primitive.cmi src/ml2c/typing/predef.cmi \
>      src/ml2c/typing/path.cmi src/ml2c/typing/outcometree.cmi \
>      src/ml2c/typing/oprint.cmi src/ml2c/utils/misc.cmi \
>      src/ml2c/parsing/longident.cmi src/ml2c/typing/ident.cmi \
>      src/ml2c/typing/env.cmi src/ml2c/typing/ctype.cmi \
>      src/ml2c/utils/clflags.cmi src/ml2c/typing/btype.cmi \
>      src/ml2c/parsing/asttypes.cmi src/ml2c/typing/printtyp.cmi
> src/ml2c/typing/printtyp.cmx: src/ml2c/typing/types.cmx \
>      src/ml2c/typing/primitive.cmx src/ml2c/typing/predef.cmx \
>      src/ml2c/typing/path.cmx src/ml2c/typing/outcometree.cmx \
>      src/ml2c/typing/oprint.cmx src/ml2c/utils/misc.cmx \
>      src/ml2c/parsing/longident.cmx src/ml2c/typing/ident.cmx \
>      src/ml2c/typing/env.cmx src/ml2c/typing/ctype.cmx \
>      src/ml2c/utils/clflags.cmx src/ml2c/typing/btype.cmx \
>      src/ml2c/parsing/asttypes.cmx src/ml2c/typing/printtyp.cmi
>
> The second rule seems to be unnecessarily demanding (unless it makes no
> sense to compile *.cmi files if we use ocamlopt). It should read:
>
> src/ml2c/typing/printtyp.cmx: src/ml2c/typing/types.cmi \
>      src/ml2c/typing/primitive.cmi src/ml2c/typing/predef.cmi \
>      src/ml2c/typing/path.cmi src/ml2c/typing/outcometree.cmi \
>      src/ml2c/typing/oprint.cmi src/ml2c/utils/misc.cmi \
>      src/ml2c/parsing/longident.cmi src/ml2c/typing/ident.cmi \
>      src/ml2c/typing/env.cmi src/ml2c/typing/ctype.cmi \
>      src/ml2c/utils/clflags.cmi src/ml2c/typing/btype.cmi \
>      src/ml2c/parsing/asttypes.cmi src/ml2c/typing/printtyp.cmi
>
> Shouldn't it?
>

Re: [Caml-list] a question about "ocamlopt" and "ocamldep"

[Gabriel Scherer]

> to build a module,
> you only need the *interfaces* of its dependencies, but it is unfortunately
> not ensured when compiling to native code

It is actually possible to have separate compilation for .cmx, just
like for .cmo: when ocamlopt looks for the .cmx for a given
dependency, if it doesn't find it, it doesn't optimize, and doesn't
record it as a dependency. So by selectively removing .cmx from your
compilation environment, you can ensure to have only
interface-dependency on some modules -- at the cost of disabled
optimizations, of course.

(I seem to remember that may happen in particular when you compile
against certain libraries that do not distribute the .cmx separately.)

> I wonder now if it would be theoretically possible to do these optimization,
> not at compile-time, but delay them until link-time, when the code is fully
> known...

Well, you could always recompile just before you want to link. What is
usually called "link-time optimizations" are optimizations that are
convenient to perform on the compiled objects directly (those that the
linkers see). The high-level inlining or memory representation
prediction optimizations are really part of "compilation" (but *when*
you compile in the lifetime of your project is up to you; you could
always distribute source only and compile just before running the
program, without saving the result).

On Tue, Mar 13, 2012 at 7:34 PM, Matthias Puech <puech@cs.unibo.it> wrote:
> Hello,
>
> This is consistent with how ocamlc/ocamlopt work: separate compilation is
> ensured the way you think by bytecode .cmo compilation: to build a module,
> you only need the *interfaces* of its dependencies, but it is unfortunately
> not ensured when compiling to native code, because of the global
> (inter-modules) optimizations performed (inlining AFAIK). Thus, to build a
> .cmx module, you need to be aware of the actual *code* of its dependencies.
>
> I wonder now if it would be theoretically possible to do these optimization,
> not at compile-time, but delay them until link-time, when the code is fully
> known...
>
> Cheers,
>    -m
>
> Le 03/13/2012 07:19 PM, Matej Košík a écrit :
>
>> Hi,
>>
>> The "ocamldep" tool generates Makefile dependencies for both situations:
>> - when we use "ocamlc"
>> - as well as when we use "ocamlopt"
>>
>> Dependencies, generated for "*.cmo" files,
>> are corresponding "*.cmi" files.
>>
>> This is not surprising.
>>
>> However, dependencies, generated for "*.cmx" files,
>> are always other "*.cmx" files.
>>
>> This is surprising.
>>
>> Why "*.cmx" files do not depend on "*.cmi" files?
>>
>> I have noticed this in a bigger project but this phenomenon appear to
>> happen for arbitrarily small projects.
>>
>> Consider the following ocamldep-generated couple of rules:
>>
>> src/ml2c/typing/printtyp.cmo: src/ml2c/typing/types.cmi \
>>     src/ml2c/typing/primitive.cmi src/ml2c/typing/predef.cmi \
>>     src/ml2c/typing/path.cmi src/ml2c/typing/outcometree.cmi \
>>     src/ml2c/typing/oprint.cmi src/ml2c/utils/misc.cmi \
>>     src/ml2c/parsing/longident.cmi src/ml2c/typing/ident.cmi \
>>     src/ml2c/typing/env.cmi src/ml2c/typing/ctype.cmi \
>>     src/ml2c/utils/clflags.cmi src/ml2c/typing/btype.cmi \
>>     src/ml2c/parsing/asttypes.cmi src/ml2c/typing/printtyp.cmi
>> src/ml2c/typing/printtyp.cmx: src/ml2c/typing/types.cmx \
>>     src/ml2c/typing/primitive.cmx src/ml2c/typing/predef.cmx \
>>     src/ml2c/typing/path.cmx src/ml2c/typing/outcometree.cmx \
>>     src/ml2c/typing/oprint.cmx src/ml2c/utils/misc.cmx \
>>     src/ml2c/parsing/longident.cmx src/ml2c/typing/ident.cmx \
>>     src/ml2c/typing/env.cmx src/ml2c/typing/ctype.cmx \
>>     src/ml2c/utils/clflags.cmx src/ml2c/typing/btype.cmx \
>>     src/ml2c/parsing/asttypes.cmx src/ml2c/typing/printtyp.cmi
>>
>> The second rule seems to be unnecessarily demanding (unless it makes no
>> sense to compile *.cmi files if we use ocamlopt). It should read:
>>
>> src/ml2c/typing/printtyp.cmx: src/ml2c/typing/types.cmi \
>>     src/ml2c/typing/primitive.cmi src/ml2c/typing/predef.cmi \
>>     src/ml2c/typing/path.cmi src/ml2c/typing/outcometree.cmi \
>>     src/ml2c/typing/oprint.cmi src/ml2c/utils/misc.cmi \
>>     src/ml2c/parsing/longident.cmi src/ml2c/typing/ident.cmi \
>>     src/ml2c/typing/env.cmi src/ml2c/typing/ctype.cmi \
>>     src/ml2c/utils/clflags.cmi src/ml2c/typing/btype.cmi \
>>     src/ml2c/parsing/asttypes.cmi src/ml2c/typing/printtyp.cmi
>>
>> Shouldn't it?
>>
>
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa-roc.inria.fr/wws/info/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>

Re: [Caml-list] a question about "ocamlopt" and "ocamldep"

[Gabriel Scherer]

Indeed, ocamlc is faster than ocamlopt (because there is less analysis
in the backend, and simply less work to do to bridge the expressivity
gap) and suited for fast compile-and-edit cycles.

There are occasional cases when I've seen a use for ocamlopt, when I
needed to integrate a "run automated tests" in my feedback loop, and
some tests where noticeably faster when natively compiled. I must
admit I have never give a thought to the .cmx handling, as that has
never appeared to be an issue in the past. I'm not sure how hard it is
to integrate a "no .cmx dependency" pass in the Makefile; intuitively,
just doing "mv *.cmx cmx_save_dir/" before each compilation command
ought to be enough.

It might be interesting to add a "don't depend on .cmx" option to the
compiler, so that it's only a flag to add somewhere in your build
system. Given the existence of ocamlc, there is few incentive for
this, and apparently the OCaml maintainers have not considered it
high-priority. See the following bug report (and do not hesitate to
comment there if you are sure that, indeed, you have a real need for
this feature):
  http://caml.inria.fr/mantis/view.php?id=4389

The following bug report is also related to .cmx/.cmxa and
implementation dependency (short story: if as a library provider you
want your users to be able to cross-optimize with the library code,
providing .cmxa is not enough, you also have to provide the separate
.cmx):
  http://caml.inria.fr/mantis/view.php?id=4772

On Wed, Mar 14, 2012 at 2:19 PM, Virgile Prevosto
<virgile.prevosto@m4x.org> wrote:
> 2012/3/14 Matej Košík <5764c029b688c1c0d24a2e97cd764f@gmail.com>:
>
>> There are two scenarios when I use the compiler:
>>
>> Scenario 1 (most frequent): when I want to incrementally remove typing
>> errors during development. Various optimizations do not matter here.
>> What matters is a short time to rebuild everything (that has to be rebuilt).
>>
>> Scenario 2 (rare one): to produce the final product
>> where quality of various optimizations matter more than
>> the amount of required compilation time
>>
>> If dropping dependencies of *.cmx files on other *.cmx files (rather
>> than on *.cmi files) requires manual intervention or careful thinking,
>> then ocamlopt, with this behavior, is not ideal tool for Scenario 1
>> (while still being perfectly suitable for Scenario 2).
>
> Scenario 1 is exactly what bytecode compilation is for, and it is
> indeed fast and without dependencies on .cmo.
>
>
>
>
>
> --
> E tutto per oggi, a la prossima volta
> Virgile
>
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa-roc.inria.fr/wws/info/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>

Re: [Caml-list] a question about "ocamlopt" and "ocamldep"

[Virgile Prevosto]

2012/3/14 Matej Košík <5764c029b688c1c0d24a2e97cd764f@gmail.com>:

> There are two scenarios when I use the compiler:
>
> Scenario 1 (most frequent): when I want to incrementally remove typing
> errors during development. Various optimizations do not matter here.
> What matters is a short time to rebuild everything (that has to be rebuilt).
>
> Scenario 2 (rare one): to produce the final product
> where quality of various optimizations matter more than
> the amount of required compilation time
>
> If dropping dependencies of *.cmx files on other *.cmx files (rather
> than on *.cmi files) requires manual intervention or careful thinking,
> then ocamlopt, with this behavior, is not ideal tool for Scenario 1
> (while still being perfectly suitable for Scenario 2).

Scenario 1 is exactly what bytecode compilation is for, and it is
indeed fast and without dependencies on .cmo.





-- 
E tutto per oggi, a la prossima volta
Virgile

Re: [Caml-list] a question about "ocamlopt" and "ocamldep"

[Matej Košík]

On 03/14/2012 11:23 AM, Gabriel Scherer wrote:
>> :-(
> 
> I don't understand. Why is it sad to have the *ability* to perform
> cross-module implementation-dependent optimizations (at the inevitable
> cost of locally damaging separate compilation) *if* you wish?

There are two scenarios when I use the compiler:

Scenario 1 (most frequent): when I want to incrementally remove typing
errors during development. Various optimizations do not matter here.
What matters is a short time to rebuild everything (that has to be rebuilt).

Scenario 2 (rare one): to produce the final product
where quality of various optimizations matter more than
the amount of required compilation time

If dropping dependencies of *.cmx files on other *.cmx files (rather
than on *.cmi files) requires manual intervention or careful thinking,
then ocamlopt, with this behavior, is not ideal tool for Scenario 1
(while still being perfectly suitable for Scenario 2).

Re: [Caml-list] a question about "ocamlopt" and "ocamldep"

[Gabriel Scherer]

> :-(

I don't understand. Why is it sad to have the *ability* to perform
cross-module implementation-dependent optimizations (at the inevitable
cost of locally damaging separate compilation) *if* you wish?

On Wed, Mar 14, 2012 at 11:31 AM, Matej Košík
<5764c029b688c1c0d24a2e97cd764f@gmail.com> wrote:
> On 03/13/2012 06:34 PM, Matthias Puech wrote:
>> This is consistent with how ocamlc/ocamlopt work: separate compilation
>> is ensured the way you think by bytecode .cmo compilation: to build a
>> module, you only need the *interfaces* of its dependencies, but it is
>> unfortunately not ensured when compiling to native code, because of the
>> global (inter-modules) optimizations performed (inlining AFAIK). Thus,
>> to build a .cmx module, you need to be aware of the actual *code* of its
>> dependencies.
>
> :-(
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa-roc.inria.fr/wws/info/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>

Re: [Caml-list] a question about "ocamlopt" and "ocamldep"

[Matej Košík]

On 03/13/2012 06:34 PM, Matthias Puech wrote:
> This is consistent with how ocamlc/ocamlopt work: separate compilation 
> is ensured the way you think by bytecode .cmo compilation: to build a 
> module, you only need the *interfaces* of its dependencies, but it is 
> unfortunately not ensured when compiling to native code, because of the 
> global (inter-modules) optimizations performed (inlining AFAIK). Thus, 
> to build a .cmx module, you need to be aware of the actual *code* of its 
> dependencies.

:-(