Re: [Caml-list] why is building ocaml hard?

[Gerd Stolpmann <info@gerd-stolpmann.de>]

[-- Attachment #1: Type: text/plain, Size: 3874 bytes --]

Am Samstag, den 09.07.2016, 21:16 -0700 schrieb Martin DeMello:
> My consistent experience with OCaml has been that the build systems
> are fiddly and hard to work with, but I've never seen a discussion of
> why this is so (as opposed to problems with specific build tools).
> Supposing you were to start from scratch and develop a new build
> system in a bottom up manner, starting with a set of libraries and
> utilities and working your way up to a framework or dsl, what would
> the difficult steps be?

Well, I don't think you can improve it that much (except... see below).
Since I've taken over omake and studied it carefully, I think I have
good insights into the problems.

In short, the big difficulty of OCaml is the strict build topology. You
need to build a module before the caller of the module. Most build-tool
related failures come from that. Note that many other languages have
more relaxed build topologies, or work around the problem by doing
2-pass builds (i.e. you first pre-compile and extract interfaces for the
whole project, and do the actual code generation in the second pass).

Let's have a closer look why it is relatively error-prone to extract the
dependencies. The tool in question is ocamldep. It is fairly dumb in so
far it is only parsing the source code, and then looks at all
module-related constructs (open, include, module, etc.). Because it
never looks into already compiled interfaces and also proceeds file by
file, it may sometimes emit wrong dependency information. For example,
when there is

open M1
open M2

at the beginning of a file, ocamldep doesn't know whether M2 is another
top-level module, or whether it is a submodule of M1. ocamldep normally
errs on the side of generating too many dependencies, which is then
tried to be corrected by only accepting those deps corresponding to
existing files. In this example, this would mean that a dependency to M2
is emitted when there is a file M2.ml. Note that this is wrong when M2
is actually a submodule of M1 AND the file M2.ml exists.

So how to fix this? In my opinion there are two solutions. You can
either have a more intelligent ocamldep (e.g. one that reads in
non-local cmi files and uses that information and also tries to
interpret all project ml files at once and not file by file - btw, did
anybody check whether there is an algorithm that precisely solves the
problem?). The other solution path is to mark toplevel modules in the
syntax of OCaml (e.g. you'd have to do "open ^M2" is M2 is a toplevel
module).

Besides ocamldep, there are also other aspects that affect the
dependency analysis. E.g. with omake there is a distinction of
project-local and other dependencies, and you need to set the
OCAMLINCLUDES variable to add other directories to the local part of the
analysis, whereas the non-local deps are nowadays handled with
ocamlfind. First of all, this distinction is not really clear to every
user, and second, there are some difficulties in processing that when
both concepts overlap (e.g. you want to also get project-local
dependency expansion with ocamlfind).

Note that recently the dependency analysis even became harder because of
flambda. Now cmx files play a bigger role. In particular a cmx file can
refer to another cmx file that isn't a direct dependency.  In other
words, there is a second kind of dependency that is added by the code
generator. Current build tools cannot record these dependencies yet.

Gerd

-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
My OCaml site:          http://www.camlcity.org
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]