Re: [Caml-list] why is building ocaml hard?

[Goswin von Brederlow <goswin-v-b@web.de>]

On Sun, Jul 10, 2016 at 01:03:26PM +0200, Gerd Stolpmann wrote:
> Am Samstag, den 09.07.2016, 21:16 -0700 schrieb Martin DeMello:
> > My consistent experience with OCaml has been that the build systems
> > are fiddly and hard to work with, but I've never seen a discussion of
> > why this is so (as opposed to problems with specific build tools).
> > Supposing you were to start from scratch and develop a new build
> > system in a bottom up manner, starting with a set of libraries and
> > utilities and working your way up to a framework or dsl, what would
> > the difficult steps be?
> 
> Well, I don't think you can improve it that much (except... see below).
> Since I've taken over omake and studied it carefully, I think I have
> good insights into the problems.

I'm using oasis with myocamlbuild and one thing I find a total pain:
mssing files (typos in file name). The error reporting when something
goes wrong in myocamlbuild is a total nightmare as it reports on the
most unlikely build chain it can find to get the module you requested.
 
> In short, the big difficulty of OCaml is the strict build topology. You
> need to build a module before the caller of the module. Most build-tool
> related failures come from that. Note that many other languages have
> more relaxed build topologies, or work around the problem by doing
> 2-pass builds (i.e. you first pre-compile and extract interfaces for the
> whole project, and do the actual code generation in the second pass).
> 
> Let's have a closer look why it is relatively error-prone to extract the
> dependencies. The tool in question is ocamldep. It is fairly dumb in so
> far it is only parsing the source code, and then looks at all
> module-related constructs (open, include, module, etc.). Because it
> never looks into already compiled interfaces and also proceeds file by
> file, it may sometimes emit wrong dependency information. For example,
> when there is
> 
> open M1
> open M2
> 
> at the beginning of a file, ocamldep doesn't know whether M2 is another
> top-level module, or whether it is a submodule of M1. ocamldep normally
> errs on the side of generating too many dependencies, which is then
> tried to be corrected by only accepting those deps corresponding to
> existing files. In this example, this would mean that a dependency to M2
> is emitted when there is a file M2.ml. Note that this is wrong when M2
> is actually a submodule of M1 AND the file M2.ml exists.
> 
> So how to fix this? In my opinion there are two solutions. You can
> either have a more intelligent ocamldep (e.g. one that reads in
> non-local cmi files and uses that information and also tries to
> interpret all project ml files at once and not file by file - btw, did
> anybody check whether there is an algorithm that precisely solves the
> problem?).

Which basically means compiling all the sources recursively and
throwing away the AST. The algorithm is simple:

1) open file
2) parse code till you hit an unknown module
3) add dependency on module
4) check for cmi file for module and get list of submodules
   or check for mli file for module and gosub 1
   or check for ml file for module and gosub 1
   or assume module has no submodules
5) add module and submodules to known modules
6) goto 2

Note that this will still fail when you have

open M1
open M2

and M1 is generated and contains M2. You have to run ocamldep again
after M1 is generated.

> The other solution path is to mark toplevel modules in the
> syntax of OCaml (e.g. you'd have to do "open ^M2" is M2 is a toplevel
> module).

That seems rather stupid. As in why do I have to tell it that. Ocamlc
already figures that out just fine. I don't like doing the computers
job.


Here is a 3rd option:

3a) fix ocamlc to build recursively. When it finds an "open M1" and M1
is not yet build then build M1 first. This could be done in 3 ways:

  - Call itself to compile M1. Bad for parallel building and fails
    is M1 is generated (e.g. from mly file).

  - Retrun an error to the build system to build M1 first and retry.
    It's not easy to dynamically add depends in most build systems.
    For most this would probably mean restarting itself till there
    is no more dependency error.

  - A mixture of the two: ocamlc asks the build system to provide
    M1 when it needs it.

The 3rd option seems to be the sanest although figuring out a good
general interface might be tricky. E.g. for GNU make you have to pass
through the filedescriptor for the job server, some ENV vars and
temporarily returns its build token when calling make again.

3b) output dependencies as a side effect of buidling

This seems not needed since 3a ensures everything is build in order.
But knowing the dependencies makes it easier the second time around
and the build system can even watch for changed files and auto
rebuild. Building files in order also prevents many instances of
ocamlc being in memory at the same time. Building from scartch (no
depends yet) might fail because memory is exhausted from all the
ocamlc being started.

Note: ocamlc is just an example. Same goes for ocamlopt, ocamlyacc,
ocamllex, ...


I believe an interactive design, where the build system and the
compiler interact to figure out the dependencies together is the only
way to cover all cases. The compiler knows about modules and
submodules and all the ways the source can create a dependency on
another module and the build system knows about generated files. Both
are needed to figure out the complete dependencies.


> Besides ocamldep, there are also other aspects that affect the
> dependency analysis. E.g. with omake there is a distinction of
> project-local and other dependencies, and you need to set the
> OCAMLINCLUDES variable to add other directories to the local part of the
> analysis, whereas the non-local deps are nowadays handled with
> ocamlfind. First of all, this distinction is not really clear to every
> user, and second, there are some difficulties in processing that when
> both concepts overlap (e.g. you want to also get project-local
> dependency expansion with ocamlfind).
> 
> Note that recently the dependency analysis even became harder because of
> flambda. Now cmx files play a bigger role. In particular a cmx file can
> refer to another cmx file that isn't a direct dependency.  In other
> words, there is a second kind of dependency that is added by the code
> generator. Current build tools cannot record these dependencies yet.
> 
> Gerd

How does that change anything? Afaik if A depends on B and B depends
on C and C changes then B always needs to be rebuild. Therefore there
can never be a change in C.cmx unless B.cmx also changes. So for A
looking at B.cmx is enough. Right?

MfG
	Goswin