Module hierarchies

[Charles Martin <martin@chasm.org>]

I am wondering how a large OCaml project might be structured, specifically
in terms of directories and files.  I see that the OCaml compiler is
structured as a one-level collection of directories, with the Makefile
drawing them all together.  There's no hierarchy here; the directory
structure is used to partition major components of the system.

I'm used to thinking about breaking large projects down into smaller and
smaller hierarchical subcomponents.  I think the Java mapping between
package structure and directory structure helps, and I wonder whether
something similar could be useful for OCaml?

I can see a natural breakdown for large projects into nested module
hierarchies.  For example, a graphics engine might have modules such as:

     Engine.Graphics.Texture.Manager...

The only way to have such a structure currently would be to put your entire
engine in a single file, which is unrealistic for the programmers and
probably the compiler:

     $ cat engine.ml
     module Graphics = struct ...
       module Texture = struct ...
         module Manager = struct ...

A previous message implies that there will be an "#include" directive in
the next OCaml release.  This would allow you to break up what the compiler
sees as a single file into multiple files for the programmers:

     $ cat engine.ml
     #include "engine/graphics.ml"
     $ cat engine/graphics.ml
     #include "graphics/texture.ml"
     $ cat engine/graphics/texture.ml
     #include "texture/manager.ml"

This would allow you to have a directory and file structure that mirrored
the module structure.  It has the drawback that the compiler still sees
only a single file, which for a large project might doom compile times.

An alternative is to adopt the Java convention, in which a module such a

     engine/graphics/texture/manager.ml

is automagically mapped to Engine.Graphics.Texture.Manager.  The difficulty
now is what to do about the file/module

     engine/graphics/texture.ml <=> Engine.Graphics.Texture

It seems to me the easiest solution is to assume that a directory/file
layout has the semantics of a single file in which the modules are
catenated in depth-first order.  For example, the directory structure:

     a/b/c.ml
     a/b/d.ml
     a/b.ml
     a/e/f.ml
     a/e.ml
     a.ml

would have the semantics of a single file (also named a.ml):

     module B = struct
       module C = struct
         (* contents of c.ml *)
         end
       module D = struct
         (* contents of d.ml *)
         end
       (* contents of b.ml *)
     end
     module E = struct
       module F = struct
         (* contents of f.ml *)
         end
       (* contents of e.ml *)
     end
     (* contents of a.ml *)

Because of the depth-first ordering, the compilation of any individual
module does not depend upon the contents of a module placed higher in the
hierarchy.  Thus, they can be separately compiled.

I think this would be helpful for structuring large projects.  It would
also make it easier to incorporate third-party utilities, since we could
then adopt Java-style naming conventions: the distribution modules would
all be wrapped inside Ocaml, and everything else would use the inverted
domain name convention:

     Ocaml.Pervasives        Pervasives library, comes with the distribution
     Ocaml.Bigarray          Bigarray library, comes with the distribution
     Fr.Inria.Caml.Com       CamlIDL run time library for COM components
     Fr.Ens.Frisch.Getopt    Alan Frisch's parsing of command line args

I'm interested in any feedback on this idea.

Charles