managing ocaml dependencies

managing ocaml dependencies

[Guillaume Rousse]

I'm trying to use autoconf for ocaml project, and I have troubles with
managing dependencies.

They are generated through a specific program, ocamldep, that basically
output makefile rules, so as to be used this way:
.depend:
	$(OCAMLDEP) *.ml *.mli > .depend

include .depend

They are two different strategies here, either generate them on
maintainer host and ship them in the distribution, either generate them
on user's host.

The first seems safe, excepted when you have conditional build options:
if the user has not the same setup as the maintainer host, there might
be inconsistencies. I also saw some comments in autotools description
than this strategy was also considered insecure for other languages.

The second strategy, however, heavily relies on make implementation.
Whereas GNU make happily generate .depend file on the fly with previous
snippet, some other implementations don't, such as Digital Unix's one
(and potentially others).
Make: Cannot open ../.depend.  Stop.

I also discovered than GNU make use current directory to resolve path of
included files, even when the inclusion directive is found in an
included makefile, whereas Digital Unix's one consider the directory
containing the makefile containing the inclusion directive instead.

Given the following setup:
|-- Makefile.rules
`-- a
    `-- Makefile

If the inclusion directive is given in top level Makefile.rules, itself
included in lower level a/Makefile, GNU make resolve the .depend file in
a directory, whereas Digital Unix resolve it in top-level directory.

Finally, it seems the safest strategy would be to use configure to
produce those .depend file, or is there any other possibility ?

Re: [Caml-list] managing ocaml dependencies

[skaller]

On Thu, 2006-11-16 at 15:20 +0100, Guillaume Rousse wrote:
> I'm trying to use autoconf for ocaml project, and I have troubles with

> They are two different strategies here, either generate them on
> maintainer host and ship them in the distribution, either generate them
> on user's host.

There is no choice: it has to be done by the developer
because sometimes the dependency is semantic, in particular
when order of initialisation matters.

For example:

(* file 1 *)
let a : int opt = ref None 
;;

(* file 2 *)
let _ = a := Some 99
;;

(* file 3 *)
let _ = 
  match !a with
  | None -> failwith "WRONG INIT ORDER"
  | Some x -> print_endline (string_of_int x)
;;

You must link in the given order. Ocamldep can tell
2 and 3 depend on 1, but not that 3 depends on 2.

> The second strategy, however, heavily relies on make implementation.
> Whereas GNU make happily generate .depend file on the fly with previous
> snippet, some other implementations don't, such as Digital Unix's one
> (and potentially others).
> Make: Cannot open ../.depend.  Stop.

This is easy to fix: generate a new_makefile and then use
a rule something like:

	cat new_makefile_prefix .depend > new_makefile
	make -f new_makefile

Since this is a rule, the order of evaluation is strict.


-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

Re: [Caml-list] managing ocaml dependencies

[Jean-Christophe Filliatre]

skaller writes:
 > On Thu, 2006-11-16 at 15:20 +0100, Guillaume Rousse wrote:
 > 
 > > They are two different strategies here, either generate them on
 > > maintainer host and ship them in the distribution, either generate them
 > > on user's host.
 > 
 > There is no choice: it has to be done by the developer
 > because sometimes the dependency is semantic, in particular
 > when order of initialisation matters.

The link order and the modules dependencies are two different matters.
Typically, the Makefile contains the link order but dependencies can
indeed be generated automatically, even on the fly; with the possible
issues mentioned by Guillaume, though.

Regarding the latter, I use the following typical Makefile for my
projects: http://www.lri.fr/~filliatr/ftp/ocaml/misc/Makefile.in
(together with http://www.lri.fr/~filliatr/ftp/ocaml/misc/configure.in)
and I ship the .depend file within the distributions.
(I don't include Makefiles; I use a single Makefile at the top
directory) But having the dependencies built by the ./configure seems
to be a good solution...

-- 
Jean-Christophe

Re: [Caml-list] managing ocaml dependencies

[Nicolas Pouillard]

On 11/16/06, skaller <skaller@users.sourceforge.net> wrote:
> On Thu, 2006-11-16 at 15:20 +0100, Guillaume Rousse wrote:
> > I'm trying to use autoconf for ocaml project, and I have troubles with
>
> > They are two different strategies here, either generate them on
> > maintainer host and ship them in the distribution, either generate them
> > on user's host.
>
> There is no choice: it has to be done by the developer
> because sometimes the dependency is semantic, in particular
> when order of initialisation matters.

When order of initialization matters, just enforce it.

>
> For example:
>
> (* file 1 *)
> let a : int opt = ref None
> ;;
>
> (* file 2 *)
> let _ = a := Some 99
> ;;
>
> (* file 3 *)

Just add something like:

open File2;; (* because initialization order matters *)

Or if really don't want use open:

module File2 = File2;; (* because initialization order matters *)

Or if really don't want re-export that module (if you don't have a .mli):

let module File2 = File2 in ();; (* because initialization order matters *)

> let _ =
>   match !a with
>   | None -> failwith "WRONG INIT ORDER"
>   | Some x -> print_endline (string_of_int x)
> ;;
>
> You must link in the given order. Ocamldep can tell
> 2 and 3 depend on 1, but not that 3 depends on 2.
>
> > The second strategy, however, heavily relies on make implementation.
> > Whereas GNU make happily generate .depend file on the fly with previous
> > snippet, some other implementations don't, such as Digital Unix's one
> > (and potentially others).
> > Make: Cannot open ../.depend.  Stop.
>
> This is easy to fix: generate a new_makefile and then use
> a rule something like:
>
>         cat new_makefile_prefix .depend > new_makefile
>         make -f new_makefile
>
> Since this is a rule, the order of evaluation is strict.
>

-- 
Nicolas Pouillard

Re: [Caml-list] managing ocaml dependencies

[skaller]

On Thu, 2006-11-16 at 22:43 +0100, Nicolas Pouillard wrote:
> On 11/16/06, skaller <skaller@users.sourceforge.net> wrote:

> When order of initialization matters, just enforce it.

> > For example:
> >
> > (* file 1 *)
> > let a : int opt = ref None
> > ;;
> >
> > (* file 2 *)
> > let _ = a := Some 99
> > ;;
> >
> > (* file 3 *)
> 
> Just add something like:
> 
> open File2;; (* because initialization order matters *)

Really there should be a directive such as:

require File2;;

or something, implied by 'open', and *mandatory* so that

List.fold_left

without a 

require/open List;;

was an error. In any case, using make for this kind of
job is a real negative for Ocaml. Felix compiles
dependencies recursively and conditionally. It makes
using it -- and upgrading it -- a breeze, and eliminates
the need for external tool support.

Hmm. It would be interesting to write a driver harness
that did this. The biggest impediment is the lack of
an enforced 'require' directive, and the existence
of Ocamlp4, both of which make it hard to use simple
analysis tools to calculate dependencies.

BTW: in the Felix compiler itself the initialisation
order problem arises with the 'version' record:
whenever any code changes, the time stamp in the
version record is updated. The version record
is first in the link order, so any part of the code
can refer to it .. but the actual time stamp is
last in link order to avoid gratuitous recompilations.
Its module uses the above trick -- the time stamp
is stored into the version record by executable 
initialisation code dynamically before any
functions are called in the mainline. This implies
that no modules refer to the version record dynamically in 
initialisations (except the mainline).

This is nasty because this version record is
Marshalled out into pre-compiled Felix code,
and is actually then used for the dependency
checking in Felix -- if the compiler version
changes the Felix sources need to be recompiled.

So there is a 'exponential' dependency situation, where
getting the order right is critical,
hard to enforce, and hard to detect when there
is an error.

Skallers Law: if a programmer can make a mistake ..
skaller will make it.

-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

Re: managing ocaml dependencies

[Ralf Wildenhues]

Hello Guillaume,

Do you use Automake?  If yes, or if you want to add support for ocaml to
Automake, then probably its list would be more appropriate.  If not, all
of the following has relevance without Automake, it's just colored that
way a bit.

* Guillaume Rousse wrote on Thu, Nov 16, 2006 at 03:20:42PM CET:
> I'm trying to use autoconf for ocaml project, and I have troubles with
> managing dependencies.
> 
> They are generated through a specific program, ocamldep, that basically
> output makefile rules, so as to be used this way:
> .depend:
> 	$(OCAMLDEP) *.ml *.mli > .depend
> 
> include .depend
> 
> They are two different strategies here, either generate them on
> maintainer host and ship them in the distribution, either generate them
> on user's host.

FWIW, Automake's strategy is building them on the user's system.  There
is a chapter in its manual that contains rationale:
http://sources.redhat.com/automake/automake.html#Dependency-Tracking-Evolution

Further, the macro AM_MAKE_INCLUDE in automake/m4/make.m4 detects the
'include' style used by 'make'.  The automake way to handle
bootstrapping is to create (almost empty) per-target files at
config.status execution time (see m4/depout.m4), and update them as a
side effect of compilation (see m4/depend.m4 and lib/depcomp).  The side
effect method is more efficient once you have a compiler that can output
dependency information on the fly (such as GCC for some languages).

> The second strategy, however, heavily relies on make implementation.
> Whereas GNU make happily generate .depend file on the fly with previous
> snippet, some other implementations don't, such as Digital Unix's one
> (and potentially others).
> Make: Cannot open ../.depend.  Stop.

Does that mean it fails if it updates a file that is included, or that
it merely fails the first time when the file does not yet exist?

> I also discovered than GNU make use current directory to resolve path of
> included files, even when the inclusion directive is found in an
> included makefile, whereas Digital Unix's one consider the directory
> containing the makefile containing the inclusion directive instead.
> 
> Given the following setup:
> |-- Makefile.rules
> `-- a
>     `-- Makefile
> 
> If the inclusion directive is given in top level Makefile.rules, itself
> included in lower level a/Makefile, GNU make resolve the .depend file in
> a directory, whereas Digital Unix resolve it in top-level directory.

Ah, thanks, I did not know that.  (Automake does not use recursive
inclusion at 'make' run time; not to be mistaken with the automake
'include' which is resolved at 'automake' run time.)

> Finally, it seems the safest strategy would be to use configure to
> produce those .depend file, or is there any other possibility ?

Yes, that's the idea.  I think adapting Automake for ocaml should be
possible.  If you go and add support for it in Autoconf first, then it
can be made quite seamless, too.  For examples how to do this, you could
peek at how Erlang support was recently added to Autoconf, and UPC to
Automake.

Cheers,
Ralf