[Caml-list] ocamldep, transitive dependencies, build systems, flambda

[Caml-list] ocamldep, transitive dependencies, build systems, flambda

[Alain Frisch]

Dear all,

I'd like to know if people have good solutions to address the problem below.

Assume a large project, with multiple libraries spread over 
sub-directories, all managed by a single global build system that tracks 
dependencies on a per-file basis (i.e. if a module depends on modules 
another library, it is not necessarily recompiled when only modules in 
that library are modified).

For instance, imagine a library in lib1/src with two modules A and B, 
B.ml and B.mli both depending on A.  Thanks to ocamldep, the build 
system learns about the following dependencies (in make syntax):

  lib1/src/B.cmx: lib1/src/A.cmi lib1/src/A.cmx
  lib1/src/B.cmi: lib1/src/A.cmi

For various reasons, one might want to "install" some build artefefacts 
(.cmi, .cmx) in staging directories.  One possible reason is to expose 
only a subset of a library internal modules to other libraries.  For our 
example, imagine that both A and B are part of the public API. So we 
create copy rules and record the associated dependencies to the build 
system:

  lib1/pub/A.cmx: lib1/src/A.cmx
  lib1/pub/A.cmi: lib1/src/A.cmi
  lib1/pub/B.cmx: lib1/src/B.cmx
  lib1/pub/B.cmi: lib1/src/B.cmi

Another library lib2/ is only allowed to see this public API, and so is 
compiled with "-I $(ROOT)/lib1/pub" (and not "-I $(ROOT)/lib1/src").  A 
module C in this library depends directly on B, and the build system 
thus infer the following dependencies:

  lib2/src/C.cmx: lib1/pub/B.cmi lib1/pub/B.cmx

C has no reference to A in its source code so ocamldep has no way to 
know that it (transitively) depends on A.  The trouble is that some 
dependencies are effectively unknown to the build system, which can lead 
to broken builds.  For instance, when lib1/pub/A.mli is modified and one 
ask the build system to refresh lib2/src/C.cmx, the dependencies above 
will force only the following files to be refreshed in the process:

  lib1/pub/B.cmi lib1/pub/B.cmx lib1/src/B.cmx lib1/src/B.cmi 
lib1/src/A.cmi lib1/src/A.cmx

So when C.ml is recompiled to produce C.cmx, it will see the old version 
of lib1/pub/A.cmi.  But even if ocamldep does not report any dependency 
from C to A, the type-checker might need to open A.cmi to expand e.g. 
type aliases, hence the broken build.  I reported this problem in 
http://caml.inria.fr/mantis/view.php?id=5624 and the fix we have in 
place at LexiFi is to compile in a "strict" mode where the compiler 
prevents itself from opening a .cmi file which is not a direct 
dependency (i.e. the compiler runs ocamldep internally and restrict its 
view of the file system accordingly).  This works fine and only forces 
us to explicitly add some dummy references.  (Typically, if one needs 
A.cmi to compile C.ml, one would add a dummy reference to A somewhere in 
C.ml.  And ocamldep will thus report that C.cmx depends on A.cmi, which 
will fix the problem above.)

I'm wondering how other groups manage this kind of problem.

Moreover, flambda makes the problem actually quite a bit worse.  Indeed, 
B.cmx can now contain symbolic references to A.cmx, and when compiling 
C.cmx, the compiler will complain that it cannot find A.cmx (typically 
when a function in B is inlined in C and calls a function in A).  This 
is warning 58.  Simply disabling the warning does not work, since an old 
version of A.cmx could remain in lib1/pub, leading to mismatched 
implementation digests and to unreliable parallel build.

One could apply the same trick as for .cmi files, i.e. prevent the 
compiler from opening A.cmx if the current unit does not depend 
(according to ocamldep) on A.  But this is not so good as for 
interfaces, for two reasons:

   - It's harder for the user to figure out that an explicit dependency 
must be forced, because this is not exposed in the published API (i.e. 
the module interfaces), but only in the implementation.  Moreover, it 
depends on internals of the compiler whether A.cmx is actually needed to 
compile C.cmx (e.g. in non-flambda mode, and perhaps in flambda mode 
with some settings, it is not needed).

   - We still want to be able *not* to install A.cmi in lib1/pub if A is 
not part of the public API of lib1.  But this would prevent the code in 
C to force a dependency to A.


A different direction would be to register extra dependencies between 
"installed" files depending on the dependencies between source units. 
In the example above, one would register:


  lib1/pub/B.cmx: lib1/pub/A.cmi lib1/pub/A.cmx
  lib1/pub/B.cmi: lib1/pub/A.cmi lib1/src/B.cmi


The problem is that this creates interactions between the copy rules 
(which are just regular copy commands with the associated dependencies) 
and the normal build rules for OCaml units (with automatic discovery of 
dependencies with "ocamldep -modules").  In our case, our build system 
is omake and these two kinds of rules are completely separated (generic 
build rules and one or several "install" rules to expose different APIs 
to various parts of the projects).  We don't see how to write our build 
rules in a modular way and keep the automatic discovery of dependencies.

The core of the problem, as I see it, is that ocamldep cannot return 
even an over-approximation of the actual dependencies of a given unit. 
It misses "implicit" dependencies related to either aliases in the type 
system or cross-module optimizations in cmx files (with flambda at 
least, the problem does not seem to exist at the implementation level 
for non-flambda mode).


So if any other group has faced the same problem and found a nice 
solution (with omake or another build system), I'd love to hear about it!


-- Alain

Re: [Caml-list] ocamldep, transitive dependencies, build systems, flambda

[Nick Chapman]

[-- Attachment #1: Type: text/plain, Size: 7558 bytes --]

Hi Alain,

We have a setup at Jane Street quite similar to how you describe. We also
install library artifacts into what you call a "pub" directory.

We sidestep the ocamldep issues you describe by using ocamldep only to
determine dependencies within a library but not between libraries.
Dependencies between libraries and handled by requiring the user explicitly
list dependent libraries (in a "jbuild" file). And then we setup
dependencies on all public .cmi's in the listed libraries.

This does mean our dependencies are not as fine grained as one might like.
Previously when we packed all our libraries this wasn't an issue since
there was only a single .cmi per library anyway - i.e. we had already lost
any chance to be more fine grained - but we could perhaps do better now.

There was a further issue we needed to solve to get our scheme working.
Suppose library A lists library B as a dependency and allows this
dependence to be exposed in its interface. Clients of library A will
require access to the .cmi's of library B to be compiled but it seems
unreasonable to require them to explicitly list library B as a dependency.
We solve this by automatically running ocamlinfo on the public .cmi's of a
library to discover additional library deps required by clients of the
library.

We use jenga to setup all the dependencies described above.

The above is from memory and represents our approach circa a year ago or
so. Some details might be wrong now.

Nick Chapman.


On 4 July 2016 at 17:49, Alain Frisch <alain.frisch@lexifi.com> wrote:

> Dear all,
>
> I'd like to know if people have good solutions to address the problem
> below.
>
> Assume a large project, with multiple libraries spread over
> sub-directories, all managed by a single global build system that tracks
> dependencies on a per-file basis (i.e. if a module depends on modules
> another library, it is not necessarily recompiled when only modules in that
> library are modified).
>
> For instance, imagine a library in lib1/src with two modules A and B, B.ml
> and B.mli both depending on A.  Thanks to ocamldep, the build system learns
> about the following dependencies (in make syntax):
>
>  lib1/src/B.cmx: lib1/src/A.cmi lib1/src/A.cmx
>  lib1/src/B.cmi: lib1/src/A.cmi
>
> For various reasons, one might want to "install" some build artefefacts
> (.cmi, .cmx) in staging directories.  One possible reason is to expose only
> a subset of a library internal modules to other libraries.  For our
> example, imagine that both A and B are part of the public API. So we create
> copy rules and record the associated dependencies to the build system:
>
>  lib1/pub/A.cmx: lib1/src/A.cmx
>  lib1/pub/A.cmi: lib1/src/A.cmi
>  lib1/pub/B.cmx: lib1/src/B.cmx
>  lib1/pub/B.cmi: lib1/src/B.cmi
>
> Another library lib2/ is only allowed to see this public API, and so is
> compiled with "-I $(ROOT)/lib1/pub" (and not "-I $(ROOT)/lib1/src").  A
> module C in this library depends directly on B, and the build system thus
> infer the following dependencies:
>
>  lib2/src/C.cmx: lib1/pub/B.cmi lib1/pub/B.cmx
>
> C has no reference to A in its source code so ocamldep has no way to know
> that it (transitively) depends on A.  The trouble is that some dependencies
> are effectively unknown to the build system, which can lead to broken
> builds.  For instance, when lib1/pub/A.mli is modified and one ask the
> build system to refresh lib2/src/C.cmx, the dependencies above will force
> only the following files to be refreshed in the process:
>
>  lib1/pub/B.cmi lib1/pub/B.cmx lib1/src/B.cmx lib1/src/B.cmi
> lib1/src/A.cmi lib1/src/A.cmx
>
> So when C.ml is recompiled to produce C.cmx, it will see the old version
> of lib1/pub/A.cmi.  But even if ocamldep does not report any dependency
> from C to A, the type-checker might need to open A.cmi to expand e.g. type
> aliases, hence the broken build.  I reported this problem in
> http://caml.inria.fr/mantis/view.php?id=5624 and the fix we have in place
> at LexiFi is to compile in a "strict" mode where the compiler prevents
> itself from opening a .cmi file which is not a direct dependency (i.e. the
> compiler runs ocamldep internally and restrict its view of the file system
> accordingly).  This works fine and only forces us to explicitly add some
> dummy references.  (Typically, if one needs A.cmi to compile C.ml, one
> would add a dummy reference to A somewhere in C.ml.  And ocamldep will thus
> report that C.cmx depends on A.cmi, which will fix the problem above.)
>
> I'm wondering how other groups manage this kind of problem.
>
> Moreover, flambda makes the problem actually quite a bit worse.  Indeed,
> B.cmx can now contain symbolic references to A.cmx, and when compiling
> C.cmx, the compiler will complain that it cannot find A.cmx (typically when
> a function in B is inlined in C and calls a function in A).  This is
> warning 58.  Simply disabling the warning does not work, since an old
> version of A.cmx could remain in lib1/pub, leading to mismatched
> implementation digests and to unreliable parallel build.
>
> One could apply the same trick as for .cmi files, i.e. prevent the
> compiler from opening A.cmx if the current unit does not depend (according
> to ocamldep) on A.  But this is not so good as for interfaces, for two
> reasons:
>
>   - It's harder for the user to figure out that an explicit dependency
> must be forced, because this is not exposed in the published API (i.e. the
> module interfaces), but only in the implementation.  Moreover, it depends
> on internals of the compiler whether A.cmx is actually needed to compile
> C.cmx (e.g. in non-flambda mode, and perhaps in flambda mode with some
> settings, it is not needed).
>
>   - We still want to be able *not* to install A.cmi in lib1/pub if A is
> not part of the public API of lib1.  But this would prevent the code in C
> to force a dependency to A.
>
>
> A different direction would be to register extra dependencies between
> "installed" files depending on the dependencies between source units. In
> the example above, one would register:
>
>
>  lib1/pub/B.cmx: lib1/pub/A.cmi lib1/pub/A.cmx
>  lib1/pub/B.cmi: lib1/pub/A.cmi lib1/src/B.cmi
>
>
> The problem is that this creates interactions between the copy rules
> (which are just regular copy commands with the associated dependencies) and
> the normal build rules for OCaml units (with automatic discovery of
> dependencies with "ocamldep -modules").  In our case, our build system is
> omake and these two kinds of rules are completely separated (generic build
> rules and one or several "install" rules to expose different APIs to
> various parts of the projects).  We don't see how to write our build rules
> in a modular way and keep the automatic discovery of dependencies.
>
> The core of the problem, as I see it, is that ocamldep cannot return even
> an over-approximation of the actual dependencies of a given unit. It misses
> "implicit" dependencies related to either aliases in the type system or
> cross-module optimizations in cmx files (with flambda at least, the problem
> does not seem to exist at the implementation level for non-flambda mode).
>
>
> So if any other group has faced the same problem and found a nice solution
> (with omake or another build system), I'd love to hear about it!
>
>
> -- Alain
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>

[-- Attachment #2: Type: text/html, Size: 8738 bytes --]

Re: [Caml-list] ocamldep, transitive dependencies, build systems, flambda

[François Bobot]

On 04/07/2016 18:49, Alain Frisch wrote:
> So if any other group has faced the same problem and found a nice solution (with omake or another
> build system), I'd love to hear about it!

The solution is not nice and similar to JaneStreet solution.

Nick Chapman wrote:
> We sidestep the ocamldep issues you describe by using ocamldep only to determine dependencies
> within a library but not between libraries. Dependencies between libraries and handled by
> requiring the user explicitly list dependent libraries

But we do everything in `Makefile` with `make` and we are using packs. The only tricky point is to 
create dumb `.ml` and `.mli` in the "pub" directory so that ocamldep add the right dependencies.

When I tried to use module-alias instead of packs I failed.

-- 
François

Re: [Caml-list] ocamldep, transitive dependencies, build systems, flambda

[Gerd Stolpmann]

[-- Attachment #1: Type: text/plain, Size: 8566 bytes --]

Hi Alain,

my thinking is that the core of the issue is more a naming problem.
Currently, cmi and cmx files are named by the module to which they
refer. This works well as long as all modules are exposed to the user.

What you need is a way of hiding the module while keeping a reference to
the "module as such" - which got kind of anonymous after removing the
cmi. Let's follow this idea for a while: Support for anonymous modules.
Imagine there was a command

ocamlhide A

which reads a.cmi and outputs _a_900150983cd24fb0d6963f7d28e17f72.cmi,
and the name consisted of the original module name and the module
checksum. Same for the cmx file. In this form, the cmi file can no
longer be directly used, but indirect references from other modules are
still possible (the compiler would use the checksum for resolving the
dependency). These files can then be installed into the pub directory.

First of all, this solves an imminent problem of just not installing the
cmi, namely a possible conflict with the name of a user's module (what
will happen when another module A enters the scene later?). Second, all
problems with different versions of the same module are gone (well, I'm
not sure about the symbols, but let's consider this as a solvable
detail). In particular, it is possible to have several versions of a
module at the same time. If a particular version of a module is demanded
for linking, it will be available and not conflict with any older
artifact.

Of course, ocamldep still has no chance to see such hidden deps by only
looking at the sources. I don't have a really good idea that would
easily integrate with build tools. If you manually manage the deps you'd
have to do it this way:

lib1/pub/B.cmx: lib1/src/B.cmx lib1/src/A.cmx
lib1/pub/B.cmi: lib1/src/B.cmi lib1/src/A.cmi

Because when A.cmi changes the reference in B.cmi to the anonymous
module also changes. You cannot use the files with the checksums,
because this is exactly the part that would be updated by the rebuild.

This is just a thought. I'm sure whether a good solution is in reach
with these ideas.

Gerd


Am Montag, den 04.07.2016, 18:49 +0200 schrieb Alain Frisch:
> Dear all,
> 
> I'd like to know if people have good solutions to address the problem below.
> 
> Assume a large project, with multiple libraries spread over 
> sub-directories, all managed by a single global build system that tracks 
> dependencies on a per-file basis (i.e. if a module depends on modules 
> another library, it is not necessarily recompiled when only modules in 
> that library are modified).
> 
> For instance, imagine a library in lib1/src with two modules A and B, 
> B.ml and B.mli both depending on A.  Thanks to ocamldep, the build 
> system learns about the following dependencies (in make syntax):
> 
>   lib1/src/B.cmx: lib1/src/A.cmi lib1/src/A.cmx
>   lib1/src/B.cmi: lib1/src/A.cmi
> 
> For various reasons, one might want to "install" some build artefefacts 
> (.cmi, .cmx) in staging directories.  One possible reason is to expose 
> only a subset of a library internal modules to other libraries.  For our 
> example, imagine that both A and B are part of the public API. So we 
> create copy rules and record the associated dependencies to the build 
> system:
> 
>   lib1/pub/A.cmx: lib1/src/A.cmx
>   lib1/pub/A.cmi: lib1/src/A.cmi
>   lib1/pub/B.cmx: lib1/src/B.cmx
>   lib1/pub/B.cmi: lib1/src/B.cmi
> 
> Another library lib2/ is only allowed to see this public API, and so is 
> compiled with "-I $(ROOT)/lib1/pub" (and not "-I $(ROOT)/lib1/src").  A 
> module C in this library depends directly on B, and the build system 
> thus infer the following dependencies:
> 
>   lib2/src/C.cmx: lib1/pub/B.cmi lib1/pub/B.cmx
> 
> C has no reference to A in its source code so ocamldep has no way to 
> know that it (transitively) depends on A.  The trouble is that some 
> dependencies are effectively unknown to the build system, which can lead 
> to broken builds.  For instance, when lib1/pub/A.mli is modified and one 
> ask the build system to refresh lib2/src/C.cmx, the dependencies above 
> will force only the following files to be refreshed in the process:
> 
>   lib1/pub/B.cmi lib1/pub/B.cmx lib1/src/B.cmx lib1/src/B.cmi 
> lib1/src/A.cmi lib1/src/A.cmx
> 
> So when C.ml is recompiled to produce C.cmx, it will see the old version 
> of lib1/pub/A.cmi.  But even if ocamldep does not report any dependency 
> from C to A, the type-checker might need to open A.cmi to expand e.g. 
> type aliases, hence the broken build.  I reported this problem in 
> http://caml.inria.fr/mantis/view.php?id=5624 and the fix we have in 
> place at LexiFi is to compile in a "strict" mode where the compiler 
> prevents itself from opening a .cmi file which is not a direct 
> dependency (i.e. the compiler runs ocamldep internally and restrict its 
> view of the file system accordingly).  This works fine and only forces 
> us to explicitly add some dummy references.  (Typically, if one needs 
> A.cmi to compile C.ml, one would add a dummy reference to A somewhere in 
> C.ml.  And ocamldep will thus report that C.cmx depends on A.cmi, which 
> will fix the problem above.)
> 
> I'm wondering how other groups manage this kind of problem.
> 
> Moreover, flambda makes the problem actually quite a bit worse.  Indeed, 
> B.cmx can now contain symbolic references to A.cmx, and when compiling 
> C.cmx, the compiler will complain that it cannot find A.cmx (typically 
> when a function in B is inlined in C and calls a function in A).  This 
> is warning 58.  Simply disabling the warning does not work, since an old 
> version of A.cmx could remain in lib1/pub, leading to mismatched 
> implementation digests and to unreliable parallel build.
> 
> One could apply the same trick as for .cmi files, i.e. prevent the 
> compiler from opening A.cmx if the current unit does not depend 
> (according to ocamldep) on A.  But this is not so good as for 
> interfaces, for two reasons:
> 
>    - It's harder for the user to figure out that an explicit dependency 
> must be forced, because this is not exposed in the published API (i.e. 
> the module interfaces), but only in the implementation.  Moreover, it 
> depends on internals of the compiler whether A.cmx is actually needed to 
> compile C.cmx (e.g. in non-flambda mode, and perhaps in flambda mode 
> with some settings, it is not needed).
> 
>    - We still want to be able *not* to install A.cmi in lib1/pub if A is 
> not part of the public API of lib1.  But this would prevent the code in 
> C to force a dependency to A.
> 
> 
> A different direction would be to register extra dependencies between 
> "installed" files depending on the dependencies between source units. 
> In the example above, one would register:
> 
> 
>   lib1/pub/B.cmx: lib1/pub/A.cmi lib1/pub/A.cmx
>   lib1/pub/B.cmi: lib1/pub/A.cmi lib1/src/B.cmi
> 
> 
> The problem is that this creates interactions between the copy rules 
> (which are just regular copy commands with the associated dependencies) 
> and the normal build rules for OCaml units (with automatic discovery of 
> dependencies with "ocamldep -modules").  In our case, our build system 
> is omake and these two kinds of rules are completely separated (generic 
> build rules and one or several "install" rules to expose different APIs 
> to various parts of the projects).  We don't see how to write our build 
> rules in a modular way and keep the automatic discovery of dependencies.
> 
> The core of the problem, as I see it, is that ocamldep cannot return 
> even an over-approximation of the actual dependencies of a given unit. 
> It misses "implicit" dependencies related to either aliases in the type 
> system or cross-module optimizations in cmx files (with flambda at 
> least, the problem does not seem to exist at the implementation level 
> for non-flambda mode).
> 
> 
> So if any other group has faced the same problem and found a nice 
> solution (with omake or another build system), I'd love to hear about it!
> 
> 
> -- Alain
> 

-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
My OCaml site:          http://www.camlcity.org
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Caml-list] ocamldep, transitive dependencies, build systems, flambda

[Alain Frisch]

Thanks to everyone who commented on my message!

On 05/07/2016 11:17, Nick Chapman wrote:
> There was a further issue we needed to solve to get our scheme working.
> Suppose library A lists library B as a dependency and allows this
> dependence to be exposed in its interface. Clients of library A will
> require access to the .cmi's of library B to be compiled but it seems
> unreasonable to require them to explicitly list library B as a
> dependency. We solve this by automatically running ocamlinfo on the
> public .cmi's of a library to discover additional library deps required
> by clients of the library.

I think this is another instance of the same problem, namely that there 
as "implicit" dependencies that cannot be seen by ocamldep.  It's just 
reduced a bit thanks to the less fine-grained dependency analysis across 
libraries (which we try to avoid in our case).

If we wanted to use the same trick on a per file-basis, one would need 
to inject dynamic dependencies (obtained after compilation), but I don't 
see how to do that with our omake-based build system.

Have you had to change some build rules to account for flambda (I guess 
you must now look for implementation deps in .cmx in addition to 
interface deps in .cmi)?

Here is the solution we you have adopted for now for our experiments 
with flambda.  We apply the same "dependency restriction" for .cmx files 
as we do for .cmi files, namely: whenever the compiler tries to open 
A.cmx when compiling B.ml, it does as if A had been compiled with 
-opaque unless A is an explicit dependency of B.ml (i.e. ocamldep would 
see "A" when analyzing "B.ml").  In effect, this will prevent some 
cross-module optimizations, but we don't see how to do better without 
significantly changing our build strategy.   This solution is similar to 
Hongbo's comment about cross-module inlining in Bucklescript.


Alain

Re: [Caml-list] ocamldep, transitive dependencies, build systems, flambda

[Goswin von Brederlow]

On 05/07/2016 11:17, Nick Chapman wrote:
> There was a further issue we needed to solve to get our scheme working.
> Suppose library A lists library B as a dependency and allows this
> dependence to be exposed in its interface. Clients of library A will
> require access to the .cmi's of library B to be compiled but it seems
> unreasonable to require them to explicitly list library B as a
> dependency. We solve this by automatically running ocamlinfo on the
> public .cmi's of a library to discover additional library deps required
> by clients of the library.

Debian has a verry simple policy for dynamic link libraries:

    A library must itself be linked against any other lib it uses.

That way applications can just link libfoo if the use it and any lib
used by libfoo is automatically included as well. If libfoo changes
what other libs it needs then they get included without the app having
to do anything.


The same should work for cmi files. As you say the cmi file already
contains the info what other libs are needed. Ocaml should include
them automatically and ocamldep and the build system should not have
to care.

So I agree with you but want to go one step further. Don't even
automatically figure out the depends so you can add the libs to the
command line when linking. Have ocaml add them implizitly itself.

MfG
	Goswin

Re: [Caml-list] ocamldep, transitive dependencies, build systems, flambda

[Daniel Bünzli]

Le mardi, 5 juillet 2016 à 11:17, Nick Chapman a écrit :
> There was a further issue we needed to solve to get our scheme working. Suppose library A lists library B as a dependency and allows this dependence to be exposed in its interface. Clients of library A will require access to the .cmi's of library B to be compiled but it seems unreasonable to require them to explicitly list library B as a dependency. We solve this by automatically running ocamlinfo on the public .cmi's of a library to discover additional library deps required by clients of the library.

I think this is the best way of having underspecified dependencies. Suppose the client starts using B aswell, it won't have anything to specify since A is using it. Now suppose that for some reason A stops using B, suddenly your package will no longer compile because it's missing a dependency towards B.

Best,  

Daniel
  

Re: [Caml-list] ocamldep, transitive dependencies, build systems, flambda

[Gabriel Scherer]

[-- Attachment #1: Type: text/plain, Size: 2685 bytes --]

I think that the question was more about how owner of large codebases
approach the problem in practice today, but in terms of "what do people
think about this" I would still stand by my comment on the mantis issue:

ocamldep is an heuristic tool based on some assumptions about how OCaml
> users typically organize their development. I'm not sure it is reasonable
> to try to evolve it into an omniscient tool capable of dealing with your
> sophisticated file manipulations. That's what "by hand" compilation scripts
> (or special-case tools designed hand-in-hand with your quite specific build
> system) are for: absolute flexibility.
>
> I'm not convinced in particular that doing deep modification to the
> semantics of .cmi files and/or the compilation process to work around
> limitations of ocamldep is such a good idea.
>
> I would rather welcome richer ways to inform the compiler of the mapping
> from Ocaml world compilation unit names to filesystem object files than the
> current rules allow; and/or potentially a type-checker design facilitating
> discovery of dependencies and interaction with library providers and build
> systems (implement what currently is ocamldep as a layer between parsing
> and type-checking, that resolves such mapping, is able to access existing
> .cmi files for semantically accurate naming information, and to make
> asynchronous queries to a build system for the missing .cmi).


That said, I also know that Alain has explored these options (eg.
intercepting calls to other files in the compiler) further than many of
thus, so he probably has good reasons for sticking to a practical
compromise.

On Tue, Jul 5, 2016 at 9:06 AM, Hongbo Zhang (BLOOMBERG/ 731 LEX) <
hzhang295@bloomberg.net> wrote:

> Hi Alain,
> I found the same problem when we did aggressive cross-module inlining in
> bucklescript. It makes it too hard to build a correct build system, so
> currently the inlining will stop at B, it will not propagate to C, what do
> other people think about this?
>
> From: alain.frisch@lexifi.com At: 07/04/16 12:49:32
> To: caml-list@inria.fr
> Subject: Re:[Caml-list] ocamldep, transitive dependencies, build systems,
> flambda
>
>
>
> Moreover, flambda makes the problem actually quite a bit worse. Indeed,
> B.cmx can now contain symbolic references to A.cmx, and when compiling
> C.cmx, the compiler will complain that it cannot find A.cmx (typically
> when a function in B is inlined in C and calls a function in A). This
> is warning 58. Simply disabling the warning does not work, since an old
> version of A.cmx could remain in lib1/pub, leading to mismatched
> implementation digests and to unreliable parallel build.
>
>
>
>

[-- Attachment #2: Type: text/html, Size: 3581 bytes --]

Re: [Caml-list] ocamldep, transitive dependencies, build systems, flambda

[Hongbo Zhang (BLOOMBERG/ 731 LEX)]

[-- Attachment #1: Type: text/plain, Size: 912 bytes --]

Hi Alain,
   I found the same problem when we did aggressive cross-module inlining in bucklescript. It makes it too hard to build a correct build system, so currently the inlining will stop at B, it will not propagate to C, what do other people think about this?

From: alain.frisch@lexifi.com At: 07/04/16 12:49:32
To: caml-list@inria.fr
Subject: Re:[Caml-list] ocamldep, transitive dependencies, build systems, flambda


Moreover, flambda makes the problem actually quite a bit worse.  Indeed, 
B.cmx can now contain symbolic references to A.cmx, and when compiling 
C.cmx, the compiler will complain that it cannot find A.cmx (typically 
when a function in B is inlined in C and calls a function in A).  This 
is warning 58.  Simply disabling the warning does not work, since an old 
version of A.cmx could remain in lib1/pub, leading to mismatched 
implementation digests and to unreliable parallel build.



[-- Attachment #2: Type: text/html, Size: 1933 bytes --]