From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: caml-list@sympa.inria.fr Delivered-To: caml-list@sympa.inria.fr Received: from mail3-relais-sop.national.inria.fr (mail3-relais-sop.national.inria.fr [192.134.164.104]) by sympa.inria.fr (Postfix) with ESMTPS id 0F0027FEFE for ; Tue, 5 Jul 2016 11:17:24 +0200 (CEST) IronPort-PHdr: 9a23:KYl8uRM2/LvxRi/8SB8l6mtUPXoX/o7sNwtQ0KIMzox0KPX+rarrMEGX3/hxlliBBdydsKMczbCK+Pm5EUU7or+5+EgYd5JNUxJXwe43pCcHRPC/NEvgMfTxZDY7FskRHHVs/nW8LFQHUJ2mPw6anHS+4HYoFwnlMkItf6KuS9aU1Zj8jrn60qaQSj0AvCC6b7J2IUf+hiTqne5Sv7FfLL0swADCuHpCdrce72ppIVWOg0S0vZ/or9ZLuh5dsPM59sNGTb6yP+FhFeQZXwIdLnst6cb3mR7GRAqJrjtAAyRF2iZPViHB5RD8FrnrszDxtqIp0S2TPMuwQ6o1Qj+ix6NmWB7miWEMMDtvo0/NjcklqqNQo1qKpxVnxofSKNWTP/p3ZqbUe/sYTGxMRdpLWiFdRIi7at1cXKI6Ie9Eotyl9BM1phykCFzpWbri Authentication-Results: mail3-smtp-sop.national.inria.fr; spf=None smtp.pra=nchapman@janestreet.com; spf=Pass smtp.mailfrom=nchapman@janestreet.com; spf=None smtp.helo=postmaster@mxout3.mail.janestreet.com Received-SPF: None (mail3-smtp-sop.national.inria.fr: no sender authenticity information available from domain of nchapman@janestreet.com) identity=pra; client-ip=38.105.200.229; receiver=mail3-smtp-sop.national.inria.fr; envelope-from="nchapman@janestreet.com"; x-sender="nchapman@janestreet.com"; x-conformance=sidf_compatible Received-SPF: Pass (mail3-smtp-sop.national.inria.fr: domain of nchapman@janestreet.com designates 38.105.200.229 as permitted sender) identity=mailfrom; client-ip=38.105.200.229; receiver=mail3-smtp-sop.national.inria.fr; envelope-from="nchapman@janestreet.com"; x-sender="nchapman@janestreet.com"; x-conformance=sidf_compatible; x-record-type="v=spf1" Received-SPF: None (mail3-smtp-sop.national.inria.fr: no sender authenticity information available from domain of postmaster@mxout3.mail.janestreet.com) identity=helo; client-ip=38.105.200.229; receiver=mail3-smtp-sop.national.inria.fr; envelope-from="nchapman@janestreet.com"; x-sender="postmaster@mxout3.mail.janestreet.com"; x-conformance=sidf_compatible X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A0B2AAD3eXtXfeXIaSZCGoQUfAaoDpMkIoV2AoEwBzwQAQEBAQEBAQERAQEJFglQgjKCGwEEARIRHQEBLAsBBAsLCxodAgIiEgEFAQoSBhMSCQeHdAMPCAMLLZtxgTE+MYpUZ4RDAQEFhkgDhBgBAQEBAQUBAQEBAQEBAQEBFQgQhheETYRtghwLLYJahlcMjHuFOoYJiD6COIxyhyuHIBIegQ8PJoQQbQGIeAEBAQ X-IPAS-Result: A0B2AAD3eXtXfeXIaSZCGoQUfAaoDpMkIoV2AoEwBzwQAQEBAQEBAQERAQEJFglQgjKCGwEEARIRHQEBLAsBBAsLCxodAgIiEgEFAQoSBhMSCQeHdAMPCAMLLZtxgTE+MYpUZ4RDAQEFhkgDhBgBAQEBAQUBAQEBAQEBAQEBFQgQhheETYRtghwLLYJahlcMjHuFOoYJiD6COIxyhyuHIBIegQ8PJoQQbQGIeAEBAQ X-IronPort-AV: E=Sophos;i="5.26,579,1459807200"; d="scan'208,217";a="183898052" Received: from mxout3.mail.janestreet.com ([38.105.200.229]) by mail3-smtp-sop.national.inria.fr with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 05 Jul 2016 11:17:22 +0200 Received: from tot-qpr-mailcore2.delacy.com ([172.27.56.106] helo=tot-qpr-mailcore2) by mxout3.mail.janestreet.com with esmtps (TLSv1:DHE-RSA-AES256-SHA:256) (Exim 4.82) (envelope-from ) id 1bKMTp-0001t3-G1 for caml-list@inria.fr; Tue, 05 Jul 2016 05:17:21 -0400 X-JS-Flow: external X-JS-Scanner-attachment: (ok) No attachments Received: by tot-qpr-mailcore2 with JS-mailcore (0.1) (envelope-from ) id BXe3sh-AAACks-OR; 2016-07-05 05:17:21.458989-04:00 Received: from mail-qt0-f197.google.com ([209.85.216.197]) by mxgoog1.mail.janestreet.com with esmtps (UNKNOWN:AES128-GCM-SHA256:128) (Exim 4.72) (envelope-from ) id 1bKMTp-00069Q-CL for caml-list@inria.fr; Tue, 05 Jul 2016 05:17:21 -0400 Received: by mail-qt0-f197.google.com with SMTP id v18so455243736qtv.0 for ; Tue, 05 Jul 2016 02:17:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=janestreet.com; s=google; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=TXmGXxP+7vXVoxGULHOeSqgCWaLnD8UP3gjHJ3WYY3A=; b=QUABYwrFk5y7MxGzgMPO19iXFgZ01C8DCn2PLUWjeeJpnSfnvPh/a7gI3dtjJIbcxU a5z8ktpEjq8dA00bqHjtBYk2RWeAkEUz3vt6OSAt2EejQHQRs/GET6BDSqWpbyDAz+kk 7GL1fEmh4n++n8r4CLOfpmoV4VdQu1BkCKiEw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=TXmGXxP+7vXVoxGULHOeSqgCWaLnD8UP3gjHJ3WYY3A=; b=ll0zckQxw2KXqAnJCAy3LCajV5RUVYasajjFT+YpsvwzT83G3g1NJ7dc0E6LAGV0u1 BoE0EgE4dCmYK1HmsNqkUDbT5/bAzZIscLNfMn1a4/HcH+xG3xsmi/vUh6d3qScIPYFl Zt3X+9B2hBF3nHolXCBjXXWRNddIhKlb5XrKR4eGgMg00ozWSAQJXlEN+KUYrluZBJm3 SUgf1cjyVZv37XEaeQkzO9FAZYyyt9AKK8D/4bEbFhXSB1IxJycSLVNx4q6O/zKfV+Ue xWNZw7j5E/jO2AIvC/nwoiegKucK1DBGrLdoBUvanPHlFeSSYyG97rEDznz3TIUfFFib ZfCQ== X-Gm-Message-State: ALyK8tJ+6tZyQccJbcgk0/jCR+SluQ6m1MwKjwCE+jNHlR8QSCMaDAyRCL1LNjC/NK6uFDZMsWiV0EnC6CBqCWW90ya39l0GM2ImMF4CV8bVTszUDpYITvd2xf7CLHjjaTFhvCZ2vUZUFjlCb9GY7XHNtk82k3kUNhpr0/iRIzc= X-Received: by 10.37.47.76 with SMTP id v73mr9380222ybv.92.1467710240879; Tue, 05 Jul 2016 02:17:20 -0700 (PDT) X-Received: by 10.37.47.76 with SMTP id v73mr9380210ybv.92.1467710240638; Tue, 05 Jul 2016 02:17:20 -0700 (PDT) MIME-Version: 1.0 Received: by 10.37.65.193 with HTTP; Tue, 5 Jul 2016 02:17:20 -0700 (PDT) In-Reply-To: References: From:Nick Chapman Date: Tue, 5 Jul 2016 10:17:20 +0100 Message-ID: To:Alain Frisch Cc:OCaml Mailing List Content-Type: multipart/alternative; boundary=001a1140b9c087078f0536dfeb94 X-JS-Processed-by: mailcore Subject: Re: [Caml-list] ocamldep, transitive dependencies, build systems, flambda --001a1140b9c087078f0536dfeb94 Content-Type: text/plain; charset=UTF-8 Hi Alain, We have a setup at Jane Street quite similar to how you describe. We also install library artifacts into what you call a "pub" directory. We sidestep the ocamldep issues you describe by using ocamldep only to determine dependencies within a library but not between libraries. Dependencies between libraries and handled by requiring the user explicitly list dependent libraries (in a "jbuild" file). And then we setup dependencies on all public .cmi's in the listed libraries. This does mean our dependencies are not as fine grained as one might like. Previously when we packed all our libraries this wasn't an issue since there was only a single .cmi per library anyway - i.e. we had already lost any chance to be more fine grained - but we could perhaps do better now. There was a further issue we needed to solve to get our scheme working. Suppose library A lists library B as a dependency and allows this dependence to be exposed in its interface. Clients of library A will require access to the .cmi's of library B to be compiled but it seems unreasonable to require them to explicitly list library B as a dependency. We solve this by automatically running ocamlinfo on the public .cmi's of a library to discover additional library deps required by clients of the library. We use jenga to setup all the dependencies described above. The above is from memory and represents our approach circa a year ago or so. Some details might be wrong now. Nick Chapman. On 4 July 2016 at 17:49, Alain Frisch wrote: > Dear all, > > I'd like to know if people have good solutions to address the problem > below. > > Assume a large project, with multiple libraries spread over > sub-directories, all managed by a single global build system that tracks > dependencies on a per-file basis (i.e. if a module depends on modules > another library, it is not necessarily recompiled when only modules in that > library are modified). > > For instance, imagine a library in lib1/src with two modules A and B, B.ml > and B.mli both depending on A. Thanks to ocamldep, the build system learns > about the following dependencies (in make syntax): > > lib1/src/B.cmx: lib1/src/A.cmi lib1/src/A.cmx > lib1/src/B.cmi: lib1/src/A.cmi > > For various reasons, one might want to "install" some build artefefacts > (.cmi, .cmx) in staging directories. One possible reason is to expose only > a subset of a library internal modules to other libraries. For our > example, imagine that both A and B are part of the public API. So we create > copy rules and record the associated dependencies to the build system: > > lib1/pub/A.cmx: lib1/src/A.cmx > lib1/pub/A.cmi: lib1/src/A.cmi > lib1/pub/B.cmx: lib1/src/B.cmx > lib1/pub/B.cmi: lib1/src/B.cmi > > Another library lib2/ is only allowed to see this public API, and so is > compiled with "-I $(ROOT)/lib1/pub" (and not "-I $(ROOT)/lib1/src"). A > module C in this library depends directly on B, and the build system thus > infer the following dependencies: > > lib2/src/C.cmx: lib1/pub/B.cmi lib1/pub/B.cmx > > C has no reference to A in its source code so ocamldep has no way to know > that it (transitively) depends on A. The trouble is that some dependencies > are effectively unknown to the build system, which can lead to broken > builds. For instance, when lib1/pub/A.mli is modified and one ask the > build system to refresh lib2/src/C.cmx, the dependencies above will force > only the following files to be refreshed in the process: > > lib1/pub/B.cmi lib1/pub/B.cmx lib1/src/B.cmx lib1/src/B.cmi > lib1/src/A.cmi lib1/src/A.cmx > > So when C.ml is recompiled to produce C.cmx, it will see the old version > of lib1/pub/A.cmi. But even if ocamldep does not report any dependency > from C to A, the type-checker might need to open A.cmi to expand e.g. type > aliases, hence the broken build. I reported this problem in > http://caml.inria.fr/mantis/view.php?id=5624 and the fix we have in place > at LexiFi is to compile in a "strict" mode where the compiler prevents > itself from opening a .cmi file which is not a direct dependency (i.e. the > compiler runs ocamldep internally and restrict its view of the file system > accordingly). This works fine and only forces us to explicitly add some > dummy references. (Typically, if one needs A.cmi to compile C.ml, one > would add a dummy reference to A somewhere in C.ml. And ocamldep will thus > report that C.cmx depends on A.cmi, which will fix the problem above.) > > I'm wondering how other groups manage this kind of problem. > > Moreover, flambda makes the problem actually quite a bit worse. Indeed, > B.cmx can now contain symbolic references to A.cmx, and when compiling > C.cmx, the compiler will complain that it cannot find A.cmx (typically when > a function in B is inlined in C and calls a function in A). This is > warning 58. Simply disabling the warning does not work, since an old > version of A.cmx could remain in lib1/pub, leading to mismatched > implementation digests and to unreliable parallel build. > > One could apply the same trick as for .cmi files, i.e. prevent the > compiler from opening A.cmx if the current unit does not depend (according > to ocamldep) on A. But this is not so good as for interfaces, for two > reasons: > > - It's harder for the user to figure out that an explicit dependency > must be forced, because this is not exposed in the published API (i.e. the > module interfaces), but only in the implementation. Moreover, it depends > on internals of the compiler whether A.cmx is actually needed to compile > C.cmx (e.g. in non-flambda mode, and perhaps in flambda mode with some > settings, it is not needed). > > - We still want to be able *not* to install A.cmi in lib1/pub if A is > not part of the public API of lib1. But this would prevent the code in C > to force a dependency to A. > > > A different direction would be to register extra dependencies between > "installed" files depending on the dependencies between source units. In > the example above, one would register: > > > lib1/pub/B.cmx: lib1/pub/A.cmi lib1/pub/A.cmx > lib1/pub/B.cmi: lib1/pub/A.cmi lib1/src/B.cmi > > > The problem is that this creates interactions between the copy rules > (which are just regular copy commands with the associated dependencies) and > the normal build rules for OCaml units (with automatic discovery of > dependencies with "ocamldep -modules"). In our case, our build system is > omake and these two kinds of rules are completely separated (generic build > rules and one or several "install" rules to expose different APIs to > various parts of the projects). We don't see how to write our build rules > in a modular way and keep the automatic discovery of dependencies. > > The core of the problem, as I see it, is that ocamldep cannot return even > an over-approximation of the actual dependencies of a given unit. It misses > "implicit" dependencies related to either aliases in the type system or > cross-module optimizations in cmx files (with flambda at least, the problem > does not seem to exist at the implementation level for non-flambda mode). > > > So if any other group has faced the same problem and found a nice solution > (with omake or another build system), I'd love to hear about it! > > > -- Alain > > -- > Caml-list mailing list. Subscription management and archives: > https://sympa.inria.fr/sympa/arc/caml-list > Beginner's list: http://groups.yahoo.com/group/ocaml_beginners > Bug reports: http://caml.inria.fr/bin/caml-bugs > --001a1140b9c087078f0536dfeb94 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

Hi Alain,

We = have a setup at Jane Street quite similar to how you describe. We also inst= all library artifacts into what you call a "pub" directory.
=

We sidestep the ocamldep issues you describe by using o= camldep only to determine dependencies within a library but not between lib= raries. Dependencies between libraries and handled by requiring the user ex= plicitly list dependent libraries (in a "jbuild" file). And then = we setup dependencies on all public .cmi's in the listed libraries.

This does mean our dependencies are not as fine grain= ed as one might like. Previously when we packed all our libraries this wasn= 't an issue since there was only a single .cmi per library anyway - i.e= . we had already lost any chance to be more fine grained - but we could per= haps do better now.

There was a further issue we n= eeded to solve to get our scheme working. Suppose library A lists library B= as a dependency and allows this dependence to be exposed in its interface.= Clients of library A will require access to the .cmi's of library B to= be compiled but it seems unreasonable to require them to explicitly list l= ibrary B as a dependency. We solve this by automatically running ocamlinfo = on the public .cmi's of a library to discover additional library deps r= equired by clients of the library.

We use jenga to= setup all the dependencies described above.

The a= bove is from memory and represents our approach circa a year ago or so. Som= e details might be wrong now.

Nick Chapman.
<= div>

On 4 July 2016 at 17:49, Alain Frisch <alain.frisch@lexifi.com> wrote:
Dear all,

I'd like to know if people have good solutions to address the problem b= elow.

Assume a large project, with multiple libraries spread over sub-directories= , all managed by a single global build system that tracks dependencies on a= per-file basis (i.e. if a module depends on modules another library, it is= not necessarily recompiled when only modules in that library are modified)= .

For instance, imagine a library in lib1/src with two modules A and B, B.ml = and B.mli both depending on A.=C2=A0 Thanks to ocamldep, the build system l= earns about the following dependencies (in make syntax):

=C2=A0lib1/src/B.cmx: lib1/src/A.cmi lib1/src/A.cmx
=C2=A0lib1/src/B.cmi: lib1/src/A.cmi

For various reasons, one might want to "install" some build artef= efacts (.cmi, .cmx) in staging directories.=C2=A0 One possible reason is to= expose only a subset of a library internal modules to other libraries.=C2= =A0 For our example, imagine that both A and B are part of the public API. = So we create copy rules and record the associated dependencies to the build= system:

=C2=A0lib1/pub/A.cmx: lib1/src/A.cmx
=C2=A0lib1/pub/A.cmi: lib1/src/A.cmi
=C2=A0lib1/pub/B.cmx: lib1/src/B.cmx
=C2=A0lib1/pub/B.cmi: lib1/src/B.cmi

Another library lib2/ is only allowed to see this public API, and so is com= piled with "-I $(ROOT)/lib1/pub" (and not "-I $(ROOT)/lib1/s= rc").=C2=A0 A module C in this library depends directly on B, and the = build system thus infer the following dependencies:

=C2=A0lib2/src/C.cmx: lib1/pub/B.cmi lib1/pub/B.cmx

C has no reference to A in its source code so ocamldep has no way to know t= hat it (transitively) depends on A.=C2=A0 The trouble is that some dependen= cies are effectively unknown to the build system, which can lead to broken = builds.=C2=A0 For instance, when lib1/pub/A.mli is modified and one ask the= build system to refresh lib2/src/C.cmx, the dependencies above will force = only the following files to be refreshed in the process:

=C2=A0lib1/pub/B.cmi lib1/pub/B.cmx lib1/src/B.cmx lib1/src/B.cmi lib1/src/= A.cmi lib1/src/A.cmx

So when C.ml is recompiled to produce C.cmx, it will see the old version of= lib1/pub/A.cmi.=C2=A0 But even if ocamldep does not report any dependency = from C to A, the type-checker might need to open A.cmi to expand e.g. type = aliases, hence the broken build.=C2=A0 I reported this problem in http://caml.inria.fr/mantis/view.php?id=3D5624 and the fi= x we have in place at LexiFi is to compile in a "strict" mode whe= re the compiler prevents itself from opening a .cmi file which is not a dir= ect dependency (i.e. the compiler runs ocamldep internally and restrict its= view of the file system accordingly).=C2=A0 This works fine and only force= s us to explicitly add some dummy references.=C2=A0 (Typically, if one need= s A.cmi to compile C.ml, one would add a dummy reference to A somewhere in = C.ml.=C2=A0 And ocamldep will thus report that C.cmx depends on A.cmi, whic= h will fix the problem above.)

I'm wondering how other groups manage this kind of problem.

Moreover, flambda makes the problem actually quite a bit worse.=C2=A0 Indee= d, B.cmx can now contain symbolic references to A.cmx, and when compiling C= .cmx, the compiler will complain that it cannot find A.cmx (typically when = a function in B is inlined in C and calls a function in A).=C2=A0 This is w= arning 58.=C2=A0 Simply disabling the warning does not work, since an old v= ersion of A.cmx could remain in lib1/pub, leading to mismatched implementat= ion digests and to unreliable parallel build.

One could apply the same trick as for .cmi files, i.e. prevent the compiler= from opening A.cmx if the current unit does not depend (according to ocaml= dep) on A.=C2=A0 But this is not so good as for interfaces, for two reasons= :

=C2=A0 - It's harder for the user to figure out that an explicit depend= ency must be forced, because this is not exposed in the published API (i.e.= the module interfaces), but only in the implementation.=C2=A0 Moreover, it= depends on internals of the compiler whether A.cmx is actually needed to c= ompile C.cmx (e.g. in non-flambda mode, and perhaps in flambda mode with so= me settings, it is not needed).

=C2=A0 - We still want to be able *not* to install A.cmi in lib1/pub if A i= s not part of the public API of lib1.=C2=A0 But this would prevent the code= in C to force a dependency to A.


A different direction would be to register extra dependencies between "= ;installed" files depending on the dependencies between source units. = In the example above, one would register:


=C2=A0lib1/pub/B.cmx: lib1/pub/A.cmi lib1/pub/A.cmx
=C2=A0lib1/pub/B.cmi: lib1/pub/A.cmi lib1/src/B.cmi


The problem is that this creates interactions between the copy rules (which= are just regular copy commands with the associated dependencies) and the n= ormal build rules for OCaml units (with automatic discovery of dependencies= with "ocamldep -modules").=C2=A0 In our case, our build system i= s omake and these two kinds of rules are completely separated (generic buil= d rules and one or several "install" rules to expose different AP= Is to various parts of the projects).=C2=A0 We don't see how to write o= ur build rules in a modular way and keep the automatic discovery of depende= ncies.

The core of the problem, as I see it, is that ocamldep cannot return even a= n over-approximation of the actual dependencies of a given unit. It misses = "implicit" dependencies related to either aliases in the type sys= tem or cross-module optimizations in cmx files (with flambda at least, the = problem does not seem to exist at the implementation level for non-flambda = mode).


So if any other group has faced the same problem and found a nice solution = (with omake or another build system), I'd love to hear about it!


-- Alain

--
Caml-list mailing list.=C2=A0 Subscription management and archives:
https://sympa.inria.fr/sympa/arc/caml-list
Beginner's list: http://groups.yahoo.com/group/ocam= l_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs

--001a1140b9c087078f0536dfeb94--