From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: caml-list@sympa.inria.fr Delivered-To: caml-list@sympa.inria.fr Received: from mail2-relais-roc.national.inria.fr (mail2-relais-roc.national.inria.fr [192.134.164.83]) by sympa.inria.fr (Postfix) with ESMTPS id BD0597FE44 for ; Mon, 4 Jul 2016 18:49:16 +0200 (CEST) IronPort-PHdr: 9a23:G1gwexac499uQ5lVZBvRnGf/LSx+4OfEezUN459isYplN5qZpcW6bnLW6fgltlLVR4KTs6sC0LuO9fq6Ej1fqb+681k6OKRWUBEEjchE1ycBO+WiTXPBEfjxciYhF95DXlI2t1uyMExSBdqsLwaK+i760zceF13FOBZvIaytQ8iJ3pzxjbr5qsKbSj4LrQL1Wal1IhSyoFeZnegtqqwmFJwMzADUqGBDYeVcyDAgD1uSmxHh+pX4p8Y7oGx48sgs/M9YUKj8Y79wDfkBVGxnYFYcv5Kw8EGfBUvSvjpPGlkRxz1jCg3BpDTzX5PwtCSykutmxGHONsTzSfUwWC++x6ZtUh7hzikdYW0X6mbS3+V0haNd6Dy7pgd0w8aAaYWcNfw4cLncZtgabW5MTsdfVmpKBYbqPNhHNPYIIesN99q1nFAJtxbrQFT0CQ== Authentication-Results: mail2-smtp-roc.national.inria.fr; spf=None smtp.pra=alain.frisch@lexifi.com; spf=None smtp.mailfrom=alain.frisch@lexifi.com; spf=None smtp.helo=postmaster@mx30.yaziba.net Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of alain.frisch@lexifi.com) identity=pra; client-ip=82.138.71.21; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="alain.frisch@lexifi.com"; x-sender="alain.frisch@lexifi.com"; x-conformance=sidf_compatible Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of alain.frisch@lexifi.com) identity=mailfrom; client-ip=82.138.71.21; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="alain.frisch@lexifi.com"; x-sender="alain.frisch@lexifi.com"; x-conformance=sidf_compatible Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of postmaster@mx30.yaziba.net) identity=helo; client-ip=82.138.71.21; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="alain.frisch@lexifi.com"; x-sender="postmaster@mx30.yaziba.net"; x-conformance=sidf_compatible X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A0BwAQB3knpXfRVHilJBGoQUKlIBuy4ihzA8EAEBAQEBAQEBEQEBCxQJUIIygkQVdgImAl8NCAEBF4gZCi2ZVo9ij3KBAYUngXiKFoJaBZkTgViEMYg+gjiHE4VfhyuIXQI1gVYMgi9sAYkLAQEB X-IPAS-Result: A0BwAQB3knpXfRVHilJBGoQUKlIBuy4ihzA8EAEBAQEBAQEBEQEBCxQJUIIygkQVdgImAl8NCAEBF4gZCi2ZVo9ij3KBAYUngXiKFoJaBZkTgViEMYg+gjiHE4VfhyuIXQI1gVYMgi9sAYkLAQEB X-IronPort-AV: E=Sophos;i="5.26,575,1459807200"; d="scan'208";a="225440630" Received: from mx30.yaziba.net ([82.138.71.21]) by mail2-smtp-roc.national.inria.fr with ESMTP/TLS/DHE-RSA-AES256-SHA; 04 Jul 2016 18:49:16 +0200 Received: from mta10.int.yaziba.net (unknown [217.117.151.14]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx30.yaziba.net (mx10.yaziba.net) with ESMTPS id 2D0B61A76DC for ; Mon, 4 Jul 2016 18:49:15 +0200 (CEST) Received: from mta10.int.yaziba.net (localhost [127.0.0.1]) by mta10.int.yaziba.net (Postfix) with ESMTPS id 55B93CA659 for ; Mon, 4 Jul 2016 18:49:15 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by mta10.int.yaziba.net (Postfix) with ESMTP id 45714CA756 for ; Mon, 4 Jul 2016 18:49:15 +0200 (CEST) X-Virus-Scanned: amavisd-new at mta10.int.yaziba.net Received: from mta10.int.yaziba.net ([127.0.0.1]) by localhost (mta10.int.yaziba.net [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id VwCEYIFT4PAJ for ; Mon, 4 Jul 2016 18:49:15 +0200 (CEST) Received: from [10.0.210.71] (unknown [185.23.92.144]) by mta10.int.yaziba.net (Postfix) with ESMTPSA id 24F5BCA659 for ; Mon, 4 Jul 2016 18:49:15 +0200 (CEST) To: OCaml Mailing List From: Alain Frisch Message-ID: Date: Mon, 4 Jul 2016 18:49:14 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.1.1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-DRWEB-SCAN: ok X-VRSPAM-SCORE: -50 X-VRSPAM-STATE: legit X-VRSPAM-CAUSE: gggruggvucftvghtrhhoucdtuddrfeeltddrvddvgddutdegucetufdoteggodetrfcurfhrohhfihhlvgemucggtfgfnhhsuhgsshgtrhhisggvpdgjtegkkfeuteenuceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmnegoteefjeefqddtgeculdehtddmnecujfgurhepvffhuffkffgfgggtgfesthejredttdefjeenucfhrhhomheptehlrghinhcuhfhrihhstghhuceorghlrghinhdrfhhrihhstghhsehlvgigihhfihdrtghomheqnecuffhomhgrihhnpehinhhrihgrrdhfrhenucfkphepudekhedrvdefrdelvddrudeggeenucfrrghrrghmpehmohguvgepshhmthhpohhuth X-VRSPAM-EXTCAUSE: mhhouggvpehsmhhtphhouhht Subject: [Caml-list] ocamldep, transitive dependencies, build systems, flambda Dear all, I'd like to know if people have good solutions to address the problem below. Assume a large project, with multiple libraries spread over sub-directories, all managed by a single global build system that tracks dependencies on a per-file basis (i.e. if a module depends on modules another library, it is not necessarily recompiled when only modules in that library are modified). For instance, imagine a library in lib1/src with two modules A and B, B.ml and B.mli both depending on A. Thanks to ocamldep, the build system learns about the following dependencies (in make syntax): lib1/src/B.cmx: lib1/src/A.cmi lib1/src/A.cmx lib1/src/B.cmi: lib1/src/A.cmi For various reasons, one might want to "install" some build artefefacts (.cmi, .cmx) in staging directories. One possible reason is to expose only a subset of a library internal modules to other libraries. For our example, imagine that both A and B are part of the public API. So we create copy rules and record the associated dependencies to the build system: lib1/pub/A.cmx: lib1/src/A.cmx lib1/pub/A.cmi: lib1/src/A.cmi lib1/pub/B.cmx: lib1/src/B.cmx lib1/pub/B.cmi: lib1/src/B.cmi Another library lib2/ is only allowed to see this public API, and so is compiled with "-I $(ROOT)/lib1/pub" (and not "-I $(ROOT)/lib1/src"). A module C in this library depends directly on B, and the build system thus infer the following dependencies: lib2/src/C.cmx: lib1/pub/B.cmi lib1/pub/B.cmx C has no reference to A in its source code so ocamldep has no way to know that it (transitively) depends on A. The trouble is that some dependencies are effectively unknown to the build system, which can lead to broken builds. For instance, when lib1/pub/A.mli is modified and one ask the build system to refresh lib2/src/C.cmx, the dependencies above will force only the following files to be refreshed in the process: lib1/pub/B.cmi lib1/pub/B.cmx lib1/src/B.cmx lib1/src/B.cmi lib1/src/A.cmi lib1/src/A.cmx So when C.ml is recompiled to produce C.cmx, it will see the old version of lib1/pub/A.cmi. But even if ocamldep does not report any dependency from C to A, the type-checker might need to open A.cmi to expand e.g. type aliases, hence the broken build. I reported this problem in http://caml.inria.fr/mantis/view.php?id=5624 and the fix we have in place at LexiFi is to compile in a "strict" mode where the compiler prevents itself from opening a .cmi file which is not a direct dependency (i.e. the compiler runs ocamldep internally and restrict its view of the file system accordingly). This works fine and only forces us to explicitly add some dummy references. (Typically, if one needs A.cmi to compile C.ml, one would add a dummy reference to A somewhere in C.ml. And ocamldep will thus report that C.cmx depends on A.cmi, which will fix the problem above.) I'm wondering how other groups manage this kind of problem. Moreover, flambda makes the problem actually quite a bit worse. Indeed, B.cmx can now contain symbolic references to A.cmx, and when compiling C.cmx, the compiler will complain that it cannot find A.cmx (typically when a function in B is inlined in C and calls a function in A). This is warning 58. Simply disabling the warning does not work, since an old version of A.cmx could remain in lib1/pub, leading to mismatched implementation digests and to unreliable parallel build. One could apply the same trick as for .cmi files, i.e. prevent the compiler from opening A.cmx if the current unit does not depend (according to ocamldep) on A. But this is not so good as for interfaces, for two reasons: - It's harder for the user to figure out that an explicit dependency must be forced, because this is not exposed in the published API (i.e. the module interfaces), but only in the implementation. Moreover, it depends on internals of the compiler whether A.cmx is actually needed to compile C.cmx (e.g. in non-flambda mode, and perhaps in flambda mode with some settings, it is not needed). - We still want to be able *not* to install A.cmi in lib1/pub if A is not part of the public API of lib1. But this would prevent the code in C to force a dependency to A. A different direction would be to register extra dependencies between "installed" files depending on the dependencies between source units. In the example above, one would register: lib1/pub/B.cmx: lib1/pub/A.cmi lib1/pub/A.cmx lib1/pub/B.cmi: lib1/pub/A.cmi lib1/src/B.cmi The problem is that this creates interactions between the copy rules (which are just regular copy commands with the associated dependencies) and the normal build rules for OCaml units (with automatic discovery of dependencies with "ocamldep -modules"). In our case, our build system is omake and these two kinds of rules are completely separated (generic build rules and one or several "install" rules to expose different APIs to various parts of the projects). We don't see how to write our build rules in a modular way and keep the automatic discovery of dependencies. The core of the problem, as I see it, is that ocamldep cannot return even an over-approximation of the actual dependencies of a given unit. It misses "implicit" dependencies related to either aliases in the type system or cross-module optimizations in cmx files (with flambda at least, the problem does not seem to exist at the implementation level for non-flambda mode). So if any other group has faced the same problem and found a nice solution (with omake or another build system), I'd love to hear about it! -- Alain