From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: caml-list@sympa.inria.fr Delivered-To: caml-list@sympa.inria.fr Received: from mail2-relais-roc.national.inria.fr (mail2-relais-roc.national.inria.fr [192.134.164.83]) by sympa.inria.fr (Postfix) with ESMTPS id B9A4E7FE44 for ; Tue, 5 Jul 2016 15:53:47 +0200 (CEST) IronPort-PHdr: 9a23:1TY0FxOBVgYhJxZqnLYl6mtUPXoX/o7sNwtQ0KIMzox0KPv8rarrMEGX3/hxlliBBdydsKMczbCK+P6/EUU7or+5+EgYd5JNUxJXwe43pCcHRPC/NEvgMfTxZDY7FskRHHVs/nW8LFQHUJ2mPw6anHS+4HYoFwnlMkItf6KuS9aU1Zj8ib/60qaQSj0AvCC6b7J2IUf+hiTqne5Sv7FfLL0swADCuHpCdrce72ppIVWOg0S0vZ/or9ZLuh5dsPM59sNGTb6yP+FhFeQZX3waNDUc4MTqs1HtVwqU7XtUBmwSmxtORQbf7QrxXr/1vzv7uOs70y6fa57YV7cxDBuv9asjcwPvjD8CPjg/uDXWjMJ3pK1Wuh7kvAB4x5bRaYeTcvZzKPCONegGTHZMC54CHxdKBZmxOs5WV7IM Authentication-Results: mail2-smtp-roc.national.inria.fr; spf=None smtp.pra=info@gerd-stolpmann.de; spf=None smtp.mailfrom=info@gerd-stolpmann.de; spf=None smtp.helo=postmaster@mout.kundenserver.de Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of info@gerd-stolpmann.de) identity=pra; client-ip=212.227.126.133; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="info@gerd-stolpmann.de"; x-sender="info@gerd-stolpmann.de"; x-conformance=sidf_compatible Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of info@gerd-stolpmann.de) identity=mailfrom; client-ip=212.227.126.133; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="info@gerd-stolpmann.de"; x-sender="info@gerd-stolpmann.de"; x-conformance=sidf_compatible Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of postmaster@mout.kundenserver.de) identity=helo; client-ip=212.227.126.133; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="info@gerd-stolpmann.de"; x-sender="postmaster@mout.kundenserver.de"; x-conformance=sidf_compatible X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A0CbAAAUu3tXf4V+49RCGoMRgQN8sVqHa4F3IoV2AoEyORMBAQEBAQEBAREBAQkLCwkfMYIyghsBBVUkEAtGVwYTCRKIFgMBCS26EQEBAQEBBQEBAQEVDoUvhUWETII9C4MHBYgahWqLD4E3AoRQiD6COIcPhWOHK4hfIAKEJGwBiHgBAQE X-IPAS-Result: A0CbAAAUu3tXf4V+49RCGoMRgQN8sVqHa4F3IoV2AoEyORMBAQEBAQEBAREBAQkLCwkfMYIyghsBBVUkEAtGVwYTCRKIFgMBCS26EQEBAQEBBQEBAQEVDoUvhUWETII9C4MHBYgahWqLD4E3AoRQiD6COIcPhWOHK4hfIAKEJGwBiHgBAQE X-IronPort-AV: E=Sophos;i="5.26,580,1459807200"; d="asc'?scan'208";a="225664301" Received: from mout.kundenserver.de ([212.227.126.133]) by mail2-smtp-roc.national.inria.fr with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 05 Jul 2016 15:53:46 +0200 Received: from office1.lan.sumadev.de ([84.59.134.100]) by mrelayeu.kundenserver.de (mreue005) with ESMTPSA (Nemesis) id 0MOETY-1bH7Ph2i60-005cwz; Tue, 05 Jul 2016 15:53:45 +0200 Received: from [IPv6:fd55:cf:6598:7f:1fe:405e:ce30:a214] (unknown [IPv6:fd55:cf:6598:7f:1fe:405e:ce30:a214]) by office1.lan.sumadev.de (Postfix) with ESMTPSA id 0E792DC05D; Tue, 5 Jul 2016 15:53:45 +0200 (CEST) Message-ID: <1467726819.17506.22.camel@e130.lan.sumadev.de> From: Gerd Stolpmann To: Alain Frisch Cc: OCaml Mailing List Date: Tue, 05 Jul 2016 15:53:39 +0200 In-Reply-To: References: Content-Type: multipart/signed; micalg="pgp-sha1"; protocol="application/pgp-signature"; boundary="=-feYDjRmraufquCtn3J1W" X-Mailer: Evolution 3.10.4-0ubuntu2 Mime-Version: 1.0 X-Provags-ID: V03:K0:zObgIi6G/ZCGk+dKdlyvjyLdfTpA25EI94VBzZ0IsSGiH8/Dcrk Zh0eVTIAzLVx9/rXAwoAkgFGZ/2DuWp0t1aqsFYVSjnBs8MewJ3d6fPyCGaHOV2/jjXJdH6 IY7GrdIlFiK4VYCH1n4/Lrt9NpTdFM+k+2RsyAQMYGrXjqaQLtw4rf5gdYqJy1ZUfBEGXlI FvK0v5oxtVsbdtBxahetg== X-UI-Out-Filterresults: notjunk:1;V01:K0:olv31OCM9tE=:jt7JD6LRpVgfB172ADbxsv Tq6XxYjuwlr41LuTSEKMfYFAykelBb8Glloa6OAdw5mjJ3YC1ADhf+eKGTzgT0hS2z83o2rB8 aUBx4ZLMztgKkuEg2OUePaSEW1qaegOYupn9Sq/Kjvw0Ye6cgHIsXBYJKRAubc+UIPBnw00hZ OLyNLFt42hZJmsQ3UNPoQdwuL+VLDZ2YBZkcl3PsIIX76YSyASccGXaIYxsIyXKc5bWfMUcwp p3BRJxpdtnE3WvCShvoabL8v9O8UxgVYo1OGwXrS8hjzg21g5xFzO6StFfxfXV8GQhExY00Vh 33c47PVQh42iTofVDZ4HJTXbsWSD0YftbJ8Y1JPtPiejzn/pljxsmWEEH+UfZIxZQy00x/603 R9oDszKxFiTdqELnPDwfPLJvxDWB+LYsHvTpG85ZtC5mbqmjFJVmhGxD3AzvtXefGYt5NMV38 5wggV2A1/oWJgedIMa7KMZYw4TWuFoiEGKpALpGXatHsSvWPULB7Ur39z1bW5AyObRleMVi2d O+rZyQ7euXWpGLlpSvXe8ZXywzoQ3EDRfXKiBjILU7cSOowcXHquNHIQT9EjkC//YnHQVW2YG SYFcbeVh56JmGvvT+uTZagxV/IDfLZD6hadf04DOuzssPYJxDxAdrn2nof10ZJELSYFnDhYPw 00ECqg7PLyrLspFKhJWGyVidCbpTc4INHmapzAzsIepd6yfOoeiWkWzTCg6oIXvWiGW0= Subject: Re: [Caml-list] ocamldep, transitive dependencies, build systems, flambda --=-feYDjRmraufquCtn3J1W Content-Type: text/plain; charset="ISO-8859-15" Content-Transfer-Encoding: quoted-printable Hi Alain, my thinking is that the core of the issue is more a naming problem. Currently, cmi and cmx files are named by the module to which they refer. This works well as long as all modules are exposed to the user. What you need is a way of hiding the module while keeping a reference to the "module as such" - which got kind of anonymous after removing the cmi. Let's follow this idea for a while: Support for anonymous modules. Imagine there was a command ocamlhide A which reads a.cmi and outputs _a_900150983cd24fb0d6963f7d28e17f72.cmi, and the name consisted of the original module name and the module checksum. Same for the cmx file. In this form, the cmi file can no longer be directly used, but indirect references from other modules are still possible (the compiler would use the checksum for resolving the dependency). These files can then be installed into the pub directory. First of all, this solves an imminent problem of just not installing the cmi, namely a possible conflict with the name of a user's module (what will happen when another module A enters the scene later?). Second, all problems with different versions of the same module are gone (well, I'm not sure about the symbols, but let's consider this as a solvable detail). In particular, it is possible to have several versions of a module at the same time. If a particular version of a module is demanded for linking, it will be available and not conflict with any older artifact. Of course, ocamldep still has no chance to see such hidden deps by only looking at the sources. I don't have a really good idea that would easily integrate with build tools. If you manually manage the deps you'd have to do it this way: lib1/pub/B.cmx: lib1/src/B.cmx lib1/src/A.cmx lib1/pub/B.cmi: lib1/src/B.cmi lib1/src/A.cmi Because when A.cmi changes the reference in B.cmi to the anonymous module also changes. You cannot use the files with the checksums, because this is exactly the part that would be updated by the rebuild. This is just a thought. I'm sure whether a good solution is in reach with these ideas. Gerd Am Montag, den 04.07.2016, 18:49 +0200 schrieb Alain Frisch: > Dear all, >=20 > I'd like to know if people have good solutions to address the problem bel= ow. >=20 > Assume a large project, with multiple libraries spread over=20 > sub-directories, all managed by a single global build system that tracks= =20 > dependencies on a per-file basis (i.e. if a module depends on modules=20 > another library, it is not necessarily recompiled when only modules in=20 > that library are modified). >=20 > For instance, imagine a library in lib1/src with two modules A and B,=20 > B.ml and B.mli both depending on A. Thanks to ocamldep, the build=20 > system learns about the following dependencies (in make syntax): >=20 > lib1/src/B.cmx: lib1/src/A.cmi lib1/src/A.cmx > lib1/src/B.cmi: lib1/src/A.cmi >=20 > For various reasons, one might want to "install" some build artefefacts=20 > (.cmi, .cmx) in staging directories. One possible reason is to expose=20 > only a subset of a library internal modules to other libraries. For our= =20 > example, imagine that both A and B are part of the public API. So we=20 > create copy rules and record the associated dependencies to the build=20 > system: >=20 > lib1/pub/A.cmx: lib1/src/A.cmx > lib1/pub/A.cmi: lib1/src/A.cmi > lib1/pub/B.cmx: lib1/src/B.cmx > lib1/pub/B.cmi: lib1/src/B.cmi >=20 > Another library lib2/ is only allowed to see this public API, and so is=20 > compiled with "-I $(ROOT)/lib1/pub" (and not "-I $(ROOT)/lib1/src"). A=20 > module C in this library depends directly on B, and the build system=20 > thus infer the following dependencies: >=20 > lib2/src/C.cmx: lib1/pub/B.cmi lib1/pub/B.cmx >=20 > C has no reference to A in its source code so ocamldep has no way to=20 > know that it (transitively) depends on A. The trouble is that some=20 > dependencies are effectively unknown to the build system, which can lead= =20 > to broken builds. For instance, when lib1/pub/A.mli is modified and one= =20 > ask the build system to refresh lib2/src/C.cmx, the dependencies above=20 > will force only the following files to be refreshed in the process: >=20 > lib1/pub/B.cmi lib1/pub/B.cmx lib1/src/B.cmx lib1/src/B.cmi=20 > lib1/src/A.cmi lib1/src/A.cmx >=20 > So when C.ml is recompiled to produce C.cmx, it will see the old version= =20 > of lib1/pub/A.cmi. But even if ocamldep does not report any dependency=20 > from C to A, the type-checker might need to open A.cmi to expand e.g.=20 > type aliases, hence the broken build. I reported this problem in=20 > http://caml.inria.fr/mantis/view.php?id=3D5624 and the fix we have in=20 > place at LexiFi is to compile in a "strict" mode where the compiler=20 > prevents itself from opening a .cmi file which is not a direct=20 > dependency (i.e. the compiler runs ocamldep internally and restrict its=20 > view of the file system accordingly). This works fine and only forces=20 > us to explicitly add some dummy references. (Typically, if one needs=20 > A.cmi to compile C.ml, one would add a dummy reference to A somewhere in= =20 > C.ml. And ocamldep will thus report that C.cmx depends on A.cmi, which=20 > will fix the problem above.) >=20 > I'm wondering how other groups manage this kind of problem. >=20 > Moreover, flambda makes the problem actually quite a bit worse. Indeed,= =20 > B.cmx can now contain symbolic references to A.cmx, and when compiling=20 > C.cmx, the compiler will complain that it cannot find A.cmx (typically=20 > when a function in B is inlined in C and calls a function in A). This=20 > is warning 58. Simply disabling the warning does not work, since an old= =20 > version of A.cmx could remain in lib1/pub, leading to mismatched=20 > implementation digests and to unreliable parallel build. >=20 > One could apply the same trick as for .cmi files, i.e. prevent the=20 > compiler from opening A.cmx if the current unit does not depend=20 > (according to ocamldep) on A. But this is not so good as for=20 > interfaces, for two reasons: >=20 > - It's harder for the user to figure out that an explicit dependency=20 > must be forced, because this is not exposed in the published API (i.e.=20 > the module interfaces), but only in the implementation. Moreover, it=20 > depends on internals of the compiler whether A.cmx is actually needed to= =20 > compile C.cmx (e.g. in non-flambda mode, and perhaps in flambda mode=20 > with some settings, it is not needed). >=20 > - We still want to be able *not* to install A.cmi in lib1/pub if A is= =20 > not part of the public API of lib1. But this would prevent the code in=20 > C to force a dependency to A. >=20 >=20 > A different direction would be to register extra dependencies between=20 > "installed" files depending on the dependencies between source units.=20 > In the example above, one would register: >=20 >=20 > lib1/pub/B.cmx: lib1/pub/A.cmi lib1/pub/A.cmx > lib1/pub/B.cmi: lib1/pub/A.cmi lib1/src/B.cmi >=20 >=20 > The problem is that this creates interactions between the copy rules=20 > (which are just regular copy commands with the associated dependencies)=20 > and the normal build rules for OCaml units (with automatic discovery of=20 > dependencies with "ocamldep -modules"). In our case, our build system=20 > is omake and these two kinds of rules are completely separated (generic=20 > build rules and one or several "install" rules to expose different APIs=20 > to various parts of the projects). We don't see how to write our build=20 > rules in a modular way and keep the automatic discovery of dependencies. >=20 > The core of the problem, as I see it, is that ocamldep cannot return=20 > even an over-approximation of the actual dependencies of a given unit.=20 > It misses "implicit" dependencies related to either aliases in the type=20 > system or cross-module optimizations in cmx files (with flambda at=20 > least, the problem does not seem to exist at the implementation level=20 > for non-flambda mode). >=20 >=20 > So if any other group has faced the same problem and found a nice=20 > solution (with omake or another build system), I'd love to hear about it! >=20 >=20 > -- Alain >=20 --=20 ------------------------------------------------------------ Gerd Stolpmann, Darmstadt, Germany gerd@gerd-stolpmann.de My OCaml site: http://www.camlcity.org Contact details: http://www.camlcity.org/contact.html Company homepage: http://www.gerd-stolpmann.de ------------------------------------------------------------ --=-feYDjRmraufquCtn3J1W Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAABAgAGBQJXe7vjAAoJEAaM4b9ZLB5T+q0H/3OUzC4HNVSQ2JCTS7FaBubM utQtgAvOTPxSJoZv6xGpJ9TQdF0nzhQ83UezJkmAO6Ss7mx/pn34mrsTWxeb8nvr JIsr5x9G5f2VlK3D9mKoT1mHoJsyW4id0seUPGFMp35CDp95YyomJ32SB9deOnl3 dpyWvbORg7ULbc+i0/bEqvCiEi+TOu7C3AoPYbqCtYLjGMDlkCnSJJjzYzBOnbi6 f8HcGAqJQAdE1wAkAsVkNHbdN3E16+A7MGHrtOY02pG8I4rsCIYeDgs8JGoDY+Vj 5XN28pWHojny6hv5dX+urL+GPa+FgxaWuzmz3sR9agLV7QkKC235qoQnhykvJS0= =3TQ+ -----END PGP SIGNATURE----- --=-feYDjRmraufquCtn3J1W--