From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: caml-list@sympa.inria.fr Delivered-To: caml-list@sympa.inria.fr Received: from mail2-relais-roc.national.inria.fr (mail2-relais-roc.national.inria.fr [192.134.164.83]) by sympa.inria.fr (Postfix) with ESMTPS id C91D17FCCD for ; Tue, 12 Jul 2016 10:19:01 +0200 (CEST) IronPort-PHdr: 9a23:SPnysRNN+KXlgAIix/Ml6mtUPXoX/o7sNwtQ0KIMzox0KP/zrarrMEGX3/hxlliBBdydsKMczbGI+Pm6ACQp2tWoiDg6aptCVhsI2409vjcLJ4q7M3D9N+PgdCcgHc5PBxdP9nC/NlVJSo6lPwWB6kO74TNaIBjjLw09fr2zQd+KyZ/pnLnootX6WEZhvHKFe7R8LRG7/036l/I9ps9cEJs30QbDuXBSeu5blitCLFOXmAvgtI/rpMYwu3cYh/V0/MdFVeD+fr8kZb1eFjUvdW4vt+PxshyWYgyU+XoaGnsRlFJiGQXJ4Qv+WYi55iT9rfV83myEeMfeUr0+HzivufQ4ACT0gTsKYmZquFrcjdZ92fpW Authentication-Results: mail2-smtp-roc.national.inria.fr; spf=None smtp.pra=goswin-v-b@web.de; spf=Pass smtp.mailfrom=goswin-v-b@web.de; spf=None smtp.helo=postmaster@mout.web.de Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of goswin-v-b@web.de) identity=pra; client-ip=212.227.15.3; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="goswin-v-b@web.de"; x-sender="goswin-v-b@web.de"; x-conformance=sidf_compatible Received-SPF: Pass (mail2-smtp-roc.national.inria.fr: domain of goswin-v-b@web.de designates 212.227.15.3 as permitted sender) identity=mailfrom; client-ip=212.227.15.3; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="goswin-v-b@web.de"; x-sender="goswin-v-b@web.de"; x-conformance=sidf_compatible; x-record-type="v=spf1" Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of postmaster@mout.web.de) identity=helo; client-ip=212.227.15.3; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="goswin-v-b@web.de"; x-sender="postmaster@mout.web.de"; x-conformance=sidf_compatible X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A0DgAACepoRXhwMP49RchBR8sHCIGIF6GoV+AoEwOhIBAQEBAQEBAREBAQEIDQkJIS+CMhWCFQEBBAE6RAsLGAklDwUoT4d6ARIIBLoTH4Q2AQEIAgEkineEK4NCgi8FjgeLE4YPgnqFQYNuhVsKhWSQEiQBgjgRC4FObIdhKoEaAQEB X-IPAS-Result: A0DgAACepoRXhwMP49RchBR8sHCIGIF6GoV+AoEwOhIBAQEBAQEBAREBAQEIDQkJIS+CMhWCFQEBBAE6RAsLGAklDwUoT4d6ARIIBLoTH4Q2AQEIAgEkineEK4NCgi8FjgeLE4YPgnqFQYNuhVsKhWSQEiQBgjgRC4FObIdhKoEaAQEB X-IronPort-AV: E=Sophos;i="5.28,350,1464645600"; d="scan'208";a="226428166" Received: from mout.web.de ([212.227.15.3]) by mail2-smtp-roc.national.inria.fr with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 12 Jul 2016 10:19:01 +0200 Received: from frosties.localnet ([134.3.242.84]) by smtp.web.de (mrweb002) with ESMTPSA (Nemesis) id 0LdVha-1awrTi1dP0-00ihdr for ; Tue, 12 Jul 2016 10:19:00 +0200 Received: from mrvn by frosties.localnet with local (Exim 4.87) (envelope-from ) id 1bMsuB-000101-Gk for caml-list@inria.fr; Tue, 12 Jul 2016 10:18:59 +0200 Date: Tue, 12 Jul 2016 10:18:59 +0200 From: Goswin von Brederlow To: caml-list@inria.fr Message-ID: <20160712081859.GA919@frosties> References: <1468148606.25014.58.camel@e130.lan.sumadev.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1468148606.25014.58.camel@e130.lan.sumadev.de> User-Agent: Mutt/1.5.23 (2014-03-12) X-Provags-ID: V03:K0:BkXYSBK8QEO50eEhaGi16Y6DQ+w5o5i4JcOuZaqicDPeLDYAMvJ uRbmg1VXsfowGlaahSBwx1mXvv3biJLkjKX5pTwYPUeUz0zxTmQd8z6ZB0arha0FTSUtXgm WJQnnfEphPLps00l8eI9lXmxJweQH3n6m3al1uA5w9MqTR4yM3Ou+MrgL1GzXG6k83Y5FT/ iInmImIgcyBlFCnF2bVLw== X-UI-Out-Filterresults: notjunk:1;V01:K0:iAT2HIQAHZ4=:NtyNicdoM9HqjhxTt+gFkX kAyD6d8ADhoy/bHSSm6r28P258rAPGCpLfCEvGy2nukyRHpdVz+x0ICno2pY9VPWKOURGBvuV AcStGlROFq98gvD/IlpF6FjVln/j7Hlh4hxDvsO7jlW+3VYgwPpznVYrBhN9s01lioR55q9sm LkEajIVK91kItJvu2foSaP2/4bsjeaEG8xWiXlFdmWQLed8UX2eBnwS9qBVO+j1/G/pa5EDZA hw/sVub1iy/OeTtNNG0uvm3AuAQS+JstNl/me94Wg/t4KFlefjxw3B64jpkeUn4k4O5Dw1mTJ xhMpURC+xH74dA0AcHL3il9P8vCZxgNz6hudqxyenH4IWPjVRD0U1rC4yHxKHFUaVnJHcf5jB D2doRLQOL3C29xjBYbvIYl4gfQUMiz2oAw5XiGbQqixR4QzzG7ZxUPvbdApf7P7ugBmic4j1O 9cRW1nTzHbnwNTw1xwg6zT1vjp/O8jBiyoFfSjzsOtMvnwj9+QRMkvy0DvOfJSTLh9RZRRonE W7RhgW5xjiCqRtIts/9qsze1oAA+ZK7zmXbJZM95Cf/KW236/fcLG3WxHj/vujLmlDwvEr1Gh PdSxsP9vGZ11rlxwGdz1RG85A4tq7XHa9zxL3PpB4n375q3ACLufCw6Azrk35eYTcZUVoDSCP r9IKhpcPEKgmud0uwg7mGZKk6ycLXeQQi6acS6Xisbu5uxoJF07bAHz/qkA5kINf9wDap0TKS GUHUAUKFqE1lcwOY3nfebp3yPnxeDWBMnIPLKw7aR6lDvNKJfisF5bxju78= Subject: Re: [Caml-list] why is building ocaml hard? On Sun, Jul 10, 2016 at 01:03:26PM +0200, Gerd Stolpmann wrote: > Am Samstag, den 09.07.2016, 21:16 -0700 schrieb Martin DeMello: > > My consistent experience with OCaml has been that the build systems > > are fiddly and hard to work with, but I've never seen a discussion of > > why this is so (as opposed to problems with specific build tools). > > Supposing you were to start from scratch and develop a new build > > system in a bottom up manner, starting with a set of libraries and > > utilities and working your way up to a framework or dsl, what would > > the difficult steps be? > > Well, I don't think you can improve it that much (except... see below). > Since I've taken over omake and studied it carefully, I think I have > good insights into the problems. I'm using oasis with myocamlbuild and one thing I find a total pain: mssing files (typos in file name). The error reporting when something goes wrong in myocamlbuild is a total nightmare as it reports on the most unlikely build chain it can find to get the module you requested. > In short, the big difficulty of OCaml is the strict build topology. You > need to build a module before the caller of the module. Most build-tool > related failures come from that. Note that many other languages have > more relaxed build topologies, or work around the problem by doing > 2-pass builds (i.e. you first pre-compile and extract interfaces for the > whole project, and do the actual code generation in the second pass). > > Let's have a closer look why it is relatively error-prone to extract the > dependencies. The tool in question is ocamldep. It is fairly dumb in so > far it is only parsing the source code, and then looks at all > module-related constructs (open, include, module, etc.). Because it > never looks into already compiled interfaces and also proceeds file by > file, it may sometimes emit wrong dependency information. For example, > when there is > > open M1 > open M2 > > at the beginning of a file, ocamldep doesn't know whether M2 is another > top-level module, or whether it is a submodule of M1. ocamldep normally > errs on the side of generating too many dependencies, which is then > tried to be corrected by only accepting those deps corresponding to > existing files. In this example, this would mean that a dependency to M2 > is emitted when there is a file M2.ml. Note that this is wrong when M2 > is actually a submodule of M1 AND the file M2.ml exists. > > So how to fix this? In my opinion there are two solutions. You can > either have a more intelligent ocamldep (e.g. one that reads in > non-local cmi files and uses that information and also tries to > interpret all project ml files at once and not file by file - btw, did > anybody check whether there is an algorithm that precisely solves the > problem?). Which basically means compiling all the sources recursively and throwing away the AST. The algorithm is simple: 1) open file 2) parse code till you hit an unknown module 3) add dependency on module 4) check for cmi file for module and get list of submodules or check for mli file for module and gosub 1 or check for ml file for module and gosub 1 or assume module has no submodules 5) add module and submodules to known modules 6) goto 2 Note that this will still fail when you have open M1 open M2 and M1 is generated and contains M2. You have to run ocamldep again after M1 is generated. > The other solution path is to mark toplevel modules in the > syntax of OCaml (e.g. you'd have to do "open ^M2" is M2 is a toplevel > module). That seems rather stupid. As in why do I have to tell it that. Ocamlc already figures that out just fine. I don't like doing the computers job. Here is a 3rd option: 3a) fix ocamlc to build recursively. When it finds an "open M1" and M1 is not yet build then build M1 first. This could be done in 3 ways: - Call itself to compile M1. Bad for parallel building and fails is M1 is generated (e.g. from mly file). - Retrun an error to the build system to build M1 first and retry. It's not easy to dynamically add depends in most build systems. For most this would probably mean restarting itself till there is no more dependency error. - A mixture of the two: ocamlc asks the build system to provide M1 when it needs it. The 3rd option seems to be the sanest although figuring out a good general interface might be tricky. E.g. for GNU make you have to pass through the filedescriptor for the job server, some ENV vars and temporarily returns its build token when calling make again. 3b) output dependencies as a side effect of buidling This seems not needed since 3a ensures everything is build in order. But knowing the dependencies makes it easier the second time around and the build system can even watch for changed files and auto rebuild. Building files in order also prevents many instances of ocamlc being in memory at the same time. Building from scartch (no depends yet) might fail because memory is exhausted from all the ocamlc being started. Note: ocamlc is just an example. Same goes for ocamlopt, ocamlyacc, ocamllex, ... I believe an interactive design, where the build system and the compiler interact to figure out the dependencies together is the only way to cover all cases. The compiler knows about modules and submodules and all the ways the source can create a dependency on another module and the build system knows about generated files. Both are needed to figure out the complete dependencies. > Besides ocamldep, there are also other aspects that affect the > dependency analysis. E.g. with omake there is a distinction of > project-local and other dependencies, and you need to set the > OCAMLINCLUDES variable to add other directories to the local part of the > analysis, whereas the non-local deps are nowadays handled with > ocamlfind. First of all, this distinction is not really clear to every > user, and second, there are some difficulties in processing that when > both concepts overlap (e.g. you want to also get project-local > dependency expansion with ocamlfind). > > Note that recently the dependency analysis even became harder because of > flambda. Now cmx files play a bigger role. In particular a cmx file can > refer to another cmx file that isn't a direct dependency. In other > words, there is a second kind of dependency that is added by the code > generator. Current build tools cannot record these dependencies yet. > > Gerd How does that change anything? Afaik if A depends on B and B depends on C and C changes then B always needs to be rebuild. Therefore there can never be a change in C.cmx unless B.cmx also changes. So for A looking at B.cmx is enough. Right? MfG Goswin