From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: caml-list@sympa.inria.fr Delivered-To: caml-list@sympa.inria.fr Received: from mail2-relais-roc.national.inria.fr (mail2-relais-roc.national.inria.fr [192.134.164.83]) by sympa.inria.fr (Postfix) with ESMTPS id 0E5A77FE36 for ; Mon, 11 Jul 2016 00:41:53 +0200 (CEST) IronPort-PHdr: 9a23:yg7SbBbMPEMGzdwrcpqcTKn/LSx+4OfEezUN459isYplN5qZpcS+bnLW6fgltlLVR4KTs6sC0LuO9fu8EjVfu96oizMrSNR0TRgLiMEbzUQLIfWuLgnFFsPsdDEwB89YVVVorDmROElRH9viNRWJ+iXhpQAbFhi3DwdpPOO9QteU1JXvkbvusMSMOU1hv3mUWftKNhK4rAHc5IE9oLBJDeIP8CbPuWZCYO9MxGlldhq5lhf44dqsrtY4q3wD89pozcNLUL37cqIkVvQYSW1+ayFmrPHs4DbOVwfH3WcbVn0TmxxOS1zE6h/9dpj8qCe/rfZ63DGfNMvwC7w5D2eM9aBuHSPvjW89NjI2/SmDlst1yqAduhOtqBE8yIfRZKmaM/1xeq7YdNIeA2FGW5ACBGR6HoqgYt5XXKI6NuFCoty4/gNWoA== Authentication-Results: mail2-smtp-roc.national.inria.fr; spf=None smtp.pra=tom.j.ridge@googlemail.com; spf=Pass smtp.mailfrom=tom.j.ridge@googlemail.com; spf=None smtp.helo=postmaster@mail-vk0-f54.google.com Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of tom.j.ridge@googlemail.com) identity=pra; client-ip=209.85.213.54; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="tom.j.ridge@googlemail.com"; x-sender="tom.j.ridge@googlemail.com"; x-conformance=sidf_compatible Received-SPF: Pass (mail2-smtp-roc.national.inria.fr: domain of tom.j.ridge@googlemail.com designates 209.85.213.54 as permitted sender) identity=mailfrom; client-ip=209.85.213.54; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="tom.j.ridge@googlemail.com"; x-sender="tom.j.ridge@googlemail.com"; x-conformance=sidf_compatible; x-record-type="v=spf1" Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of postmaster@mail-vk0-f54.google.com) identity=helo; client-ip=209.85.213.54; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="tom.j.ridge@googlemail.com"; x-sender="postmaster@mail-vk0-f54.google.com"; x-conformance=sidf_compatible X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A0AAAQDRzoJXhjbVVdFdhBR8BqYKjgOHHIcUBzwQAQEBAQEBAQERAQEBCAsLCSEvgjIEARIBghMBBAESER0BFCQBAwELAQUDAgsDNAICIhIBBQEcBhMJEgeHcwEDDwgOklCPQoExPjGLO4REBYUSChknDYQUAQEIAQEBAQEaAgYQimSHQoJaBYgckHyGDYYyghOPLI5QEh6BDzVVgVMcgU07MgGIbwEBAQ X-IPAS-Result: A0AAAQDRzoJXhjbVVdFdhBR8BqYKjgOHHIcUBzwQAQEBAQEBAQERAQEBCAsLCSEvgjIEARIBghMBBAESER0BFCQBAwELAQUDAgsDNAICIhIBBQEcBhMJEgeHcwEDDwgOklCPQoExPjGLO4REBYUSChknDYQUAQEIAQEBAQEaAgYQimSHQoJaBYgckHyGDYYyghOPLI5QEh6BDzVVgVMcgU07MgGIbwEBAQ X-IronPort-AV: E=Sophos;i="5.28,343,1464645600"; d="scan'208,217";a="226247902" Received: from mail-vk0-f54.google.com ([209.85.213.54]) by mail2-smtp-roc.national.inria.fr with ESMTP/TLS/AES128-GCM-SHA256; 11 Jul 2016 00:41:51 +0200 Received: by mail-vk0-f54.google.com with SMTP id b192so116982065vke.0 for ; Sun, 10 Jul 2016 15:41:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=20120113; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=KPs4JLr5F54Cw61o1JPKgv2+sOhsKk6C6eiT6C0TIVE=; b=J/JZqY3dYLWvFNHrLX0Y8/YRC+h8FYwlOslF4zgybBpG+TOCoA5hL8/KS+ifjvItxz ueZVOVP18gfjJL6zcaucDAhrdNxOvUQxFACYeWmA1cqF2G3NQSnSUnkL4r1eDuoIwV5S QHtvCxMApXvt/zZ+rgptaX5ylKU1LSzm4Qln4MkjHCDyhNDJyliPvoP7IGjtFPcffTOZ MCkPmqL2RgDEfc2Frc/KnHGntWur2Lh1BJyE6efnM5DqOr6pytzTrDwWIfVuK7F2+uBT P5C5dDiPltxuPJjwctRtaxxSt9BWgHw0XacrgZp58whKNNtGCY5rkGKlaoQwzS3blr20 Qu4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=KPs4JLr5F54Cw61o1JPKgv2+sOhsKk6C6eiT6C0TIVE=; b=D/BA3ZQgGzMteMRj1ZTb+ERj6CTaJdHB7irnNMrjuA4pheHheWQt/eH8O3iIy1NwzY fkkuIEOijde1qTmC5mNOCenZ/v4OUXZ+a0gX7uM/ZGQL4ODozvMyXns5k80jmGfXlHSN puHQVY098wUD2DDzy3gMSrJgcFA/FmWzz9fGIeCAZuhanmYE1WG5pNWf2WuhMS1BzMYd mHP4x+MaHIDFgFzS5oK3nC6XCWSjjqvlDLTS8w4OsqZsq4fLWAAO5fcxqBtaj3rDTaa8 XLAc9iBqh+/zFq+HjSPuEBX9aapkcM8pmuyjTfRaXGR1RfJ/MHDWx3B371FCfYzmuOJ8 gTig== X-Gm-Message-State: ALyK8tLKlPdoieZ5spG/UbRVLZy5WPNpIvdxaggyRBZUjcY7E16Gs3moPNteslE01ZFXJJviltACKdThG3nptw== X-Received: by 10.159.36.120 with SMTP id 111mr6288127uaq.106.1468190509831; Sun, 10 Jul 2016 15:41:49 -0700 (PDT) MIME-Version: 1.0 Sender: tom.j.ridge@googlemail.com Received: by 10.103.107.5 with HTTP; Sun, 10 Jul 2016 15:41:30 -0700 (PDT) In-Reply-To: <1468148606.25014.58.camel@e130.lan.sumadev.de> References: <1468148606.25014.58.camel@e130.lan.sumadev.de> From: Tom Ridge Date: Sun, 10 Jul 2016 23:41:30 +0100 X-Google-Sender-Auth: 0o-myKSZt9qZ_7mEVYxs4uGrriU Message-ID: To: Gerd Stolpmann Cc: Martin DeMello , "caml-list@inria.fr" Content-Type: multipart/alternative; boundary=001a11352274cd1d7205374fbdf2 X-Validation-by: tom.j.ridge+caml@googlemail.com Subject: Re: [Caml-list] why is building ocaml hard? --001a11352274cd1d7205374fbdf2 Content-Type: text/plain; charset=UTF-8 Gerd raises interesting technical points. A possibly-too-simple point of view which is not really technical at all, but usability: for a src/ directory, with (possibly nested) subdirectories (used for organization of files not naming of modules), and with no external dependencies, I think most users would like to write `ocamlfind ocamlc src/` and have it just work. If there are external dependencies, then `ocamlfind -pkgs x,y,z ocamlc src/` (or similar) should just work. Now, ocaml's build process is very flexible, so we should not expect everything to be so simple. But in the simple case, things should ideally be simple. On 10 July 2016 at 12:03, Gerd Stolpmann wrote: > Am Samstag, den 09.07.2016, 21:16 -0700 schrieb Martin DeMello: > > My consistent experience with OCaml has been that the build systems > > are fiddly and hard to work with, but I've never seen a discussion of > > why this is so (as opposed to problems with specific build tools). > > Supposing you were to start from scratch and develop a new build > > system in a bottom up manner, starting with a set of libraries and > > utilities and working your way up to a framework or dsl, what would > > the difficult steps be? > > Well, I don't think you can improve it that much (except... see below). > Since I've taken over omake and studied it carefully, I think I have > good insights into the problems. > > In short, the big difficulty of OCaml is the strict build topology. You > need to build a module before the caller of the module. Most build-tool > related failures come from that. Note that many other languages have > more relaxed build topologies, or work around the problem by doing > 2-pass builds (i.e. you first pre-compile and extract interfaces for the > whole project, and do the actual code generation in the second pass). > > Let's have a closer look why it is relatively error-prone to extract the > dependencies. The tool in question is ocamldep. It is fairly dumb in so > far it is only parsing the source code, and then looks at all > module-related constructs (open, include, module, etc.). Because it > never looks into already compiled interfaces and also proceeds file by > file, it may sometimes emit wrong dependency information. For example, > when there is > > open M1 > open M2 > > at the beginning of a file, ocamldep doesn't know whether M2 is another > top-level module, or whether it is a submodule of M1. ocamldep normally > errs on the side of generating too many dependencies, which is then > tried to be corrected by only accepting those deps corresponding to > existing files. In this example, this would mean that a dependency to M2 > is emitted when there is a file M2.ml. Note that this is wrong when M2 > is actually a submodule of M1 AND the file M2.ml exists. > > So how to fix this? In my opinion there are two solutions. You can > either have a more intelligent ocamldep (e.g. one that reads in > non-local cmi files and uses that information and also tries to > interpret all project ml files at once and not file by file - btw, did > anybody check whether there is an algorithm that precisely solves the > problem?). The other solution path is to mark toplevel modules in the > syntax of OCaml (e.g. you'd have to do "open ^M2" is M2 is a toplevel > module). > > Besides ocamldep, there are also other aspects that affect the > dependency analysis. E.g. with omake there is a distinction of > project-local and other dependencies, and you need to set the > OCAMLINCLUDES variable to add other directories to the local part of the > analysis, whereas the non-local deps are nowadays handled with > ocamlfind. First of all, this distinction is not really clear to every > user, and second, there are some difficulties in processing that when > both concepts overlap (e.g. you want to also get project-local > dependency expansion with ocamlfind). > > Note that recently the dependency analysis even became harder because of > flambda. Now cmx files play a bigger role. In particular a cmx file can > refer to another cmx file that isn't a direct dependency. In other > words, there is a second kind of dependency that is added by the code > generator. Current build tools cannot record these dependencies yet. > > Gerd > > -- > ------------------------------------------------------------ > Gerd Stolpmann, Darmstadt, Germany gerd@gerd-stolpmann.de > My OCaml site: http://www.camlcity.org > Contact details: http://www.camlcity.org/contact.html > Company homepage: http://www.gerd-stolpmann.de > ------------------------------------------------------------ > > --001a11352274cd1d7205374fbdf2 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Gerd raises interesting technical points.=C2=A0

A possibly-too-simple point of view which is not really technical at all= , but usability: for a src/ directory, with (possibly nested) subdirectorie= s (used for organization of files not naming of modules), and with no exter= nal dependencies, I think most users would like to write `ocamlfind ocamlc = src/` and have it just work. If there are external dependencies, then `ocam= lfind -pkgs x,y,z ocamlc src/` (or similar) should just work.

Now, ocaml's build process is very flexible, so we should not exp= ect everything to be so simple. But in the simple case, things should ideal= ly be simple.=C2=A0


On 10 July 2016 at 12:03, Gerd Sto= lpmann <info@gerd-stolpmann.de> wrote:
Am Samstag, den 09.07.2016, 21:16 -0700 = schrieb Martin DeMello:
> My consistent experience with OCaml has been that the build systems
> are fiddly and hard to work with, but I've never seen a discussion= of
> why this is so (as opposed to problems with specific build tools).
> Supposing you were to start from scratch and develop a new build
> system in a bottom up manner, starting with a set of libraries and
> utilities and working your way up to a framework or dsl, what would
> the difficult steps be?

Well, I don't think you can improve it that much (except... see = below).
Since I've taken over omake and studied it carefully, I think I have
good insights into the problems.

In short, the big difficulty of OCaml is the strict build topology. You
need to build a module before the caller of the module. Most build-tool
related failures come from that. Note that many other languages have
more relaxed build topologies, or work around the problem by doing
2-pass builds (i.e. you first pre-compile and extract interfaces for the
whole project, and do the actual code generation in the second pass).

Let's have a closer look why it is relatively error-prone to extract th= e
dependencies. The tool in question is ocamldep. It is fairly dumb in so
far it is only parsing the source code, and then looks at all
module-related constructs (open, include, module, etc.). Because it
never looks into already compiled interfaces and also proceeds file by
file, it may sometimes emit wrong dependency information. For example,
when there is

open M1
open M2

at the beginning of a file, ocamldep doesn't know whether M2 is another=
top-level module, or whether it is a submodule of M1. ocamldep normally
errs on the side of generating too many dependencies, which is then
tried to be corrected by only accepting those deps corresponding to
existing files. In this example, this would mean that a dependency to M2
is emitted when there is a file M2.ml. Note that this is wrong when M2
is actually a submodule of M1 AND the file M2.ml exists.

So how to fix this? In my opinion there are two solutions. You can
either have a more intelligent ocamldep (e.g. one that reads in
non-local cmi files and uses that information and also tries to
interpret all project ml files at once and not file by file - btw, did
anybody check whether there is an algorithm that precisely solves the
problem?). The other solution path is to mark toplevel modules in the
syntax of OCaml (e.g. you'd have to do "open ^M2" is M2 is a = toplevel
module).

Besides ocamldep, there are also other aspects that affect the
dependency analysis. E.g. with omake there is a distinction of
project-local and other dependencies, and you need to set the
OCAMLINCLUDES variable to add other directories to the local part of the
analysis, whereas the non-local deps are nowadays handled with
ocamlfind. First of all, this distinction is not really clear to every
user, and second, there are some difficulties in processing that when
both concepts overlap (e.g. you want to also get project-local
dependency expansion with ocamlfind).

Note that recently the dependency analysis even became harder because of
flambda. Now cmx files play a bigger role. In particular a cmx file can
refer to another cmx file that isn't a direct dependency.=C2=A0 In othe= r
words, there is a second kind of dependency that is added by the code
generator. Current build tools cannot record these dependencies yet.

Gerd

--
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany=C2=A0 =C2=A0 gerd@gerd-stolpmann.de
My OCaml site:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 http://www.camlcity.org<= br> Contact details:=C2=A0 =C2=A0 =C2=A0 =C2=A0 http://www.camlcity.= org/contact.html
Company homepage:=C2=A0 =C2=A0 =C2=A0 =C2=A0http://www.gerd-stolpmann.de=
------------------------------------------------------------


--001a11352274cd1d7205374fbdf2--