From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: caml-list@sympa.inria.fr Delivered-To: caml-list@sympa.inria.fr Received: from mail2-relais-roc.national.inria.fr (mail2-relais-roc.national.inria.fr [192.134.164.83]) by sympa.inria.fr (Postfix) with ESMTPS id 856347F75C for ; Tue, 23 Sep 2014 12:58:24 +0200 (CEST) Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of mshinwell@janestreet.com) identity=pra; client-ip=38.105.200.233; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="mshinwell@janestreet.com"; x-sender="mshinwell@janestreet.com"; x-conformance=sidf_compatible Received-SPF: Pass (mail2-smtp-roc.national.inria.fr: domain of mshinwell@janestreet.com designates 38.105.200.233 as permitted sender) identity=mailfrom; client-ip=38.105.200.233; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="mshinwell@janestreet.com"; x-sender="mshinwell@janestreet.com"; x-conformance=sidf_compatible; x-record-type="v=spf1" Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of postmaster@mxout4.mail.janestreet.com) identity=helo; client-ip=38.105.200.233; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="mshinwell@janestreet.com"; x-sender="postmaster@mxout4.mail.janestreet.com"; x-conformance=sidf_compatible X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AskBAH9RIVQmacjpnGdsb2JhbABghDgEgn3OGAGBCAgWAREBAQEBAQgUCT+EBAEBBBIRHQEBNwEPCwsNAgImAgIiEgEFARwGEyKIHAOiEm2KOHiFAgEFkTIBEQYKgSKOA1gHgniBU50ah0yMFBgpgx4bgVprgQaBRAEBAQ X-IPAS-Result: AskBAH9RIVQmacjpnGdsb2JhbABghDgEgn3OGAGBCAgWAREBAQEBAQgUCT+EBAEBBBIRHQEBNwEPCwsNAgImAgIiEgEFARwGEyKIHAOiEm2KOHiFAgEFkTIBEQYKgSKOA1gHgniBU50ah0yMFBgpgx4bgVprgQaBRAEBAQ X-IronPort-AV: E=Sophos;i="5.04,579,1406584800"; d="scan'208";a="97417783" Received: from mxout4.mail.janestreet.com ([38.105.200.233]) by mail2-smtp-roc.national.inria.fr with ESMTP/TLS/DHE-RSA-AES256-SHA; 23 Sep 2014 12:58:23 +0200 Received: from tot-oib-smtp1.delacy.com ([172.27.22.15] helo=tot-smtp) by mxout4.mail.janestreet.com with esmtps (UNKNOWN:DHE-RSA-AES256-GCM-SHA384:256) (Exim 4.82) (envelope-from ) id 1XWNnZ-0006GC-R4 for caml-list@inria.fr; Tue, 23 Sep 2014 06:58:21 -0400 Received: from [172.27.229.230] (helo=mxgoog1.mail.janestreet.com) by tot-smtp with esmtps (UNKNOWN:AES256-GCM-SHA384:256) (Exim 4.72) (envelope-from ) id 1XWNnZ-0001VU-Lp for caml-list@inria.fr; Tue, 23 Sep 2014 06:58:21 -0400 Received: from mail-pa0-f41.google.com ([209.85.220.41]) by mxgoog1.mail.janestreet.com with esmtps (TLSv1:RC4-SHA:128) (Exim 4.72) (envelope-from ) id 1XWNnZ-0007Tq-Fk for caml-list@inria.fr; Tue, 23 Sep 2014 06:58:21 -0400 Received: by mail-pa0-f41.google.com with SMTP id ey11so7101004pad.28 for ; Tue, 23 Sep 2014 03:58:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=janestreet.com; s=google; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=MU4RXogVmNbKnj2r/gmIoj3DYpsysxZtols3kmfG1i0=; b=XeqhD5+ZNs+NQiXYzTzRESdeoS1MNpTyljZ4QgEkJN98GEsCBCIE0p5k8j1vR216gM f52F5+bZUyATFMU/wMf9AHqJW7GWAgYNAj4uhK+jZObPt/Ot51HDfbSQeJ6fkK61IqRq NN8UJUXaspbnNeRIepoT/db3ORyvD0pcrEQoc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=MU4RXogVmNbKnj2r/gmIoj3DYpsysxZtols3kmfG1i0=; b=gM0fbtkeSqLIU+5AfEKEOp6eJtjmvY9ng+RcdoUp+2RvLPZDKZqBAQuL5NjV8JfCFO IOxZT6aqeXor1AA6wgM4CoLiaToSVb54HnYSKWgQu02Pnub9Pgq7Tpv4x1ec+31xHqYL QRWrA+59z0XguFRafIS0t7G0so9/HM+CEJIu3aD6UfVXVMyot6bjQfcOI2Ag7ddm0Mko oAAMTNt3FwqF216Pnyb8H/BhaznTf8GXFJBM879KeUV3b4470mVVv8ct50A8W3fGiaqm 5pUwbVGvkCb3cBgp2jj3Vp9NdMUEs005BlcWPZxuS++Rgazp92WI1VcmvFqAN1WlO2iy SoWg== X-Gm-Message-State: ALoCoQmGBOKPOrwd6OEdnqWNc0VygJCwO+Uu5weAVNGYwtWP750cfIljhL5YaO8LujufDeGhO1JqCQskmep5pILUU/3WrEPtftcIWN7hYy60ptPBPTGEJYi6hBo3LEpk3sHj8IHd52zD X-Received: by 10.68.57.198 with SMTP id k6mr32498863pbq.60.1411469900777; Tue, 23 Sep 2014 03:58:20 -0700 (PDT) MIME-Version: 1.0 X-Received: by 10.68.57.198 with SMTP id k6mr32498838pbq.60.1411469900651; Tue, 23 Sep 2014 03:58:20 -0700 (PDT) Received: by 10.70.128.177 with HTTP; Tue, 23 Sep 2014 03:58:20 -0700 (PDT) In-Reply-To: <54214E1C.4010507@frisch.fr> References: <541C2037.5030303@frisch.fr> <1411133763.2930.28.camel@zotac> <541C36FF.3010603@frisch.fr> <54214E1C.4010507@frisch.fr> Date: Tue, 23 Sep 2014 11:58:20 +0100 Message-ID: From: Mark Shinwell To: Alain Frisch Cc: Yaron Minsky , Gerd Stolpmann , Bob Zhang , Caml List Content-Type: text/plain; charset=UTF-8 X-Validation-by: mshinwell@janestreet.com Subject: Re: [Caml-list] improve omake [was One build system to rule them all] On 23 September 2014 11:40, Alain Frisch wrote: > On 09/19/2014 05:18 PM, Yaron Minsky wrote: >> >> - The build itself has limited parallelism because of various >> bottlenecks inside omake, like the fact that it computes its md5sums >> in a single process. > > I think that omake doesn't usually recompute the md5 when the file hasn't > changed (probably based on the mtime, maybe inode as well), so I'm surprised > by the claim. Did you actually confirm that the computation of md5 was one > of the bottlenecks or is it just an hypothesis? Without wishing to prolong a discussion about improving omake... I think the details are hazy now, but as far as we recall: your statement about avoiding recomputation is correct, but a lot of md5sum work happened subsequent to each compilation action in serial, within the omake process. By no means was this the only problem; another, for example, was a vast amount of time spent constructing rules for an entire tree even if only a small portion was being demanded to be built. (Of course, the worst problem with omake was probably the DSL, which we found to be wholly unsuitable for large-scale reliable rule engineering. Programmability really is key.) Mark