From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: caml-list@sympa.inria.fr Delivered-To: caml-list@sympa.inria.fr Received: from mail3-relais-sop.national.inria.fr (mail3-relais-sop.national.inria.fr [192.134.164.104]) by sympa.inria.fr (Postfix) with ESMTPS id E319F7FACD for ; Fri, 19 Sep 2014 19:49:27 +0200 (CEST) Received-SPF: None (mail3-smtp-sop.national.inria.fr: no sender authenticity information available from domain of yminsky@janestreet.com) identity=pra; client-ip=38.105.200.115; receiver=mail3-smtp-sop.national.inria.fr; envelope-from="yminsky@janestreet.com"; x-sender="yminsky@janestreet.com"; x-conformance=sidf_compatible Received-SPF: None (mail3-smtp-sop.national.inria.fr: no sender authenticity information available from domain of yminsky@janestreet.com) identity=mailfrom; client-ip=38.105.200.115; receiver=mail3-smtp-sop.national.inria.fr; envelope-from="yminsky@janestreet.com"; x-sender="yminsky@janestreet.com"; x-conformance=sidf_compatible Received-SPF: None (mail3-smtp-sop.national.inria.fr: no sender authenticity information available from domain of postmaster@mxout2.mail.janestreet.com) identity=helo; client-ip=38.105.200.115; receiver=mail3-smtp-sop.national.inria.fr; envelope-from="yminsky@janestreet.com"; x-sender="postmaster@mxout2.mail.janestreet.com"; x-conformance=sidf_compatible X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AnACANFrHFQmachznGdsb2JhbABWCoNgVwSCfbU2kTcKh00BewgWAREBAQEBAQYWCT6EAwEBAQMBEhEdAQETGQYFAQQLCwsDCgICCR0CAiISAQUBChIGEwkJEIgIAwkIAwqgc22KOHiFAgEFiWcDhzgBEQYKgSKIboULBDImB4I3QRKBQYUSBYEOj2yEeoINgWGFa4wQGCmDGxyBdVABgQWBRAEBAQ X-IPAS-Result: AnACANFrHFQmachznGdsb2JhbABWCoNgVwSCfbU2kTcKh00BewgWAREBAQEBAQYWCT6EAwEBAQMBEhEdAQETGQYFAQQLCwsDCgICCR0CAiISAQUBChIGEwkJEIgIAwkIAwqgc22KOHiFAgEFiWcDhzgBEQYKgSKIboULBDImB4I3QRKBQYUSBYEOj2yEeoINgWGFa4wQGCmDGxyBdVABgQWBRAEBAQ X-IronPort-AV: E=Sophos;i="5.04,556,1406584800"; d="scan'208";a="80098835" Received: from mxout2.mail.janestreet.com ([38.105.200.115]) by mail3-smtp-sop.national.inria.fr with ESMTP/TLS/DHE-RSA-AES256-SHA; 19 Sep 2014 19:49:02 +0200 Received: from tot-smtp.delacy.com ([172.27.22.15] helo=tot-smtp) by mxout2.mail.janestreet.com with esmtps (TLSv1.2:DHE-RSA-AES256-GCM-SHA384:256) (Exim 4.82) (envelope-from ) id 1XV2Im-0008Ra-AM for caml-list@inria.fr; Fri, 19 Sep 2014 13:49:00 -0400 Received: from [172.27.229.230] (helo=mxgoog1.mail.janestreet.com) by tot-smtp with esmtps (UNKNOWN:AES256-GCM-SHA384:256) (Exim 4.72) (envelope-from ) id 1XV2Im-0000qR-5K for caml-list@inria.fr; Fri, 19 Sep 2014 13:49:00 -0400 Received: from mail-lb0-f174.google.com ([209.85.217.174]) by mxgoog1.mail.janestreet.com with esmtps (TLSv1:RC4-SHA:128) (Exim 4.72) (envelope-from ) id 1XV2Il-0007an-Qd for caml-list@inria.fr; Fri, 19 Sep 2014 13:49:00 -0400 Received: by mail-lb0-f174.google.com with SMTP id l4so3664046lbv.33 for ; Fri, 19 Sep 2014 10:48:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=janestreet.com; s=google; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=y3/sRPbEoWTvBdliuVmlcTHuppxx+fUtm21+tVosJtc=; b=Yehf++FSRxu1KdHK9FX+saWEcJp+GTnf0AytUQK83AGD3VNnAHdlEjUk+0Spt4pBT9 GsqlAdKcceMCIvb4UXdk4b/IaR3KqaIiPc7V+GPxJqujGoxBmvLxzW38cM//eISf0DhH BgNovIRacK92MItIQ9N2pVfqCYJiqdhWA8mj8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=y3/sRPbEoWTvBdliuVmlcTHuppxx+fUtm21+tVosJtc=; b=d8IMMnUt3rc26nNLwbqC99+2BAE83AhGp2wJRU2k4giw/f4SKf+ez24UW/M4pdzSis lAPkxesJuWjSxuqUEuCfVvpBaOMiypyEBuVnhVPQddtv4IODRJAt3RrRGjZv1rMfH/CR ELtKwBTjSkUNtsw+WJtLX9krTFZOqSA1S3bAYCzaNC7puUK8sVhX4EQF3XG5HP41NJKG yC1zSOG4pg6R0hHtlowRFLG4ZnFZMq5qRsmNcboji8gJgnvtwaN0jjz1RMeKeWE9GcWC ATqR2qXUIRUjbOE957agm08hPi+/rXq28E4f5jJE1ugj5iQEhCnmitPw873jsJzsId5J TtEA== X-Received: by 10.112.54.135 with SMTP id j7mr7696702lbp.51.1411148938651; Fri, 19 Sep 2014 10:48:58 -0700 (PDT) X-Gm-Message-State: ALoCoQkSD6+i4sM0PZXueb1MMuL3LKxeRw/Ho09kj0k0obK0tmk3MRRUSrnyVa/KQXFmyOp5wzMYVsWZyLrXBXlTGRBMc0re73GBIHNSuDEwJSij6GXX4YcH//qkxC72+Z0K0i6Sm44p MIME-Version: 1.0 X-Received: by 10.112.54.135 with SMTP id j7mr7696687lbp.51.1411148938475; Fri, 19 Sep 2014 10:48:58 -0700 (PDT) Received: by 10.112.56.136 with HTTP; Fri, 19 Sep 2014 10:48:58 -0700 (PDT) In-Reply-To: <1411147089.2930.42.camel@zotac> References: <541C2037.5030303@frisch.fr> <1411133763.2930.28.camel@zotac> <541C36FF.3010603@frisch.fr> <1411147089.2930.42.camel@zotac> Date: Fri, 19 Sep 2014 13:48:58 -0400 Message-ID: From: Yaron Minsky To: Gerd Stolpmann Cc: Alain Frisch , Bob Zhang , Caml List Content-Type: text/plain; charset=UTF-8 Subject: Re: [Caml-list] improve omake [was One build system to rule them all] On Fri, Sep 19, 2014 at 1:18 PM, Gerd Stolpmann wrote: > Am Freitag, den 19.09.2014, 11:18 -0400 schrieb Yaron Minsky: >> We had a fair number of problems with omake >> >> - We've run into lots of performance problems on large builds (2-3 >> million lines, many thousands of targets) was that omake took a very >> long time (a few minutes) to be restarted. > > Well, never got into that dimensions. The largest builds had only around > 250Kloc, and omake worked well for that size. I don't know much about > the internals of omake, but I can imagine that certain things are > recomputed too often, and then you run into performance problems at a > certain size. But why can't this be nailed down? > >> - The build itself has limited parallelism because of various >> bottlenecks inside omake, like the fact that it computes its md5sums >> in a single process. > > Hmm, is that evident? Maybe switching to a faster hash algorithm could > solve this? (E.g. just use RC4 and XOR up the output stream; should be > ultimately fast.) I guess. It seems better to instead get rid of the single-threaded bottleneck. >> - The default rules didn't do a good job of clearing out stale build >> artifacts (important for getting reliable incremental builds), and >> we had to put quite a lot of painful engineering in place to make >> that work. We needed to do similar work in Jenga to make the same >> thing happen, but it was a lot more fun writing that code in OCaml! > > If you just stick to the built-in macros, incremental builds are very > reliable. If you start writing your own rules, there is some chance that > you overlook dependencies. But that's independent of which build system > you use. > >> I am not convinced that putting more complicated calculations into >> programs will work well. I know of no system that does a good job >> allowing you to deal with complex dependencies that are discovered >> over the course of a build (which you get with staged programming) >> that take this approach. It seems that a more expressive, monadic >> structure fits the bill, and hammering that into the round peg of >> shell invocations won't work well, I suspect. > > After all, I think that it would be naive to think that one size fits > all. When you have a large build like yours you probably have quite > special requirements, and are willing to do without comfort features > like a macro language. But there's no reason to settle for less. You can have a powerful core and a nice DSL for configuring your builds! And we certainly don't do without the comfort of easy to specify builds. Quite the opposite, we have a very simple, very easy-to-use way of expressing our builds, that doesn't require dipping in to OCaml. It's just built on top of a set of core rules that are in OCaml. And I don't think our requirements are particularly special. We build more code, at higher levels of parallelism, than any other OCaml user, but that's most of it. We build many of the same packages that everyone else does, and I think most of the benefits of a system like Jenga could be appreciated by other users who operate at somewhat smaller scales. y > So: no one build system to rule them all. Nevertheless, I'm very much > for improving omake. > > Gerd > > >> y >> >> On Fri, Sep 19, 2014 at 10:00 AM, Alain Frisch wrote: >> > On 09/19/2014 03:36 PM, Gerd Stolpmann wrote: >> >> >> >> Well, I run frequently into the difficulty that I need some special >> >> omake function that would be trivial to develop in OCaml (e.g. >> >> associating some data with some other data, filtering things, doing >> >> string transformations), but for writing it in the omake language I need >> >> some time for developing and testing. I have a quite simple idea to >> >> improve this: Besides OMakefile there could also be an OMakefile.ml, and >> >> you can define any helper functions there, and they would be >> >> automatically callable from the OMakefile. I think this is not really >> >> complicated to do - you'd need to build a custom omake executable >> >> whenever OMakefile.ml changes, and need to scan the OMakefile.ml >> >> interface for function signatures that match the form that is callable, >> >> and you need to generate some glue code. (Later this idea could be >> >> extended by allowing OCaml code to emit new rules, as described in an >> >> earlier post.) >> > >> > >> > I can see some cases where it would indeed be more comfortable to implement >> > build-system calculations in OCaml. But couldn't most of these cases be >> > implemented as external programs called by omake functions and implemented >> > e.g. in OCaml? This forces to pass explicitly all the data required by the >> > calculation to those external programs, but how often is this a problem? >> > With some convention, the external program could even return description of >> > new dependencies, to be interpreted by some omake code and injected into the >> > actual graph. AFAICT, all this is already possible. >> > >> > >> >> I see what you mean. In a recent project I had to define all variables >> >> with library names, findlib names, intra-project library dependencies >> >> etc. in the global OMakefile, because they are needed in potentially all >> >> sub-OMakefiles. That's of course not where these things are best >> >> naturally defined. >> > >> > >> > A variant is to have e.g. a OPreOMakefile file in each directory and arrange >> > so that the toplevel OMakefile includes all of them (with a proper ordering) >> > without processing the rest of the project. This way, you only need to >> > "lift" the full list of directories, and actual data definitions can be put >> > where they belong. >> > >> >> Maybe we should allow to switch to global context anywhere? I think this >> >> is solvable. >> > >> > >> > I'm not sure this would easily fit in the current functional semantics. >> > >> >> Could be something simple, like matching the wildcard rules against the >> >> real files. >> > >> > >> > Reading the directory content should be quite cheap, and then it is just >> > string processing, which should be even cheaper (if done properly). It >> > would be great to identify such hot spots; maybe some very local tweaks to >> > algorithmics or data structures could improve performance a lot. >> > >> > >> > >> > Alain >> > >> > >> > -- >> > Caml-list mailing list. Subscription management and archives: >> > https://sympa.inria.fr/sympa/arc/caml-list >> > Beginner's list: http://groups.yahoo.com/group/ocaml_beginners >> > Bug reports: http://caml.inria.fr/bin/caml-bugs >> > > -- > ------------------------------------------------------------ > Gerd Stolpmann, Darmstadt, Germany gerd@gerd-stolpmann.de > My OCaml site: http://www.camlcity.org > Contact details: http://www.camlcity.org/contact.html > Company homepage: http://www.gerd-stolpmann.de > ------------------------------------------------------------