From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: caml-list@sympa.inria.fr Delivered-To: caml-list@sympa.inria.fr Received: from mail2-relais-roc.national.inria.fr (mail2-relais-roc.national.inria.fr [192.134.164.83]) by sympa.inria.fr (Postfix) with ESMTPS id 2FB697FACF for ; Fri, 19 Sep 2014 17:18:30 +0200 (CEST) Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of yminsky@janestreet.com) identity=pra; client-ip=38.105.200.112; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="yminsky@janestreet.com"; x-sender="yminsky@janestreet.com"; x-conformance=sidf_compatible Received-SPF: Pass (mail2-smtp-roc.national.inria.fr: domain of yminsky@janestreet.com designates 38.105.200.112 as permitted sender) identity=mailfrom; client-ip=38.105.200.112; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="yminsky@janestreet.com"; x-sender="yminsky@janestreet.com"; x-conformance=sidf_compatible; x-record-type="v=spf1" Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of postmaster@mxout1.mail.janestreet.com) identity=helo; client-ip=38.105.200.112; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="yminsky@janestreet.com"; x-sender="postmaster@mxout1.mail.janestreet.com"; x-conformance=sidf_compatible X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Am8CAPVIHFQmachwnGdsb2JhbABWCoNgVwSCfbU1kUGHTQF6CBYBEQEBAQEBBhYJPoQEAQEEEhEdAQEsBgUBDwsLDQICCR0CAiISAQUBChIGExIQiAgDEQMKoGRtijh4hQIBBYltA4c4AREGCoEiiG6FCwRYB4J4gVOFEgWQeocHgWGFa4wQGCmFLFCBBoFEAQEB X-IPAS-Result: Am8CAPVIHFQmachwnGdsb2JhbABWCoNgVwSCfbU1kUGHTQF6CBYBEQEBAQEBBhYJPoQEAQEEEhEdAQEsBgUBDwsLDQICCR0CAiISAQUBChIGExIQiAgDEQMKoGRtijh4hQIBBYltA4c4AREGCoEiiG6FCwRYB4J4gVOFEgWQeocHgWGFa4wQGCmFLFCBBoFEAQEB X-IronPort-AV: E=Sophos;i="5.04,555,1406584800"; d="scan'208";a="96674692" Received: from mxout1.mail.janestreet.com ([38.105.200.112]) by mail2-smtp-roc.national.inria.fr with ESMTP/TLS/DHE-RSA-AES256-SHA; 19 Sep 2014 17:18:18 +0200 Received: from tot-smtp.delacy.com ([172.27.22.15] helo=tot-smtp) by mxout1.mail.janestreet.com with esmtps (TLSv1.2:DHE-RSA-AES256-GCM-SHA384:256) (Exim 4.82) (envelope-from ) id 1XUzwu-0004ok-OF for caml-list@inria.fr; Fri, 19 Sep 2014 11:18:16 -0400 Received: from [172.27.229.230] (helo=mxgoog1.mail.janestreet.com) by tot-smtp with esmtps (UNKNOWN:AES256-GCM-SHA384:256) (Exim 4.72) (envelope-from ) id 1XUzwu-0002e2-JK for caml-list@inria.fr; Fri, 19 Sep 2014 11:18:16 -0400 Received: from mail-la0-f43.google.com ([209.85.215.43]) by mxgoog1.mail.janestreet.com with esmtps (TLSv1:RC4-SHA:128) (Exim 4.72) (envelope-from ) id 1XUzwu-0008In-C9 for caml-list@inria.fr; Fri, 19 Sep 2014 11:18:16 -0400 Received: by mail-la0-f43.google.com with SMTP id gi9so3371229lab.16 for ; Fri, 19 Sep 2014 08:18:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=janestreet.com; s=google; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=5Ct8l5DqLVSSbZzwzD4LMe+5b0J5CDfqz/qfWrIN1TE=; b=B0RJRBT+8VF/ZkuSPpWl6lhqJBBjKuUxV2SRhaS3JLgjZs6c02eoaRmfmsCUXZmSc3 oZvpYX5+dkU7I7FOSZDmVlZ5hFu3GCQ/ikgUgB9z+JU52GqCzHpIYwGw6n63Yh7/gKgh VRzANHFYFaqz7Q23J/WbK0+WjN/Wrf6CSreoA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=5Ct8l5DqLVSSbZzwzD4LMe+5b0J5CDfqz/qfWrIN1TE=; b=Qk7ZHUoGNOnnUpdQQjlcNIXn+J+rXqU2i+ujsE3ajOJpAq5uCQ8+HKNSICxWXOcgUE qit+fn1jYmjtW3D2uscVd5TC788vjMCLzQv2qwHyucablie+kh782CfbGO1d334IpRj8 KEEUbRqGSlCRApGMAlHJft8zSXAxhzo7gp2jNVcXHVPfX/+9dipuU8aFxr0ekUnF/VRY LcYDKot6hJU8ac0nQ0yywtnclzIk5z6aesWf06v5D5AB07fBhESRqnImNXWr9EBPQ0yJ COqaxmBxeKU0Sawao7QN5jkGwW7hskQFZUe9S4efBjtRKZu9cpn/RegTJ15b9AKKIID0 vlvg== X-Gm-Message-State: ALoCoQmDns9pXHExJbrPNHDqrIki3zl/Rsby1OMMQpefF38XGU/hwHRbaNqxfRtyo9nx1isYcfWt8i56UdqACrDdfOo5HpzCCH5JOEmpy4l/sa2CpisT5tI7+hfpDUdi+ENyHvFI9QNP X-Received: by 10.112.209.70 with SMTP id mk6mr7306591lbc.44.1411139895249; Fri, 19 Sep 2014 08:18:15 -0700 (PDT) MIME-Version: 1.0 X-Received: by 10.112.209.70 with SMTP id mk6mr7306577lbc.44.1411139895135; Fri, 19 Sep 2014 08:18:15 -0700 (PDT) Received: by 10.112.56.136 with HTTP; Fri, 19 Sep 2014 08:18:15 -0700 (PDT) In-Reply-To: <541C36FF.3010603@frisch.fr> References: <541C2037.5030303@frisch.fr> <1411133763.2930.28.camel@zotac> <541C36FF.3010603@frisch.fr> Date: Fri, 19 Sep 2014 11:18:15 -0400 Message-ID: From: Yaron Minsky To: Alain Frisch Cc: Gerd Stolpmann , Bob Zhang , Caml List Content-Type: text/plain; charset=UTF-8 Subject: Re: [Caml-list] improve omake [was One build system to rule them all] We had a fair number of problems with omake - We've run into lots of performance problems on large builds (2-3 million lines, many thousands of targets) was that omake took a very long time (a few minutes) to be restarted. - The build itself has limited parallelism because of various bottlenecks inside omake, like the fact that it computes its md5sums in a single process. - The default rules didn't do a good job of clearing out stale build artifacts (important for getting reliable incremental builds), and we had to put quite a lot of painful engineering in place to make that work. We needed to do similar work in Jenga to make the same thing happen, but it was a lot more fun writing that code in OCaml! I am not convinced that putting more complicated calculations into programs will work well. I know of no system that does a good job allowing you to deal with complex dependencies that are discovered over the course of a build (which you get with staged programming) that take this approach. It seems that a more expressive, monadic structure fits the bill, and hammering that into the round peg of shell invocations won't work well, I suspect. y On Fri, Sep 19, 2014 at 10:00 AM, Alain Frisch wrote: > On 09/19/2014 03:36 PM, Gerd Stolpmann wrote: >> >> Well, I run frequently into the difficulty that I need some special >> omake function that would be trivial to develop in OCaml (e.g. >> associating some data with some other data, filtering things, doing >> string transformations), but for writing it in the omake language I need >> some time for developing and testing. I have a quite simple idea to >> improve this: Besides OMakefile there could also be an OMakefile.ml, and >> you can define any helper functions there, and they would be >> automatically callable from the OMakefile. I think this is not really >> complicated to do - you'd need to build a custom omake executable >> whenever OMakefile.ml changes, and need to scan the OMakefile.ml >> interface for function signatures that match the form that is callable, >> and you need to generate some glue code. (Later this idea could be >> extended by allowing OCaml code to emit new rules, as described in an >> earlier post.) > > > I can see some cases where it would indeed be more comfortable to implement > build-system calculations in OCaml. But couldn't most of these cases be > implemented as external programs called by omake functions and implemented > e.g. in OCaml? This forces to pass explicitly all the data required by the > calculation to those external programs, but how often is this a problem? > With some convention, the external program could even return description of > new dependencies, to be interpreted by some omake code and injected into the > actual graph. AFAICT, all this is already possible. > > >> I see what you mean. In a recent project I had to define all variables >> with library names, findlib names, intra-project library dependencies >> etc. in the global OMakefile, because they are needed in potentially all >> sub-OMakefiles. That's of course not where these things are best >> naturally defined. > > > A variant is to have e.g. a OPreOMakefile file in each directory and arrange > so that the toplevel OMakefile includes all of them (with a proper ordering) > without processing the rest of the project. This way, you only need to > "lift" the full list of directories, and actual data definitions can be put > where they belong. > >> Maybe we should allow to switch to global context anywhere? I think this >> is solvable. > > > I'm not sure this would easily fit in the current functional semantics. > >> Could be something simple, like matching the wildcard rules against the >> real files. > > > Reading the directory content should be quite cheap, and then it is just > string processing, which should be even cheaper (if done properly). It > would be great to identify such hot spots; maybe some very local tweaks to > algorithmics or data structures could improve performance a lot. > > > > Alain > > > -- > Caml-list mailing list. Subscription management and archives: > https://sympa.inria.fr/sympa/arc/caml-list > Beginner's list: http://groups.yahoo.com/group/ocaml_beginners > Bug reports: http://caml.inria.fr/bin/caml-bugs