From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: caml-list@sympa.inria.fr Delivered-To: caml-list@sympa.inria.fr Received: from mail2-relais-roc.national.inria.fr (mail2-relais-roc.national.inria.fr [192.134.164.83]) by sympa.inria.fr (Postfix) with ESMTPS id B064A7F75C for ; Wed, 24 Sep 2014 17:47:24 +0200 (CEST) Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of alain.frisch@lexifi.com) identity=pra; client-ip=193.252.23.212; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="alain.frisch@lexifi.com"; x-sender="alain.frisch@lexifi.com"; x-conformance=sidf_compatible Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of alain.frisch@lexifi.com) identity=mailfrom; client-ip=193.252.23.212; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="alain.frisch@lexifi.com"; x-sender="alain.frisch@lexifi.com"; x-conformance=sidf_compatible Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of postmaster@msa.smtpout.orange.fr) identity=helo; client-ip=193.252.23.212; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="alain.frisch@lexifi.com"; x-sender="postmaster@msa.smtpout.orange.fr"; x-conformance=sidf_compatible X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: As4BAD7nIlTB/BfUnGdsb2JhbABWCoNhV4MBtgKRRYdNAYEdAREBAQEBAQgLCQkUK4QEAQEEIxUdGQYEARALDgoCAgUEEgsCAgkDAgECATMSBgEMAQUCAQEFCYgYAxUJrSaOdwOHDgEXgSyIfoUYBDImB4J4gVMFhRAFgQ6MPYM1hHuCDYFhhWuPcx6BW2oBgQWBRAEBAQ X-IPAS-Result: As4BAD7nIlTB/BfUnGdsb2JhbABWCoNhV4MBtgKRRYdNAYEdAREBAQEBAQgLCQkUK4QEAQEEIxUdGQYEARALDgoCAgUEEgsCAgkDAgECATMSBgEMAQUCAQEFCYgYAxUJrSaOdwOHDgEXgSyIfoUYBDImB4J4gVMFhRAFgQ6MPYM1hHuCDYFhhWuPcx6BW2oBgQWBRAEBAQ X-IronPort-AV: E=Sophos;i="5.04,589,1406584800"; d="scan'208";a="97685274" Received: from msa03.smtpout.orange.fr (HELO msa.smtpout.orange.fr) ([193.252.23.212]) by mail2-smtp-roc.national.inria.fr with ESMTP; 24 Sep 2014 17:47:23 +0200 Received: from [192.168.1.132] ([92.151.37.5]) by mwinf5d52 with ME id v3nN1o00606fm10033nNlY; Wed, 24 Sep 2014 17:47:23 +0200 X-ME-Helo: [192.168.1.132] X-ME-Auth: bGV4aWZpQHdhbmFkb28uZnI= X-ME-Date: Wed, 24 Sep 2014 17:47:23 +0200 X-ME-IP: 92.151.37.5 Message-ID: <5422E78D.1040601@lexifi.com> Date: Wed, 24 Sep 2014 17:47:25 +0200 From: Alain Frisch Organization: LexiFi User-Agent: Mozilla/5.0 (X11; Linux i686 on x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.1.1 MIME-Version: 1.0 To: Gerd Stolpmann , Bob Zhang CC: nogin@metaprl.org, Caml List References: <1411565826.2930.76.camel@zotac> In-Reply-To: <1411565826.2930.76.camel@zotac> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Validation-by: alain.frisch@lexifi.com Subject: Re: [Caml-list] improve omake [was One build system to rule them all] LexiFi might be willing to provide some funding for improving omake, esp. its performances. Please contact me if you (Gerd, Hongbo, anyone) want to discuss that. -- Alain On 09/24/2014 03:37 PM, Gerd Stolpmann wrote: > Although I don't have much time now, I'll try to find some for helping > here. > > Gerd > > Am Montag, den 22.09.2014, 11:33 -0400 schrieb Bob Zhang: >> Hi all, >> I am glad there are some people who like omake. I have a full time >> job right now, but I am still very interested in improving omake, it >> would be nice that some more knowledgeable people would contribute(and >> particular windows developer). >> Currently my fork is here : https://github.com/bobzhang/omake-fork >> It's in a re-factorization status . >> The main things I can think about to improve omake(long term >> goal): >> a. remove some unused struct in the omake language, add a doc >> string and reflection abilities >> b. make debug easier, error messages easy to follow >> c. add a new backend (for example to ninja) for faster >> performance >> d. expose an API to allow people write rules in ocaml language >> itself. >> >> The short term goal is bug fix and keep it up to date with the >> latest ocaml compiler >> -- Hongbo >> On Fri, Sep 19, 2014 at 1:18 PM, Gerd Stolpmann >> wrote: >> Am Freitag, den 19.09.2014, 11:18 -0400 schrieb Yaron Minsky: >> > We had a fair number of problems with omake >> > >> > - We've run into lots of performance problems on large >> builds (2-3 >> > million lines, many thousands of targets) was that omake >> took a very >> > long time (a few minutes) to be restarted. >> >> Well, never got into that dimensions. The largest builds had >> only around >> 250Kloc, and omake worked well for that size. I don't know >> much about >> the internals of omake, but I can imagine that certain things >> are >> recomputed too often, and then you run into performance >> problems at a >> certain size. But why can't this be nailed down? >> >> > - The build itself has limited parallelism because of >> various >> > bottlenecks inside omake, like the fact that it computes >> its md5sums >> > in a single process. >> >> Hmm, is that evident? Maybe switching to a faster hash >> algorithm could >> solve this? (E.g. just use RC4 and XOR up the output stream; >> should be >> ultimately fast.) >> >> > - The default rules didn't do a good job of clearing out >> stale build >> > artifacts (important for getting reliable incremental >> builds), and >> > we had to put quite a lot of painful engineering in place >> to make >> > that work. We needed to do similar work in Jenga to make >> the same >> > thing happen, but it was a lot more fun writing that code >> in OCaml! >> >> If you just stick to the built-in macros, incremental builds >> are very >> reliable. If you start writing your own rules, there is some >> chance that >> you overlook dependencies. But that's independent of which >> build system >> you use. >> >> > I am not convinced that putting more complicated >> calculations into >> > programs will work well. I know of no system that does a >> good job >> > allowing you to deal with complex dependencies that are >> discovered >> > over the course of a build (which you get with staged >> programming) >> > that take this approach. It seems that a more expressive, >> monadic >> > structure fits the bill, and hammering that into the round >> peg of >> > shell invocations won't work well, I suspect. >> >> After all, I think that it would be naive to think that one >> size fits >> all. When you have a large build like yours you probably have >> quite >> special requirements, and are willing to do without comfort >> features >> like a macro language. >> >> So: no one build system to rule them all. Nevertheless, I'm >> very much >> for improving omake. >> >> Gerd >> >> >> > y >> > >> > On Fri, Sep 19, 2014 at 10:00 AM, Alain Frisch >> wrote: >> > > On 09/19/2014 03:36 PM, Gerd Stolpmann wrote: >> > >> >> > >> Well, I run frequently into the difficulty that I need >> some special >> > >> omake function that would be trivial to develop in OCaml >> (e.g. >> > >> associating some data with some other data, filtering >> things, doing >> > >> string transformations), but for writing it in the omake >> language I need >> > >> some time for developing and testing. I have a quite >> simple idea to >> > >> improve this: Besides OMakefile there could also be an >> OMakefile.ml, and >> > >> you can define any helper functions there, and they would >> be >> > >> automatically callable from the OMakefile. I think this >> is not really >> > >> complicated to do - you'd need to build a custom omake >> executable >> > >> whenever OMakefile.ml changes, and need to scan the >> OMakefile.ml >> > >> interface for function signatures that match the form >> that is callable, >> > >> and you need to generate some glue code. (Later this idea >> could be >> > >> extended by allowing OCaml code to emit new rules, as >> described in an >> > >> earlier post.) >> > > >> > > >> > > I can see some cases where it would indeed be more >> comfortable to implement >> > > build-system calculations in OCaml. But couldn't most of >> these cases be >> > > implemented as external programs called by omake functions >> and implemented >> > > e.g. in OCaml? This forces to pass explicitly all the >> data required by the >> > > calculation to those external programs, but how often is >> this a problem? >> > > With some convention, the external program could even >> return description of >> > > new dependencies, to be interpreted by some omake code and >> injected into the >> > > actual graph. AFAICT, all this is already possible. >> > > >> > > >> > >> I see what you mean. In a recent project I had to define >> all variables >> > >> with library names, findlib names, intra-project library >> dependencies >> > >> etc. in the global OMakefile, because they are needed in >> potentially all >> > >> sub-OMakefiles. That's of course not where these things >> are best >> > >> naturally defined. >> > > >> > > >> > > A variant is to have e.g. a OPreOMakefile file in each >> directory and arrange >> > > so that the toplevel OMakefile includes all of them (with >> a proper ordering) >> > > without processing the rest of the project. This way, you >> only need to >> > > "lift" the full list of directories, and actual data >> definitions can be put >> > > where they belong. >> > > >> > >> Maybe we should allow to switch to global context >> anywhere? I think this >> > >> is solvable. >> > > >> > > >> > > I'm not sure this would easily fit in the current >> functional semantics. >> > > >> > >> Could be something simple, like matching the wildcard >> rules against the >> > >> real files. >> > > >> > > >> > > Reading the directory content should be quite cheap, and >> then it is just >> > > string processing, which should be even cheaper (if done >> properly). It >> > > would be great to identify such hot spots; maybe some very >> local tweaks to >> > > algorithmics or data structures could improve performance >> a lot. >> > > >> > > >> > > >> > > Alain >> > > >> > > >> > > -- >> > > Caml-list mailing list. Subscription management and >> archives: >> > > https://sympa.inria.fr/sympa/arc/caml-list >> > > Beginner's list: >> http://groups.yahoo.com/group/ocaml_beginners >> > > Bug reports: http://caml.inria.fr/bin/caml-bugs >> > >> >> >> -- >> ------------------------------------------------------------ >> Gerd Stolpmann, Darmstadt, Germany gerd@gerd-stolpmann.de >> My OCaml site: http://www.camlcity.org >> Contact details: http://www.camlcity.org/contact.html >> Company homepage: http://www.gerd-stolpmann.de >> ------------------------------------------------------------ >> >> >> >> >> >> -- >> Regards >> -- Hongbo Zhang >