From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: caml-list@sympa.inria.fr Delivered-To: caml-list@sympa.inria.fr Received: from mail2-relais-roc.national.inria.fr (mail2-relais-roc.national.inria.fr [192.134.164.83]) by sympa.inria.fr (Postfix) with ESMTPS id 09C9E7FACE for ; Wed, 24 Sep 2014 15:37:11 +0200 (CEST) Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of info@gerd-stolpmann.de) identity=pra; client-ip=212.227.17.10; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="info@gerd-stolpmann.de"; x-sender="info@gerd-stolpmann.de"; x-conformance=sidf_compatible Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of info@gerd-stolpmann.de) identity=mailfrom; client-ip=212.227.17.10; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="info@gerd-stolpmann.de"; x-sender="info@gerd-stolpmann.de"; x-conformance=sidf_compatible Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of postmaster@mout.kundenserver.de) identity=helo; client-ip=212.227.17.10; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="info@gerd-stolpmann.de"; x-sender="postmaster@mout.kundenserver.de"; x-conformance=sidf_compatible X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ai8EAL3IIlTU4xEKm2dsb2JhbABWCoMjPleDAbYBkTsKh00BgQkWAREBAQEBAQYLCwkUK4QEAQEEIzIZBgUQCxgNHQICRRIGEwkJiBgDFQmtCBQhbw2NNwOHOQEXig4chRgEMiYHgjdBEoFBBYUQBYEOi3ODf4R7g26FZgWPcx6BW2oBgQWBRAEBAQ X-IPAS-Result: Ai8EAL3IIlTU4xEKm2dsb2JhbABWCoMjPleDAbYBkTsKh00BgQkWAREBAQEBAQYLCwkUK4QEAQEEIzIZBgUQCxgNHQICRRIGEwkJiBgDFQmtCBQhbw2NNwOHOQEXig4chRgEMiYHgjdBEoFBBYUQBYEOi3ODf4R7g26FZgWPcx6BW2oBgQWBRAEBAQ X-IronPort-AV: E=Sophos;i="5.04,589,1406584800"; d="asc'?scan'208";a="97659448" Received: from mout.kundenserver.de ([212.227.17.10]) by mail2-smtp-roc.national.inria.fr with ESMTP/TLS/DHE-RSA-AES256-SHA; 24 Sep 2014 15:37:10 +0200 Received: from office1.lan.sumadev.de (dslb-178-004-068-137.178.004.pools.vodafone-ip.de [178.4.68.137]) by mrelayeu.kundenserver.de (node=mreue102) with ESMTP (Nemesis) id 0M57wk-1YUXcI00zO-00zHJL; Wed, 24 Sep 2014 15:37:08 +0200 Received: from [192.168.10.103] (unknown [5.146.48.81]) by office1.lan.sumadev.de (Postfix) with ESMTPSA id 49E27DC270; Wed, 24 Sep 2014 15:37:07 +0200 (CEST) Message-ID: <1411565826.2930.76.camel@zotac> From: Gerd Stolpmann To: Bob Zhang Cc: nogin@metaprl.org, Alain Frisch , Caml List Date: Wed, 24 Sep 2014 15:37:06 +0200 In-Reply-To: References: Content-Type: multipart/signed; micalg="pgp-sha1"; protocol="application/pgp-signature"; boundary="=-6soFSa+CFg4CRG35vZBL" X-Mailer: Evolution 3.10.4-0ubuntu1 Mime-Version: 1.0 X-Provags-ID: V02:K0:VrptAsPozYG1n8p23iVRvp6h7fVHBBwvjk3l4pqFZ/R K3TRciz2NfQvZVWhEym7qP9mfdc3ggvaIV3vG9Hclx8ekUfZnI v9rW2utb1AWytyB8VnW5JdrbTMFBthFyHd8Rp+cLUkCBhIqZxR WlsFCM6K5BeNO/CNOOTBYqGYPlcCs7U/rquFRXxBMMhi/GQnjz v1YkLfAv65hIISnfQYw3EcJZ8D5y0Tj/eDtQN+zWG6LIIk9bZ/ +m1gFMrUNfqwQSMXYjGEMm7DZZ0+MFPJgtRehdAh/u4/Yu9OxQ Vc+GGDP7EVv6eMoESBI1Oage6N5qVfTH5QKZbJsDEI2CmV/jtr YxusxGKxQxkffDpX4t/lR8qeM8vFGlFVN2EdFl5s8 X-UI-Out-Filterresults: notjunk:1; Subject: Re: [Caml-list] improve omake [was One build system to rule them all] --=-6soFSa+CFg4CRG35vZBL Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Although I don't have much time now, I'll try to find some for helping here. Gerd Am Montag, den 22.09.2014, 11:33 -0400 schrieb Bob Zhang: > Hi all, > I am glad there are some people who like omake. I have a full time > job right now, but I am still very interested in improving omake, it > would be nice that some more knowledgeable people would contribute(and > particular windows developer). > Currently my fork is here : https://github.com/bobzhang/omake-fork > It's in a re-factorization status . > The main things I can think about to improve omake(long term > goal): > a. remove some unused struct in the omake language, add a doc > string and reflection abilities > b. make debug easier, error messages easy to follow > c. add a new backend (for example to ninja) for faster > performance > d. expose an API to allow people write rules in ocaml language > itself. >=20 > The short term goal is bug fix and keep it up to date with the > latest ocaml compiler > -- Hongbo > On Fri, Sep 19, 2014 at 1:18 PM, Gerd Stolpmann > wrote: > Am Freitag, den 19.09.2014, 11:18 -0400 schrieb Yaron Minsky: > > We had a fair number of problems with omake > > > > - We've run into lots of performance problems on large > builds (2-3 > > million lines, many thousands of targets) was that omake > took a very > > long time (a few minutes) to be restarted. >=20=20=20=20=20=20=20=20=20 > Well, never got into that dimensions. The largest builds had > only around > 250Kloc, and omake worked well for that size. I don't know > much about > the internals of omake, but I can imagine that certain things > are > recomputed too often, and then you run into performance > problems at a > certain size. But why can't this be nailed down? >=20=20=20=20=20=20=20=20=20 > > - The build itself has limited parallelism because of > various > > bottlenecks inside omake, like the fact that it computes > its md5sums > > in a single process. >=20=20=20=20=20=20=20=20=20 > Hmm, is that evident? Maybe switching to a faster hash > algorithm could > solve this? (E.g. just use RC4 and XOR up the output stream; > should be > ultimately fast.) >=20=20=20=20=20=20=20=20=20 > > - The default rules didn't do a good job of clearing out > stale build > > artifacts (important for getting reliable incremental > builds), and > > we had to put quite a lot of painful engineering in place > to make > > that work. We needed to do similar work in Jenga to make > the same > > thing happen, but it was a lot more fun writing that code > in OCaml! >=20=20=20=20=20=20=20=20=20 > If you just stick to the built-in macros, incremental builds > are very > reliable. If you start writing your own rules, there is some > chance that > you overlook dependencies. But that's independent of which > build system > you use. >=20=20=20=20=20=20=20=20=20 > > I am not convinced that putting more complicated > calculations into > > programs will work well. I know of no system that does a > good job > > allowing you to deal with complex dependencies that are > discovered > > over the course of a build (which you get with staged > programming) > > that take this approach. It seems that a more expressive, > monadic > > structure fits the bill, and hammering that into the round > peg of > > shell invocations won't work well, I suspect. >=20=20=20=20=20=20=20=20=20 > After all, I think that it would be naive to think that one > size fits > all. When you have a large build like yours you probably have > quite > special requirements, and are willing to do without comfort > features > like a macro language. >=20=20=20=20=20=20=20=20=20 > So: no one build system to rule them all. Nevertheless, I'm > very much > for improving omake. >=20=20=20=20=20=20=20=20=20 > Gerd >=20=20=20=20=20=20=20=20=20 >=20=20=20=20=20=20=20=20=20 > > y > > > > On Fri, Sep 19, 2014 at 10:00 AM, Alain Frisch > wrote: > > > On 09/19/2014 03:36 PM, Gerd Stolpmann wrote: > > >> > > >> Well, I run frequently into the difficulty that I need > some special > > >> omake function that would be trivial to develop in OCaml > (e.g. > > >> associating some data with some other data, filtering > things, doing > > >> string transformations), but for writing it in the omake > language I need > > >> some time for developing and testing. I have a quite > simple idea to > > >> improve this: Besides OMakefile there could also be an > OMakefile.ml, and > > >> you can define any helper functions there, and they would > be > > >> automatically callable from the OMakefile. I think this > is not really > > >> complicated to do - you'd need to build a custom omake > executable > > >> whenever OMakefile.ml changes, and need to scan the > OMakefile.ml > > >> interface for function signatures that match the form > that is callable, > > >> and you need to generate some glue code. (Later this idea > could be > > >> extended by allowing OCaml code to emit new rules, as > described in an > > >> earlier post.) > > > > > > > > > I can see some cases where it would indeed be more > comfortable to implement > > > build-system calculations in OCaml. But couldn't most of > these cases be > > > implemented as external programs called by omake functions > and implemented > > > e.g. in OCaml? This forces to pass explicitly all the > data required by the > > > calculation to those external programs, but how often is > this a problem? > > > With some convention, the external program could even > return description of > > > new dependencies, to be interpreted by some omake code and > injected into the > > > actual graph. AFAICT, all this is already possible. > > > > > > > > >> I see what you mean. In a recent project I had to define > all variables > > >> with library names, findlib names, intra-project library > dependencies > > >> etc. in the global OMakefile, because they are needed in > potentially all > > >> sub-OMakefiles. That's of course not where these things > are best > > >> naturally defined. > > > > > > > > > A variant is to have e.g. a OPreOMakefile file in each > directory and arrange > > > so that the toplevel OMakefile includes all of them (with > a proper ordering) > > > without processing the rest of the project. This way, you > only need to > > > "lift" the full list of directories, and actual data > definitions can be put > > > where they belong. > > > > > >> Maybe we should allow to switch to global context > anywhere? I think this > > >> is solvable. > > > > > > > > > I'm not sure this would easily fit in the current > functional semantics. > > > > > >> Could be something simple, like matching the wildcard > rules against the > > >> real files. > > > > > > > > > Reading the directory content should be quite cheap, and > then it is just > > > string processing, which should be even cheaper (if done > properly). It > > > would be great to identify such hot spots; maybe some very > local tweaks to > > > algorithmics or data structures could improve performance > a lot. > > > > > > > > > > > > Alain > > > > > > > > > -- > > > Caml-list mailing list. Subscription management and > archives: > > > https://sympa.inria.fr/sympa/arc/caml-list > > > Beginner's list: > http://groups.yahoo.com/group/ocaml_beginners > > > Bug reports: http://caml.inria.fr/bin/caml-bugs > > >=20=20=20=20=20=20=20=20=20 >=20=20=20=20=20=20=20=20=20 > -- > ------------------------------------------------------------ > Gerd Stolpmann, Darmstadt, Germany gerd@gerd-stolpmann.de > My OCaml site: http://www.camlcity.org > Contact details: http://www.camlcity.org/contact.html > Company homepage: http://www.gerd-stolpmann.de > ------------------------------------------------------------ >=20=20=20=20=20=20=20=20=20 >=20 >=20 >=20 >=20 > --=20 > Regards > -- Hongbo Zhang --=20 ------------------------------------------------------------ Gerd Stolpmann, Darmstadt, Germany gerd@gerd-stolpmann.de My OCaml site: http://www.camlcity.org Contact details: http://www.camlcity.org/contact.html Company homepage: http://www.gerd-stolpmann.de ------------------------------------------------------------ --=-6soFSa+CFg4CRG35vZBL Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAABAgAGBQJUIskCAAoJEAaM4b9ZLB5THEsH/A/tE0PhiInB1j4eBsk7RIkP uMRdl0fm1VbnIRYHCb+sw6mFOvqsg8HMx3hCp06miFGk5hGnuM56NwqDH3MOSgP8 Vc1cRtbzrIlodyRi1JjAo/DCJAEIG3AyQNU7A7tj4f2qj4sayagzdR648KrzsnGx bxAQwc2x9Jor9cncTdQDBj1YJrLSPqOpT5Y8S8OPIVZ5OKmgy1SnSTnpUisZnNgf QG59N+f++S9YCGYlJgAjQQoq6MlDpENrtgfFL5+nteBYy6MlqketItdX1MqwVVA9 kIagfyHZibzsMlkZcPkDZtzHQ02CWXe1xALpuJ7ZKRbbAiZaz0FDywORZ+kSmYQ= =fYkY -----END PGP SIGNATURE----- --=-6soFSa+CFg4CRG35vZBL--