From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: caml-list@sympa.inria.fr Delivered-To: caml-list@sympa.inria.fr Received: from mail3-relais-sop.national.inria.fr (mail3-relais-sop.national.inria.fr [192.134.164.104]) by sympa.inria.fr (Postfix) with ESMTPS id E95237FACD for ; Fri, 19 Sep 2014 19:18:17 +0200 (CEST) Received-SPF: None (mail3-smtp-sop.national.inria.fr: no sender authenticity information available from domain of info@gerd-stolpmann.de) identity=pra; client-ip=212.227.17.24; receiver=mail3-smtp-sop.national.inria.fr; envelope-from="info@gerd-stolpmann.de"; x-sender="info@gerd-stolpmann.de"; x-conformance=sidf_compatible Received-SPF: None (mail3-smtp-sop.national.inria.fr: no sender authenticity information available from domain of info@gerd-stolpmann.de) identity=mailfrom; client-ip=212.227.17.24; receiver=mail3-smtp-sop.national.inria.fr; envelope-from="info@gerd-stolpmann.de"; x-sender="info@gerd-stolpmann.de"; x-conformance=sidf_compatible Received-SPF: None (mail3-smtp-sop.national.inria.fr: no sender authenticity information available from domain of postmaster@mout.kundenserver.de) identity=helo; client-ip=212.227.17.24; receiver=mail3-smtp-sop.national.inria.fr; envelope-from="info@gerd-stolpmann.de"; x-sender="postmaster@mout.kundenserver.de"; x-conformance=sidf_compatible X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AtsBAJ5kHFTU4xEYnGdsb2JhbABWCoNgV4MBtTWRNwqHTQGBAhYBEQEBAQEBBg0JCRQqhAMBAQEDASMyGQYFBQsLDgoNHQICRRIGEwkJiBgDCQwJrRYUIW8NjTwDhzgBF4l+HIULBDImB4I3QRKBQQWFDQWBDotvg32IaIVmBY9sHIFbagGBBYFEAQEB X-IPAS-Result: AtsBAJ5kHFTU4xEYnGdsb2JhbABWCoNgV4MBtTWRNwqHTQGBAhYBEQEBAQEBBg0JCRQqhAMBAQEDASMyGQYFBQsLDgoNHQICRRIGEwkJiBgDCQwJrRYUIW8NjTwDhzgBF4l+HIULBDImB4I3QRKBQQWFDQWBDotvg32IaIVmBY9sHIFbagGBBYFEAQEB X-IronPort-AV: E=Sophos;i="5.04,556,1406584800"; d="asc'?scan'208";a="80096412" Received: from mout.kundenserver.de ([212.227.17.24]) by mail3-smtp-sop.national.inria.fr with ESMTP/TLS/DHE-RSA-AES256-SHA; 19 Sep 2014 19:18:17 +0200 Received: from office1.lan.sumadev.de (dslb-178-004-068-137.178.004.pools.vodafone-ip.de [178.4.68.137]) by mrelayeu.kundenserver.de (node=mreue101) with ESMTP (Nemesis) id 0Lecog-1Y60Gv0gfe-00qRkV; Fri, 19 Sep 2014 19:18:15 +0200 Received: from [192.168.10.103] (ip-37-201-182-45.hsi13.unitymediagroup.de [37.201.182.45]) by office1.lan.sumadev.de (Postfix) with ESMTPSA id 883D5DC270; Fri, 19 Sep 2014 19:18:14 +0200 (CEST) Message-ID: <1411147089.2930.42.camel@zotac> From: Gerd Stolpmann To: Yaron Minsky Cc: Alain Frisch , Bob Zhang , Caml List Date: Fri, 19 Sep 2014 19:18:09 +0200 In-Reply-To: References: <541C2037.5030303@frisch.fr> <1411133763.2930.28.camel@zotac> <541C36FF.3010603@frisch.fr> Content-Type: multipart/signed; micalg="pgp-sha1"; protocol="application/pgp-signature"; boundary="=-wlMsVE3wH62wdVNiTjko" X-Mailer: Evolution 3.10.4-0ubuntu1 Mime-Version: 1.0 X-Provags-ID: V02:K0:sUMyBA7Vply7Q0RsttalTUqfiUPa6LmJIwKEDthpkCj XVhcj6sPiNOlU7wWlZXXMuyvfLZ0P/Cux6UAnfk9Vwonn4vxaM tIk+UWW98ryRmSvcstkYYCJTf3+eH34mgSz8ZZ6Ch7++LQZnDy TyLswXYrJ0RXG3y+zB/7UArVDmJoSPr/O7zjZ8FAv9ilIzKhE+ 42o91YafGilhUlpliyc/3loaUal5KjGZIxWT1YDVdO0ZVztrpS XNCW+uISe6AI4EWLLECPakCwirJwxaTnyqfGUkqA8JOwFBDE5e ZzGXItFYwLmDcsI4ygAmS3g9apZavlZLhBmH1FnbVsEhmlJYr2 6RTK12JJJJIPoqRYNQF7WkXEkv9O4c9YmNgey2SLl X-UI-Out-Filterresults: notjunk:1; Subject: Re: [Caml-list] improve omake [was One build system to rule them all] --=-wlMsVE3wH62wdVNiTjko Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Am Freitag, den 19.09.2014, 11:18 -0400 schrieb Yaron Minsky: > We had a fair number of problems with omake >=20 > - We've run into lots of performance problems on large builds (2-3 > million lines, many thousands of targets) was that omake took a very > long time (a few minutes) to be restarted. Well, never got into that dimensions. The largest builds had only around 250Kloc, and omake worked well for that size. I don't know much about the internals of omake, but I can imagine that certain things are recomputed too often, and then you run into performance problems at a certain size. But why can't this be nailed down? > - The build itself has limited parallelism because of various > bottlenecks inside omake, like the fact that it computes its md5sums > in a single process. Hmm, is that evident? Maybe switching to a faster hash algorithm could solve this? (E.g. just use RC4 and XOR up the output stream; should be ultimately fast.) > - The default rules didn't do a good job of clearing out stale build > artifacts (important for getting reliable incremental builds), and > we had to put quite a lot of painful engineering in place to make > that work. We needed to do similar work in Jenga to make the same > thing happen, but it was a lot more fun writing that code in OCaml! If you just stick to the built-in macros, incremental builds are very reliable. If you start writing your own rules, there is some chance that you overlook dependencies. But that's independent of which build system you use. > I am not convinced that putting more complicated calculations into > programs will work well. I know of no system that does a good job > allowing you to deal with complex dependencies that are discovered > over the course of a build (which you get with staged programming) > that take this approach. It seems that a more expressive, monadic > structure fits the bill, and hammering that into the round peg of > shell invocations won't work well, I suspect. After all, I think that it would be naive to think that one size fits all. When you have a large build like yours you probably have quite special requirements, and are willing to do without comfort features like a macro language. So: no one build system to rule them all. Nevertheless, I'm very much for improving omake. Gerd > y >=20 > On Fri, Sep 19, 2014 at 10:00 AM, Alain Frisch wrote: > > On 09/19/2014 03:36 PM, Gerd Stolpmann wrote: > >> > >> Well, I run frequently into the difficulty that I need some special > >> omake function that would be trivial to develop in OCaml (e.g. > >> associating some data with some other data, filtering things, doing > >> string transformations), but for writing it in the omake language I ne= ed > >> some time for developing and testing. I have a quite simple idea to > >> improve this: Besides OMakefile there could also be an OMakefile.ml, a= nd > >> you can define any helper functions there, and they would be > >> automatically callable from the OMakefile. I think this is not really > >> complicated to do - you'd need to build a custom omake executable > >> whenever OMakefile.ml changes, and need to scan the OMakefile.ml > >> interface for function signatures that match the form that is callable, > >> and you need to generate some glue code. (Later this idea could be > >> extended by allowing OCaml code to emit new rules, as described in an > >> earlier post.) > > > > > > I can see some cases where it would indeed be more comfortable to imple= ment > > build-system calculations in OCaml. But couldn't most of these cases be > > implemented as external programs called by omake functions and implemen= ted > > e.g. in OCaml? This forces to pass explicitly all the data required by= the > > calculation to those external programs, but how often is this a problem? > > With some convention, the external program could even return descripti= on of > > new dependencies, to be interpreted by some omake code and injected int= o the > > actual graph. AFAICT, all this is already possible. > > > > > >> I see what you mean. In a recent project I had to define all variables > >> with library names, findlib names, intra-project library dependencies > >> etc. in the global OMakefile, because they are needed in potentially a= ll > >> sub-OMakefiles. That's of course not where these things are best > >> naturally defined. > > > > > > A variant is to have e.g. a OPreOMakefile file in each directory and ar= range > > so that the toplevel OMakefile includes all of them (with a proper orde= ring) > > without processing the rest of the project. This way, you only need to > > "lift" the full list of directories, and actual data definitions can be= put > > where they belong. > > > >> Maybe we should allow to switch to global context anywhere? I think th= is > >> is solvable. > > > > > > I'm not sure this would easily fit in the current functional semantics. > > > >> Could be something simple, like matching the wildcard rules against the > >> real files. > > > > > > Reading the directory content should be quite cheap, and then it is just > > string processing, which should be even cheaper (if done properly). It > > would be great to identify such hot spots; maybe some very local tweaks= to > > algorithmics or data structures could improve performance a lot. > > > > > > > > Alain > > > > > > -- > > Caml-list mailing list. Subscription management and archives: > > https://sympa.inria.fr/sympa/arc/caml-list > > Beginner's list: http://groups.yahoo.com/group/ocaml_beginners > > Bug reports: http://caml.inria.fr/bin/caml-bugs >=20 --=20 ------------------------------------------------------------ Gerd Stolpmann, Darmstadt, Germany gerd@gerd-stolpmann.de My OCaml site: http://www.camlcity.org Contact details: http://www.camlcity.org/contact.html Company homepage: http://www.gerd-stolpmann.de ------------------------------------------------------------ --=-wlMsVE3wH62wdVNiTjko Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAABAgAGBQJUHGVRAAoJEAaM4b9ZLB5TZMEIAIVxf+JZEgycLCzDewBubs6g praF1hmOn/LxCjcisuAMzVRrz6kSHZeWoLcPaERZyaGxMICzqAMvgkDfbaonzHcZ aWzEV0E85w2DtnT4U+2ZCJlSkbyH2RI42qY1vmKgXy3Yktl51vatIRxEKm/xt74j K6lfSZdo6LKb9sl+mkR9jQHL1uapKYETU96f32ZwVrS2qD6ZLxW/hxWmyKU3nV5t XBHwyBKX8BHDMKiR6PJuIPehK3PNxW1v5v4vYXg0BZ+hibpL6oXPWYFexePWp+3f CxNvPtWbDlEIrSP9NB2h6FCsfq0aOh5X/NcQX64M4pu4K7/a2R/lRPuZzcyaRiE= =XnRp -----END PGP SIGNATURE----- --=-wlMsVE3wH62wdVNiTjko--