From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: caml-list@sympa.inria.fr Delivered-To: caml-list@sympa.inria.fr Received: from mail3-relais-sop.national.inria.fr (mail3-relais-sop.national.inria.fr [192.134.164.104]) by sympa.inria.fr (Postfix) with ESMTPS id B1A9A7F75C for ; Mon, 22 Sep 2014 17:33:35 +0200 (CEST) Received-SPF: None (mail3-smtp-sop.national.inria.fr: no sender authenticity information available from domain of bobzhang1988@gmail.com) identity=pra; client-ip=209.85.216.44; receiver=mail3-smtp-sop.national.inria.fr; envelope-from="bobzhang1988@gmail.com"; x-sender="bobzhang1988@gmail.com"; x-conformance=sidf_compatible Received-SPF: Pass (mail3-smtp-sop.national.inria.fr: domain of bobzhang1988@gmail.com designates 209.85.216.44 as permitted sender) identity=mailfrom; client-ip=209.85.216.44; receiver=mail3-smtp-sop.national.inria.fr; envelope-from="bobzhang1988@gmail.com"; x-sender="bobzhang1988@gmail.com"; x-conformance=sidf_compatible; x-record-type="v=spf1" Received-SPF: None (mail3-smtp-sop.national.inria.fr: no sender authenticity information available from domain of postmaster@mail-qa0-f44.google.com) identity=helo; client-ip=209.85.216.44; receiver=mail3-smtp-sop.national.inria.fr; envelope-from="bobzhang1988@gmail.com"; x-sender="postmaster@mail-qa0-f44.google.com"; x-conformance=sidf_compatible X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: At8BAAhAIFTRVdgslGdsb2JhbABWCoNhVwSCfbYPj1iBa4dNgQYIFgERAQEBAQcLCwkSLIQDAQEBAxMRHQEUBxIGBQEDAREDAgsDCg0dAiQBEQEFAQoYARIJCRCIBwEDCQgNkgWQLG2LMIFygxCJAwoZJwMKZoYWAREBBQ6KDYUPBDIiBIJ/gVMFhQ0FgQ6Pb4R6gg2BYZF/GCmDHB2BdSEvAYEFgUQBAQE X-IPAS-Result: At8BAAhAIFTRVdgslGdsb2JhbABWCoNhVwSCfbYPj1iBa4dNgQYIFgERAQEBAQcLCwkSLIQDAQEBAxMRHQEUBxIGBQEDAREDAgsDCg0dAiQBEQEFAQoYARIJCRCIBwEDCQgNkgWQLG2LMIFygxCJAwoZJwMKZoYWAREBBQ6KDYUPBDIiBIJ/gVMFhQ0FgQ6Pb4R6gg2BYZF/GCmDHB2BdSEvAYEFgUQBAQE X-IronPort-AV: E=Sophos;i="5.04,572,1406584800"; d="scan'208";a="80459379" Received: from mail-qa0-f44.google.com ([209.85.216.44]) by mail3-smtp-sop.national.inria.fr with ESMTP/TLS/RC4-SHA; 22 Sep 2014 17:33:33 +0200 Received: by mail-qa0-f44.google.com with SMTP id x12so278703qac.31 for ; Mon, 22 Sep 2014 08:33:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:cc:content-type; bh=pBDSwXSIeW3f9hWSsFXK0it/oJKmO5I6yLZV/j0CcP4=; b=dMsBFMAIIEBMx2e5+huj/KXg0ny7xh42sQZWEiDgY8mpS+qK/PU4oYDKIgEl7sj0jp BviM/6Dia40ec4X4AQPHSp/enFUfsB4sV0ZeJZCBj4Qwk35nqNTvg6IWsls0v5/0QFEA Hf6WUtXDoVfbyZ6JWktC8mdgQGJExc7eCY4VNJAlHIRy/odb4cv5VPsRAA9KUZC8X2eS W7lOh6sF4/Ss4PFV7YDqx8Ez0f6FtxZ0BRm/rcvy9MVzNTHYYhcy9MFa9zJxxEIk+ol6 d3TwSfCcvqOzhbDM5HWI1rbnp4Bqp6qOpzoAP+9RgSdqWTx+y5MN+D6TtWfyfHyvTLKe VDdg== MIME-Version: 1.0 X-Received: by 10.140.93.43 with SMTP id c40mr18709115qge.10.1411400012888; Mon, 22 Sep 2014 08:33:32 -0700 (PDT) Received: by 10.140.235.193 with HTTP; Mon, 22 Sep 2014 08:33:32 -0700 (PDT) Date: Mon, 22 Sep 2014 11:33:32 -0400 Message-ID: From: Bob Zhang To: Gerd Stolpmann , nogin@metaprl.org Cc: Alain Frisch , Caml List Content-Type: multipart/alternative; boundary=001a113abd606783580503a92c0d Subject: Re: [Caml-list] improve omake [was One build system to rule them all] --001a113abd606783580503a92c0d Content-Type: text/plain; charset=UTF-8 Hi all, I am glad there are some people who like omake. I have a full time job right now, but I am still very interested in improving omake, it would be nice that some more knowledgeable people would contribute(and particular windows developer). Currently my fork is here : https://github.com/bobzhang/omake-fork It's in a re-factorization status . The main things I can think about to improve omake(long term goal): a. remove some unused struct in the omake language, add a doc string and reflection abilities b. make debug easier, error messages easy to follow c. add a new backend (for example to ninja) for faster performance d. expose an API to allow people write rules in ocaml language itself. The short term goal is bug fix and keep it up to date with the latest ocaml compiler -- Hongbo On Fri, Sep 19, 2014 at 1:18 PM, Gerd Stolpmann wrote: > Am Freitag, den 19.09.2014, 11:18 -0400 schrieb Yaron Minsky: > > We had a fair number of problems with omake > > > > - We've run into lots of performance problems on large builds (2-3 > > million lines, many thousands of targets) was that omake took a very > > long time (a few minutes) to be restarted. > > Well, never got into that dimensions. The largest builds had only around > 250Kloc, and omake worked well for that size. I don't know much about > the internals of omake, but I can imagine that certain things are > recomputed too often, and then you run into performance problems at a > certain size. But why can't this be nailed down? > > > - The build itself has limited parallelism because of various > > bottlenecks inside omake, like the fact that it computes its md5sums > > in a single process. > > Hmm, is that evident? Maybe switching to a faster hash algorithm could > solve this? (E.g. just use RC4 and XOR up the output stream; should be > ultimately fast.) > > > - The default rules didn't do a good job of clearing out stale build > > artifacts (important for getting reliable incremental builds), and > > we had to put quite a lot of painful engineering in place to make > > that work. We needed to do similar work in Jenga to make the same > > thing happen, but it was a lot more fun writing that code in OCaml! > > If you just stick to the built-in macros, incremental builds are very > reliable. If you start writing your own rules, there is some chance that > you overlook dependencies. But that's independent of which build system > you use. > > > I am not convinced that putting more complicated calculations into > > programs will work well. I know of no system that does a good job > > allowing you to deal with complex dependencies that are discovered > > over the course of a build (which you get with staged programming) > > that take this approach. It seems that a more expressive, monadic > > structure fits the bill, and hammering that into the round peg of > > shell invocations won't work well, I suspect. > > After all, I think that it would be naive to think that one size fits > all. When you have a large build like yours you probably have quite > special requirements, and are willing to do without comfort features > like a macro language. > > So: no one build system to rule them all. Nevertheless, I'm very much > for improving omake. > > Gerd > > > > y > > > > On Fri, Sep 19, 2014 at 10:00 AM, Alain Frisch wrote: > > > On 09/19/2014 03:36 PM, Gerd Stolpmann wrote: > > >> > > >> Well, I run frequently into the difficulty that I need some special > > >> omake function that would be trivial to develop in OCaml (e.g. > > >> associating some data with some other data, filtering things, doing > > >> string transformations), but for writing it in the omake language I > need > > >> some time for developing and testing. I have a quite simple idea to > > >> improve this: Besides OMakefile there could also be an OMakefile.ml, > and > > >> you can define any helper functions there, and they would be > > >> automatically callable from the OMakefile. I think this is not really > > >> complicated to do - you'd need to build a custom omake executable > > >> whenever OMakefile.ml changes, and need to scan the OMakefile.ml > > >> interface for function signatures that match the form that is > callable, > > >> and you need to generate some glue code. (Later this idea could be > > >> extended by allowing OCaml code to emit new rules, as described in an > > >> earlier post.) > > > > > > > > > I can see some cases where it would indeed be more comfortable to > implement > > > build-system calculations in OCaml. But couldn't most of these cases > be > > > implemented as external programs called by omake functions and > implemented > > > e.g. in OCaml? This forces to pass explicitly all the data required > by the > > > calculation to those external programs, but how often is this a > problem? > > > With some convention, the external program could even return > description of > > > new dependencies, to be interpreted by some omake code and injected > into the > > > actual graph. AFAICT, all this is already possible. > > > > > > > > >> I see what you mean. In a recent project I had to define all variables > > >> with library names, findlib names, intra-project library dependencies > > >> etc. in the global OMakefile, because they are needed in potentially > all > > >> sub-OMakefiles. That's of course not where these things are best > > >> naturally defined. > > > > > > > > > A variant is to have e.g. a OPreOMakefile file in each directory and > arrange > > > so that the toplevel OMakefile includes all of them (with a proper > ordering) > > > without processing the rest of the project. This way, you only need to > > > "lift" the full list of directories, and actual data definitions can > be put > > > where they belong. > > > > > >> Maybe we should allow to switch to global context anywhere? I think > this > > >> is solvable. > > > > > > > > > I'm not sure this would easily fit in the current functional semantics. > > > > > >> Could be something simple, like matching the wildcard rules against > the > > >> real files. > > > > > > > > > Reading the directory content should be quite cheap, and then it is > just > > > string processing, which should be even cheaper (if done properly). It > > > would be great to identify such hot spots; maybe some very local > tweaks to > > > algorithmics or data structures could improve performance a lot. > > > > > > > > > > > > Alain > > > > > > > > > -- > > > Caml-list mailing list. Subscription management and archives: > > > https://sympa.inria.fr/sympa/arc/caml-list > > > Beginner's list: http://groups.yahoo.com/group/ocaml_beginners > > > Bug reports: http://caml.inria.fr/bin/caml-bugs > > > > -- > ------------------------------------------------------------ > Gerd Stolpmann, Darmstadt, Germany gerd@gerd-stolpmann.de > My OCaml site: http://www.camlcity.org > Contact details: http://www.camlcity.org/contact.html > Company homepage: http://www.gerd-stolpmann.de > ------------------------------------------------------------ > -- Regards -- Hongbo Zhang --001a113abd606783580503a92c0d Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi all,
=C2=A0 =C2=A0I am glad there are some people wh= o like omake. I have a full time job right now, but I am still very interes= ted in improving omake, it would be nice that some more knowledgeable peopl= e would contribute(and particular windows developer).
=C2=A0 =C2=A0 Curr= ently my fork is here :=C2=A0https://github.com/bobzhang/omake-fork
=C2=A0 =C2=A0 It's i= n a re-factorization status .
=C2=A0 =C2=A0 The main things I can think = about to improve omake(long term goal):
=C2=A0 =C2=A0 =C2=A0a. remove s= ome unused struct in the omake language, add a doc string and reflection ab= ilities
=C2=A0 =C2=A0 =C2=A0b. make debug easier, error messages easy to= follow
=C2=A0 =C2=A0 =C2=A0c. add a new backend (for example to ninja) = for faster performance
=C2=A0 =C2=A0 =C2=A0d. expose an API to allow peo= ple write rules in ocaml language itself.

=C2=A0 =C2=A0 =C2=A0The sh= ort term goal is bug fix and keep it up to date with the latest ocaml compi= ler
-- Hongbo
O= n Fri, Sep 19, 2014 at 1:18 PM, Gerd Stolpmann <info@gerd-stolpmann.d= e> wrote:
Am Freitag, den 19.09.2014, 11:= 18 -0400 schrieb Yaron Minsky:
> We had a fair number of problems with omake
>
> - We've run into lots of performance problems on large builds (2-3=
>=C2=A0 =C2=A0million lines, many thousands of targets) was that omake t= ook a very
>=C2=A0 =C2=A0long time (a few minutes) to be restarted.

Well, never got into that dimensions. The largest builds had only ar= ound
250Kloc, and omake worked well for that size. I don't know much about the internals of omake, but I can imagine that certain things are
recomputed too often, and then you run into performance problems at a
certain size. But why can't this be nailed down?

> - The build itself has limited parallelism because of various
>=C2=A0 =C2=A0bottlenecks inside omake, like the fact that it computes i= ts md5sums
>=C2=A0 =C2=A0in a single process.

Hmm, is that evident? Maybe switching to a faster hash algorithm cou= ld
solve this? (E.g. just use RC4 and XOR up the output stream; should be
ultimately fast.)

> - The default rules didn't do a good job of clearing out stale bui= ld
>=C2=A0 =C2=A0artifacts (important for getting reliable incremental buil= ds), and
>=C2=A0 =C2=A0we had to put quite a lot of painful engineering in place = to make
>=C2=A0 =C2=A0that work.=C2=A0 We needed to do similar work in Jenga to = make the same
>=C2=A0 =C2=A0thing happen, but it was a lot more fun writing that code = in OCaml!

If you just stick to the built-in macros, incremental builds are ver= y
reliable. If you start writing your own rules, there is some chance that
you overlook dependencies. But that's independent of which build system=
you use.

> I am not convinced that putting more complicated calculations into
> programs will work well.=C2=A0 I know of no system that does a good jo= b
> allowing you to deal with complex dependencies that are discovered
> over the course of a build (which you get with staged programming)
> that take this approach.=C2=A0 It seems that a more expressive, monadi= c
> structure fits the bill, and hammering that into the round peg of
> shell invocations won't work well, I suspect.

After all, I think that it would be naive to think that one size fit= s
all. When you have a large build like yours you probably have quite
special requirements, and are willing to do without comfort features
like a macro language.

So: no one build system to rule them all. Nevertheless, I'm very much for improving omake.

Gerd


> y
>
> On Fri, Sep 19, 2014 at 10:00 AM, Alain Frisch <alain@frisch.fr> wrote:
> > On 09/19/2014 03:36 PM, Gerd Stolpmann wrote:
> >>
> >> Well, I run frequently into the difficulty that I need some s= pecial
> >> omake function that would be trivial to develop in OCaml (e.g= .
> >> associating some data with some other data, filtering things,= doing
> >> string transformations), but for writing it in the omake lang= uage I need
> >> some time for developing and testing. I have a quite simple i= dea to
> >> improve this: Besides OMakefile there could also be an OMakef= ile.ml, and
> >> you can define any helper functions there, and they would be<= br> > >> automatically callable from the OMakefile. I think this is no= t really
> >> complicated to do - you'd need to build a custom omake ex= ecutable
> >> whenever OMakefile.ml changes, and need to scan the OMakefile= .ml
> >> interface for function signatures that match the form that is= callable,
> >> and you need to generate some glue code. (Later this idea cou= ld be
> >> extended by allowing OCaml code to emit new rules, as describ= ed in an
> >> earlier post.)
> >
> >
> > I can see some cases where it would indeed be more comfortable to= implement
> > build-system calculations in OCaml.=C2=A0 But couldn't most o= f these cases be
> > implemented as external programs called by omake functions and im= plemented
> > e.g. in OCaml?=C2=A0 This forces to pass explicitly all the data = required by the
> > calculation to those external programs, but how often is this a p= roblem?
> > With some convention,=C2=A0 the external program could even retur= n description of
> > new dependencies, to be interpreted by some omake code and inject= ed into the
> > actual graph. AFAICT, all this is already possible.
> >
> >
> >> I see what you mean. In a recent project I had to define all = variables
> >> with library names, findlib names, intra-project library depe= ndencies
> >> etc. in the global OMakefile, because they are needed in pote= ntially all
> >> sub-OMakefiles. That's of course not where these things a= re best
> >> naturally defined.
> >
> >
> > A variant is to have e.g. a OPreOMakefile file in each directory = and arrange
> > so that the toplevel OMakefile includes all of them (with a prope= r ordering)
> > without processing the rest of the project.=C2=A0 This way, you o= nly need to
> > "lift" the full list of directories, and actual data de= finitions can be put
> > where they belong.
> >
> >> Maybe we should allow to switch to global context anywhere? I= think this
> >> is solvable.
> >
> >
> > I'm not sure this would easily fit in the current functional = semantics.
> >
> >> Could be something simple, like matching the wildcard rules a= gainst the
> >> real files.
> >
> >
> > Reading the directory content should be quite cheap, and then it = is just
> > string processing, which should be even cheaper (if done properly= ).=C2=A0 It
> > would be great to identify such hot spots; maybe some very local = tweaks to
> > algorithmics or data structures could improve performance a lot.<= br> > >
> >
> >
> > Alain
> >
> >
> > --
> > Caml-list mailing list.=C2=A0 Subscription management and archive= s:
> > https://sympa.inria.fr/sympa/arc/caml-list
> > Beginner's list: http://groups.yahoo.com/group/ocaml_beginne= rs
> > Bug reports: http://caml.inria.fr/bin/caml-bugs
>

--
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany=C2=A0 =C2=A0 gerd@gerd-stolpmann.de
My OCaml site:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 http://www.camlcity.org
Contact details:=C2=A0 =C2=A0 =C2=A0 =C2=A0 http://www.camlcity.org/contact.html
Company homepage:=C2=A0 =C2=A0 =C2=A0 =C2=A0http://www.gerd-stolpmann.de
------------------------------------------------------------



--
= Regards
-- Hongbo Zhang
--001a113abd606783580503a92c0d--