Re: [Caml-list] improve omake [was One build system to rule them all]

[Gerd Stolpmann <info@gerd-stolpmann.de>]

[-- Attachment #1: Type: text/plain, Size: 9652 bytes --]

Although I don't have much time now, I'll try to find some for helping
here.

Gerd

Am Montag, den 22.09.2014, 11:33 -0400 schrieb Bob Zhang:
> Hi all,
>    I am glad there are some people who like omake. I have a full time
> job right now, but I am still very interested in improving omake, it
> would be nice that some more knowledgeable people would contribute(and
> particular windows developer).
>     Currently my fork is here : https://github.com/bobzhang/omake-fork
>     It's in a re-factorization status .
>     The main things I can think about to improve omake(long term
> goal):
>      a. remove some unused struct in the omake language, add a doc
> string and reflection abilities
>      b. make debug easier, error messages easy to follow
>      c. add a new backend (for example to ninja) for faster
> performance
>      d. expose an API to allow people write rules in ocaml language
> itself.
> 
>      The short term goal is bug fix and keep it up to date with the
> latest ocaml compiler
> -- Hongbo
> On Fri, Sep 19, 2014 at 1:18 PM, Gerd Stolpmann
> <info@gerd-stolpmann.de> wrote:
>         Am Freitag, den 19.09.2014, 11:18 -0400 schrieb Yaron Minsky:
>         > We had a fair number of problems with omake
>         >
>         > - We've run into lots of performance problems on large
>         builds (2-3
>         >   million lines, many thousands of targets) was that omake
>         took a very
>         >   long time (a few minutes) to be restarted.
>         
>         Well, never got into that dimensions. The largest builds had
>         only around
>         250Kloc, and omake worked well for that size. I don't know
>         much about
>         the internals of omake, but I can imagine that certain things
>         are
>         recomputed too often, and then you run into performance
>         problems at a
>         certain size. But why can't this be nailed down?
>         
>         > - The build itself has limited parallelism because of
>         various
>         >   bottlenecks inside omake, like the fact that it computes
>         its md5sums
>         >   in a single process.
>         
>         Hmm, is that evident? Maybe switching to a faster hash
>         algorithm could
>         solve this? (E.g. just use RC4 and XOR up the output stream;
>         should be
>         ultimately fast.)
>         
>         > - The default rules didn't do a good job of clearing out
>         stale build
>         >   artifacts (important for getting reliable incremental
>         builds), and
>         >   we had to put quite a lot of painful engineering in place
>         to make
>         >   that work.  We needed to do similar work in Jenga to make
>         the same
>         >   thing happen, but it was a lot more fun writing that code
>         in OCaml!
>         
>         If you just stick to the built-in macros, incremental builds
>         are very
>         reliable. If you start writing your own rules, there is some
>         chance that
>         you overlook dependencies. But that's independent of which
>         build system
>         you use.
>         
>         > I am not convinced that putting more complicated
>         calculations into
>         > programs will work well.  I know of no system that does a
>         good job
>         > allowing you to deal with complex dependencies that are
>         discovered
>         > over the course of a build (which you get with staged
>         programming)
>         > that take this approach.  It seems that a more expressive,
>         monadic
>         > structure fits the bill, and hammering that into the round
>         peg of
>         > shell invocations won't work well, I suspect.
>         
>         After all, I think that it would be naive to think that one
>         size fits
>         all. When you have a large build like yours you probably have
>         quite
>         special requirements, and are willing to do without comfort
>         features
>         like a macro language.
>         
>         So: no one build system to rule them all. Nevertheless, I'm
>         very much
>         for improving omake.
>         
>         Gerd
>         
>         
>         > y
>         >
>         > On Fri, Sep 19, 2014 at 10:00 AM, Alain Frisch
>         <alain@frisch.fr> wrote:
>         > > On 09/19/2014 03:36 PM, Gerd Stolpmann wrote:
>         > >>
>         > >> Well, I run frequently into the difficulty that I need
>         some special
>         > >> omake function that would be trivial to develop in OCaml
>         (e.g.
>         > >> associating some data with some other data, filtering
>         things, doing
>         > >> string transformations), but for writing it in the omake
>         language I need
>         > >> some time for developing and testing. I have a quite
>         simple idea to
>         > >> improve this: Besides OMakefile there could also be an
>         OMakefile.ml, and
>         > >> you can define any helper functions there, and they would
>         be
>         > >> automatically callable from the OMakefile. I think this
>         is not really
>         > >> complicated to do - you'd need to build a custom omake
>         executable
>         > >> whenever OMakefile.ml changes, and need to scan the
>         OMakefile.ml
>         > >> interface for function signatures that match the form
>         that is callable,
>         > >> and you need to generate some glue code. (Later this idea
>         could be
>         > >> extended by allowing OCaml code to emit new rules, as
>         described in an
>         > >> earlier post.)
>         > >
>         > >
>         > > I can see some cases where it would indeed be more
>         comfortable to implement
>         > > build-system calculations in OCaml.  But couldn't most of
>         these cases be
>         > > implemented as external programs called by omake functions
>         and implemented
>         > > e.g. in OCaml?  This forces to pass explicitly all the
>         data required by the
>         > > calculation to those external programs, but how often is
>         this a problem?
>         > > With some convention,  the external program could even
>         return description of
>         > > new dependencies, to be interpreted by some omake code and
>         injected into the
>         > > actual graph. AFAICT, all this is already possible.
>         > >
>         > >
>         > >> I see what you mean. In a recent project I had to define
>         all variables
>         > >> with library names, findlib names, intra-project library
>         dependencies
>         > >> etc. in the global OMakefile, because they are needed in
>         potentially all
>         > >> sub-OMakefiles. That's of course not where these things
>         are best
>         > >> naturally defined.
>         > >
>         > >
>         > > A variant is to have e.g. a OPreOMakefile file in each
>         directory and arrange
>         > > so that the toplevel OMakefile includes all of them (with
>         a proper ordering)
>         > > without processing the rest of the project.  This way, you
>         only need to
>         > > "lift" the full list of directories, and actual data
>         definitions can be put
>         > > where they belong.
>         > >
>         > >> Maybe we should allow to switch to global context
>         anywhere? I think this
>         > >> is solvable.
>         > >
>         > >
>         > > I'm not sure this would easily fit in the current
>         functional semantics.
>         > >
>         > >> Could be something simple, like matching the wildcard
>         rules against the
>         > >> real files.
>         > >
>         > >
>         > > Reading the directory content should be quite cheap, and
>         then it is just
>         > > string processing, which should be even cheaper (if done
>         properly).  It
>         > > would be great to identify such hot spots; maybe some very
>         local tweaks to
>         > > algorithmics or data structures could improve performance
>         a lot.
>         > >
>         > >
>         > >
>         > > Alain
>         > >
>         > >
>         > > --
>         > > Caml-list mailing list.  Subscription management and
>         archives:
>         > > https://sympa.inria.fr/sympa/arc/caml-list
>         > > Beginner's list:
>         http://groups.yahoo.com/group/ocaml_beginners
>         > > Bug reports: http://caml.inria.fr/bin/caml-bugs
>         >
>         
>         
>         --
>         ------------------------------------------------------------
>         Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
>         My OCaml site:          http://www.camlcity.org
>         Contact details:        http://www.camlcity.org/contact.html
>         Company homepage:       http://www.gerd-stolpmann.de
>         ------------------------------------------------------------
>         
> 
> 
> 
> 
> -- 
> Regards
> -- Hongbo Zhang

-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
My OCaml site:          http://www.camlcity.org
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]