[Caml-list] improve omake [was One build system to rule them all]

[Caml-list] improve omake [was One build system to rule them all]

[Bob Zhang]

[-- Attachment #1: Type: text/plain, Size: 1622 bytes --]

Dear camlers,
   I have done some work to  improve omake available here:
https://github.com/bobzhang/omake-fork/tree/work
   Before deciding spending some time in improving omake, I have tried
various build systems.
  1. ocamlbuild
      ocamlbuild is really nice for small to medium projects and I have
used it pervasively in my personal projects and corporation projects. It
works pretty well in most cases.
     There are mainly three drawbacks:
      a. Easy things hard to do.
          Even for some very trivial things, if you don't write
myocamlbuild.m for a long time, you have to google ocamlbuild API and
figure it out how to do it correctly.
     b. Error messages hard to understand
         It's cool that ocamlbuild detect dependencies dynamically, when it
does not work out, in general, I would turn on -verbose and search which
part goes wrong.
     c. no parallellism
        This is fatal and main reason that I gave it up
   2. ocp-build
      I tried it for my hobby project, it's not close to maturity yet.
   3. jenga
      Jenga looks promising, but I don't think it would be usable inside
our company, the dependency is huge, more importantly, its dependency chain
includes Camlp4 which we can not rely on. Also, looking at the examples, it
is quite verbose even for trivial projects.

   omake has its own drawbacks as well, for example, the language is overly
complex and error message is hard to understand(still better than
ocamlbuild), startup speed is slow, no easy FFI interface to write rules in
OCaml language itself, but that's all we can find a way to fix.

-- 
Regards
-- Hongbo Zhang

[-- Attachment #2: Type: text/html, Size: 1998 bytes --]

Re: [Caml-list] improve omake [was One build system to rule them all]

[Aleksey Nogin]

All,

As you obviously know, OMake have not had a proper maintainer for a few
years now - while I did not completely abandon it, I did not have time
to devote to even little things (like pushing out a new 0.9.8.6 release
which I have been hoping to call "version 1.0", and which have been
lingering in "release candidate" mode for almost four years).

It is clear that there are quite a few people on this list with good
ideas on how to improve OMake (e.g. ability to write rules in OCaml
instead of/in addition to the OMake language seems like a good idea) -
so I am wondering - is there somebody who would be willing to take over
as the omake maintainer - ideally somebody whom people on this list
would trust with this role?

If there was some sort of consensus on this list about a new maintainer,
I would be happy to pass on this role (redirect omake.metaprl.org
accordingly, etc).

Aleksey


On 18.09.2014 13:14, Bob Zhang wrote:

> Dear camlers,
>    I have done some work to  improve omake available here:
>  https://github.com/bobzhang/omake-fork/tree/work
>    Before deciding spending some time in improving omake, I have tried
> various build systems.   
>   1. ocamlbuild
>       ocamlbuild is really nice for small to medium projects and I have
> used it pervasively in my personal projects and corporation projects. It
> works pretty well in most cases.
>      There are mainly three drawbacks:
>       a. Easy things hard to do.
>           Even for some very trivial things, if you don't write
> myocamlbuild.m for a long time, you have to google ocamlbuild API and
> figure it out how to do it correctly.
>      b. Error messages hard to understand
>          It's cool that ocamlbuild detect dependencies dynamically, when
> it does not work out, in general, I would turn on -verbose and search
> which part goes wrong.
>      c. no parallellism
>         This is fatal and main reason that I gave it up
>    2. ocp-build
>       I tried it for my hobby project, it's not close to maturity yet.
>    3. jenga
>       Jenga looks promising, but I don't think it would be usable inside
> our company, the dependency is huge, more importantly, its dependency
> chain includes Camlp4 which we can not rely on. Also, looking at the
> examples, it is quite verbose even for trivial projects.
> 
>    omake has its own drawbacks as well, for example, the language is
> overly complex and error message is hard to understand(still better than
> ocamlbuild), startup speed is slow, no easy FFI interface to write rules
> in OCaml language itself, but that's all we can find a way to fix. 
> 
> -- 
> Regards
> -- Hongbo Zhang

Re: [Caml-list] improve omake [was One build system to rule them all]

[Sébastien Dailly]

Le 18/09/2014 22:14, Bob Zhang a écrit :
> Dear camlers,
>     I have done some work to  improve omake available here:
> https://github.com/bobzhang/omake-fork/tree/work

Hello,

What did you changed with the legacy omake code ?

Regards

-- 
Sébastien Dailly

Re: [Caml-list] improve omake [was One build system to rule them all]

[Yaron Minsky]

This is not the main point of your post, but to reply to your concerns
on Jenga:

- I'm not sure why the dependency chain of Jenga matters much.  You
  need camlp4 to build it, but not to use it.  In any case, I expect
  that in a few months, that dependency will be gone as we migrate to
  extension points.

- The examples are big, but that's mostly because it takes a lot of
  code to precisely express OCaml's dependency and build rules.  Other
  tools either hide that code from you (ocamlbuild) or have incomplete
  rules that don't quite do the right thing (omake).

Note that with our Jenga rules, we do have simple build specification
files.  Here's the one for async_kernel, which emphasizes how simple
these can be.

    (library
     ((name async_kernel)
      (libraries (core))))

Here's the one for async_unix, which is a little more complicated.

    (library
     ((name async_unix)
      (libraries (async_kernel
                  core))
      (c_names (dump_core_on_job_delay_stubs))
      (preprocess (((pps (JANE pa_macro)) All)))))


And here's a much more complicated one, for core:

    (rule
     ((targets (config.mlh config.h))
      (deps (discover.sh))
      (action "${<} ${OCAML} ${OCAMLC} config.mlh config.h -DLINUX_EXT
-DPOSIX_TIMERS -DWORDEXP")))

    (rule
     ((targets (version_defaults.mlh))
      (deps ())
      (action "echo 'DEFINE DEFAULT_VERSION   = Version_util.version'
   > version_defaults.mlh
               echo 'DEFINE DEFAULT_BUILDINFO =
Version_util.build_info' >> version_defaults.mlh
               echo 'DEFINE BUILD_VERSION_UTIL'
  >> version_defaults.mlh")))

    (library
     ((name core)
      (libraries
       (core_kernel
        sexplib_unix
        version_util_fallback))
      (interfaces (core_kernel))
      (extra_cflags (-D_LARGEFILE64_SOURCE))
      ;; We omit [-pedantic] because
      ;; [bigstring_recvmmsg_assume_fd_is_nonblocking_stub] in
      ;; bigstring_stubs.c uses variable-length arrays rather than [alloca].
      (avoid_cflags (-pedantic))
      (library_flags (-cclib -lrt))
      (c_names
       (backtrace_stubs
        bigstring_stubs
        crc_stubs
        iobuf_stubs
        linux_ext_stubs
        ocaml_utils_stubs
        recvmmsg
        signal_stubs
        syslog_stubs
        time_stamp_counter_stubs
        time_stubs
        timespec
        unix_stubs))
      (preprocess (((pps (JANE pa_macro)) All)))
      (preprocessor_deps (version_defaults.mlh config.mlh))))

The point is, there is a simple DSL embedded that we use for
specifying individual projects, but it's _very_ simple, with more
complicated things done by extending the build rules, in OCaml.
Really, I think this is the right design.  Build DSLs are typically
horrible programming languages, and you don't want to do anything
complicated with them.

That said, it's not ready for prime time yet, mostly in that it needs
some portability work, and some work to get good default rules that
people can use externally from Jane Street, with a good story for
interacting with ocamlfind and the like.

But we'd love to have people trying it out and helping us work out
those use cases.  Indeed, we've already exported a version of our
build rules that are suitable for building with OCaml 4.02:

    https://github.com/janestreet/jenga/tree/master/examples/js-build-style

y

On Thu, Sep 18, 2014 at 4:14 PM, Bob Zhang <bobzhang1988@gmail.com> wrote:
> Dear camlers,
>    I have done some work to  improve omake available here:
> https://github.com/bobzhang/omake-fork/tree/work
>    Before deciding spending some time in improving omake, I have tried
> various build systems.
>   1. ocamlbuild
>       ocamlbuild is really nice for small to medium projects and I have used
> it pervasively in my personal projects and corporation projects. It works
> pretty well in most cases.
>      There are mainly three drawbacks:
>       a. Easy things hard to do.
>           Even for some very trivial things, if you don't write
> myocamlbuild.m for a long time, you have to google ocamlbuild API and figure
> it out how to do it correctly.
>      b. Error messages hard to understand
>          It's cool that ocamlbuild detect dependencies dynamically, when it
> does not work out, in general, I would turn on -verbose and search which
> part goes wrong.
>      c. no parallellism
>         This is fatal and main reason that I gave it up
>    2. ocp-build
>       I tried it for my hobby project, it's not close to maturity yet.
>    3. jenga
>       Jenga looks promising, but I don't think it would be usable inside our
> company, the dependency is huge, more importantly, its dependency chain
> includes Camlp4 which we can not rely on. Also, looking at the examples, it
> is quite verbose even for trivial projects.
>
>    omake has its own drawbacks as well, for example, the language is overly
> complex and error message is hard to understand(still better than
> ocamlbuild), startup speed is slow, no easy FFI interface to write rules in
> OCaml language itself, but that's all we can find a way to fix.
>
> --
> Regards
> -- Hongbo Zhang

Re: [Caml-list] improve omake [was One build system to rule them all]

[Alain Frisch]

Hi Hongbo,

It's great to hear that you'll put some energy into omake.  Despite some 
of its shortcomings, it's a great and mature tool, which is nice for 
both simple projects and large ones.  It certainly deserves some good 
treatment :-)   and it's actually the only build system developed around 
OCaml which put so much emphasis, right from the beginning, on good 
Windows support.

I'm not sure of the benefit of using OCaml to write custom rules (but 
why not).  The omake language could certainly be improved, both from a 
syntactic and from a semantic point of view.  (I think there was some 
project, in the latest development version, to introduce a syntax closer 
to programming languages, with un-prefixed variables and delimited 
string literals.)   Personally, I don't care too much about the syntax. 
  The most important problem I can see with the language is the 
difficulty to "pass" information from one part of the project to another 
one.  The only two ways I'm aware of to achieve that are:  (i) rely on 
the scoping rules, which in practice means a one-directional flow of 
data (typically from a toplevel OMakefile to OMakefile in 
sub-directories)  or (ii) piggy-back the more "imperative" dependency 
graph (attaching dependencies to dummy "tag" files can be used to 
implement Boolean markers than can be put on a target in one place and 
observed from another place).   A typical situation where information 
should flow from one part to another:  each library (in its own 
directory) exports some variables (such as some link flags), to be used 
by client parts.


Several people complained about the startup performance of omake on big 
projects.  It would be very useful to know whether this comes from the 
processing of the omake "scripts" (in which case some energy might be 
put into improving the interpreter and the internal data structures -- I 
don't see a deep reason for spending several seconds on interpreting 
even quite large scripts) or from scanning the file system for file 
changes (in which case nothing much could be done about it).


Alain



On 09/18/2014 10:14 PM, Bob Zhang wrote:
> Dear camlers,
>     I have done some work to  improve omake available here:
> https://github.com/bobzhang/omake-fork/tree/work
>     Before deciding spending some time in improving omake, I have tried
> various build systems.
>    1. ocamlbuild
>        ocamlbuild is really nice for small to medium projects and I have
> used it pervasively in my personal projects and corporation projects. It
> works pretty well in most cases.
>       There are mainly three drawbacks:
>        a. Easy things hard to do.
>            Even for some very trivial things, if you don't write
> myocamlbuild.m for a long time, you have to google ocamlbuild API and
> figure it out how to do it correctly.
>       b. Error messages hard to understand
>           It's cool that ocamlbuild detect dependencies dynamically,
> when it does not work out, in general, I would turn on -verbose and
> search which part goes wrong.
>       c. no parallellism
>          This is fatal and main reason that I gave it up
>     2. ocp-build
>        I tried it for my hobby project, it's not close to maturity yet.
>     3. jenga
>        Jenga looks promising, but I don't think it would be usable
> inside our company, the dependency is huge, more importantly, its
> dependency chain includes Camlp4 which we can not rely on. Also, looking
> at the examples, it is quite verbose even for trivial projects.
>
>     omake has its own drawbacks as well, for example, the language is
> overly complex and error message is hard to understand(still better than
> ocamlbuild), startup speed is slow, no easy FFI interface to write rules
> in OCaml language itself, but that's all we can find a way to fix.
>
> --
> Regards
> -- Hongbo Zhang

Re: [Caml-list] improve omake [was One build system to rule them all]

[Nicolas Boulay]

[-- Attachment #1: Type: text/plain, Size: 4423 bytes --]

Ocaml could be executed as shell code. Why create another langage and not
use an ocaml program (+ library) to build an other ocaml program ? The
first program could be interpreted as shell code.

2014-09-19 14:23 GMT+02:00 Alain Frisch <alain@frisch.fr>:

> Hi Hongbo,
>
> It's great to hear that you'll put some energy into omake.  Despite some
> of its shortcomings, it's a great and mature tool, which is nice for both
> simple projects and large ones.  It certainly deserves some good treatment
> :-)   and it's actually the only build system developed around OCaml which
> put so much emphasis, right from the beginning, on good Windows support.
>
> I'm not sure of the benefit of using OCaml to write custom rules (but why
> not).  The omake language could certainly be improved, both from a
> syntactic and from a semantic point of view.  (I think there was some
> project, in the latest development version, to introduce a syntax closer to
> programming languages, with un-prefixed variables and delimited string
> literals.)   Personally, I don't care too much about the syntax.  The most
> important problem I can see with the language is the difficulty to "pass"
> information from one part of the project to another one.  The only two ways
> I'm aware of to achieve that are:  (i) rely on the scoping rules, which in
> practice means a one-directional flow of data (typically from a toplevel
> OMakefile to OMakefile in sub-directories)  or (ii) piggy-back the more
> "imperative" dependency graph (attaching dependencies to dummy "tag" files
> can be used to implement Boolean markers than can be put on a target in one
> place and observed from another place).   A typical situation where
> information should flow from one part to another:  each library (in its own
> directory) exports some variables (such as some link flags), to be used by
> client parts.
>
>
> Several people complained about the startup performance of omake on big
> projects.  It would be very useful to know whether this comes from the
> processing of the omake "scripts" (in which case some energy might be put
> into improving the interpreter and the internal data structures -- I don't
> see a deep reason for spending several seconds on interpreting even quite
> large scripts) or from scanning the file system for file changes (in which
> case nothing much could be done about it).
>
>
> Alain
>
>
>
>
> On 09/18/2014 10:14 PM, Bob Zhang wrote:
>
>> Dear camlers,
>>     I have done some work to  improve omake available here:
>> https://github.com/bobzhang/omake-fork/tree/work
>>     Before deciding spending some time in improving omake, I have tried
>> various build systems.
>>    1. ocamlbuild
>>        ocamlbuild is really nice for small to medium projects and I have
>> used it pervasively in my personal projects and corporation projects. It
>> works pretty well in most cases.
>>       There are mainly three drawbacks:
>>        a. Easy things hard to do.
>>            Even for some very trivial things, if you don't write
>> myocamlbuild.m for a long time, you have to google ocamlbuild API and
>> figure it out how to do it correctly.
>>       b. Error messages hard to understand
>>           It's cool that ocamlbuild detect dependencies dynamically,
>> when it does not work out, in general, I would turn on -verbose and
>> search which part goes wrong.
>>       c. no parallellism
>>          This is fatal and main reason that I gave it up
>>     2. ocp-build
>>        I tried it for my hobby project, it's not close to maturity yet.
>>     3. jenga
>>        Jenga looks promising, but I don't think it would be usable
>> inside our company, the dependency is huge, more importantly, its
>> dependency chain includes Camlp4 which we can not rely on. Also, looking
>> at the examples, it is quite verbose even for trivial projects.
>>
>>     omake has its own drawbacks as well, for example, the language is
>> overly complex and error message is hard to understand(still better than
>> ocamlbuild), startup speed is slow, no easy FFI interface to write rules
>> in OCaml language itself, but that's all we can find a way to fix.
>>
>> --
>> Regards
>> -- Hongbo Zhang
>>
>
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>

[-- Attachment #2: Type: text/html, Size: 5568 bytes --]

Re: [Caml-list] improve omake [was One build system to rule them all]

[Gerd Stolpmann]

[-- Attachment #1: Type: text/plain, Size: 6693 bytes --]

Am Freitag, den 19.09.2014, 14:23 +0200 schrieb Alain Frisch:
> I'm not sure of the benefit of using OCaml to write custom rules (but 
> why not). 

Well, I run frequently into the difficulty that I need some special
omake function that would be trivial to develop in OCaml (e.g.
associating some data with some other data, filtering things, doing
string transformations), but for writing it in the omake language I need
some time for developing and testing. I have a quite simple idea to
improve this: Besides OMakefile there could also be an OMakefile.ml, and
you can define any helper functions there, and they would be
automatically callable from the OMakefile. I think this is not really
complicated to do - you'd need to build a custom omake executable
whenever OMakefile.ml changes, and need to scan the OMakefile.ml
interface for function signatures that match the form that is callable,
and you need to generate some glue code. (Later this idea could be
extended by allowing OCaml code to emit new rules, as described in an
earlier post.)

>  The omake language could certainly be improved, both from a 
> syntactic and from a semantic point of view.  (I think there was some 
> project, in the latest development version, to introduce a syntax closer 
> to programming languages, with un-prefixed variables and delimited 
> string literals.)

omake picks up ideas from shell programming, and this is a different
thinking than in programming languages. For example, if you define

files = x1 x2 x3

the variable is not set to a string, but to a sequence of words, so when
you later call

ls $(files)

the command is started with three arguments, and not one. If you define

files = "x1 x2" x3

this is a sequence with two words only (i.e. there are string
delimiters, only the outermost delimiters are normally omitted). Using a
special datatype for word sequences is a very nice idea, as it
eliminates the need for quoting (as in the shell). However, it is a
somewhat unusual thinking for most OCaml programmers, especially because
these sequences can get recursive (i.e. it's not a string list, but a
string tree, a bit like s-expressions in Lisp).

>    Personally, I don't care too much about the syntax. 
>   The most important problem I can see with the language is the 
> difficulty to "pass" information from one part of the project to another 
> one.  The only two ways I'm aware of to achieve that are:  (i) rely on 
> the scoping rules, which in practice means a one-directional flow of 
> data (typically from a toplevel OMakefile to OMakefile in 
> sub-directories)  or (ii) piggy-back the more "imperative" dependency 
> graph (attaching dependencies to dummy "tag" files can be used to 
> implement Boolean markers than can be put on a target in one place and 
> observed from another place).   A typical situation where information 
> should flow from one part to another:  each library (in its own 
> directory) exports some variables (such as some link flags), to be used 
> by client parts.

I see what you mean. In a recent project I had to define all variables
with library names, findlib names, intra-project library dependencies
etc. in the global OMakefile, because they are needed in potentially all
sub-OMakefiles. That's of course not where these things are best
naturally defined.

Maybe we should allow to switch to global context anywhere? I think this
is solvable.

> Several people complained about the startup performance of omake on big 
> projects.  It would be very useful to know whether this comes from the 
> processing of the omake "scripts" (in which case some energy might be 
> put into improving the interpreter and the internal data structures -- I 
> don't see a deep reason for spending several seconds on interpreting 
> even quite large scripts) or from scanning the file system for file 
> changes (in which case nothing much could be done about it).

Could be something simple, like matching the wildcard rules against the
real files.

Another wish I have: There could be a mode that explains why a certain
file is NOT rebuilt. More than once I forgot to create a manual
dependency for something, and some file was not rebuilt that should
have. It would be cool if I could get help from omake:

omake -why-not file

would list all potential build rules and why they aren't activated.

Gerd

> 
> Alain
> 
> 
> 
> On 09/18/2014 10:14 PM, Bob Zhang wrote:
> > Dear camlers,
> >     I have done some work to  improve omake available here:
> > https://github.com/bobzhang/omake-fork/tree/work
> >     Before deciding spending some time in improving omake, I have tried
> > various build systems.
> >    1. ocamlbuild
> >        ocamlbuild is really nice for small to medium projects and I have
> > used it pervasively in my personal projects and corporation projects. It
> > works pretty well in most cases.
> >       There are mainly three drawbacks:
> >        a. Easy things hard to do.
> >            Even for some very trivial things, if you don't write
> > myocamlbuild.m for a long time, you have to google ocamlbuild API and
> > figure it out how to do it correctly.
> >       b. Error messages hard to understand
> >           It's cool that ocamlbuild detect dependencies dynamically,
> > when it does not work out, in general, I would turn on -verbose and
> > search which part goes wrong.
> >       c. no parallellism
> >          This is fatal and main reason that I gave it up
> >     2. ocp-build
> >        I tried it for my hobby project, it's not close to maturity yet.
> >     3. jenga
> >        Jenga looks promising, but I don't think it would be usable
> > inside our company, the dependency is huge, more importantly, its
> > dependency chain includes Camlp4 which we can not rely on. Also, looking
> > at the examples, it is quite verbose even for trivial projects.
> >
> >     omake has its own drawbacks as well, for example, the language is
> > overly complex and error message is hard to understand(still better than
> > ocamlbuild), startup speed is slow, no easy FFI interface to write rules
> > in OCaml language itself, but that's all we can find a way to fix.
> >
> > --
> > Regards
> > -- Hongbo Zhang
> 
> 

-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
My OCaml site:          http://www.camlcity.org
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Caml-list] improve omake [was One build system to rule them all]

[Alain Frisch]

On 09/19/2014 03:36 PM, Gerd Stolpmann wrote:
> Well, I run frequently into the difficulty that I need some special
> omake function that would be trivial to develop in OCaml (e.g.
> associating some data with some other data, filtering things, doing
> string transformations), but for writing it in the omake language I need
> some time for developing and testing. I have a quite simple idea to
> improve this: Besides OMakefile there could also be an OMakefile.ml, and
> you can define any helper functions there, and they would be
> automatically callable from the OMakefile. I think this is not really
> complicated to do - you'd need to build a custom omake executable
> whenever OMakefile.ml changes, and need to scan the OMakefile.ml
> interface for function signatures that match the form that is callable,
> and you need to generate some glue code. (Later this idea could be
> extended by allowing OCaml code to emit new rules, as described in an
> earlier post.)

I can see some cases where it would indeed be more comfortable to 
implement build-system calculations in OCaml.  But couldn't most of 
these cases be implemented as external programs called by omake 
functions and implemented e.g. in OCaml?  This forces to pass explicitly 
all the data required by the calculation to those external programs, but 
how often is this a problem?  With some convention,  the external 
program could even return description of new dependencies, to be 
interpreted by some omake code and injected into the actual graph. 
AFAICT, all this is already possible.


> I see what you mean. In a recent project I had to define all variables
> with library names, findlib names, intra-project library dependencies
> etc. in the global OMakefile, because they are needed in potentially all
> sub-OMakefiles. That's of course not where these things are best
> naturally defined.

A variant is to have e.g. a OPreOMakefile file in each directory and 
arrange so that the toplevel OMakefile includes all of them (with a 
proper ordering) without processing the rest of the project.  This way, 
you only need to "lift" the full list of directories, and actual data 
definitions can be put where they belong.

> Maybe we should allow to switch to global context anywhere? I think this
> is solvable.

I'm not sure this would easily fit in the current functional semantics.

> Could be something simple, like matching the wildcard rules against the
> real files.

Reading the directory content should be quite cheap, and then it is just 
string processing, which should be even cheaper (if done properly).  It 
would be great to identify such hot spots; maybe some very local tweaks 
to algorithmics or data structures could improve performance a lot.



Alain

Re: [Caml-list] improve omake [was One build system to rule them all]

[Yaron Minsky]

We had a fair number of problems with omake

- We've run into lots of performance problems on large builds (2-3
  million lines, many thousands of targets) was that omake took a very
  long time (a few minutes) to be restarted.
- The build itself has limited parallelism because of various
  bottlenecks inside omake, like the fact that it computes its md5sums
  in a single process.
- The default rules didn't do a good job of clearing out stale build
  artifacts (important for getting reliable incremental builds), and
  we had to put quite a lot of painful engineering in place to make
  that work.  We needed to do similar work in Jenga to make the same
  thing happen, but it was a lot more fun writing that code in OCaml!

I am not convinced that putting more complicated calculations into
programs will work well.  I know of no system that does a good job
allowing you to deal with complex dependencies that are discovered
over the course of a build (which you get with staged programming)
that take this approach.  It seems that a more expressive, monadic
structure fits the bill, and hammering that into the round peg of
shell invocations won't work well, I suspect.

y

On Fri, Sep 19, 2014 at 10:00 AM, Alain Frisch <alain@frisch.fr> wrote:
> On 09/19/2014 03:36 PM, Gerd Stolpmann wrote:
>>
>> Well, I run frequently into the difficulty that I need some special
>> omake function that would be trivial to develop in OCaml (e.g.
>> associating some data with some other data, filtering things, doing
>> string transformations), but for writing it in the omake language I need
>> some time for developing and testing. I have a quite simple idea to
>> improve this: Besides OMakefile there could also be an OMakefile.ml, and
>> you can define any helper functions there, and they would be
>> automatically callable from the OMakefile. I think this is not really
>> complicated to do - you'd need to build a custom omake executable
>> whenever OMakefile.ml changes, and need to scan the OMakefile.ml
>> interface for function signatures that match the form that is callable,
>> and you need to generate some glue code. (Later this idea could be
>> extended by allowing OCaml code to emit new rules, as described in an
>> earlier post.)
>
>
> I can see some cases where it would indeed be more comfortable to implement
> build-system calculations in OCaml.  But couldn't most of these cases be
> implemented as external programs called by omake functions and implemented
> e.g. in OCaml?  This forces to pass explicitly all the data required by the
> calculation to those external programs, but how often is this a problem?
> With some convention,  the external program could even return description of
> new dependencies, to be interpreted by some omake code and injected into the
> actual graph. AFAICT, all this is already possible.
>
>
>> I see what you mean. In a recent project I had to define all variables
>> with library names, findlib names, intra-project library dependencies
>> etc. in the global OMakefile, because they are needed in potentially all
>> sub-OMakefiles. That's of course not where these things are best
>> naturally defined.
>
>
> A variant is to have e.g. a OPreOMakefile file in each directory and arrange
> so that the toplevel OMakefile includes all of them (with a proper ordering)
> without processing the rest of the project.  This way, you only need to
> "lift" the full list of directories, and actual data definitions can be put
> where they belong.
>
>> Maybe we should allow to switch to global context anywhere? I think this
>> is solvable.
>
>
> I'm not sure this would easily fit in the current functional semantics.
>
>> Could be something simple, like matching the wildcard rules against the
>> real files.
>
>
> Reading the directory content should be quite cheap, and then it is just
> string processing, which should be even cheaper (if done properly).  It
> would be great to identify such hot spots; maybe some very local tweaks to
> algorithmics or data structures could improve performance a lot.
>
>
>
> Alain
>
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs

Re: [Caml-list] improve omake [was One build system to rule them all]

[Gerd Stolpmann]

[-- Attachment #1: Type: text/plain, Size: 6074 bytes --]

Am Freitag, den 19.09.2014, 11:18 -0400 schrieb Yaron Minsky:
> We had a fair number of problems with omake
> 
> - We've run into lots of performance problems on large builds (2-3
>   million lines, many thousands of targets) was that omake took a very
>   long time (a few minutes) to be restarted.

Well, never got into that dimensions. The largest builds had only around
250Kloc, and omake worked well for that size. I don't know much about
the internals of omake, but I can imagine that certain things are
recomputed too often, and then you run into performance problems at a
certain size. But why can't this be nailed down?

> - The build itself has limited parallelism because of various
>   bottlenecks inside omake, like the fact that it computes its md5sums
>   in a single process.

Hmm, is that evident? Maybe switching to a faster hash algorithm could
solve this? (E.g. just use RC4 and XOR up the output stream; should be
ultimately fast.)

> - The default rules didn't do a good job of clearing out stale build
>   artifacts (important for getting reliable incremental builds), and
>   we had to put quite a lot of painful engineering in place to make
>   that work.  We needed to do similar work in Jenga to make the same
>   thing happen, but it was a lot more fun writing that code in OCaml!

If you just stick to the built-in macros, incremental builds are very
reliable. If you start writing your own rules, there is some chance that
you overlook dependencies. But that's independent of which build system
you use.

> I am not convinced that putting more complicated calculations into
> programs will work well.  I know of no system that does a good job
> allowing you to deal with complex dependencies that are discovered
> over the course of a build (which you get with staged programming)
> that take this approach.  It seems that a more expressive, monadic
> structure fits the bill, and hammering that into the round peg of
> shell invocations won't work well, I suspect.

After all, I think that it would be naive to think that one size fits
all. When you have a large build like yours you probably have quite
special requirements, and are willing to do without comfort features
like a macro language.

So: no one build system to rule them all. Nevertheless, I'm very much
for improving omake.

Gerd


> y
> 
> On Fri, Sep 19, 2014 at 10:00 AM, Alain Frisch <alain@frisch.fr> wrote:
> > On 09/19/2014 03:36 PM, Gerd Stolpmann wrote:
> >>
> >> Well, I run frequently into the difficulty that I need some special
> >> omake function that would be trivial to develop in OCaml (e.g.
> >> associating some data with some other data, filtering things, doing
> >> string transformations), but for writing it in the omake language I need
> >> some time for developing and testing. I have a quite simple idea to
> >> improve this: Besides OMakefile there could also be an OMakefile.ml, and
> >> you can define any helper functions there, and they would be
> >> automatically callable from the OMakefile. I think this is not really
> >> complicated to do - you'd need to build a custom omake executable
> >> whenever OMakefile.ml changes, and need to scan the OMakefile.ml
> >> interface for function signatures that match the form that is callable,
> >> and you need to generate some glue code. (Later this idea could be
> >> extended by allowing OCaml code to emit new rules, as described in an
> >> earlier post.)
> >
> >
> > I can see some cases where it would indeed be more comfortable to implement
> > build-system calculations in OCaml.  But couldn't most of these cases be
> > implemented as external programs called by omake functions and implemented
> > e.g. in OCaml?  This forces to pass explicitly all the data required by the
> > calculation to those external programs, but how often is this a problem?
> > With some convention,  the external program could even return description of
> > new dependencies, to be interpreted by some omake code and injected into the
> > actual graph. AFAICT, all this is already possible.
> >
> >
> >> I see what you mean. In a recent project I had to define all variables
> >> with library names, findlib names, intra-project library dependencies
> >> etc. in the global OMakefile, because they are needed in potentially all
> >> sub-OMakefiles. That's of course not where these things are best
> >> naturally defined.
> >
> >
> > A variant is to have e.g. a OPreOMakefile file in each directory and arrange
> > so that the toplevel OMakefile includes all of them (with a proper ordering)
> > without processing the rest of the project.  This way, you only need to
> > "lift" the full list of directories, and actual data definitions can be put
> > where they belong.
> >
> >> Maybe we should allow to switch to global context anywhere? I think this
> >> is solvable.
> >
> >
> > I'm not sure this would easily fit in the current functional semantics.
> >
> >> Could be something simple, like matching the wildcard rules against the
> >> real files.
> >
> >
> > Reading the directory content should be quite cheap, and then it is just
> > string processing, which should be even cheaper (if done properly).  It
> > would be great to identify such hot spots; maybe some very local tweaks to
> > algorithmics or data structures could improve performance a lot.
> >
> >
> >
> > Alain
> >
> >
> > --
> > Caml-list mailing list.  Subscription management and archives:
> > https://sympa.inria.fr/sympa/arc/caml-list
> > Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> > Bug reports: http://caml.inria.fr/bin/caml-bugs
> 

-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
My OCaml site:          http://www.camlcity.org
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Caml-list] improve omake [was One build system to rule them all]

[Yaron Minsky]

On Fri, Sep 19, 2014 at 1:18 PM, Gerd Stolpmann <info@gerd-stolpmann.de> wrote:
> Am Freitag, den 19.09.2014, 11:18 -0400 schrieb Yaron Minsky:
>> We had a fair number of problems with omake
>>
>> - We've run into lots of performance problems on large builds (2-3
>>   million lines, many thousands of targets) was that omake took a very
>>   long time (a few minutes) to be restarted.
>
> Well, never got into that dimensions. The largest builds had only around
> 250Kloc, and omake worked well for that size. I don't know much about
> the internals of omake, but I can imagine that certain things are
> recomputed too often, and then you run into performance problems at a
> certain size. But why can't this be nailed down?
>
>> - The build itself has limited parallelism because of various
>>   bottlenecks inside omake, like the fact that it computes its md5sums
>>   in a single process.
>
> Hmm, is that evident? Maybe switching to a faster hash algorithm could
> solve this? (E.g. just use RC4 and XOR up the output stream; should be
> ultimately fast.)

I guess.  It seems better to instead get rid of the single-threaded
bottleneck.

>> - The default rules didn't do a good job of clearing out stale build
>>   artifacts (important for getting reliable incremental builds), and
>>   we had to put quite a lot of painful engineering in place to make
>>   that work.  We needed to do similar work in Jenga to make the same
>>   thing happen, but it was a lot more fun writing that code in OCaml!
>
> If you just stick to the built-in macros, incremental builds are very
> reliable. If you start writing your own rules, there is some chance that
> you overlook dependencies. But that's independent of which build system
> you use.
>
>> I am not convinced that putting more complicated calculations into
>> programs will work well.  I know of no system that does a good job
>> allowing you to deal with complex dependencies that are discovered
>> over the course of a build (which you get with staged programming)
>> that take this approach.  It seems that a more expressive, monadic
>> structure fits the bill, and hammering that into the round peg of
>> shell invocations won't work well, I suspect.
>
> After all, I think that it would be naive to think that one size fits
> all. When you have a large build like yours you probably have quite
> special requirements, and are willing to do without comfort features
> like a macro language.

But there's no reason to settle for less.  You can have a powerful
core and a nice DSL for configuring your builds!  And we certainly
don't do without the comfort of easy to specify builds.  Quite the
opposite, we have a very simple, very easy-to-use way of expressing
our builds, that doesn't require dipping in to OCaml.  It's just built
on top of a set of core rules that are in OCaml.

And I don't think our requirements are particularly special.  We build
more code, at higher levels of parallelism, than any other OCaml user,
but that's most of it.  We build many of the same packages that
everyone else does, and I think most of the benefits of a system like
Jenga could be appreciated by other users who operate at somewhat
smaller scales.

y

> So: no one build system to rule them all. Nevertheless, I'm very much
> for improving omake.
>
> Gerd
>
>
>> y
>>
>> On Fri, Sep 19, 2014 at 10:00 AM, Alain Frisch <alain@frisch.fr> wrote:
>> > On 09/19/2014 03:36 PM, Gerd Stolpmann wrote:
>> >>
>> >> Well, I run frequently into the difficulty that I need some special
>> >> omake function that would be trivial to develop in OCaml (e.g.
>> >> associating some data with some other data, filtering things, doing
>> >> string transformations), but for writing it in the omake language I need
>> >> some time for developing and testing. I have a quite simple idea to
>> >> improve this: Besides OMakefile there could also be an OMakefile.ml, and
>> >> you can define any helper functions there, and they would be
>> >> automatically callable from the OMakefile. I think this is not really
>> >> complicated to do - you'd need to build a custom omake executable
>> >> whenever OMakefile.ml changes, and need to scan the OMakefile.ml
>> >> interface for function signatures that match the form that is callable,
>> >> and you need to generate some glue code. (Later this idea could be
>> >> extended by allowing OCaml code to emit new rules, as described in an
>> >> earlier post.)
>> >
>> >
>> > I can see some cases where it would indeed be more comfortable to implement
>> > build-system calculations in OCaml.  But couldn't most of these cases be
>> > implemented as external programs called by omake functions and implemented
>> > e.g. in OCaml?  This forces to pass explicitly all the data required by the
>> > calculation to those external programs, but how often is this a problem?
>> > With some convention,  the external program could even return description of
>> > new dependencies, to be interpreted by some omake code and injected into the
>> > actual graph. AFAICT, all this is already possible.
>> >
>> >
>> >> I see what you mean. In a recent project I had to define all variables
>> >> with library names, findlib names, intra-project library dependencies
>> >> etc. in the global OMakefile, because they are needed in potentially all
>> >> sub-OMakefiles. That's of course not where these things are best
>> >> naturally defined.
>> >
>> >
>> > A variant is to have e.g. a OPreOMakefile file in each directory and arrange
>> > so that the toplevel OMakefile includes all of them (with a proper ordering)
>> > without processing the rest of the project.  This way, you only need to
>> > "lift" the full list of directories, and actual data definitions can be put
>> > where they belong.
>> >
>> >> Maybe we should allow to switch to global context anywhere? I think this
>> >> is solvable.
>> >
>> >
>> > I'm not sure this would easily fit in the current functional semantics.
>> >
>> >> Could be something simple, like matching the wildcard rules against the
>> >> real files.
>> >
>> >
>> > Reading the directory content should be quite cheap, and then it is just
>> > string processing, which should be even cheaper (if done properly).  It
>> > would be great to identify such hot spots; maybe some very local tweaks to
>> > algorithmics or data structures could improve performance a lot.
>> >
>> >
>> >
>> > Alain
>> >
>> >
>> > --
>> > Caml-list mailing list.  Subscription management and archives:
>> > https://sympa.inria.fr/sympa/arc/caml-list
>> > Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>> > Bug reports: http://caml.inria.fr/bin/caml-bugs
>>
>
> --
> ------------------------------------------------------------
> Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
> My OCaml site:          http://www.camlcity.org
> Contact details:        http://www.camlcity.org/contact.html
> Company homepage:       http://www.gerd-stolpmann.de
> ------------------------------------------------------------

Re: [Caml-list] improve omake [was One build system to rule them all]

[Jun Furuse]

[-- Attachment #1: Type: text/plain, Size: 3252 bytes --]

Hi Aleksey,

Quickly grepped the opam-repository, I am the one who has released the most
packages require omake. I can be a maintainer, though I have no brilliant
idea to enhance it for now...

Jun

On Fri, Sep 19, 2014 at 4:30 AM, Aleksey Nogin <nogin@metaprl.org> wrote:

> All,
>
> As you obviously know, OMake have not had a proper maintainer for a few
> years now - while I did not completely abandon it, I did not have time
> to devote to even little things (like pushing out a new 0.9.8.6 release
> which I have been hoping to call "version 1.0", and which have been
> lingering in "release candidate" mode for almost four years).
>
> It is clear that there are quite a few people on this list with good
> ideas on how to improve OMake (e.g. ability to write rules in OCaml
> instead of/in addition to the OMake language seems like a good idea) -
> so I am wondering - is there somebody who would be willing to take over
> as the omake maintainer - ideally somebody whom people on this list
> would trust with this role?
>
> If there was some sort of consensus on this list about a new maintainer,
> I would be happy to pass on this role (redirect omake.metaprl.org
> accordingly, etc).
>
> Aleksey
>
>
> On 18.09.2014 13:14, Bob Zhang wrote:
>
> > Dear camlers,
> >    I have done some work to  improve omake available here:
> >  https://github.com/bobzhang/omake-fork/tree/work
> >    Before deciding spending some time in improving omake, I have tried
> > various build systems.
> >   1. ocamlbuild
> >       ocamlbuild is really nice for small to medium projects and I have
> > used it pervasively in my personal projects and corporation projects. It
> > works pretty well in most cases.
> >      There are mainly three drawbacks:
> >       a. Easy things hard to do.
> >           Even for some very trivial things, if you don't write
> > myocamlbuild.m for a long time, you have to google ocamlbuild API and
> > figure it out how to do it correctly.
> >      b. Error messages hard to understand
> >          It's cool that ocamlbuild detect dependencies dynamically, when
> > it does not work out, in general, I would turn on -verbose and search
> > which part goes wrong.
> >      c. no parallellism
> >         This is fatal and main reason that I gave it up
> >    2. ocp-build
> >       I tried it for my hobby project, it's not close to maturity yet.
> >    3. jenga
> >       Jenga looks promising, but I don't think it would be usable inside
> > our company, the dependency is huge, more importantly, its dependency
> > chain includes Camlp4 which we can not rely on. Also, looking at the
> > examples, it is quite verbose even for trivial projects.
> >
> >    omake has its own drawbacks as well, for example, the language is
> > overly complex and error message is hard to understand(still better than
> > ocamlbuild), startup speed is slow, no easy FFI interface to write rules
> > in OCaml language itself, but that's all we can find a way to fix.
> >
> > --
> > Regards
> > -- Hongbo Zhang
>
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>

[-- Attachment #2: Type: text/html, Size: 4405 bytes --]

Re: [Caml-list] improve omake [was One build system to rule them all]

[Alain Frisch]

On 09/19/2014 05:18 PM, Yaron Minsky wrote:
> - The build itself has limited parallelism because of various
>    bottlenecks inside omake, like the fact that it computes its md5sums
>    in a single process.

I think that omake doesn't usually recompute the md5 when the file 
hasn't changed (probably based on the mtime, maybe inode as well), so 
I'm surprised by the claim.  Did you actually confirm that the 
computation of md5 was one of the bottlenecks or is it just an hypothesis?

-- Alain

Re: [Caml-list] improve omake [was One build system to rule them all]

[Mark Shinwell]

On 23 September 2014 11:40, Alain Frisch <alain@frisch.fr> wrote:
> On 09/19/2014 05:18 PM, Yaron Minsky wrote:
>>
>> - The build itself has limited parallelism because of various
>>    bottlenecks inside omake, like the fact that it computes its md5sums
>>    in a single process.
>
> I think that omake doesn't usually recompute the md5 when the file hasn't
> changed (probably based on the mtime, maybe inode as well), so I'm surprised
> by the claim.  Did you actually confirm that the computation of md5 was one
> of the bottlenecks or is it just an hypothesis?

Without wishing to prolong a discussion about improving omake...

I think the details are hazy now, but as far as we recall: your
statement about avoiding recomputation is correct, but a lot of md5sum
work happened subsequent to each compilation action in serial, within
the omake process.

By no means was this the only problem; another, for example, was a
vast amount of time spent constructing rules for an entire tree even
if only a small portion was being demanded to be built.

(Of course, the worst problem with omake was probably the DSL, which
we found to be wholly unsuitable for large-scale reliable rule
engineering.  Programmability really is key.)

Mark

Re: [Caml-list] improve omake [was One build system to rule them all]

[Alain Frisch]

On 9/23/2014 12:58 PM, Mark Shinwell wrote:
> I think the details are hazy now, but as far as we recall: your
> statement about avoiding recomputation is correct, but a lot of md5sum
> work happened subsequent to each compilation action in serial, within
> the omake process.

Ok, this could affect total compilation built time, not startup (and 
total time when nothing needs to be rebuilt).

> By no means was this the only problem; another, for example, was a
> vast amount of time spent constructing rules for an entire tree even
> if only a small portion was being demanded to be built.

Ok.  Do you remember if you had some clue about the source of 
inefficiency (evaluator, data structure to represent the environment, etc)?

> (Of course, the worst problem with omake was probably the DSL, which
> we found to be wholly unsuitable for large-scale reliable rule
> engineering.  Programmability really is key.)

YMMV.  LexiFi's experience with the omake DSL is that even if sometime 
annoying, it's possible to achieve a reasonable effect with some 
efforts.  At least, the issues with the DSL did not compensate the other 
nice properties of omake.  I assume that other groups have functional 
build systems implemented with omake as well, so if people working on 
omake can improve its performances, it would be an immediate gain. 
That's not to say that we don't keep an eye open on alternative build 
systems such as Jenga (but switching an entire project to a different 
build system is a daunting task, of course).


Alain

Re: [Caml-list] improve omake [was One build system to rule them all]

[Yaron Minsky]

No joke.  The switch required a fair bit of cleverness on our part.
We actually had some clever intermediate stage when we were building
jenga build description files (full of s-expressions, not OCaml code)
from omake build rules, as a way of bridging the gap as we did the
conversion.  Flipping a big codebase is not for the faint of heart.

y

On Tue, Sep 23, 2014 at 4:12 PM, Alain Frisch <alain@frisch.fr> wrote:
> On 9/23/2014 12:58 PM, Mark Shinwell wrote:
>>
>> I think the details are hazy now, but as far as we recall: your
>> statement about avoiding recomputation is correct, but a lot of md5sum
>> work happened subsequent to each compilation action in serial, within
>> the omake process.
>
>
> Ok, this could affect total compilation built time, not startup (and total
> time when nothing needs to be rebuilt).
>
>> By no means was this the only problem; another, for example, was a
>> vast amount of time spent constructing rules for an entire tree even
>> if only a small portion was being demanded to be built.
>
>
> Ok.  Do you remember if you had some clue about the source of inefficiency
> (evaluator, data structure to represent the environment, etc)?
>
>> (Of course, the worst problem with omake was probably the DSL, which
>> we found to be wholly unsuitable for large-scale reliable rule
>> engineering.  Programmability really is key.)
>
>
> YMMV.  LexiFi's experience with the omake DSL is that even if sometime
> annoying, it's possible to achieve a reasonable effect with some efforts.
> At least, the issues with the DSL did not compensate the other nice
> properties of omake.  I assume that other groups have functional build
> systems implemented with omake as well, so if people working on omake can
> improve its performances, it would be an immediate gain. That's not to say
> that we don't keep an eye open on alternative build systems such as Jenga
> (but switching an entire project to a different build system is a daunting
> task, of course).
>
>
> Alain

Re: [Caml-list] improve omake [was One build system to rule them all]

[Alain Frisch]

LexiFi might be willing to provide some funding for improving omake, 
esp. its performances.  Please contact me if you (Gerd, Hongbo, anyone) 
want to discuss that.

-- Alain


On 09/24/2014 03:37 PM, Gerd Stolpmann wrote:
> Although I don't have much time now, I'll try to find some for helping
> here.
>
> Gerd
>
> Am Montag, den 22.09.2014, 11:33 -0400 schrieb Bob Zhang:
>> Hi all,
>>     I am glad there are some people who like omake. I have a full time
>> job right now, but I am still very interested in improving omake, it
>> would be nice that some more knowledgeable people would contribute(and
>> particular windows developer).
>>      Currently my fork is here : https://github.com/bobzhang/omake-fork
>>      It's in a re-factorization status .
>>      The main things I can think about to improve omake(long term
>> goal):
>>       a. remove some unused struct in the omake language, add a doc
>> string and reflection abilities
>>       b. make debug easier, error messages easy to follow
>>       c. add a new backend (for example to ninja) for faster
>> performance
>>       d. expose an API to allow people write rules in ocaml language
>> itself.
>>
>>       The short term goal is bug fix and keep it up to date with the
>> latest ocaml compiler
>> -- Hongbo
>> On Fri, Sep 19, 2014 at 1:18 PM, Gerd Stolpmann
>> <info@gerd-stolpmann.de> wrote:
>>          Am Freitag, den 19.09.2014, 11:18 -0400 schrieb Yaron Minsky:
>>          > We had a fair number of problems with omake
>>          >
>>          > - We've run into lots of performance problems on large
>>          builds (2-3
>>          >   million lines, many thousands of targets) was that omake
>>          took a very
>>          >   long time (a few minutes) to be restarted.
>>
>>          Well, never got into that dimensions. The largest builds had
>>          only around
>>          250Kloc, and omake worked well for that size. I don't know
>>          much about
>>          the internals of omake, but I can imagine that certain things
>>          are
>>          recomputed too often, and then you run into performance
>>          problems at a
>>          certain size. But why can't this be nailed down?
>>
>>          > - The build itself has limited parallelism because of
>>          various
>>          >   bottlenecks inside omake, like the fact that it computes
>>          its md5sums
>>          >   in a single process.
>>
>>          Hmm, is that evident? Maybe switching to a faster hash
>>          algorithm could
>>          solve this? (E.g. just use RC4 and XOR up the output stream;
>>          should be
>>          ultimately fast.)
>>
>>          > - The default rules didn't do a good job of clearing out
>>          stale build
>>          >   artifacts (important for getting reliable incremental
>>          builds), and
>>          >   we had to put quite a lot of painful engineering in place
>>          to make
>>          >   that work.  We needed to do similar work in Jenga to make
>>          the same
>>          >   thing happen, but it was a lot more fun writing that code
>>          in OCaml!
>>
>>          If you just stick to the built-in macros, incremental builds
>>          are very
>>          reliable. If you start writing your own rules, there is some
>>          chance that
>>          you overlook dependencies. But that's independent of which
>>          build system
>>          you use.
>>
>>          > I am not convinced that putting more complicated
>>          calculations into
>>          > programs will work well.  I know of no system that does a
>>          good job
>>          > allowing you to deal with complex dependencies that are
>>          discovered
>>          > over the course of a build (which you get with staged
>>          programming)
>>          > that take this approach.  It seems that a more expressive,
>>          monadic
>>          > structure fits the bill, and hammering that into the round
>>          peg of
>>          > shell invocations won't work well, I suspect.
>>
>>          After all, I think that it would be naive to think that one
>>          size fits
>>          all. When you have a large build like yours you probably have
>>          quite
>>          special requirements, and are willing to do without comfort
>>          features
>>          like a macro language.
>>
>>          So: no one build system to rule them all. Nevertheless, I'm
>>          very much
>>          for improving omake.
>>
>>          Gerd
>>
>>
>>          > y
>>          >
>>          > On Fri, Sep 19, 2014 at 10:00 AM, Alain Frisch
>>          <alain@frisch.fr> wrote:
>>          > > On 09/19/2014 03:36 PM, Gerd Stolpmann wrote:
>>          > >>
>>          > >> Well, I run frequently into the difficulty that I need
>>          some special
>>          > >> omake function that would be trivial to develop in OCaml
>>          (e.g.
>>          > >> associating some data with some other data, filtering
>>          things, doing
>>          > >> string transformations), but for writing it in the omake
>>          language I need
>>          > >> some time for developing and testing. I have a quite
>>          simple idea to
>>          > >> improve this: Besides OMakefile there could also be an
>>          OMakefile.ml, and
>>          > >> you can define any helper functions there, and they would
>>          be
>>          > >> automatically callable from the OMakefile. I think this
>>          is not really
>>          > >> complicated to do - you'd need to build a custom omake
>>          executable
>>          > >> whenever OMakefile.ml changes, and need to scan the
>>          OMakefile.ml
>>          > >> interface for function signatures that match the form
>>          that is callable,
>>          > >> and you need to generate some glue code. (Later this idea
>>          could be
>>          > >> extended by allowing OCaml code to emit new rules, as
>>          described in an
>>          > >> earlier post.)
>>          > >
>>          > >
>>          > > I can see some cases where it would indeed be more
>>          comfortable to implement
>>          > > build-system calculations in OCaml.  But couldn't most of
>>          these cases be
>>          > > implemented as external programs called by omake functions
>>          and implemented
>>          > > e.g. in OCaml?  This forces to pass explicitly all the
>>          data required by the
>>          > > calculation to those external programs, but how often is
>>          this a problem?
>>          > > With some convention,  the external program could even
>>          return description of
>>          > > new dependencies, to be interpreted by some omake code and
>>          injected into the
>>          > > actual graph. AFAICT, all this is already possible.
>>          > >
>>          > >
>>          > >> I see what you mean. In a recent project I had to define
>>          all variables
>>          > >> with library names, findlib names, intra-project library
>>          dependencies
>>          > >> etc. in the global OMakefile, because they are needed in
>>          potentially all
>>          > >> sub-OMakefiles. That's of course not where these things
>>          are best
>>          > >> naturally defined.
>>          > >
>>          > >
>>          > > A variant is to have e.g. a OPreOMakefile file in each
>>          directory and arrange
>>          > > so that the toplevel OMakefile includes all of them (with
>>          a proper ordering)
>>          > > without processing the rest of the project.  This way, you
>>          only need to
>>          > > "lift" the full list of directories, and actual data
>>          definitions can be put
>>          > > where they belong.
>>          > >
>>          > >> Maybe we should allow to switch to global context
>>          anywhere? I think this
>>          > >> is solvable.
>>          > >
>>          > >
>>          > > I'm not sure this would easily fit in the current
>>          functional semantics.
>>          > >
>>          > >> Could be something simple, like matching the wildcard
>>          rules against the
>>          > >> real files.
>>          > >
>>          > >
>>          > > Reading the directory content should be quite cheap, and
>>          then it is just
>>          > > string processing, which should be even cheaper (if done
>>          properly).  It
>>          > > would be great to identify such hot spots; maybe some very
>>          local tweaks to
>>          > > algorithmics or data structures could improve performance
>>          a lot.
>>          > >
>>          > >
>>          > >
>>          > > Alain
>>          > >
>>          > >
>>          > > --
>>          > > Caml-list mailing list.  Subscription management and
>>          archives:
>>          > > https://sympa.inria.fr/sympa/arc/caml-list
>>          > > Beginner's list:
>>          http://groups.yahoo.com/group/ocaml_beginners
>>          > > Bug reports: http://caml.inria.fr/bin/caml-bugs
>>          >
>>
>>
>>          --
>>          ------------------------------------------------------------
>>          Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
>>          My OCaml site:          http://www.camlcity.org
>>          Contact details:        http://www.camlcity.org/contact.html
>>          Company homepage:       http://www.gerd-stolpmann.de
>>          ------------------------------------------------------------
>>
>>
>>
>>
>>
>> --
>> Regards
>> -- Hongbo Zhang
>

Re: [Caml-list] improve omake [was One build system to rule them all]

[Gerd Stolpmann]

[-- Attachment #1: Type: text/plain, Size: 9652 bytes --]

Although I don't have much time now, I'll try to find some for helping
here.

Gerd

Am Montag, den 22.09.2014, 11:33 -0400 schrieb Bob Zhang:
> Hi all,
>    I am glad there are some people who like omake. I have a full time
> job right now, but I am still very interested in improving omake, it
> would be nice that some more knowledgeable people would contribute(and
> particular windows developer).
>     Currently my fork is here : https://github.com/bobzhang/omake-fork
>     It's in a re-factorization status .
>     The main things I can think about to improve omake(long term
> goal):
>      a. remove some unused struct in the omake language, add a doc
> string and reflection abilities
>      b. make debug easier, error messages easy to follow
>      c. add a new backend (for example to ninja) for faster
> performance
>      d. expose an API to allow people write rules in ocaml language
> itself.
> 
>      The short term goal is bug fix and keep it up to date with the
> latest ocaml compiler
> -- Hongbo
> On Fri, Sep 19, 2014 at 1:18 PM, Gerd Stolpmann
> <info@gerd-stolpmann.de> wrote:
>         Am Freitag, den 19.09.2014, 11:18 -0400 schrieb Yaron Minsky:
>         > We had a fair number of problems with omake
>         >
>         > - We've run into lots of performance problems on large
>         builds (2-3
>         >   million lines, many thousands of targets) was that omake
>         took a very
>         >   long time (a few minutes) to be restarted.
>         
>         Well, never got into that dimensions. The largest builds had
>         only around
>         250Kloc, and omake worked well for that size. I don't know
>         much about
>         the internals of omake, but I can imagine that certain things
>         are
>         recomputed too often, and then you run into performance
>         problems at a
>         certain size. But why can't this be nailed down?
>         
>         > - The build itself has limited parallelism because of
>         various
>         >   bottlenecks inside omake, like the fact that it computes
>         its md5sums
>         >   in a single process.
>         
>         Hmm, is that evident? Maybe switching to a faster hash
>         algorithm could
>         solve this? (E.g. just use RC4 and XOR up the output stream;
>         should be
>         ultimately fast.)
>         
>         > - The default rules didn't do a good job of clearing out
>         stale build
>         >   artifacts (important for getting reliable incremental
>         builds), and
>         >   we had to put quite a lot of painful engineering in place
>         to make
>         >   that work.  We needed to do similar work in Jenga to make
>         the same
>         >   thing happen, but it was a lot more fun writing that code
>         in OCaml!
>         
>         If you just stick to the built-in macros, incremental builds
>         are very
>         reliable. If you start writing your own rules, there is some
>         chance that
>         you overlook dependencies. But that's independent of which
>         build system
>         you use.
>         
>         > I am not convinced that putting more complicated
>         calculations into
>         > programs will work well.  I know of no system that does a
>         good job
>         > allowing you to deal with complex dependencies that are
>         discovered
>         > over the course of a build (which you get with staged
>         programming)
>         > that take this approach.  It seems that a more expressive,
>         monadic
>         > structure fits the bill, and hammering that into the round
>         peg of
>         > shell invocations won't work well, I suspect.
>         
>         After all, I think that it would be naive to think that one
>         size fits
>         all. When you have a large build like yours you probably have
>         quite
>         special requirements, and are willing to do without comfort
>         features
>         like a macro language.
>         
>         So: no one build system to rule them all. Nevertheless, I'm
>         very much
>         for improving omake.
>         
>         Gerd
>         
>         
>         > y
>         >
>         > On Fri, Sep 19, 2014 at 10:00 AM, Alain Frisch
>         <alain@frisch.fr> wrote:
>         > > On 09/19/2014 03:36 PM, Gerd Stolpmann wrote:
>         > >>
>         > >> Well, I run frequently into the difficulty that I need
>         some special
>         > >> omake function that would be trivial to develop in OCaml
>         (e.g.
>         > >> associating some data with some other data, filtering
>         things, doing
>         > >> string transformations), but for writing it in the omake
>         language I need
>         > >> some time for developing and testing. I have a quite
>         simple idea to
>         > >> improve this: Besides OMakefile there could also be an
>         OMakefile.ml, and
>         > >> you can define any helper functions there, and they would
>         be
>         > >> automatically callable from the OMakefile. I think this
>         is not really
>         > >> complicated to do - you'd need to build a custom omake
>         executable
>         > >> whenever OMakefile.ml changes, and need to scan the
>         OMakefile.ml
>         > >> interface for function signatures that match the form
>         that is callable,
>         > >> and you need to generate some glue code. (Later this idea
>         could be
>         > >> extended by allowing OCaml code to emit new rules, as
>         described in an
>         > >> earlier post.)
>         > >
>         > >
>         > > I can see some cases where it would indeed be more
>         comfortable to implement
>         > > build-system calculations in OCaml.  But couldn't most of
>         these cases be
>         > > implemented as external programs called by omake functions
>         and implemented
>         > > e.g. in OCaml?  This forces to pass explicitly all the
>         data required by the
>         > > calculation to those external programs, but how often is
>         this a problem?
>         > > With some convention,  the external program could even
>         return description of
>         > > new dependencies, to be interpreted by some omake code and
>         injected into the
>         > > actual graph. AFAICT, all this is already possible.
>         > >
>         > >
>         > >> I see what you mean. In a recent project I had to define
>         all variables
>         > >> with library names, findlib names, intra-project library
>         dependencies
>         > >> etc. in the global OMakefile, because they are needed in
>         potentially all
>         > >> sub-OMakefiles. That's of course not where these things
>         are best
>         > >> naturally defined.
>         > >
>         > >
>         > > A variant is to have e.g. a OPreOMakefile file in each
>         directory and arrange
>         > > so that the toplevel OMakefile includes all of them (with
>         a proper ordering)
>         > > without processing the rest of the project.  This way, you
>         only need to
>         > > "lift" the full list of directories, and actual data
>         definitions can be put
>         > > where they belong.
>         > >
>         > >> Maybe we should allow to switch to global context
>         anywhere? I think this
>         > >> is solvable.
>         > >
>         > >
>         > > I'm not sure this would easily fit in the current
>         functional semantics.
>         > >
>         > >> Could be something simple, like matching the wildcard
>         rules against the
>         > >> real files.
>         > >
>         > >
>         > > Reading the directory content should be quite cheap, and
>         then it is just
>         > > string processing, which should be even cheaper (if done
>         properly).  It
>         > > would be great to identify such hot spots; maybe some very
>         local tweaks to
>         > > algorithmics or data structures could improve performance
>         a lot.
>         > >
>         > >
>         > >
>         > > Alain
>         > >
>         > >
>         > > --
>         > > Caml-list mailing list.  Subscription management and
>         archives:
>         > > https://sympa.inria.fr/sympa/arc/caml-list
>         > > Beginner's list:
>         http://groups.yahoo.com/group/ocaml_beginners
>         > > Bug reports: http://caml.inria.fr/bin/caml-bugs
>         >
>         
>         
>         --
>         ------------------------------------------------------------
>         Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
>         My OCaml site:          http://www.camlcity.org
>         Contact details:        http://www.camlcity.org/contact.html
>         Company homepage:       http://www.gerd-stolpmann.de
>         ------------------------------------------------------------
>         
> 
> 
> 
> 
> -- 
> Regards
> -- Hongbo Zhang

-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
My OCaml site:          http://www.camlcity.org
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Caml-list] improve omake [was One build system to rule them all]

[Bob Zhang]

[-- Attachment #1: Type: text/plain, Size: 7174 bytes --]

Hi all,
   I am glad there are some people who like omake. I have a full time job
right now, but I am still very interested in improving omake, it would be
nice that some more knowledgeable people would contribute(and particular
windows developer).
    Currently my fork is here : https://github.com/bobzhang/omake-fork
    It's in a re-factorization status .
    The main things I can think about to improve omake(long term goal):
     a. remove some unused struct in the omake language, add a doc string
and reflection abilities
     b. make debug easier, error messages easy to follow
     c. add a new backend (for example to ninja) for faster performance
     d. expose an API to allow people write rules in ocaml language itself.

     The short term goal is bug fix and keep it up to date with the latest
ocaml compiler
-- Hongbo
On Fri, Sep 19, 2014 at 1:18 PM, Gerd Stolpmann <info@gerd-stolpmann.de>
wrote:

> Am Freitag, den 19.09.2014, 11:18 -0400 schrieb Yaron Minsky:
> > We had a fair number of problems with omake
> >
> > - We've run into lots of performance problems on large builds (2-3
> >   million lines, many thousands of targets) was that omake took a very
> >   long time (a few minutes) to be restarted.
>
> Well, never got into that dimensions. The largest builds had only around
> 250Kloc, and omake worked well for that size. I don't know much about
> the internals of omake, but I can imagine that certain things are
> recomputed too often, and then you run into performance problems at a
> certain size. But why can't this be nailed down?
>
> > - The build itself has limited parallelism because of various
> >   bottlenecks inside omake, like the fact that it computes its md5sums
> >   in a single process.
>
> Hmm, is that evident? Maybe switching to a faster hash algorithm could
> solve this? (E.g. just use RC4 and XOR up the output stream; should be
> ultimately fast.)
>
> > - The default rules didn't do a good job of clearing out stale build
> >   artifacts (important for getting reliable incremental builds), and
> >   we had to put quite a lot of painful engineering in place to make
> >   that work.  We needed to do similar work in Jenga to make the same
> >   thing happen, but it was a lot more fun writing that code in OCaml!
>
> If you just stick to the built-in macros, incremental builds are very
> reliable. If you start writing your own rules, there is some chance that
> you overlook dependencies. But that's independent of which build system
> you use.
>
> > I am not convinced that putting more complicated calculations into
> > programs will work well.  I know of no system that does a good job
> > allowing you to deal with complex dependencies that are discovered
> > over the course of a build (which you get with staged programming)
> > that take this approach.  It seems that a more expressive, monadic
> > structure fits the bill, and hammering that into the round peg of
> > shell invocations won't work well, I suspect.
>
> After all, I think that it would be naive to think that one size fits
> all. When you have a large build like yours you probably have quite
> special requirements, and are willing to do without comfort features
> like a macro language.
>
> So: no one build system to rule them all. Nevertheless, I'm very much
> for improving omake.
>
> Gerd
>
>
> > y
> >
> > On Fri, Sep 19, 2014 at 10:00 AM, Alain Frisch <alain@frisch.fr> wrote:
> > > On 09/19/2014 03:36 PM, Gerd Stolpmann wrote:
> > >>
> > >> Well, I run frequently into the difficulty that I need some special
> > >> omake function that would be trivial to develop in OCaml (e.g.
> > >> associating some data with some other data, filtering things, doing
> > >> string transformations), but for writing it in the omake language I
> need
> > >> some time for developing and testing. I have a quite simple idea to
> > >> improve this: Besides OMakefile there could also be an OMakefile.ml,
> and
> > >> you can define any helper functions there, and they would be
> > >> automatically callable from the OMakefile. I think this is not really
> > >> complicated to do - you'd need to build a custom omake executable
> > >> whenever OMakefile.ml changes, and need to scan the OMakefile.ml
> > >> interface for function signatures that match the form that is
> callable,
> > >> and you need to generate some glue code. (Later this idea could be
> > >> extended by allowing OCaml code to emit new rules, as described in an
> > >> earlier post.)
> > >
> > >
> > > I can see some cases where it would indeed be more comfortable to
> implement
> > > build-system calculations in OCaml.  But couldn't most of these cases
> be
> > > implemented as external programs called by omake functions and
> implemented
> > > e.g. in OCaml?  This forces to pass explicitly all the data required
> by the
> > > calculation to those external programs, but how often is this a
> problem?
> > > With some convention,  the external program could even return
> description of
> > > new dependencies, to be interpreted by some omake code and injected
> into the
> > > actual graph. AFAICT, all this is already possible.
> > >
> > >
> > >> I see what you mean. In a recent project I had to define all variables
> > >> with library names, findlib names, intra-project library dependencies
> > >> etc. in the global OMakefile, because they are needed in potentially
> all
> > >> sub-OMakefiles. That's of course not where these things are best
> > >> naturally defined.
> > >
> > >
> > > A variant is to have e.g. a OPreOMakefile file in each directory and
> arrange
> > > so that the toplevel OMakefile includes all of them (with a proper
> ordering)
> > > without processing the rest of the project.  This way, you only need to
> > > "lift" the full list of directories, and actual data definitions can
> be put
> > > where they belong.
> > >
> > >> Maybe we should allow to switch to global context anywhere? I think
> this
> > >> is solvable.
> > >
> > >
> > > I'm not sure this would easily fit in the current functional semantics.
> > >
> > >> Could be something simple, like matching the wildcard rules against
> the
> > >> real files.
> > >
> > >
> > > Reading the directory content should be quite cheap, and then it is
> just
> > > string processing, which should be even cheaper (if done properly).  It
> > > would be great to identify such hot spots; maybe some very local
> tweaks to
> > > algorithmics or data structures could improve performance a lot.
> > >
> > >
> > >
> > > Alain
> > >
> > >
> > > --
> > > Caml-list mailing list.  Subscription management and archives:
> > > https://sympa.inria.fr/sympa/arc/caml-list
> > > Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> > > Bug reports: http://caml.inria.fr/bin/caml-bugs
> >
>
> --
> ------------------------------------------------------------
> Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
> My OCaml site:          http://www.camlcity.org
> Contact details:        http://www.camlcity.org/contact.html
> Company homepage:       http://www.gerd-stolpmann.de
> ------------------------------------------------------------
>



-- 
Regards
-- Hongbo Zhang

[-- Attachment #2: Type: text/html, Size: 9392 bytes --]