[Caml-list] One build system to rule them all?

[Caml-list] One build system to rule them all?

[Yotam Barnoy]

[-- Attachment #1: Type: text/plain, Size: 1123 bytes --]

It appears to me that every couple of months we hear of someone
implementing yet another build system in ocaml. Given the success of opam,
I think it's clear that sometimes a monolithic solution, behind which the
entire community can organize, is the best solution -- especially for
infrastructure. Looking at haskell, having cabal as the main build system
has really helped them advance in terms of supporting other platforms (such
as windows), and since all community efforts in this realm are focused on
cabal, they can improve it rapidly.

a. Is there any build system we can organize behind to crown as the
official build system?
b. What are the use-cases missing from specific build systems, that have
driven people to use other build systems?
c. To pick one possible candidate, if ocamlbuild were spun out of the
compiler, could it be enhanced to cover all the main use-cases so (almost)
everyone would be happy with it?

It's just such a shame to see the ocaml community re-inventing the wheel
over and over again, each time with some limitation so that the next person
needs to do the same thing yet again.

Yotam

[-- Attachment #2: Type: text/html, Size: 1262 bytes --]

Re: [Caml-list] One build system to rule them all?

[Simon Cruanes]

[-- Attachment #1: Type: text/plain, Size: 2143 bytes --]

I'd suggest oasis, based on my limited experience (I don't compile C
bindings nor to js_of_ocaml, but I write libraries, sub-libraries, and
programs that use ocamlfind).

Le Wed, 10 Sep 2014, Yotam Barnoy a écrit :
> It appears to me that every couple of months we hear of someone
> implementing yet another build system in ocaml. Given the success of opam,
> I think it's clear that sometimes a monolithic solution, behind which the
> entire community can organize, is the best solution -- especially for
> infrastructure. Looking at haskell, having cabal as the main build system
> has really helped them advance in terms of supporting other platforms (such
> as windows), and since all community efforts in this realm are focused on
> cabal, they can improve it rapidly.
> 
> a. Is there any build system we can organize behind to crown as the
> official build system?

I like oasis a lot, especially the fact that every information related
to a project is written down in a single file. It can generate
documentation, libraries, executables, ocamlfind-related files (META...),
configure/Makefile, and run tests. Even if oasis isn't blessed as the
canonical build system, its project description should be a good model
for the winner.

> b. What are the use-cases missing from specific build systems, that have
> driven people to use other build systems?
> c. To pick one possible candidate, if ocamlbuild were spun out of the
> compiler, could it be enhanced to cover all the main use-cases so (almost)
> everyone would be happy with it?

In the case of oasis, it already relies on ocamlbuild, so both could be
merged somehow (and the redundancies, hopefully, removed)...

> It's just such a shame to see the ocaml community re-inventing the wheel
> over and over again, each time with some limitation so that the next person
> needs to do the same thing yet again.

My 2 ¢

-- 
Simon

full disclosure: I don't have any interest in promoting oasis, nor have
I been paid (sadly) to do so ;)

http://weusepgp.info/
key 49AA62B6, fingerprint 949F EB87 8F06 59C6 D7D3  7D8D 4AC0 1D08 49AA 62B6

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [Caml-list] One build system to rule them all?

[Adrien Nader]

On Wed, Sep 10, 2014, Yotam Barnoy wrote:
> It appears to me that every couple of months we hear of someone
> implementing yet another build system in ocaml. Given the success of opam,
> I think it's clear that sometimes a monolithic solution, behind which the
> entire community can organize, is the best solution -- especially for
> infrastructure. Looking at haskell, having cabal as the main build system
> has really helped them advance in terms of supporting other platforms (such
> as windows), and since all community efforts in this realm are focused on
> cabal, they can improve it rapidly.
> 
> a. Is there any build system we can organize behind to crown as the
> official build system?

It's a bit like asking if there could be a single programming language
that would fit all the needs of everyone at the same time. I'm not sure
you'd spend a lot of time chosing Coq, OCaml, shell scripts and asm for
a given project.

> b. What are the use-cases missing from specific build systems, that have
> driven people to use other build systems?

I won't answer exactly on this but considering that building a .ml
requires that its dependencies have already been built and are
reachable. You can mix C in too. Some build systems automatically
discover dependencies. You can produce .ml files from .mll/.mly ones.
You can add camlp4 or another preprocess to the mix. You might want
support for linking against C libraries, and also discovering them.
And that list is still incomplete.

> c. To pick one possible candidate, if ocamlbuild were spun out of the
> compiler, could it be enhanced to cover all the main use-cases so (almost)
> everyone would be happy with it?

Even then, not everyone would be happy about it. Some people don't like
some of the approaches used in ocamlbuild.

That said, and for unrelated reasons, I'd like to know if ocamlbuild is
going to be split or not (with 4.02 getting out and ICFP and other confs
around, I'm not very surprised there hasn't been an answer on that yet).

-- 
Adrien Nader

Re: [Caml-list] One build system to rule them all?

[David Sheets]

On Wed, Sep 10, 2014 at 1:49 PM, Yotam Barnoy <yotambarnoy@gmail.com> wrote:
> It appears to me that every couple of months we hear of someone implementing
> yet another build system in ocaml. Given the success of opam, I think it's
> clear that sometimes a monolithic solution, behind which the entire
> community can organize, is the best solution -- especially for
> infrastructure.

opam is not a monolithic solution. For instance, it does not try to
mandate a build system and it uses external solvers if available.

> Looking at haskell, having cabal as the main build system
> has really helped them advance in terms of supporting other platforms (such
> as windows), and since all community efforts in this realm are focused on
> cabal, they can improve it rapidly.

Like all system design decisions, this is a trade-off. Cabal does
build and packaging. Cabal has some issues and limitations that may
not be fixable without major reworking.

> a. Is there any build system we can organize behind to crown as the official
> build system?

What is "official"? People get hit by buses, governments fall,
requirements change. I don't want an "official" build system.

> b. What are the use-cases missing from specific build systems, that have
> driven people to use other build systems?

Code-base size, dependency management, foreign-function binding, code
generation and staging, transparency, documentation, speed...

> c. To pick one possible candidate, if ocamlbuild were spun out of the
> compiler, could it be enhanced to cover all the main use-cases so (almost)
> everyone would be happy with it?

Probably not. One size does not fit all.

> It's just such a shame to see the ocaml community re-inventing the wheel
> over and over again, each time with some limitation so that the next person
> needs to do the same thing yet again.

A build system is considerably more complicated than a wheel. In many
cases, the issues responsible for re-work stem from design and social
factors, not technical problems. It's not clear to me that *any*
language community has actually completely solved the build system
problem. Writing and promoting a new build system which solves old
problems and does not regress the status quo seems like a laudable, if
thankless, goal.

If you'd like to contribute to a build system project to help it cover
more use cases and become The One True Build System, I suggest getting
involved with the project you feel has the best design, project
management, and philosophy and contributing your time and patches.

Best regards,

David

Re: [Caml-list] One build system to rule them all?

[Mark Shinwell]

On 10 September 2014 13:49, Yotam Barnoy <yotambarnoy@gmail.com> wrote:
> a. Is there any build system we can organize behind to crown as the official
> build system?

It may be a worthy goal, but I think it's unlikely that we will
achieve complete consensus on this front.  (Also, some of the concerns
that might be attributed to "build systems" might actually better be
the responsibility of other entities, for example package management
systems.)

> b. What are the use-cases missing from specific build systems, that have
> driven people to use other build systems?
> c. To pick one possible candidate, if ocamlbuild were spun out of the
> compiler, could it be enhanced to cover all the main use-cases so (almost)
> everyone would be happy with it?

Jane Street developed jenga, a general-purpose build tool, to address
the fact that we did not know of any other tool with suitable
properties of scalability, programmability, and efficiency.  The
target use case is very large code bases (millions of lines) with rule
sets to encode all of the complexities Adrien just identified in his
email, and more.

I'm not going to make claims as to whether jenga is superior within
the wider community, although we are committed to trying to make it
work there as well as possible.

Mark

Re: [Caml-list] One build system to rule them all?

[Francois Berenger]

On 09/10/2014 02:49 PM, Yotam Barnoy wrote:
> It appears to me that every couple of months we hear of someone
> implementing yet another build system in ocaml. Given the success of
> opam, I think it's clear that sometimes a monolithic solution, behind
> which the entire community can organize, is the best solution --
> especially for infrastructure. Looking at haskell, having cabal as the
> main build system has really helped them advance in terms of supporting
> other platforms (such as windows), and since all community efforts in
> this realm are focused on cabal, they can improve it rapidly.
>
> a. Is there any build system we can organize behind to crown as the
> official build system?
> b. What are the use-cases missing from specific build systems, that have
> driven people to use other build systems?
> c. To pick one possible candidate, if ocamlbuild were spun out of the
> compiler, could it be enhanced to cover all the main use-cases so
> (almost) everyone would be happy with it?
>
> It's just such a shame to see the ocaml community re-inventing the wheel
> over and over again, each time with some limitation so that the next
> person needs to do the same thing yet again.

It is not a shame, it is a tradition in the ocaml community! :-D
For example, almost each ocaml programmer that I know of have written a 
logger (me included, it's even in opam so that I can reinvent another 
wheel next time).

More seriously, concerning build systems, we clearly have quite some
choice on the OCaml shelf:
- obuild
- ocamlbuild
- omake
- oasis (which in fact uses ocamlbuild, don't forget that)
- jenga
- [...]

My preffered is obuild (https://github.com/ocaml-obuild/obuild),
for the terseness, readability and centralization of its build 
descriptions. I would love to see the user community of obuild grow,
so that we can get rid of more bugs, be able to compile _any_
OCaml project with it and implement even more cool features 
(contributors are very welcome).

I don't want a ring to rule them all, jut a ring that fits _my_ finger. ;)

-- 
Regards,
Francois.

Re: [Caml-list] One build system to rule them all?

[Jacques-Pascal Deplaix]

Hi,

I did know obuild but I forgot a fact about it: it doesn't use ocamlfind.

As you took the defense of obuild, do you know why it doesn't use it ? I
think I know how it works for simple packages but how is it supposed to
work with packages which uses the linkopts field or even if the cma has
not the same name as the library itself ? Why not having used findlib
(for the speed declaimed in DESIGN.md) ?

Did I missed something ?

Cheers,

On 09/10/2014 03:29 PM, Francois Berenger wrote:
> On 09/10/2014 02:49 PM, Yotam Barnoy wrote:
>> It appears to me that every couple of months we hear of someone
>> implementing yet another build system in ocaml. Given the success of
>> opam, I think it's clear that sometimes a monolithic solution, behind
>> which the entire community can organize, is the best solution --
>> especially for infrastructure. Looking at haskell, having cabal as the
>> main build system has really helped them advance in terms of supporting
>> other platforms (such as windows), and since all community efforts in
>> this realm are focused on cabal, they can improve it rapidly.
>>
>> a. Is there any build system we can organize behind to crown as the
>> official build system?
>> b. What are the use-cases missing from specific build systems, that have
>> driven people to use other build systems?
>> c. To pick one possible candidate, if ocamlbuild were spun out of the
>> compiler, could it be enhanced to cover all the main use-cases so
>> (almost) everyone would be happy with it?
>>
>> It's just such a shame to see the ocaml community re-inventing the wheel
>> over and over again, each time with some limitation so that the next
>> person needs to do the same thing yet again.
>
> It is not a shame, it is a tradition in the ocaml community! :-D
> For example, almost each ocaml programmer that I know of have written
> a logger (me included, it's even in opam so that I can reinvent
> another wheel next time).
>
> More seriously, concerning build systems, we clearly have quite some
> choice on the OCaml shelf:
> - obuild
> - ocamlbuild
> - omake
> - oasis (which in fact uses ocamlbuild, don't forget that)
> - jenga
> - [...]
>
> My preffered is obuild (https://github.com/ocaml-obuild/obuild),
> for the terseness, readability and centralization of its build
> descriptions. I would love to see the user community of obuild grow,
> so that we can get rid of more bugs, be able to compile _any_
> OCaml project with it and implement even more cool features
> (contributors are very welcome).
>
> I don't want a ring to rule them all, jut a ring that fits _my_
> finger. ;)
>

Re: [Caml-list] One build system to rule them all?

[Francois Berenger]

On 09/10/2014 03:53 PM, Jacques-Pascal Deplaix wrote:
> Hi,
>
> I did know obuild but I forgot a fact about it: it doesn't use ocamlfind.
>
> As you took the defense of obuild, do you know why it doesn't use it ? I
> think I know how it works for simple packages but how is it supposed to
> work with packages which uses the linkopts field or even if the cma has
> not the same name as the library itself ? Why not having used findlib
> (for the speed declaimed in DESIGN.md) ?

I don't know, I am just an obuild user, not a real contributor.

> Did I missed something ?
>
> Cheers,
>
> On 09/10/2014 03:29 PM, Francois Berenger wrote:
>> On 09/10/2014 02:49 PM, Yotam Barnoy wrote:
>>> It appears to me that every couple of months we hear of someone
>>> implementing yet another build system in ocaml. Given the success of
>>> opam, I think it's clear that sometimes a monolithic solution, behind
>>> which the entire community can organize, is the best solution --
>>> especially for infrastructure. Looking at haskell, having cabal as the
>>> main build system has really helped them advance in terms of supporting
>>> other platforms (such as windows), and since all community efforts in
>>> this realm are focused on cabal, they can improve it rapidly.
>>>
>>> a. Is there any build system we can organize behind to crown as the
>>> official build system?
>>> b. What are the use-cases missing from specific build systems, that have
>>> driven people to use other build systems?
>>> c. To pick one possible candidate, if ocamlbuild were spun out of the
>>> compiler, could it be enhanced to cover all the main use-cases so
>>> (almost) everyone would be happy with it?
>>>
>>> It's just such a shame to see the ocaml community re-inventing the wheel
>>> over and over again, each time with some limitation so that the next
>>> person needs to do the same thing yet again.
>>
>> It is not a shame, it is a tradition in the ocaml community! :-D
>> For example, almost each ocaml programmer that I know of have written
>> a logger (me included, it's even in opam so that I can reinvent
>> another wheel next time).
>>
>> More seriously, concerning build systems, we clearly have quite some
>> choice on the OCaml shelf:
>> - obuild
>> - ocamlbuild
>> - omake
>> - oasis (which in fact uses ocamlbuild, don't forget that)
>> - jenga
>> - [...]
>>
>> My preffered is obuild (https://github.com/ocaml-obuild/obuild),
>> for the terseness, readability and centralization of its build
>> descriptions. I would love to see the user community of obuild grow,
>> so that we can get rid of more bugs, be able to compile _any_
>> OCaml project with it and implement even more cool features
>> (contributors are very welcome).
>>
>> I don't want a ring to rule them all, jut a ring that fits _my_
>> finger. ;)
>>
>

-- 
Regards,
Francois.

Re: [Caml-list] One build system to rule them all?

[Thomas Braibant]

> Code-base size, dependency management, foreign-function binding, code
> generation and staging, transparency, documentation, speed...

and cross compilation, maybe?

Re: [Caml-list] One build system to rule them all?

[Adrien Nader]

On that topic, I've recently cleaned up my build scripts for use in
win-builds.org (which should see a release in a few weeks hopefully).

For the oasis case, I only set OCAMLFIND_CONF to a file which contains:
  destdir="/opt/windows_32/lib/ocaml/site-lib"
  path="/opt/windows_32/lib/ocaml/site-lib"
  # Where the cross-compiler is installed
  stdlib="/opt/cross_toolchain_32/lib/ocaml"
  # I have this variable but I only do native compile so I don't
  # actually use it
  ldconf="/opt/windows_32/lib/ocaml/ld.conf"
  # Below, the .exe versions are the cross-compiler binaries; their
  # proper name should be something like "i686-w64-mingw32-ocamlopt.opt"
  # instead of "ocamlopt.opt.exe"
  ocamlc="ocamlc.opt.exe"
  ocamlopt="ocamlopt.opt.exe"
  # Btw, that one is needed too for C stubs libraries and hasn't been in
  # oasis since the beginning; only something like >= 0.4
  ocamlmklib="ocamlmklib.exe"
  # And these are from the native toolchain, not the cross one (with my
  # most recent patches, these are cross-compiled and will be windows
  # executables)
  ocamldep="ocamldep.opt"
  ocamldoc="ocamldoc.opt"

Then, my oasis configure invocation looks like:
  ocaml setup.ml -configure --prefix "/${PREFIX}" --mandir "/${PREFIX}/man" --override ext_dll .dll

Very simple in practice.

I also need to mention that this relies on oasis not using all the
values it finds during configure. For instance it finds the host C
compiler but it'll let ocamlbuild and in turn ocamlc call the C
compiler.

-- 
Adrien Nader

Re: [Caml-list] One build system to rule them all?

[Maxence Guesdon]

On Wed, 10 Sep 2014 15:29:18 +0200
Francois Berenger <francois.berenger@inria.fr> wrote:

> On 09/10/2014 02:49 PM, Yotam Barnoy wrote:
> > It appears to me that every couple of months we hear of someone
> > implementing yet another build system in ocaml. Given the success of
> > opam, I think it's clear that sometimes a monolithic solution, behind
> > which the entire community can organize, is the best solution --
> > especially for infrastructure. Looking at haskell, having cabal as the
> > main build system has really helped them advance in terms of supporting
> > other platforms (such as windows), and since all community efforts in
> > this realm are focused on cabal, they can improve it rapidly.
> >
> > a. Is there any build system we can organize behind to crown as the
> > official build system?
> > b. What are the use-cases missing from specific build systems, that have
> > driven people to use other build systems?
> > c. To pick one possible candidate, if ocamlbuild were spun out of the
> > compiler, could it be enhanced to cover all the main use-cases so
> > (almost) everyone would be happy with it?
> >
> > It's just such a shame to see the ocaml community re-inventing the wheel
> > over and over again, each time with some limitation so that the next
> > person needs to do the same thing yet again.
> 
> It is not a shame, it is a tradition in the ocaml community! :-D
> For example, almost each ocaml programmer that I know of have written a 
> logger (me included, it's even in opam so that I can reinvent another 
> wheel next time).
> 
> More seriously, concerning build systems, we clearly have quite some
> choice on the OCaml shelf:
> - obuild
> - ocamlbuild
> - omake
> - oasis (which in fact uses ocamlbuild, don't forget that)
> - jenga
> - [...]

I can't resist to add good old Make to the list, not specific to OCaml,
yet powerful enough and known by a lot of developers.

Regards,

Maxence

Re: [Caml-list] One build system to rule them all?

[Gerd Stolpmann]

[-- Attachment #1: Type: text/plain, Size: 1597 bytes --]

Am Mittwoch, den 10.09.2014, 15:29 +0200 schrieb Francois Berenger:
> More seriously, concerning build systems, we clearly have quite some
> choice on the OCaml shelf:
> - obuild
> - ocamlbuild
> - omake
> - oasis (which in fact uses ocamlbuild, don't forget that)

oasis is not a build system. It is a package description format that
describes how to invoke the build (and more). So far there is only
built-in knowledge for ocamlbuild, but this may change, and you can
already call any external tool, so you can wrap any build system you
want. The intention is here more to create a uniform API for starting
the build, which may help packagers and other people who routinely build
software.

Gerd

> - jenga
> - [...]
> 
> My preffered is obuild (https://github.com/ocaml-obuild/obuild),
> for the terseness, readability and centralization of its build 
> descriptions. I would love to see the user community of obuild grow,
> so that we can get rid of more bugs, be able to compile _any_
> OCaml project with it and implement even more cool features 
> (contributors are very welcome).
> 
> I don't want a ring to rule them all, jut a ring that fits _my_ finger. ;)
> 
> -- 
> Regards,
> Francois.
> 

-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
My OCaml site:          http://www.camlcity.org
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Caml-list] One build system to rule them all?

[Leonardo Laguna Ruiz]

My pick is ocamlbuild because:

- It works the same way in all platforms that I work (linux,osx, windows 
msvc port, cygwin)
- If I can compile ocaml, then I have ocamlbuild
- It does not require external libraries

I have to say that I feel a little bit annoyed by the fact that some 
other build systems or tools, do not work correctly in all the platforms 
that I want to support.

Leonardo


On 9/10/2014 4:23 PM, Gerd Stolpmann wrote:
> Am Mittwoch, den 10.09.2014, 15:29 +0200 schrieb Francois Berenger:
>> More seriously, concerning build systems, we clearly have quite some
>> choice on the OCaml shelf:
>> - obuild
>> - ocamlbuild
>> - omake
>> - oasis (which in fact uses ocamlbuild, don't forget that)
> oasis is not a build system. It is a package description format that
> describes how to invoke the build (and more). So far there is only
> built-in knowledge for ocamlbuild, but this may change, and you can
> already call any external tool, so you can wrap any build system you
> want. The intention is here more to create a uniform API for starting
> the build, which may help packagers and other people who routinely build
> software.
>
> Gerd
>
>> - jenga
>> - [...]
>>
>> My preffered is obuild (https://github.com/ocaml-obuild/obuild),
>> for the terseness, readability and centralization of its build
>> descriptions. I would love to see the user community of obuild grow,
>> so that we can get rid of more bugs, be able to compile _any_
>> OCaml project with it and implement even more cool features
>> (contributors are very welcome).
>>
>> I don't want a ring to rule them all, jut a ring that fits _my_ finger. ;)
>>
>> -- 
>> Regards,
>> Francois.
>>

Re: [Caml-list] One build system to rule them all?

[Yotam Barnoy]

[-- Attachment #1: Type: text/plain, Size: 5050 bytes --]

Here's part of my motivation for starting this discussion now: I recently
experienced a hard-drive failure and had to switch to my backup windows
laptop. I was dismayed that opam is still not available for Windows, and
while wodi is available, it's not nearly as well developed, and apparently
requires many hacks for different packages.

It seems to me that these hacks are some of the things that make platform
compatibility hard for opam. Each build system has different requirements
and methods of compilation, and making this approach cross-platform
compatible is difficult. Ideally, a build system will abstract away the
things that are not inter-platform compatible. This is why I don't like
people using makefiles. Makefiles are not available natively on Windows,
and they often contain other bits of shell code that isn't available on all
POSIX platforms, let alone on Windows. Not to mention the fact that
makefiles have a very tricky and sensitive syntax.

I don't expect one build system to match everyone's requirements, but I
think if we get together, make a list of requirements, and try to get over
our personal biases, we can find something that works for most people, and
with some group effort, can work for even more people in more use-cases.
I'm not suggesting that we invent something new, but that we take something
good and make it better, as well as make an effort to learn that tool and
convert packages to use that tool.

So here are some requirements I can think of (using some of the suggestions
that have been brought up):
- Easy to use, especially for small projects (large projects can afford to
put more time into their build systems)
- Abstract away platform considerations as much as possible. No dependence
on specific shells and POSIX utilities.
- Allows compilation of C files, which is quite common in ocaml packages.
- Scalable to many directories and files
- Uses ocamlfind to locate packages
- Handles camlp4 and ppx
- Parallel & incremental compilation

About Jenga: I took a quick look at Jenga, and even though I'm impressed by
its capabilities, the amount of code needed to be written even for simple
projects is overwhelming. It's clearly a very flexible and powerful tool,
but I'd say it's too flexible for the mainstream. There's always room for a
build system that's integrated into ocaml itself (like Shake), but I think
declarative build systems tend to be easier to comprehend for the average
user.

ocp-build actually looks very interesting. The manual (which is here:
http://github.com/OCamlPro/ocp-build/blob/master/docs/user-manual/user-manual.pdf?raw=true)
is incomplete, but contains a nice survey of the existing build tools, and
motivation for making ocp-build. Has anyone had experience with ocp-build?
Opam seems to be using it, but they also use a makefile (why?) with a bunch
of shell commands inside (which is precisely the problem from my
perspective). ocp-build is supposedly compatible with Windows, too.

Yotam


On Wed, Sep 10, 2014 at 11:17 AM, Leonardo Laguna Ruiz <modlfo@gmail.com>
wrote:

> My pick is ocamlbuild because:
>
> - It works the same way in all platforms that I work (linux,osx, windows
> msvc port, cygwin)
> - If I can compile ocaml, then I have ocamlbuild
> - It does not require external libraries
>
> I have to say that I feel a little bit annoyed by the fact that some other
> build systems or tools, do not work correctly in all the platforms that I
> want to support.
>
> Leonardo
>
>
>
> On 9/10/2014 4:23 PM, Gerd Stolpmann wrote:
>
>> Am Mittwoch, den 10.09.2014, 15:29 +0200 schrieb Francois Berenger:
>>
>>> More seriously, concerning build systems, we clearly have quite some
>>> choice on the OCaml shelf:
>>> - obuild
>>> - ocamlbuild
>>> - omake
>>> - oasis (which in fact uses ocamlbuild, don't forget that)
>>>
>> oasis is not a build system. It is a package description format that
>> describes how to invoke the build (and more). So far there is only
>> built-in knowledge for ocamlbuild, but this may change, and you can
>> already call any external tool, so you can wrap any build system you
>> want. The intention is here more to create a uniform API for starting
>> the build, which may help packagers and other people who routinely build
>> software.
>>
>> Gerd
>>
>>  - jenga
>>> - [...]
>>>
>>> My preffered is obuild (https://github.com/ocaml-obuild/obuild),
>>> for the terseness, readability and centralization of its build
>>> descriptions. I would love to see the user community of obuild grow,
>>> so that we can get rid of more bugs, be able to compile _any_
>>> OCaml project with it and implement even more cool features
>>> (contributors are very welcome).
>>>
>>> I don't want a ring to rule them all, jut a ring that fits _my_ finger.
>>> ;)
>>>
>>> --
>>> Regards,
>>> Francois.
>>>
>>>
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>

[-- Attachment #2: Type: text/html, Size: 6669 bytes --]

Re: [Caml-list] One build system to rule them all?

[Drup]

Le 10/09/2014 16:17, Maxence Guesdon a écrit :
> I can't resist to add good old Make to the list, not specific to OCaml,
> yet powerful enough and known by a lot of developers.
>
No.
Every single makefile to compile ocaml (except OCaml-makefile[1]) I have 
encountered are broken. Every single one of them. OCaml compilation 
process is too complex to be able to fire some rules and be done with 
it; especially if you want portability.

Please, don't write makefiles.

[1]: https://github.com/mmottl/ocaml-makefile

Re: [Caml-list] One build system to rule them all?

[Peter Zotov]

On 2014-09-10 22:59, Yotam Barnoy wrote:
> ocp-build actually looks very interesting. The manual (which is here:
> http://github.com/OCamlPro/ocp-build/blob/master/docs/user-manual/user-manual.pdf?raw=true
> [2]) is incomplete, but contains a nice survey of the existing build
> tools, and motivation for making ocp-build. Has anyone had experience
> with ocp-build? Opam seems to be using it, but they also use a
> makefile (why?) with a bunch of shell commands inside (which is
> precisely the problem from my perspective). ocp-build is supposedly
> compatible with Windows, too.

Every single time I had to use ocp-build, it broke in an odd and hard to
fix way. It was so bad that eventually I just ported the ocp-build-using
projects (ocp-index and its dependencies) to OASIS. Most worryingly it 
has
some strange requirement to ship bytecode, which ties it to a released 
OCaml
version; no other buildsystem needs that.

-- 
Peter Zotov

Re: [Caml-list] One build system to rule them all?

[Sebastien Mondet]

[-- Attachment #1: Type: text/plain, Size: 2552 bytes --]

On Wed, Sep 10, 2014 at 3:16 PM, Peter Zotov <whitequark@whitequark.org>
wrote:

> On 2014-09-10 22:59, Yotam Barnoy wrote:
>
>> ocp-build actually looks very interesting. The manual (which is here:
>> http://github.com/OCamlPro/ocp-build/blob/master/docs/
>> user-manual/user-manual.pdf?raw=true
>> [2]) is incomplete, but contains a nice survey of the existing build
>> tools, and motivation for making ocp-build. Has anyone had experience
>> with ocp-build? Opam seems to be using it, but they also use a
>> makefile (why?) with a bunch of shell commands inside (which is
>> precisely the problem from my perspective). ocp-build is supposedly
>> compatible with Windows, too.
>>
>
> Every single time I had to use ocp-build, it broke in an odd and hard to
> fix way. It was so bad that eventually I just ported the ocp-build-using
> projects (ocp-index and its dependencies) to OASIS. Most worryingly it has
> some strange requirement to ship bytecode, which ties it to a released
> OCaml
> version; no other buildsystem needs that.
>
>
Yes ocp-build is broken for many corner cases, I've been trying to push it
to its maximum; look at that ugly shell script:
https://github.com/hammerlab/ketrew/blob/master/please.sh that even has to
create a yojson library out of the one in ~/.opam to please ocp-build's
assumptions.

Also, like every build system based on flat files (oasis, obuild), it is
fundamentally broken. You'll always need a programming language to extended
your build (adding targets, like build documentation/websites, special
tests, .merlin files, code generation...).

- omake did that with yet another obscure and weird language (I guess the
goal was to "look" like `make` but with even more broken string escaping).
- ocamlbuild and jenga picked the right language.
    - ocamlbuild's API is very limited, there is not even a clear way to
replace all the crazy small files required everywhere (_tags, mllib, ...)
with function calls within a myocamlbuild.ml plugin. It is also painfully
slow.
    - Jenga is not for "normal" projects. It takes half-an-hour to build
jenga itself, and it's dependency tree is not very portable. The API is
very convoluted even for simple projects.

Look at https://github.com/samoht/assemblage/ certainly going to in the
right direction.






> --
> Peter Zotov
>
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>

[-- Attachment #2: Type: text/html, Size: 4118 bytes --]

Re: [Caml-list] One build system to rule them all?

[Adrien Nader]

On Wed, Sep 10, 2014, Yotam Barnoy wrote:
> Here's part of my motivation for starting this discussion now: I recently
> experienced a hard-drive failure and had to switch to my backup windows
> laptop. I was dismayed that opam is still not available for Windows, and
> while wodi is available, it's not nearly as well developed, and apparently
> requires many hacks for different packages.
> 
> It seems to me that these hacks are some of the things that make platform
> compatibility hard for opam. Each build system has different requirements
> and methods of compilation, and making this approach cross-platform
> compatible is difficult. Ideally, a build system will abstract away the
> things that are not inter-platform compatible. This is why I don't like
> people using makefiles. Makefiles are not available natively on Windows,
> and they often contain other bits of shell code that isn't available on all
> POSIX platforms, let alone on Windows. Not to mention the fact that
> makefiles have a very tricky and sensitive syntax.

You're taking the wrong approach IMHO. The issue is not with the build
systems (usually) but with the build environements and how you're using
them.

On Windows you have three choices to build from. Chose one. Stick to it.
Don't mix.

1- Cygwin or MSYS/MSYS2 (MSYS is a fork of Cygwin; MSYS2 is a fork of
newer Cygwin).
They are POSIXish (Cygwin) or POSIX-like-but-actually-doing-dirtytricks
(MSYS*). They can run both regular Windows apps and specific cygwin/msys
apps too.
Consider that when from Cygwin, you will be doing a special
cases of cross-compilation which allows you to run what you've built
(but you will probably have to post-process the output of what you run
if it contains paths).
On MSYS*, you'll try to mix both that not-really-posix universe with
windows and it should work most of the time. If you look deeper at how
it's mixing things, you will probably feel like never using it ever
again.

2- Visual Studio.
First you need to click on the "I accept this EULA" tickbox to sell your
soul to Microsoft.
Then you will discover a world of ad-hoc solutions, collection of blobs
and random binaries, build cargo cult and "I'm keeping this installation
since it builds what I need and it's too risky to change it; I know it's
running Windows 98 and Visual Studio 6 but it works!".
This beast is also the reason CMake is engineered the fail at finding
dependencies except when you give it their full path along with every
possible bit of information it might need during build since it won't
find anything (handy, hey?).
[ that's really the design idea behind cmake's configuration
"discovery": you list paths and libs and it tries them until it finds
something which passes the insufficient match tests given; you're
supposed to avoid pkg-config or at least make it a secondary citizen,
which means it's broken and every lib detection script has to reinvent
pkg-config usage in a broken way using the horrible cmake macro language
]

3- Other IDEs.
Some tend to use a make. A native one which spawns cmd.exe. I guess they
write Makefiles which contain very simple commands. It seems they work
fine. Makefiles are generated from some IDE-specific code. They share
some of the bad things MSVC but at least they're not building using a
dump of blobs and they don't change their project description
fileformats with each version.

Pick your broken system. Send love letters to Microsoft.

As far as I'm concerned, I'm cross-compiling from Linux. Or sometimes
from Cygwin. And when I really need to, from MSYS. In other words, I
solve the issue by avoiding Windows as much as possible, even when on
Windows.

Pick one. Understand it. Don't cross the streams. Eat five vegetables
and fruits per day. Alcohol and other drugs are not the answer (violence
is).

-- 
Adrien Nader

Re: [Caml-list] One build system to rule them all?

[Gabriel Scherer]

[-- Attachment #1: Type: text/plain, Size: 3776 bytes --]

I'm not convinced by the starting point of this discussion. Why would we
need a consensus on a single build system? It seems to me that build
systems are fine as per-project choices; I don't lose much if some other
project uses another build system. The libraries I rely on may use any
build system they like, it doesn't affect my own work.


Of course there is also a question of the workforce supporting any given
build system. OCamlbuild for example could certainly do with more people
improving the tool; but it seems highly unclear that developers working
today on tool X would contribute on tool Y if X didn't exist.

(Fragmentation of, say, standard libraries seem more problematic as it
tends to create separate ecosystems.)


PS:

> if ocamlbuild were spun out of the compiler, could it be enhanced to
cover all the main use-cases so (almost) everyone would be happy with it

OCamlbuild accepts patches today (through caml-list, mantis, or as github
pull requests). There is no reason to wait for the "spinning out" to
contribute.

On Wed, Sep 10, 2014 at 9:56 PM, Sebastien Mondet <
sebastien.mondet@gmail.com> wrote:

>
>
> On Wed, Sep 10, 2014 at 3:16 PM, Peter Zotov <whitequark@whitequark.org>
> wrote:
>
>> On 2014-09-10 22:59, Yotam Barnoy wrote:
>>
>>> ocp-build actually looks very interesting. The manual (which is here:
>>> http://github.com/OCamlPro/ocp-build/blob/master/docs/
>>> user-manual/user-manual.pdf?raw=true
>>> [2]) is incomplete, but contains a nice survey of the existing build
>>> tools, and motivation for making ocp-build. Has anyone had experience
>>> with ocp-build? Opam seems to be using it, but they also use a
>>> makefile (why?) with a bunch of shell commands inside (which is
>>> precisely the problem from my perspective). ocp-build is supposedly
>>> compatible with Windows, too.
>>>
>>
>> Every single time I had to use ocp-build, it broke in an odd and hard to
>> fix way. It was so bad that eventually I just ported the ocp-build-using
>> projects (ocp-index and its dependencies) to OASIS. Most worryingly it has
>> some strange requirement to ship bytecode, which ties it to a released
>> OCaml
>> version; no other buildsystem needs that.
>>
>>
> Yes ocp-build is broken for many corner cases, I've been trying to push it
> to its maximum; look at that ugly shell script:
> https://github.com/hammerlab/ketrew/blob/master/please.sh that even has
> to create a yojson library out of the one in ~/.opam to please ocp-build's
> assumptions.
>
> Also, like every build system based on flat files (oasis, obuild), it is
> fundamentally broken. You'll always need a programming language to extended
> your build (adding targets, like build documentation/websites, special
> tests, .merlin files, code generation...).
>
> - omake did that with yet another obscure and weird language (I guess the
> goal was to "look" like `make` but with even more broken string escaping).
> - ocamlbuild and jenga picked the right language.
>     - ocamlbuild's API is very limited, there is not even a clear way to
> replace all the crazy small files required everywhere (_tags, mllib, ...)
> with function calls within a myocamlbuild.ml plugin. It is also painfully
> slow.
>     - Jenga is not for "normal" projects. It takes half-an-hour to build
> jenga itself, and it's dependency tree is not very portable. The API is
> very convoluted even for simple projects.
>
> Look at https://github.com/samoht/assemblage/ certainly going to in the
> right direction.
>
>
>
>
>
>
>> --
>> Peter Zotov
>>
>>
>> --
>> Caml-list mailing list.  Subscription management and archives:
>> https://sympa.inria.fr/sympa/arc/caml-list
>> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>> Bug reports: http://caml.inria.fr/bin/caml-bugs
>>
>
>

[-- Attachment #2: Type: text/html, Size: 5769 bytes --]

Re: [Caml-list] One build system to rule them all?

[Gerd Stolpmann]

[-- Attachment #1: Type: text/plain, Size: 1156 bytes --]

Am Mittwoch, den 10.09.2014, 21:13 +0200 schrieb Drup:
> Le 10/09/2014 16:17, Maxence Guesdon a écrit :
> > I can't resist to add good old Make to the list, not specific to OCaml,
> > yet powerful enough and known by a lot of developers.
> >
> No.
> Every single makefile to compile ocaml (except OCaml-makefile[1]) I have 
> encountered are broken. Every single one of them. OCaml compilation 
> process is too complex to be able to fire some rules and be done with 
> it; especially if you want portability.
> 
> Please, don't write makefiles.

Yes, they are broken, but at least you know very well HOW they are
broken. That's the big advantage to other build systems which are
sometimes difficult to debug.

Gerd

> 
> [1]: https://github.com/mmottl/ocaml-makefile
> 
> 

-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
My OCaml site:          http://www.camlcity.org
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Caml-list] One build system to rule them all?

[Gerd Stolpmann]

[-- Attachment #1: Type: text/plain, Size: 2724 bytes --]

Am Mittwoch, den 10.09.2014, 15:56 -0400 schrieb Sebastien Mondet:

> 
> - omake did that with yet another obscure and weird language (I guess
> the goal was to "look" like `make` but with even more broken string
> escaping).

omake has at least a working, powerful and quite clean model of
dependencies, and the build rules for the supported languages (like
OCaml) are read from OMakefiles and are not hard-coded. There was some
excellent analysis how to achieve this. I also think that the omake
language is a bit obscure as language. It's a dynamic functional
language that tries to do too much; you can run code as functions, as
statements, and as commands, and all these three forms use a slightly
different syntax (I guess that's why you complain about broken string
escaping - it is sometimes confusing which syntactical rules are
applicable at a certain point in the code). I'd love when this part
would be dropped, and I could develop all the helper functionality for a
build directly in OCaml, enhanced by syntax extensions for running shell
commands and for entering build rules into the dependency tree. E.g. you
could have:

let variable = ...

let do_something() = ...

<:make<
file_to_create: file_dep1 file_dep2
    ./build_tool.sh $(variable)
    $(do_something())
>>

just to illustrate the idea of mixing several notations in the same
file. Every OMakefile could e.g. be understood as functor that takes an
environment with build rules and returns a new, extended environment.
Finally everything is combined and executed. (Too bad that I don't have
more time to work on something like this.)

Gerd



> - ocamlbuild and jenga picked the right language.
> 
>     - ocamlbuild's API is very limited, there is not even a clear way
> to replace all the crazy small files required everywhere (_tags,
> mllib, ...) with function calls within a myocamlbuild.ml plugin. It is
> also painfully slow.
> 
>     - Jenga is not for "normal" projects. It takes half-an-hour to
> build jenga itself, and it's dependency tree is not very portable. The
> API is very convoluted even for simple projects.
> 
> 
> Look at https://github.com/samoht/assemblage/ certainly going to in
> the right direction.
> 
> 
> 
> 
> 
>  
>         -- 
>         Peter Zotov
>         
>         
>         
> 
> 
-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
My OCaml site:          http://www.camlcity.org
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Caml-list] One build system to rule them all?

[Francois Berenger]

On 09/10/2014 08:59 PM, Yotam Barnoy wrote:
> Here's part of my motivation for starting this discussion now: I
> recently experienced a hard-drive failure and had to switch to my backup
> windows laptop. I was dismayed that opam is still not available for
> Windows, and while wodi is available, it's not nearly as well developed,
> and apparently requires many hacks for different packages.
>
> It seems to me that these hacks are some of the things that make
> platform compatibility hard for opam. Each build system has different
> requirements and methods of compilation, and making this approach
> cross-platform compatible is difficult. Ideally, a build system will
> abstract away the things that are not inter-platform compatible. This is
> why I don't like people using makefiles. Makefiles are not available
> natively on Windows, and they often contain other bits of shell code
> that isn't available on all POSIX platforms, let alone on Windows. Not
> to mention the fact that makefiles have a very tricky and sensitive syntax.
>
> I don't expect one build system to match everyone's requirements, but I
> think if we get together, make a list of requirements, and try to get
> over our personal biases, we can find something that works for most
> people, and with some group effort, can work for even more people in
> more use-cases. I'm not suggesting that we invent something new, but
> that we take something good and make it better, as well as make an
> effort to learn that tool and convert packages to use that tool.
>
> So here are some requirements I can think of (using some of the
> suggestions that have been brought up):
> - Easy to use, especially for small projects (large projects can afford
> to put more time into their build systems)
> - Abstract away platform considerations as much as possible. No
> dependence on specific shells and POSIX utilities.
> - Allows compilation of C files, which is quite common in ocaml packages.
> - Scalable to many directories and files
> - Uses ocamlfind to locate packages
> - Handles camlp4 and ppx
> - Parallel & incremental compilation
>
> About Jenga: I took a quick look at Jenga, and even though I'm impressed
> by its capabilities, the amount of code needed to be written even for
> simple projects is overwhelming. It's clearly a very flexible and
> powerful tool, but I'd say it's too flexible for the mainstream. There's
> always room for a build system that's integrated into ocaml itself (like
> Shake), but I think declarative build systems tend to be easier to
> comprehend for the average user.
>
> ocp-build actually looks very interesting. The manual (which is here:
> http://github.com/OCamlPro/ocp-build/blob/master/docs/user-manual/user-manual.pdf?raw=true)
> is incomplete, but contains a nice survey of the existing build tools,
> and motivation for making ocp-build. Has anyone had experience with
> ocp-build? Opam seems to be using it, but they also use a makefile
> (why?) with a bunch of shell commands inside (which is precisely the
> problem from my perspective). ocp-build is supposedly compatible with
> Windows, too.

Here is one thing I dislike in ocp-build: you have to explicitely list 
all the source files (and maybe same thing for Jenga).

I am way too lazy to do that, especially since other tools are able
to infer this list automatically (a library is just a list of modules,
an executable has a single entry point, the main function in a single file).

> Yotam
>
>
> On Wed, Sep 10, 2014 at 11:17 AM, Leonardo Laguna Ruiz <modlfo@gmail.com
> <mailto:modlfo@gmail.com>> wrote:
>
>     My pick is ocamlbuild because:
>
>     - It works the same way in all platforms that I work (linux,osx,
>     windows msvc port, cygwin)
>     - If I can compile ocaml, then I have ocamlbuild
>     - It does not require external libraries
>
>     I have to say that I feel a little bit annoyed by the fact that some
>     other build systems or tools, do not work correctly in all the
>     platforms that I want to support.
>
>     Leonardo
>
>
>
>     On 9/10/2014 4:23 PM, Gerd Stolpmann wrote:
>
>         Am Mittwoch, den 10.09.2014, 15:29 +0200 schrieb Francois Berenger:
>
>             More seriously, concerning build systems, we clearly have
>             quite some
>             choice on the OCaml shelf:
>             - obuild
>             - ocamlbuild
>             - omake
>             - oasis (which in fact uses ocamlbuild, don't forget that)
>
>         oasis is not a build system. It is a package description format that
>         describes how to invoke the build (and more). So far there is only
>         built-in knowledge for ocamlbuild, but this may change, and you can
>         already call any external tool, so you can wrap any build system you
>         want. The intention is here more to create a uniform API for
>         starting
>         the build, which may help packagers and other people who
>         routinely build
>         software.
>
>         Gerd
>
>             - jenga
>             - [...]
>
>             My preffered is obuild
>             (https://github.com/ocaml-__obuild/obuild
>             <https://github.com/ocaml-obuild/obuild>),
>             for the terseness, readability and centralization of its build
>             descriptions. I would love to see the user community of
>             obuild grow,
>             so that we can get rid of more bugs, be able to compile _any_
>             OCaml project with it and implement even more cool features
>             (contributors are very welcome).
>
>             I don't want a ring to rule them all, jut a ring that fits
>             _my_ finger. ;)
>
>             --
>             Regards,
>             Francois.
>
>
>
>     --
>     Caml-list mailing list.  Subscription management and archives:
>     https://sympa.inria.fr/sympa/__arc/caml-list
>     <https://sympa.inria.fr/sympa/arc/caml-list>
>     Beginner's list: http://groups.yahoo.com/group/__ocaml_beginners
>     <http://groups.yahoo.com/group/ocaml_beginners>
>     Bug reports: http://caml.inria.fr/bin/caml-__bugs
>     <http://caml.inria.fr/bin/caml-bugs>
>
>

-- 
Regards,
Francois.

Re: [Caml-list] One build system to rule them all?

[Yaron Minsky]

On Thu, Sep 11, 2014 at 3:53 AM, Francois Berenger
<francois.berenger@inria.fr> wrote:
> On 09/10/2014 08:59 PM, Yotam Barnoy wrote:
>>
>> Here's part of my motivation for starting this discussion now: I
>> recently experienced a hard-drive failure and had to switch to my backup
>> windows laptop. I was dismayed that opam is still not available for
>> Windows, and while wodi is available, it's not nearly as well developed,
>> and apparently requires many hacks for different packages.
>>
>> It seems to me that these hacks are some of the things that make
>> platform compatibility hard for opam. Each build system has different
>> requirements and methods of compilation, and making this approach
>> cross-platform compatible is difficult. Ideally, a build system will
>> abstract away the things that are not inter-platform compatible. This is
>> why I don't like people using makefiles. Makefiles are not available
>> natively on Windows, and they often contain other bits of shell code
>> that isn't available on all POSIX platforms, let alone on Windows. Not
>> to mention the fact that makefiles have a very tricky and sensitive
>> syntax.
>>
>> I don't expect one build system to match everyone's requirements, but I
>> think if we get together, make a list of requirements, and try to get
>> over our personal biases, we can find something that works for most
>> people, and with some group effort, can work for even more people in
>> more use-cases. I'm not suggesting that we invent something new, but
>> that we take something good and make it better, as well as make an
>> effort to learn that tool and convert packages to use that tool.
>>
>> So here are some requirements I can think of (using some of the
>> suggestions that have been brought up):
>> - Easy to use, especially for small projects (large projects can afford
>> to put more time into their build systems)
>> - Abstract away platform considerations as much as possible. No
>> dependence on specific shells and POSIX utilities.
>> - Allows compilation of C files, which is quite common in ocaml packages.
>> - Scalable to many directories and files
>> - Uses ocamlfind to locate packages
>> - Handles camlp4 and ppx
>> - Parallel & incremental compilation
>>
>> About Jenga: I took a quick look at Jenga, and even though I'm impressed
>> by its capabilities, the amount of code needed to be written even for
>> simple projects is overwhelming. It's clearly a very flexible and
>> powerful tool, but I'd say it's too flexible for the mainstream. There's
>> always room for a build system that's integrated into ocaml itself (like
>> Shake), but I think declarative build systems tend to be easier to
>> comprehend for the average user.
>>
>> ocp-build actually looks very interesting. The manual (which is here:
>>
>> http://github.com/OCamlPro/ocp-build/blob/master/docs/user-manual/user-manual.pdf?raw=true)
>> is incomplete, but contains a nice survey of the existing build tools,
>> and motivation for making ocp-build. Has anyone had experience with
>> ocp-build? Opam seems to be using it, but they also use a makefile
>> (why?) with a bunch of shell commands inside (which is precisely the
>> problem from my perspective). ocp-build is supposedly compatible with
>> Windows, too.
>
>
> Here is one thing I dislike in ocp-build: you have to explicitely list all
> the source files (and maybe same thing for Jenga).

Jenga is quite flexible, so you could write your jengaroot either way
(to demand or not demand explicit source file lists.)  Our internal
jengaroot does not require explicit source file lists.

Not to confuse people, though, Jenga isn't really suitable for people
just picking up and using yet.  We haven't yet written a suitable
jengaroot for general use yet (though there is a version of our
internal jengaroot that we've released, but it's not yet ready for
prime time.)  So Jenga is interesting as an option to discuss, but not
yet a practical option we want to encourage casual users to try.

> I am way too lazy to do that, especially since other tools are able
> to infer this list automatically (a library is just a list of modules,
> an executable has a single entry point, the main function in a single file).
>
>> Yotam
>>
>>
>> On Wed, Sep 10, 2014 at 11:17 AM, Leonardo Laguna Ruiz <modlfo@gmail.com
>> <mailto:modlfo@gmail.com>> wrote:
>>
>>     My pick is ocamlbuild because:
>>
>>     - It works the same way in all platforms that I work (linux,osx,
>>     windows msvc port, cygwin)
>>     - If I can compile ocaml, then I have ocamlbuild
>>     - It does not require external libraries
>>
>>     I have to say that I feel a little bit annoyed by the fact that some
>>     other build systems or tools, do not work correctly in all the
>>     platforms that I want to support.
>>
>>     Leonardo
>>
>>
>>
>>     On 9/10/2014 4:23 PM, Gerd Stolpmann wrote:
>>
>>         Am Mittwoch, den 10.09.2014, 15:29 +0200 schrieb Francois
>> Berenger:
>>
>>             More seriously, concerning build systems, we clearly have
>>             quite some
>>             choice on the OCaml shelf:
>>             - obuild
>>             - ocamlbuild
>>             - omake
>>             - oasis (which in fact uses ocamlbuild, don't forget that)
>>
>>         oasis is not a build system. It is a package description format
>> that
>>         describes how to invoke the build (and more). So far there is only
>>         built-in knowledge for ocamlbuild, but this may change, and you
>> can
>>         already call any external tool, so you can wrap any build system
>> you
>>         want. The intention is here more to create a uniform API for
>>         starting
>>         the build, which may help packagers and other people who
>>         routinely build
>>         software.
>>
>>         Gerd
>>
>>             - jenga
>>             - [...]
>>
>>             My preffered is obuild
>>             (https://github.com/ocaml-__obuild/obuild
>>             <https://github.com/ocaml-obuild/obuild>),
>>             for the terseness, readability and centralization of its build
>>             descriptions. I would love to see the user community of
>>             obuild grow,
>>             so that we can get rid of more bugs, be able to compile _any_
>>             OCaml project with it and implement even more cool features
>>             (contributors are very welcome).
>>
>>             I don't want a ring to rule them all, jut a ring that fits
>>             _my_ finger. ;)
>>
>>             --
>>             Regards,
>>             Francois.
>>
>>
>>
>>     --
>>     Caml-list mailing list.  Subscription management and archives:
>>     https://sympa.inria.fr/sympa/__arc/caml-list
>>     <https://sympa.inria.fr/sympa/arc/caml-list>
>>     Beginner's list: http://groups.yahoo.com/group/__ocaml_beginners
>>     <http://groups.yahoo.com/group/ocaml_beginners>
>>     Bug reports: http://caml.inria.fr/bin/caml-__bugs
>>     <http://caml.inria.fr/bin/caml-bugs>
>>
>>
>
> --
> Regards,
> Francois.
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs

Re: [Caml-list] One build system to rule them all?

[Yotam Barnoy]

[-- Attachment #1: Type: text/plain, Size: 9923 bytes --]

OK so here's my summary of what I've heard here so far:

- Ocaml does not yet have a (public) build system that the majority of
users can be happy with for all of their project needs.
- It seems difficult to have a build system that's simple enough to use for
small projects, flexible enough to handle the more complicated projects,
and that can scale.
- While declarative, dsl-based tools are appealing, they get bogged down as
you add features to them, making them become unwieldy.
- A better solution seems to be using a real programming language to
describe packages. In our case that should clearly be ocaml itself. The
trick is making the API simple.

Looking at some of the the available options:
- ocamlbuild (dsl-based) is a decent basic choice, but lacks advanced
features and has become weighed down by multiple file types (_tags,
myocamlbuild etc).
- OASIS (dsl-based) seems to be a good description layer for projects,
currently farming out the difficult work of building stuff to ocamlbuild,
but not really a contender in this race except as an abstraction layer.
- omake (dsl-based) has some fans, but becomes overly-complex for more
complicated projects.
- ocp-build (dsl-based) has had nothing but criticism on this thread. Is
there any ongoing work to address these criticisms?
- Jenga (uses ocaml) seems promising but has heavy dependencies, and is
currently not geared towards light projects.
- assemblage (uses ocaml) seems promising as well, and seems focused on
simplicity. Any comment from assemblage contributors on its status?

Also, while I understand some of the windows hatred, the fact is, most of
the world uses windows. Even using cygwin as a translation layer hurts
ocaml's ability to reach people. Ideally, a windows programmer could
download opam, ocaml and perhaps a mingw compiler, and begin to build
projects without any need for a shell or anything POSIX-like. This means
that the build system needs to obviate the need for any shell commands or
makefiles.

On Thu, Sep 11, 2014 at 6:37 AM, Yaron Minsky <yminsky@janestreet.com>
wrote:

> On Thu, Sep 11, 2014 at 3:53 AM, Francois Berenger
> <francois.berenger@inria.fr> wrote:
> > On 09/10/2014 08:59 PM, Yotam Barnoy wrote:
> >>
> >> Here's part of my motivation for starting this discussion now: I
> >> recently experienced a hard-drive failure and had to switch to my backup
> >> windows laptop. I was dismayed that opam is still not available for
> >> Windows, and while wodi is available, it's not nearly as well developed,
> >> and apparently requires many hacks for different packages.
> >>
> >> It seems to me that these hacks are some of the things that make
> >> platform compatibility hard for opam. Each build system has different
> >> requirements and methods of compilation, and making this approach
> >> cross-platform compatible is difficult. Ideally, a build system will
> >> abstract away the things that are not inter-platform compatible. This is
> >> why I don't like people using makefiles. Makefiles are not available
> >> natively on Windows, and they often contain other bits of shell code
> >> that isn't available on all POSIX platforms, let alone on Windows. Not
> >> to mention the fact that makefiles have a very tricky and sensitive
> >> syntax.
> >>
> >> I don't expect one build system to match everyone's requirements, but I
> >> think if we get together, make a list of requirements, and try to get
> >> over our personal biases, we can find something that works for most
> >> people, and with some group effort, can work for even more people in
> >> more use-cases. I'm not suggesting that we invent something new, but
> >> that we take something good and make it better, as well as make an
> >> effort to learn that tool and convert packages to use that tool.
> >>
> >> So here are some requirements I can think of (using some of the
> >> suggestions that have been brought up):
> >> - Easy to use, especially for small projects (large projects can afford
> >> to put more time into their build systems)
> >> - Abstract away platform considerations as much as possible. No
> >> dependence on specific shells and POSIX utilities.
> >> - Allows compilation of C files, which is quite common in ocaml
> packages.
> >> - Scalable to many directories and files
> >> - Uses ocamlfind to locate packages
> >> - Handles camlp4 and ppx
> >> - Parallel & incremental compilation
> >>
> >> About Jenga: I took a quick look at Jenga, and even though I'm impressed
> >> by its capabilities, the amount of code needed to be written even for
> >> simple projects is overwhelming. It's clearly a very flexible and
> >> powerful tool, but I'd say it's too flexible for the mainstream. There's
> >> always room for a build system that's integrated into ocaml itself (like
> >> Shake), but I think declarative build systems tend to be easier to
> >> comprehend for the average user.
> >>
> >> ocp-build actually looks very interesting. The manual (which is here:
> >>
> >>
> http://github.com/OCamlPro/ocp-build/blob/master/docs/user-manual/user-manual.pdf?raw=true
> )
> >> is incomplete, but contains a nice survey of the existing build tools,
> >> and motivation for making ocp-build. Has anyone had experience with
> >> ocp-build? Opam seems to be using it, but they also use a makefile
> >> (why?) with a bunch of shell commands inside (which is precisely the
> >> problem from my perspective). ocp-build is supposedly compatible with
> >> Windows, too.
> >
> >
> > Here is one thing I dislike in ocp-build: you have to explicitely list
> all
> > the source files (and maybe same thing for Jenga).
>
> Jenga is quite flexible, so you could write your jengaroot either way
> (to demand or not demand explicit source file lists.)  Our internal
> jengaroot does not require explicit source file lists.
>
> Not to confuse people, though, Jenga isn't really suitable for people
> just picking up and using yet.  We haven't yet written a suitable
> jengaroot for general use yet (though there is a version of our
> internal jengaroot that we've released, but it's not yet ready for
> prime time.)  So Jenga is interesting as an option to discuss, but not
> yet a practical option we want to encourage casual users to try.
>
> > I am way too lazy to do that, especially since other tools are able
> > to infer this list automatically (a library is just a list of modules,
> > an executable has a single entry point, the main function in a single
> file).
> >
> >> Yotam
> >>
> >>
> >> On Wed, Sep 10, 2014 at 11:17 AM, Leonardo Laguna Ruiz <
> modlfo@gmail.com
> >> <mailto:modlfo@gmail.com>> wrote:
> >>
> >>     My pick is ocamlbuild because:
> >>
> >>     - It works the same way in all platforms that I work (linux,osx,
> >>     windows msvc port, cygwin)
> >>     - If I can compile ocaml, then I have ocamlbuild
> >>     - It does not require external libraries
> >>
> >>     I have to say that I feel a little bit annoyed by the fact that some
> >>     other build systems or tools, do not work correctly in all the
> >>     platforms that I want to support.
> >>
> >>     Leonardo
> >>
> >>
> >>
> >>     On 9/10/2014 4:23 PM, Gerd Stolpmann wrote:
> >>
> >>         Am Mittwoch, den 10.09.2014, 15:29 +0200 schrieb Francois
> >> Berenger:
> >>
> >>             More seriously, concerning build systems, we clearly have
> >>             quite some
> >>             choice on the OCaml shelf:
> >>             - obuild
> >>             - ocamlbuild
> >>             - omake
> >>             - oasis (which in fact uses ocamlbuild, don't forget that)
> >>
> >>         oasis is not a build system. It is a package description format
> >> that
> >>         describes how to invoke the build (and more). So far there is
> only
> >>         built-in knowledge for ocamlbuild, but this may change, and you
> >> can
> >>         already call any external tool, so you can wrap any build system
> >> you
> >>         want. The intention is here more to create a uniform API for
> >>         starting
> >>         the build, which may help packagers and other people who
> >>         routinely build
> >>         software.
> >>
> >>         Gerd
> >>
> >>             - jenga
> >>             - [...]
> >>
> >>             My preffered is obuild
> >>             (https://github.com/ocaml-__obuild/obuild
> >>             <https://github.com/ocaml-obuild/obuild>),
> >>             for the terseness, readability and centralization of its
> build
> >>             descriptions. I would love to see the user community of
> >>             obuild grow,
> >>             so that we can get rid of more bugs, be able to compile
> _any_
> >>             OCaml project with it and implement even more cool features
> >>             (contributors are very welcome).
> >>
> >>             I don't want a ring to rule them all, jut a ring that fits
> >>             _my_ finger. ;)
> >>
> >>             --
> >>             Regards,
> >>             Francois.
> >>
> >>
> >>
> >>     --
> >>     Caml-list mailing list.  Subscription management and archives:
> >>     https://sympa.inria.fr/sympa/__arc/caml-list
> >>     <https://sympa.inria.fr/sympa/arc/caml-list>
> >>     Beginner's list: http://groups.yahoo.com/group/__ocaml_beginners
> >>     <http://groups.yahoo.com/group/ocaml_beginners>
> >>     Bug reports: http://caml.inria.fr/bin/caml-__bugs
> >>     <http://caml.inria.fr/bin/caml-bugs>
> >>
> >>
> >
> > --
> > Regards,
> > Francois.
> >
> > --
> > Caml-list mailing list.  Subscription management and archives:
> > https://sympa.inria.fr/sympa/arc/caml-list
> > Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> > Bug reports: http://caml.inria.fr/bin/caml-bugs
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>

[-- Attachment #2: Type: text/html, Size: 13533 bytes --]

Re: [Caml-list] One build system to rule them all?

[Francois Berenger]

On 09/12/2014 04:08 PM, Yotam Barnoy wrote:
> OK so here's my summary of what I've heard here so far:
>
> - Ocaml does not yet have a (public) build system that the majority of
> users can be happy with for all of their project needs.
> - It seems difficult to have a build system that's simple enough to use
> for small projects, flexible enough to handle the more complicated
> projects, and that can scale.
> - While declarative, dsl-based tools are appealing, they get bogged down
> as you add features to them, making them become unwieldy.
> - A better solution seems to be using a real programming language to
> describe packages. In our case that should clearly be ocaml itself. The
> trick is making the API simple.

There is maybe an API in Haskell already:
https://hackage.haskell.org/package/shake

> Looking at some of the the available options:
> - ocamlbuild (dsl-based) is a decent basic choice, but lacks advanced
> features and has become weighed down by multiple file types (_tags,
> myocamlbuild etc).
> - OASIS (dsl-based) seems to be a good description layer for projects,
> currently farming out the difficult work of building stuff to
> ocamlbuild, but not really a contender in this race except as an
> abstraction layer.
> - omake (dsl-based) has some fans, but becomes overly-complex for more
> complicated projects.
> - ocp-build (dsl-based) has had nothing but criticism on this thread. Is
> there any ongoing work to address these criticisms?
> - Jenga (uses ocaml) seems promising but has heavy dependencies, and is
> currently not geared towards light projects.
> - assemblage (uses ocaml) seems promising as well, and seems focused on
> simplicity. Any comment from assemblage contributors on its status?
>
> Also, while I understand some of the windows hatred, the fact is, most
> of the world uses windows. Even using cygwin as a translation layer
> hurts ocaml's ability to reach people. Ideally, a windows programmer
> could download opam, ocaml and perhaps a mingw compiler, and begin to
> build projects without any need for a shell or anything POSIX-like. This
> means that the build system needs to obviate the need for any shell
> commands or makefiles.
>
> On Thu, Sep 11, 2014 at 6:37 AM, Yaron Minsky <yminsky@janestreet.com
> <mailto:yminsky@janestreet.com>> wrote:
>
>     On Thu, Sep 11, 2014 at 3:53 AM, Francois Berenger
>     <francois.berenger@inria.fr <mailto:francois.berenger@inria.fr>> wrote:
>      > On 09/10/2014 08:59 PM, Yotam Barnoy wrote:
>      >>
>      >> Here's part of my motivation for starting this discussion now: I
>      >> recently experienced a hard-drive failure and had to switch to
>     my backup
>      >> windows laptop. I was dismayed that opam is still not available for
>      >> Windows, and while wodi is available, it's not nearly as well
>     developed,
>      >> and apparently requires many hacks for different packages.
>      >>
>      >> It seems to me that these hacks are some of the things that make
>      >> platform compatibility hard for opam. Each build system has
>     different
>      >> requirements and methods of compilation, and making this approach
>      >> cross-platform compatible is difficult. Ideally, a build system will
>      >> abstract away the things that are not inter-platform compatible.
>     This is
>      >> why I don't like people using makefiles. Makefiles are not available
>      >> natively on Windows, and they often contain other bits of shell code
>      >> that isn't available on all POSIX platforms, let alone on
>     Windows. Not
>      >> to mention the fact that makefiles have a very tricky and sensitive
>      >> syntax.
>      >>
>      >> I don't expect one build system to match everyone's
>     requirements, but I
>      >> think if we get together, make a list of requirements, and try
>     to get
>      >> over our personal biases, we can find something that works for most
>      >> people, and with some group effort, can work for even more people in
>      >> more use-cases. I'm not suggesting that we invent something new, but
>      >> that we take something good and make it better, as well as make an
>      >> effort to learn that tool and convert packages to use that tool.
>      >>
>      >> So here are some requirements I can think of (using some of the
>      >> suggestions that have been brought up):
>      >> - Easy to use, especially for small projects (large projects can
>     afford
>      >> to put more time into their build systems)
>      >> - Abstract away platform considerations as much as possible. No
>      >> dependence on specific shells and POSIX utilities.
>      >> - Allows compilation of C files, which is quite common in ocaml
>     packages.
>      >> - Scalable to many directories and files
>      >> - Uses ocamlfind to locate packages
>      >> - Handles camlp4 and ppx
>      >> - Parallel & incremental compilation
>      >>
>      >> About Jenga: I took a quick look at Jenga, and even though I'm
>     impressed
>      >> by its capabilities, the amount of code needed to be written
>     even for
>      >> simple projects is overwhelming. It's clearly a very flexible and
>      >> powerful tool, but I'd say it's too flexible for the mainstream.
>     There's
>      >> always room for a build system that's integrated into ocaml
>     itself (like
>      >> Shake), but I think declarative build systems tend to be easier to
>      >> comprehend for the average user.
>      >>
>      >> ocp-build actually looks very interesting. The manual (which is
>     here:
>      >>
>      >>
>     http://github.com/OCamlPro/ocp-build/blob/master/docs/user-manual/user-manual.pdf?raw=true)
>      >> is incomplete, but contains a nice survey of the existing build
>     tools,
>      >> and motivation for making ocp-build. Has anyone had experience with
>      >> ocp-build? Opam seems to be using it, but they also use a makefile
>      >> (why?) with a bunch of shell commands inside (which is precisely the
>      >> problem from my perspective). ocp-build is supposedly compatible
>     with
>      >> Windows, too.
>      >
>      >
>      > Here is one thing I dislike in ocp-build: you have to explicitely
>     list all
>      > the source files (and maybe same thing for Jenga).
>
>     Jenga is quite flexible, so you could write your jengaroot either way
>     (to demand or not demand explicit source file lists.)  Our internal
>     jengaroot does not require explicit source file lists.
>
>     Not to confuse people, though, Jenga isn't really suitable for people
>     just picking up and using yet.  We haven't yet written a suitable
>     jengaroot for general use yet (though there is a version of our
>     internal jengaroot that we've released, but it's not yet ready for
>     prime time.)  So Jenga is interesting as an option to discuss, but not
>     yet a practical option we want to encourage casual users to try.
>
>      > I am way too lazy to do that, especially since other tools are able
>      > to infer this list automatically (a library is just a list of
>     modules,
>      > an executable has a single entry point, the main function in a
>     single file).
>      >
>      >> Yotam
>      >>
>      >>
>      >> On Wed, Sep 10, 2014 at 11:17 AM, Leonardo Laguna Ruiz
>     <modlfo@gmail.com <mailto:modlfo@gmail.com>
>      >> <mailto:modlfo@gmail.com <mailto:modlfo@gmail.com>>> wrote:
>      >>
>      >>     My pick is ocamlbuild because:
>      >>
>      >>     - It works the same way in all platforms that I work (linux,osx,
>      >>     windows msvc port, cygwin)
>      >>     - If I can compile ocaml, then I have ocamlbuild
>      >>     - It does not require external libraries
>      >>
>      >>     I have to say that I feel a little bit annoyed by the fact
>     that some
>      >>     other build systems or tools, do not work correctly in all the
>      >>     platforms that I want to support.
>      >>
>      >>     Leonardo
>      >>
>      >>
>      >>
>      >>     On 9/10/2014 4:23 PM, Gerd Stolpmann wrote:
>      >>
>      >>         Am Mittwoch, den 10.09.2014, 15:29 +0200 schrieb Francois
>      >> Berenger:
>      >>
>      >>             More seriously, concerning build systems, we clearly
>     have
>      >>             quite some
>      >>             choice on the OCaml shelf:
>      >>             - obuild
>      >>             - ocamlbuild
>      >>             - omake
>      >>             - oasis (which in fact uses ocamlbuild, don't forget
>     that)
>      >>
>      >>         oasis is not a build system. It is a package description
>     format
>      >> that
>      >>         describes how to invoke the build (and more). So far
>     there is only
>      >>         built-in knowledge for ocamlbuild, but this may change,
>     and you
>      >> can
>      >>         already call any external tool, so you can wrap any
>     build system
>      >> you
>      >>         want. The intention is here more to create a uniform API for
>      >>         starting
>      >>         the build, which may help packagers and other people who
>      >>         routinely build
>      >>         software.
>      >>
>      >>         Gerd
>      >>
>      >>             - jenga
>      >>             - [...]
>      >>
>      >>             My preffered is obuild
>      >>             (https://github.com/ocaml-__obuild/obuild
>      >>             <https://github.com/ocaml-obuild/obuild>),
>      >>             for the terseness, readability and centralization of
>     its build
>      >>             descriptions. I would love to see the user community of
>      >>             obuild grow,
>      >>             so that we can get rid of more bugs, be able to
>     compile _any_
>      >>             OCaml project with it and implement even more cool
>     features
>      >>             (contributors are very welcome).
>      >>
>      >>             I don't want a ring to rule them all, jut a ring
>     that fits
>      >>             _my_ finger. ;)
>      >>
>      >>             --
>      >>             Regards,
>      >>             Francois.
>      >>
>      >>
>      >>
>      >>     --
>      >>     Caml-list mailing list.  Subscription management and archives:
>      >> https://sympa.inria.fr/sympa/__arc/caml-list
>      >>     <https://sympa.inria.fr/sympa/arc/caml-list>
>      >>     Beginner's list: http://groups.yahoo.com/group/__ocaml_beginners
>      >>     <http://groups.yahoo.com/group/ocaml_beginners>
>      >>     Bug reports: http://caml.inria.fr/bin/caml-__bugs
>      >>     <http://caml.inria.fr/bin/caml-bugs>
>      >>
>      >>
>      >
>      > --
>      > Regards,
>      > Francois.
>      >
>      > --
>      > Caml-list mailing list.  Subscription management and archives:
>      > https://sympa.inria.fr/sympa/arc/caml-list
>      > Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>      > Bug reports: http://caml.inria.fr/bin/caml-bugs
>
>     --
>     Caml-list mailing list.  Subscription management and archives:
>     https://sympa.inria.fr/sympa/arc/caml-list
>     Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>     Bug reports: http://caml.inria.fr/bin/caml-bugs
>
>

-- 
Regards,
Francois.

Re: [Caml-list] One build system to rule them all?

[Anil Madhavapeddy]

[-- Attachment #1: Type: text/plain, Size: 683 bytes --]

On 12 Sep 2014, at 15:08, Yotam Barnoy <yotambarnoy@gmail.com> wrote:

> - assemblage (uses ocaml) seems promising as well, and seems focused on simplicity. Any comment from assemblage contributors on its status?

It's being actively developed, is still unreleased, and the build API is still subject to change.

Early adopters and contributors are most welcome to get involved and give feedback, as long as you understand the above caveats that it's a moving target.

I'd suggest reading through some of the discussions and pull requests on https://github.com/samoht/assemblage/issues to get a sense of the outstanding tasks left before a public beta release.

best,
Anil

[-- Attachment #2: Type: text/html, Size: 1492 bytes --]

Re: [Caml-list] One build system to rule them all?

[SF Markus Elfring]

> Looking at some of the the available options:

How do you think about the other software that is also usable for this
application domain?
https://en.wikipedia.org/wiki/List_of_build_automation_software#Comparison_of_build_automation_software

Regards,
Markus

Re: [Caml-list] One build system to rule them all?

[Adrien Nader]

On Fri, Sep 12, 2014, Yotam Barnoy wrote:
> Also, while I understand some of the windows hatred, the fact is, most of
> the world uses windows. Even using cygwin as a translation layer hurts
> ocaml's ability to reach people. Ideally, a windows programmer could
> download opam, ocaml and perhaps a mingw compiler, and begin to build
> projects without any need for a shell or anything POSIX-like. This means
> that the build system needs to obviate the need for any shell commands or
> makefiles.

I'm worried it my message which sounded like Windows hatred.
I don't hate Windows, really. I hate cmake and qt however because the
design they _chose_ was already a step back more than a decade ago. It's
similar to advocating static linking despite the current needs to
regularly update libraries because of security issues.
(NB: similar feelings towards people who do not use ocamlfind; hopefully
they're very uncommon nowadays)

As for using only cmd.exe. Well. You probably don't know cmd.exe. You
probably don't know how command-line argument parsing works on Windows.
I can't believe you would and not do your best to avoid them.
The main issue is that the newly-launched process does not receive its
arguments as an array of strings. Instead it receives them as a string.
When you call execv() on windows, it concatenates the string array
you've given and it will be the new process which will have to split
them back into a "char* []" for use with main().
So you have to prepare your arguments. But how? According to the spec of
course! Ah, easy! Except there is no real logic behind it: it's
completely ad-hoc stuff and while probably doable with only a few lines
of code (probably as many as this email), it's looks error-prone (and,
let's face it, boring). Oh and, it might well break compat and the issue
has been in OCaml since 1994.

This is the reason ocamlbuild depends on an sh.exe from msys/cygwin btw
(I'm actually wondering if cygwin's is fit for this task...).
It will receive a single string of arguments, parse it like it would do
on POSIX; in other words, its only purpose is to get back to well-known
and manageable parsing rules.

To be honest, OCamlbuild also has a large tendency to shell-out but in
90% of the cases, it's not a big issue although re-implementing "cp -r"
is annoying (did someone mention that if ocamlbuild moves out of the
compiler, it can depend on Fileutils which implements this kind of
operations?).

That said, ocamlbuild works well on Windows because besides these few
shell-outs, it's written in OCaml.

Btw, there are open bug reports on mantis about that. If you're
interested in getting these fixed, the links to the docs are already
there and the work is definitely manageable so speak up.

Btw2, there's a fairly recent "zsh for windows" (last release from 2011
or 2012) and which, if it works, and if it works for our use-case, could
maybe be integrated in the ocaml installers and could simplify the
process.
(note the number of "ifs" and conditionals used above)


In any case, if you plan on integrating with something else, and you
will, unless you don't want to use any C stubs, including the ones for
the OS, you will have to understand and integrate into the existing
C-building setups which already exist one way or another.


-- 
Adrien Nader

[Caml-list] Re : One build system to rule them all?

[r.3]

a. Is there any build system we can organize behind to crown as the official build system?
b. What are the use-cases missing from specific build systems, that have driven people to use other build systems?
c. To pick one possible candidate, if ocamlbuild were spun out of the compiler, could it be enhanced to cover all the main use-cases so (almost) everyone would be happy with it? 


a) Yes, ocamlbuild is official.
b) I would rather see the following question : has someone abandonned ocamlbuild, and why ?
c) If ocamlbuild were spun out of the compiler, then it would not be official anymore ? So it would fail a) ?

Re: [Caml-list] One build system to rule them all?

[Yaron Minsky]

Jenga, as I said, isn't currently ready for wide use.  But I do think
it's a very promising tool.  A few notes:

- Easy vs Sophisticated

  We think Jenga provides a pretty good balance between easy and
  sophisticated.  Even though its core is quite powerful and so not
  trivial to use, it is also amenable to using with other, simpler
  build-declaration systems.

  We ourselves use a simple s-expression config format for specifying
  individual libraries, and there has been some talk of integrating
  Assemblage into Jenga eventually as another such simple DSL.  We've
  even done some work making Jenga compile from Ninja build
  descriptions.

- Bootstrapping

  One concern with Jenga is boot-strapping, or other ways of exporting
  packages to be built where Jenga itself is not available.

  We have a pretty good solution to this in flight, and after talking
  with Neil Mitchell, he's going to try the same trick out with Shake.
  The idea is to be able to generate makefiles that can be used as
  part of a package.

  Systems like Jenga and Shake do well because they have rich, monadic
  languages for expressing complex dependencies that are discovered
  during a build.  But we can use Jenga to do enough of the build to
  discover the dependencies, and then export a makefile reflecting
  those dependencies.  This should make it possible to build all of
  Jenga's dependencies using make, and indeed for any Jenga package to
  be exported in a portable and dependency-free way.

  (Windows make should be supportable this way too.)

- Dependencies

  We're working on lightening the dependencies of Jenga.  We are
  planning to release a new version of Async in a couple of months
  whose base (called Async_kernel) no longer depends on Core (which
  depends on some Unix-isms) and instead only depends on the (highly
  portable) Core_kernel.

  That should bring us a good step closer to having Windows support.
  I think we'd still need some work, in particular, someone to write
  better windows support for Async, perhaps on top of libuv.  But once
  we release our cleaned up async, I think that's more approachable.

In the short term, I'm quite hopeful that Assemblage will provide
something that's easy to use right now.  But I think Jenga is an
important part of the longer term plan for OCaml builds.

y

On Fri, Sep 12, 2014 at 10:36 AM, Anil Madhavapeddy <anil@recoil.org> wrote:
> On 12 Sep 2014, at 15:08, Yotam Barnoy <yotambarnoy@gmail.com> wrote:
>
> - assemblage (uses ocaml) seems promising as well, and seems focused on
> simplicity. Any comment from assemblage contributors on its status?
>
>
> It's being actively developed, is still unreleased, and the build API is
> still subject to change.
>
> Early adopters and contributors are most welcome to get involved and give
> feedback, as long as you understand the above caveats that it's a moving
> target.
>
> I'd suggest reading through some of the discussions and pull requests on
> https://github.com/samoht/assemblage/issues to get a sense of the
> outstanding tasks left before a public beta release.
>
> best,
> Anil

Re: [Caml-list] One build system to rule them all?

[Fabrice Le Fessant]

[-- Attachment #1: Type: text/plain, Size: 560 bytes --]

On Fri, Sep 12, 2014 at 4:08 PM, Yotam Barnoy <yotambarnoy@gmail.com> wrote:

> - ocp-build (dsl-based) has had nothing but criticism on this thread. Is
> there any ongoing work to address these criticisms?
>

Yes, ocp-build is still alive. I would not advise anybody to use it for
anything complicated (no syntax extension, only predefined compilation
rules for .ml/.mli/.mll and .mly). My plan is to rewrite the DSL to
introduce better abstractions to allow cross-compilation, composable
package configuration and a better handling of META files.

--Fabrice

[-- Attachment #2: Type: text/html, Size: 910 bytes --]

Re: [Caml-list] One build system to rule them all?

[rixed]

> Every single makefile to compile ocaml (except OCaml-makefile[1]) I
> have encountered are broken.

Please, define broken.

Re: [Caml-list] One build system to rule them all?

[Drup]

Le 13/09/2014 14:01, rixed@happyleptic.org a écrit :
>> Every single makefile to compile ocaml (except OCaml-makefile[1]) I
>> have encountered are broken.
> Please, define broken.
>
>
Pick several:
- Work only with native or byte compilation, but not both (especially 
common for C bindings).
- Don't work with dynlink.
- Don't build some cmFOO or link the right things in the .so/.a, which 
is usually the cause for the two above (but not always, people are 
creative sometime).
- Don't offer anything to build documentation.
- Don't use ocamlfind correctly (and install things in the wrong places).
- Worse: Don't use ocamlfind at all.
- Don't have an uninstall target.
- Break under -j (ocamlbuild parallelize badly, but at least, it doesn't 
break)
- Assume linux (and GNU Make) (ask adrien about the compiler's build 
system for tasty war stories)

I could go on, but I'm pretty you got the point.

Re: [Caml-list] One build system to rule them all?

[rixed]

Ok, here are my requirements:

- be able to compile any language (personal favorites: OCaml, C)
- be able to run a preprocessor (personal favorites: a tool to filter OCaml v4
  only library functions or variants)
- be able to build the preprocessor
- be able to generate the code (personal favorite: literate programming)
- be able to build the tools that generates the code (personal favorite: ibid)
- be able to generate the data from other input files (personal favorite: true-type
  fonts generated from description)
- be able to build the tools that does that (of course)
- be able to generate the doc
- be able to run some tool on some source files and check some property (personal
  favorite: a tool to check circular dependencies)
- be able to build the tools that perform those checks
- be able to use custom ld script to link to exotic environments (personal favorite:
  phones)
- be reasonably portable to unix like environments
- don't trap users into using only one language or one tool.

Well this is a never ending list, unless you realize that it all sums up to:

- be able to run any command to change one set of input files to some output file
- be declarative as much as possible

I don't believe there can be anything between generic make-like tools and
community centered tools that simplify but one use case.
<insert fish vs fishing allegory here>.

Re: [Caml-list] One build system to rule them all?

[rixed]

-[ Sat, Sep 13, 2014 at 02:21:53PM +0200, Drup ]----
> >Please, define broken.
> Pick several:
> (...)
> I could go on, but I'm pretty you got the point.

Yes, the point was that you tried to build things that were not supported (doc,
dynlink) on architectures the author didn't plan to support, or to do things
outside of building (installing, deinstalling) and many time it failed.

I reckon that "make" is a presumptious name that may leads to wrong
expectations :-)

Was it hard to fix the makefiles?

Re: [Caml-list] One build system to rule them all?

[Adrien Nader]

On Sat, Sep 13, 2014, rixed@happyleptic.org wrote:
> -[ Sat, Sep 13, 2014 at 02:21:53PM +0200, Drup ]----
> > >Please, define broken.
> > Pick several:
> > (...)
> > I could go on, but I'm pretty you got the point.
> 
> Yes, the point was that you tried to build things that were not supported (doc,
> dynlink) on architectures the author didn't plan to support, or to do things
> outside of building (installing, deinstalling) and many time it failed.
> 
> I reckon that "make" is a presumptious name that may leads to wrong
> expectations :-)
> 
> Was it hard to fix the makefiles?

We could go a bit further and say that assembly, brainfuck and malbolge
are not proper languages for build system and that only the
implementations are wrong.

Sounds satiric? Maybe a bit bit.
But I regularly hear the same for C and buffer overflows: it's not the
language, it's the code. Not wrong but...

The point is that in any case, if you want to make your makefile from
scratch, you're going to be missing some things. OCamlMakefile is a few
thousands lines; you can be sure that you're going to be missing stuff
in your 30 lines Makefile.
And it is not a packager's role to rewrite build systems for features
(there's already enough to do with bugs).

There's a funny game though (at least I find it funny; people say it
makes me sound bitter): you show a custom build system, I find practical
issues with it. To be honest it's probably not a good thing to start on
the ML.


Also, Drup got things a bit wrong: the build system of the compiler uses
makefiles and it works most of the time. As far as I know, people still
get issues when building in parallel; less than before but it appears to
not be reliable.
The main issues with the compiler build system are: age, size,
duplication between directories and windows/unix.

Re: [Caml-list] One build system to rule them all?

[Drup]

Le 13/09/2014 14:37, rixed@happyleptic.org a écrit :
> -[ Sat, Sep 13, 2014 at 02:21:53PM +0200, Drup ]----
>>> Please, define broken.
>> Pick several:
>> (...)
>> I could go on, but I'm pretty you got the point.
> Yes, the point was that you tried to build things that were not supported (doc,
> dynlink) on architectures the author didn't plan to support, or to do things
> outside of building (installing, deinstalling) and many time it failed.
The point is that It's supposed to be supported by default in any 
reasonable build system (like those named in this thread, for example).

> I reckon that "make" is a presumptious name that may leads to wrong
> expectations :-)
>
> Was it hard to fix the makefiles?
I usually don't even bother trying, it's easier to replace the whole 
build system by something else.

Re: [Caml-list] One build system to rule them all?

[Richard W.M. Jones]

On Wed, Sep 10, 2014 at 08:49:06AM -0400, Yotam Barnoy wrote:
> b. What are the use-cases missing from specific build systems, that have
> driven people to use other build systems?

In a non-OCaml-centric world you need to fit into other projects.  The
virt-* tools are a good example here, since they are mostly written in
C, and use automake/autoconf.  And the OCaml virt tools have to link
with C and so are also built using only autotools.

Example of mixed C / OCaml program built using autotools:

https://github.com/libguestfs/libguestfs/tree/master/v2v

> c. To pick one possible candidate, if ocamlbuild were spun out of the
> compiler, could it be enhanced to cover all the main use-cases so (almost)
> everyone would be happy with it?

ocamlbuild is horribly obscure.

Rich.

-- 
Richard Jones
Red Hat

Re: [Caml-list] One build system to rule them all?

[Richard W.M. Jones]

On Fri, Sep 12, 2014 at 10:08:14AM -0400, Yotam Barnoy wrote:
> Also, while I understand some of the windows hatred, the fact is, most of
> the world uses windows.

Android is expected to ship on a billion devices this year.  Is there
even a build system out of the ones we've discussed that works on
Android?  (Maybe ocamlbuild ..?)

Rich.

-- 
Richard Jones
Red Hat

Re: [Caml-list] One build system to rule them all?

[Stéphane Glondu]

Le 10/09/2014 20:59, Yotam Barnoy a écrit :
> So here are some requirements I can think of (using some of the
> suggestions that have been brought up):
> - Easy to use, especially for small projects (large projects can afford
> to put more time into their build systems)
> - Abstract away platform considerations as much as possible. No
> dependence on specific shells and POSIX utilities.
> - Allows compilation of C files, which is quite common in ocaml packages.
> - Scalable to many directories and files
> - Uses ocamlfind to locate packages
> - Handles camlp4 and ppx
> - Parallel & incremental compilation

 - Support of platforms without ocamlopt. Many build systems assume that
ocamlopt is available and have to be called specially (or even patched)
when it is missing, which complicates packaging on these platforms.

-- 
Stéphane

Re: [Caml-list] One build system to rule them all?

[Yotam Barnoy]

[-- Attachment #1: Type: text/plain, Size: 2565 bytes --]

While there was no 'conclusion' to this thread, if I had to come up with
one, it would be that we have a bunch of build tools which are all not
amazing at this point in time. We have some DSL-based build tools and some
ocaml-based build tools, and all of them need a lot of love to get to a
good state.

My personal view is that we (as a community) should work at getting at
least one DSL tool to be really great. I'm sure Jenga (an ocaml-based tool
which seems more like a build-tool engine) will continue to be developed by
Jane Street no matter what, so is there a DSL-based build system that we
can converge on to use and improve? The contenders for this slot appear to
be omake, obuild, and ocp-build. I'm more than willing to switch to one of
these if I know that other people will as well, and that it will be
actively developed (preferably on github). More users = more invested
parties = more development potential. Conversely, continuing to spread the
community's attention between these tools (as well as ocamlbuild, which
seems destined to stagnate) before any one of them is top notch seems to me
to be detrimental to ocaml's health as an ecosystem.

BTW Anil: is assemblage supposed to be an entire build toolchain, or is it
only supposed to write makefiles (as the description in the readme states)?

-Yotam

On Mon, Sep 15, 2014 at 9:34 AM, Stéphane Glondu <steph@glondu.net> wrote:

> Le 10/09/2014 20:59, Yotam Barnoy a écrit :
> > So here are some requirements I can think of (using some of the
> > suggestions that have been brought up):
> > - Easy to use, especially for small projects (large projects can afford
> > to put more time into their build systems)
> > - Abstract away platform considerations as much as possible. No
> > dependence on specific shells and POSIX utilities.
> > - Allows compilation of C files, which is quite common in ocaml packages.
> > - Scalable to many directories and files
> > - Uses ocamlfind to locate packages
> > - Handles camlp4 and ppx
> > - Parallel & incremental compilation
>
>  - Support of platforms without ocamlopt. Many build systems assume that
> ocamlopt is available and have to be called specially (or even patched)
> when it is missing, which complicates packaging on these platforms.
>
> --
> Stéphane
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>

[-- Attachment #2: Type: text/html, Size: 3360 bytes --]

Re: [Caml-list] One build system to rule them all?

[Anil Madhavapeddy]

On 18 Sep 2014, at 22:15, Yotam Barnoy <yotambarnoy@gmail.com> wrote:

> BTW Anil: is assemblage supposed to be an entire build toolchain, or is it only supposed to write makefiles (as the description in the readme states)?

The README in Assemblage is accurate:
https://github.com/samoht/assemblage/blob/master/README.md

> The library can be used to describe OCaml projects with static dependencies.
> 
> [...]
> 
> A projects description consists of a list of libraries, binaries and/or toplevels. Each containing a list of compilation units with precise (and static) dependency relationships.
> 
> Using a project description, the tools can generate:
> 
> 	• a .merlin to help edit the project;
> 	• a Makefile to build the project;
> 	• a .install to install the project; and
> 	• a META file to use the project.

While none currently exist, other tools could use the same
description API to generate other targets (such as OMake or Jenga)
files.  This is best done after the existing APIs stabilise, which
is ongoing.

best,
Anil

Re: [Caml-list] One build system to rule them all?

[Yaron Minsky]

I think the conclusion is that we're not "as a community" going to
settle in on a single solution before there's one that's ready.
There's a lot of work on competing approaches, which I think is
healthy.  Even those people who are working on competing approaches
are cooperating pretty effectively (e.g., Thomas is working on
Assemblage and at the same time looking into Jenga to see where the
two can connect.)

Rather than exhort everyone to focus in on a solution, I recommend you
pick the one you think looks best, and see if you can contribute to
it.

I think consensus should follow code, not the other way around.

y

On Thu, Sep 18, 2014 at 5:15 PM, Yotam Barnoy <yotambarnoy@gmail.com> wrote:
> While there was no 'conclusion' to this thread, if I had to come up with
> one, it would be that we have a bunch of build tools which are all not
> amazing at this point in time. We have some DSL-based build tools and some
> ocaml-based build tools, and all of them need a lot of love to get to a good
> state.
>
> My personal view is that we (as a community) should work at getting at least
> one DSL tool to be really great. I'm sure Jenga (an ocaml-based tool which
> seems more like a build-tool engine) will continue to be developed by Jane
> Street no matter what, so is there a DSL-based build system that we can
> converge on to use and improve? The contenders for this slot appear to be
> omake, obuild, and ocp-build. I'm more than willing to switch to one of
> these if I know that other people will as well, and that it will be actively
> developed (preferably on github). More users = more invested parties = more
> development potential. Conversely, continuing to spread the community's
> attention between these tools (as well as ocamlbuild, which seems destined
> to stagnate) before any one of them is top notch seems to me to be
> detrimental to ocaml's health as an ecosystem.
>
> BTW Anil: is assemblage supposed to be an entire build toolchain, or is it
> only supposed to write makefiles (as the description in the readme states)?
>
> -Yotam
>
> On Mon, Sep 15, 2014 at 9:34 AM, Stéphane Glondu <steph@glondu.net> wrote:
>>
>> Le 10/09/2014 20:59, Yotam Barnoy a écrit :
>> > So here are some requirements I can think of (using some of the
>> > suggestions that have been brought up):
>> > - Easy to use, especially for small projects (large projects can afford
>> > to put more time into their build systems)
>> > - Abstract away platform considerations as much as possible. No
>> > dependence on specific shells and POSIX utilities.
>> > - Allows compilation of C files, which is quite common in ocaml
>> > packages.
>> > - Scalable to many directories and files
>> > - Uses ocamlfind to locate packages
>> > - Handles camlp4 and ppx
>> > - Parallel & incremental compilation
>>
>>  - Support of platforms without ocamlopt. Many build systems assume that
>> ocamlopt is available and have to be called specially (or even patched)
>> when it is missing, which complicates packaging on these platforms.
>>
>> --
>> Stéphane
>>
>> --
>> Caml-list mailing list.  Subscription management and archives:
>> https://sympa.inria.fr/sympa/arc/caml-list
>> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>> Bug reports: http://caml.inria.fr/bin/caml-bugs
>
>

Re: [Caml-list] One build system to rule them all?

[Yaron Minsky]

On Thu, Sep 18, 2014 at 5:21 PM, Anil Madhavapeddy <anil@recoil.org> wrote:
> On 18 Sep 2014, at 22:15, Yotam Barnoy <yotambarnoy@gmail.com> wrote:
>
>> BTW Anil: is assemblage supposed to be an entire build toolchain, or is it only supposed to write makefiles (as the description in the readme states)?
>
> The README in Assemblage is accurate:
> https://github.com/samoht/assemblage/blob/master/README.md
>
>> The library can be used to describe OCaml projects with static dependencies.
>>
>> [...]
>>
>> A projects description consists of a list of libraries, binaries and/or toplevels. Each containing a list of compilation units with precise (and static) dependency relationships.
>>
>> Using a project description, the tools can generate:
>>
>>       • a .merlin to help edit the project;
>>       • a Makefile to build the project;
>>       • a .install to install the project; and
>>       • a META file to use the project.
>
> While none currently exist, other tools could use the same
> description API to generate other targets (such as OMake or Jenga)
> files.  This is best done after the existing APIs stabilise, which
> is ongoing.

Generating Jenga rules or OMake rules I think is note quite the right
approach.  That's because Jenga and OMake both use general purpose
programming languages for specifying builds, and I think you'll get
bad behavior if you try to treat that language as a compilation
target.

For Jenga, I think it's better to write a Jenga plug-in for reading
Assemblage files.  It has the same effect, but is an easier and more
typeful programming experience than spitting out a bunch of OCaml AST.

We already have a toy plugin for Ninja build rules, and I believe
others could be added pretty naturally.

Shake, for what it's worth, takes a similar approach (and has a Ninja
plugin too), and I think it works well there also.

y

> best,
> Anil
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs

Re: [Caml-list] One build system to rule them all?

[Gabriel Scherer]

[-- Attachment #1: Type: text/plain, Size: 6362 bytes --]

On Thu, Sep 18, 2014 at 11:23 PM, Yaron Minsky <yminsky@janestreet.com>
wrote:

> Rather than exhort everyone to focus in on a solution, I recommend you
> pick the one you think looks best, and see if you can contribute to
> it.
>

Agreed.

My very personal point of focus is to help current ocamlbuild users (there
are a fair number of them)

is there a DSL-based build system that we can converge on to use and
> improve?
>

I think this distinction between ocaml-using and DSL-using build systems
does not matter in the long run. It matters when you look at a tool at a
given point in time, but here you are discussing the long-term evolution of
the tooling, and those distinctions get blurred assuming enough
contributions:
1) you can turn an ocaml-using build system into a DSL-using build system
by having an input mode that take DSL descriptions instead of rules, and
interpreting them in term of those rules
2) a well-designed DSL-based system has an underlying library encoding the
tool's engines, and exposing this library to advanced users effectively
turns it into an ocaml-using tool as well

Conversely, continuing to spread the community's attention between these
> tools (as well as ocamlbuild, which seems destined to stagnate) before any
> one of them is top notch seems to me to be detrimental to ocaml's health as
> an ecosystem.


I haven't seen any good reason in this thread justifying ocamlbuild's
stagnation (except, of course, the fact that few people are currently
interested in working on that; but I thought you tried to abstract from
this aspect):
- people have mentioned having to edit several files as a source of
annoyance with ocamlbuild; it would be rather easy to work on having a file
with sub-sections corresponding to several of today's configuration file
- the disappointing parallelism has been mentioned as a problem with
ocamlbuild's implementation today -- it is a problem. I have discussed a
practical approach to improving parallelism in (
http://gallium.inria.fr/blog/ocamlbuild-parallelization/ ), which would
require a bit more work but is not out of reach. For example that would be
a reasonable Google Summer of Code (GSoC) topic, had the community
succeeded in establishing a presence at GSoC.

Note that I am not particularly attached to ocamlbuild in its current
state; I care about its users to which it provides good service and should
not be left in the dust.

Jenga's design is in fact surprisingly close to ocamlbuild's, and one thing
I have been thinking about is trying to reimplement ocamlbuild as a kind of
"frontend" on top of Jenga's implementation -- the goal being the maintain
compatibility with existing ocamlbuild setups, and gain some of the
benefits of Jenga's engineering resources, namely arguably better
parallelization and support continuous build. That's a rather bold move,
there would be many less invasive changes that could have a positive impact
on users -- for example, turning a succesful build into a Makefile should
be doable in ocamlbuild as well, and interesting for many reasons
(bare-boneness, performances).



> I think consensus should follow code, not the other way around.
>
> y
>
> On Thu, Sep 18, 2014 at 5:15 PM, Yotam Barnoy <yotambarnoy@gmail.com>
> wrote:
> > While there was no 'conclusion' to this thread, if I had to come up with
> > one, it would be that we have a bunch of build tools which are all not
> > amazing at this point in time. We have some DSL-based build tools and
> some
> > ocaml-based build tools, and all of them need a lot of love to get to a
> good
> > state.
> >
> > My personal view is that we (as a community) should work at getting at
> least
> > one DSL tool to be really great. I'm sure Jenga (an ocaml-based tool
> which
> > seems more like a build-tool engine) will continue to be developed by
> Jane
> > Street no matter what, so is there a DSL-based build system that we can
> > converge on to use and improve? The contenders for this slot appear to be
> > omake, obuild, and ocp-build. I'm more than willing to switch to one of
> > these if I know that other people will as well, and that it will be
> actively
> > developed (preferably on github). More users = more invested parties =
> more
> > development potential. Conversely, continuing to spread the community's
> > attention between these tools (as well as ocamlbuild, which seems
> destined
> > to stagnate) before any one of them is top notch seems to me to be
> > detrimental to ocaml's health as an ecosystem.
> >
> > BTW Anil: is assemblage supposed to be an entire build toolchain, or is
> it
> > only supposed to write makefiles (as the description in the readme
> states)?
> >
> > -Yotam
> >
> > On Mon, Sep 15, 2014 at 9:34 AM, Stéphane Glondu <steph@glondu.net>
> wrote:
> >>
> >> Le 10/09/2014 20:59, Yotam Barnoy a écrit :
> >> > So here are some requirements I can think of (using some of the
> >> > suggestions that have been brought up):
> >> > - Easy to use, especially for small projects (large projects can
> afford
> >> > to put more time into their build systems)
> >> > - Abstract away platform considerations as much as possible. No
> >> > dependence on specific shells and POSIX utilities.
> >> > - Allows compilation of C files, which is quite common in ocaml
> >> > packages.
> >> > - Scalable to many directories and files
> >> > - Uses ocamlfind to locate packages
> >> > - Handles camlp4 and ppx
> >> > - Parallel & incremental compilation
> >>
> >>  - Support of platforms without ocamlopt. Many build systems assume that
> >> ocamlopt is available and have to be called specially (or even patched)
> >> when it is missing, which complicates packaging on these platforms.
> >>
> >> --
> >> Stéphane
> >>
> >> --
> >> Caml-list mailing list.  Subscription management and archives:
> >> https://sympa.inria.fr/sympa/arc/caml-list
> >> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> >> Bug reports: http://caml.inria.fr/bin/caml-bugs
> >
> >
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>

[-- Attachment #2: Type: text/html, Size: 8365 bytes --]

Re: [Caml-list] One build system to rule them all?

[r.3]

[-- Attachment #1: Type: text/plain, Size: 1263 bytes --]

Hello, 

Some time ago I set up a cross tool chain from linux to windows. The project was using lablgtk2, camlimages, xml-light, lablgl and cairo. None of these packages were ready for cross-compilation, and I had to patch them. 
My experience : 
Makefile (small ones) : was ok 
oasis (for my own packages) : was not possible to cross-compile 
ocamlbuild through Makefiles (for my own packages) : ok 
autotools + ocaml-autoconf + Makefiles (lablgtk2, cairo) : my preferred 
oMake (camlimages) : seemed obscure to me, and did not succeeded in patching it. In the end, I reverted to a previous version of camlimages using autotools + ocaml-autoconf + Makefiles 

In the end, to my point of view, I found ocamlbuild to be nice for small to medium ocaml projects, though documentation was bad. Then, when C sources were required with cross compilation, or for bigger projects, I found autotools to be nice. 

Finally, I also tend to think that the best tool is the one used by most people :-) 

Just to get fun, I designed two polls : 

"best tool" : 

http://vote.pollcode.com/55577581#sthash.7DIWIfiP.dpuf http://vote.pollcode.com/55577581 

"most used tool" : 
http://vote.pollcode.com/114772714 

William 


http://vote.pollcode.com/55577581#sthash.7DIWIfiP.dpuf 

[-- Attachment #2: Type: text/html, Size: 2493 bytes --]

Re: [Caml-list] One build system to rule them all?

[Matej Kosik]

On 10/09/14 20:13, Drup wrote:
> Le 10/09/2014 16:17, Maxence Guesdon a écrit :
>> I can't resist to add good old Make to the list, not specific to OCaml,
>> yet powerful enough and known by a lot of developers.
>>
> No.
> Every single makefile to compile ocaml (except OCaml-makefile[1]) I have encountered are broken. Every single one of them. OCaml compilation process is too complex to be able to fire some rules and be
> done with it; especially if you want portability.
> 
> Please, don't write makefiles.

Non sequitur.

Re: [Caml-list] One build system to rule them all?

[Daniel Bünzli]

Le jeudi, 18 septembre 2014 à 23:36, Yaron Minsky a écrit :
> Generating Jenga rules or OMake rules I think is note quite the right
> approach. That's because Jenga and OMake both use general purpose
> programming languages for specifying builds, and I think you'll get
> bad behavior if you try to treat that language as a compilation
> target.
>  
> For Jenga, I think it's better to write a Jenga plug-in for reading
> Assemblage files. It has the same effect, but is an easier and more
> typeful programming experience than spitting out a bunch of OCaml AST.

Yes that seems much more sensitive. I'm currently working on the API and everything will be exposed to allow you to do that. We'll just need to figure out a way so that you can easily take a handle on the project description value.

Usually I don't comment or advertise unfinished things but I'd also like to point out that the goal with assemblage is also a little bit larger than just generating a build system. I'd like to be able to express most of the package release and installation process inside it (or at least provide helpers to do so) in order to replace all the ad-hoc scripts that is use now (e.g. topkg), to make quick and flawless releases by gathering information from the right places, tags in the vcs for version information, synchronizing opam file with correct package dependencies, opam-submit integration etc.  

Some people do find my release process complex, I basically apply a function on a git checkout and do ugly things like a global key value substitution on all the files, but at least it is flawless and traceable. For example package never lie to you about their current version, even if you pin them. The only way to get that flawlessly is to automate it — can't count the number of times opam lied to me about its version because its version number needs to be bumped manually.

Last year I tried the other new build systems and to influence them in that direction (e.g. generating .install files) without much success. I guess most people don't understand these problem as they only publish one package or two and can afford to spend some time to do their releases or hack an install process. I can't. That being said, I think we should not rush on that one, it will be ready when it is ready, (I personally soon won't have the time to work on this as I'll have to earn my life for the winter, I can't speak for Thomas though).

> I think consensus should follow code, not the other way around.
Yes. That whole discussion is largely pointless. Some people don't get that healthy online communities self-organize.

Daniel

Re: [Caml-list] One build system to rule them all?

[Anil Madhavapeddy]

On 19 Sep 2014, at 13:31, Daniel Bünzli <daniel.buenzli@erratique.ch> wrote:

> Le jeudi, 18 septembre 2014 à 23:36, Yaron Minsky a écrit :
>> Generating Jenga rules or OMake rules I think is note quite the right
>> approach. That's because Jenga and OMake both use general purpose
>> programming languages for specifying builds, and I think you'll get
>> bad behavior if you try to treat that language as a compilation
>> target.
>> 
>> For Jenga, I think it's better to write a Jenga plug-in for reading
>> Assemblage files. It has the same effect, but is an easier and more
>> typeful programming experience than spitting out a bunch of OCaml AST.
> 
> Yes that seems much more sensitive. I'm currently working on the API and everything will be exposed to allow you to do that. We'll just need to figure out a way so that you can easily take a handle on the project description value.
> 
> Usually I don't comment or advertise unfinished things but I'd also like to point out that the goal with assemblage is also a little bit larger than just generating a build system. I'd like to be able to express most of the package release and installation process inside it (or at least provide helpers to do so) in order to replace all the ad-hoc scripts that is use now (e.g. topkg), to make quick and flawless releases by gathering information from the right places, tags in the vcs for version information, synchronizing opam file with correct package dependencies, opam-submit integration etc.  
> 
> Some people do find my release process complex, I basically apply a function on a git checkout and do ugly things like a global key value substitution on all the files, but at least it is flawless and traceable. For example package never lie to you about their current version, even if you pin them. The only way to get that flawlessly is to automate it — can't count the number of times opam lied to me about its version because its version number needs to be bumped manually.

I'm in complete agreement here about automating the full release cycle being important.  A significant number of bugs in the OPAM repository have been around fixing broken invariants around out-of-sync metadata (most simply: remove the files that you installed when you are uninstalled).  It should be straightforward to automate much of this process, but it currently is very difficult to do so.

What excites me about Assemblage is that it provides the very lowest level of functionality to enable other tools to provide this automation.  This can range from file generation (Makefile), acting as a library for other build systems to interface with the package (Jenga, OMake), reliably interfacing with third-party tools such as ocamlfind, and also providing an easy extension mechanism for functionality that is still inflight (such as integrated documentation generation).

It's going to take a lot of time and effort to bring the global OCaml repositories out there into sync, and will probably never fully happen.  However, the aim with Assemblage is to solve the problem for a smaller set of interested people (i.e. Daniel's own repositories, or the Mirage library releases) and expand it out from there.  For instance, we'd love reliable compilation of Mirage on Windows (it is, after all, quite portable), and so this would motivate providing a suitable mechanism to go from an Assemblage project description to a Windows build (perhaps interfacing with WODI).

best,
Anil

Re: [Caml-list] One build system to rule them all?

[Yaron Minsky]

On Fri, Sep 19, 2014 at 3:27 AM, Gabriel Scherer
<gabriel.scherer@gmail.com> wrote:
> On Thu, Sep 18, 2014 at 11:23 PM, Yaron Minsky <yminsky@janestreet.com>
> wrote:
>>
>> Rather than exhort everyone to focus in on a solution, I recommend you
>> pick the one you think looks best, and see if you can contribute to
>> it.
>
>
> Agreed.
>
> My very personal point of focus is to help current ocamlbuild users (there
> are a fair number of them)
>
>> is there a DSL-based build system that we can converge on to use and
>> improve?
>
>
> I think this distinction between ocaml-using and DSL-using build systems
> does not matter in the long run. It matters when you look at a tool at a
> given point in time, but here you are discussing the long-term evolution of
> the tooling, and those distinctions get blurred assuming enough
> contributions:
> 1) you can turn an ocaml-using build system into a DSL-using build system by
> having an input mode that take DSL descriptions instead of rules, and
> interpreting them in term of those rules
> 2) a well-designed DSL-based system has an underlying library encoding the
> tool's engines, and exposing this library to advanced users effectively
> turns it into an ocaml-using tool as well
>
>> Conversely, continuing to spread the community's attention between these
>> tools (as well as ocamlbuild, which seems destined to stagnate) before any
>> one of them is top notch seems to me to be detrimental to ocaml's health as
>> an ecosystem.
>
>
> I haven't seen any good reason in this thread justifying ocamlbuild's
> stagnation (except, of course, the fact that few people are currently
> interested in working on that; but I thought you tried to abstract from this
> aspect):
> - people have mentioned having to edit several files as a source of
> annoyance with ocamlbuild; it would be rather easy to work on having a file
> with sub-sections corresponding to several of today's configuration file
> - the disappointing parallelism has been mentioned as a problem with
> ocamlbuild's implementation today -- it is a problem. I have discussed a
> practical approach to improving parallelism in (
> http://gallium.inria.fr/blog/ocamlbuild-parallelization/ ), which would
> require a bit more work but is not out of reach. For example that would be a
> reasonable Google Summer of Code (GSoC) topic, had the community succeeded
> in establishing a presence at GSoC.
>
> Note that I am not particularly attached to ocamlbuild in its current state;
> I care about its users to which it provides good service and should not be
> left in the dust.
>
> Jenga's design is in fact surprisingly close to ocamlbuild's, and one thing
> I have been thinking about is trying to reimplement ocamlbuild as a kind of
> "frontend" on top of Jenga's implementation -- the goal being the maintain
> compatibility with existing ocamlbuild setups, and gain some of the benefits
> of Jenga's engineering resources, namely arguably better parallelization and
> support continuous build. That's a rather bold move, there would be many
> less invasive changes that could have a positive impact on users -- for
> example, turning a succesful build into a Makefile should be doable in
> ocamlbuild as well, and interesting for many reasons (bare-boneness,
> performances).

We would very much welcome that kind of move, and would be happy to help.

>>
>> I think consensus should follow code, not the other way around.
>>
>> y
>>
>> On Thu, Sep 18, 2014 at 5:15 PM, Yotam Barnoy <yotambarnoy@gmail.com>
>> wrote:
>> > While there was no 'conclusion' to this thread, if I had to come up with
>> > one, it would be that we have a bunch of build tools which are all not
>> > amazing at this point in time. We have some DSL-based build tools and
>> > some
>> > ocaml-based build tools, and all of them need a lot of love to get to a
>> > good
>> > state.
>> >
>> > My personal view is that we (as a community) should work at getting at
>> > least
>> > one DSL tool to be really great. I'm sure Jenga (an ocaml-based tool
>> > which
>> > seems more like a build-tool engine) will continue to be developed by
>> > Jane
>> > Street no matter what, so is there a DSL-based build system that we can
>> > converge on to use and improve? The contenders for this slot appear to
>> > be
>> > omake, obuild, and ocp-build. I'm more than willing to switch to one of
>> > these if I know that other people will as well, and that it will be
>> > actively
>> > developed (preferably on github). More users = more invested parties =
>> > more
>> > development potential. Conversely, continuing to spread the community's
>> > attention between these tools (as well as ocamlbuild, which seems
>> > destined
>> > to stagnate) before any one of them is top notch seems to me to be
>> > detrimental to ocaml's health as an ecosystem.
>> >
>> > BTW Anil: is assemblage supposed to be an entire build toolchain, or is
>> > it
>> > only supposed to write makefiles (as the description in the readme
>> > states)?
>> >
>> > -Yotam
>> >
>> > On Mon, Sep 15, 2014 at 9:34 AM, Stéphane Glondu <steph@glondu.net>
>> > wrote:
>> >>
>> >> Le 10/09/2014 20:59, Yotam Barnoy a écrit :
>> >> > So here are some requirements I can think of (using some of the
>> >> > suggestions that have been brought up):
>> >> > - Easy to use, especially for small projects (large projects can
>> >> > afford
>> >> > to put more time into their build systems)
>> >> > - Abstract away platform considerations as much as possible. No
>> >> > dependence on specific shells and POSIX utilities.
>> >> > - Allows compilation of C files, which is quite common in ocaml
>> >> > packages.
>> >> > - Scalable to many directories and files
>> >> > - Uses ocamlfind to locate packages
>> >> > - Handles camlp4 and ppx
>> >> > - Parallel & incremental compilation
>> >>
>> >>  - Support of platforms without ocamlopt. Many build systems assume
>> >> that
>> >> ocamlopt is available and have to be called specially (or even patched)
>> >> when it is missing, which complicates packaging on these platforms.
>> >>
>> >> --
>> >> Stéphane
>> >>
>> >> --
>> >> Caml-list mailing list.  Subscription management and archives:
>> >> https://sympa.inria.fr/sympa/arc/caml-list
>> >> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>> >> Bug reports: http://caml.inria.fr/bin/caml-bugs
>> >
>> >
>>
>> --
>> Caml-list mailing list.  Subscription management and archives:
>> https://sympa.inria.fr/sympa/arc/caml-list
>> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>> Bug reports: http://caml.inria.fr/bin/caml-bugs
>
>