[Caml-list] About ocamlbuild

[Caml-list] About ocamlbuild

[Pierre-Etienne Meunier]

Hi,

Several ocamlbuild-related questions:

1 - Is there a simple reason why ocamlbuild is a tool with plugins and not a library ? It would be much easier to adapt it to other contexts than ocaml compiling, and to larger projects. In my projects, a myocamlbuild.ml file must be written most of the time, so I do not understand the point.
Another point I see is that I usually spend a non-negligible part of my time writing makefiles and myocamlbuild files. Since I must do it anyway, I'd like to write all that in pure ocaml, doing argument parsing as I want, adding arguments if I want, and even doing some configure steps in ocaml before starting to compile anything. Using output_value to do this (as I did for http://lama.univ-savoie.fr/~meunier/camlimages for instance) is much easier than outputing _tags and .mlpack in subdirectories.
Also, it would be nice to allow it to depend on other modules, even of some files of the project you are compiling. Examples of uses include configuration options, versions, etc.

2 - Also, I do not understand why I get the following message:

Failure: Included or excluded directories must be implicit (not "../_build").

Is it a technical restriction to avoid copying entire filesystems to _build if the _tags are ill-written, or is there another reason ?

3 - Finally, I find the idea of tags good, for backward compatibility reasons (you do not have to change your code), but not enough. For instance, in haskell (and some compilers written in ocaml), you can add "tags" at the beginning of your files. You would start your ocaml files with comments such as:

(* #OPTIONS -rectypes *)


Thanks,
Pierre

Re: [Caml-list] About ocamlbuild

[Xavier Clerc]

----- Mail original -----
> Hi,
> 
> Several ocamlbuild-related questions:
> 
> 1 - Is there a simple reason why ocamlbuild is a tool with plugins
> and not a library ? It would be much easier to adapt it to other
> contexts than ocaml compiling, and to larger projects. In my
> projects, a myocamlbuild.ml file must be written most of the time,
> so I do not understand the point.

I am afraid I do not grasp the dichotomy you draw between a tool
and a library, as ocamlbuild plugins are actually compiled against
the ocamlbuild library. Could you elaborate on what you feel wrong
with this approach?


> Another point I see is that I usually spend a non-negligible part of
> my time writing makefiles and myocamlbuild files. Since I must do it
> anyway, I'd like to write all that in pure ocaml, doing argument
> parsing as I want, adding arguments if I want, and even doing some
> configure steps in ocaml before starting to compile anything. Using
> output_value to do this (as I did for
> http://lama.univ-savoie.fr/~meunier/camlimages for instance) is much
> easier than outputing _tags and .mlpack in subdirectories.

Well, the first thing to notice is that the ocamlbuild API is not
cast in stone, and a tool like camlp4 provides an API allowing to add
command-line switches via plugins. However, I have found myself that
the question of parameter passing can often be done through tags.
As for auxiliary files, it is frequent for me (in personal projects)
to create mlpack files from the myocamlbuild file. Indeed, I think
that it is even necessary to be really flexible about file addition
or deletion (but, of course, this implies to have some sort of "rule"
to determine which modules should be packed).


> Also, it would be nice to allow it to depend on other modules, even
> of some files of the project you are compiling. Examples of uses
> include configuration options, versions, etc.

I beg to differ with this ability to interleave application code and
build code. That being said, nothing prevents you from tweaking your
myocamlbuild file at configuration step.


> 2 - Also, I do not understand why I get the following message:
> 
> Failure: Included or excluded directories must be implicit (not
> "../_build").
> 
> Is it a technical restriction to avoid copying entire filesystems to
> _build if the _tags are ill-written, or is there another reason ?

Let me ask you the dual question: why is this restriction causing you
some trouble?


> 3 - Finally, I find the idea of tags good, for backward compatibility
> reasons (you do not have to change your code), but not enough. For
> instance, in haskell (and some compilers written in ocaml), you can
> add "tags" at the beginning of your files. You would start your
> ocaml files with comments such as:
> 
> (* #OPTIONS -rectypes *)

Well, I find tags pretty convenient, and do not dislike the comment
approach. However, I am pretty sure I do not want the related information
to be scattered over multiple locations.


Kind regards,

Xavier Clerc

Re: [Caml-list] About ocamlbuild

[Edgar Friendly]

On 11/15/2012 4:48 PM, Xavier Clerc wrote:
>> 3 - Finally, I find the idea of tags good, for backward compatibility
>> reasons (you do not have to change your code), but not enough. For
>> instance, in haskell (and some compilers written in ocaml), you can
>> add "tags" at the beginning of your files. You would start your
>> ocaml files with comments such as:
>>
>> (* #OPTIONS -rectypes *)
> Well, I find tags pretty convenient, and do not dislike the comment
> approach. However, I am pretty sure I do not want the related information
> to be scattered over multiple locations.
It's exactly this reason that I'm strongly in favor of magic comments or 
pragmas or something in .ml files to do do this kind of thing.  
Splitting the data needed to compile a module into two parts: the module 
and a makefile (or _tags or OMakefile or whatever) seems a violation of 
your "multiple locations" policy.  While I grant that some compilation 
options (whether to compile in debug/profiling/native mode, location of 
external libraries) should not be included in a source file, I'm 
strongly in favor of having the build system work out the details of 
dependencies (internal to project and external, installed in global 
location) and even what camlp4 to run based on the contents of the file 
itself and not elsewhere.

E.

Re: [Caml-list] About ocamlbuild

[Xavier Clerc]

----- Mail original -----
> On 11/15/2012 4:48 PM, Xavier Clerc wrote:
> >> 3 - Finally, I find the idea of tags good, for backward
> >> compatibility
> >> reasons (you do not have to change your code), but not enough. For
> >> instance, in haskell (and some compilers written in ocaml), you
> >> can
> >> add "tags" at the beginning of your files. You would start your
> >> ocaml files with comments such as:
> >>
> >> (* #OPTIONS -rectypes *)
> > Well, I find tags pretty convenient, and do not dislike the comment
> > approach. However, I am pretty sure I do not want the related
> > information
> > to be scattered over multiple locations.
> It's exactly this reason that I'm strongly in favor of magic comments
> or
> pragmas or something in .ml files to do do this kind of thing.
> Splitting the data needed to compile a module into two parts: the
> module
> and a makefile (or _tags or OMakefile or whatever) seems a violation
> of
> your "multiple locations" policy.

I do not follow your line of thought: every information about how to
compile is clearly in build files, and not in source code files.
If I wonder how the command-line for a given compilation is determined,
there is only one place to look.

Again, I do not claim that the current state of affairs (everything
in build files) is better than a pure pragma-based solution (everything
in source files). My objection is about a mixed system where one should
have to consult both build and source file in order to know which
options are passed to compile a given module.


> While I grant that some
> compilation
> options (whether to compile in debug/profiling/native mode, location
> of
> external libraries) should not be included in a source file, I'm
> strongly in favor of having the build system work out the details of
> dependencies (internal to project and external, installed in global
> location) and even what camlp4 to run based on the contents of the
> file
> itself and not elsewhere.

To be fair, I should point out that Coq is extracting camlp4 information
from source files in order to determine how to treat them.


Just a final remark: compared to Haskell, OCaml compilers feature a limited
number of command-line switches having an influence on the semantics of a
source file(*). I regard this as a pleasant thing, and think this makes the
issue at hand quite minor.


Kind regards,

Xavier Clerc




(*) we are only talking about labels, applicative functors, and recursive types

Re: [Caml-list] About ocamlbuild

[Arnaud Spiwack]

[-- Attachment #1: Type: text/plain, Size: 4372 bytes --]

Well,

I must say the sort of mafioso-like reasoning ­- "you really do not want to
do that" - doesn't really make sense to me.

Here are valid reasons not to include a feature X in a tool:

   - I don't have time to do X
   - I don't know how to do X
   - Having X would prevent me from having Y which I'd rather

Any other reason is bad.

People want to have tags and compilation pragmas? Well, let them, they
probably have valid reasons to (some options may be better seen global to
the project and other local to files). People want to interleave compiling
the build system and compiling the project. They have a good reason for
that too. Also: ocamlbuild doesn't only compile ocaml files.

Related to that last point, and to the initial question: as far as I'm
aware, an ocamlbuild plugin cannot depend on a custom library (to go around
that we can use a myocamlbuild.ml which compiles a mymyocamlbuild.ml (or
something) with the appropriate linking options, then copy it in a
_build_stage2 directory, then Unix.excv an ocamlbuild -no-plugin -build-dir
_build_stage2. It would be nice if there were a standard and simple way to
do this sort of thing). In particular there is no principled way to
distribute a tool together with an ocamlbuild library against which to
compile our myocamlbuild.ml plugins (hence extending what ocamlbuild can
"natively" compile).



On 15 November 2012 23:36, Xavier Clerc <xavier.clerc@inria.fr> wrote:

> ----- Mail original -----
> > On 11/15/2012 4:48 PM, Xavier Clerc wrote:
> > >> 3 - Finally, I find the idea of tags good, for backward
> > >> compatibility
> > >> reasons (you do not have to change your code), but not enough. For
> > >> instance, in haskell (and some compilers written in ocaml), you
> > >> can
> > >> add "tags" at the beginning of your files. You would start your
> > >> ocaml files with comments such as:
> > >>
> > >> (* #OPTIONS -rectypes *)
> > > Well, I find tags pretty convenient, and do not dislike the comment
> > > approach. However, I am pretty sure I do not want the related
> > > information
> > > to be scattered over multiple locations.
> > It's exactly this reason that I'm strongly in favor of magic comments
> > or
> > pragmas or something in .ml files to do do this kind of thing.
> > Splitting the data needed to compile a module into two parts: the
> > module
> > and a makefile (or _tags or OMakefile or whatever) seems a violation
> > of
> > your "multiple locations" policy.
>
> I do not follow your line of thought: every information about how to
> compile is clearly in build files, and not in source code files.
> If I wonder how the command-line for a given compilation is determined,
> there is only one place to look.
>
> Again, I do not claim that the current state of affairs (everything
> in build files) is better than a pure pragma-based solution (everything
> in source files). My objection is about a mixed system where one should
> have to consult both build and source file in order to know which
> options are passed to compile a given module.
>
>
> > While I grant that some
> > compilation
> > options (whether to compile in debug/profiling/native mode, location
> > of
> > external libraries) should not be included in a source file, I'm
> > strongly in favor of having the build system work out the details of
> > dependencies (internal to project and external, installed in global
> > location) and even what camlp4 to run based on the contents of the
> > file
> > itself and not elsewhere.
>
> To be fair, I should point out that Coq is extracting camlp4 information
> from source files in order to determine how to treat them.
>
>
> Just a final remark: compared to Haskell, OCaml compilers feature a limited
> number of command-line switches having an influence on the semantics of a
> source file(*). I regard this as a pleasant thing, and think this makes the
> issue at hand quite minor.
>
>
> Kind regards,
>
> Xavier Clerc
>
>
>
>
> (*) we are only talking about labels, applicative functors, and recursive
> types
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>

[-- Attachment #2: Type: text/html, Size: 5495 bytes --]

Re: [Caml-list] About ocamlbuild

[Gabriel Scherer]

Regardless of the overly heated tone of the discussion, I find the
idea of having "ocamlbuild as a library" interesting. Let me detail
what I believe was the idea of the proposal.

Currently we have a shared ocamlbuild executable that would contain
the core logic, and be passed plugins that are invoked by the core
logic at some point of the program. We could consider an inversion of
control where the executable logic is exposed as a library, and users
write scripts use this library (to invoke the ocamlbuild processing),
the script being compiled as whole programs rather than plugins. In
effect, you give the tool to let any project define it's own "tailored
ocamlbuild" easily. There could be little change of usage and
behavior, it's just a different point of view, that has the advantages
already evoked in this thread (facility to link with other stuff,
etc.).

This idea has already been used in other projects where "configuration
is compilation", such as the XMonad window manager (
http://xmonad.org/ ); the xmonad.hs users write is not compiled to a
dynamically loaded plugin, it itself invokes the "xmonad program" as a
library function, passing it (from the language) configuration
information.

I find this idea interesting, but it will require some amount of work.
My guess as to why Xavier Clerc's reply is a bit grumpy is that he is
often the one tasked with implementing those changes. On the other
hand, everyone is free to contribute to ocamlbuild, and I've been
trying to publicly encourage people to contribute, with little success
so far. So Pierre, if you want to have a try at implementing this
idea, by all means, do! We'll see when someone has some working patch
if it can be integrated upstream.

Regarding in-file tags/pragmas, I find such changes should be
considered much more cautiously, as those could mean changes to the
OCaml tooling (or worse, language) as a whole, not simply ocamlbuild
itself. Again, nobody's forbidden to come up with a good,
well-specified, simple proposal that would please everyone, but that
is a quite difficult task and any such proposal should expect to get
rejected.

On Fri, Nov 16, 2012 at 11:33 AM, Arnaud Spiwack
<Arnaud.Spiwack@lix.polytechnique.fr> wrote:
> Well,
>
> I must say the sort of mafioso-like reasoning ­- "you really do not want to
> do that" - doesn't really make sense to me.
>
> Here are valid reasons not to include a feature X in a tool:
>
> I don't have time to do X
> I don't know how to do X
> Having X would prevent me from having Y which I'd rather
>
> Any other reason is bad.
>
> People want to have tags and compilation pragmas? Well, let them, they
> probably have valid reasons to (some options may be better seen global to
> the project and other local to files). People want to interleave compiling
> the build system and compiling the project. They have a good reason for that
> too. Also: ocamlbuild doesn't only compile ocaml files.
>
> Related to that last point, and to the initial question: as far as I'm
> aware, an ocamlbuild plugin cannot depend on a custom library (to go around
> that we can use a myocamlbuild.ml which compiles a mymyocamlbuild.ml (or
> something) with the appropriate linking options, then copy it in a
> _build_stage2 directory, then Unix.excv an ocamlbuild -no-plugin -build-dir
> _build_stage2. It would be nice if there were a standard and simple way to
> do this sort of thing). In particular there is no principled way to
> distribute a tool together with an ocamlbuild library against which to
> compile our myocamlbuild.ml plugins (hence extending what ocamlbuild can
> "natively" compile).
>
>
>
>
> On 15 November 2012 23:36, Xavier Clerc <xavier.clerc@inria.fr> wrote:
>>
>> ----- Mail original -----
>> > On 11/15/2012 4:48 PM, Xavier Clerc wrote:
>> > >> 3 - Finally, I find the idea of tags good, for backward
>> > >> compatibility
>> > >> reasons (you do not have to change your code), but not enough. For
>> > >> instance, in haskell (and some compilers written in ocaml), you
>> > >> can
>> > >> add "tags" at the beginning of your files. You would start your
>> > >> ocaml files with comments such as:
>> > >>
>> > >> (* #OPTIONS -rectypes *)
>> > > Well, I find tags pretty convenient, and do not dislike the comment
>> > > approach. However, I am pretty sure I do not want the related
>> > > information
>> > > to be scattered over multiple locations.
>> > It's exactly this reason that I'm strongly in favor of magic comments
>> > or
>> > pragmas or something in .ml files to do do this kind of thing.
>> > Splitting the data needed to compile a module into two parts: the
>> > module
>> > and a makefile (or _tags or OMakefile or whatever) seems a violation
>> > of
>> > your "multiple locations" policy.
>>
>> I do not follow your line of thought: every information about how to
>> compile is clearly in build files, and not in source code files.
>> If I wonder how the command-line for a given compilation is determined,
>> there is only one place to look.
>>
>> Again, I do not claim that the current state of affairs (everything
>> in build files) is better than a pure pragma-based solution (everything
>> in source files). My objection is about a mixed system where one should
>> have to consult both build and source file in order to know which
>> options are passed to compile a given module.
>>
>>
>> > While I grant that some
>> > compilation
>> > options (whether to compile in debug/profiling/native mode, location
>> > of
>> > external libraries) should not be included in a source file, I'm
>> > strongly in favor of having the build system work out the details of
>> > dependencies (internal to project and external, installed in global
>> > location) and even what camlp4 to run based on the contents of the
>> > file
>> > itself and not elsewhere.
>>
>> To be fair, I should point out that Coq is extracting camlp4 information
>> from source files in order to determine how to treat them.
>>
>>
>> Just a final remark: compared to Haskell, OCaml compilers feature a
>> limited
>> number of command-line switches having an influence on the semantics of a
>> source file(*). I regard this as a pleasant thing, and think this makes
>> the
>> issue at hand quite minor.
>>
>>
>> Kind regards,
>>
>> Xavier Clerc
>>
>>
>>
>>
>> (*) we are only talking about labels, applicative functors, and recursive
>> types
>>
>> --
>> Caml-list mailing list.  Subscription management and archives:
>> https://sympa.inria.fr/sympa/arc/caml-list
>> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>> Bug reports: http://caml.inria.fr/bin/caml-bugs
>
>

Re: [Caml-list] About ocamlbuild

[Daniel Bünzli]

Le vendredi, 16 novembre 2012 à 11:33, Arnaud Spiwack a écrit :
> Well,
>  
> I must say the sort of mafioso-like reasoning ­- "you really do not want to do that" - doesn't really make sense to me.
>  
> Here are valid reasons not to include a feature X in a tool:
> I don't have time to do X
> I don't know how to do X
> Having X would prevent me from having Y which I'd rather
>  
> Any other reason is bad.
This is a little bit OT, but

  X doesn't fit the philosophy of my tool  

is also a very good reason not to include a feature X in a tool. It's not mafioso-like reasoning, it's called design (and if you are not happy with my design, make your own tool). Sticking to your three valid reasons is a recipe for disaster and feature creep. As is thinking that people have necessarily good reasons for what they want (we all have blind spots).  

Best,

Daniel

Re: [Caml-list] About ocamlbuild

[Pierre-Etienne Meunier]

Hi,

My original idea was not at all to start a new ocaml-tools war, I'm sorry if my first message was poorly formulated. Let me make myself clear, I wrote a typesetting system in ocaml, in order to write my PhD thesis. The way it works is a little strange, it compiles your source to ocaml source code (our language is quite cool, because based on a dypgen grammar, but you can use another one if you have time to write a parser and code generator). Then it compiles this code with a library called "Typography", that does all the document processing stuff, optimization of the page breaks, and so on. Finally, it calls another library to output to different formats, such as pdf, plain text, svg, opengl, or anything else.

These are clearly "separable but not independent" projects (drivers, document formats, font library, typesetting library, parser/code generator). So compiling everything without duplication is not that trivial, and I really would like to write an ocaml program to compile it, maybe using a more complex build system than just one file with the ocamlbuild api. I understand that ocamlbuild is still in an early stage of development, this is why I'm sending these mails. The code is split among different directories, and using a library interface in the compilation of another library still seems difficult in ocamlbuild today. The documentation explains how to use an internal library in program, but it forces you to use a particular layout of your files. This is why I feel that a library would be more versatile: we can imagine a makefile compiling the build system first, possibly split between several modules, then calling it to compile the rest, like xmonad does, for instance.

Also, if you are writing a document with our system, separated between different files, you want the system to compile it automatically, without too much recompilation. The problem is extensibility: for instance, if you are writing a bibliography library (I wrote one, but there may be others in the future), you certainly do not want to touch our code in order to compile it. Instead, you want to write a dynlinkable module explaining to our system how the documents depend on bibliography files, and how to compile everything in what order.
To do this, I had to rewrite a (small) ocamlbuild-like library. My point is that ocamlbuild exposes (or documents) a too small part of its internals to be re-used inside another project.


Now, I completely understand Xavier's points:

- about _tags, and this is also my philosophy when I write ocaml code myself. I asked the question because I needed magic comments in the build system of my typesetting stuff, mostly because the ocaml source code that I compile is produced automatically, and meant to be executed only once. Writing the correct _tags files without conflicts between the different files, with the correct directory inclusions for instance, would be much more complicated. This is the only case where I see the point of a mixed system.

- about the command line options of ghc: clearly, this also has to do with the fact that ocamlopt manages to compile files of equivalent sizes about twenty or thirty times faster than ghc in the cases I tried. The theoretical problems solved are not the same, either.

Finally, Gabriel, I would be happy to contribute code to ocaml, only my time is a scarce resource right now. Maybe some day…

Cheers,
Pierre


Em 16/11/2012, às 11:53, Gabriel Scherer escreveu:

> Regardless of the overly heated tone of the discussion, I find the
> idea of having "ocamlbuild as a library" interesting. Let me detail
> what I believe was the idea of the proposal.
> 
> Currently we have a shared ocamlbuild executable that would contain
> the core logic, and be passed plugins that are invoked by the core
> logic at some point of the program. We could consider an inversion of
> control where the executable logic is exposed as a library, and users
> write scripts use this library (to invoke the ocamlbuild processing),
> the script being compiled as whole programs rather than plugins. In
> effect, you give the tool to let any project define it's own "tailored
> ocamlbuild" easily. There could be little change of usage and
> behavior, it's just a different point of view, that has the advantages
> already evoked in this thread (facility to link with other stuff,
> etc.).
> 
> This idea has already been used in other projects where "configuration
> is compilation", such as the XMonad window manager (
> http://xmonad.org/ ); the xmonad.hs users write is not compiled to a
> dynamically loaded plugin, it itself invokes the "xmonad program" as a
> library function, passing it (from the language) configuration
> information.
> 
> I find this idea interesting, but it will require some amount of work.
> My guess as to why Xavier Clerc's reply is a bit grumpy is that he is
> often the one tasked with implementing those changes. On the other
> hand, everyone is free to contribute to ocamlbuild, and I've been
> trying to publicly encourage people to contribute, with little success
> so far. So Pierre, if you want to have a try at implementing this
> idea, by all means, do! We'll see when someone has some working patch
> if it can be integrated upstream.
> 
> Regarding in-file tags/pragmas, I find such changes should be
> considered much more cautiously, as those could mean changes to the
> OCaml tooling (or worse, language) as a whole, not simply ocamlbuild
> itself. Again, nobody's forbidden to come up with a good,
> well-specified, simple proposal that would please everyone, but that
> is a quite difficult task and any such proposal should expect to get
> rejected.
> 
> On Fri, Nov 16, 2012 at 11:33 AM, Arnaud Spiwack
> <Arnaud.Spiwack@lix.polytechnique.fr> wrote:
>> Well,
>> 
>> I must say the sort of mafioso-like reasoning - "you really do not want to
>> do that" - doesn't really make sense to me.
>> 
>> Here are valid reasons not to include a feature X in a tool:
>> 
>> I don't have time to do X
>> I don't know how to do X
>> Having X would prevent me from having Y which I'd rather
>> 
>> Any other reason is bad.
>> 
>> People want to have tags and compilation pragmas? Well, let them, they
>> probably have valid reasons to (some options may be better seen global to
>> the project and other local to files). People want to interleave compiling
>> the build system and compiling the project. They have a good reason for that
>> too. Also: ocamlbuild doesn't only compile ocaml files.
>> 
>> Related to that last point, and to the initial question: as far as I'm
>> aware, an ocamlbuild plugin cannot depend on a custom library (to go around
>> that we can use a myocamlbuild.ml which compiles a mymyocamlbuild.ml (or
>> something) with the appropriate linking options, then copy it in a
>> _build_stage2 directory, then Unix.excv an ocamlbuild -no-plugin -build-dir
>> _build_stage2. It would be nice if there were a standard and simple way to
>> do this sort of thing). In particular there is no principled way to
>> distribute a tool together with an ocamlbuild library against which to
>> compile our myocamlbuild.ml plugins (hence extending what ocamlbuild can
>> "natively" compile).
>> 
>> 
>> 
>> 
>> On 15 November 2012 23:36, Xavier Clerc <xavier.clerc@inria.fr> wrote:
>>> 
>>> ----- Mail original -----
>>>> On 11/15/2012 4:48 PM, Xavier Clerc wrote:
>>>>>> 3 - Finally, I find the idea of tags good, for backward
>>>>>> compatibility
>>>>>> reasons (you do not have to change your code), but not enough. For
>>>>>> instance, in haskell (and some compilers written in ocaml), you
>>>>>> can
>>>>>> add "tags" at the beginning of your files. You would start your
>>>>>> ocaml files with comments such as:
>>>>>> 
>>>>>> (* #OPTIONS -rectypes *)
>>>>> Well, I find tags pretty convenient, and do not dislike the comment
>>>>> approach. However, I am pretty sure I do not want the related
>>>>> information
>>>>> to be scattered over multiple locations.
>>>> It's exactly this reason that I'm strongly in favor of magic comments
>>>> or
>>>> pragmas or something in .ml files to do do this kind of thing.
>>>> Splitting the data needed to compile a module into two parts: the
>>>> module
>>>> and a makefile (or _tags or OMakefile or whatever) seems a violation
>>>> of
>>>> your "multiple locations" policy.
>>> 
>>> I do not follow your line of thought: every information about how to
>>> compile is clearly in build files, and not in source code files.
>>> If I wonder how the command-line for a given compilation is determined,
>>> there is only one place to look.
>>> 
>>> Again, I do not claim that the current state of affairs (everything
>>> in build files) is better than a pure pragma-based solution (everything
>>> in source files). My objection is about a mixed system where one should
>>> have to consult both build and source file in order to know which
>>> options are passed to compile a given module.
>>> 
>>> 
>>>> While I grant that some
>>>> compilation
>>>> options (whether to compile in debug/profiling/native mode, location
>>>> of
>>>> external libraries) should not be included in a source file, I'm
>>>> strongly in favor of having the build system work out the details of
>>>> dependencies (internal to project and external, installed in global
>>>> location) and even what camlp4 to run based on the contents of the
>>>> file
>>>> itself and not elsewhere.
>>> 
>>> To be fair, I should point out that Coq is extracting camlp4 information
>>> from source files in order to determine how to treat them.
>>> 
>>> 
>>> Just a final remark: compared to Haskell, OCaml compilers feature a
>>> limited
>>> number of command-line switches having an influence on the semantics of a
>>> source file(*). I regard this as a pleasant thing, and think this makes
>>> the
>>> issue at hand quite minor.
>>> 
>>> 
>>> Kind regards,
>>> 
>>> Xavier Clerc
>>> 
>>> 
>>> 
>>> 
>>> (*) we are only talking about labels, applicative functors, and recursive
>>> types
>>> 
>>> --
>>> Caml-list mailing list.  Subscription management and archives:
>>> https://sympa.inria.fr/sympa/arc/caml-list
>>> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>>> Bug reports: http://caml.inria.fr/bin/caml-bugs
>> 
>> 
> 
> -- 
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs

[Caml-list] Re: About ocamlbuild

[Hongbo Zhang]

On 11/16/12 6:43 AM, Pierre-Etienne Meunier wrote:
> Hi,
>
> My original idea was not at all to start a new ocaml-tools war, I'm sorry if my first message was poorly formulated. Let me make myself clear, I wrote a typesetting system in ocaml, in order to write my PhD thesis. The way it works is a little strange, it compiles your source to ocaml source code (our language is quite cool, because based on a dypgen grammar, but you can use another one if you have time to write a parser and code generator). Then it compiles this code with a library called "Typography", that does all the document processing stuff, optimization of the page breaks, and so on. Finally, it calls another library to output to different formats, such as pdf, plain text, svg, opengl, or anything else.
>
> These are clearly "separable but not independent" projects (drivers, document formats, font library, typesetting library, parser/code generator). So compiling everything without duplication is not that trivial, and I really would like to write an ocaml program to compile it, maybe using a more complex build system than just one file with the ocamlbuild api. I understand that ocamlbuild is still in an early stage of development, this is why I'm sending these mails. The code is split among different directories, and using a library interface in the compilation of another library still seems difficult in ocamlbuild today. The documentation explains how to use an internal library in program, but it forces you to use a particular layout of your files. This is why I feel that a library would be more versatile: we can imagine a makefile compiling the build system first, possibly split between several modules, then calling it to compile the rest, like xmonad does, for instance.
>
> Also, if you are writing a document with our system, separated between different files, you want the system to compile it automatically, without too much recompilation. The problem is extensibility: for instance, if you are writing a bibliography library (I wrote one, but there may be others in the future), you certainly do not want to touch our code in order to compile it. Instead, you want to write a dynlinkable module explaining to our system how the documents depend on bibliography files, and how to compile everything in what order.
> To do this, I had to rewrite a (small) ocamlbuild-like library. My point is that ocamlbuild exposes (or documents) a too small part of its internals to be re-used inside another project.

I fully agree with you, it would be really nice that ocamlbuild provide 
a library interface instead a poor myocamlbuild.ml.

Currently my situation is  that I have to copy my 2000lines 
myocamlbuild.ml everywhere, and do some changes locally(probably some 
bug fixes will be forgotten to sync up since it's scattered everywhere 
and I am lazy), that's not the ideal case.

>
> Now, I completely understand Xavier's points:
>
> - about _tags, and this is also my philosophy when I write ocaml code myself. I asked the question because I needed magic comments in the build system of my typesetting stuff, mostly because the ocaml source code that I compile is produced automatically, and meant to be executed only once. Writing the correct _tags files without conflicts between the different files, with the correct directory inclusions for instance, would be much more complicated. This is the only case where I see the point of a mixed system.
>
> - about the command line options of ghc: clearly, this also has to do with the fact that ocamlopt manages to compile files of equivalent sizes about twenty or thirty times faster than ghc in the cases I tried. The theoretical problems solved are not the same, either.
>
> Finally, Gabriel, I would be happy to contribute code to ocaml, only my time is a scarce resource right now. Maybe some day…
>
> Cheers,
> Pierre
>
>
> Em 16/11/2012, às 11:53, Gabriel Scherer escreveu:
>
>> Regardless of the overly heated tone of the discussion, I find the
>> idea of having "ocamlbuild as a library" interesting. Let me detail
>> what I believe was the idea of the proposal.
>>
>> Currently we have a shared ocamlbuild executable that would contain
>> the core logic, and be passed plugins that are invoked by the core
>> logic at some point of the program. We could consider an inversion of
>> control where the executable logic is exposed as a library, and users
>> write scripts use this library (to invoke the ocamlbuild processing),
>> the script being compiled as whole programs rather than plugins. In
>> effect, you give the tool to let any project define it's own "tailored
>> ocamlbuild" easily. There could be little change of usage and
>> behavior, it's just a different point of view, that has the advantages
>> already evoked in this thread (facility to link with other stuff,
>> etc.).
>>
>> This idea has already been used in other projects where "configuration
>> is compilation", such as the XMonad window manager (
>> http://xmonad.org/ ); the xmonad.hs users write is not compiled to a
>> dynamically loaded plugin, it itself invokes the "xmonad program" as a
>> library function, passing it (from the language) configuration
>> information.
>>
>> I find this idea interesting, but it will require some amount of work.
>> My guess as to why Xavier Clerc's reply is a bit grumpy is that he is
>> often the one tasked with implementing those changes. On the other
>> hand, everyone is free to contribute to ocamlbuild, and I've been
>> trying to publicly encourage people to contribute, with little success
>> so far. So Pierre, if you want to have a try at implementing this
>> idea, by all means, do! We'll see when someone has some working patch
>> if it can be integrated upstream.
>>
>> Regarding in-file tags/pragmas, I find such changes should be
>> considered much more cautiously, as those could mean changes to the
>> OCaml tooling (or worse, language) as a whole, not simply ocamlbuild
>> itself. Again, nobody's forbidden to come up with a good,
>> well-specified, simple proposal that would please everyone, but that
>> is a quite difficult task and any such proposal should expect to get
>> rejected.
>>
>> On Fri, Nov 16, 2012 at 11:33 AM, Arnaud Spiwack
>> <Arnaud.Spiwack@lix.polytechnique.fr> wrote:
>>> Well,
>>>
>>> I must say the sort of mafioso-like reasoning - "you really do not want to
>>> do that" - doesn't really make sense to me.
>>>
>>> Here are valid reasons not to include a feature X in a tool:
>>>
>>> I don't have time to do X
>>> I don't know how to do X
>>> Having X would prevent me from having Y which I'd rather
>>>
>>> Any other reason is bad.
>>>
>>> People want to have tags and compilation pragmas? Well, let them, they
>>> probably have valid reasons to (some options may be better seen global to
>>> the project and other local to files). People want to interleave compiling
>>> the build system and compiling the project. They have a good reason for that
>>> too. Also: ocamlbuild doesn't only compile ocaml files.
>>>
>>> Related to that last point, and to the initial question: as far as I'm
>>> aware, an ocamlbuild plugin cannot depend on a custom library (to go around
>>> that we can use a myocamlbuild.ml which compiles a mymyocamlbuild.ml (or
>>> something) with the appropriate linking options, then copy it in a
>>> _build_stage2 directory, then Unix.excv an ocamlbuild -no-plugin -build-dir
>>> _build_stage2. It would be nice if there were a standard and simple way to
>>> do this sort of thing). In particular there is no principled way to
>>> distribute a tool together with an ocamlbuild library against which to
>>> compile our myocamlbuild.ml plugins (hence extending what ocamlbuild can
>>> "natively" compile).
>>>
>>>
>>>
>>>
>>> On 15 November 2012 23:36, Xavier Clerc <xavier.clerc@inria.fr> wrote:
>>>>
>>>> ----- Mail original -----
>>>>> On 11/15/2012 4:48 PM, Xavier Clerc wrote:
>>>>>>> 3 - Finally, I find the idea of tags good, for backward
>>>>>>> compatibility
>>>>>>> reasons (you do not have to change your code), but not enough. For
>>>>>>> instance, in haskell (and some compilers written in ocaml), you
>>>>>>> can
>>>>>>> add "tags" at the beginning of your files. You would start your
>>>>>>> ocaml files with comments such as:
>>>>>>>
>>>>>>> (* #OPTIONS -rectypes *)
>>>>>> Well, I find tags pretty convenient, and do not dislike the comment
>>>>>> approach. However, I am pretty sure I do not want the related
>>>>>> information
>>>>>> to be scattered over multiple locations.
>>>>> It's exactly this reason that I'm strongly in favor of magic comments
>>>>> or
>>>>> pragmas or something in .ml files to do do this kind of thing.
>>>>> Splitting the data needed to compile a module into two parts: the
>>>>> module
>>>>> and a makefile (or _tags or OMakefile or whatever) seems a violation
>>>>> of
>>>>> your "multiple locations" policy.
>>>>
>>>> I do not follow your line of thought: every information about how to
>>>> compile is clearly in build files, and not in source code files.
>>>> If I wonder how the command-line for a given compilation is determined,
>>>> there is only one place to look.
>>>>
>>>> Again, I do not claim that the current state of affairs (everything
>>>> in build files) is better than a pure pragma-based solution (everything
>>>> in source files). My objection is about a mixed system where one should
>>>> have to consult both build and source file in order to know which
>>>> options are passed to compile a given module.
>>>>
>>>>
>>>>> While I grant that some
>>>>> compilation
>>>>> options (whether to compile in debug/profiling/native mode, location
>>>>> of
>>>>> external libraries) should not be included in a source file, I'm
>>>>> strongly in favor of having the build system work out the details of
>>>>> dependencies (internal to project and external, installed in global
>>>>> location) and even what camlp4 to run based on the contents of the
>>>>> file
>>>>> itself and not elsewhere.
>>>>
>>>> To be fair, I should point out that Coq is extracting camlp4 information
>>>> from source files in order to determine how to treat them.
>>>>
>>>>
>>>> Just a final remark: compared to Haskell, OCaml compilers feature a
>>>> limited
>>>> number of command-line switches having an influence on the semantics of a
>>>> source file(*). I regard this as a pleasant thing, and think this makes
>>>> the
>>>> issue at hand quite minor.
>>>>
>>>>
>>>> Kind regards,
>>>>
>>>> Xavier Clerc
>>>>
>>>>
>>>>
>>>>
>>>> (*) we are only talking about labels, applicative functors, and recursive
>>>> types
>>>>
>>>> --
>>>> Caml-list mailing list.  Subscription management and archives:
>>>> https://sympa.inria.fr/sympa/arc/caml-list
>>>> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>>>> Bug reports: http://caml.inria.fr/bin/caml-bugs
>>>
>>>
>>
>> --
>> Caml-list mailing list.  Subscription management and archives:
>> https://sympa.inria.fr/sympa/arc/caml-list
>> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>> Bug reports: http://caml.inria.fr/bin/caml-bugs
>
>

Re: [Caml-list] About ocamlbuild

[Wojciech Meyer]

[-- Attachment #1: Type: text/plain, Size: 8044 bytes --]

Hi,

I had the idea of fleshing out the ocamlbuild internals some time ago. On
OUD I expressed this to Nicolas P (co author of ocamlbuid) and Xavier
Clerk. I also spoke to Sylvain about turning Oasis from compiler to
interpreter, not directly generating files but instead calling ocamlbuild
API. The good start would be to be able to dynlink directly libraries with
side effects. Then we could do further and provide .build pacakges for
ocamlind for each project that uses complex build steps for instance due to
metaprogramming or several stages of compilation.

On Friday, November 16, 2012, Hongbo Zhang <bobzhang1988@gmail.com> wrote:
> On 11/16/12 6:43 AM, Pierre-Etienne Meunier wrote:
>>
>> Hi,
>>
>> My original idea was not at all to start a new ocaml-tools war, I'm
sorry if my first message was poorly formulated. Let me make myself clear,
I wrote a typesetting system in ocaml, in order to write my PhD thesis. The
way it works is a little strange, it compiles your source to ocaml source
code (our language is quite cool, because based on a dypgen grammar, but
you can use another one if you have time to write a parser and code
generator). Then it compiles this code with a library called "Typography",
that does all the document processing stuff, optimization of the page
breaks, and so on. Finally, it calls another library to output to different
formats, such as pdf, plain text, svg, opengl, or anything else.
>>
>> These are clearly "separable but not independent" projects (drivers,
document formats, font library, typesetting library, parser/code
generator). So compiling everything without duplication is not that
trivial, and I really would like to write an ocaml program to compile it,
maybe using a more complex build system than just one file with the
ocamlbuild api. I understand that ocamlbuild is still in an early stage of
development, this is why I'm sending these mails. The code is split among
different directories, and using a library interface in the compilation of
another library still seems difficult in ocamlbuild today. The
documentation explains how to use an internal library in program, but it
forces you to use a particular layout of your files. This is why I feel
that a library would be more versatile: we can imagine a makefile compiling
the build system first, possibly split between several modules, then
calling it to compile the rest, like xmonad does, for instance.
>>
>> Also, if you are writing a document with our system, separated between
different files, you want the system to compile it automatically, without
too much recompilation. The problem is extensibility: for instance, if you
are writing a bibliography library (I wrote one, but there may be others in
the future), you certainly do not want to touch our code in order to
compile it. Instead, you want to write a dynlinkable module explaining to
our system how the documents depend on bibliography files, and how to
compile everything in what order.
>> To do this, I had to rewrite a (small) ocamlbuild-like library. My point
is that ocamlbuild exposes (or documents) a too small part of its internals
to be re-used inside another project.
>
> I fully agree with you, it would be really nice that ocamlbuild provide a
library interface instead a poor myocamlbuild.ml.
>
> Currently my situation is  that I have to copy my 2000lines
myocamlbuild.ml everywhere, and do some changes locally(probably some bug
fixes will be forgotten to sync up since it's scattered everywhere and I am
lazy), that's not the ideal case.
>
>
> Now, I completely understand Xavier's points:
>
> - about _tags, and this is also my philosophy when I write ocaml code
myself. I asked the question because I needed magic comments in the build
system of my typesetting stuff, mostly because the ocaml source code that I
compile is produced automatically, and meant to be executed only once.
Writing the correct _tags files without conflicts between the different
files, with the correct directory inclusions for instance, would be much
more complicated. This is the only case where I see the point of a mixed
system.
>
> - about the command line options of ghc: clearly, this also has to do
with the fact that ocamlopt manages to compile files of equivalent sizes
about twenty or thirty times faster than ghc in the cases I tried. The
theoretical problems solved are not the same, either.
>
> Finally, Gabriel, I would be happy to contribute code to ocaml, only my
time is a scarce resource right now. Maybe some day…
>
> Cheers,
> Pierre
>
>
> Em 16/11/2012, às 11:53, Gabriel Scherer escreveu:
>
> Regardless of the overly heated tone of the discussion, I find the
> idea of having "ocamlbuild as a library" interesting. Let me detail
> what I believe was the idea of the proposal.
>
> Currently we have a shared ocamlbuild executable that would contain
> the core logic, and be passed plugins that are invoked by the core
> logic at some point of the program. We could consider an inversion of
> control where the executable logic is exposed as a library, and users
> write scripts use this library (to invoke the ocamlbuild processing),
> the script being compiled as whole programs rather than plugins. In
> effect, you give the tool to let any project define it's own "tailored
> ocamlbuild" easily. There could be little change of usage and
> behavior, it's just a different point of view, that has the advantages
> already evoked in this thread (facility to link with other stuff,
> etc.).
>
> This idea has already been used in other projects where "configuration
> is compilation", such as the XMonad window manager (
> http://xmonad.org/ ); the xmonad.hs users write is not compiled to a
> dynamically loaded plugin, it itself invokes the "xmonad program" as a
> library function, passing it (from the language) configuration
> information.
>
> I find this idea interesting, but it will require some amount of work.
> My guess as to why Xavier Clerc's reply is a bit grumpy is that he is
> often the one tasked with implementing those changes. On the other
> hand, everyone is free to contribute to ocamlbuild, and I've been
> trying to publicly encourage people to contribute, with little success
> so far. So Pierre, if you want to have a try at implementing this
> idea, by all means, do! We'll see when someone has some working patch
> if it can be integrated upstream.
>
> Regarding in-file tags/pragmas, I find such changes should be
> considered much more cautiously, as those could mean changes to the
> OCaml tooling (or worse, language) as a whole, not simply ocamlbuild
> itself. Again, nobody's forbidden to come up with a good,
> well-specified, simple proposal that would please everyone, but that
> is a quite difficult task and any such proposal should expect to get
> rejected.
>
> On Fri, Nov 16, 2012 at 11:33 AM, Arnaud Spiwack
> <Arnaud.Spiwack@lix.polytechnique.fr> wrote:
>
> Well,
>
> I must say the sort of mafioso-like reasoning - "you really do not want to
> do that" - doesn't really make sense to me.
>
> Here are valid reasons not to include a feature X in a tool:
>
> I don't have time to do X
> I don't know how to do X
> Having X would prevent me from having Y which I'd rather
>
> Any other reason is bad.
>
> People want to have tags and compilation pragmas? Well, let them, they
> probably have valid reasons to (some options may be better seen global to
> the project and other local to files). People want to interleave compiling
> the build system and compiling the project. They have a good reason for
that
> too. Also: ocamlbuild doesn't only compile ocaml files.
>
> Related to that last point, and to the initial question: as far as I'm
> aware, an ocamlbuild plugin cannot depend on a custom library (to go
around
> that we can use a myocamlbuild.ml which compiles a mymyocamlbuild.ml (or
> something) with the appropriate linking options, th

[-- Attachment #2: Type: text/html, Size: 8988 bytes --]

Re: [Caml-list] About ocamlbuild

[Fabrice Le Fessant]

Hi,

On Fri, Nov 16, 2012 at 12:43 PM, Pierre-Etienne Meunier
<pierreetienne.meunier@gmail.com> wrote:
> So compiling everything without duplication is not that trivial, and I really would like to write an ocaml program to compile it, maybe using a more complex build system than just one file with the ocamlbuild api. I understand that ocamlbuild is still in an early stage of development, this is why I'm sending these mails. The code is split among different directories, and using a library interface in the compilation of another library still seems difficult in ocamlbuild today. The documentation explains how to use an internal library in program, but it forces you to use a particular layout of your files. This is why I feel that a library would be more versatile: we can imagine a makefile compiling the build system first, possibly split between several modules, then calling it to compile the rest, like xmonad does, for instance.


Actually, this use case is exactly the reason why I developed the
"ocp-build" tool (you might have used it if you tried to compile
opam). It's much less powerful than ocamlbuild, but for what it does,
it is much simpler to use. The idea is to write a simple description
of the package contained in each directory. The tool then scans all
the sub-directories looking for these descriptions (in files with .ocp
extensions), use the specified dependencies between packages to order
them, and then build all of them incrementally (no unnecessary
rebuild) and in parallel.

Since the description is only for OCaml files, it is very simple and
short, but of course, if you are doing complicated things, you won't
be able to express them in such a simple language...

There is currently no documentation, and 'opam install ocp-build' will
install a one-year old version (but latest sources are in the typerex2
branch of typerex), so I would not advise anybody to use it curently,
except for playing, but I (and some others in my team) found that it
is a convenient tool for most of our needs.

For example, the opam file looks like :

begin
  comp += [ "-g" "-annot" "-warn-error" "A" ]
  link += [ "-g" ]
begin library "opam-lib"
  subdir = [ "src" ]
  files   = [
    "opamGlobals.ml"     ...
    "repo/opamGit.ml"    "opamSolver.ml"    "opamClient.ml"
  ]
  requires = [    "cudf"    "dose"    "unix"    "graph"     "re_glob" ]
end
begin program "opam"
  subdir  = [ "src" ]
  files    = [   "opamMain.ml" ]
  requires = [    "opam-lib"    "arg" ]
end
begin program "opam-mk-repo"
  files = [ "src/scripts/opam_mk_repo.ml"]
  requires = [ "opam-lib" ]
end
begin program "opam-check"
  files = [ "src/scripts/opam_check.ml" ]
  requires = [ "opam-lib" ]
end
begin program "opam-repo-check"
  files = [ "src/scripts/opam_repo_check.ml" ]
  requires = [ "opam-lib" ]
end
end

Among the current drawbacks, there is no compatibility with findlib
descriptions, nor any easy way to guess the current installed
packages, so you have to create a file describing your environment.
For example, in opam, it comes with a file 'ocaml-libs.ocp'
containing:

begin
  generated = true
  dirname = [ "%{OCAMLLIB}%" ]
  has_byte = true
  has_asm = true
  begin library "unix"  end
  begin library "str"
    requires = [ "unix" ]
  end
  begin library "dynlink"  end
  begin library "camlp4fulllib"
    requires = [ "dynlink" ]
    has_asm = false
  end
  begin library "bigarray"  end
  begin library "threads"
    dirname = [ "%{OCAMLLIB}%/threads" ]
  end
end

to describe all the default libraries. We hope that the tool, at some
point, will be able to either read META files, or packages will come
with an .ocp file describing them...

Finally, another nice thing is the "packing" syntax. For example, to
compile sexplib with ocp-build, we use :

begin library "sexplib"
   files = [
      pack Sexplib [
        "type.ml"; "parser.mly"; "lexer.mll"
        "pre_sexp.ml" (  pp = [ "cpp"; "-undef";  "-traditional" ]        );
        "sexp_intf.ml"; "sexp.ml";
        "path.ml" "conv.ml" "conv_error.ml" "exn_magic.ml" "std.ml"
      ]
   ]
   requires = [ "bigarray" "nums" ]
end

to pack all the modules within the "Sexplib" module.

--Fabrice