[Caml-list] META file standards for ppx extensions

[Caml-list] META file standards for ppx extensions

[Dario Teixeira]

Hi,

I think we need to standardise the naming and structure of META
files associated with ppx extensions.  Case in point: I was hoping
to start using sedlex instead of ulex, because the latter has been
deprecated for a while now and all the cool kids are switching to
the former.  However, sedlex currently ships with a broken META
file that severely restricts its usefulness [1].  Therefore, in
order to avoid similar problems with other ppx extensions and to
foster some uniformity in the ecosystem, here's what I suggest:

a) Suppose you have a ppx extension called 'foobar' that relies
    on a runtime library foobar.cma (and native code counterparts,
    which I'll omit from now on).  The top-level declarations in
    foobar's META should reference foobar.cma, but should *not*
    include a ppx declaration.  The ppx declaration should be
    part of a 'ppx' sub-package.

b) If the ppx extension 'foobar' does not have a runtime library,
    then the top-level declarations in the META file should consist
    only of generic version and description information.  The actual
    ppx declaration should be part of a 'ppx' sub-package, as in a).

There is obviously an alternative approach where the ppx invocations
are located at the top-level and any potential runtime libraries
are moved into a 'lib' sub-package.  I prefer the approach outlined
above, however, as it makes ppx invocations explicit and follows the
'syntax' sub-package tradition from Camlp4's glory days.

Thoughts or suggestions?

Best regards,
Dario Teixeira

[1] https://github.com/alainfrisch/sedlex/issues/22

Re: [Caml-list] META file standards for ppx extensions

[Drup]

I'm not fond of b). ppx_deriving et ppx_import don't follow this scheme, 
and that's rather understandable given the absence of runtime library.

Otherwise yes. lwt, js_of_ocaml follow that and it was widely adopted 
for camlp4, so it shouldn't be an issue.

Le 08/04/2015 20:20, Dario Teixeira a écrit :
> Hi,
>
> I think we need to standardise the naming and structure of META
> files associated with ppx extensions.  Case in point: I was hoping
> to start using sedlex instead of ulex, because the latter has been
> deprecated for a while now and all the cool kids are switching to
> the former.  However, sedlex currently ships with a broken META
> file that severely restricts its usefulness [1].  Therefore, in
> order to avoid similar problems with other ppx extensions and to
> foster some uniformity in the ecosystem, here's what I suggest:
>
> a) Suppose you have a ppx extension called 'foobar' that relies
>    on a runtime library foobar.cma (and native code counterparts,
>    which I'll omit from now on).  The top-level declarations in
>    foobar's META should reference foobar.cma, but should *not*
>    include a ppx declaration.  The ppx declaration should be
>    part of a 'ppx' sub-package.
>
> b) If the ppx extension 'foobar' does not have a runtime library,
>    then the top-level declarations in the META file should consist
>    only of generic version and description information.  The actual
>    ppx declaration should be part of a 'ppx' sub-package, as in a).
>
> There is obviously an alternative approach where the ppx invocations
> are located at the top-level and any potential runtime libraries
> are moved into a 'lib' sub-package.  I prefer the approach outlined
> above, however, as it makes ppx invocations explicit and follows the
> 'syntax' sub-package tradition from Camlp4's glory days.
>
> Thoughts or suggestions?
>
> Best regards,
> Dario Teixeira
>
> [1] https://github.com/alainfrisch/sedlex/issues/22
>
>

Re: [Caml-list] META file standards for ppx extensions

[Dario Teixeira]

Hi,

> I'm not fond of b). ppx_deriving et ppx_import don't follow
> this scheme, and that's rather understandable given the absence
> of runtime library.

Well, if the most popular ppx extensions don't have a runtime
library, this is an argument in favour of the alternative scheme
of declaring the ppx at the top-level of the META file, and the
runtime as a 'lib' or 'runtime' sub-package where applicable.
Personally, I don't think this argument is strong enough to tip
the balance, and I still favour the original scheme I proposed,
but I'm not very adamant about it.

*However*, regardless of what scheme is adopted, can we at least
agree on the need for it to be uniformly applied in the ecosystem?
I think the worst path is the one we're currently following, where
each ppx extension does as it pleases.


> Otherwise yes. lwt, js_of_ocaml follow that and it was widely
> adopted for camlp4, so it shouldn't be an issue.

Well, newcomers to OCaml will not recall this, but we had a similar
discussion some years ago about Camlp4.  At the time there was no
'syntax' standard, and most Camlp4 extensions were not even well
integrated with findlib.  I think we can all agree that using Camlp4
extensions became much easier and saner after that standard was
adopted by the community.

Best regards,
Dario Teixeira

Re: [Caml-list] META file standards for ppx extensions

[Daniel Bünzli]

It seems to me that Dario's scheme, that is always have PKG.ppx, is more forward looking. We could perfectly imagine that a library in the future will implement its functionality using another mechanism (for example in a non-ugly way, using a macro system). 

We can imagine current packages that are using camlp4 providing both a legacy PKG.camlp4 and a PKG.ppx package.

It also makes it more clear in the build system specification what crap pre-processing technology is being pulled in. 

Best,

Daniel

Re: [Caml-list] META file standards for ppx extensions

[Dario Teixeira]

Hi,

> It seems to me that Dario's scheme, that is always have PKG.ppx, is
> more forward looking. We could perfectly imagine that a library in
> the future will implement its functionality using another mechanism
> (for example in a non-ugly way, using a macro system).

Yes, and in the meantime I thought of another reason why the original
scheme (runtime -- if any -- at the META top-level and ppx as a 'ppx'
sub-package) is superior to the alternative (ppx at the top-level
and runtime as 'lib' sub-package): there are libraries where the use
of the syntax/ppx extension is just a nice bonus (LWT, PG'OCaml).
In these cases, the original scheme is the only one that makes sense.

Though there are of course extensions without a runtime where
the alternative scheme would be more sensible, we then run into
the consistency argument: it's preferable to bend one side to the
other than to have a hodgepodge of schemes.  And below you'll find
additional reasons why the original scheme wins decisively.


> We can imagine current packages that are using camlp4 providing
> both a legacy PKG.camlp4 and a PKG.ppx package.

Agreed.  Case in point: LWT.  I don't know when the Ocsigen team
plans to eliminate the LWT Camlp4 syntax extension, but I suspect
that for some time both the Camlp4 and ppx extensions will ship
with LWT.  The former can and should remain accessible via the
lwt.syntax sub-package, whereas the latter is available via
lwt.ppx.

Btw, currently legacy Camlp4 syntax extensions are available as a
PKG.syntax sub-package (not PKG.camlp4), and I see no compelling
reason to change this.


> It also makes it more clear in the build system specification
> what crap pre-processing technology is being pulled in.

Well, apart from the inflammatory use of 'crap', I fully agree!

Best regards,
Dario Teixeira

Re: [Caml-list] META file standards for ppx extensions

[Gerd Stolpmann]

[-- Attachment #1: Type: text/plain, Size: 2650 bytes --]

> *However*, regardless of what scheme is adopted, can we at least
> agree on the need for it to be uniformly applied in the ecosystem?
> I think the worst path is the one we're currently following, where
> each ppx extension does as it pleases.

There is still the question how to set the standard. Just talking about
it isn't sufficient. My thinking here is that there should be a couple
of pointers into the right direction giving advice to the developers.
There could be a good ppx example in the findlib manual (or maybe a link
to a blog post explaining the issue). I'm also thinking about a helper
inside ocamlfind, maybe

ocamlfind check META

that would (a) search for hard errors, and (b) give recommendations. For
example, if a library cannot be linked at all, or not be loaded into the
toploop, this would be (a). If the ppx preprocessor isn't separated from
the runtime, this would be (b) (i.e. using ppx together with archive).

The question is, of course, what to recommend.

> Well, newcomers to OCaml will not recall this, but we had a similar
> discussion some years ago about Camlp4.  At the time there was no
> 'syntax' standard, and most Camlp4 extensions were not even well
> integrated with findlib.  I think we can all agree that using Camlp4
> extensions became much easier and saner after that standard was
> adopted by the community.

Well, camlp4 support in findlib is pretty old (already version 0.4
contained it, and I think it was released around 2001 - even before
camlp4 was integrated into the main ocaml tree). It took some time until
it was really usable (in particular, that subpackages were added), but
it also took some time until it was discovered by the developers. Then
the question was how to do it (i.e. about conventions). So far I
remember, good examples were quite helpful to drive the adoption of
easily usable camlp4 extensions.

(A side note: for camlp4 extensions, you always need to specify a
-syntax in order to activate the extension. Because of this, it is not
that important to cleanly separate the preprocessor from the runtime.
Maybe we want something similar for ppx?)

Do we have a project that could be used as reference for ppx?

Gerd

> 
> Best regards,
> Dario Teixeira
> 
> 

-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
My OCaml site:          http://www.camlcity.org
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Caml-list] META file standards for ppx extensions

[Dario Teixeira]

Hi,

> There is still the question how to set the standard. Just talking
> about it isn't sufficient. My thinking here is that there should
> be a couple of pointers into the right direction giving advice to
> the developers.  There could be a good ppx example in the findlib
> manual (or maybe a link to a blog post explaining the issue).

Fair point.  At the end of this email I'm posting a first draft of
META samples for all ppx variations.  (It's bound to be incomplete
and/or contain errors, so please share additions and/or corrections!)


> I'm also thinking about a helper inside ocamlfind, maybe
> 
> ocamlfind check META
> 
> that would (a) search for hard errors, and (b) give
> recommendations. For example, if a library cannot be linked at all,
> or not be loaded into the toploop, this would be (a). If the ppx
> preprocessor isn't separated from the runtime, this would be (b)
> (i.e. using ppx together with archive).

That would be interesting indeed.


> Do we have a project that could be used as reference for ppx?

LWT is doing things the right way, I think.

Best regards,
Dario Teixeira


####################################################################

META file skeletons for ppx extensions, 1st draft:

Broadly, there's three types of packages defining ppx extensions:
1) libraries where ppx is optional candy, 2) primarily ppx extensions
with mandatory runtime, and 3) primarily ppx extensions without
runtime.  I've named the sample packages for each case as Fooloot,
Foolex, and Foonad, respectively.  Below you'll find sample META
files for each case.


1) Package "Fooloot" is like LWT: it's a library that can be used
    without a camlp4/ppx extension, but it also offers an optional
    camlp4 and/or ppx extension.

    ###### META begins #####
    version = "1.0"
    description = "Fooloot library"
    archive(byte) = "fooloot.cma"
    archive(byte, plugin) = "fooloot.cma"
    archive(native) = "fooloot.cmxa"
    archive(native, plugin) = "fooloot.cmxs"

    package "syntax" (
      version = "1.0"
      description = "Camlp4-based syntax extension for Fooloot"
      requires = "camlp4 fooloot"
      archive(syntax, preprocessor) = "pa_fooloot.cma"
      archive(syntax, toploop) = "pa_fooloot.cma"
      archive(syntax, preprocessor, native) = "pa_fooloot.cmxa"
      archive(syntax, preprocessor, native, plugin) = "pa_fooloot.cmxs"
    )

    package "ppx" (
      version = "1.0"
      description = "New-style (ppx) syntax extension for Fooloot"
      requires = "fooloot"
      ppx = "ppx_fooloot"
    )
    ##### META ends #####


2) Package "Foolex" is like Sedlex: it's used only as a ppx
    extension, but it requires a runtime.

    ###### META begins #####
    version = "1.0"
    description = "Foolex library"
    archive(byte) = "foolex.cma"
    archive(byte, plugin) = "foolex.cma"
    archive(native) = "foolex.cmxa"
    archive(native, plugin) = "foolex.cmxs"

    package "ppx" (
      version = "1.0"
      description = "New-style (ppx) syntax extension for Foolex"
      requires = "foolex"
      ppx = "ppx_foolex"
    )
    ##### META ends #####


3) Package "Foonad" is like ppx_monad: it's used only as a ppx
    extension, and it requires no runtime.

    ###### META begins #####
    version = "1.0"
    description = "Foonad"

    package "ppx" (
      version = "1.0"
      description = "New-style (ppx) syntax extension for Foonad"
      ppx = "ppx_foonad"
    )
    ##### META ends #####

Re: [Caml-list] META file standards for ppx extensions

[Daniel Bünzli]

In the assemblage project we are quite interested in seeing packages as named entities providing flags/and or objects to use in a given command execution context (defined as pre-processing, compiling, linking, etc. see [1] for more details).

During the first build ("configure") we want to `ocamlfind query` packages for the right flags in each context and cache them so that we don't have to go through ocamlfind during the compilation (which slows down things quite a bit). 

Is there an invocation that allow us to get the flags or objects to use, the following don't seem to work:

> ocamlfind query -predicates ppx -pp lwt.ppx
> ocamlfind query -pp lwt.ppx 


and I don't see an obvious specifier for `-format` I can use. 

It would be nice in general if ocamlfind would allow us to use `ocamlfind query` in obvious ways so that we can exactly reproduce what it does when whatever `ocamlfind TOOL -pkg PKG1,PKG2,PKG3` is being invoked. 

I'm not working on this at the moment and don't have these things in my head but I remember that it proved to be quite challenging (if not impossible) to achieve that goal. 

Best,

Daniel

[1] http://samoht.github.io/assemblage/Assemblage.Ctx.html

Re: [Caml-list] META file standards for ppx extensions

[Thomas Gazagnaire]

Hi,

first I think it's a good idea to have a proper convention for ppx pre-processors. And I like your proposal. I just have a question, though:

>   package "syntax" (
>     version = "1.0"
>     description = "Camlp4-based syntax extension for Fooloot"
>     requires = "camlp4 fooloot"
>     archive(syntax, preprocessor) = "pa_fooloot.cma"
>     archive(syntax, toploop) = "pa_fooloot.cma"
>     archive(syntax, preprocessor, native) = "pa_fooloot.cmxa"
>     archive(syntax, preprocessor, native, plugin) = "pa_fooloot.cmxs"
>   )
> 
>   package "ppx" (
>     version = "1.0"
>     description = "New-style (ppx) syntax extension for Fooloot"
>     requires = "fooloot"
>     ppx = "ppx_fooloot"
>   )

How do you compose ppx extensions using META? Can you require "barloot.ppx"? The camlp4 "syntax" packages are usual ocaml libraries, so the rules of composition are (almost) clear. How does it work when you want to compose ppx rewriters?

Thomas

Re: [Caml-list] META file standards for ppx extensions

[Dario Teixeira]

Hi,

> How do you compose ppx extensions using META? Can you require
> "barloot.ppx"? The camlp4 "syntax" packages are usual ocaml
> libraries, so the rules of composition are (almost) clear.
> How does it work when you want to compose ppx rewriters?

Gerd can provide a more authoritative answer, but I reckon
ocamlfind is smart enough to handle such cases.  You can
already require a sub-package after all, and involving ppx
should not change this.

Best regards,
Dario Teixeira

Re: [Caml-list] META file standards for ppx extensions

[Gerd Stolpmann]

[-- Attachment #1: Type: text/plain, Size: 1884 bytes --]

Am Donnerstag, den 09.04.2015, 17:33 +0200 schrieb Daniel Bünzli:
> In the assemblage project we are quite interested in seeing packages as named entities providing flags/and or objects to use in a given command execution context (defined as pre-processing, compiling, linking, etc. see [1] for more details).
> 
> During the first build ("configure") we want to `ocamlfind query` packages for the right flags in each context and cache them so that we don't have to go through ocamlfind during the compilation (which slows down things quite a bit). 
> 
> Is there an invocation that allow us to get the flags or objects to use, the following don't seem to work:
> 
> > ocamlfind query -predicates ppx -pp lwt.ppx
> > ocamlfind query -pp lwt.ppx 
> 
> 
> and I don't see an obvious specifier for `-format` I can use. 

ocamlfind query -format '%(ppx)' lwt.ppx

-pp is not meant for ppx.

> It would be nice in general if ocamlfind would allow us to use `ocamlfind query` in obvious ways so that we can exactly reproduce what it does when whatever `ocamlfind TOOL -pkg PKG1,PKG2,PKG3` is being invoked. 

E.g.

ocamlfind ocamlc -only-show ...

then get the command from stdout and run it as often as you want.

Gerd

> I'm not working on this at the moment and don't have these things in my head but I remember that it proved to be quite challenging (if not impossible) to achieve that goal. 
> 
> Best,
> 
> Daniel
> 
> [1] http://samoht.github.io/assemblage/Assemblage.Ctx.html
> 
> 
> 

-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
My OCaml site:          http://www.camlcity.org
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Caml-list] META file standards for ppx extensions

[Gerd Stolpmann]

[-- Attachment #1: Type: text/plain, Size: 1225 bytes --]

Am Donnerstag, den 09.04.2015, 17:28 +0100 schrieb Dario Teixeira:
> Hi,
> 
> > How do you compose ppx extensions using META? Can you require
> > "barloot.ppx"? The camlp4 "syntax" packages are usual ocaml
> > libraries, so the rules of composition are (almost) clear.
> > How does it work when you want to compose ppx rewriters?
> 
> Gerd can provide a more authoritative answer, but I reckon
> ocamlfind is smart enough to handle such cases.  You can
> already require a sub-package after all, and involving ppx
> should not change this.

As ppx rewriters are commands, and as you can specify the -ppx option
several times to invoke several commands, the answer should be as clear
as for the library case: The commands are just run one after the other,
and if there is a dependency "ppx1 requires ppx2", the command for ppx2
is run first.

Gerd
-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
My OCaml site:          http://www.camlcity.org
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Caml-list] META file standards for ppx extensions

[Daniel Bünzli]

Le jeudi, 9 avril 2015 à 18:45, Gerd Stolpmann a écrit :
> ocamlfind ocamlc -only-show ...
>  
> then get the command from stdout and run it as often as you want.
Ah thanks ! That seems much better than the outputs of `ocamlfind query`. Is there any way to separate compilation instructions from pre-processing instructions ?  

Best,

Daniel

Re: [Caml-list] META file standards for ppx extensions

[Daniel Bünzli]

Le jeudi, 9 avril 2015 à 19:27, Daniel Bünzli a écrit :
> Ah thanks ! That seems much better than the outputs of `ocamlfind query`. Is there any way to separate compilation instructions from pre-processing instructions ?

Also why is -ppx passed during the linking phase ?  

> ocamlfind ocamlc -linkpkg -only-show -package react,lwt.ppx
ocamlc.opt -I /Users/dbuenzli/.opam/4.02.1/lib/react -I /Users/dbuenzli/.opam/4.02.1/lib/bytes -I /Users/dbuenzli/.opam/4.02.1/lib/lwt -ppx ppx_lwt /Users/dbuenzli/.opam/4.02.1/lib/react/react.cma /Users/dbuenzli/.opam/4.02.1/lib/lwt/lwt.cma /Users/dbuenzli/.opam/4.02.1/lib/lwt/ppx.cma



Daniel

Re: [Caml-list] META file standards for ppx extensions

[Gerd Stolpmann]

[-- Attachment #1: Type: text/plain, Size: 858 bytes --]

Am Donnerstag, den 09.04.2015, 19:27 +0200 schrieb Daniel Bünzli:
> Le jeudi, 9 avril 2015 à 18:45, Gerd Stolpmann a écrit :
> > ocamlfind ocamlc -only-show ...
> >  
> > then get the command from stdout and run it as often as you want.
> Ah thanks ! That seems much better than the outputs of `ocamlfind query`. Is there any way to separate compilation instructions from pre-processing instructions ?  

No - you'd need to parse the command and watch out for -pp and -ppx
options.

Gerd
-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
My OCaml site:          http://www.camlcity.org
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Caml-list] META file standards for ppx extensions

[Gerd Stolpmann]

[-- Attachment #1: Type: text/plain, Size: 1385 bytes --]

Am Donnerstag, den 09.04.2015, 20:05 +0200 schrieb Daniel Bünzli:
> Le jeudi, 9 avril 2015 à 19:27, Daniel Bünzli a écrit :
> > Ah thanks ! That seems much better than the outputs of `ocamlfind query`. Is there any way to separate compilation instructions from pre-processing instructions ?
> 
> Also why is -ppx passed during the linking phase ?  

-linkpkg enables linking but does not disable compilation. You could
also have something like

ocamlfind ocamlc -linkpkg -o executable ... module.ml

I think the -ppx option is ignored by ocamlc if there is no module
source, so ocamlfind doesn't care about it.

Gerd

> > ocamlfind ocamlc -linkpkg -only-show -package react,lwt.ppx
> ocamlc.opt -I /Users/dbuenzli/.opam/4.02.1/lib/react -I /Users/dbuenzli/.opam/4.02.1/lib/bytes -I /Users/dbuenzli/.opam/4.02.1/lib/lwt -ppx ppx_lwt /Users/dbuenzli/.opam/4.02.1/lib/react/react.cma /Users/dbuenzli/.opam/4.02.1/lib/lwt/lwt.cma /Users/dbuenzli/.opam/4.02.1/lib/lwt/ppx.cma
> 
> 
> 
> Daniel
> 
> 
> 

-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
My OCaml site:          http://www.camlcity.org
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Caml-list] META file standards for ppx extensions

[Daniel Bünzli]

Le vendredi, 10 avril 2015 à 00:21, Gerd Stolpmann a écrit :
> No - you'd need to parse the command and watch out for -pp and -ppx
> options.

[…]
> I think the -ppx option is ignored by ocamlc if there is no module
> source, so ocamlfind doesn't care about it.

That seems quite brittle and unprecise and are things that get in the way/you have to consider when you need to debug build systems.

It seems to me that OCaml's build rules are already sufficiently complex so that the tool that is supposed to help us to manage them doesn't introduce more noise in the system.

More specifically I think that most build system developers out there would agree that having a clear ocamlfind sub-command that allows us to query the *exact* flags a package is supposed to provide us along the well defined phases: pre-processing, compilation and linking in the native and bytecode dimensions would be of great help in general.  

Would you maybe consider implementing something along these lines ?  

Best,

Daniel

Re: [Caml-list] META file standards for ppx extensions

[François Bobot]

On 10/04/2015 01:06, Daniel Bünzli wrote:
>  well defined phases: pre-processing, compilation and linking in the native and bytecode dimensions 
would be of great help in general.

Just by curiosity, why do you want to separate the preprocessing from the compilation and not let 
ocaml do the piping? Is it for efficiency?

Regards,

-- 
François

Re: [Caml-list] META file standards for ppx extensions

[Daniel Bünzli]

Le vendredi, 10 avril 2015 à 10:53, François Bobot a écrit :
> Just by curiosity, why do you want to separate the preprocessing from the compilation and not let
> ocaml do the piping? Is it for efficiency?

Maybe Thomas will have other comments to make since I personally stay away from the pre-processing cancer.  

But for one thing you can perfectly imagine doing a source pre-processing step before making a distribution, so that final users don't need to install the pre-processor you are using. Besides if you have a decent build system which finely tracks dependencies for minimal rebuilds you can perfectly imagine to not re-pre-process a file if only a *compilation* option changes.

In general it's better if the base compilation tools do not try to do too much/output many files in one go so that build systems can 1) trivially infer their outputs 2) reasonably rely on fs modification times to equate them with concrete content modification 3) be really in charge of composing the build pipeline.  

It's all about having a *composable* toolchain vs. inflexible monolithic black boxes.  

Best,

Daniel

Re: [Caml-list] META file standards for ppx extensions

[Alain Frisch]

On 04/10/2015 11:42 AM, Daniel Bünzli wrote:
> But for one thing you can perfectly imagine doing a source pre-processing step before making a distribution, so that final users don't need to install the pre-processor you are using. Besides if you have a decent build system which finely tracks dependencies for minimal rebuilds you can perfectly imagine to not re-pre-process a file if only a *compilation* option changes.

It's better than that: you can avoid pre-processing if dependencies 
change but not the current unit's code (if you know that the 
preprocessors is not going to read compilation artefacts, but they 
usually don't).  This is far more common than changing compilation options.

Alain

Re: [Caml-list] META file standards for ppx extensions

[François Bobot]

On 10/04/2015 11:42, Daniel Bünzli wrote:
> It's all about having a *composable* toolchain vs. inflexible monolithic black boxes.

In order to gain flexibility, do you tried to link with findlib? I found this library very clean, 
and powerful. I think Gerd is good at keeping backward compatibility.

My two cents,

-- 
François

Re: [Caml-list] META file standards for ppx extensions

[Gerd Stolpmann]

[-- Attachment #1: Type: text/plain, Size: 2948 bytes --]

Am Freitag, den 10.04.2015, 01:06 +0200 schrieb Daniel Bünzli:
> Le vendredi, 10 avril 2015 à 00:21, Gerd Stolpmann a écrit :
> > No - you'd need to parse the command and watch out for -pp and -ppx
> > options.
> 
> […]
> > I think the -ppx option is ignored by ocamlc if there is no module
> > source, so ocamlfind doesn't care about it.
> 
> That seems quite brittle and unprecise 

No, it is well-defined.

> and are things that get in the way/you have to consider when you need to debug build systems.

Why is it in the way? In a build system you compile normally module by
module, and separate linking out, but only because you can then specify
the options module by module, or by linking situation. However, ocamlc
already offers a superset of this, and why should ocamlfind restrict
that? It has always been the philosophy of findlib that it supports
everything ocamlc does, and in the same way, i.e. you can turn

ocamlc <options>

always into

ocamlfind ocamlc <options>

only that you have additional options. This philosophy is part of the
success story of findlib, and I won't change that.

> It seems to me that OCaml's build rules are already sufficiently complex so that the tool that is supposed to help us to manage them doesn't introduce more noise in the system.
> 
> More specifically I think that most build system developers out there would agree that having a clear ocamlfind sub-command that allows us to query the *exact* flags a package is supposed to provide us along the well defined phases: pre-processing, compilation and linking in the native and bytecode dimensions would be of great help in general.  

I think you are hunting a ghost animal. The ocamlfind wrapper is fast
enough that you can ignore it, and skipping it will only so minimally
speed up builds that it is not worth doing it.

If you want to develop something in this direction, I'd suggest a
different path, namely a new driver for the ocaml compilers that runs as
a server, and that communicates with the build system over a pipe. Also,
what we can talk about is that ocamlfind functionality is completely
available as library (as of now there is only package lookup in the
library, but not command construction). This would allow it to link it
into your build system, and simply call the functions directly. But as
said, alone this will not be the big accelerator, only in conjunction
with a true compile server.

Gerd

> Would you maybe consider implementing something along these lines ?  
> 
> Best,
> 
> Daniel
> 
> 
> 

-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
My OCaml site:          http://www.camlcity.org
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Caml-list] META file standards for ppx extensions

[Thomas Gazagnaire]

>> But for one thing you can perfectly imagine doing a source pre-processing step before making a distribution, so that final users don't need to install the pre-processor you are using. Besides if you have a decent build system which finely tracks dependencies for minimal rebuilds you can perfectly imagine to not re-pre-process a file if only a *compilation* option changes.
> 
> It's better than that: you can avoid pre-processing if dependencies change but not the current unit's code (if you know that the preprocessors is not going to read compilation artefacts, but they usually don't).  This is far more common than changing compilation options.

Exactly. And also, most of the current build systems call the pre-processor three times: one with ocamldep -pp, one with ocamlc -pp and one with ocamlopt -pp. The preprocessor use byte-code (as the native libraries for the pre-processors are usually not installed) and dynamic linking which is slow -- various naive (and probably biais) experiments showed that the compilation time is usually negligible compared to the pre-processing time, so you usually have a 2/3 speed-up.

Thomas

Re: [Caml-list] META file standards for ppx extensions

[Daniel Bünzli]

Le vendredi, 10 avril 2015 à 13:25, Gerd Stolpmann a écrit :
> > and are things that get in the way/you have to consider when you need to debug build systems.
>  
> Why is it in the way?

Passing uneeded arguments to utility invocations is noisy and confusing when I need to understand what is happening. These are instructions I have to consider to eventually realize they end up doing nothing.  

I guess you wouldn't find it acceptable to have a program that calls functions passing them random arguments that have no effects whatsoever. Why should it be so on the command line ?

> I think you are hunting a ghost animal. The ocamlfind wrapper is fast
> enough that you can ignore it, and skipping it will only so minimally
> speed up builds that it is not worth doing it.

As was already mentioned in the discussion, there are benefits beyond saving the cost to the ocamlfind wrapper to be able to cleanly separate the compilation phases.

Best,

Daniel

Re: [Caml-list] META file standards for ppx extensions

[Daniel Bünzli]

Le vendredi, 10 avril 2015 à 13:04, François Bobot a écrit :
> On 10/04/2015 11:42, Daniel Bünzli wrote:
> > It's all about having a *composable* toolchain vs. inflexible monolithic black boxes.
>  
> In order to gain flexibility, do you tried to link with findlib?

If you are refering to this  

https://github.com/whitequark/ocaml-findlib/blob/master/src/findlib/findlib.mli

then it doesn't seem to be much different from `ocamlfind query`, which as I already mentioned I had hard time to use to achieve my goals. But then maybe I should simply short-circuit all the ocamlfind flags logic and simply query the objects in the META files (which I didn't understand at the time could be done by simply using  "(%property)" notation; I used to fight with the various other % formatters).  

Best,

Daniel

Re: [Caml-list] META file standards for ppx extensions

[Daniel Bünzli]

Le jeudi, 9 avril 2015 à 14:24, Dario Teixeira a écrit :
> Below you'll find sample META files for each case.

One thing this doesn't seem to answer is where the ppx binary is installed. Doing a  

> find .opam/4.02.1/lib -name "META" | xargs cat | grep "^ *ppx"

On my machine yields the following:  
  
ppx = "ppx_lwt"
ppx = "./ppx_metaquot"
ppx(-byte)="./ppx_xtmpl"

ppx(byte)="./ppx_xtmpl.byte"

which seems inconsistant with the semantics of fields in META files which usually express their object w.r.t. to the package directory.  

Best,

Daniel

Re: [Caml-list] META file standards for ppx extensions

[Gerd Stolpmann]

[-- Attachment #1: Type: text/plain, Size: 1195 bytes --]

Am Freitag, den 10.04.2015, 14:23 +0200 schrieb Daniel Bünzli:
> Le jeudi, 9 avril 2015 à 14:24, Dario Teixeira a écrit :
> > Below you'll find sample META files for each case.
> 
> One thing this doesn't seem to answer is where the ppx binary is installed. Doing a  
> 
> > find .opam/4.02.1/lib -name "META" | xargs cat | grep "^ *ppx"
> 
> On my machine yields the following:  
>   
> ppx = "ppx_lwt"
> ppx = "./ppx_metaquot"
> ppx(-byte)="./ppx_xtmpl"
> 
> ppx(byte)="./ppx_xtmpl.byte"
> 
> which seems inconsistant with the semantics of fields in META files which usually express their object w.r.t. to the package directory.  

It is described in the man page for META. A command is not an object
file, and it would be inconsistent with the conceptual peculiarities if
we handled commands as objects.

Gerd
-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
My OCaml site:          http://www.camlcity.org
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Caml-list] META file standards for ppx extensions

[François Bobot]

On 10/04/2015 13:55, Daniel Bünzli wrote:
> Le vendredi, 10 avril 2015 à 13:04, François Bobot a écrit :
>> On 10/04/2015 11:42, Daniel Bünzli wrote:
>>> It's all about having a *composable* toolchain vs. inflexible monolithic black boxes.
>>
>> In order to gain flexibility, do you tried to link with findlib?
>
> If you are refering to this
>
> https://github.com/whitequark/ocaml-findlib/blob/master/src/findlib/findlib.mli
>
> then it doesn't seem to be much different from `ocamlfind query`, which as I already mentioned I
> had hard time to use to achieve my goals. But then maybe I should simply short-circuit all the
> ocamlfind flags logic and simply query the objects in the META files (which I didn't understand
> at the time could be done by simply using  "(%property)" notation; I used to fight with the
> various other % formatters).

findlib.mli doesn't implement the various % formatters. And it gives a much easier way to access the 
properties than to parse the output of ocamlfind.

You can look at the implementation of the frontend for how to use the library (ex: to use 
Fl_split.in_words) :
https://github.com/whitequark/ocaml-findlib/blob/master/src/findlib/frontend.ml

Regards,

-- 
François

Re: [Caml-list] META file standards for ppx extensions

[Daniel Bünzli]

Le vendredi, 10 avril 2015 à 18:33, François Bobot a écrit :
> findlib.mli doesn't implement the various % formatters. And it gives a much easier way to access the
> properties than to parse the output of ocamlfind.

The %(property) queries seems reliable (in contrast to other queries which sport seemingly random duplicate outputs and/or empty lines according to which predicates are present) so I'll rather go with that. assemblage's package support tries to be rather generic so that other systems can be plugged in --- like pkg-config for C and/or anything better we may get in the future — so I'd rather not make the core assemblage library depend on this.

Best,

Daniel

Re: [Caml-list] META file standards for ppx extensions

[Anil Madhavapeddy]

On 10 Apr 2015, at 17:42, Daniel Bünzli <daniel.buenzli@erratique.ch> wrote:
> 
> In general it's better if the base compilation tools do not try to do too much/output many files in one go so that build systems can 1) trivially infer their outputs 2) reasonably rely on fs modification times to equate them with concrete content modification 3) be really in charge of composing the build pipeline.  
> 
> It's all about having a *composable* toolchain vs. inflexible monolithic black boxes.  

From a longer term perspective, the disentangling of preprocessing from compilation also makes it easier to do multi-stage preprocessing -- for instance, to perform type-driven I/O generation separately from an Lwt extension. The former affects dependency calculation, and the latter doesn't and is usually quite a local transformation within a module.  Why should they be clumped together in the build system just because they happen to use the same ppx/camlp4 mechanism?

Those examples aren't multistage yet, but the lines between MetaCaml, Camlp4, and (a hypothetical) OCaml macro system are pretty blurred.  OCaml is one of the few languages with a really good story for a statically typed processing pipeline, and it would be a great pity if build systems become the blocker for experimenting with them easily...

-Anil