[Caml-list] proposal for finding, loading and composing ppx preprocessors

[Caml-list] proposal for finding, loading and composing ppx preprocessors

[Benjamin Canou]

   Dear OCaml'ers,

To accompany the arrival of annotations in the forthcoming release, we 
propose to discuss and hopefully agree on a standard way of selecting 
the adequate preprocessor(s) for a given file. Gerd already introduced a 
mechanism in the last version of OCamlFind, that we'd like to extend as 
follows.

Our idea is to indicate via a pragma inside the file, how it has to be 
processed. You can see that as a machine readable way of writing (* this 
file has to be preprocessed with blah using options blah *). For this, 
we propose that build systems use a default preprocessor systematically, 
that would interpret these pragmas and find and call its fellow 
preprocessors accordingly.

The compiler already provides built-in annotations. In a way, it's a 
second predefined layer, this time for the build system.

We designed and prototyped such a tool [1], which understand annotations 
of the form [@@@ syntax "package"], to preprocess the following code 
using the ppx declared in the corresponding OCamlFind package.

The tool also defines a way to register plug-ins instead of executables 
so preprocessors can be dynlinked instead of called as processes.

Extensions could be:
  - delimiting a fragment of the code to process
  - add a way to associate a given annotation identifier to a 
preprocessor, so that only the mother preprocessor would do the tree 
traversal, specific preprocessors only seeing the nodes they are 
actually interested in.
  - other kind of pragmas for the build system ?

Again, the goal is to agree now on a standard way and avoid the mess we 
had with camlp4. What do you think ?

   Benjamin, Grégoire and Pierre @ OCamlPro.

[1] https://github.com/hnrgrgr/ppx_loader

Re: [Caml-list] proposal for finding, loading and composing ppx preprocessors

[Daniel Bünzli]

Le vendredi, 11 juillet 2014 à 14:31, Benjamin Canou a écrit :
> Again, the goal is to agree now on a standard way and avoid the mess we
> had with camlp4.  

Rather than *avoid* I'd rather say *improve* over the mess…

> What do you think ?
If I understand correctly this is a ppx for selectively using ppxs over a file. Bonus points for the higher-order ppx.  

Since apparently we have to deal with this crap until we actually get something decent for static meta programming in this language, wouldn't it be easier to just define a standard annotation as you suggest, and then have some kind of "ppxdep" tool that just spits you the flags you will have to put on the command line in order to be able to compile it ?  

I think I'd rather see explicitly and statically what crap extension is being pulled by the sources I'm trying to compile rather than being hidden from all that by that
meta ppx extension.

Daniel

P.S. What happens if I do [@@@ syntax "ppx_loader"] ?  

Re: [Caml-list] proposal for finding, loading and composing ppx preprocessors

[Peter Zotov]

On 2014-07-11 18:21, Daniel Bünzli wrote:
> Le vendredi, 11 juillet 2014 à 14:31, Benjamin Canou a écrit :
>> What do you think ?
> If I understand correctly this is a ppx for selectively using ppxs
> over a file. Bonus points for the higher-order ppx.
> 
> Since apparently we have to deal with this crap until we actually get
> something decent for static meta programming in this language,
> wouldn't it be easier to just define a standard annotation as you
> suggest, and then have some kind of "ppxdep" tool that just spits you
> the flags you will have to put on the command line in order to be able
> to compile it ?

This would require extensive changes to buildsystems (or just to 
ocamldep;
that doesn't make it easier), which, as I understand, this proposal is 
aiming
to avoid.

Personally I don't see anything wrong with just supplying -package to
ocamlfind c invocation, directly, via _tags, via _oasis (and possibly
_tags too, for more finely grained control) or whatever build system are
you using.

Alain Frisch suggested that constant forking might slow down builds, and
suggested some kind of a frontend that would automatically link (not 
even
dynlink) a compound ppx to make builds faster, especially on Windows.

I wonder if we should just get rid of a Unix-like build pipeline 
entirely,
and just make a hybrid frontend-buildsystem that would use compiler-libs
and never execute an external OCaml process. This would make builds even
faster and it's the logical conclusion of Alain's suggestion.

-- 
Peter Zotov
sip:whitequark@sipnet.ru

Re: [Caml-list] proposal for finding, loading and composing ppx preprocessors

[Daniel Bünzli]

Le vendredi, 11 juillet 2014 à 15:36, Peter Zotov a écrit :
> This would require extensive changes to buildsystems (or just to
> ocamldep;

Changes will be required anyway, if only to support the *ultimate* ppx: ppx_loader. I wasn't suggesting in anyway that ocamldep should be changed, ocamldep looks for dependencies among compilation units and that's the only thing it should do (and if one day we can hope to get meta programming in the language itself, it's all that will be needed).  

Best,

Daniel
  

Re: [Caml-list] proposal for finding, loading and composing ppx preprocessors

[Mark Shinwell]

On 11 July 2014 15:36, Peter Zotov <whitequark@whitequark.org> wrote:
> I wonder if we should just get rid of a Unix-like build pipeline entirely,
> and just make a hybrid frontend-buildsystem that would use compiler-libs
> and never execute an external OCaml process. This would make builds even
> faster and it's the logical conclusion of Alain's suggestion.

(There have already been discussions about such a system at Cambridge.)

Mark

Re: [Caml-list] proposal for finding, loading and composing ppx preprocessors

[Benjamin Canou]

Le 11/07/2014 16:21, Daniel Bünzli a écrit :
>
> What do you think ?
> If I understand correctly this is a ppx for selectively using ppxs over a file. Bonus points for the higher-order ppx.
>
> Since apparently we have to deal with this crap until we actually get something decent for static meta programming in this language, wouldn't it be easier to just define a standard annotation as you suggest, and then have some kind of "ppxdep" tool that just spits you the flags you will have to put on the command line in order to be able to compile it ?
>
Sure, the annotation mechanism itself is the core of what we propose. 
The ppx_loader is just a convenience utility to interpret them fully 
automatically. The tool you propose is another valid way to use them. 
We'll integrate it into ppx_loader as another option.
> Daniel
>
> P.S. What happens if I do [@@@ syntax "ppx_loader"] ?
>
It works wonders :)

   Benjamin.

Re: [Caml-list] proposal for finding, loading and composing ppx preprocessors

[Peter Zotov]

On 2014-07-11 18:49, Daniel Bünzli wrote:
> Le vendredi, 11 juillet 2014 à 15:36, Peter Zotov a écrit :
>> This would require extensive changes to buildsystems (or just to
>> ocamldep;
> 
> Changes will be required anyway, if only to support the *ultimate*
> ppx: ppx_loader.

As a matter of fact, no--with ocamlfind 1.5, ppx_loader is supported
on any system that works with ocamlfind.

-- 
Peter Zotov
sip:whitequark@sipnet.ru

Re: [Caml-list] proposal for finding, loading and composing ppx preprocessors

[Daniel Bünzli]

Le vendredi, 11 juillet 2014 à 15:55, Benjamin Canou a écrit :
> Sure, the annotation mechanism itself is the core of what we propose.
> The ppx_loader is just a convenience utility to interpret them fully  
> automatically. The tool you propose is another valid way to use them.  
> We'll integrate it into ppx_loader as another option.

But if I understand correctly ppx_loader selectively applies ppxs on AST subparts (?). If that the case you won't be able to provide matching functionality by simply listing the ppx to use on the command line, which would apply each ppx on the whole file.  

(which illustrates the fundamental uncomposable nature of these things).

Best,

Daniel
  

Re: [Caml-list] proposal for finding, loading and composing ppx preprocessors

[Anil Madhavapeddy]

On 11 Jul 2014, at 15:36, Peter Zotov <whitequark@whitequark.org> wrote:
> 
> I wonder if we should just get rid of a Unix-like build pipeline entirely,
> and just make a hybrid frontend-buildsystem that would use compiler-libs
> and never execute an external OCaml process. This would make builds even
> faster and it's the logical conclusion of Alain's suggestion.

I prototyped one of these a couple of years ago while trying to speed up
the Mirage build system, but required some extensive patching of the compiler
sources and was definitely not suitable for upstreaming.

These days, much of the required functionality is exposed via the
compiler-libs package, so it should be possible to have a go at this
much more easily.

Personally, I'd love to see the use of Sys replaced with the Irmin Git
library database (http://github.com/mirage/irmin).  This would allow a model
where the ML sources are imported into a Git DAG, passed through to the 
compiler-libs as an in-memory structure, and the intermediate results 
recorded in the DAG, still all in-memory.

When the build has completed, the Irmin DAG could be dumped to disk in
the Git file format, permitting the entire build process to be inspected
if necessary (or GCed if not).  All the Git bits are currently supported
by Irmin, so this project just requires an intrepid hacker to connect
compiler-libs, Irmin, and some coordination glue.

-anil

Re: [Caml-list] proposal for finding, loading and composing ppx preprocessors

[Mark Shinwell]

On 11 July 2014 16:27, Anil Madhavapeddy <anil@recoil.org> wrote:
> On 11 Jul 2014, at 15:36, Peter Zotov <whitequark@whitequark.org> wrote:
> Personally, I'd love to see the use of Sys replaced with the Irmin Git
> library database (http://github.com/mirage/irmin).  This would allow a model
> where the ML sources are imported into a Git DAG, passed through to the
> compiler-libs as an in-memory structure, and the intermediate results
> recorded in the DAG, still all in-memory.
>
> When the build has completed, the Irmin DAG could be dumped to disk in
> the Git file format, permitting the entire build process to be inspected
> if necessary (or GCed if not).  All the Git bits are currently supported
> by Irmin, so this project just requires an intrepid hacker to connect
> compiler-libs, Irmin, and some coordination glue.

For those who don't know, we have discussed the idea of using Jenga, a
new dependency engine and build system under development at Jane
Street to do the "coordination" part.  This isn't (yet) being actively
developed, but I'd be pleased to hear from anyone who might be
interested in hacking on this.

Mark

Re: [Caml-list] proposal for finding, loading and composing ppx preprocessors

[Benjamin Canou]

Le 11/07/2014 17:23, Daniel Bünzli a écrit :
> Le vendredi, 11 juillet 2014 à 15:55, Benjamin Canou a écrit :
>> Sure, the annotation mechanism itself is the core of what we propose.
>> The ppx_loader is just a convenience utility to interpret them fully
>> automatically. The tool you propose is another valid way to use them.
>> We'll integrate it into ppx_loader as another option.
> But if I understand correctly ppx_loader selectively applies ppxs on AST subparts (?). If that the case you won't be able to provide matching functionality by simply listing the ppx to use on the command line, which would apply each ppx on the whole file.
True enough. We could force the annotations to be placed at the 
beginning of files. After all, the interest of selectively applying ppxs 
to parts of files is not clear. The responsibility of deciding which 
parts to rewrite probably belongs to each ppx and not ppx_loader.

Re: [Caml-list] proposal for finding, loading and composing ppx preprocessors

[Alain Frisch]

Hi Benjamin,

This topic of how to specify which ppx processors to run, and avoiding 
multiple processes, is indeed still largely opened.

I don't see what's the benefit of restricting the processors to 
subtrees.  It's easy enough for each processor to traverse extension 
nodes it doesn't support (this is the behavior of the default mapper). 
And I don't think it's a good idea to introduce a composition model 
different from the successive application of different processors on the 
entire tree. I.e. function composition, which is quite well understood 
and easy to reason about.  In particular, you only need to understand 
the behavior of each processor to predict what the composition will do, 
not exactly how each processor is implemented (and which state it 
carries across the tree, internally).

With ocamlfind 1.5, requiring a package when compiling a file adds the 
required -ppx flags in addition to the -I flags.  If we want to avoid 
multiple process, one could create a meta ppx driver which dynamically 
loads and runs other drivers (specified as .cmxs files).  To be able to 
use such as meta driver from ocamlfind, ocamlfind needs to know about 
how to build each ppx processor (i.e. which libraries should be 
invoked).  Dynamic linking might not be the best solution, though: it is 
not available on all platforms, and it requires all libraries on which 
ppx processor depend to provide a corresponding .cmxs.  The alternative 
is to have ocamlfind link a specific meta driver statically (using its 
knowledge of package dependencies) for each actual combination of ppx to 
be used, relying on an internal cache to avoid linking the same driver 
repeatedly, of course.  The next step is to create not ppx drivers (that 
incorporate multiple precessors), but compiler drivers (to avoid 
completely extra process creations and marshaling of the AST).  If this 
is encapsulated in ocamlfind (or a similar tool), this can still be used 
by any build system which currently relies on ocamlfind.

Specifying ppx requirements in the source code is a different topic.  It 
might be a good direction, but then I don't see why this should be 
restricted to ppx requirements and not -I flags.  It would seem logical 
to specify package requirements, which would add both -I and -ppx flags 
(and maybe more).

Actually, it would have been more important to specify package 
requirements for Camlp4 processors, since this knowledge might be 
required by tools that are not under the control of your build system, 
such as your editor/IDE (to load the corresponding syntactic support). 
Since the concrete syntax doesn't change anymore with ppx processors, it 
seems less critical to specify them in the source code (I'd say, not 
more and not less than general package requirement).


-- Alain

Re: [Caml-list] proposal for finding, loading and composing ppx preprocessors

[Benjamin Canou]

Le 11/07/2014 18:13, Alain Frisch a écrit :
> Hi Benjamin,
>
> This topic of how to specify which ppx processors to run, and avoiding 
> multiple processes, is indeed still largely opened.
>
> I don't see what's the benefit of restricting the processors to 
> subtrees.  It's easy enough for each processor to traverse extension 
> nodes it doesn't support (this is the behavior of the default mapper). 
> And I don't think it's a good idea to introduce a composition model 
> different from the successive application of different processors on 
> the entire tree. I.e. function composition, which is quite well 
> understood and easy to reason about.  In particular, you only need to 
> understand the behavior of each processor to predict what the 
> composition will do, not exactly how each processor is implemented 
> (and which state it carries across the tree, internally).
My belief is that there is room for a less ambitious, plug-in based 
(statically or dynamically linked) annotation mechanism, on top of PPX. 
For instance, if we consider the current use of camlp4, we can assume 
that most ppxs will probably just define some annotation on some 
specific kind of AST node in order to rewrite them and / or insert 
auxiliary code, without carrying so much state. Having a common 
mechanism for registering such common simple tasks and assigning to 
names to them could be useful, without breaking the model.

In practice, an annotation would simply be declared as an OCaml module 
calling predefined specific registration functions. Such a module, 
linked with a predefined main module would produce a stand alone ppx 
binary processing only this annotation. However, it gives the 
possibility to compose annotations more finely, without changing the 
semantics. As I mentioned previously, It could limit the number of tree 
rewrites, but it could have other advantages such as detecting 
annotation name clashes.

We launched this thread because we thought that such a mechanism has 
more chances to be adopted if previously discussed and understood by 
build systems, but perhaps the best solution is that we propose such a 
ppx helper and then wait and see.

>
> With ocamlfind 1.5, requiring a package when compiling a file adds the 
> required -ppx flags in addition to the -I flags.  If we want to avoid 
> multiple process, one could create a meta ppx driver which dynamically 
> loads and runs other drivers (specified as .cmxs files).  To be able 
> to use such as meta driver from ocamlfind, ocamlfind needs to know 
> about how to build each ppx processor (i.e. which libraries should be 
> invoked).  Dynamic linking might not be the best solution, though: it 
> is not available on all platforms, and it requires all libraries on 
> which ppx processor depend to provide a corresponding .cmxs.  The 
> alternative is to have ocamlfind link a specific meta driver 
> statically (using its knowledge of package dependencies) for each 
> actual combination of ppx to be used, relying on an internal cache to 
> avoid linking the same driver repeatedly, of course.  The next step is 
> to create not ppx drivers (that incorporate multiple precessors), but 
> compiler drivers (to avoid completely extra process creations and 
> marshaling of the AST).  If this is encapsulated in ocamlfind (or a 
> similar tool), this can still be used by any build system which 
> currently relies on ocamlfind.
>
> Specifying ppx requirements in the source code is a different topic.  
> It might be a good direction, but then I don't see why this should be 
> restricted to ppx requirements and not -I flags. It would seem logical 
> to specify package requirements, which would add both -I and -ppx 
> flags (and maybe more).
I agree on this. Annotations give us the possibility to make OCaml files 
more self content and build-system independent. I though see a 
distinction between compiler directed pragmas and build-system directed 
ones.
>
> Actually, it would have been more important to specify package 
> requirements for Camlp4 processors, since this knowledge might be 
> required by tools that are not under the control of your build system, 
> such as your editor/IDE (to load the corresponding syntactic support). 
> Since the concrete syntax doesn't change anymore with ppx processors, 
> it seems less critical to specify them in the source code (I'd say, 
> not more and not less than general package requirement).
Well, one of the use cases of extension nodes is to integrate external 
notations into literals such as [%json{| ... |}]. I believe IDEs could 
still use a little help to know how to format these, instead of showing 
plain OCaml strings.
>
>
> -- Alain
>
   Benjamin.

Re: [Caml-list] proposal for finding, loading and composing ppx preprocessors

[Alain Frisch]

On 07/15/2014 03:59 PM, Benjamin Canou wrote:
> My belief is that there is room for a less ambitious, plug-in based
> (statically or dynamically linked) annotation mechanism, on top of PPX.
> For instance, if we consider the current use of camlp4, we can assume
> that most ppxs will probably just define some annotation on some
> specific kind of AST node in order to rewrite them and / or insert
> auxiliary code, without carrying so much state. Having a common
> mechanism for registering such common simple tasks and assigning to
> names to them could be useful, without breaking the model.

The registration mechanism exists:  a ppx processor is supposed to call 
Ast_mapper.register.  By default, this runs directly the main program to 
support for ppx protocol for this single processor, but you can very 
well override the behavior of Ast_mapper.register by calling 
Ast_mapper.register_function first.

You can look at https://github.com/alainfrisch/ppx_drivers for an 
experiment of creating such a driver.

But this notion of registration is quite independent on how multiple 
processors interact.  I'm yet to be convinced that a composition mode 
more fine-grained that function composition is worth the extra mental 
overhead.  It would also be necessary to decide between top-down and 
bottom-up rewriting, none of them being the best pick for all 
processors.  Typically, a macro expander would want other processors to 
apply to the result of macro expansions (i.e. top-down rewriting style), 
while a type-conv-like processor might prefer a bottom-up rewriting (so 
that the macro expander can run first on type declarations before it 
takes control).

Concerning the notion of state in ppx processors, I've written the 
following ppx processors:

  - sedlex (a lexer generator): it needs to remember about all lexer 
definitions in the current unit to be able to generate some global 
function definitions (shared by various lexers)  (for instance, we don't 
want to generate definitions used only by code that would be excluded by 
a conditional compilation processor)

  - ppx_metaquot: some extension nodes with very short names %e, %t, %p 
are recognized only in the context of an enclosing extension node such 
as %expr ( 
https://github.com/alainfrisch/ppx_tools/blob/master/ppx_metaquot.ml )

I expect most other ppx processors to be slightly more complex than 
purely local transformations, which makes it even more difficult to 
reason on local composition (and "purely local" term rewriting systems 
are already hard to reason about their composition).

> Well, one of the use cases of extension nodes is to integrate external
> notations into literals such as [%json{| ... |}]. I believe IDEs could
> still use a little help to know how to format these, instead of showing
> plain OCaml strings.

Indeed, although I think this is going to be a rather rare use of 
extension points / quoted strings.


-- Alain