Re: [Caml-list] proposal for finding, loading and composing ppx preprocessors

[Benjamin Canou <benjamin.canou@gmail.com>]

Le 11/07/2014 18:13, Alain Frisch a écrit :
> Hi Benjamin,
>
> This topic of how to specify which ppx processors to run, and avoiding 
> multiple processes, is indeed still largely opened.
>
> I don't see what's the benefit of restricting the processors to 
> subtrees.  It's easy enough for each processor to traverse extension 
> nodes it doesn't support (this is the behavior of the default mapper). 
> And I don't think it's a good idea to introduce a composition model 
> different from the successive application of different processors on 
> the entire tree. I.e. function composition, which is quite well 
> understood and easy to reason about.  In particular, you only need to 
> understand the behavior of each processor to predict what the 
> composition will do, not exactly how each processor is implemented 
> (and which state it carries across the tree, internally).
My belief is that there is room for a less ambitious, plug-in based 
(statically or dynamically linked) annotation mechanism, on top of PPX. 
For instance, if we consider the current use of camlp4, we can assume 
that most ppxs will probably just define some annotation on some 
specific kind of AST node in order to rewrite them and / or insert 
auxiliary code, without carrying so much state. Having a common 
mechanism for registering such common simple tasks and assigning to 
names to them could be useful, without breaking the model.

In practice, an annotation would simply be declared as an OCaml module 
calling predefined specific registration functions. Such a module, 
linked with a predefined main module would produce a stand alone ppx 
binary processing only this annotation. However, it gives the 
possibility to compose annotations more finely, without changing the 
semantics. As I mentioned previously, It could limit the number of tree 
rewrites, but it could have other advantages such as detecting 
annotation name clashes.

We launched this thread because we thought that such a mechanism has 
more chances to be adopted if previously discussed and understood by 
build systems, but perhaps the best solution is that we propose such a 
ppx helper and then wait and see.

>
> With ocamlfind 1.5, requiring a package when compiling a file adds the 
> required -ppx flags in addition to the -I flags.  If we want to avoid 
> multiple process, one could create a meta ppx driver which dynamically 
> loads and runs other drivers (specified as .cmxs files).  To be able 
> to use such as meta driver from ocamlfind, ocamlfind needs to know 
> about how to build each ppx processor (i.e. which libraries should be 
> invoked).  Dynamic linking might not be the best solution, though: it 
> is not available on all platforms, and it requires all libraries on 
> which ppx processor depend to provide a corresponding .cmxs.  The 
> alternative is to have ocamlfind link a specific meta driver 
> statically (using its knowledge of package dependencies) for each 
> actual combination of ppx to be used, relying on an internal cache to 
> avoid linking the same driver repeatedly, of course.  The next step is 
> to create not ppx drivers (that incorporate multiple precessors), but 
> compiler drivers (to avoid completely extra process creations and 
> marshaling of the AST).  If this is encapsulated in ocamlfind (or a 
> similar tool), this can still be used by any build system which 
> currently relies on ocamlfind.
>
> Specifying ppx requirements in the source code is a different topic.  
> It might be a good direction, but then I don't see why this should be 
> restricted to ppx requirements and not -I flags. It would seem logical 
> to specify package requirements, which would add both -I and -ppx 
> flags (and maybe more).
I agree on this. Annotations give us the possibility to make OCaml files 
more self content and build-system independent. I though see a 
distinction between compiler directed pragmas and build-system directed 
ones.
>
> Actually, it would have been more important to specify package 
> requirements for Camlp4 processors, since this knowledge might be 
> required by tools that are not under the control of your build system, 
> such as your editor/IDE (to load the corresponding syntactic support). 
> Since the concrete syntax doesn't change anymore with ppx processors, 
> it seems less critical to specify them in the source code (I'd say, 
> not more and not less than general package requirement).
Well, one of the use cases of extension nodes is to integrate external 
notations into literals such as [%json{| ... |}]. I believe IDEs could 
still use a little help to know how to format these, instead of showing 
plain OCaml strings.
>
>
> -- Alain
>
   Benjamin.