Re: [Caml-list] PPX is harmful to our community in the long term

[Alain Frisch <alain.frisch@lexifi.com>]

On 21/04/2017 18:04, Yotam Barnoy wrote:
> My 2 cents: I personally think we did PPX wrong. PPX should have used
> the syntax as its starting point for full backwards compatibility.
> Currently, the PPX cycle is
>
> OCaml: { syntax -> AST -> serialized AST } -> PPX: {serialized AST ->
> AST -> PPX modification -> serialized AST} -> OCaml: {serialized AST
> -> AST -> ... }
>
> Exposing the AST is the cause of the problem you mention. If instead,
> every PPX extension had a particular OCaml compiler's syntax parser
> and syntax printer integrated into it, and the PPX cycle was:
>
> PPX: { syntax -> AST -> PPX modification -> syntax } -> compiler: {
> syntax -> AST -> ...}
>
> We would have far fewer issues with PPX plugins, since they would be
> as backwards-compatible as the syntax.

I think

   https://github.com/let-def/ocaml-migrate-parsetree

is a much stronger approach.  If I understand correctly, Jane Street 
moved from using a migration system based on concrete syntax to using 
this new project.


In your suggested approach, imagine you have a PPX processor written for 
OCaml 4.04.  It assumes that the Parsetree it works on is the one of 
4.04.  Sure, you can compile this PPX processor (+ embedded 
parser/printer) into a stand-alone executable, using OCaml 4.04, and 
then apply it as a preprocessor called from OCaml 4.05.  But this is 
very impractical: users would need to install OCaml 4.04 just to produce 
the PPX executable.  Moreover, you loose the ability to combine multiple 
rewritings in a single process, and you have to pay the price of 
multiple parsing/printing/processes.  On top of that:

   (i) you put in the critical loop "pprintast.ml", which has always 
been more or less buggy;

   (ii) there is no hope that an "old PPX" applies to source code using 
newest syntactic features;

   (iii) as you mention, locations would need to be added for each node 
in the AST, which makes the parsing/printing even slower;  moreover, it 
is not technically straightforward to do so, since many places in the 
AST contains locations but do not support attaching attributes;

   (iv) we loose the ability to change the concrete syntax of OCaml, 
either to use alternative syntaxes such as Reason, or to remove, at some 
point, weird corner cases in the current syntax.


With ocaml-migrate-parsetree, you can still compile your PPX written for 
4.04 with newer versions of OCaml.  And it would even be possible (I 
don't know if this is already the case) to apply it to source code using 
newer syntactic features, as long as the rewriting done by the PPX 
doesn't interact with those features.  (This could be achieved by 
turning new syntactic features into attributes/extension nodes when 
mapping to the older AST; and then recognizing these forms and 
reconstitute the correct new AST forms when mapping back to the new 
version.)


-- Alain