Re: [Caml-list] PPX is harmful to our community in the long term

[Jeremie Dimino <jdimino@janestreet.com>]

[-- Attachment #1: Type: text/plain, Size: 5394 bytes --]

Yes, we are now using ocaml-migrate-parsetree at Jane Street. I think that
with this library the versioning story is much better. We are heavy user of
ppx rewriters and all of the code we release now builds with OCaml 4.03 to
4.06 without problems.

Initially we considered using the concrete syntax to solve the versioning
problem and we have a solution that should work in theory. Although it's
not great when there is no source file, such as in the toplevel. The method
we considered is described in [1]. Switching to ocaml-migrate-parsetree was
easier since we didn't have to change anything to the way things worked,
just port the code.

Regarding the idea of cutting down the dependency for release tarballs by
including the generated code, while this is an interesting idea, there are
a lot of small annoying problems that makes this technique hard in
practice. The two main problems are:

1. it doesn't work for .ml files that are generated. Basically all the code
you generate with custom code generators as to be ppx free. It is fine
however for pure generators, since you can just embed the generated code

2. it doesn't work if you use conditional compilation. Conditional
compilation is not great in general, but it is a lot of work to completely
get rid of it

However, one thing that we kept from these experiments is the idea to use
ppx in the same way that expectation tests work. This idea is described in
[1] and we use it for the Base library [2]. Additionally it is a good
testing/debugging tool. It is a viable solution if all you need is
[@@deriving] plugins and don't want to depend on ppx.

The idea is to let the ppx rewriter insert the code generated by
[@@deriving] plugins in the source code:

type t = A | B [@@deriving_inline sexp_of]

let sexp_of_t = function
  | A -> Sexp.Atom "A"
  | B -> Sexp.Atom "B"

[@@@end_of_derived_code]

The code builds without ppx and you still don't have to write the
boilerplate yourself, you let the ppx tool do it for you.

[1] https://blogs.janestreet.com/an-solution-to-the-ppx-versioning-problem/
[2] https://github.com/janestreet/base


On Fri, Apr 21, 2017 at 6:11 PM, Alain Frisch <alain.frisch@lexifi.com>
wrote:

> On 21/04/2017 18:04, Yotam Barnoy wrote:
>
>> My 2 cents: I personally think we did PPX wrong. PPX should have used
>> the syntax as its starting point for full backwards compatibility.
>> Currently, the PPX cycle is
>>
>> OCaml: { syntax -> AST -> serialized AST } -> PPX: {serialized AST ->
>> AST -> PPX modification -> serialized AST} -> OCaml: {serialized AST
>> -> AST -> ... }
>>
>> Exposing the AST is the cause of the problem you mention. If instead,
>> every PPX extension had a particular OCaml compiler's syntax parser
>> and syntax printer integrated into it, and the PPX cycle was:
>>
>> PPX: { syntax -> AST -> PPX modification -> syntax } -> compiler: {
>> syntax -> AST -> ...}
>>
>> We would have far fewer issues with PPX plugins, since they would be
>> as backwards-compatible as the syntax.
>>
>
> I think
>
>   https://github.com/let-def/ocaml-migrate-parsetree
>
> is a much stronger approach.  If I understand correctly, Jane Street moved
> from using a migration system based on concrete syntax to using this new
> project.
>
>
> In your suggested approach, imagine you have a PPX processor written for
> OCaml 4.04.  It assumes that the Parsetree it works on is the one of 4.04.
> Sure, you can compile this PPX processor (+ embedded parser/printer) into a
> stand-alone executable, using OCaml 4.04, and then apply it as a
> preprocessor called from OCaml 4.05.  But this is very impractical: users
> would need to install OCaml 4.04 just to produce the PPX executable.
> Moreover, you loose the ability to combine multiple rewritings in a single
> process, and you have to pay the price of multiple
> parsing/printing/processes.  On top of that:
>
>   (i) you put in the critical loop "pprintast.ml", which has always been
> more or less buggy;
>
>   (ii) there is no hope that an "old PPX" applies to source code using
> newest syntactic features;
>
>   (iii) as you mention, locations would need to be added for each node in
> the AST, which makes the parsing/printing even slower;  moreover, it is not
> technically straightforward to do so, since many places in the AST contains
> locations but do not support attaching attributes;
>
>   (iv) we loose the ability to change the concrete syntax of OCaml, either
> to use alternative syntaxes such as Reason, or to remove, at some point,
> weird corner cases in the current syntax.
>
>
> With ocaml-migrate-parsetree, you can still compile your PPX written for
> 4.04 with newer versions of OCaml.  And it would even be possible (I don't
> know if this is already the case) to apply it to source code using newer
> syntactic features, as long as the rewriting done by the PPX doesn't
> interact with those features.  (This could be achieved by turning new
> syntactic features into attributes/extension nodes when mapping to the
> older AST; and then recognizing these forms and reconstitute the correct
> new AST forms when mapping back to the new version.)
>
>
> -- Alain
>
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>



-- 
Jeremie

[-- Attachment #2: Type: text/html, Size: 9138 bytes --]