Yes, we are now using ocaml-migrate-parsetree at Jane Street. I think that with this library the versioning story is much better. We are heavy user of ppx rewriters and all of the code we release now builds with OCaml 4.03 to 4.06 without problems.

Initially we considered using the concrete syntax to solve the versioning problem and we have a solution that should work in theory. Although it's not great when there is no source file, such as in the toplevel. The method we considered is described in [1]. Switching to ocaml-migrate-parsetree was easier since we didn't have to change anything to the way things worked, just port the code. 

Regarding the idea of cutting down the dependency for release tarballs by including the generated code, while this is an interesting idea, there are a lot of small annoying problems that makes this technique hard in practice. The two main problems are:

1. it doesn't work for .ml files that are generated. Basically all the code you generate with custom code generators as to be ppx free. It is fine however for pure generators, since you can just embed the generated code

2. it doesn't work if you use conditional compilation. Conditional compilation is not great in general, but it is a lot of work to completely get rid of it

However, one thing that we kept from these experiments is the idea to use ppx in the same way that expectation tests work. This idea is described in [1] and we use it for the Base library [2]. Additionally it is a good testing/debugging tool. It is a viable solution if all you need is [@@deriving] plugins and don't want to depend on ppx.

The idea is to let the ppx rewriter insert the code generated by [@@deriving] plugins in the source code:

type t = A | B [@@deriving_inline sexp_of]

let sexp_of_t = function
  | A -> Sexp.Atom "A"
  | B -> Sexp.Atom "B"

[@@@end_of_derived_code]

The code builds without ppx and you still don't have to write the boilerplate yourself, you let the ppx tool do it for you.

[1] https://blogs.janestreet.com/an-solution-to-the-ppx-versioning-problem/
[2] https://github.com/janestreet/base


On Fri, Apr 21, 2017 at 6:11 PM, Alain Frisch <alain.frisch@lexifi.com> wrote:
On 21/04/2017 18:04, Yotam Barnoy wrote:
My 2 cents: I personally think we did PPX wrong. PPX should have used
the syntax as its starting point for full backwards compatibility.
Currently, the PPX cycle is

OCaml: { syntax -> AST -> serialized AST } -> PPX: {serialized AST ->
AST -> PPX modification -> serialized AST} -> OCaml: {serialized AST
-> AST -> ... }

Exposing the AST is the cause of the problem you mention. If instead,
every PPX extension had a particular OCaml compiler's syntax parser
and syntax printer integrated into it, and the PPX cycle was:

PPX: { syntax -> AST -> PPX modification -> syntax } -> compiler: {
syntax -> AST -> ...}

We would have far fewer issues with PPX plugins, since they would be
as backwards-compatible as the syntax.

I think

   https://github.com/let-def/ocaml-migrate-parsetree

is a much stronger approach.  If I understand correctly, Jane Street moved from using a migration system based on concrete syntax to using this new project.


In your suggested approach, imagine you have a PPX processor written for OCaml 4.04.  It assumes that the Parsetree it works on is the one of 4.04.  Sure, you can compile this PPX processor (+ embedded parser/printer) into a stand-alone executable, using OCaml 4.04, and then apply it as a preprocessor called from OCaml 4.05.  But this is very impractical: users would need to install OCaml 4.04 just to produce the PPX executable.  Moreover, you loose the ability to combine multiple rewritings in a single process, and you have to pay the price of multiple parsing/printing/processes.  On top of that:

  (i) you put in the critical loop "pprintast.ml", which has always been more or less buggy;

  (ii) there is no hope that an "old PPX" applies to source code using newest syntactic features;

  (iii) as you mention, locations would need to be added for each node in the AST, which makes the parsing/printing even slower;  moreover, it is not technically straightforward to do so, since many places in the AST contains locations but do not support attaching attributes;

  (iv) we loose the ability to change the concrete syntax of OCaml, either to use alternative syntaxes such as Reason, or to remove, at some point, weird corner cases in the current syntax.


With ocaml-migrate-parsetree, you can still compile your PPX written for 4.04 with newer versions of OCaml.  And it would even be possible (I don't know if this is already the case) to apply it to source code using newer syntactic features, as long as the rewriting done by the PPX doesn't interact with those features.  (This could be achieved by turning new syntactic features into attributes/extension nodes when mapping to the older AST; and then recognizing these forms and reconstitute the correct new AST forms when mapping back to the new version.)


-- Alain


--
Caml-list mailing list.  Subscription management and archives:
https://sympa.inria.fr/sympa/arc/caml-list
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs



--
Jeremie