From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: caml-list@sympa.inria.fr Delivered-To: caml-list@sympa.inria.fr Received: from mail3-relais-sop.national.inria.fr (mail3-relais-sop.national.inria.fr [192.134.164.104]) by sympa.inria.fr (Postfix) with ESMTPS id 4E0977FA5E for ; Fri, 21 Apr 2017 20:28:19 +0200 (CEST) Authentication-Results: mail3-smtp-sop.national.inria.fr; spf=None smtp.pra=jdimino@janestreet.com; spf=Pass smtp.mailfrom=jdimino@janestreet.com; spf=None smtp.helo=postmaster@mxout1.mail.janestreet.com Received-SPF: None (mail3-smtp-sop.national.inria.fr: no sender authenticity information available from domain of jdimino@janestreet.com) identity=pra; client-ip=38.105.200.78; receiver=mail3-smtp-sop.national.inria.fr; envelope-from="jdimino@janestreet.com"; x-sender="jdimino@janestreet.com"; x-conformance=sidf_compatible Received-SPF: Pass (mail3-smtp-sop.national.inria.fr: domain of jdimino@janestreet.com designates 38.105.200.78 as permitted sender) identity=mailfrom; client-ip=38.105.200.78; receiver=mail3-smtp-sop.national.inria.fr; envelope-from="jdimino@janestreet.com"; x-sender="jdimino@janestreet.com"; x-conformance=sidf_compatible; x-record-type="v=spf1" Received-SPF: None (mail3-smtp-sop.national.inria.fr: no sender authenticity information available from domain of postmaster@mxout1.mail.janestreet.com) identity=helo; client-ip=38.105.200.78; receiver=mail3-smtp-sop.national.inria.fr; envelope-from="jdimino@janestreet.com"; x-sender="postmaster@mxout1.mail.janestreet.com"; x-conformance=sidf_compatible IronPort-PHdr: =?us-ascii?q?9a23=3AvLLZQhE5hscCeB6EvlwhAJ1GYnF86YWxBRYc798d?= =?us-ascii?q?s5kLTJ7yoMqwAkXT6L1XgUPTWs2DsrQf2raQ6/iocFdDyK7JiGoFfp1IWk1Nou?= =?us-ascii?q?QttCtkPvS4D1bmJuXhdS0wEZcKflZk+3amLRodQ56mNBXdrXKo8DEdBAj0OxZr?= =?us-ascii?q?KeTpAI7SiNm82/yv95HJbQhFgDWwbaluIBmqsA7cqtQYjYx+J6gr1xDHuGFIe+?= =?us-ascii?q?NYxWNpIVKcgRPx7dqu8ZBg7ipdpesv+9ZPXqvmcas4S6dYDCk9PGAu+MLrrxjD?= =?us-ascii?q?QhCR6XYaT24bjwBHAwnB7BH9Q5fxri73vfdz1SWGIcH7S60/VC+85Kl3VhDnlC?= =?us-ascii?q?YHNyY48G7JjMxwkLlbqw+lqxBm3oLYfJ2ZOP94c6zaYN0aWHFBXt5PWCNdHoOy?= =?us-ascii?q?YYwPD+8bMuZZqYn2ul8CoBS6CAWpAu7k1z1GiWLs3aAi0eshHwHI0gIjEdwTrn?= =?us-ascii?q?rbsM74O70OXe2v1qTE0SnPYvFQ1Dzg6IbIaBchofSUULx0b8XR01cgFwbEjlqO?= =?us-ascii?q?tIfrMTKV1uEMs2id6OprSOWii2w6pAFwpzivx8Esio7Si4IX0F/E8zt2wJ0pJd?= =?us-ascii?q?2iVkF0f8KkHIFMuCGdMot7W8UvSHxmtiY9z70Jo5+7fC4SxZQmwR7fcf2HfJKS?= =?us-ascii?q?7hLtTuadOTh4hHN5eLK/mha96lKsxfH7VsmxyFpLrjBKktnLtnAKzRDc9s+HSv?= =?us-ascii?q?5l8kejwzmP0R7c6vpYIUAui6XUNYIhzqQsmZoUtETPBjf2mF3sjK+XcEUk5vKn?= =?us-ascii?q?6//7Yrn8o5+cM4l5gRz9PKQ2gsGyD+c1PhITU2SH+emwzqPv8VHlTLlQjPA7kb?= =?us-ascii?q?HVvI7GKckfvKK0AA9Y3pw95xqiDzqqytYVkWUBIVlYYhyIlZLpNEvLIP3gDfew?= =?us-ascii?q?nVCskDBzyvDDMbzhBYjNLmTenLv7eLZy8U9cyA4pwdBd/Z1UDK8OIOnvWk/rqt?= =?us-ascii?q?PXEAM5PxaozObgDdVxzoIeWWSRDa+FKK7fv1yF6vgyL+WQZIIZoijxJ+Q56/L0?= =?us-ascii?q?j3I0mkcRfayz0psWbHC4EO5mI0KcYXf0gNcODGYKvg8/TOzsj12PSjxTaGy0X6?= =?us-ascii?q?0i/TE7FJimApndSYCxmrCOwCC7HphOamBcFl+MCWvod5mDW/oUdC2dOMphkjgd?= =?us-ascii?q?WbilSo8hzg2uuRThy7tnK+rU4jcXuYji1Nhz/e3TlAs9+SZ6D8SHgCmxSDRdmm?= =?us-ascii?q?oJQXcYxqFkqkw1nl6H2Kl+xfJCFMdY59tNXxc7O5+axOt/XYPcQAXEK/WEUluj?= =?us-ascii?q?Q9juLTA0T9Q2i4sMb0d4GtOlphLK2SexH6UYmqDND5sxpPGPl0PtLtpwni6VnJ?= =?us-ascii?q?IqiEMrF44SbGA=3D?= X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: =?us-ascii?q?A0CdAQCXTfpYfU7IaSZcHAEBBAEBCgEBF?= =?us-ascii?q?wEBBAEBCgEBhAqBDAeDYJt9gjwqjUmFNYIPLIQegVoChAQHQBcBAQEBAQEBAQE?= =?us-ascii?q?BARIBAQkWCFeCMyKCQAEBAQECASMEGQEBLAsBBAsLCw0NHQICIhIBBQEKEgYTE?= =?us-ascii?q?gIHiWkDDQgDC50tP4sdaIFsOoMIAQEFhB8Dg28BAQEBAQEBAQIBAQEBAQEBAQE?= =?us-ascii?q?BARUIEoZBg2yBCoUEglmCX4diDJVYhxeLb4JViCCGYpJRFB+BFQ8RAYE8Jh0IG?= =?us-ascii?q?BVEGAaEEioPHIFkdIk2AQEB?= X-IPAS-Result: =?us-ascii?q?A0CdAQCXTfpYfU7IaSZcHAEBBAEBCgEBFwEBBAEBCgEBhAq?= =?us-ascii?q?BDAeDYJt9gjwqjUmFNYIPLIQegVoChAQHQBcBAQEBAQEBAQEBARIBAQkWCFeCM?= =?us-ascii?q?yKCQAEBAQECASMEGQEBLAsBBAsLCw0NHQICIhIBBQEKEgYTEgIHiWkDDQgDC50?= =?us-ascii?q?tP4sdaIFsOoMIAQEFhB8Dg28BAQEBAQEBAQIBAQEBAQEBAQEBARUIEoZBg2yBC?= =?us-ascii?q?oUEglmCX4diDJVYhxeLb4JViCCGYpJRFB+BFQ8RAYE8Jh0IGBVEGAaEEioPHIF?= =?us-ascii?q?kdIk2AQEB?= X-IronPort-AV: E=Sophos;i="5.37,230,1488841200"; d="scan'208,217";a="221127327" Received: from mxout1.mail.janestreet.com ([38.105.200.78]) by mail3-smtp-sop.national.inria.fr with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 21 Apr 2017 20:28:17 +0200 Received: from [172.27.56.106] (helo=tot-qpr-mailcore2) by mxout1.mail.janestreet.com with esmtps (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.89) (envelope-from ) id 1d1dI0-00012W-0L for caml-list@inria.fr; Fri, 21 Apr 2017 14:28:16 -0400 X-JS-Flow: external X-JS-Scanner-attachment: (ok) No attachments Received: by tot-qpr-mailcore2 with JS-mailcore (0.1) (envelope-from ) id BY-k8_-A2qNUG-fO; 2017-04-21 14:28:16.002596-04:00 Received: from mail-lf0-f70.google.com ([209.85.215.70]) by mxgoog1.mail.janestreet.com with esmtps (TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128) (Exim 4.89) (envelope-from ) id 1d1dHz-0005x5-RM for caml-list@inria.fr; Fri, 21 Apr 2017 14:28:15 -0400 Received: by mail-lf0-f70.google.com with SMTP id h16so14753560lfi.18 for ; Fri, 21 Apr 2017 11:28:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=janestreet.com; s=google; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=ZwBjuF4/GeLnpGugoIhC1cK5+GUp9xnwGtswPP39F0c=; b=vYWFdpxGIznrS7wekME8sOlZ4i/bknmU6LCfNJ/NhfrCewsHBZ9sxMJ01kRWR1+ko2 kNPBhNjosdenjq1zAU6L2AqiXyAXZs27i8PqLGmdfQsXbT0N4TyboafxVsdxXR0vjIHu xkTZeoWI8eWUB8cMdPRmjmguihvV0M/1J+2vg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=ZwBjuF4/GeLnpGugoIhC1cK5+GUp9xnwGtswPP39F0c=; b=ZlMM0eBnZPG8w9s1akbzBeBnuyrHILVsvnch/4WDc9Z3UpPNr2KVfs218GULTtiyY0 v7m06Dnndusg12sXVOuFC7Np3LViCj96LGzrboVPXQUGLrVWUXk1Lt6mWEFqbNWOxngt BVB+C0xMmuLzgdYP6I2MPaU7xtHCJcwFHbm/pvT0XMtwu5cnmEBRNi8MHRsGDQipNUFC rLpOqwJ62Qbw6pkpmEMW+e8hI/ulNLjgcLqtktJv2sxoZzHDj3xPaeoIqG9cvqJ0dZlj h0NMKlk254y2iH6gmk5HFDWf5CxQ8VH9HnjwnDTpVUnJ7xG9z+UZoVUZpR9FMyyndcyk OGmg== X-Gm-Message-State: AN3rC/7e8SIJC4YsuLSiBDQHuoUutaWV30/r688WXA7Z8R2/zS91cmwP ldUTwdsn7zR/fsF3eIl89+YrZGmu7febviAYgbnui+zeviVSgkQE7okcezpQXVTRK5kNu2grvS4 RMOONbst2BI2Zd/u3 X-Received: by 10.46.20.15 with SMTP id u15mr5404476ljd.81.1492799294544; Fri, 21 Apr 2017 11:28:14 -0700 (PDT) X-Received: by 10.46.20.15 with SMTP id u15mr5404467ljd.81.1492799294266; Fri, 21 Apr 2017 11:28:14 -0700 (PDT) MIME-Version: 1.0 Received: by 10.25.199.3 with HTTP; Fri, 21 Apr 2017 11:28:13 -0700 (PDT) In-Reply-To: <703d59cf-a1f9-c12f-9325-44aa23b03e36@lexifi.com> References: <58FA282A021A076200390603_0_57023@msllnjpmsgsv06> <703d59cf-a1f9-c12f-9325-44aa23b03e36@lexifi.com> From:Jeremie Dimino Date: Fri, 21 Apr 2017 19:28:13 +0100 Message-ID: To:Alain Frisch Cc:Yotam Barnoy , Hongbo Zhang , Ocaml Mailing List Content-Type: multipart/alternative; boundary=f403045fc074a800b9054db16ba7 X-JS-Exim-Data-Received: 2017-04-21 14:28:15-0400 X-JS-Processed-by: mailcore Subject: Re: [Caml-list] PPX is harmful to our community in the long term --f403045fc074a800b9054db16ba7 Content-Type: text/plain; charset=UTF-8 Yes, we are now using ocaml-migrate-parsetree at Jane Street. I think that with this library the versioning story is much better. We are heavy user of ppx rewriters and all of the code we release now builds with OCaml 4.03 to 4.06 without problems. Initially we considered using the concrete syntax to solve the versioning problem and we have a solution that should work in theory. Although it's not great when there is no source file, such as in the toplevel. The method we considered is described in [1]. Switching to ocaml-migrate-parsetree was easier since we didn't have to change anything to the way things worked, just port the code. Regarding the idea of cutting down the dependency for release tarballs by including the generated code, while this is an interesting idea, there are a lot of small annoying problems that makes this technique hard in practice. The two main problems are: 1. it doesn't work for .ml files that are generated. Basically all the code you generate with custom code generators as to be ppx free. It is fine however for pure generators, since you can just embed the generated code 2. it doesn't work if you use conditional compilation. Conditional compilation is not great in general, but it is a lot of work to completely get rid of it However, one thing that we kept from these experiments is the idea to use ppx in the same way that expectation tests work. This idea is described in [1] and we use it for the Base library [2]. Additionally it is a good testing/debugging tool. It is a viable solution if all you need is [@@deriving] plugins and don't want to depend on ppx. The idea is to let the ppx rewriter insert the code generated by [@@deriving] plugins in the source code: type t = A | B [@@deriving_inline sexp_of] let sexp_of_t = function | A -> Sexp.Atom "A" | B -> Sexp.Atom "B" [@@@end_of_derived_code] The code builds without ppx and you still don't have to write the boilerplate yourself, you let the ppx tool do it for you. [1] https://blogs.janestreet.com/an-solution-to-the-ppx-versioning-problem/ [2] https://github.com/janestreet/base On Fri, Apr 21, 2017 at 6:11 PM, Alain Frisch wrote: > On 21/04/2017 18:04, Yotam Barnoy wrote: > >> My 2 cents: I personally think we did PPX wrong. PPX should have used >> the syntax as its starting point for full backwards compatibility. >> Currently, the PPX cycle is >> >> OCaml: { syntax -> AST -> serialized AST } -> PPX: {serialized AST -> >> AST -> PPX modification -> serialized AST} -> OCaml: {serialized AST >> -> AST -> ... } >> >> Exposing the AST is the cause of the problem you mention. If instead, >> every PPX extension had a particular OCaml compiler's syntax parser >> and syntax printer integrated into it, and the PPX cycle was: >> >> PPX: { syntax -> AST -> PPX modification -> syntax } -> compiler: { >> syntax -> AST -> ...} >> >> We would have far fewer issues with PPX plugins, since they would be >> as backwards-compatible as the syntax. >> > > I think > > https://github.com/let-def/ocaml-migrate-parsetree > > is a much stronger approach. If I understand correctly, Jane Street moved > from using a migration system based on concrete syntax to using this new > project. > > > In your suggested approach, imagine you have a PPX processor written for > OCaml 4.04. It assumes that the Parsetree it works on is the one of 4.04. > Sure, you can compile this PPX processor (+ embedded parser/printer) into a > stand-alone executable, using OCaml 4.04, and then apply it as a > preprocessor called from OCaml 4.05. But this is very impractical: users > would need to install OCaml 4.04 just to produce the PPX executable. > Moreover, you loose the ability to combine multiple rewritings in a single > process, and you have to pay the price of multiple > parsing/printing/processes. On top of that: > > (i) you put in the critical loop "pprintast.ml", which has always been > more or less buggy; > > (ii) there is no hope that an "old PPX" applies to source code using > newest syntactic features; > > (iii) as you mention, locations would need to be added for each node in > the AST, which makes the parsing/printing even slower; moreover, it is not > technically straightforward to do so, since many places in the AST contains > locations but do not support attaching attributes; > > (iv) we loose the ability to change the concrete syntax of OCaml, either > to use alternative syntaxes such as Reason, or to remove, at some point, > weird corner cases in the current syntax. > > > With ocaml-migrate-parsetree, you can still compile your PPX written for > 4.04 with newer versions of OCaml. And it would even be possible (I don't > know if this is already the case) to apply it to source code using newer > syntactic features, as long as the rewriting done by the PPX doesn't > interact with those features. (This could be achieved by turning new > syntactic features into attributes/extension nodes when mapping to the > older AST; and then recognizing these forms and reconstitute the correct > new AST forms when mapping back to the new version.) > > > -- Alain > > > -- > Caml-list mailing list. Subscription management and archives: > https://sympa.inria.fr/sympa/arc/caml-list > Beginner's list: http://groups.yahoo.com/group/ocaml_beginners > Bug reports: http://caml.inria.fr/bin/caml-bugs > -- Jeremie --f403045fc074a800b9054db16ba7 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Yes, we are now using ocaml-migrate-parsetree at Jane S= treet. I think that with this library the versioning story is much better. = We are heavy user of ppx rewriters and all of the code we release now build= s with OCaml 4.03 to 4.06 without problems.

Initially = we considered using the concrete syntax to solve the versioning problem and= we have a solution that should work in theory. Although it's not great= when there is no source file, such as in the toplevel. The method we consi= dered is described in [1]. Switching to ocaml-migrate-parsetree was easier = since we didn't have to change anything to the way things worked, just = port the code.=C2=A0

Regarding the idea of cutting d= own the dependency for release tarballs by including the generated code, wh= ile this is an interesting idea, there are a lot of small annoying problems= that makes this technique hard in practice. The two main problems are:

1. it doesn't work for .ml files that are generated. = Basically all the code you generate with custom code generators as to be pp= x free. It is fine however for pure generators, since you can just embed th= e generated code

2. it doesn't work if you use con= ditional compilation. Conditional compilation is not great in general, but = it is a lot of work to completely get rid of it

Howeve= r, one thing that we kept from these experiments is the idea to use ppx in = the same way that expectation tests work. This idea is described in [1] and= we use it for the Base library [2]. Additionally it is a good testing/debu= gging tool. It is a viable solution if all you need is [@@deriving] plugins= and don't want to depend on ppx.

The idea is to l= et the ppx rewriter insert the code generated by [@@deriving] plugins in th= e source code:

type t =3D A | = B [@@deriving_inline sexp_of]

let sexp_of_t =3D fu= nction
=C2=A0 | A -> Sexp.Atom "A"
=C2=A0 = | B -> Sexp.Atom "B"
=
[@@@end_of_deriv= ed_code]

The code builds without ppx and= you still don't have to write the boilerplate yourself, you let the pp= x tool do it for you.

[1]=C2=A0https://blogs.janestreet.com/an-solution-to-the-ppx-versioni= ng-problem/
[2] https://github.com/janestreet/base

<= /div>

On Fri= , Apr 21, 2017 at 6:11 PM, Alain Frisch <alain.frisch@lexifi.com= > wrote:
On 21= /04/2017 18:04, Yotam Barnoy wrote:
My 2 cents: I personally think we did PPX wrong. PPX should have used
the syntax as its starting point for full backwards compatibility.
Currently, the PPX cycle is

OCaml: { syntax -> AST -> serialized AST } -> PPX: {serialized AST= ->
AST -> PPX modification -> serialized AST} -> OCaml: {serialized A= ST
-> AST -> ... }

Exposing the AST is the cause of the problem you mention. If instead,
every PPX extension had a particular OCaml compiler's syntax parser
and syntax printer integrated into it, and the PPX cycle was:

PPX: { syntax -> AST -> PPX modification -> syntax } -> compile= r: {
syntax -> AST -> ...}

We would have far fewer issues with PPX plugins, since they would be
as backwards-compatible as the syntax.

I think

 =C2=A0 https://github.com/let-def/ocaml-mig= rate-parsetree

is a much stronger approach.=C2=A0 If I understand correctly, Jane Street m= oved from using a migration system based on concrete syntax to using this n= ew project.


In your suggested approach, imagine you have a PPX processor written for OC= aml 4.04.=C2=A0 It assumes that the Parsetree it works on is the one of 4.0= 4.=C2=A0 Sure, you can compile this PPX processor (+ embedded parser/printe= r) into a stand-alone executable, using OCaml 4.04, and then apply it as a = preprocessor called from OCaml 4.05.=C2=A0 But this is very impractical: us= ers would need to install OCaml 4.04 just to produce the PPX executable.=C2= =A0 Moreover, you loose the ability to combine multiple rewritings in a sin= gle process, and you have to pay the price of multiple parsing/printing/pro= cesses.=C2=A0 On top of that:

=C2=A0 (i) you put in the critical loop "pprintast.ml", which has a= lways been more or less buggy;

=C2=A0 (ii) there is no hope that an "old PPX" applies to source = code using newest syntactic features;

=C2=A0 (iii) as you mention, locations would need to be added for each node= in the AST, which makes the parsing/printing even slower;=C2=A0 moreover, = it is not technically straightforward to do so, since many places in the AS= T contains locations but do not support attaching attributes;

=C2=A0 (iv) we loose the ability to change the concrete syntax of OCaml, ei= ther to use alternative syntaxes such as Reason, or to remove, at some poin= t, weird corner cases in the current syntax.


With ocaml-migrate-parsetree, you can still compile your PPX written for 4.= 04 with newer versions of OCaml.=C2=A0 And it would even be possible (I don= 't know if this is already the case) to apply it to source code using n= ewer syntactic features, as long as the rewriting done by the PPX doesn'= ;t interact with those features.=C2=A0 (This could be achieved by turning n= ew syntactic features into attributes/extension nodes when mapping to the o= lder AST; and then recognizing these forms and reconstitute the correct new= AST forms when mapping back to the new version.)


-- Alain


--
Caml-list mailing list.=C2=A0 Subscription management and archives:
https://sympa.inria.fr/sympa/arc/caml-list
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs



--
=
Jeremie
--f403045fc074a800b9054db16ba7--