Re: [Caml-list] PPX is harmful to our community in the long term

Re: [Caml-list] PPX is harmful to our community in the long term

[Hongbo Zhang (BLOOMBERG/ 731 LEX)]

[-- Attachment #1: Type: text/plain, Size: 3934 bytes --]

Exactly, for companies like janestreet, they may not feel its pain since they didn't depend on external ppx, release software only works on latest compilers, and have a very good orientation section about those conventions, it makes perfect sense under such internal organizations.

But for external open source communities, it's too bad that PPX is so widely used, migrate-parsetree maybe useful, but it adds yet another layer of complexity, the real solution is checking in generated code which will cut the dependency completely or just don't use ppx.

Anyway , I don't want to make anyone unhappy, but I believe acknowledging what the awesome OCaml language offers and keeping build high quality software instead of writing ppx is the right way to benefit our community . People built amazing software in less expressive languages like GoLang, I am sure we can do much better!

----- Original Message -----
From: Robert Muller <robert.muller2@gmail.com>
To: HONGBO ZHANG
CC: caml-list@inria.fr, ssp.mryau@gmail.com
At: 22-Apr-2017 15:47:47


The VAX Fortran compiler was a state-of-the-art compiler in its day. The compiler was implemented in BLISS, a systems programming language with a powerful macro system. Each compiler developer invariably authored their own macros to suit their styles. So when one wandered into foreign bits of the compiler they appeared to be written in different dialects of BLISS. We had the dialects of the present developers as well as legacy dialects from years of earlier developers. It was not for nothing that our development machine was called "Babel". 
New hires faced a steeper slope in mastering the compiler than they would have had there been no macros. I concluded that macros, at least managed in that style, were injurious to the engineering process.
- Bob Muller


On Sat, Apr 22, 2017 at 10:47 AM, Hongbo Zhang (BLOOMBERG/ 731 LEX) <hzhang295@bloomberg.net> wrote:

Yes, I agree it's useful , and that's why I wrote hundreds of thousands of lines of code in syntactic meta-programming (camlp4, fan, ppx)
But in the end of day, the conclusion is the cost is just so huge that it should not be widely used, at least , it should not be *leaked* to end users. ( I remember I had a conversation with the original maintainer of camlp4, Nicolas, about 5 years ago, he had similar ideas with me)


----- Original Message -----
From: Serge Sivkov <ssp.mryau@gmail.com>
To: caml-list@inria.fr
At: 22-Apr-2017 08:49:40


Hence, my two cents: PPX has problems in cross-compilation use cases, but I suppose something like new tag in META can reslove this issue.
As for me, just ppx_deriving* by whitequark is yet one example of usefullness of PPX.

WBR, ssp

2017-04-22 5:10 GMT+06:00 Emilio Jesús Gallego Arias <e@x80.org>:

"Hongbo Zhang (BLOOMBERG/ 731 LEX)" <hzhang295@bloomberg.net> writes:

> Yes, that's exactly what I suggested in the beginning!

Maybe I interpret the word "harmful" differently, but IMVHO I have to
strongly disagree with your choice of subject in the original mail.

Not only PPX has not been harmful for me, but it has been a life-saver
tool that has enabled significant progress towards more productive
research.

"Hongbo Zhang (BLOOMBERG/ 731 LEX)" <hzhang295@bloomberg.net> writes:

> calling it 'madness' is disrespectful

Personally, I fully subscribe Yaron's message and I see nothing
disrespectful in suggesting that abandoning syntactic abstractions is a
very bad idea.

You wrote:

 "the OCaml library developer should avoid PPX as much as you can",

but if you meant:

 "PPX seems quite unstable these days, I wonder how could we improve
  long-term stability?"

I'd have to admit that message didn't reach to me.

Best regards!
Emilio

--
Caml-list mailing list.  Subscription management and archives:
https://sympa.inria.fr/sympa/arc/caml-list
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs



[-- Attachment #2: Type: text/html, Size: 9473 bytes --]

Re: [Caml-list] PPX is harmful to our community in the long term

[Yaron Minsky]

[-- Attachment #1: Type: text/plain, Size: 5245 bytes --]

On Sat, Apr 22, 2017 at 8:06 PM, Hongbo Zhang (BLOOMBERG/ 731 LEX) <
hzhang295@bloomberg.net> wrote:

> Exactly, for companies like janestreet, they may not feel its pain since
> they didn't depend on external ppx, release software only works on latest
> compilers, and have a very good orientation section about those
> conventions, it makes perfect sense under such internal organizations.
>

Jane Street has no special immunity. We use external PPXs, and we in fact
had a bunch of pain around versioning of PPXs, which is precisely why we've
spent real effort on making versioning easier.

Syntactic abstraction is amazingly useful for many different kinds of
programming, open source and industrial alike. Again, I the emphasis should
be on identifying and fixing the problems with PPX, not pushing to avoid it.


> But for external open source communities, it's too bad that PPX is so
> widely used, migrate-parsetree maybe useful, but it adds yet another layer
> of complexity, the real solution is checking in generated code which will
> cut the dependency completely or just don't use ppx.
>

Did you read Jeremie's post? That solution was considered, but it turns out
to have a lot of issues.


> Anyway , I don't want to make anyone unhappy, but I believe acknowledging
> what the awesome OCaml language offers and keeping build high quality
> software instead of writing ppx is the right way to benefit our community .
> People built amazing software in less expressive languages like GoLang, I
> am sure we can do much better!
>

But as OCaml users know, more expressive languages really do help. I'm not
in favor of adding expressiveness at all costs, but the upside of PPX is
massive, and the problems are eminently manageable, as the emergence of
tools like migrate-parsetree has demonstrated.

y


> ----- Original Message -----
> From: Robert Muller <robert.muller2@gmail.com>
> To: HONGBO ZHANG
> CC: caml-list@inria.fr, ssp.mryau@gmail.com
> At: 22-Apr-2017 15:47:47
>
> The VAX Fortran compiler was a state-of-the-art compiler in its day. The
> compiler was implemented in BLISS, a systems programming language with a
> powerful macro system. Each compiler developer invariably authored their
> own macros to suit their styles. So when one wandered into foreign bits of
> the compiler they appeared to be written in different dialects of BLISS. We
> had the dialects of the present developers as well as legacy dialects from
> years of earlier developers. It was not for nothing that our development
> machine was called "Babel".
> New hires faced a steeper slope in mastering the compiler than they would
> have had there been no macros. I concluded that macros, at least managed in
> that style, were injurious to the engineering process.
> - Bob Muller
>
>
> On Sat, Apr 22, 2017 at 10:47 AM, Hongbo Zhang (BLOOMBERG/ 731 LEX) <
> hzhang295@bloomberg.net> wrote:
>
>> Yes, I agree it's useful , and that's why I wrote hundreds of thousands
>> of lines of code in syntactic meta-programming (camlp4, fan, ppx)
>> But in the end of day, the conclusion is the cost is just so huge that it
>> should not be widely used, at least , it should not be *leaked* to end
>> users. ( I remember I had a conversation with the original maintainer of
>> camlp4, Nicolas, about 5 years ago, he had similar ideas with me)
>>
>>
>> ----- Original Message -----
>> From: Serge Sivkov <ssp.mryau@gmail.com>
>> To: caml-list@inria.fr
>> At: 22-Apr-2017 08:49:40
>>
>> Hence, my two cents: PPX has problems in cross-compilation use cases, but
>> I suppose something like new tag in META can reslove this issue.
>> As for me, just ppx_deriving* by whitequark is yet one example of
>> usefullness of PPX.
>>
>> WBR, ssp
>>
>> 2017-04-22 5:10 GMT+06:00 Emilio Jesús Gallego Arias <e@x80.org>:
>>
>>> "Hongbo Zhang (BLOOMBERG/ 731 LEX)" <hzhang295@bloomberg.net> writes:
>>>
>>> > Yes, that's exactly what I suggested in the beginning!
>>>
>>> Maybe I interpret the word "harmful" differently, but IMVHO I have to
>>> strongly disagree with your choice of subject in the original mail.
>>>
>>> Not only PPX has not been harmful for me, but it has been a life-saver
>>> tool that has enabled significant progress towards more productive
>>> research.
>>>
>>> "Hongbo Zhang (BLOOMBERG/ 731 LEX)" <hzhang295@bloomberg.net> writes:
>>>
>>> > calling it 'madness' is disrespectful
>>>
>>> Personally, I fully subscribe Yaron's message and I see nothing
>>> disrespectful in suggesting that abandoning syntactic abstractions is a
>>> very bad idea.
>>>
>>> You wrote:
>>>
>>>  "the OCaml library developer should avoid PPX as much as you can",
>>>
>>> but if you meant:
>>>
>>>  "PPX seems quite unstable these days, I wonder how could we improve
>>>   long-term stability?"
>>>
>>> I'd have to admit that message didn't reach to me.
>>>
>>> Best regards!
>>> Emilio
>>>
>>> --
>>> Caml-list mailing list.  Subscription management and archives:
>>> https://sympa.inria.fr/sympa/arc/caml-list
>>> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>>> Bug reports: http://caml.inria.fr/bin/caml-bugs
>>>
>>
>>
>

[-- Attachment #2: Type: text/html, Size: 10529 bytes --]

Re: [Caml-list] PPX is harmful to our community in the long term

[Hongbo Zhang (BLOOMBERG/ 731 LEX)]

[-- Attachment #1: Type: text/plain, Size: 2018 bytes --]

Yes, I agree it's useful , and that's why I wrote hundreds of thousands of lines of code in syntactic meta-programming (camlp4, fan, ppx)
But in the end of day, the conclusion is the cost is just so huge that it should not be widely used, at least , it should not be *leaked* to end users. ( I remember I had a conversation with the original maintainer of camlp4, Nicolas, about 5 years ago, he had similar ideas with me)

----- Original Message -----
From: Serge Sivkov <ssp.mryau@gmail.com>
To: caml-list@inria.fr
At: 22-Apr-2017 08:49:40


Hence, my two cents: PPX has problems in cross-compilation use cases, but I suppose something like new tag in META can reslove this issue.
As for me, just ppx_deriving* by whitequark is yet one example of usefullness of PPX.

WBR, ssp

2017-04-22 5:10 GMT+06:00 Emilio Jesús Gallego Arias <e@x80.org>:

"Hongbo Zhang (BLOOMBERG/ 731 LEX)" <hzhang295@bloomberg.net> writes:

> Yes, that's exactly what I suggested in the beginning!

Maybe I interpret the word "harmful" differently, but IMVHO I have to
strongly disagree with your choice of subject in the original mail.

Not only PPX has not been harmful for me, but it has been a life-saver
tool that has enabled significant progress towards more productive
research.

"Hongbo Zhang (BLOOMBERG/ 731 LEX)" <hzhang295@bloomberg.net> writes:

> calling it 'madness' is disrespectful

Personally, I fully subscribe Yaron's message and I see nothing
disrespectful in suggesting that abandoning syntactic abstractions is a
very bad idea.

You wrote:

 "the OCaml library developer should avoid PPX as much as you can",

but if you meant:

 "PPX seems quite unstable these days, I wonder how could we improve
  long-term stability?"

I'd have to admit that message didn't reach to me.

Best regards!
Emilio

--
Caml-list mailing list.  Subscription management and archives:
https://sympa.inria.fr/sympa/arc/caml-list
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs



[-- Attachment #2: Type: text/html, Size: 4965 bytes --]

Re: [Caml-list] PPX is harmful to our community in the long term

[Robert Muller]

[-- Attachment #1: Type: text/plain, Size: 3111 bytes --]

The VAX Fortran compiler was a state-of-the-art compiler in its day. The
compiler was implemented in BLISS, a systems programming language with a
powerful macro system. Each compiler developer invariably authored their
own macros to suit their styles. So when one wandered into foreign bits of
the compiler they appeared to be written in different dialects of BLISS. We
had the dialects of the present developers as well as legacy dialects from
years of earlier developers. It was not for nothing that our development
machine was called "Babel".
New hires faced a steeper slope in mastering the compiler than they would
have had there been no macros. I concluded that macros, at least managed in
that style, were injurious to the engineering process.
- Bob Muller


On Sat, Apr 22, 2017 at 10:47 AM, Hongbo Zhang (BLOOMBERG/ 731 LEX) <
hzhang295@bloomberg.net> wrote:

> Yes, I agree it's useful , and that's why I wrote hundreds of thousands of
> lines of code in syntactic meta-programming (camlp4, fan, ppx)
> But in the end of day, the conclusion is the cost is just so huge that it
> should not be widely used, at least , it should not be *leaked* to end
> users. ( I remember I had a conversation with the original maintainer of
> camlp4, Nicolas, about 5 years ago, he had similar ideas with me)
>
>
> ----- Original Message -----
> From: Serge Sivkov <ssp.mryau@gmail.com>
> To: caml-list@inria.fr
> At: 22-Apr-2017 08:49:40
>
> Hence, my two cents: PPX has problems in cross-compilation use cases, but
> I suppose something like new tag in META can reslove this issue.
> As for me, just ppx_deriving* by whitequark is yet one example of
> usefullness of PPX.
>
> WBR, ssp
>
> 2017-04-22 5:10 GMT+06:00 Emilio Jesús Gallego Arias <e@x80.org>:
>
>> "Hongbo Zhang (BLOOMBERG/ 731 LEX)" <hzhang295@bloomberg.net> writes:
>>
>> > Yes, that's exactly what I suggested in the beginning!
>>
>> Maybe I interpret the word "harmful" differently, but IMVHO I have to
>> strongly disagree with your choice of subject in the original mail.
>>
>> Not only PPX has not been harmful for me, but it has been a life-saver
>> tool that has enabled significant progress towards more productive
>> research.
>>
>> "Hongbo Zhang (BLOOMBERG/ 731 LEX)" <hzhang295@bloomberg.net> writes:
>>
>> > calling it 'madness' is disrespectful
>>
>> Personally, I fully subscribe Yaron's message and I see nothing
>> disrespectful in suggesting that abandoning syntactic abstractions is a
>> very bad idea.
>>
>> You wrote:
>>
>>  "the OCaml library developer should avoid PPX as much as you can",
>>
>> but if you meant:
>>
>>  "PPX seems quite unstable these days, I wonder how could we improve
>>   long-term stability?"
>>
>> I'd have to admit that message didn't reach to me.
>>
>> Best regards!
>> Emilio
>>
>> --
>> Caml-list mailing list.  Subscription management and archives:
>> https://sympa.inria.fr/sympa/arc/caml-list
>> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>> Bug reports: http://caml.inria.fr/bin/caml-bugs
>>
>
>

[-- Attachment #2: Type: text/html, Size: 5727 bytes --]

Re: [Caml-list] PPX is harmful to our community in the long term

[Yaron Minsky]

[-- Attachment #1: Type: text/plain, Size: 3791 bytes --]

I very much agree with this sentiment. Restraint in the use of PPXs is
important. We've added a fairly modest set of PPXs that mostly act as
language extensions, turned on nearly everywhere in our codebase.

We consider the addition of a PPX to be expensive, and especially a
project-specific one. We only do it if it makes for a really large
improvement. I totally agree that you don't want different subsets of your
codebase to feel as if they were written in different languages.

y

On Sat, Apr 22, 2017 at 3:47 PM, Robert Muller <robert.muller2@gmail.com>
wrote:

> The VAX Fortran compiler was a state-of-the-art compiler in its day. The
> compiler was implemented in BLISS, a systems programming language with a
> powerful macro system. Each compiler developer invariably authored their
> own macros to suit their styles. So when one wandered into foreign bits of
> the compiler they appeared to be written in different dialects of BLISS. We
> had the dialects of the present developers as well as legacy dialects from
> years of earlier developers. It was not for nothing that our development
> machine was called "Babel".
> New hires faced a steeper slope in mastering the compiler than they would
> have had there been no macros. I concluded that macros, at least managed in
> that style, were injurious to the engineering process.
> - Bob Muller
>
>
> On Sat, Apr 22, 2017 at 10:47 AM, Hongbo Zhang (BLOOMBERG/ 731 LEX) <
> hzhang295@bloomberg.net> wrote:
>
>> Yes, I agree it's useful , and that's why I wrote hundreds of thousands
>> of lines of code in syntactic meta-programming (camlp4, fan, ppx)
>> But in the end of day, the conclusion is the cost is just so huge that it
>> should not be widely used, at least , it should not be *leaked* to end
>> users. ( I remember I had a conversation with the original maintainer of
>> camlp4, Nicolas, about 5 years ago, he had similar ideas with me)
>>
>>
>> ----- Original Message -----
>> From: Serge Sivkov <ssp.mryau@gmail.com>
>> To: caml-list@inria.fr
>> At: 22-Apr-2017 08:49:40
>>
>> Hence, my two cents: PPX has problems in cross-compilation use cases, but
>> I suppose something like new tag in META can reslove this issue.
>> As for me, just ppx_deriving* by whitequark is yet one example of
>> usefullness of PPX.
>>
>> WBR, ssp
>>
>> 2017-04-22 5:10 GMT+06:00 Emilio Jesús Gallego Arias <e@x80.org>:
>>
>>> "Hongbo Zhang (BLOOMBERG/ 731 LEX)" <hzhang295@bloomberg.net> writes:
>>>
>>> > Yes, that's exactly what I suggested in the beginning!
>>>
>>> Maybe I interpret the word "harmful" differently, but IMVHO I have to
>>> strongly disagree with your choice of subject in the original mail.
>>>
>>> Not only PPX has not been harmful for me, but it has been a life-saver
>>> tool that has enabled significant progress towards more productive
>>> research.
>>>
>>> "Hongbo Zhang (BLOOMBERG/ 731 LEX)" <hzhang295@bloomberg.net> writes:
>>>
>>> > calling it 'madness' is disrespectful
>>>
>>> Personally, I fully subscribe Yaron's message and I see nothing
>>> disrespectful in suggesting that abandoning syntactic abstractions is a
>>> very bad idea.
>>>
>>> You wrote:
>>>
>>>  "the OCaml library developer should avoid PPX as much as you can",
>>>
>>> but if you meant:
>>>
>>>  "PPX seems quite unstable these days, I wonder how could we improve
>>>   long-term stability?"
>>>
>>> I'd have to admit that message didn't reach to me.
>>>
>>> Best regards!
>>> Emilio
>>>
>>> --
>>> Caml-list mailing list.  Subscription management and archives:
>>> https://sympa.inria.fr/sympa/arc/caml-list
>>> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>>> Bug reports: http://caml.inria.fr/bin/caml-bugs
>>>
>>
>>
>

[-- Attachment #2: Type: text/html, Size: 6921 bytes --]

Re: [Caml-list] PPX is harmful to our community in the long term

[Hongbo Zhang (BLOOMBERG/ 731 LEX)]

[-- Attachment #1: Type: text/plain, Size: 4886 bytes --]

Yes, that's exactly what I suggested in the beginning! 
calling it 'madness' is disrespectful 

----- Original Message -----
From: Fabrice Le Fessant <Fabrice.Le_fessant@inria.fr>
To: HONGBO ZHANG, yminsky@janestreet.com
CC: caml-list@inria.fr
At: 21-Apr-2017 17:18:32


A lot of people use `autoconf` to generate `./configure` scripts, and the standard practice is to keep the `./configure` script so that people don't need to run `autoconf` to just compile and install the software. Maybe projects could do the same with ppx, i.e. store pre-processed files in the project sources so that the ppx would only be needed when the developer modifies the sources. For example, there could be a `_ppx` directory, where pre-processed files would be stored under the hash of a combination of their original source code and the  `-ppx` arguments. The compiler (or the build system) would use these files when available instead of calling the ppx. That might reduce the problem at least for `opam`, since users are not supposed to edit the packages when installing them.
On Fri, Apr 21, 2017 at 9:24 PM Hongbo Zhang (BLOOMBERG/ 731 LEX) <hzhang295@bloomberg.net> wrote:

Yes, when you never depend on other people's PPX, it is perfectly fine to provide a customized
suite of PPX. 
Think about the software which only works against 4.03, 4.04, this is equivalent to say that you release
a c++ library only works with gcc 7 while most enterprises are still using 4.8, nobody even think it is a 
serious piece of software.
It is a different story in Haskell deriving or Scheme macro system, there is no issue that you software 
in use today will be not compilable in the future.

From: yminsky@janestreet.com At: 04/21/17 15:12:29
To: Hongbo Zhang (BLOOMBERG/ 731 LEX)
Cc: caml-list@inria.fr
Subject: Re: [Caml-list] PPX is harmful to our community in the long term

I understand the frustration, but I think your conclusion is
misplaced. PPXs are massively helpful when building serious software.
Having automatic generation of pretty-printers, comparison functions,
hash functions, binary protocols, and the like is a huge win for both
programmer efficiency and correctness. The Haskell folk aren't wrong
to care about deriving, and the schemers aren't crazy to want their
macro systems.

In short, I think abandoning syntactic abstractions is madness.

I agree that the portability problems are serious and should be
addressed, but ocaml-migrate-parsetree seems like a solid step in the
right direction.

y

On Fri, Apr 21, 2017 at 11:41 AM, Hongbo Zhang (BLOOMBERG/ 731 LEX)
<hzhang295@bloomberg.net> wrote:
> Dear OCaml developers:
> Given that bitten by PPX from time to time, finally, I think it is a time to
> spend two hours sharing my experience with PPX and why you(the OCaml library
> developer) should avoid PPX as much as you can.
>
> Here is a story I just experienced this morning, I tried to install a
> package from opam, and it complained my compiler is too old - 4.02.3, to be
> honest, 4.02.3 is still a pretty modern OCaml compiler, even debian stable
> still stays on 4.01. Anyway, I switched to 4.04.1, after half an hour, it
> failed to compile again, complaning about some ppx error message. This is
> not my first time experience, and finally it made me to write an essay about
> why PPX is harmful.
> PPX is a compiler plugin, it imposes a very large compiler surface API to
> your library, and we don't have any backward compatibility guarantee from
> the compiler, which means your library will only work against a specific
> compiler. Even worse, OCaml is an elegant but small community, we don't have
> too many maintainers for a library, if you have a library which relies on
> PPX (the dependency chain could be really really huge, for example,
> ppx_metaquot depends on typing environment, you can find lots of stories
> about node_modules in Node community), it will probably not work against
> next version of OCaml compiler, and it will be a huge maintenance overhead
> for other people to pick it up.
>
> OCaml is already a very expressive language, probably more expressive than
> any other mainstream language, (Go, Java, C/C++, etc), it is fine to write
> some boilerplate code, or we can cut PPX as a dev dependency, after your
> PPXed your code, check in the generated source code(via -dsource), so it
> will not bring dependency to end users.
> There are some valid use cases of PPX, for example, in BuckleScript or
> JS_of_OCaml, we want to customize OCaml language a bit for external FFI, or
> if you have a very large team, and committed effort to maintain your PPX.
> Happy hacking in OCaml without PPX, Thanks -- Hongbo
>
>


--
Caml-list mailing list.  Subscription management and archives:
https://sympa.inria.fr/sympa/arc/caml-list
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs



[-- Attachment #2: Type: text/html, Size: 9463 bytes --]

Re: [Caml-list] PPX is harmful to our community in the long term

[Emilio Jesús Gallego Arias]

"Hongbo Zhang (BLOOMBERG/ 731 LEX)" <hzhang295@bloomberg.net> writes:

> Yes, that's exactly what I suggested in the beginning!

Maybe I interpret the word "harmful" differently, but IMVHO I have to
strongly disagree with your choice of subject in the original mail.

Not only PPX has not been harmful for me, but it has been a life-saver
tool that has enabled significant progress towards more productive
research.

"Hongbo Zhang (BLOOMBERG/ 731 LEX)" <hzhang295@bloomberg.net> writes:

> calling it 'madness' is disrespectful

Personally, I fully subscribe Yaron's message and I see nothing
disrespectful in suggesting that abandoning syntactic abstractions is a
very bad idea.

You wrote:

 "the OCaml library developer should avoid PPX as much as you can",

but if you meant:

 "PPX seems quite unstable these days, I wonder how could we improve
  long-term stability?"

I'd have to admit that message didn't reach to me.

Best regards!
Emilio

Re: [Caml-list] PPX is harmful to our community in the long term

[Serge Sivkov]

[-- Attachment #1: Type: text/plain, Size: 1586 bytes --]

Hence, my two cents: PPX has problems in cross-compilation use cases, but I
suppose something like new tag in META can reslove this issue.
As for me, just ppx_deriving* by whitequark is yet one example of
usefullness of PPX.

WBR, ssp

2017-04-22 5:10 GMT+06:00 Emilio Jesús Gallego Arias <e@x80.org>:

> "Hongbo Zhang (BLOOMBERG/ 731 LEX)" <hzhang295@bloomberg.net> writes:
>
> > Yes, that's exactly what I suggested in the beginning!
>
> Maybe I interpret the word "harmful" differently, but IMVHO I have to
> strongly disagree with your choice of subject in the original mail.
>
> Not only PPX has not been harmful for me, but it has been a life-saver
> tool that has enabled significant progress towards more productive
> research.
>
> "Hongbo Zhang (BLOOMBERG/ 731 LEX)" <hzhang295@bloomberg.net> writes:
>
> > calling it 'madness' is disrespectful
>
> Personally, I fully subscribe Yaron's message and I see nothing
> disrespectful in suggesting that abandoning syntactic abstractions is a
> very bad idea.
>
> You wrote:
>
>  "the OCaml library developer should avoid PPX as much as you can",
>
> but if you meant:
>
>  "PPX seems quite unstable these days, I wonder how could we improve
>   long-term stability?"
>
> I'd have to admit that message didn't reach to me.
>
> Best regards!
> Emilio
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>

[-- Attachment #2: Type: text/html, Size: 2600 bytes --]

Re: [Caml-list] PPX is harmful to our community in the long term

[Hongbo Zhang (BLOOMBERG/ 731 LEX)]

[-- Attachment #1: Type: text/plain, Size: 3724 bytes --]

Yes, when you never depend on other people's PPX, it is perfectly fine to provide a customized
suite of PPX. 
Think about the software which only works against 4.03, 4.04, this is equivalent to say that you release
a c++ library only works with gcc 7 while most enterprises are still using 4.8, nobody even think it is a 
serious piece of software.
It is a different story in Haskell deriving or Scheme macro system, there is no issue that you software 
in use today will be not compilable in the future.

From: yminsky@janestreet.com At: 04/21/17 15:12:29
To: Hongbo Zhang (BLOOMBERG/ 731 LEX)
Cc: caml-list@inria.fr
Subject: Re: [Caml-list] PPX is harmful to our community in the long term

I understand the frustration, but I think your conclusion is
misplaced. PPXs are massively helpful when building serious software.
Having automatic generation of pretty-printers, comparison functions,
hash functions, binary protocols, and the like is a huge win for both
programmer efficiency and correctness. The Haskell folk aren't wrong
to care about deriving, and the schemers aren't crazy to want their
macro systems.

In short, I think abandoning syntactic abstractions is madness.

I agree that the portability problems are serious and should be
addressed, but ocaml-migrate-parsetree seems like a solid step in the
right direction.

y

On Fri, Apr 21, 2017 at 11:41 AM, Hongbo Zhang (BLOOMBERG/ 731 LEX)
<hzhang295@bloomberg.net> wrote:
> Dear OCaml developers:
> Given that bitten by PPX from time to time, finally, I think it is a time to
> spend two hours sharing my experience with PPX and why you(the OCaml library
> developer) should avoid PPX as much as you can.
>
> Here is a story I just experienced this morning, I tried to install a
> package from opam, and it complained my compiler is too old - 4.02.3, to be
> honest, 4.02.3 is still a pretty modern OCaml compiler, even debian stable
> still stays on 4.01. Anyway, I switched to 4.04.1, after half an hour, it
> failed to compile again, complaning about some ppx error message. This is
> not my first time experience, and finally it made me to write an essay about
> why PPX is harmful.
> PPX is a compiler plugin, it imposes a very large compiler surface API to
> your library, and we don't have any backward compatibility guarantee from
> the compiler, which means your library will only work against a specific
> compiler. Even worse, OCaml is an elegant but small community, we don't have
> too many maintainers for a library, if you have a library which relies on
> PPX (the dependency chain could be really really huge, for example,
> ppx_metaquot depends on typing environment, you can find lots of stories
> about node_modules in Node community), it will probably not work against
> next version of OCaml compiler, and it will be a huge maintenance overhead
> for other people to pick it up.
>
> OCaml is already a very expressive language, probably more expressive than
> any other mainstream language, (Go, Java, C/C++, etc), it is fine to write
> some boilerplate code, or we can cut PPX as a dev dependency, after your
> PPXed your code, check in the generated source code(via -dsource), so it
> will not bring dependency to end users.
> There are some valid use cases of PPX, for example, in BuckleScript or
> JS_of_OCaml, we want to customize OCaml language a bit for external FFI, or
> if you have a very large team, and committed effort to maintain your PPX.
> Happy hacking in OCaml without PPX, Thanks -- Hongbo
>
>

-- 
Caml-list mailing list.  Subscription management and archives:
https://sympa.inria.fr/sympa/arc/caml-list
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs



[-- Attachment #2: Type: text/html, Size: 5588 bytes --]

Re: [Caml-list] PPX is harmful to our community in the long term

[Fabrice Le Fessant]

[-- Attachment #1: Type: text/plain, Size: 4777 bytes --]

A lot of people use `autoconf` to generate `./configure` scripts, and the
standard practice is to keep the `./configure` script so that people don't
need to run `autoconf` to just compile and install the software. Maybe
projects could do the same with ppx, i.e. store pre-processed files in the
project sources so that the ppx would only be needed when the developer
modifies the sources. For example, there could be a `_ppx` directory, where
pre-processed files would be stored under the hash of a combination of
their original source code and the  `-ppx` arguments. The compiler (or the
build system) would use these files when available instead of calling the
ppx. That might reduce the problem at least for `opam`, since users are not
supposed to edit the packages when installing them.

On Fri, Apr 21, 2017 at 9:24 PM Hongbo Zhang (BLOOMBERG/ 731 LEX) <
hzhang295@bloomberg.net> wrote:

> Yes, when you never depend on other people's PPX, it is perfectly fine to
> provide a customized
> suite of PPX.
> Think about the software which only works against 4.03, 4.04, this is
> equivalent to say that you release
> a c++ library only works with gcc 7 while most enterprises are still using
> 4.8, nobody even think it is a
> serious piece of software.
> It is a different story in Haskell deriving or Scheme macro system, there
> is no issue that you software
> in use today will be not compilable in the future.
>
> From: yminsky@janestreet.com At: 04/21/17 15:12:29
> To: Hongbo Zhang (BLOOMBERG/ 731 LEX)
> Cc: caml-list@inria.fr
> Subject: Re: [Caml-list] PPX is harmful to our community in the long term
>
> I understand the frustration, but I think your conclusion is
> misplaced. PPXs are massively helpful when building serious software.
> Having automatic generation of pretty-printers, comparison functions,
> hash functions, binary protocols, and the like is a huge win for both
> programmer efficiency and correctness. The Haskell folk aren't wrong
> to care about deriving, and the schemers aren't crazy to want their
> macro systems.
>
> In short, I think abandoning syntactic abstractions is madness.
>
> I agree that the portability problems are serious and should be
> addressed, but ocaml-migrate-parsetree seems like a solid step in the
> right direction.
>
> y
>
> On Fri, Apr 21, 2017 at 11:41 AM, Hongbo Zhang (BLOOMBERG/ 731 LEX)
> <hzhang295@bloomberg.net> wrote:
> > Dear OCaml developers:
> > Given that bitten by PPX from time to time, finally, I think it is a
> time to
> > spend two hours sharing my experience with PPX and why you(the OCaml
> library
> > developer) should avoid PPX as much as you can.
> >
> > Here is a story I just experienced this morning, I tried to install a
> > package from opam, and it complained my compiler is too old - 4.02.3, to
> be
> > honest, 4.02.3 is still a pretty modern OCaml compiler, even debian
> stable
> > still stays on 4.01. Anyway, I switched to 4.04.1, after half an hour, it
> > failed to compile again, complaning about some ppx error message. This is
> > not my first time experience, and finally it made me to write an essay
> about
> > why PPX is harmful.
> > PPX is a compiler plugin, it imposes a very large compiler surface API to
> > your library, and we don't have any backward compatibility guarantee from
> > the compiler, which means your library will only work against a specific
> > compiler. Even worse, OCaml is an elegant but small community, we don't
> have
> > too many maintainers for a library, if you have a library which relies on
> > PPX (the dependency chain could be really really huge, for example,
> > ppx_metaquot depends on typing environment, you can find lots of stories
> > about node_modules in Node community), it will probably not work against
> > next version of OCaml compiler, and it will be a huge maintenance
> overhead
> > for other people to pick it up.
> >
> > OCaml is already a very expressive language, probably more expressive
> than
> > any other mainstream language, (Go, Java, C/C++, etc), it is fine to
> write
> > some boilerplate code, or we can cut PPX as a dev dependency, after your
> > PPXed your code, check in the generated source code(via -dsource), so it
> > will not bring dependency to end users.
> > There are some valid use cases of PPX, for example, in BuckleScript or
> > JS_of_OCaml, we want to customize OCaml language a bit for external FFI,
> or
> > if you have a very large team, and committed effort to maintain your PPX.
> > Happy hacking in OCaml without PPX, Thanks -- Hongbo
> >
> >
>
> --
> Caml-list mailing list. Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>
>

[-- Attachment #2: Type: text/html, Size: 6754 bytes --]

Re: [Caml-list] PPX is harmful to our community in the long term

[Olaf Hering]

[-- Attachment #1: Type: text/plain, Size: 638 bytes --]

On Fri, Apr 21, Fabrice Le Fessant wrote:

> A lot of people use `autoconf` to generate `./configure` scripts, and the
> standard practice is to keep the `./configure` script so that people don't need
> to run `autoconf` to just compile and install the software. Maybe projects

This is and was a huge mistake to promote 'configure&&make' instead of
autogen.sh&&configure&&make. Having a set of uptodate autotools
installed is easy and cheap, they are not runtime dependencies. The
result of that wrongdoing is a huge pile of broken and/or incomplete configure.ac.

Do not repeat that mistake, whatever it means in the OCaml world.

Olaf

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [Caml-list] PPX is harmful to our community in the long term

[Anil Madhavapeddy]

On 28 Apr 2017, at 12:07, Olaf Hering <olaf@aepfle.de> wrote:
> 
> On Fri, Apr 21, Fabrice Le Fessant wrote:
> 
>> A lot of people use `autoconf` to generate `./configure` scripts, and the
>> standard practice is to keep the `./configure` script so that people don't need
>> to run `autoconf` to just compile and install the software. Maybe projects
> 
> This is and was a huge mistake to promote 'configure&&make' instead of
> autogen.sh&&configure&&make. Having a set of uptodate autotools
> installed is easy and cheap, they are not runtime dependencies. The
> result of that wrongdoing is a huge pile of broken and/or incomplete configure.ac.

Indeed -- we spent years in OpenBSD dealing with patching broken versions
of libtool scripts that embedded incorrect behaviour on our particular platforms,
and were stubbornly included in upstream distributions without being regenerated.

> Do not repeat that mistake, whatever it means in the OCaml world.

A similar analogue in the OCaml world may be the inclusion of autogenerated
files from _oasis.  The inclusion of the autogenerated files like myocamlbuild.ml
was a holdover from a pre-opam world when it was painful to install all the
build dependencies of OASIS.

Nowadays, it's quite easy to install oasis and run `oasis setup` in a project
to get the latest build rules, and the checked in autogenerated files only
get in the way and/or are increasingly complex due to having to deal with
multiple OCaml versions (e.g. for String.lowercase vs String.lowercase_ascii).

Bundling pre-evaluated ppx output in a release tarball runs the same risk...

Our experience in Mirage with PPX has been to find a balance -- we do not let
it proliferate beyond type_conv usage or ppx_cstruct for binary formats.  Some
libraries (such as the TLS stack) do not use it all. One of the heaviest uses
of camlp4 in the past for us was the pa_lwt syntax extension, and most libraries
have gone towards explicit Lwt.bind() calls instead of using the ppx alternative.

I'm hopeful that ocaml-migrate-parsetree will make it easier for us to test
common libraries on dev versions of OCaml.  In practise with 4.05, it has been
non-ppx changes that have been blocking testing -- for instance the close-on-exec
flag addition to the Unix module caused rippling breakage through Lwt and other
platform libraries.  That's not to say that PPX isn't a problem, but it has
gotten significantly easier to deal with since Fred's work on migrate-parsetree.

regards,
Anil

Re: [Caml-list] PPX is harmful to our community in the long term

[Yaron Minsky]

[-- Attachment #1: Type: text/plain, Size: 3110 bytes --]

We're very similar, except that we now do use a monadic syntax pretty
pervasively. I wrote about this here:

https://blogs.janestreet.com/let-syntax-and-why-you-should-use-it/

and am if anything more convinced that it's a worthwhile idiom. Monads and
Applicatives are useful in so many places (beyond Async and Lwt) that
having syntactic support for them is really nice, especially for the let ..
and constructs.

y

On Fri, Apr 28, 2017 at 9:04 AM, Anil Madhavapeddy <anil@recoil.org> wrote:

> On 28 Apr 2017, at 12:07, Olaf Hering <olaf@aepfle.de> wrote:
> >
> > On Fri, Apr 21, Fabrice Le Fessant wrote:
> >
> >> A lot of people use `autoconf` to generate `./configure` scripts, and
> the
> >> standard practice is to keep the `./configure` script so that people
> don't need
> >> to run `autoconf` to just compile and install the software. Maybe
> projects
> >
> > This is and was a huge mistake to promote 'configure&&make' instead of
> > autogen.sh&&configure&&make. Having a set of uptodate autotools
> > installed is easy and cheap, they are not runtime dependencies. The
> > result of that wrongdoing is a huge pile of broken and/or incomplete
> configure.ac.
>
> Indeed -- we spent years in OpenBSD dealing with patching broken versions
> of libtool scripts that embedded incorrect behaviour on our particular
> platforms,
> and were stubbornly included in upstream distributions without being
> regenerated.
>
> > Do not repeat that mistake, whatever it means in the OCaml world.
>
> A similar analogue in the OCaml world may be the inclusion of autogenerated
> files from _oasis.  The inclusion of the autogenerated files like
> myocamlbuild.ml
> was a holdover from a pre-opam world when it was painful to install all the
> build dependencies of OASIS.
>
> Nowadays, it's quite easy to install oasis and run `oasis setup` in a
> project
> to get the latest build rules, and the checked in autogenerated files only
> get in the way and/or are increasingly complex due to having to deal with
> multiple OCaml versions (e.g. for String.lowercase vs
> String.lowercase_ascii).
>
> Bundling pre-evaluated ppx output in a release tarball runs the same
> risk...
>
> Our experience in Mirage with PPX has been to find a balance -- we do not
> let
> it proliferate beyond type_conv usage or ppx_cstruct for binary formats.
> Some
> libraries (such as the TLS stack) do not use it all. One of the heaviest
> uses
> of camlp4 in the past for us was the pa_lwt syntax extension, and most
> libraries
> have gone towards explicit Lwt.bind() calls instead of using the ppx
> alternative.
>
> I'm hopeful that ocaml-migrate-parsetree will make it easier for us to test
> common libraries on dev versions of OCaml.  In practise with 4.05, it has
> been
> non-ppx changes that have been blocking testing -- for instance the
> close-on-exec
> flag addition to the Unix module caused rippling breakage through Lwt and
> other
> platform libraries.  That's not to say that PPX isn't a problem, but it has
> gotten significantly easier to deal with since Fred's work on
> migrate-parsetree.
>
> regards,
> Anil
>
>

[-- Attachment #2: Type: text/html, Size: 4004 bytes --]

Re: [Caml-list] PPX is harmful to our community in the long term

[Jacques Carette]

[-- Attachment #1: Type: text/plain, Size: 3638 bytes --]

As co-author of the now-obsolete pa_monad, I emphatically agree!
Jacques

On 2017-04-28 10:50 AM, Yaron Minsky wrote:
> We're very similar, except that we now do use a monadic syntax pretty 
> pervasively. I wrote about this here:
>
> https://blogs.janestreet.com/let-syntax-and-why-you-should-use-it/
>
> and am if anything more convinced that it's a worthwhile idiom. Monads 
> and Applicatives are useful in so many places (beyond Async and Lwt) 
> that having syntactic support for them is really nice, especially for 
> the let .. and constructs.
>
> y
>
> On Fri, Apr 28, 2017 at 9:04 AM, Anil Madhavapeddy <anil@recoil.org 
> <mailto:anil@recoil.org>> wrote:
>
>     On 28 Apr 2017, at 12:07, Olaf Hering <olaf@aepfle.de
>     <mailto:olaf@aepfle.de>> wrote:
>     >
>     > On Fri, Apr 21, Fabrice Le Fessant wrote:
>     >
>     >> A lot of people use `autoconf` to generate `./configure`
>     scripts, and the
>     >> standard practice is to keep the `./configure` script so that
>     people don't need
>     >> to run `autoconf` to just compile and install the software.
>     Maybe projects
>     >
>     > This is and was a huge mistake to promote 'configure&&make'
>     instead of
>     > autogen.sh&&configure&&make. Having a set of uptodate autotools
>     > installed is easy and cheap, they are not runtime dependencies. The
>     > result of that wrongdoing is a huge pile of broken and/or
>     incomplete configure.ac <http://configure.ac>.
>
>     Indeed -- we spent years in OpenBSD dealing with patching broken
>     versions
>     of libtool scripts that embedded incorrect behaviour on our
>     particular platforms,
>     and were stubbornly included in upstream distributions without
>     being regenerated.
>
>     > Do not repeat that mistake, whatever it means in the OCaml world.
>
>     A similar analogue in the OCaml world may be the inclusion of
>     autogenerated
>     files from _oasis.  The inclusion of the autogenerated files like
>     myocamlbuild.ml <http://myocamlbuild.ml>
>     was a holdover from a pre-opam world when it was painful to
>     install all the
>     build dependencies of OASIS.
>
>     Nowadays, it's quite easy to install oasis and run `oasis setup`
>     in a project
>     to get the latest build rules, and the checked in autogenerated
>     files only
>     get in the way and/or are increasingly complex due to having to
>     deal with
>     multiple OCaml versions (e.g. for String.lowercase vs
>     String.lowercase_ascii).
>
>     Bundling pre-evaluated ppx output in a release tarball runs the
>     same risk...
>
>     Our experience in Mirage with PPX has been to find a balance -- we
>     do not let
>     it proliferate beyond type_conv usage or ppx_cstruct for binary
>     formats.  Some
>     libraries (such as the TLS stack) do not use it all. One of the
>     heaviest uses
>     of camlp4 in the past for us was the pa_lwt syntax extension, and
>     most libraries
>     have gone towards explicit Lwt.bind() calls instead of using the
>     ppx alternative.
>
>     I'm hopeful that ocaml-migrate-parsetree will make it easier for
>     us to test
>     common libraries on dev versions of OCaml.  In practise with 4.05,
>     it has been
>     non-ppx changes that have been blocking testing -- for instance
>     the close-on-exec
>     flag addition to the Unix module caused rippling breakage through
>     Lwt and other
>     platform libraries.  That's not to say that PPX isn't a problem,
>     but it has
>     gotten significantly easier to deal with since Fred's work on
>     migrate-parsetree.
>
>     regards,
>     Anil
>
>


[-- Attachment #2: Type: text/html, Size: 5926 bytes --]

Re: [Caml-list] PPX is harmful to our community in the long term

[Jeremie Dimino]

[-- Attachment #1: Type: text/plain, Size: 4148 bytes --]

Coming back to this thread. I had a simple idea recently for a ppx that
makes it easy to do pattern matching on  abstract types. I wrote some
experiments here [1]. This essentially allows to make the AST fully
abstract while still being able to deconstruct it conveniently. In fact the
patterns are even nicer than ones matching the raw data type directly since
you can build your own helper patterns.

Making the AST abstract will allow to make the API evolve in a backward
compatible way even though the underlying AST keeps changing.

I just did some experiments for now. I think we'll eventually implement
this solution properly and use it in our ppx code.

[1] https://github.com/diml/ppx_view_pattern

On Fri, Apr 28, 2017 at 3:55 PM, Jacques Carette <carette@mcmaster.ca>
wrote:

> As co-author of the now-obsolete pa_monad, I emphatically agree!
> Jacques
>
>
> On 2017-04-28 10:50 AM, Yaron Minsky wrote:
>
> We're very similar, except that we now do use a monadic syntax pretty
> pervasively. I wrote about this here:
>
> https://blogs.janestreet.com/let-syntax-and-why-you-should-use-it/
>
> and am if anything more convinced that it's a worthwhile idiom. Monads and
> Applicatives are useful in so many places (beyond Async and Lwt) that
> having syntactic support for them is really nice, especially for the let ..
> and constructs.
>
> y
>
> On Fri, Apr 28, 2017 at 9:04 AM, Anil Madhavapeddy <anil@recoil.org>
> wrote:
>
>> On 28 Apr 2017, at 12:07, Olaf Hering <olaf@aepfle.de> wrote:
>> >
>> > On Fri, Apr 21, Fabrice Le Fessant wrote:
>> >
>> >> A lot of people use `autoconf` to generate `./configure` scripts, and
>> the
>> >> standard practice is to keep the `./configure` script so that people
>> don't need
>> >> to run `autoconf` to just compile and install the software. Maybe
>> projects
>> >
>> > This is and was a huge mistake to promote 'configure&&make' instead of
>> > autogen.sh&&configure&&make. Having a set of uptodate autotools
>> > installed is easy and cheap, they are not runtime dependencies. The
>> > result of that wrongdoing is a huge pile of broken and/or incomplete
>> configure.ac.
>>
>> Indeed -- we spent years in OpenBSD dealing with patching broken versions
>> of libtool scripts that embedded incorrect behaviour on our particular
>> platforms,
>> and were stubbornly included in upstream distributions without being
>> regenerated.
>>
>> > Do not repeat that mistake, whatever it means in the OCaml world.
>>
>> A similar analogue in the OCaml world may be the inclusion of
>> autogenerated
>> files from _oasis.  The inclusion of the autogenerated files like
>> myocamlbuild.ml
>> was a holdover from a pre-opam world when it was painful to install all
>> the
>> build dependencies of OASIS.
>>
>> Nowadays, it's quite easy to install oasis and run `oasis setup` in a
>> project
>> to get the latest build rules, and the checked in autogenerated files only
>> get in the way and/or are increasingly complex due to having to deal with
>> multiple OCaml versions (e.g. for String.lowercase vs
>> String.lowercase_ascii).
>>
>> Bundling pre-evaluated ppx output in a release tarball runs the same
>> risk...
>>
>> Our experience in Mirage with PPX has been to find a balance -- we do not
>> let
>> it proliferate beyond type_conv usage or ppx_cstruct for binary formats.
>> Some
>> libraries (such as the TLS stack) do not use it all. One of the heaviest
>> uses
>> of camlp4 in the past for us was the pa_lwt syntax extension, and most
>> libraries
>> have gone towards explicit Lwt.bind() calls instead of using the ppx
>> alternative.
>>
>> I'm hopeful that ocaml-migrate-parsetree will make it easier for us to
>> test
>> common libraries on dev versions of OCaml.  In practise with 4.05, it has
>> been
>> non-ppx changes that have been blocking testing -- for instance the
>> close-on-exec
>> flag addition to the Unix module caused rippling breakage through Lwt and
>> other
>> platform libraries.  That's not to say that PPX isn't a problem, but it
>> has
>> gotten significantly easier to deal with since Fred's work on
>> migrate-parsetree.
>>
>> regards,
>> Anil
>>
>>
>
>


-- 
Jeremie

[-- Attachment #2: Type: text/html, Size: 7600 bytes --]

[Caml-list] PPX is harmful to our community in the long term

[Hongbo Zhang (BLOOMBERG/ 731 LEX)]

[-- Attachment #1: Type: text/plain, Size: 2031 bytes --]

Dear OCaml developers:
   Given that  bitten by PPX from time to time, finally, I think it is a time to spend two hours sharing my experience with PPX and why you(the OCaml library developer) should avoid PPX as much as you can.

   Here is a story I just experienced this morning, I tried to install a package from opam, and it complained my compiler is too old - 4.02.3, to be honest, 4.02.3 is still a pretty modern OCaml compiler, even debian stable still stays on 4.01. Anyway, I switched to 4.04.1, after half an hour, it failed to compile again, complaning about some ppx error message. This is not my first time experience, and finally it made me to write an essay about why PPX is harmful.
  
    PPX is a compiler plugin, it imposes a very large compiler surface API to your library, and we don't have any backward compatibility guarantee from the compiler, which means your library will only work against a specific compiler. Even worse, OCaml is an elegant but small community, we don't have too many maintainers for a library, if you have a library which relies on PPX (the dependency chain could be really really huge, for example, ppx_metaquot depends on typing environment, you can find lots of stories about node_modules in Node community), it will probably not work against next version of OCaml compiler, and it will be a huge maintenance overhead for other people to pick it up.

    OCaml is already a very expressive language, probably more expressive than any other mainstream language, (Go, Java, C/C++, etc), it is fine to write some boilerplate code, or we can cut PPX as a dev dependency, after your PPXed your code, check in the generated source code(via -dsource), so it will not bring dependency to end users.
   
    There are some valid use cases of PPX, for example, in BuckleScript or JS_of_OCaml, we want to customize OCaml language a bit for external FFI, or if you have a very large team, and committed effort to maintain your PPX.
  
    Happy hacking in OCaml without PPX, Thanks -- Hongbo



[-- Attachment #2: Type: text/html, Size: 2642 bytes --]

Re: [Caml-list] PPX is harmful to our community in the long term

[Yotam Barnoy]

My 2 cents: I personally think we did PPX wrong. PPX should have used
the syntax as its starting point for full backwards compatibility.
Currently, the PPX cycle is

OCaml: { syntax -> AST -> serialized AST } -> PPX: {serialized AST ->
AST -> PPX modification -> serialized AST} -> OCaml: {serialized AST
-> AST -> ... }

Exposing the AST is the cause of the problem you mention. If instead,
every PPX extension had a particular OCaml compiler's syntax parser
and syntax printer integrated into it, and the PPX cycle was:

PPX: { syntax -> AST -> PPX modification -> syntax } -> compiler: {
syntax -> AST -> ...}

We would have far fewer issues with PPX plugins, since they would be
as backwards-compatible as the syntax.

I think this mistake happened because everyone was trying to move away
from camlp4's complexity, and we didn't notice that the problem with
camlp4 wasn't the approach, but rather the fact that it wasn't making
use of the same code as the compiler and therefore required its own
heavy learning-curve and parallel maintenance to keep up with the
compiler. To be fair though, camlp4 came before the age of OCaml on
github, opam, and the associated easy code/library sharing.

The only issue I can see with this idea of improving PPX is that the
PPX-generated syntax wouldn't have the original source locations. We'd
need an annotation that indicated the original source locations
(perhaps row and column offsets) when a PPX translated to modified
syntax and printed it out, so that the compiler's errors would have
proper locations relative to the original source.

-Yotam


On Fri, Apr 21, 2017 at 11:41 AM, Hongbo Zhang (BLOOMBERG/ 731 LEX)
<hzhang295@bloomberg.net> wrote:
> Dear OCaml developers:
> Given that bitten by PPX from time to time, finally, I think it is a time to
> spend two hours sharing my experience with PPX and why you(the OCaml library
> developer) should avoid PPX as much as you can.
>
> Here is a story I just experienced this morning, I tried to install a
> package from opam, and it complained my compiler is too old - 4.02.3, to be
> honest, 4.02.3 is still a pretty modern OCaml compiler, even debian stable
> still stays on 4.01. Anyway, I switched to 4.04.1, after half an hour, it
> failed to compile again, complaning about some ppx error message. This is
> not my first time experience, and finally it made me to write an essay about
> why PPX is harmful.
> PPX is a compiler plugin, it imposes a very large compiler surface API to
> your library, and we don't have any backward compatibility guarantee from
> the compiler, which means your library will only work against a specific
> compiler. Even worse, OCaml is an elegant but small community, we don't have
> too many maintainers for a library, if you have a library which relies on
> PPX (the dependency chain could be really really huge, for example,
> ppx_metaquot depends on typing environment, you can find lots of stories
> about node_modules in Node community), it will probably not work against
> next version of OCaml compiler, and it will be a huge maintenance overhead
> for other people to pick it up.
>
> OCaml is already a very expressive language, probably more expressive than
> any other mainstream language, (Go, Java, C/C++, etc), it is fine to write
> some boilerplate code, or we can cut PPX as a dev dependency, after your
> PPXed your code, check in the generated source code(via -dsource), so it
> will not bring dependency to end users.
> There are some valid use cases of PPX, for example, in BuckleScript or
> JS_of_OCaml, we want to customize OCaml language a bit for external FFI, or
> if you have a very large team, and committed effort to maintain your PPX.
> Happy hacking in OCaml without PPX, Thanks -- Hongbo
>
>

Re: [Caml-list] PPX is harmful to our community in the long term

[Gerd Stolpmann]

So far I understand, with "syntax" you really mean syntax-as-string.

I think most often the AST changes by adding more fields or more AST 
nodes. Wouldn't the situation already improve a lot if we switched to 
record variants for the AST types? (Besides that the compiler would get 
way more readable.) New fields wouldn't be a big problem anymore because 
they are looked up by name and not by position.

Gerd


On 21.04.17 18:04, Yotam Barnoy wrote:
> My 2 cents: I personally think we did PPX wrong. PPX should have used
> the syntax as its starting point for full backwards compatibility.
> Currently, the PPX cycle is
>
> OCaml: { syntax -> AST -> serialized AST } -> PPX: {serialized AST ->
> AST -> PPX modification -> serialized AST} -> OCaml: {serialized AST
> -> AST -> ... }
>
> Exposing the AST is the cause of the problem you mention. If instead,
> every PPX extension had a particular OCaml compiler's syntax parser
> and syntax printer integrated into it, and the PPX cycle was:
>
> PPX: { syntax -> AST -> PPX modification -> syntax } -> compiler: {
> syntax -> AST -> ...}
>
> We would have far fewer issues with PPX plugins, since they would be
> as backwards-compatible as the syntax.
>
> I think this mistake happened because everyone was trying to move away
> from camlp4's complexity, and we didn't notice that the problem with
> camlp4 wasn't the approach, but rather the fact that it wasn't making
> use of the same code as the compiler and therefore required its own
> heavy learning-curve and parallel maintenance to keep up with the
> compiler. To be fair though, camlp4 came before the age of OCaml on
> github, opam, and the associated easy code/library sharing.
>
> The only issue I can see with this idea of improving PPX is that the
> PPX-generated syntax wouldn't have the original source locations. We'd
> need an annotation that indicated the original source locations
> (perhaps row and column offsets) when a PPX translated to modified
> syntax and printed it out, so that the compiler's errors would have
> proper locations relative to the original source.
>
> -Yotam
>
>
> On Fri, Apr 21, 2017 at 11:41 AM, Hongbo Zhang (BLOOMBERG/ 731 LEX)
> <hzhang295@bloomberg.net> wrote:
>> Dear OCaml developers:
>> Given that bitten by PPX from time to time, finally, I think it is a time to
>> spend two hours sharing my experience with PPX and why you(the OCaml library
>> developer) should avoid PPX as much as you can.
>>
>> Here is a story I just experienced this morning, I tried to install a
>> package from opam, and it complained my compiler is too old - 4.02.3, to be
>> honest, 4.02.3 is still a pretty modern OCaml compiler, even debian stable
>> still stays on 4.01. Anyway, I switched to 4.04.1, after half an hour, it
>> failed to compile again, complaning about some ppx error message. This is
>> not my first time experience, and finally it made me to write an essay about
>> why PPX is harmful.
>> PPX is a compiler plugin, it imposes a very large compiler surface API to
>> your library, and we don't have any backward compatibility guarantee from
>> the compiler, which means your library will only work against a specific
>> compiler. Even worse, OCaml is an elegant but small community, we don't have
>> too many maintainers for a library, if you have a library which relies on
>> PPX (the dependency chain could be really really huge, for example,
>> ppx_metaquot depends on typing environment, you can find lots of stories
>> about node_modules in Node community), it will probably not work against
>> next version of OCaml compiler, and it will be a huge maintenance overhead
>> for other people to pick it up.
>>
>> OCaml is already a very expressive language, probably more expressive than
>> any other mainstream language, (Go, Java, C/C++, etc), it is fine to write
>> some boilerplate code, or we can cut PPX as a dev dependency, after your
>> PPXed your code, check in the generated source code(via -dsource), so it
>> will not bring dependency to end users.
>> There are some valid use cases of PPX, for example, in BuckleScript or
>> JS_of_OCaml, we want to customize OCaml language a bit for external FFI, or
>> if you have a very large team, and committed effort to maintain your PPX.
>> Happy hacking in OCaml without PPX, Thanks -- Hongbo
>>
>>

Re: [Caml-list] PPX is harmful to our community in the long term

[Alain Frisch]

On 21/04/2017 18:04, Yotam Barnoy wrote:
> My 2 cents: I personally think we did PPX wrong. PPX should have used
> the syntax as its starting point for full backwards compatibility.
> Currently, the PPX cycle is
>
> OCaml: { syntax -> AST -> serialized AST } -> PPX: {serialized AST ->
> AST -> PPX modification -> serialized AST} -> OCaml: {serialized AST
> -> AST -> ... }
>
> Exposing the AST is the cause of the problem you mention. If instead,
> every PPX extension had a particular OCaml compiler's syntax parser
> and syntax printer integrated into it, and the PPX cycle was:
>
> PPX: { syntax -> AST -> PPX modification -> syntax } -> compiler: {
> syntax -> AST -> ...}
>
> We would have far fewer issues with PPX plugins, since they would be
> as backwards-compatible as the syntax.

I think

   https://github.com/let-def/ocaml-migrate-parsetree

is a much stronger approach.  If I understand correctly, Jane Street 
moved from using a migration system based on concrete syntax to using 
this new project.


In your suggested approach, imagine you have a PPX processor written for 
OCaml 4.04.  It assumes that the Parsetree it works on is the one of 
4.04.  Sure, you can compile this PPX processor (+ embedded 
parser/printer) into a stand-alone executable, using OCaml 4.04, and 
then apply it as a preprocessor called from OCaml 4.05.  But this is 
very impractical: users would need to install OCaml 4.04 just to produce 
the PPX executable.  Moreover, you loose the ability to combine multiple 
rewritings in a single process, and you have to pay the price of 
multiple parsing/printing/processes.  On top of that:

   (i) you put in the critical loop "pprintast.ml", which has always 
been more or less buggy;

   (ii) there is no hope that an "old PPX" applies to source code using 
newest syntactic features;

   (iii) as you mention, locations would need to be added for each node 
in the AST, which makes the parsing/printing even slower;  moreover, it 
is not technically straightforward to do so, since many places in the 
AST contains locations but do not support attaching attributes;

   (iv) we loose the ability to change the concrete syntax of OCaml, 
either to use alternative syntaxes such as Reason, or to remove, at some 
point, weird corner cases in the current syntax.


With ocaml-migrate-parsetree, you can still compile your PPX written for 
4.04 with newer versions of OCaml.  And it would even be possible (I 
don't know if this is already the case) to apply it to source code using 
newer syntactic features, as long as the rewriting done by the PPX 
doesn't interact with those features.  (This could be achieved by 
turning new syntactic features into attributes/extension nodes when 
mapping to the older AST; and then recognizing these forms and 
reconstitute the correct new AST forms when mapping back to the new 
version.)


-- Alain

Re: [Caml-list] PPX is harmful to our community in the long term

[Jeremie Dimino]

[-- Attachment #1: Type: text/plain, Size: 5394 bytes --]

Yes, we are now using ocaml-migrate-parsetree at Jane Street. I think that
with this library the versioning story is much better. We are heavy user of
ppx rewriters and all of the code we release now builds with OCaml 4.03 to
4.06 without problems.

Initially we considered using the concrete syntax to solve the versioning
problem and we have a solution that should work in theory. Although it's
not great when there is no source file, such as in the toplevel. The method
we considered is described in [1]. Switching to ocaml-migrate-parsetree was
easier since we didn't have to change anything to the way things worked,
just port the code.

Regarding the idea of cutting down the dependency for release tarballs by
including the generated code, while this is an interesting idea, there are
a lot of small annoying problems that makes this technique hard in
practice. The two main problems are:

1. it doesn't work for .ml files that are generated. Basically all the code
you generate with custom code generators as to be ppx free. It is fine
however for pure generators, since you can just embed the generated code

2. it doesn't work if you use conditional compilation. Conditional
compilation is not great in general, but it is a lot of work to completely
get rid of it

However, one thing that we kept from these experiments is the idea to use
ppx in the same way that expectation tests work. This idea is described in
[1] and we use it for the Base library [2]. Additionally it is a good
testing/debugging tool. It is a viable solution if all you need is
[@@deriving] plugins and don't want to depend on ppx.

The idea is to let the ppx rewriter insert the code generated by
[@@deriving] plugins in the source code:

type t = A | B [@@deriving_inline sexp_of]

let sexp_of_t = function
  | A -> Sexp.Atom "A"
  | B -> Sexp.Atom "B"

[@@@end_of_derived_code]

The code builds without ppx and you still don't have to write the
boilerplate yourself, you let the ppx tool do it for you.

[1] https://blogs.janestreet.com/an-solution-to-the-ppx-versioning-problem/
[2] https://github.com/janestreet/base


On Fri, Apr 21, 2017 at 6:11 PM, Alain Frisch <alain.frisch@lexifi.com>
wrote:

> On 21/04/2017 18:04, Yotam Barnoy wrote:
>
>> My 2 cents: I personally think we did PPX wrong. PPX should have used
>> the syntax as its starting point for full backwards compatibility.
>> Currently, the PPX cycle is
>>
>> OCaml: { syntax -> AST -> serialized AST } -> PPX: {serialized AST ->
>> AST -> PPX modification -> serialized AST} -> OCaml: {serialized AST
>> -> AST -> ... }
>>
>> Exposing the AST is the cause of the problem you mention. If instead,
>> every PPX extension had a particular OCaml compiler's syntax parser
>> and syntax printer integrated into it, and the PPX cycle was:
>>
>> PPX: { syntax -> AST -> PPX modification -> syntax } -> compiler: {
>> syntax -> AST -> ...}
>>
>> We would have far fewer issues with PPX plugins, since they would be
>> as backwards-compatible as the syntax.
>>
>
> I think
>
>   https://github.com/let-def/ocaml-migrate-parsetree
>
> is a much stronger approach.  If I understand correctly, Jane Street moved
> from using a migration system based on concrete syntax to using this new
> project.
>
>
> In your suggested approach, imagine you have a PPX processor written for
> OCaml 4.04.  It assumes that the Parsetree it works on is the one of 4.04.
> Sure, you can compile this PPX processor (+ embedded parser/printer) into a
> stand-alone executable, using OCaml 4.04, and then apply it as a
> preprocessor called from OCaml 4.05.  But this is very impractical: users
> would need to install OCaml 4.04 just to produce the PPX executable.
> Moreover, you loose the ability to combine multiple rewritings in a single
> process, and you have to pay the price of multiple
> parsing/printing/processes.  On top of that:
>
>   (i) you put in the critical loop "pprintast.ml", which has always been
> more or less buggy;
>
>   (ii) there is no hope that an "old PPX" applies to source code using
> newest syntactic features;
>
>   (iii) as you mention, locations would need to be added for each node in
> the AST, which makes the parsing/printing even slower;  moreover, it is not
> technically straightforward to do so, since many places in the AST contains
> locations but do not support attaching attributes;
>
>   (iv) we loose the ability to change the concrete syntax of OCaml, either
> to use alternative syntaxes such as Reason, or to remove, at some point,
> weird corner cases in the current syntax.
>
>
> With ocaml-migrate-parsetree, you can still compile your PPX written for
> 4.04 with newer versions of OCaml.  And it would even be possible (I don't
> know if this is already the case) to apply it to source code using newer
> syntactic features, as long as the rewriting done by the PPX doesn't
> interact with those features.  (This could be achieved by turning new
> syntactic features into attributes/extension nodes when mapping to the
> older AST; and then recognizing these forms and reconstitute the correct
> new AST forms when mapping back to the new version.)
>
>
> -- Alain
>
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>



-- 
Jeremie

[-- Attachment #2: Type: text/html, Size: 9138 bytes --]

Re: [Caml-list] PPX is harmful to our community in the long term

[Francois BERENGER]

On 04/21/2017 10:41 AM, Hongbo Zhang (BLOOMBERG/ 731 LEX) wrote:
> Dear OCaml developers:
> Given that bitten by PPX from time to time, finally, I think it is a
> time to spend two hours sharing my experience with PPX and why you(the
> OCaml library developer) should avoid PPX as much as you can.
>
> Here is a story I just experienced this morning, I tried to install a
> package from opam, and it complained my compiler is too old - 4.02.3, to
> be honest, 4.02.3 is still a pretty modern OCaml compiler, even debian
> stable still stays on 4.01. Anyway, I switched to 4.04.1, after half an
> hour, it failed to compile again, complaning about some ppx error
> message. This is not my first time experience, and finally it made me to
> write an essay about why PPX is harmful.
> PPX is a compiler plugin, it imposesa very large compiler surface API to
> your library, and we don't have any backward compatibility guarantee
> from the compiler, which means your library will only work against a
> specific compiler. Even worse, OCaml is an elegant but small community,
> we don't have too many maintainers for a library, if you have a library
> which relies on PPX (the dependency chain could be really really huge,
> for example, ppx_metaquot depends on typing environment, you can find
> lots of stories about node_modules in Node community), it will probably
> not work against next version of OCaml compiler, and it will be a huge
> maintenance overhead for other people to pick it up.
>
> OCaml is already a very expressive language, probably more expressive
> than any other mainstream language, (Go, Java, C/C++, etc), it is fine
> to write some boilerplate code, or we can cut PPX as a dev dependency,
> after your PPXed your code, check in the generated source code(via
> -dsource), so it will not bring dependency to end users.

Does this -dsource trick makes the code backward compatible with older 
compiler versions?
Or is it just upward compatible with future compiler versions?

 From an end-user point of view, it is quite useful if we can have more 
opam packages that can install, whatever the version of the compiler is.

Regards,
F.

Re: [Caml-list] PPX is harmful to our community in the long term

[Yaron Minsky]

I understand the frustration, but I think your conclusion is
misplaced. PPXs are massively helpful when building serious software.
Having automatic generation of pretty-printers, comparison functions,
hash functions, binary protocols, and the like is a huge win for both
programmer efficiency and correctness. The Haskell folk aren't wrong
to care about deriving, and the schemers aren't crazy to want their
macro systems.

In short, I think abandoning syntactic abstractions is madness.

I agree that the portability problems are serious and should be
addressed, but ocaml-migrate-parsetree seems like a solid step in the
right direction.

y

On Fri, Apr 21, 2017 at 11:41 AM, Hongbo Zhang (BLOOMBERG/ 731 LEX)
<hzhang295@bloomberg.net> wrote:
> Dear OCaml developers:
> Given that bitten by PPX from time to time, finally, I think it is a time to
> spend two hours sharing my experience with PPX and why you(the OCaml library
> developer) should avoid PPX as much as you can.
>
> Here is a story I just experienced this morning, I tried to install a
> package from opam, and it complained my compiler is too old - 4.02.3, to be
> honest, 4.02.3 is still a pretty modern OCaml compiler, even debian stable
> still stays on 4.01. Anyway, I switched to 4.04.1, after half an hour, it
> failed to compile again, complaning about some ppx error message. This is
> not my first time experience, and finally it made me to write an essay about
> why PPX is harmful.
> PPX is a compiler plugin, it imposes a very large compiler surface API to
> your library, and we don't have any backward compatibility guarantee from
> the compiler, which means your library will only work against a specific
> compiler. Even worse, OCaml is an elegant but small community, we don't have
> too many maintainers for a library, if you have a library which relies on
> PPX (the dependency chain could be really really huge, for example,
> ppx_metaquot depends on typing environment, you can find lots of stories
> about node_modules in Node community), it will probably not work against
> next version of OCaml compiler, and it will be a huge maintenance overhead
> for other people to pick it up.
>
> OCaml is already a very expressive language, probably more expressive than
> any other mainstream language, (Go, Java, C/C++, etc), it is fine to write
> some boilerplate code, or we can cut PPX as a dev dependency, after your
> PPXed your code, check in the generated source code(via -dsource), so it
> will not bring dependency to end users.
> There are some valid use cases of PPX, for example, in BuckleScript or
> JS_of_OCaml, we want to customize OCaml language a bit for external FFI, or
> if you have a very large team, and committed effort to maintain your PPX.
> Happy hacking in OCaml without PPX, Thanks -- Hongbo
>
>

Re: [Caml-list] PPX is harmful to our community in the long term

[Emilio Jesús Gallego Arias]

Hi all,

indeed I think this problem is real as I have been trying to design a
plan for Coq 8.8 (expected Q1 2018) to depend on PPX, and indeed it has
not been easy to come up with something that wouldn't be turned down by
the developers due to compatibility concerns.

It is great to hear that ocaml-migrate-parsetree helps and that the
situation is improving, but maybe the biggest concern would Coq adopt
that solution, is how fast would it become "obsolete", to be replaced by
a new one.

From my attendance to the Coq WG I got the personal feeling that there
was some amount of uncertainty about what the OCaml roadmap regarding
certain features is. This in turn difficults the elaboration of Coq's
own roadmap.

Best regards,
Emilio