[Caml-list] Working Group: the future of syntax extensions in OCaml, after camlp4

[Caml-list] Working Group: the future of syntax extensions in OCaml, after camlp4

[Alain Frisch]

Dear caml-list,

There is a growing opinion that camlp4 is overly complex considering the 
benefits it brings to OCaml developers.  I would personally go as far as 
to say that the future of OCaml and the OCaml community would be 
brighter if camlp4 could be removed from our "basic ecosystem".  In 
particular, most of the current uses of camlp4 to create syntax 
extensions could probably be replaced by the new "-ppx" technology (see 
below) and small extensions to the compilers.

A lot needs to happens for this camlp4-free OCaml ecosystem to become a 
reality and we have to come up with a solid transition plan.  A new 
community-driven working group, chaired by Leo White and me, is being 
set up today to elaborate this plan.  If this topic is of interest to 
you, please join our mailing list:

   http://lists.ocaml.org/listinfo/wg-camlp4


Some of the most important tasks for our new working group will be:

  - Gather information from the community about the use of camlp4.
    (Please consider sharing information about your use of camlp4
    and camlp4-based extensions, even if you don't plan to participate
    to the working group discussions!)

  - Finalize support for -ppx and make technical proposals for
    extensions of the OCaml compilers in order to enable a transition of
    camlp4-based extensions to -ppx (in particular, we need to come up
    with a concrete syntax for generic extension points in the grammar).

  - Write some "canonical" examples of extensions based on -ppx and
    provide information and support to developers of extensions for
    switching from camlp4 to -ppx.

  - Discuss integration of -ppx with existing tools (findlib, build
    systems, etc).

  - Find a solution in the community for the future of camlp4
    (in particular, discuss how / how long / by who it will be
    maintained).

  - Discuss longer-terms plans beyond -ppx, including extra language
    support, to facilitate light syntactic meta-programming for OCaml
    (Leo has some clever ideas!).



The discussion on the mailing list will start in a few days, to give 
some time for interested people to join.  In the meanwhile, Leo has 
written a blog post to get the discussion started:

  http://www.lpw25.net/2013/01/23/camlp4-alternative-part-1.html

You can also read about -ppx:

  http://www.lexifi.com/blog/syntax-extensions-without-camlp4
  http://www.lexifi.com/blog/syntax-extensions-without-camlp4-lets-do-it

Some projects have already started to replace camlp4 by -ppx:

   bisect (supports both camlp4 and ppx since version 1.3)
   sedlex (unicode-friendly lexer generator, successor of ulex)
   omonad (syntax for monadic code, similar to pa_monad)


Many thanks to Anil Madhavapeddy and to OCamlLabs for setting up the 
working group and its mailing list!


Alain

Re: [Caml-list] Working Group: the future of syntax extensions in OCaml, after camlp4

[Török Edwin]

On 01/24/2013 04:31 PM, Alain Frisch wrote:

> 
> The discussion on the mailing list will start in a few days, to give some time for interested people to join.  In the meanwhile, Leo has written a blog post to get the discussion started:
> 
>  http://www.lpw25.net/2013/01/23/camlp4-alternative-part-1.html
> 
> You can also read about -ppx:
> 
>  http://www.lexifi.com/blog/syntax-extensions-without-camlp4
>  http://www.lexifi.com/blog/syntax-extensions-without-camlp4-lets-do-it

Interesting posts, could these be made available in the ocaml.org/ocaml planet news feed too?

On 01/24/2013 04:31 PM, Alain Frisch wrote:
>    (Please consider sharing information about your use of camlp4
>    and camlp4-based extensions, even if you don't plan to participate
>    to the working group discussions!)

Will do.

Best regards,
--Edwin

Re: [Caml-list] Working Group: the future of syntax extensions in OCaml, after camlp4

[Ashish Agarwal]

[-- Attachment #1: Type: text/plain, Size: 312 bytes --]

On Thu, Jan 24, 2013 at 10:52 AM, Török Edwin <edwin+ml-ocaml@etorok.net>wrote:

Interesting posts, could these be made available in the
ocaml.org/ocamlplanet news feed too?
>

The feed aggregation is run by Sylvain. See his instructions here [1] for
adding a blog.
[1] http://www.ocamlcore.org/planet/

[-- Attachment #2: Type: text/html, Size: 734 bytes --]

Re: [Caml-list] Working Group: the future of syntax extensions in OCaml, after camlp4

[Daniel Bünzli]

Le jeudi, 24 janvier 2013 à 15:31, Alain Frisch a écrit :
> - Gather information from the community about the use of camlp4.
> (Please consider sharing information about your use of camlp4
> and camlp4-based extensions, even if you don't plan to participate
> to the working group discussions!)

I tried to disable camlp4 to compile OCaml 4.00.1 on a raspberry pi --- takes half the (long) compilation time, and allows you to compile OCaml without having to increase the swap size of the default raspbian image to prevent the system from crashing during camlp4 compilation...

However one thing I learned is that currently, it's pretty a no go as camlp4 is needed by ocamlfind which means that if you use something like opam to install software you can't install any of the 'required by' packages mentioned on this page [1].  

So it seems that to get rid of camlp4, ocamlfind needs to get rid of camlp4, I'd be interested in what Gerd has to say about that (I have no idea if camlp4 is used by the tool itself or if it's just needed to provide support for camlp4).  

Best,

Daniel

[1] http://opam.ocamlpro.com/pkg/ocamlfind.1.3.3.html

Re: [Caml-list] Working Group: the future of syntax extensions in OCaml, after camlp4

[Esther Baruk]

[-- Attachment #1: Type: text/plain, Size: 1419 bytes --]

Hi,

The blog is already added to the subscription of the news feed.
The news feed shows recent posts and these are old blog posts (2011) so
that's why they do not appear.

Esther Baruk


On Thu, Jan 24, 2013 at 4:52 PM, Török Edwin <edwin+ml-ocaml@etorok.net>wrote:

> On 01/24/2013 04:31 PM, Alain Frisch wrote:
>
> >
> > The discussion on the mailing list will start in a few days, to give
> some time for interested people to join.  In the meanwhile, Leo has written
> a blog post to get the discussion started:
> >
> >  http://www.lpw25.net/2013/01/23/camlp4-alternative-part-1.html
> >
> > You can also read about -ppx:
> >
> >  http://www.lexifi.com/blog/syntax-extensions-without-camlp4
> >  http://www.lexifi.com/blog/syntax-extensions-without-camlp4-lets-do-it
>
> Interesting posts, could these be made available in the ocaml.org/ocamlplanet news feed too?
>
> On 01/24/2013 04:31 PM, Alain Frisch wrote:
> >    (Please consider sharing information about your use of camlp4
> >    and camlp4-based extensions, even if you don't plan to participate
> >    to the working group discussions!)
>
> Will do.
>
> Best regards,
> --Edwin
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>

[-- Attachment #2: Type: text/html, Size: 2540 bytes --]

Re: [Caml-list] Working Group: the future of syntax extensions in OCaml, after camlp4

[rixed]

Are we interrested here striclty in _pre_processing or is runtime code
generation also on topic?

AW: [Caml-list] Working Group: the future of syntax extensions in OCaml, after camlp4

[Gerd Stolpmann]

Am 24.01.2013 16:57:37 schrieb(en) Daniel Bünzli:
> Le jeudi, 24 janvier 2013 à 15:31, Alain Frisch a écrit :
> > - Gather information from the community about the use of camlp4.
> > (Please consider sharing information about your use of camlp4
> > and camlp4-based extensions, even if you don't plan to participate
> > to the working group discussions!)
> 
> I tried to disable camlp4 to compile OCaml 4.00.1 on a raspberry pi  
> --- takes half the (long) compilation time, and allows you to compile  
> OCaml without having to increase the swap size of the default  
> raspbian image to prevent the system from crashing during camlp4  
> compilation...
> 
> However one thing I learned is that currently, it's pretty a no go as  
> camlp4 is needed by ocamlfind which means that if you use something  
> like opam to install software you can't install any of the 'required  
> by' packages mentioned on this page [1].
> 
> So it seems that to get rid of camlp4, ocamlfind needs to get rid of  
> camlp4, I'd be interested in what Gerd has to say about that (I have  
> no idea if camlp4 is used by the tool itself or if it's just needed  
> to provide support for camlp4).

It's used in the tool, but only for stream parsing. I could also  
distribute the already-preprocessed file (and maybe I'll do so in the  
next release).

Stream parsing is certainly one of the topics to discuss.

Just fyi, findlib was originally developed at a time when camlp4 was  
separately distributed, and stream parsing was supported by core OCaml  
w/o camlp4.

Gerd


> 
> Best,
> 
> Daniel
> 
> [1] http://opam.ocamlpro.com/pkg/ocamlfind.1.3.3.html
> 
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
> 



-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
Creator of GODI and camlcity.org.
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------

Re: [Caml-list] Working Group: the future of syntax extensions in OCaml, after camlp4

[Alain Frisch]

On 01/24/2013 05:16 PM, rixed@happyleptic.org wrote:
> Are we interrested here striclty in _pre_processing or is runtime code
> generation also on topic?

Runtime code generation, and meta-programming a la MetaOCaml are indeed 
quite a different story, and they are not specifically in the scope of 
the working group.  I can imagine that some outcomes of this 
de-camlp4-ifcation might benefit to such projects, though.  For 
instance, a more liberal concrete syntax (with attributes/quotations) 
might allow, say,  MetaOCaml, to use directly the official parser, thus 
avoiding problems related to the fact that it currently needs a custom 
parser. For instance, I guess that camlp4 extensions cannot be directly 
used by MetaOCaml users (except if someone decided to port Camlp4 to 
MetaOCaml).  Discussions related to these syntactic aspects are very 
much welcome in the new mailing list.


Alain

Re: [Caml-list] Working Group: the future of syntax extensions in OCaml, after camlp4

[Jacques Carette]

On 13-01-24 12:08 PM, Alain Frisch wrote:
> On 01/24/2013 05:16 PM, rixed@happyleptic.org wrote:
>> Are we interrested here striclty in _pre_processing or is runtime code
>> generation also on topic?
>
> Runtime code generation, and meta-programming a la MetaOCaml are 
> indeed quite a different story, and they are not specifically in the 
> scope of the working group.  I can imagine that some outcomes of this 
> de-camlp4-ifcation might benefit to such projects, though.  For 
> instance, a more liberal concrete syntax (with attributes/quotations) 
> might allow, say,  MetaOCaml, to use directly the official parser, 
> thus avoiding problems related to the fact that it currently needs a 
> custom parser. For instance, I guess that camlp4 extensions cannot be 
> directly used by MetaOCaml users (except if someone decided to port 
> Camlp4 to MetaOCaml). Discussions related to these syntactic aspects 
> are very much welcome in the new mailing list.

I was going to bring this up on the new mailing list, after waiting a 
bit for it to get populated.  I am glad that you intend to keep 
metaocaml-style metaprogramming as 'a different story', as it really is 
exactly that.

Note that the metocaml based on ocaml 3.09 did have a full camlp4 port 
(I did it), but it never worked for the 3.11 version (I tried and did 
not succeed).

Anyways, I've joined the new mailing list, and will participate in such 
a discussion over there.

Jacques

Re: AW: [Caml-list] Working Group: the future of syntax extensions in OCaml, after camlp4

[Alain Frisch]

On 01/24/2013 05:24 PM, Gerd Stolpmann wrote:
> It's used in the tool, but only for stream parsing. I could also
> distribute the already-preprocessed file (and maybe I'll do so in the
> next release).
>
> Stream parsing is certainly one of the topics to discuss.

I've no idea how widely stream parsing is used.  Has anyone some 
intuition about this?

Stream parsers probably fall in the same category as bitstring or sedlex 
(custom notions of pattern matching).  It seems that stream parsers 
(which I'm not familiar with) require to be able to write expressions 
within "left-hand sides", which might require special support.  Or maybe 
the whole left-hand sides should just be quotations.

Anyway, for a basic infrastructure tool such as ocamlfind, I'd probably 
advocate for a "manual" solution which works out of the box with a basic 
OCaml installation (ocamlyacc or manual top-down parser).  Gerd: does 
that sound reasonable to you?


Alain

Re: [wg-camlp4] [Caml-list] Working Group: the future of syntax extensions in OCaml, after camlp4

[Alain Frisch]

(Sorry for the noise, I did not realize that caml-list was still in Cc:. 
  Please continue the discussion on wg-camlp4 only!)


On 01/28/2013 01:15 PM, Alain Frisch wrote:
> On 01/24/2013 05:24 PM, Gerd Stolpmann wrote:
>> It's used in the tool, but only for stream parsing. I could also
>> distribute the already-preprocessed file (and maybe I'll do so in the
>> next release).
>>
>> Stream parsing is certainly one of the topics to discuss.
>
> I've no idea how widely stream parsing is used.  Has anyone some
> intuition about this?
>
> Stream parsers probably fall in the same category as bitstring or sedlex
> (custom notions of pattern matching).  It seems that stream parsers
> (which I'm not familiar with) require to be able to write expressions
> within "left-hand sides", which might require special support.  Or maybe
> the whole left-hand sides should just be quotations.
>
> Anyway, for a basic infrastructure tool such as ocamlfind, I'd probably
> advocate for a "manual" solution which works out of the box with a basic
> OCaml installation (ocamlyacc or manual top-down parser).  Gerd: does
> that sound reasonable to you?
>
>
> Alain
>
> _______________________________________________
> wg-camlp4 mailing list
> wg-camlp4@lists.ocaml.org
> http://lists.ocaml.org/listinfo/wg-camlp4

RE: AW: [Caml-list] Working Group: the future of syntax extensions in OCaml, after camlp4

[Jon Harrop]

I worked on a commercial project written in OCaml in Q4 2012 that used
stream parsing. However, my project was to translate the whole thing into
F#...

I never used stream parsing in any commercial code I wrote myself. I did use
camlp4 quite a bit though and, I must say, the only problem I had was that
it was never finished (the docs end in "..."!). Moreover, my main practical
application of camlp4 was in using it to write parsers. Parsers written
using Camlp4 are nicer than with any other tool I have ever used. Would be a
shame if OCaml lost this or, if it did, gained the ability to write lex/yacc
in-line without having to battle with multi-stage compilation and a wide
selection of incomplete/broken build tools.

Cheers,
Jon.

> -----Original Message-----
> From: caml-list-request@inria.fr [mailto:caml-list-request@inria.fr] On
Behalf
> Of Alain Frisch
> Sent: 28 January 2013 12:15
> To: Gerd Stolpmann
> Cc: Daniel Bünzli; caml-list; wg-camlp4@lists.ocaml.org; Leo P White; Anil
> Madhavapeddy
> Subject: Re: AW: [Caml-list] Working Group: the future of syntax
extensions in
> OCaml, after camlp4
> 
> On 01/24/2013 05:24 PM, Gerd Stolpmann wrote:
> > It's used in the tool, but only for stream parsing. I could also
> > distribute the already-preprocessed file (and maybe I'll do so in the
> > next release).
> >
> > Stream parsing is certainly one of the topics to discuss.
> 
> I've no idea how widely stream parsing is used.  Has anyone some intuition
> about this?
> 
> Stream parsers probably fall in the same category as bitstring or sedlex
(custom
> notions of pattern matching).  It seems that stream parsers (which I'm not
> familiar with) require to be able to write expressions within "left-hand
sides",
> which might require special support.  Or maybe the whole left-hand sides
should
> just be quotations.
> 
> Anyway, for a basic infrastructure tool such as ocamlfind, I'd probably
advocate
> for a "manual" solution which works out of the box with a basic OCaml
> installation (ocamlyacc or manual top-down parser).  Gerd: does that sound
> reasonable to you?
> 
> 
> Alain
> 
> 
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs

RE: AW: [Caml-list] Working Group: the future of syntax extensions in OCaml, after camlp4

[Simon Cruanes]

[-- Attachment #1: Type: text/plain, Size: 3089 bytes --]

I like the idea of inline lexer/parsers. A problem I have with ocamllex/ocamlyacc is the fact that circular dependencies prevent from using/exporting a parser for some type in the same module where this type is defined. Inline parsers would allow to keep the type and its parser(s) defined together. First-class parsers, even better, would be great for composition.

Cheers, 
-- 
Simon


Jon Harrop <jon@ffconsultancy.com> a écrit :

>
>I worked on a commercial project written in OCaml in Q4 2012 that used
>stream parsing. However, my project was to translate the whole thing
>into
>F#...
>
>I never used stream parsing in any commercial code I wrote myself. I
>did use
>camlp4 quite a bit though and, I must say, the only problem I had was
>that
>it was never finished (the docs end in "..."!). Moreover, my main
>practical
>application of camlp4 was in using it to write parsers. Parsers written
>using Camlp4 are nicer than with any other tool I have ever used. Would
>be a
>shame if OCaml lost this or, if it did, gained the ability to write
>lex/yacc
>in-line without having to battle with multi-stage compilation and a
>wide
>selection of incomplete/broken build tools.
>
>Cheers,
>Jon.
>
>> -----Original Message-----
>> From: caml-list-request@inria.fr [mailto:caml-list-request@inria.fr]
>On
>Behalf
>> Of Alain Frisch
>> Sent: 28 January 2013 12:15
>> To: Gerd Stolpmann
>> Cc: Daniel Bünzli; caml-list; wg-camlp4@lists.ocaml.org; Leo P White;
>Anil
>> Madhavapeddy
>> Subject: Re: AW: [Caml-list] Working Group: the future of syntax
>extensions in
>> OCaml, after camlp4
>> 
>> On 01/24/2013 05:24 PM, Gerd Stolpmann wrote:
>> > It's used in the tool, but only for stream parsing. I could also
>> > distribute the already-preprocessed file (and maybe I'll do so in
>the
>> > next release).
>> >
>> > Stream parsing is certainly one of the topics to discuss.
>> 
>> I've no idea how widely stream parsing is used.  Has anyone some
>intuition
>> about this?
>> 
>> Stream parsers probably fall in the same category as bitstring or
>sedlex
>(custom
>> notions of pattern matching).  It seems that stream parsers (which
>I'm not
>> familiar with) require to be able to write expressions within
>"left-hand
>sides",
>> which might require special support.  Or maybe the whole left-hand
>sides
>should
>> just be quotations.
>> 
>> Anyway, for a basic infrastructure tool such as ocamlfind, I'd
>probably
>advocate
>> for a "manual" solution which works out of the box with a basic OCaml
>> installation (ocamlyacc or manual top-down parser).  Gerd: does that
>sound
>> reasonable to you?
>> 
>> 
>> Alain
>> 
>> 
>> --
>> Caml-list mailing list.  Subscription management and archives:
>> https://sympa.inria.fr/sympa/arc/caml-list
>> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>> Bug reports: http://caml.inria.fr/bin/caml-bugs
>
>
>-- 
>Caml-list mailing list.  Subscription management and archives:
>https://sympa.inria.fr/sympa/arc/caml-list
>Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>Bug reports: http://caml.inria.fr/bin/caml-bugs

[-- Attachment #2: Type: text/html, Size: 4594 bytes --]