[Caml-list] Syntax extensions without Camlp4

[Caml-list] Syntax extensions without Camlp4

[Alexandre Pilkiewicz]

Hi,

I was surfing the web, and found this one year old blog post:
http://www.lexifi.com/blog/syntax-extensions-without-camlp4 . Are they
any news on that front? That seemed very promising!

Cheers,
Alex

[Caml-list] Re: Syntax extensions without Camlp4

[Hongbo Zhang]

On 5/27/12 11:06 AM, Alexandre Pilkiewicz wrote:
> Hi,
>
> I was surfing the web, and found this one year old blog post:
> http://www.lexifi.com/blog/syntax-extensions-without-camlp4 . Are they
> any news on that front? That seemed very promising!
>
Well, I don't think it's good to downplay camlp4. The problem is we need 
more documentation and more tests.
> Cheers,
> Alex
>

Re: [Caml-list] Re: Syntax extensions without Camlp4

[Daniel Bünzli]

Le dimanche, 27 mai 2012 à 18:53, Hongbo Zhang a écrit :

> Well, I don't think it's good to downplay camlp4. The problem is we need
> more documentation and more tests.


camlp4 is one of the (conceptually) ugliest component of the OCaml system. Pre-processors in general are a wrong solution to a real problem; meta-programming facilities should be part of the core language and play by its scoping rules, they should not be layered on top of it.  

The aformentioned proposal is certainly not the best I'm hoping for (the definition and semantics of the attributes is defined by the build system). However it may be a step in the right direction. For one thing by preventing the extension of OCaml's concrete syntax and making attributes part of it, using them doesn't break other tools that act on OCaml sources files; the first one being your syntax highlighter.  

Best,

Daniel

Re: [Caml-list] Re: Syntax extensions without Camlp4

[Hongbo Zhang]

[-- Attachment #1: Type: text/plain, Size: 1156 bytes --]

On 5/27/12 2:04 PM, Daniel Bünzli wrote:
>
> Le dimanche, 27 mai 2012 à 18:53, Hongbo Zhang a écrit :
>
>> Well, I don't think it's good to downplay camlp4. The problem is we need
>> more documentation and more tests.
>
> camlp4 is one of the (conceptually) ugliest component of the OCaml system. Pre-processors in general are a wrong solution to a real problem; meta-programming facilities should be part of the core language and play by its scoping rules, they should not be layered on top of it.
Come on, Parsetree.structure->Parsetree.structure does not solve the 
scope rule problem.
It's essentially a less powerful p4 tool and does not solve the problem 
at all. It just fragment the community again.

> The aformentioned proposal is certainly not the best I'm hoping for (the definition and semantics of the attributes is defined by the build system). However it may be a step in the right direction. For one thing by preventing the extension of OCaml's concrete syntax and making attributes part of it, using them doesn't break other tools that act on OCaml sources files; the first one being your syntax highlighter.
>
> Best,
>
> Daniel
>
>


[-- Attachment #2: Type: text/html, Size: 5717 bytes --]

[Caml-list] Re: Syntax extensions without Camlp4

[Hongbo Zhang]

On 5/27/12 2:04 PM, Daniel Bünzli wrote:
>
>
> Le dimanche, 27 mai 2012 à 18:53, Hongbo Zhang a écrit :
>
>> Well, I don't think it's good to downplay camlp4. The problem is we need
>> more documentation and more tests.
>
>
> camlp4 is one of the (conceptually) ugliest component of the OCaml system. Pre-processors in general are a wrong solution to a real problem; meta-programming facilities should be part of the core language and play by its scoping rules, they should not be layered on top of it.
>
> The aformentioned proposal is certainly not the best I'm hoping for (the definition and semantics of the attributes is defined by the build system). However it may be a step in the right direction. For one thing by preventing the extension of OCaml's concrete syntax and making attributes part of it, using them doesn't break other tools that act on OCaml sources files; the first one being your syntax highlighter.
>
> Best,
>
> Daniel
>
>
>
I agree that it would be nice if we have metaocaml in.

Re: [Caml-list] Re: Syntax extensions without Camlp4

[Daniel Bünzli]

Le dimanche, 27 mai 2012 à 20:18, Hongbo Zhang a écrit :
> Come on, Parsetree.structure -> Parsetree.structure does not solve the scope rule problem.


I didn't say it did.  

As I said in my message and in the comment of the blog post I'm not really fond of the proposal either but I still think it's a better thing than camlp4.

Regarding your comments about fragmentation, camlp4 is actually much more prone to lead to fragmentation and confusion (e.g. this revised syntax thing) than the proposed mechanism.

> I agree that it would be nice if we have metaocaml in.

Why not. But metaocaml is not about static meta-programming, it provides abstractions for runtime meta-programming.

Best,

Daniel

Re: [Caml-list] Re: Syntax extensions without Camlp4

[Wojciech Meyer]

Daniel Bünzli <daniel.buenzli@erratique.ch> writes:

>> I agree that it would be nice if we have metaocaml in.

Agreed.

> Why not. But metaocaml is not about static meta-programming, it provides abstractions for runtime meta-programming.

Runtime meta-programming is a generalisation of static meta
programming. MetaOCaml has a nice set of abstraction to generate
typechecking code - yes - either at runtime or during compile time. The
problem is that it's purely for partial evaluation and not extending the
syntax. Therefore the main purpose of syntactical abstraction is missing
- but that's not a problem - MetaOCaml wasn't designed for it.

Things that I would like to see in future "incarnation" or integrating of
meta programming facilities to the language  would be (beware that's my
blue dreams!):

- first of all non destructive updates to the grammar e.g: "let open
  lang Sexp in ..."  should open the Sexp syntax extension, install the
  grammar, but when it goes out of scope it should vanish. Currently
  Camlp4 can install, delete the rules after the functor is applied, and
  no way of saying OK - let's go back.

- Composable - in particular one language should behave like a module,
  or functor, should have an interface consisting of grammar rules, AST,
  AST transforms etc. So one could parametrise one syntax extension
  over another, and possibly reuse the language grammar or AST in
  other. Currently Camlp4 syntax extension is just a single separate
  module which when loaded possibly expects some existing grammar rules
  to be in place and mutates them as it's needed.

- should be type safe and as mentioned before obey scoping rules. We
  should be able to propagate type information even when the syntax
  changes. This is difficult part - but I've seen it can be done with some
  extra annotations - not talking about Camlp4

- Recursive - it should be able to apply the grammar rules not only
  once but expand until it reached the fixpoint.

- Reflective - it should be possible after each successful expansion have
  the type information available for the next expansion.

- Grammar itself should be lexer-less - memoizing PEG with left
  recursion - it's hurdle to define new grammar in terms of old lexer,
  or having a stateful lexer that depends on context.

- It should not be external tool - like previously observed - it's
  difficult to support for code highlighters or refactoring (tools in
  general) - if it depends on a build step or command line options.

That's said I find Camlp4 extremely useful for code generation purposes
- when I need to generate some ML code through quotations. Also, some
very important projects depend on Camlp4 (or Camlp5) like Coq. I don't
see that ML can live without some meta programming facilities out of the
box.

-- 
Wojciech Meyer
http://danmey.org

Re: [Caml-list] Re: Syntax extensions without Camlp4

[Paolo Donadeo]

[-- Attachment #1: Type: text/plain, Size: 1281 bytes --]

On Sun, May 27, 2012 at 8:04 PM, Daniel Bünzli <daniel.buenzli@erratique.ch>
wrote:

> camlp4 is one of the (conceptually) ugliest component of the OCaml system.
> Pre-processors in general are a wrong solution to a real problem;
> meta-programming facilities should be part of the core language and
> play by its scoping rules, they should not be layered on top of it.


I subscribe. At the present state OCaml, which is otherwise an almost
perfect language, needs some tool, I mean any tool, to accomplish
*simple*metaprogramming tasks.

Camlp4, which could be an awful or an amazing tool, is known by an inner
circle of wizards, and the activity of creating a new syntax extension is
a... sorcery for initiates only.

My experience of average and maybe dumb OCaml programmer: last week I tried
to write a simple ORM, something resembling the Django
ORM<https://docs.djangoproject.com/en/1.4/topics/db/models/>and, after
some (random) trial and error, I simply gave up. I have started
thinking to use cpp as a preprocessor to create very simple "templates",
which is awful, but the only alternative I have is to define the ORM
classes in something like XML, parse them and generate OCaml sources.

It's definitively a lose-lose game.


-- 
*Paolo*

[-- Attachment #2: Type: text/html, Size: 1572 bytes --]

Re: [Caml-list] Re: Syntax extensions without Camlp4

[Daniel Bünzli]

Le lundi, 28 mai 2012 à 00:43, Wojciech Meyer a écrit :

> Runtime meta-programming is a generalisation of static meta
> programming. MetaOCaml has a nice set of abstraction to generate
> typechecking code - yes - either at runtime or during compile time.


You meant 'typechecked' (?). It's obviously a generalization but I didn't know there was support to use it at compile time.  
  
> The
> problem is that it's purely for partial evaluation and not extending the
> syntax.  


Then it's perfect ! I think it's wrong to try to extend the language per se. Most of the time, except for very particular things (e.g. introducing a monad notation), the dsl approach is perfectly sufficient. Don't think you absolutely need to extend the OCaml grammar, embed your dsl directly into OCaml, using OCaml language binders if you need variables.  

Make libraries, not pet syntactic constructs.  

Many things that can be done with camlp4, can be done with that approach. Not only is it very elegant, it's much easier to maintain w.r.t. the evolution of the OCaml language itself. The techniques in these papers [1] should be more known and used.

> - It should not be external tool - like previously observed - it's
> difficult to support for code highlighters or refactoring (tools in
> general) - if it depends on a build step or command line options.




If you extend the grammar itself, code highlighters or any tool expecting an OCaml expression is going to break whether the tool is external or not. But for the rest of your comments I agree wholeheartedly (even though I'm not sure all that power is needed, but at least it would make the tool non-ugly to me).

Best,

Daniel

[1]  
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.31.9782&rep=rep1&type=pdf
http://research.microsoft.com/apps/pubs/default.aspx?id=67334




  
  
> Therefore the main purpose of syntactical abstraction is missing
> - but that's not a problem - MetaOCaml wasn't designed for it.
>  
> Things that I would like to see in future "incarnation" or integrating of
> meta programming facilities to the language would be (beware that's my
> blue dreams!):
>  
> - first of all non destructive updates to the grammar e.g: "let open
> lang Sexp in ..." should open the Sexp syntax extension, install the
> grammar, but when it goes out of scope it should vanish. Currently
> Camlp4 can install, delete the rules after the functor is applied, and
> no way of saying OK - let's go back.
>  
> - Composable - in particular one language should behave like a module,
> or functor, should have an interface consisting of grammar rules, AST,
> AST transforms etc. So one could parametrise one syntax extension
> over another, and possibly reuse the language grammar or AST in
> other. Currently Camlp4 syntax extension is just a single separate
> module which when loaded possibly expects some existing grammar rules
> to be in place and mutates them as it's needed.
>  
> - should be type safe and as mentioned before obey scoping rules. We
> should be able to propagate type information even when the syntax
> changes. This is difficult part - but I've seen it can be done with some
> extra annotations - not talking about Camlp4
>  
> - Recursive - it should be able to apply the grammar rules not only
> once but expand until it reached the fixpoint.
>  
> - Reflective - it should be possible after each successful expansion have
> the type information available for the next expansion.
>  
> - Grammar itself should be lexer-less - memoizing PEG with left
> recursion - it's hurdle to define new grammar in terms of old lexer,
> or having a stateful lexer that depends on context.
>  
> - It should not be external tool - like previously observed - it's
> difficult to support for code highlighters or refactoring (tools in
> general) - if it depends on a build step or command line options.
>  
> That's said I find Camlp4 extremely useful for code generation purposes
> - when I need to generate some ML code through quotations. Also, some
> very important projects depend on Camlp4 (or Camlp5) like Coq. I don't
> see that ML can live without some meta programming facilities out of the
> box.
>  
> --  
> Wojciech Meyer
> http://danmey.org
>  
> --  
> Caml-list mailing list. Subscription management and archives:
> https://sympa-roc.inria.fr/wws/info/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs

Re: [Caml-list] Re: Syntax extensions without Camlp4

[Gabriel Scherer]

[-- Attachment #1: Type: text/plain, Size: 7015 bytes --]

As this seems to be the kind of "christmas shopping list" discussion where
everyone throws something in, here is my opinion on camlp4 evolution:
- we should forget about the general problem of extending syntax rules;
it's too complex, doesn't compose well, and is only of arguable benefit
- we should focus on blessing specific *fixed* syntactical extensibility
points and making them enter the standard syntax for the language; there
are two elephants in the rooms: quotations and type-conv-like annotations.

Quotations are useful, used in practice (not always under the
camlp4-blessed form), simple to define, understand and use, and compose
extremely well. One problem with current camlp4 quotations is that they are
a bit lexically heavy (<:name< ... >>) and inflexible (you basically can't
use >> inside the quotation); users have tried to workaround this by
re-coding quotations, eg. for smart string literals u"blah" or regexp
syntactic sugar s/foo/bar/. I think we could consider having lexing-level
extensibility (have a tool to modify lexing rule instead of grammar rules
like Camlp4; hopefully lexing rules are simpler and compose better).

Type-conv like annotations are a bit less well-defined, and would need some
design work: what exactly is their intended scope/expressivity? Are they a
general annotation mechanism, a rigid way to just add new phrases after
each annotation phrase, or something in between?

Other point of extensibility could be considered (eg. Jeremy Yallop's work
on pattern-matching extensibility), but only added if they are simple
enough, useful enough, and compose well.

I would still welcome having central, well-supported libraries to
manipulate Ocaml syntactic AST and, why not, typed AST (typedtree). Among
its many features, Camlp4 currently provides the former (in a relatively
convenient way thanks to its AST quotations), and this is quite useful.

On Mon, May 28, 2012 at 11:35 AM, Daniel Bünzli <daniel.buenzli@erratique.ch
> wrote:

>
>
> Le lundi, 28 mai 2012 à 00:43, Wojciech Meyer a écrit :
>
> > Runtime meta-programming is a generalisation of static meta
> > programming. MetaOCaml has a nice set of abstraction to generate
> > typechecking code - yes - either at runtime or during compile time.
>
>
> You meant 'typechecked' (?). It's obviously a generalization but I didn't
> know there was support to use it at compile time.
>
> > The
> > problem is that it's purely for partial evaluation and not extending the
> > syntax.
>
>
> Then it's perfect ! I think it's wrong to try to extend the language per
> se. Most of the time, except for very particular things (e.g. introducing a
> monad notation), the dsl approach is perfectly sufficient. Don't think you
> absolutely need to extend the OCaml grammar, embed your dsl directly into
> OCaml, using OCaml language binders if you need variables.
>
> Make libraries, not pet syntactic constructs.
>
> Many things that can be done with camlp4, can be done with that approach.
> Not only is it very elegant, it's much easier to maintain w.r.t. the
> evolution of the OCaml language itself. The techniques in these papers [1]
> should be more known and used.
>
> > - It should not be external tool - like previously observed - it's
> > difficult to support for code highlighters or refactoring (tools in
> > general) - if it depends on a build step or command line options.
>
>
>
>
> If you extend the grammar itself, code highlighters or any tool expecting
> an OCaml expression is going to break whether the tool is external or not.
> But for the rest of your comments I agree wholeheartedly (even though I'm
> not sure all that power is needed, but at least it would make the tool
> non-ugly to me).
>
> Best,
>
> Daniel
>
> [1]
>
> http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.31.9782&rep=rep1&type=pdf
> http://research.microsoft.com/apps/pubs/default.aspx?id=67334
>
>
>
>
>
>
> > Therefore the main purpose of syntactical abstraction is missing
> > - but that's not a problem - MetaOCaml wasn't designed for it.
> >
> > Things that I would like to see in future "incarnation" or integrating of
> > meta programming facilities to the language would be (beware that's my
> > blue dreams!):
> >
> > - first of all non destructive updates to the grammar e.g: "let open
> > lang Sexp in ..." should open the Sexp syntax extension, install the
> > grammar, but when it goes out of scope it should vanish. Currently
> > Camlp4 can install, delete the rules after the functor is applied, and
> > no way of saying OK - let's go back.
> >
> > - Composable - in particular one language should behave like a module,
> > or functor, should have an interface consisting of grammar rules, AST,
> > AST transforms etc. So one could parametrise one syntax extension
> > over another, and possibly reuse the language grammar or AST in
> > other. Currently Camlp4 syntax extension is just a single separate
> > module which when loaded possibly expects some existing grammar rules
> > to be in place and mutates them as it's needed.
> >
> > - should be type safe and as mentioned before obey scoping rules. We
> > should be able to propagate type information even when the syntax
> > changes. This is difficult part - but I've seen it can be done with some
> > extra annotations - not talking about Camlp4
> >
> > - Recursive - it should be able to apply the grammar rules not only
> > once but expand until it reached the fixpoint.
> >
> > - Reflective - it should be possible after each successful expansion have
> > the type information available for the next expansion.
> >
> > - Grammar itself should be lexer-less - memoizing PEG with left
> > recursion - it's hurdle to define new grammar in terms of old lexer,
> > or having a stateful lexer that depends on context.
> >
> > - It should not be external tool - like previously observed - it's
> > difficult to support for code highlighters or refactoring (tools in
> > general) - if it depends on a build step or command line options.
> >
> > That's said I find Camlp4 extremely useful for code generation purposes
> > - when I need to generate some ML code through quotations. Also, some
> > very important projects depend on Camlp4 (or Camlp5) like Coq. I don't
> > see that ML can live without some meta programming facilities out of the
> > box.
> >
> > --
> > Wojciech Meyer
> > http://danmey.org
> >
> > --
> > Caml-list mailing list. Subscription management and archives:
> > https://sympa-roc.inria.fr/wws/info/caml-list
> > Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> > Bug reports: http://caml.inria.fr/bin/caml-bugs
>
>
>
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa-roc.inria.fr/wws/info/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>
>

[-- Attachment #2: Type: text/html, Size: 8680 bytes --]

Re: [Caml-list] Re: Syntax extensions without Camlp4

[Wojciech Meyer]

On Mon, May 28, 2012 at 10:35 AM, Daniel Bünzli
<daniel.buenzli@erratique.ch> wrote:

> You meant 'typechecked' (?). It's obviously a generalization but I didn't know there was support to use it at compile time.

I meant with small amendments you can make MetaOCaml to work at compile
time, one would need just special "force" construct to generate the code
as usually at runtime but during compilation and assign the resulting
code to the let bind.

>
>> The
>> problem is that it's purely for partial evaluation and not extending the
>> syntax.
>
>
> Then it's perfect ! I think it's wrong to try to extend the language per se. Most of the time, except for very particular things (e.g. introducing a monad notation), the dsl approach is perfectly sufficient. Don't think you absolutely need to extend the OCaml grammar, embed your dsl directly into OCaml, using OCaml language binders if you need variables.

I agree, that using combinators and run time meta programming you can do
it without performance hit because you can partially evaluate your
combinators.

>
> Make libraries, not pet syntactic constructs.

Well, I agree and disagree at the same time. Some of the stuff can't be
represented without extending the syntax. Introspecting types
(type-conv) and generating code, bitstring matcher, monadic programming
extension etc. there is a very fine grained line what can be done with
combinators and what can't be. In particular, syntax extensions give the
developer way to abstract things at the syntactic level, and they lift
restriction that usual forms (like let, if, fun etc.) are not first class
citizens. They also allow to change the evaulation order if it's
needed. How much of this power is useful for daily programming - I don't
know, but certainly having a better tool to the job would be great.

It's just the existing tools make it ugly to use and degradate them to
"pet syntactic constructs".

>
> Many things that can be done with camlp4, can be done with that approach. Not only is it very elegant, it's much easier to maintain w.r.t. the evolution of the OCaml language itself. The techniques in these papers [1] should be more known and used.

Thanks for the links, certainly I will read them.

> If you extend the grammar itself, code highlighters or any tool expecting an OCaml expression is going to break whether the tool is external or not. But for the rest of your comments I agree wholeheartedly (even though I'm not sure all that power is needed, but at least it would make the tool non-ugly to me).

Of course you can, using the approach taken by Typerex server, the
difference is of course that then Typerex does not need any build step,
because the extension is self contained in the ML file.

- Wojciech

[Caml-list] Re: Syntax extensions without Camlp4

[Jeffrey Scofield]

Daniel Bünzli <daniel.buenzli@erratique.ch> writes:

> > The problem is that it's purely for partial evaluation and not
> >  extending the syntax.
> 
> Then it's perfect ! I think it's wrong to try to extend the language
> per se. Most of the time, except for very particular things
> (e.g. introducing a monad notation), the dsl approach is perfectly
> sufficient. Don't think you absolutely need to extend the OCaml
> grammar, embed your dsl directly into OCaml, using OCaml language
> binders if you need variables.
> 
> Make libraries, not pet syntactic constructs.

For what it's worth, I heartily endorse this approach.  Use of campl4
when something simpler will do is a real burden on people learning
the language and people combining modules.  OCaml has already
fragmented into two different syntaxes, and I've already chosen to
hack several different modules for my own use by stripping out all the
syntax extensions.  In the long run this doesn't scale very well
(in my opinion).

The baseline for OCaml is already a large (rich, complex, amazing)
language.  But it's hard to pitch it to potential adopters if it's
also a rapidly moving and ramifying target.

(I should say that the syntactic flexibility is fantastic for language
researchers, it's users who are solving practical problems with the
language that I'm talking about.  I would put myself in this
category.)

Just my opinion.

Jeffrey Scofield
Seattle

Re: [Caml-list] Re: Syntax extensions without Camlp4

[Alain Frisch]

On 05/27/2012 06:53 PM, Hongbo Zhang wrote:
> Well, I don't think it's good to downplay camlp4. The problem is we need
> more documentation and more tests.

(The new) Camlp4 has been here for several years.  Documentation and 
tests are still lagging behind.  Nevertheless, it burns a non-negligible 
fraction of the total resource spent by maintainers on OCaml (fixing 
bugs, struggling with camlp4 to take new language features into account).

In addition, there is a growing consensus that the most common uses of 
camlp4 (such as code generation driven by type declaration) might be 
based on a much simpler approach, and this would actually have 
advantages for the end-users (like not changing the concrete syntax)
and for developers (much less information to grasp in order to write 
such an extension).

Coq is one of the few examples of a big project that relies on other 
aspects of camlp4 (e.g. its parsing framework with extensible grammars). 
  As far as I know, it also still works with camlp5, which is actively 
maintained, and I don't believe the Coq guys enjoy so much maintaining 
support for both camlp4 and campl5.

All that considered, and this is only a personal opinion: I don't see 
compelling arguments to continue investing efforts in camlp4 itself (at 
least, for the core OCaml team) and I believe it is a good time to start 
considering a (medium-term) future of OCaml without camlp4.  Of course, 
alternative solutions need to be developed and streamlined before 
killing camlp4 is even considered.


Alain

Re: [Caml-list] Re: Syntax extensions without Camlp4

[Markus Mottl]

On Wed, May 30, 2012 at 8:41 AM, Alain Frisch <alain@frisch.fr> wrote:
> In addition, there is a growing consensus that the most common uses of
> camlp4 (such as code generation driven by type declaration) might be based
> on a much simpler approach, and this would actually have advantages for the
> end-users (like not changing the concrete syntax)
> and for developers (much less information to grasp in order to write such an
> extension).

I agree that most OCaml developers only need a tiny subset of the
powerful but also overly complex features provided by camlp4.  These
features could surely be provided in a much simpler framework.  Making
camlp4 an external tool for heavy-duty grammar modifications would
make sense to me.  I'd rather see a well-supported light-weight
solution with AST attributes.  Something like MetaOCaml for (runtime)
code generation would seem exciting, too.

Regards,
Markus

-- 
Markus Mottl        http://www.ocaml.info        markus.mottl@gmail.com

Re: [Caml-list] Re: Syntax extensions without Camlp4

[Dan Bensen]

[-- Attachment #1: Type: text/plain, Size: 660 bytes --]


> > I don't think it's good to downplay camlp4.
> > we need more documentation and more tests.

> (The new) Camlp4 has been here for several years. 
> Documentation and tests are still lagging behind. 

Is it possible for Inria or someone else to provide 
funding for that?  I would like to help out starting 
sometime in the summer.

> the most common uses of camlp4 might be based on a much
> simpler approach ... Coq is one of the few examples of
> a big project that relies on other aspects of camlp4

Why not focus on optimizing new lightweight tools for 
small problems and keep camlp4 for big ones?  It wouldn't 
be so bad if it were documented better.

[-- Attachment #2: Type: text/html, Size: 946 bytes --]

Re: [Caml-list] Re: Syntax extensions without Camlp4

[Hongbo Zhang]

[-- Attachment #1: Type: text/plain, Size: 4182 bytes --]

On 5/30/12 8:41 AM, Alain Frisch wrote:
Hi Alain,
    Thanks for your message.
    I am building a framework on top of camlp4, which provides those 
features you mentioned in the post in a flexible way. It needs to be 
polished. So far, writing generic plugins is really easy. (generic 
printing, comparison, lift filter, etc , each about 20 lines).
    The conclusion below is based on my observation and experiences. 
Feel free to correct me if I am wrong.

    1. Why camlp4 is buggy?
    The main buggy part is its parsing technology.  So is its parsing 
technology a bad idea? No, it's really convenient for rapid ad-hoc 
prototyping, suppose you don't need write a lexer, and its parser simply 
works in the toplevel, the parser itself it really shorter even compared 
with menhir, if you refactor your parser part.
    The main buggy part lies in camlp4of, which is not tested at all. 
it's easy to test camlp4o, camlp4rf. There's large code base written 
both in camlp4o and camlp4rf. So in my opinion, we should just *drop 
camlp4of*, unless it's really heavily tested. *camlp4rf* is a good 
syntax, in my experience, it just took me one day to get used to, and I 
wrote my daily caml code in revised syntax. It's not hard to learn. It's 
much easier compared with the complexity of camlp4ast or ocamlast. So, 
IMHO, we should just advise camlp4 developers to use revised syntax. It 
would be nice if Daniel would write some articles how internal parser 
mechanism works. Revised syntax just catch up with the trunk, it should 
not be exactly the same, this gives the caml develop team enough freedom 
to introducing new constructs(monad support, etc)

   2. About the proposal.
     There are mainly 2 pieces. About the hook 
Parsetree.structure->Parsetree. structure, given that camlp4 already 
imports Parsetree, it's really trivial to
add another hook after camlp4astdump2ocamlast.
     The other piece about introducing attribute grammar, whcich is 
orthogonal to camlp4, IMHO.
     It's still nontrivial to write a robust Parsetree.structure -> 
Parsetree.structure,  it would be nice if we could provide a quotation 
syntax for Parsetree.types. I would contribute if there was some funding 
support.

  3. Extending the syntax.
     It is really nice to extend your syntax for ad-hoc hacking. Your 
nice software ulex, and bitstring is a good example. It could be even 
better to just introduce a new quotation for robustness.

     But I am against mutating syntax for personal taste. (try finally, 
monad syntax), those improvement should be in the trunk.

Many Thanks
> On 05/27/2012 06:53 PM, Hongbo Zhang wrote:
>> Well, I don't think it's good to downplay camlp4. The problem is we need
>> more documentation and more tests.
>
> (The new) Camlp4 has been here for several years.  Documentation and 
> tests are still lagging behind.  Nevertheless, it burns a 
> non-negligible fraction of the total resource spent by maintainers on 
> OCaml (fixing bugs, struggling with camlp4 to take new language 
> features into account).
>
> In addition, there is a growing consensus that the most common uses of 
> camlp4 (such as code generation driven by type declaration) might be 
> based on a much simpler approach, and this would actually have 
> advantages for the end-users (like not changing the concrete syntax)
> and for developers (much less information to grasp in order to write 
> such an extension).
>
> Coq is one of the few examples of a big project that relies on other 
> aspects of camlp4 (e.g. its parsing framework with extensible 
> grammars).  As far as I know, it also still works with camlp5, which 
> is actively maintained, and I don't believe the Coq guys enjoy so much 
> maintaining support for both camlp4 and campl5.
>
> All that considered, and this is only a personal opinion: I don't see 
> compelling arguments to continue investing efforts in camlp4 itself 
> (at least, for the core OCaml team) and I believe it is a good time to 
> start considering a (medium-term) future of OCaml without camlp4.  Of 
> course, alternative solutions need to be developed and streamlined 
> before killing camlp4 is even considered.
>
>
> Alain


[-- Attachment #2: Type: text/html, Size: 9454 bytes --]

Re: [Caml-list] Re: Syntax extensions without Camlp4

[Hongbo Zhang]

[-- Attachment #1: Type: text/plain, Size: 960 bytes --]

On 5/30/12 9:37 AM, Dan Bensen wrote:
>
> > > I don't think it's good to downplay camlp4.
> > > we need more documentation and more tests.
>
> > (The new) Camlp4 has been here for several years.
> > Documentation and tests are still lagging behind.
>
> Is it possible for Inria or someone else to provide
> funding for that?  I would like to help out starting
> sometime in the summer.
>
Actually I am writing a book about the internal part of camlp4 (100~200 
pages),
but nobody seems to care about it.
> > the most common uses of camlp4 might be based on a much
> > simpler approach ... Coq is one of the few examples of
> > a big project that relies on other aspects of camlp4
>
> Why not focus on optimizing new lightweight tools for
> small problems and keep camlp4 for big ones?  It wouldn't
> be so bad if it were documented better.
I think it's a good idea to build simple tools on top of existing more 
powerful tools instead of dropping the latter.


[-- Attachment #2: Type: text/html, Size: 1949 bytes --]

Re: [Caml-list] Re: Syntax extensions without Camlp4

[Paolo Donadeo]

[-- Attachment #1: Type: text/plain, Size: 246 bytes --]

On Wed, May 30, 2012 at 4:16 PM, Hongbo Zhang <bobzhang1988@gmail.com>wrote:
>
>  Actually I am writing a book about the internal part of camlp4 (100~200
> pages),
> but nobody seems to care about it.
>

I didn't know. Where is it?


-- 
*Paolo*

[-- Attachment #2: Type: text/html, Size: 607 bytes --]

[Caml-list] Re: Syntax extensions without Camlp4

[Hongbo Zhang]

On 5/28/12 5:59 AM, Gabriel Scherer wrote:
Hi,
    It would be cool to combine camlp4ast and typedtree to generate more 
meaningful code. one typical examples is tracer. did any one give a try?
> As this seems to be the kind of "christmas shopping list" discussion
> where everyone throws something in, here is my opinion on camlp4 evolution:
> - we should forget about the general problem of extending syntax rules;
> it's too complex, doesn't compose well, and is only of arguable benefit
> - we should focus on blessing specific *fixed* syntactical extensibility
> points and making them enter the standard syntax for the language; there
> are two elephants in the rooms: quotations and type-conv-like annotations.
>
> Quotations are useful, used in practice (not always under the
> camlp4-blessed form), simple to define, understand and use, and compose
> extremely well. One problem with current camlp4 quotations is that they
> are a bit lexically heavy (<:name< ... >>) and inflexible (you basically
> can't use >> inside the quotation); users have tried to workaround this
> by re-coding quotations, eg. for smart string literals u"blah" or regexp
> syntactic sugar s/foo/bar/. I think we could consider having
> lexing-level extensibility (have a tool to modify lexing rule instead of
> grammar rules like Camlp4; hopefully lexing rules are simpler and
> compose better).
>
> Type-conv like annotations are a bit less well-defined, and would need
> some design work: what exactly is their intended scope/expressivity? Are
> they a general annotation mechanism, a rigid way to just add new phrases
> after each annotation phrase, or something in between?
>
> Other point of extensibility could be considered (eg. Jeremy Yallop's
> work on pattern-matching extensibility), but only added if they are
> simple enough, useful enough, and compose well.
>
> I would still welcome having central, well-supported libraries to
> manipulate Ocaml syntactic AST and, why not, typed AST (typedtree).
> Among its many features, Camlp4 currently provides the former (in a
> relatively convenient way thanks to its AST quotations), and this is
> quite useful.
>
> On Mon, May 28, 2012 at 11:35 AM, Daniel Bünzli
> <daniel.buenzli@erratique.ch <mailto:daniel.buenzli@erratique.ch>> wrote:
>
>
>
>     Le lundi, 28 mai 2012 à 00:43, Wojciech Meyer a écrit :
>
>      > Runtime meta-programming is a generalisation of static meta
>      > programming. MetaOCaml has a nice set of abstraction to generate
>      > typechecking code - yes - either at runtime or during compile time.
>
>
>     You meant 'typechecked' (?). It's obviously a generalization but I
>     didn't know there was support to use it at compile time.
>
>      > The
>      > problem is that it's purely for partial evaluation and not
>     extending the
>      > syntax.
>
>
>     Then it's perfect ! I think it's wrong to try to extend the language
>     per se. Most of the time, except for very particular things (e.g.
>     introducing a monad notation), the dsl approach is perfectly
>     sufficient. Don't think you absolutely need to extend the OCaml
>     grammar, embed your dsl directly into OCaml, using OCaml language
>     binders if you need variables.
>
>     Make libraries, not pet syntactic constructs.
>
>     Many things that can be done with camlp4, can be done with that
>     approach. Not only is it very elegant, it's much easier to maintain
>     w.r.t. the evolution of the OCaml language itself. The techniques in
>     these papers [1] should be more known and used.
>
>      > - It should not be external tool - like previously observed - it's
>      > difficult to support for code highlighters or refactoring (tools in
>      > general) - if it depends on a build step or command line options.
>
>
>
>
>     If you extend the grammar itself, code highlighters or any tool
>     expecting an OCaml expression is going to break whether the tool is
>     external or not. But for the rest of your comments I agree
>     wholeheartedly (even though I'm not sure all that power is needed,
>     but at least it would make the tool non-ugly to me).
>
>     Best,
>
>     Daniel
>
>     [1]
>     http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.31.9782&rep=rep1&type=pdf
>     <http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.31.9782&rep=rep1&type=pdf>
>     http://research.microsoft.com/apps/pubs/default.aspx?id=67334
>
>
>
>
>
>
>      > Therefore the main purpose of syntactical abstraction is missing
>      > - but that's not a problem - MetaOCaml wasn't designed for it.
>      >
>      > Things that I would like to see in future "incarnation" or
>     integrating of
>      > meta programming facilities to the language would be (beware
>     that's my
>      > blue dreams!):
>      >
>      > - first of all non destructive updates to the grammar e.g: "let open
>      > lang Sexp in ..." should open the Sexp syntax extension, install the
>      > grammar, but when it goes out of scope it should vanish. Currently
>      > Camlp4 can install, delete the rules after the functor is
>     applied, and
>      > no way of saying OK - let's go back.
>      >
>      > - Composable - in particular one language should behave like a
>     module,
>      > or functor, should have an interface consisting of grammar rules,
>     AST,
>      > AST transforms etc. So one could parametrise one syntax extension
>      > over another, and possibly reuse the language grammar or AST in
>      > other. Currently Camlp4 syntax extension is just a single separate
>      > module which when loaded possibly expects some existing grammar rules
>      > to be in place and mutates them as it's needed.
>      >
>      > - should be type safe and as mentioned before obey scoping rules. We
>      > should be able to propagate type information even when the syntax
>      > changes. This is difficult part - but I've seen it can be done
>     with some
>      > extra annotations - not talking about Camlp4
>      >
>      > - Recursive - it should be able to apply the grammar rules not only
>      > once but expand until it reached the fixpoint.
>      >
>      > - Reflective - it should be possible after each successful
>     expansion have
>      > the type information available for the next expansion.
>      >
>      > - Grammar itself should be lexer-less - memoizing PEG with left
>      > recursion - it's hurdle to define new grammar in terms of old lexer,
>      > or having a stateful lexer that depends on context.
>      >
>      > - It should not be external tool - like previously observed - it's
>      > difficult to support for code highlighters or refactoring (tools in
>      > general) - if it depends on a build step or command line options.
>      >
>      > That's said I find Camlp4 extremely useful for code generation
>     purposes
>      > - when I need to generate some ML code through quotations. Also, some
>      > very important projects depend on Camlp4 (or Camlp5) like Coq. I
>     don't
>      > see that ML can live without some meta programming facilities out
>     of the
>      > box.
>      >
>      > --
>      > Wojciech Meyer
>      > http://danmey.org
>      >
>      > --
>      > Caml-list mailing list. Subscription management and archives:
>      > https://sympa-roc.inria.fr/wws/info/caml-list
>      > Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>      > Bug reports: http://caml.inria.fr/bin/caml-bugs
>
>
>
>
>     --
>     Caml-list mailing list.  Subscription management and archives:
>     https://sympa-roc.inria.fr/wws/info/caml-list
>     Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>     Bug reports: http://caml.inria.fr/bin/caml-bugs
>
>

Re: [Caml-list] Re: Syntax extensions without Camlp4

[Paolo Donadeo]

[-- Attachment #1: Type: text/plain, Size: 484 bytes --]

On Thu, May 31, 2012 at 10:19 AM, <rixed@happyleptic.org> wrote:
>
>  > I didn't know. Where is it?
> https://github.com/bobzhang/ocaml-book


Thanks, just cloned :-)

Out of curiosity, does anyone have news about Real World
OCaml<http://realworldocaml.org/>,
by Jason Hickey, Anil Madhavapeddy and Yaron Minsky, that is supposed to be
available this Autumn? The Twitter
channel<https://twitter.com/#!/realworldocaml>is threateningly silent:
is the project still alive?


-- 
*Paolo*

[-- Attachment #2: Type: text/html, Size: 941 bytes --]

Re: [Caml-list] Re: Syntax extensions without Camlp4

[Anil Madhavapeddy]

On Thu, May 31, 2012 at 02:26:18PM +0200, Paolo Donadeo wrote:
> On Thu, May 31, 2012 at 10:19 AM, <rixed@happyleptic.org> wrote:
> >
> >  > I didn't know. Where is it?
> > https://github.com/bobzhang/ocaml-book
> 
> 
> Thanks, just cloned :-)
> 
> Out of curiosity, does anyone have news about Real World
> OCaml<http://realworldocaml.org/>,
> by Jason Hickey, Anil Madhavapeddy and Yaron Minsky, that is supposed to be
> available this Autumn? The Twitter
> channel<https://twitter.com/#!/realworldocaml>is threateningly silent:
> is the project still alive?

That's an interesting leap of logic. It'll be available in Autumn, so you
assume the project is dead in July? :-)

Yes, we're working away on it.  The book will also be freely available
under a CC license, so don't worry: you'll have ample time to read it and
comment on the beta before it goes into 1st edition print.  We'll talk
more about the content after the first snapshot is out after the summer.

-anil

Re: [Caml-list] Re: Syntax extensions without Camlp4

[Anil Madhavapeddy]

On Thu, May 31, 2012 at 01:38:02PM +0100, Anil Madhavapeddy wrote:
> On Thu, May 31, 2012 at 02:26:18PM +0200, Paolo Donadeo wrote:
> > On Thu, May 31, 2012 at 10:19 AM, <rixed@happyleptic.org> wrote:
> > >
> > >  > I didn't know. Where is it?
> > > https://github.com/bobzhang/ocaml-book
> > 
> > 
> > Thanks, just cloned :-)
> > 
> > Out of curiosity, does anyone have news about Real World
> > OCaml<http://realworldocaml.org/>,
> > by Jason Hickey, Anil Madhavapeddy and Yaron Minsky, that is supposed to be
> > available this Autumn? The Twitter
> > channel<https://twitter.com/#!/realworldocaml>is threateningly silent:
> > is the project still alive?
> 
> That's an interesting leap of logic. It'll be available in Autumn, so you
> assume the project is dead in July? :-)
> 
> Yes, we're working away on it.  The book will also be freely available
> under a CC license, so don't worry: you'll have ample time to read it and
> comment on the beta before it goes into 1st edition print.  We'll talk
> more about the content after the first snapshot is out after the summer.

To clarify further, the book will not going into print before the end of
the year.  "Autumn" refers to when the online snapshots will be
available for feedback and editorial review.

-- 
Anil Madhavapeddy                                 http://anil.recoil.org

Re: [Caml-list] Re: Syntax extensions without Camlp4

[Yaron Minsky]

[-- Attachment #1: Type: text/plain, Size: 1843 bytes --]

It's definitely not dead!  I work on it a bit every day.  I'm happily
hacking away on the first-class modules section today...

y

On Thu, May 31, 2012 at 8:40 AM, Anil Madhavapeddy <anil@recoil.org> wrote:

> On Thu, May 31, 2012 at 01:38:02PM +0100, Anil Madhavapeddy wrote:
> > On Thu, May 31, 2012 at 02:26:18PM +0200, Paolo Donadeo wrote:
> > > On Thu, May 31, 2012 at 10:19 AM, <rixed@happyleptic.org> wrote:
> > > >
> > > >  > I didn't know. Where is it?
> > > > https://github.com/bobzhang/ocaml-book
> > >
> > >
> > > Thanks, just cloned :-)
> > >
> > > Out of curiosity, does anyone have news about Real World
> > > OCaml<http://realworldocaml.org/>,
> > > by Jason Hickey, Anil Madhavapeddy and Yaron Minsky, that is supposed
> to be
> > > available this Autumn? The Twitter
> > > channel<https://twitter.com/#!/realworldocaml>is threateningly silent:
> > > is the project still alive?
> >
> > That's an interesting leap of logic. It'll be available in Autumn, so you
> > assume the project is dead in July? :-)
> >
> > Yes, we're working away on it.  The book will also be freely available
> > under a CC license, so don't worry: you'll have ample time to read it and
> > comment on the beta before it goes into 1st edition print.  We'll talk
> > more about the content after the first snapshot is out after the summer.
>
> To clarify further, the book will not going into print before the end of
> the year.  "Autumn" refers to when the online snapshots will be
> available for feedback and editorial review.
>
> --
> Anil Madhavapeddy                                 http://anil.recoil.org
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa-roc.inria.fr/wws/info/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>
>

[-- Attachment #2: Type: text/html, Size: 3050 bytes --]

Re: [Caml-list] Re: Syntax extensions without Camlp4

[Gabriel Scherer]

And the OCaml Users and Developers workshop/meeting, co-organized by
Anil, will be an excellent occasion to discuss all that. Please
attend!

  http://oud.ocaml.org/2012/

On Thu, May 31, 2012 at 2:40 PM, Anil Madhavapeddy <anil@recoil.org> wrote:
> On Thu, May 31, 2012 at 01:38:02PM +0100, Anil Madhavapeddy wrote:
>> On Thu, May 31, 2012 at 02:26:18PM +0200, Paolo Donadeo wrote:
>> > On Thu, May 31, 2012 at 10:19 AM, <rixed@happyleptic.org> wrote:
>> > >
>> > >  > I didn't know. Where is it?
>> > > https://github.com/bobzhang/ocaml-book
>> >
>> >
>> > Thanks, just cloned :-)
>> >
>> > Out of curiosity, does anyone have news about Real World
>> > OCaml<http://realworldocaml.org/>,
>> > by Jason Hickey, Anil Madhavapeddy and Yaron Minsky, that is supposed to be
>> > available this Autumn? The Twitter
>> > channel<https://twitter.com/#!/realworldocaml>is threateningly silent:
>> > is the project still alive?
>>
>> That's an interesting leap of logic. It'll be available in Autumn, so you
>> assume the project is dead in July? :-)
>>
>> Yes, we're working away on it.  The book will also be freely available
>> under a CC license, so don't worry: you'll have ample time to read it and
>> comment on the beta before it goes into 1st edition print.  We'll talk
>> more about the content after the first snapshot is out after the summer.
>
> To clarify further, the book will not going into print before the end of
> the year.  "Autumn" refers to when the online snapshots will be
> available for feedback and editorial review.
>
> --
> Anil Madhavapeddy                                 http://anil.recoil.org
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa-roc.inria.fr/wws/info/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>

Re: [Caml-list] Re: Syntax extensions without Camlp4

[Alain Frisch]

On 05/30/2012 04:14 PM, Hongbo Zhang wrote:
> 1. Why camlp4 is buggy?
> The main buggy part is its parsing technology.

I don't consider that the main problem with Camlp4 is that it is buggy, 
but rather that (i) it is overly complex for the benefits it delivers,
and (ii) it is actually not such a great idea to change the concrete syntax.

Some example of useless complexity:

  - A custom notion of AST. Why not simply use the OCaml one?  (Extended 
with nodes for a new nodes, like quotations.)

  - The use of concrete syntax for manipulating the AST.  The developer 
needs to understand not only the new AST, but also how it is reflected 
exactly by the concrete syntax quotations (and this is non trivial), and 
where anti-quotations are allowed, etc. What's wrong with normal pattern 
matching and expression building with the standard AST?  It might be a 
little bit more verbose, but it's so much simpler to understand.

  - A different syntax (the revised one).  I understand the benefits of 
this new syntax, but it seems kind of crazy to have a "low-level" tool 
implemented in (and encouraging) a syntax different from the core system.

  - A complicated bootstrapping cycle (partly a consequence of the fact 
that Camlp4 is itself written in a custom syntax).  That's mostly for 
OCaml maintainers, but in the past, it has slowed down development in a 
non-negligible way.


> 2. About the proposal.
> There are mainly 2 pieces. About the hook
> Parsetree.structure->Parsetree. structure, given that camlp4 already
> imports Parsetree, it's really trivial to
> add another hook after camlp4astdump2ocamlast.

I've absolutely no doubt that Camlp4 can be extended to be at least as 
powerful as this "Parsetree rewriting" proposal.  What's important is 
that this proposal is so simple that it can be implemented in a few 
dozens line of code in the core compiler. We should not create a 
dependency on a complex tool for problems which can be solved with 
something so simple.

> It's still nontrivial to write a robust Parsetree.structure ->
> Parsetree.structure, it would be nice if we could provide a quotation
> syntax for Parsetree.types.

I believe the opposite: it's simpler to write a robust AST->AST 
rewriting function if you work directly on the "real" AST definition, 
rather than a slighlty different one and with a custom syntax.  Just for 
an example, consider a left-hand side like:

   | <:pat< $p1$ | $p2$ | $p3$ >> -> ...

Will it capture both (p1|p2)|p3 and p1|(p2|p3)?  Or only one of them?
Another example: controlling precisely locations introduced in the AST 
fragments created with quotations is quite tricky.



Alain

Re: [Caml-list] Re: Syntax extensions without Camlp4

[Dmitry Grebeniuk]

Hello.

>  - A different syntax (the revised one).  I understand the benefits of this
> new syntax, but it seems kind of crazy to have a "low-level" tool
> implemented in (and encouraging) a syntax different from the core system.

  But how will people otherwise discover the beauty of
revised syntax if camlp4 will accept original syntax?

Re: [Caml-list] Re: Syntax extensions without Camlp4

[Daniel Bünzli]

Le jeudi, 31 mai 2012 à 15:21, Dmitry Grebeniuk a écrit :  
> But how will people otherwise discover the beauty of
> revised syntax if camlp4 will accept original syntax?


It could have been beautiful if the dev team had gradually forced us to transition to it. Having not, it should be considered inexistant in my opinion. It just brings more confusion and fragmentation around the OCaml system.  

I don't have problems with OCaml's lack of popularity, but still let's try not to make it totally irrelevant by having at least two different syntaxes for it.  

Best,

Daniel

Re: [Caml-list] Re: Syntax extensions without Camlp4

[bob zhang]

[-- Attachment #1: Type: text/plain, Size: 3815 bytes --]

On Thu, May 31, 2012 at 8:59 AM, Alain Frisch <alain@frisch.fr> wrote:

> On 05/30/2012 04:14 PM, Hongbo Zhang wrote:
>
>> 1. Why camlp4 is buggy?
>> The main buggy part is its parsing technology.
>>
>
> I don't consider that the main problem with Camlp4 is that it is buggy,
> but rather that (i) it is overly complex for the benefits it delivers,
> and (ii) it is actually not such a great idea to change the concrete
> syntax.
>
> Hi,Alain,
   Thanks for your message.
   Some opinions below: (Feel free to correct if I am wrong)
   Why do you think it's overly complex? The other part is not complex IMO
except the internal mechanism of parsing. There are other benefits of
revised syntax. One point is that write an error
recover parser is straitforward for revised syntax. And it's more friendly
to IDE.

Some example of useless complexity:
>
>  - A custom notion of AST. Why not simply use the OCaml one?  (Extended
> with nodes for a new nodes, like quotations.)
>
> The simple answer is you can not. (or it's at least not an easy way once
you want to support quotation and *antiquotation* *everywhere*. Just take a
look at how ugly the Template Haskell program is)

>  - The use of concrete syntax for manipulating the AST.  The developer
> needs to understand not only the new AST, but also how it is reflected
> exactly by the concrete syntax quotations (and this is non trivial), and
> where anti-quotations are allowed, etc. What's wrong with normal pattern
> matching and expression building with the standard AST?  It might be a
> little bit more verbose, but it's so much simpler to understand.
>
> The answer is also you can not. There are some ambiguities that you can
not support quotations
and antiquotations. Currenty Camlp4 support quotations and antiquotations
for revised syntax in
all branches except Ast.TyDcl.

I agree it would be useful to write a simple quoation expander for
Parsetree.structure_item, (with limited antiquotation support) in Camlp4
and add another hook for Parsetree.

>  - A different syntax (the revised one).  I understand the benefits of
> this new syntax, but it seems kind of crazy to have a "low-level" tool
> implemented in (and encouraging) a syntax different from the core system.
>
>  - A complicated bootstrapping cycle (partly a consequence of the fact
> that Camlp4 is itself written in a custom syntax).  That's mostly for OCaml
> maintainers, but in the past, it has slowed down development in a
> non-negligible way.
>
>
>  2. About the proposal.
>> There are mainly 2 pieces. About the hook
>> Parsetree.structure->**Parsetree. structure, given that camlp4 already
>>
>> imports Parsetree, it's really trivial to
>> add another hook after camlp4astdump2ocamlast.
>>
>
> I've absolutely no doubt that Camlp4 can be extended to be at least as
> powerful as this "Parsetree rewriting" proposal.  What's important is that
> this proposal is so simple that it can be implemented in a few dozens line
> of code in the core compiler. We should not create a dependency on a
> complex tool for problems which can be solved with something so simple.
>
>
>  It's still nontrivial to write a robust Parsetree.structure ->
>> Parsetree.structure, it would be nice if we could provide a quotation
>> syntax for Parsetree.types.
>>
>
> I believe the opposite: it's simpler to write a robust AST->AST rewriting
> function if you work directly on the "real" AST definition, rather than a
> slighlty different one and with a custom syntax.  Just for an example,
> consider a left-hand side like:
>
>  | <:pat< $p1$ | $p2$ | $p3$ >> -> ...
>
> Will it capture both (p1|p2)|p3 and p1|(p2|p3)?  Or only one of them?
> Another example: controlling precisely locations introduced in the AST
> fragments created with quotations is quite tricky.
>
>
>
> Alain
>



-- 
-- Bob

[-- Attachment #2: Type: text/html, Size: 5261 bytes --]

Re: [Caml-list] Re: Syntax extensions without Camlp4

[Gerd Stolpmann]

Am Donnerstag, den 31.05.2012, 12:01 -0400 schrieb bob zhang:
> 
> 
> On Thu, May 31, 2012 at 8:59 AM, Alain Frisch <alain@frisch.fr> wrote:
>         On 05/30/2012 04:14 PM, Hongbo Zhang wrote:
>                 1. Why camlp4 is buggy?
>                 The main buggy part is its parsing technology.
>         
>         
>         I don't consider that the main problem with Camlp4 is that it
>         is buggy, but rather that (i) it is overly complex for the
>         benefits it delivers,
>         and (ii) it is actually not such a great idea to change the
>         concrete syntax.
>         
> Hi,Alain, 
>    Thanks for your message.
>    Some opinions below: (Feel free to correct if I am wrong)
>    Why do you think it's overly complex? The other part is not complex
> IMO except the internal mechanism of parsing. There are other benefits
> of revised syntax. One point is that write an error  
> recover parser is straitforward for revised syntax. And it's more
> friendly to IDE.

IMHO, there is no point in discussing "revised" (actually, alternate)
syntax. It might be easier to parse, but most programmers are used to
the standard syntax. The problem here is that it is difficult for a
human to switch between two syntaxes.

That said, I'm for dropping the alternate syntax entirely.

> 
> 
>         Some example of useless complexity:
>         
>          - A custom notion of AST. Why not simply use the OCaml one?
>          (Extended with nodes for a new nodes, like quotations.)
>         
> The simple answer is you can not. (or it's at least not an easy way
> once you want to support quotation and *antiquotation* *everywhere*.
> Just take a look at how ugly the Template Haskell program is) 

What's the problem with antiquotations? I imagine that you can simply
call the parser, and it is only a matter of selecting the right entry
point, and passing the tokens down.

>          - The use of concrete syntax for manipulating the AST.  The
>         developer needs to understand not only the new AST, but also
>         how it is reflected exactly by the concrete syntax quotations
>         (and this is non trivial), and where anti-quotations are
>         allowed, etc. What's wrong with normal pattern matching and
>         expression building with the standard AST?  It might be a
>         little bit more verbose, but it's so much simpler to
>         understand.
>         
> The answer is also you can not. There are some ambiguities that you
> can not support quotations
> and antiquotations. Currenty Camlp4 support quotations and
> antiquotations for revised syntax in 
> all branches except Ast.TyDcl.

I guess you are missing Alain's point here. He suggests to use normal
pattern matching directly on parsetrees, as in

match expr with
 | Pexp_ident id -> Pexp_ident (tranform id)
 | Pexp_contant c -> Pexp_constant c
 | ...

instead of using camlp4 syntax quotations (no <<...>> stuff for syntax
manipulation anymore).

Gerd

> I agree it would be useful to write a simple quoation expander for
> Parsetree.structure_item, (with limited antiquotation support) in
> Camlp4 and add another hook for Parsetree.
>          - A different syntax (the revised one).  I understand the
>         benefits of this new syntax, but it seems kind of crazy to
>         have a "low-level" tool implemented in (and encouraging) a
>         syntax different from the core system.
>         
>          - A complicated bootstrapping cycle (partly a consequence of
>         the fact that Camlp4 is itself written in a custom syntax).
>          That's mostly for OCaml maintainers, but in the past, it has
>         slowed down development in a non-negligible way.
>         
>         
>                 2. About the proposal.
>                 There are mainly 2 pieces. About the hook
>                 
>                 Parsetree.structure->Parsetree. structure, given that
>                 camlp4 already
>                 
>                 imports Parsetree, it's really trivial to
>                 add another hook after camlp4astdump2ocamlast.
>                 
>         
>         I've absolutely no doubt that Camlp4 can be extended to be at
>         least as powerful as this "Parsetree rewriting" proposal.
>          What's important is that this proposal is so simple that it
>         can be implemented in a few dozens line of code in the core
>         compiler. We should not create a dependency on a complex tool
>         for problems which can be solved with something so simple.
>         
>         
>                 It's still nontrivial to write a robust
>                 Parsetree.structure ->
>                 Parsetree.structure, it would be nice if we could
>                 provide a quotation
>                 syntax for Parsetree.types.
>         
>         
>         I believe the opposite: it's simpler to write a robust
>         AST->AST rewriting function if you work directly on the "real"
>         AST definition, rather than a slighlty different one and with
>         a custom syntax.  Just for an example, consider a left-hand
>         side like:
>         
>          | <:pat< $p1$ | $p2$ | $p3$ >> -> ...
>         
>         Will it capture both (p1|p2)|p3 and p1|(p2|p3)?  Or only one
>         of them?
>         Another example: controlling precisely locations introduced in
>         the AST fragments created with quotations is quite tricky.
>         
>         
>         
>         Alain
> 
> 
> 
> 
> -- 
> -- Bob

-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
Creator of GODI and camlcity.org.
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
*** Searching for new projects! Need consulting for system
*** programming in Ocaml? Gerd Stolpmann can help you.
------------------------------------------------------------

Re: [Caml-list] Re: Syntax extensions without Camlp4

[Wojciech Meyer]

Gerd Stolpmann <info@gerd-stolpmann.de> writes:

> Am Donnerstag, den 31.05.2012, 12:01 -0400 schrieb bob zhang:
> I guess you are missing Alain's point here. He suggests to use normal
> pattern matching directly on parsetrees, as in
>
> match expr with
>  | Pexp_ident id -> Pexp_ident (tranform id)
>  | Pexp_contant c -> Pexp_constant c
>  | ...
>
> instead of using camlp4 syntax quotations (no <<...>> stuff for syntax
> manipulation anymore).

I admit quotations are *way* better to construct the code, they are
syntax aware but also reassemble string manipulation at the same
time. If somebody has generated some more serious program using
quotations then she can't stand using normal way by constructing via the
data constructors. I can compare, because I have used both ways for the
same project. Of course normal ASTs have it's wide applications, what is
shown above is a transform, and not strictly a code generation - usually
I have intermediate AST on the left hand side pattern match and on the
right hand side quotations. So I can't say quotations are not useful,
they are quite much. If the quotations could be as lightweight as
brackets-escape notations of MetaOCaml it would be brilliant. (I'm
really talking about code generation - and not extending the syntax).

Please see [1].

I'd support the idea of having quotations for the parse trees.

-- 
Wojciech Meyer
http://danmey.org

[1] http://www.eecs.harvard.edu/~mainland/ghc-quasiquoting/mainland07quasiquoting.pdf

Re: [Caml-list] Re: Syntax extensions without Camlp4

[Hongbo Zhang]

On 5/31/12 1:28 PM, Gerd Stolpmann wrote:
> Am Donnerstag, den 31.05.2012, 12:01 -0400 schrieb bob zhang:
>>
>> On Thu, May 31, 2012 at 8:59 AM, Alain Frisch<alain@frisch.fr>  wrote:
>>          On 05/30/2012 04:14 PM, Hongbo Zhang wrote:
>>                  1. Why camlp4 is buggy?
>>                  The main buggy part is its parsing technology.
>>
>>
>>          I don't consider that the main problem with Camlp4 is that it
>>          is buggy, but rather that (i) it is overly complex for the
>>          benefits it delivers,
>>          and (ii) it is actually not such a great idea to change the
>>          concrete syntax.
>>
>> Hi,Alain,
>>     Thanks for your message.
>>     Some opinions below: (Feel free to correct if I am wrong)
>>     Why do you think it's overly complex? The other part is not complex
>> IMO except the internal mechanism of parsing. There are other benefits
>> of revised syntax. One point is that write an error
>> recover parser is straitforward for revised syntax. And it's more
>> friendly to IDE.
> IMHO, there is no point in discussing "revised" (actually, alternate)
> syntax. It might be easier to parse, but most programmers are used to
> the standard syntax. The problem here is that it is difficult for a
> human to switch between two syntaxes.
>
> That said, I'm for dropping the alternate syntax entirely.
It depends. I wrote camlp4 related programs in revised syntax, and
normal caml code in original syntax, it may seem awful at first,
but once you get used to it, it's really a minor issue compared with
waiting for patches for camlp4of. If you look at mantis, most camlp4
related bugs are about camlp4of. Dropping camlp4of will help alleviate
the burden a lot
>>
>>          Some example of useless complexity:
>>
>>           - A custom notion of AST. Why not simply use the OCaml one?
>>           (Extended with nodes for a new nodes, like quotations.)
>>
>> The simple answer is you can not. (or it's at least not an easy way
>> once you want to support quotation and *antiquotation* *everywhere*.
>> Just take a look at how ugly the Template Haskell program is)
> What's the problem with antiquotations? I imagine that you can simply
> call the parser, and it is only a matter of selecting the right entry
> point, and passing the tokens down.
Campl4 Ast is *quite loose*, and designed this way to tolerate parital
Ast, and way more better support for quosiquotation than
Template Haskell (etc).

This is the last post I commented on this thread.
IMHO, we should do some work to improve camlp4, a community to build 
libraries
on top of it to avoid reinventing the wheels instead of dropping 
existing powerful
tools.


Many Thanks
>
>>           - The use of concrete syntax for manipulating the AST.  The
>>          developer needs to understand not only the new AST, but also
>>          how it is reflected exactly by the concrete syntax quotations
>>          (and this is non trivial), and where anti-quotations are
>>          allowed, etc. What's wrong with normal pattern matching and
>>          expression building with the standard AST?  It might be a
>>          little bit more verbose, but it's so much simpler to
>>          understand.
>>
>> The answer is also you can not. There are some ambiguities that you
>> can not support quotations
>> and antiquotations. Currenty Camlp4 support quotations and
>> antiquotations for revised syntax in
>> all branches except Ast.TyDcl.
> I guess you are missing Alain's point here. He suggests to use normal
> pattern matching directly on parsetrees, as in
>
> match expr with
>   | Pexp_ident id ->  Pexp_ident (tranform id)
>   | Pexp_contant c ->  Pexp_constant c
>   | ...
>
> instead of using camlp4 syntax quotations (no<<...>>  stuff for syntax
> manipulation anymore).
>
> Gerd
>
>> I agree it would be useful to write a simple quoation expander for
>> Parsetree.structure_item, (with limited antiquotation support) in
>> Camlp4 and add another hook for Parsetree.
>>           - A different syntax (the revised one).  I understand the
>>          benefits of this new syntax, but it seems kind of crazy to
>>          have a "low-level" tool implemented in (and encouraging) a
>>          syntax different from the core system.
>>
>>           - A complicated bootstrapping cycle (partly a consequence of
>>          the fact that Camlp4 is itself written in a custom syntax).
>>           That's mostly for OCaml maintainers, but in the past, it has
>>          slowed down development in a non-negligible way.
>>
>>
>>                  2. About the proposal.
>>                  There are mainly 2 pieces. About the hook
>>
>>                  Parsetree.structure->Parsetree. structure, given that
>>                  camlp4 already
>>
>>                  imports Parsetree, it's really trivial to
>>                  add another hook after camlp4astdump2ocamlast.
>>
>>
>>          I've absolutely no doubt that Camlp4 can be extended to be at
>>          least as powerful as this "Parsetree rewriting" proposal.
>>           What's important is that this proposal is so simple that it
>>          can be implemented in a few dozens line of code in the core
>>          compiler. We should not create a dependency on a complex tool
>>          for problems which can be solved with something so simple.
>>
>>
>>                  It's still nontrivial to write a robust
>>                  Parsetree.structure ->
>>                  Parsetree.structure, it would be nice if we could
>>                  provide a quotation
>>                  syntax for Parsetree.types.
>>
>>
>>          I believe the opposite: it's simpler to write a robust
>>          AST->AST rewriting function if you work directly on the "real"
>>          AST definition, rather than a slighlty different one and with
>>          a custom syntax.  Just for an example, consider a left-hand
>>          side like:
>>
>>           |<:pat<  $p1$ | $p2$ | $p3$>>  ->  ...
>>
>>          Will it capture both (p1|p2)|p3 and p1|(p2|p3)?  Or only one
>>          of them?
>>          Another example: controlling precisely locations introduced in
>>          the AST fragments created with quotations is quite tricky.
>>
>>
>>
>>          Alain
>>
>>
>>
>>
>> -- 
>> -- Bob

Re: [Caml-list] Re: Syntax extensions without Camlp4

[Gerd Stolpmann]

Am Donnerstag, den 31.05.2012, 19:03 +0100 schrieb Wojciech Meyer:
> Gerd Stolpmann <info@gerd-stolpmann.de> writes:
> 
> > Am Donnerstag, den 31.05.2012, 12:01 -0400 schrieb bob zhang:
> > I guess you are missing Alain's point here. He suggests to use normal
> > pattern matching directly on parsetrees, as in
> >
> > match expr with
> >  | Pexp_ident id -> Pexp_ident (tranform id)
> >  | Pexp_contant c -> Pexp_constant c
> >  | ...
> >
> > instead of using camlp4 syntax quotations (no <<...>> stuff for syntax
> > manipulation anymore).
> 
> I admit quotations are *way* better to construct the code, they are
> syntax aware but also reassemble string manipulation at the same
> time. If somebody has generated some more serious program using
> quotations then she can't stand using normal way by constructing via the
> data constructors. I can compare, because I have used both ways for the
> same project. Of course normal ASTs have it's wide applications, what is
> shown above is a transform, and not strictly a code generation - usually
> I have intermediate AST on the left hand side pattern match and on the
> right hand side quotations. So I can't say quotations are not useful,
> they are quite much. If the quotations could be as lightweight as
> brackets-escape notations of MetaOCaml it would be brilliant. (I'm
> really talking about code generation - and not extending the syntax).
> 
> Please see [1].
> 
> I'd support the idea of having quotations for the parse trees.

I'm wondering whether you could run the normal ocaml parser in a special
mode for constructing code. In this mode, all non-keyword identifiers
(and possibly other placeholders) can also be given in antiquotation
form $name$, and the values for such injected names are taken from a
table you have to supply when calling the parser in this mode. This
would then look like

let antiquot_map =
  StrMap.add "var" "x" StrMap.empty

let ast =
  parse_struct_item antiquot_map "let $var$ = x+1"

And, given quotations are available in a renovated parser, one could
even write <:struct_item< let $var$ = x+1 >>, which already looks very
camlp4-ish (but would not use camlp4 at all), and would just be a thin
layer on top of the standard parser.

I mean the whole point is that we can do a lot without camlp4 by just
pimping the normal parser a bit. Actually, we need camlp4 only for
extensible grammars and the revised syntax, and Alain is probably right
that this is not worth the effort.

Gerd
-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
Creator of GODI and camlcity.org.
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
*** Searching for new projects! Need consulting for system
*** programming in Ocaml? Gerd Stolpmann can help you.
------------------------------------------------------------

Re: [Caml-list] Re: Syntax extensions without Camlp4

[Paolo Donadeo]

[-- Attachment #1: Type: text/plain, Size: 311 bytes --]

On Thu, May 31, 2012 at 2:38 PM, Anil Madhavapeddy <anil@recoil.org> wrote:

> That's an interesting leap of logic. It'll be available in Autumn, so you
> assume the project is dead in July? :-)
>

I admit you are right :-)

Since I didn't see any activity on Twitter, I deduced... forget it! :-)


-- 
*Paolo*

[-- Attachment #2: Type: text/html, Size: 681 bytes --]