As this seems to be the kind of "christmas shopping list" discussion where everyone throws something in, here is my opinion on camlp4 evolution:
- we should forget about the general problem of extending syntax rules; it's too complex, doesn't compose well, and is only of arguable benefit
- we should focus on blessing specific *fixed* syntactical extensibility points and making them enter the standard syntax for the language; there are two elephants in the rooms: quotations and type-conv-like annotations.

Quotations are useful, used in practice (not always under the camlp4-blessed form), simple to define, understand and use, and compose extremely well. One problem with current camlp4 quotations is that they are a bit lexically heavy (<:name< ... >>) and inflexible (you basically can't use >> inside the quotation); users have tried to workaround this by re-coding quotations, eg. for smart string literals u"blah" or regexp syntactic sugar s/foo/bar/. I think we could consider having lexing-level extensibility (have a tool to modify lexing rule instead of grammar rules like Camlp4; hopefully lexing rules are simpler and compose better).

Type-conv like annotations are a bit less well-defined, and would need some design work: what exactly is their intended scope/expressivity? Are they a general annotation mechanism, a rigid way to just add new phrases after each annotation phrase, or something in between?

Other point of extensibility could be considered (eg. Jeremy Yallop's work on pattern-matching extensibility), but only added if they are simple enough, useful enough, and compose well.

I would still welcome having central, well-supported libraries to manipulate Ocaml syntactic AST and, why not, typed AST (typedtree). Among its many features, Camlp4 currently provides the former (in a relatively convenient way thanks to its AST quotations), and this is quite useful.

On Mon, May 28, 2012 at 11:35 AM, Daniel Bünzli <daniel.buenzli@erratique.ch> wrote:


Le lundi, 28 mai 2012 à 00:43, Wojciech Meyer a écrit :

> Runtime meta-programming is a generalisation of static meta
> programming. MetaOCaml has a nice set of abstraction to generate
> typechecking code - yes - either at runtime or during compile time.


You meant 'typechecked' (?). It's obviously a generalization but I didn't know there was support to use it at compile time.

> The
> problem is that it's purely for partial evaluation and not extending the
> syntax.


Then it's perfect ! I think it's wrong to try to extend the language per se. Most of the time, except for very particular things (e.g. introducing a monad notation), the dsl approach is perfectly sufficient. Don't think you absolutely need to extend the OCaml grammar, embed your dsl directly into OCaml, using OCaml language binders if you need variables.

Make libraries, not pet syntactic constructs.

Many things that can be done with camlp4, can be done with that approach. Not only is it very elegant, it's much easier to maintain w.r.t. the evolution of the OCaml language itself. The techniques in these papers [1] should be more known and used.

> - It should not be external tool - like previously observed - it's
> difficult to support for code highlighters or refactoring (tools in
> general) - if it depends on a build step or command line options.




If you extend the grammar itself, code highlighters or any tool expecting an OCaml expression is going to break whether the tool is external or not. But for the rest of your comments I agree wholeheartedly (even though I'm not sure all that power is needed, but at least it would make the tool non-ugly to me).

Best,

Daniel

[1]
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.31.9782&rep=rep1&type=pdf
http://research.microsoft.com/apps/pubs/default.aspx?id=67334






> Therefore the main purpose of syntactical abstraction is missing
> - but that's not a problem - MetaOCaml wasn't designed for it.
>
> Things that I would like to see in future "incarnation" or integrating of
> meta programming facilities to the language would be (beware that's my
> blue dreams!):
>
> - first of all non destructive updates to the grammar e.g: "let open
> lang Sexp in ..." should open the Sexp syntax extension, install the
> grammar, but when it goes out of scope it should vanish. Currently
> Camlp4 can install, delete the rules after the functor is applied, and
> no way of saying OK - let's go back.
>
> - Composable - in particular one language should behave like a module,
> or functor, should have an interface consisting of grammar rules, AST,
> AST transforms etc. So one could parametrise one syntax extension
> over another, and possibly reuse the language grammar or AST in
> other. Currently Camlp4 syntax extension is just a single separate
> module which when loaded possibly expects some existing grammar rules
> to be in place and mutates them as it's needed.
>
> - should be type safe and as mentioned before obey scoping rules. We
> should be able to propagate type information even when the syntax
> changes. This is difficult part - but I've seen it can be done with some
> extra annotations - not talking about Camlp4
>
> - Recursive - it should be able to apply the grammar rules not only
> once but expand until it reached the fixpoint.
>
> - Reflective - it should be possible after each successful expansion have
> the type information available for the next expansion.
>
> - Grammar itself should be lexer-less - memoizing PEG with left
> recursion - it's hurdle to define new grammar in terms of old lexer,
> or having a stateful lexer that depends on context.
>
> - It should not be external tool - like previously observed - it's
> difficult to support for code highlighters or refactoring (tools in
> general) - if it depends on a build step or command line options.
>
> That's said I find Camlp4 extremely useful for code generation purposes
> - when I need to generate some ML code through quotations. Also, some
> very important projects depend on Camlp4 (or Camlp5) like Coq. I don't
> see that ML can live without some meta programming facilities out of the
> box.
>
> --
> Wojciech Meyer
> http://danmey.org
>
> --
> Caml-list mailing list. Subscription management and archives:
> https://sympa-roc.inria.fr/wws/info/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs




--
Caml-list mailing list.  Subscription management and archives:
https://sympa-roc.inria.fr/wws/info/caml-list
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs