As this seems to be the kind of "christmas shopping list" discussion where
everyone throws something in, here is my opinion on camlp4 evolution:
- we should forget about the general problem of extending syntax rules;
it's too complex, doesn't compose well, and is only of arguable benefit
- we should focus on blessing specific *fixed* syntactical extensibility
points and making them enter the standard syntax for the language; there
are two elephants in the rooms: quotations and type-conv-like annotations.

Quotations are useful, used in practice (not always under the
camlp4-blessed form), simple to define, understand and use, and compose
extremely well. One problem with current camlp4 quotations is that they are
a bit lexically heavy (<:name< ... >>) and inflexible (you basically can't
use >> inside the quotation); users have tried to workaround this by
re-coding quotations, eg. for smart string literals u"blah" or regexp
syntactic sugar s/foo/bar/. I think we could consider having lexing-level
extensibility (have a tool to modify lexing rule instead of grammar rules
like Camlp4; hopefully lexing rules are simpler and compose better).

Type-conv like annotations are a bit less well-defined, and would need some
design work: what exactly is their intended scope/expressivity? Are they a
general annotation mechanism, a rigid way to just add new phrases after
each annotation phrase, or something in between?

Other point of extensibility could be considered (eg. Jeremy Yallop's work
on pattern-matching extensibility), but only added if they are simple
enough, useful enough, and compose well.

I would still welcome having central, well-supported libraries to
manipulate Ocaml syntactic AST and, why not, typed AST (typedtree). Among
its many features, Camlp4 currently provides the former (in a relatively
convenient way thanks to its AST quotations), and this is quite useful.

On Mon, May 28, 2012 at 11:35 AM, Daniel Bünzli <daniel.buenzli@erratique.ch
> wrote:

>
>
> Le lundi, 28 mai 2012 à 00:43, Wojciech Meyer a écrit :
>
> > Runtime meta-programming is a generalisation of static meta
> > programming. MetaOCaml has a nice set of abstraction to generate
> > typechecking code - yes - either at runtime or during compile time.
>
>
> You meant 'typechecked' (?). It's obviously a generalization but I didn't
> know there was support to use it at compile time.
>
> > The
> > problem is that it's purely for partial evaluation and not extending the
> > syntax.
>
>
> Then it's perfect ! I think it's wrong to try to extend the language per
> se. Most of the time, except for very particular things (e.g. introducing a
> monad notation), the dsl approach is perfectly sufficient. Don't think you
> absolutely need to extend the OCaml grammar, embed your dsl directly into
> OCaml, using OCaml language binders if you need variables.
>
> Make libraries, not pet syntactic constructs.
>
> Many things that can be done with camlp4, can be done with that approach.
> Not only is it very elegant, it's much easier to maintain w.r.t. the
> evolution of the OCaml language itself. The techniques in these papers [1]
> should be more known and used.
>
> > - It should not be external tool - like previously observed - it's
> > difficult to support for code highlighters or refactoring (tools in
> > general) - if it depends on a build step or command line options.
>
>
>
>
> If you extend the grammar itself, code highlighters or any tool expecting
> an OCaml expression is going to break whether the tool is external or not.
> But for the rest of your comments I agree wholeheartedly (even though I'm
> not sure all that power is needed, but at least it would make the tool
> non-ugly to me).
>
> Best,
>
> Daniel
>
> [1]
>
> http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.31.9782&rep=rep1&type=pdf
> http://research.microsoft.com/apps/pubs/default.aspx?id=67334
>
>
>
>
>
>
> > Therefore the main purpose of syntactical abstraction is missing
> > - but that's not a problem - MetaOCaml wasn't designed for it.
> >
> > Things that I would like to see in future "incarnation" or integrating of
> > meta programming facilities to the language would be (beware that's my
> > blue dreams!):
> >
> > - first of all non destructive updates to the grammar e.g: "let open
> > lang Sexp in ..." should open the Sexp syntax extension, install the
> > grammar, but when it goes out of scope it should vanish. Currently
> > Camlp4 can install, delete the rules after the functor is applied, and
> > no way of saying OK - let's go back.
> >
> > - Composable - in particular one language should behave like a module,
> > or functor, should have an interface consisting of grammar rules, AST,
> > AST transforms etc. So one could parametrise one syntax extension
> > over another, and possibly reuse the language grammar or AST in
> > other. Currently Camlp4 syntax extension is just a single separate
> > module which when loaded possibly expects some existing grammar rules
> > to be in place and mutates them as it's needed.
> >
> > - should be type safe and as mentioned before obey scoping rules. We
> > should be able to propagate type information even when the syntax
> > changes. This is difficult part - but I've seen it can be done with some
> > extra annotations - not talking about Camlp4
> >
> > - Recursive - it should be able to apply the grammar rules not only
> > once but expand until it reached the fixpoint.
> >
> > - Reflective - it should be possible after each successful expansion have
> > the type information available for the next expansion.
> >
> > - Grammar itself should be lexer-less - memoizing PEG with left
> > recursion - it's hurdle to define new grammar in terms of old lexer,
> > or having a stateful lexer that depends on context.
> >
> > - It should not be external tool - like previously observed - it's
> > difficult to support for code highlighters or refactoring (tools in
> > general) - if it depends on a build step or command line options.
> >
> > That's said I find Camlp4 extremely useful for code generation purposes
> > - when I need to generate some ML code through quotations. Also, some
> > very important projects depend on Camlp4 (or Camlp5) like Coq. I don't
> > see that ML can live without some meta programming facilities out of the
> > box.
> >
> > --
> > Wojciech Meyer
> > http://danmey.org
> >
> > --
> > Caml-list mailing list. Subscription management and archives:
> > https://sympa-roc.inria.fr/wws/info/caml-list
> > Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> > Bug reports: http://caml.inria.fr/bin/caml-bugs
>
>
>
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa-roc.inria.fr/wws/info/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>
>