[Caml-list] Re: Call for collaboration on the future of camlp4

[Hongbo Zhang <bobzhang1988@gmail.com>]

On 9/19/12 3:55 PM, Wojciech Meyer wrote:
> Hi Hongboz,
>
Greetings,
> Thanks for the slides, it's good. Indeed it makes sense now to focus on
> extending Camlp4, however as usually there are some gotchas, the process
> of extending syntax with Camlp4 is a bit not well known.  Camlp5 has an
> excellent documentation, and great support for some of the things. Yet,
> Camlp4 is more modular, and is just easier to write application on top
> of Camlp4.
>
Yes, camlp5 documentation is much more comprehensive for the users.
But the underlying mechanism is not explained either, I did not see any
difference for low-level users, another problem is that camlp5 is
  not heavily tested.

> On OUD we had open ended discussions, and many people share (like me)
> the same impression - -ppx that exist on trunk is powerful and that's
> the way to implement most of the meta programming facilities (deriving
> the code from type definitions, or using quotations to embed DSLs in
> OCaml), but I believe that the direct syntax extensions (opening the
> OCaml syntax) certainly has some benefits. How to do this in a clean way
> myself - I don't know. I posted just some time ago what would be a macro
> system of my dreams:
-ppx is orthogonal to what camlp4 did, it's pretty easy to integrate
'ast rewriter' in Fan or camlp4. There are 3 problems with Ast Rewriter

1. No Quasiquotaion support
    It's ok to construct the Ast by hand for the first order macros,
    as I pointed out before, you can not write macros which write macros
    without quasi-quotation support. Remember that macros are itself 
programs,
    and there are dupliated code in macros as well, so you still need to
    scrap those dupliated code. So the expressivity to Ast Rewriter is like
    higher-order programming to first-order programming. It's a shame 
that so far
    there are not too many camlp4 plugins which generate macros to 
remove the duplicated
    code for camlp4 plugins. The camlp4 source tree itself is also 
written in a verbose way.
    Adding Quasiquotation support directly to  current parsetree is not 
an easy way,
    for macro programming, you really want to quote, and antiquote 
everywhere, and
    create some illegal Ast. I have programmed  a lot in Template 
Haskell, and I
    really appreciate that camlp4 have a much better quote-antiquot support.

2. No Delimited Syntax Extension support
    The Ast Rewriter can only override existing syntax(which is a bit 
fragile, IMO). To make
    things worse, if you want to link the compiler to your 
pre-processor, you really want to
    mark which piece of code should be eval during pre-processing  or 
delayed at run-time. So,
    even the minimal delimited syntax extension will increase the 
expressivety a lot

3. General Syntax Extension
    General Syntax Extension is definitely very useful when I finished 
the functional parser part.
    A problem with delimited syntax extension support is that sometimes 
you really want to mix the
    existing grammars with your new grammars, currently there's no easy 
way to do that in delimited
    syntax extension. When we finished functional parser, we can 
restrict the syntax extension scope
    to the minimal expression level

Btw, all the problems that camlp4 or Fan came across will not disappear 
in Ast Rewriter, you don't get
any benefit from them, hygenic, no, the same problem.
The beauty of camlp4 is that it is bootstrapped itself, when you evolve 
camlp4, you are making a more and more
expressive system, part of my work is to re-write camlp4 to make it much 
more succinct. :-)
>
> https://sympa.inria.fr/sympa/arc/caml-list/2012-05/msg00184.html
About your proposals:

Runtime meta-programming is a generalisation of static meta
programming. MetaOCaml has a nice set of abstraction to generate
typechecking code - yes - either at runtime or during compile time.

Meta-ocaml is orthogonal to camlp4 as well, but I did not see any hope 
that it will be pushed into ocaml compiler,
since the main purpose of meta-ocaml is to do some run-time 
optimization, without a native jit compiler support, It does
not make sense.

" - first of all non destructive updates to the grammar e.g: "let open
lang Sexp in ..." should open the Sexp syntax extension, install the
grammar, but when it goes out of scope it should vanish. Currently
Camlp4 can install, delete the rules after the functor is applied, and
no way of saying OK - let's go back."
Yes!

- Composable - in particular one language should behave like a module,
or functor, should have an interface consisting of grammar rules, AST,
AST transforms etc. So one could parametrise one syntax extension
over another, and possibly reuse the language grammar or AST in
other. Currently Camlp4 syntax extension is just a single separate
module which when loaded possibly expects some existing grammar rules
to be in place and mutates them as it's needed.

Yes!

- should be type safe and as mentioned before obey scoping rules. We
should be able to propagate type information even when the syntax
changes. This is difficult part - but I've seen it can be done with some
extra annotations - not talking about Camlp4
Personally, I prefer common lisp's macros to scheme. It's painful
to play with racket's macros as well

- Recursive - it should be able to apply the grammar rules not only
once but expand until it reached the fixpoint.

Yes!
- Reflective - it should be possible after each successful expansion have
the type information available for the next expansion.
Possibly with the compiler library help

- Grammar itself should be lexer-less - memoizing PEG with left
recursion - it's hurdle to define new grammar in terms of old lexer,
or having a stateful lexer that depends on context.

Yes!
- It should not be external tool - like previously observed - it's
difficult to support for code highlighters or refactoring (tools in
general) - if it depends on a build step or command line options.
Yes!