[Caml-list] Call for collaboration on the future of camlp4

[Caml-list] Call for collaboration on the future of camlp4

[bob zhang]

[-- Attachment #1: Type: text/plain, Size: 2773 bytes --]

Dear OCaml programmers,
   Last week, I give a talk about the future of Camlp4 in the ML workshop,
the slides are here
http://www.lexifi.com/ml2012/slides_panel_hongbo.pdf(some are already
done, some are work in progress).
   For a long time, in the caml community, thanks to the talented work of
Daniel and Michel,  we know there is a very very powerful tool called
*camlp4*, if there are some bolierpolate code you write here and there,
someone will tell you "hey, you can do it in camlp4" though he may not know
how to do it in camlp4.
   But it's a bit embarassing that camlp4 did not evolve very well(partly
due to the fragmentation of camlp[4,5]), another fact is that camlp4 is not
*designed*, it's like a prototype that works but not carefully designed,
and it does not provide anything out of the box  and itself was written in
a verbose way.
   It's time to bring the powerful tool back, my advisor Steve and I
started a new project Fan, which is mainly to evolve the camlp4 macro
system to be more expressive and more powerful, push the Camlp4 to the next
level. For me, I am a long-term Lisp programmer, I appreciated the value of
macros, I would really be happy to see we could make a such powerful macro
systems.
   Here is my repo https://bitbucket.org/HongboZhang/camlp4
   I already finished some cool staff,  to mention just a few:
   1. A very robust bootstrapping system, previously it takes me 20 minutes
to ver*ify my patches to camlp4 can reach a fixpoint or not, but now you
can compile your modification within seconds, and reach a fixpoint under 2
minutes, this accelerate the development cycles immediately.*
*   2. Now you can customize your lexer now, previously it's impossible(
and a number of bug fixes) you can do deep anti-quotation like *
*  <:expr< <:expr< $($(deepantiquot)) >> >> *
   3. Linking the compiler and a number of cool features (see the slides)
   4. A macro which write macros to scrap all the bolierpolate code for
generic programming, now you can customize your deriving stuff  in ten
lines (previously thousands of lines of code to write camlp4 plugin)
    5. A number of mini-DSLs  and more to be expected
   I love macros, I would commit to the project for a long time(probably my
Ph.D term), I promise that I would write the documentation to make users
happy( I am also writing a book about macros, it's un-readable though). And
I am really happy to co-laborate with anyone who love camlp4.
   Syntax matters, if you see that coffescript is now already a success,
camlp4 or Fan is actually more expressive than that, Dear ocaml
programmers, let us find a way to make ocaml more beautiful. I am also open
minded to any discussion about the future of camlp4
   Many thanks (sorry for my bad English)
-- 
Regards
-- Bob

[-- Attachment #2: Type: text/html, Size: 3165 bytes --]

Re: [Caml-list] Call for collaboration on the future of camlp4

[Wojciech Meyer]

Hi Hongboz,

Thanks for the slides, it's good. Indeed it makes sense now to focus on
extending Camlp4, however as usually there are some gotchas, the process
of extending syntax with Camlp4 is a bit not well known.  Camlp5 has an
excellent documentation, and great support for some of the things. Yet,
Camlp4 is more modular, and is just easier to write application on top
of Camlp4.

On OUD we had open ended discussions, and many people share (like me)
the same impression - -ppx that exist on trunk is powerful and that's
the way to implement most of the meta programming facilities (deriving
the code from type definitions, or using quotations to embed DSLs in
OCaml), but I believe that the direct syntax extensions (opening the
OCaml syntax) certainly has some benefits. How to do this in a clean way
myself - I don't know. I posted just some time ago what would be a macro
system of my dreams:

https://sympa.inria.fr/sympa/arc/caml-list/2012-05/msg00184.html

If the current Camlp4 could be used to do this, it would be great but
that's need to be somewhat kept up to date with all up-coming compiler
meta programming facilities.

I will have a closer look at the Fan project today.

Thanks,

--
Wojciech Meyer
http://danmey.org


bob zhang <bobzhang1988@gmail.com> writes:

> Dear OCaml programmers,
>
>    Last week, I give a talk about the future of Camlp4 in the ML
> workshop, the slides are here
> http://www.lexifi.com/ml2012/slides_panel_hongbo.pdf (some are already
> done, some are work in progress).
>    For a long time, in the caml community, thanks to the talented work
> of Daniel and Michel,  we know there is a very very powerful tool
> called *camlp4*, if there are some bolierpolate code you write here
> and there, someone will tell you "hey, you can do it in camlp4" though
> he may not know how to do it in camlp4. 
>    But it's a bit embarassing that camlp4 did not evolve very well
> (partly due to the fragmentation of camlp[4,5]), another fact is that
> camlp4 is not *designed*, it's like a prototype that works but not
> carefully designed, and it does not provide anything out of the box
>  and itself was written in a verbose way.
>    It's time to bring the powerful tool back, my advisor Steve and I
> started a new project Fan, which is mainly to evolve the camlp4 macro
> system to be more expressive and more powerful, push the Camlp4 to the
> next level. For me, I am a long-term Lisp programmer, I appreciated
> the value of macros, I would really be happy to see we could make a
> such powerful macro systems.
>    Here is my repo https://bitbucket.org/HongboZhang/camlp4
>    I already finished some cool staff,  to mention just a few:
>    1. A very robust bootstrapping system, previously it takes me 20
> minutes to verify my patches to camlp4 can reach a fixpoint or not,
> but now you can compile your modification within seconds, and reach a
> fixpoint under 2 minutes, this accelerate the development cycles
> immediately.
>    2. Now you can customize your lexer now, previously it's impossible
> ( and a number of bug fixes) you can do deep anti-quotation like 
>   <:expr< <:expr< $($(deepantiquot)) >> >> 
>    3. Linking the compiler and a number of cool features (see the
> slides)
>    4. A macro which write macros to scrap all the bolierpolate code
> for generic programming, now you can customize your deriving stuff  in
> ten lines (previously thousands of lines of code to write camlp4
> plugin)
>     5. A number of mini-DSLs  and more to be expected
>    I love macros, I would commit to the project for a long time
> (probably my Ph.D term), I promise that I would write the
> documentation to make users happy( I am also writing a book about
> macros, it's un-readable though). And I am really happy to co-laborate
> with anyone who love camlp4. 
>    Syntax matters, if you see that coffescript is now already a
> success, camlp4 or Fan is actually more expressive than that, Dear
> ocaml programmers, let us find a way to make ocaml more beautiful. I
> am also open minded to any discussion about the future of camlp4
>    Many thanks (sorry for my bad English)

[Caml-list] Re: Call for collaboration on the future of camlp4

[Hongbo Zhang]

On 9/19/12 3:55 PM, Wojciech Meyer wrote:
> Hi Hongboz,
>
Greetings,
> Thanks for the slides, it's good. Indeed it makes sense now to focus on
> extending Camlp4, however as usually there are some gotchas, the process
> of extending syntax with Camlp4 is a bit not well known.  Camlp5 has an
> excellent documentation, and great support for some of the things. Yet,
> Camlp4 is more modular, and is just easier to write application on top
> of Camlp4.
>
Yes, camlp5 documentation is much more comprehensive for the users.
But the underlying mechanism is not explained either, I did not see any
difference for low-level users, another problem is that camlp5 is
  not heavily tested.

> On OUD we had open ended discussions, and many people share (like me)
> the same impression - -ppx that exist on trunk is powerful and that's
> the way to implement most of the meta programming facilities (deriving
> the code from type definitions, or using quotations to embed DSLs in
> OCaml), but I believe that the direct syntax extensions (opening the
> OCaml syntax) certainly has some benefits. How to do this in a clean way
> myself - I don't know. I posted just some time ago what would be a macro
> system of my dreams:
-ppx is orthogonal to what camlp4 did, it's pretty easy to integrate
'ast rewriter' in Fan or camlp4. There are 3 problems with Ast Rewriter

1. No Quasiquotaion support
    It's ok to construct the Ast by hand for the first order macros,
    as I pointed out before, you can not write macros which write macros
    without quasi-quotation support. Remember that macros are itself 
programs,
    and there are dupliated code in macros as well, so you still need to
    scrap those dupliated code. So the expressivity to Ast Rewriter is like
    higher-order programming to first-order programming. It's a shame 
that so far
    there are not too many camlp4 plugins which generate macros to 
remove the duplicated
    code for camlp4 plugins. The camlp4 source tree itself is also 
written in a verbose way.
    Adding Quasiquotation support directly to  current parsetree is not 
an easy way,
    for macro programming, you really want to quote, and antiquote 
everywhere, and
    create some illegal Ast. I have programmed  a lot in Template 
Haskell, and I
    really appreciate that camlp4 have a much better quote-antiquot support.

2. No Delimited Syntax Extension support
    The Ast Rewriter can only override existing syntax(which is a bit 
fragile, IMO). To make
    things worse, if you want to link the compiler to your 
pre-processor, you really want to
    mark which piece of code should be eval during pre-processing  or 
delayed at run-time. So,
    even the minimal delimited syntax extension will increase the 
expressivety a lot

3. General Syntax Extension
    General Syntax Extension is definitely very useful when I finished 
the functional parser part.
    A problem with delimited syntax extension support is that sometimes 
you really want to mix the
    existing grammars with your new grammars, currently there's no easy 
way to do that in delimited
    syntax extension. When we finished functional parser, we can 
restrict the syntax extension scope
    to the minimal expression level

Btw, all the problems that camlp4 or Fan came across will not disappear 
in Ast Rewriter, you don't get
any benefit from them, hygenic, no, the same problem.
The beauty of camlp4 is that it is bootstrapped itself, when you evolve 
camlp4, you are making a more and more
expressive system, part of my work is to re-write camlp4 to make it much 
more succinct. :-)
>
> https://sympa.inria.fr/sympa/arc/caml-list/2012-05/msg00184.html
About your proposals:

Runtime meta-programming is a generalisation of static meta
programming. MetaOCaml has a nice set of abstraction to generate
typechecking code - yes - either at runtime or during compile time.

Meta-ocaml is orthogonal to camlp4 as well, but I did not see any hope 
that it will be pushed into ocaml compiler,
since the main purpose of meta-ocaml is to do some run-time 
optimization, without a native jit compiler support, It does
not make sense.

" - first of all non destructive updates to the grammar e.g: "let open
lang Sexp in ..." should open the Sexp syntax extension, install the
grammar, but when it goes out of scope it should vanish. Currently
Camlp4 can install, delete the rules after the functor is applied, and
no way of saying OK - let's go back."
Yes!

- Composable - in particular one language should behave like a module,
or functor, should have an interface consisting of grammar rules, AST,
AST transforms etc. So one could parametrise one syntax extension
over another, and possibly reuse the language grammar or AST in
other. Currently Camlp4 syntax extension is just a single separate
module which when loaded possibly expects some existing grammar rules
to be in place and mutates them as it's needed.

Yes!

- should be type safe and as mentioned before obey scoping rules. We
should be able to propagate type information even when the syntax
changes. This is difficult part - but I've seen it can be done with some
extra annotations - not talking about Camlp4
Personally, I prefer common lisp's macros to scheme. It's painful
to play with racket's macros as well

- Recursive - it should be able to apply the grammar rules not only
once but expand until it reached the fixpoint.

Yes!
- Reflective - it should be possible after each successful expansion have
the type information available for the next expansion.
Possibly with the compiler library help

- Grammar itself should be lexer-less - memoizing PEG with left
recursion - it's hurdle to define new grammar in terms of old lexer,
or having a stateful lexer that depends on context.

Yes!
- It should not be external tool - like previously observed - it's
difficult to support for code highlighters or refactoring (tools in
general) - if it depends on a build step or command line options.
Yes!

[Caml-list] Re: Call for collaboration on the future of camlp4

[Hongbo Zhang]

On 9/19/12 5:21 PM, Hongbo Zhang wrote:
> On 9/19/12 3:55 PM, Wojciech Meyer wrote:

> - It should not be external tool - like previously observed - it's
> difficult to support for code highlighters or refactoring (tools in
> general) - if it depends on a build step or command line options.
> Yes!
Sorry, a bit typo. IMHO, making it an external tool gains a lot of 
flexibility. The main purpose of macros is to define your own language 
without the support of compiler. If it's tightly coupled with compiler, 
any minimal change will be delayed. Given the compiler library was 
exported, I don't see any sense that P4 still stays in the compiler.
Regards, Bob
>
>
>

[Caml-list] Re: Call for collaboration on the future of camlp4

[bobzhang]

Dear all,
For those who are concerned:
   The latest version of building camlp4 is under 8s.
real	0m7.653s
user	0m5.595s
sys	0m1.736s
On 9/18/12 3:11 PM, bob zhang wrote:
> Dear OCaml programmers,
>     Last week, I give a talk about the future of Camlp4 in the ML
> workshop, the slides are here
> http://www.lexifi.com/ml2012/slides_panel_hongbo.pdf (some are already
> done, some are work in progress).
>     For a long time, in the caml community, thanks to the talented work
> of Daniel and Michel,  we know there is a very very powerful tool called
> *camlp4*, if there are some bolierpolate code you write here and there,
> someone will tell you "hey, you can do it in camlp4" though he may not
> know how to do it in camlp4.
>     But it's a bit embarassing that camlp4 did not evolve very
> well(partly due to the fragmentation of camlp[4,5]), another fact is
> that camlp4 is not *designed*, it's like a prototype that works but not
> carefully designed, and it does not provide anything out of the box  and
> itself was written in a verbose way.
>     It's time to bring the powerful tool back, my advisor Steve and I
> started a new project Fan, which is mainly to evolve the camlp4 macro
> system to be more expressive and more powerful, push the Camlp4 to the
> next level. For me, I am a long-term Lisp programmer, I appreciated the
> value of macros, I would really be happy to see we could make a such
> powerful macro systems.
>     Here is my repo https://bitbucket.org/HongboZhang/camlp4
>     I already finished some cool staff,  to mention just a few:
>     1. A very robust bootstrapping system, previously it takes me 20
> minutes to ver/ify my patches to camlp4 can reach a fixpoint or not, but
> now you can compile your modification within seconds, and reach a
> fixpoint under 2 minutes, this accelerate the development cycles
> immediately./
> /   2. Now you can customize your lexer now, previously it's impossible(
> and a number of bug fixes) you can do deep anti-quotation like /
> /  <:expr< <:expr< $($(deepantiquot)) >> >> /
>     3. Linking the compiler and a number of cool features (see the slides)
>     4. A macro which write macros to scrap all the bolierpolate code for
> generic programming, now you can customize your deriving stuff  in ten
> lines (previously thousands of lines of code to write camlp4 plugin)
>      5. A number of mini-DSLs  and more to be expected
>     I love macros, I would commit to the project for a long
> time(probably my Ph.D term), I promise that I would write the
> documentation to make users happy( I am also writing a book about
> macros, it's un-readable though). And I am really happy to co-laborate
> with anyone who love camlp4.
>     Syntax matters, if you see that coffescript is now already a
> success, camlp4 or Fan is actually more expressive than that, Dear ocaml
> programmers, let us find a way to make ocaml more beautiful. I am also
> open minded to any discussion about the future of camlp4
>     Many thanks (sorry for my bad English)
> --
> Regards
> -- Bob

[Caml-list] Call for collaboration on the future of camlp4

[oleg]

Hongbo Zhang wrote:
  > Meta-ocaml is orthogonal to camlp4 as well, but I did not see any hope
  > that it will be pushed into ocaml compiler,
  > since the main purpose of meta-ocaml is to do some run-time
  > optimization, without a native jit compiler support, It does
  > not make sense.

I believe you might be short-changing MetaOCaml. First of all, why
MetaOCaml needs JIT? It is already possible in MetaOCaml to generate
code at run-time, to compile it and link-in. The new code is generated
by the OCaml compiler itself, and works as if it were part of the
original code. Such run-time optimization can indeed be useful (e.g.,
profile-based optimization). I don't see why we need JIT if we already
have the original compiler itself.

However viewing MetaOCaml as a run-time optimizer misses a
lot. MetaOCaml is a code generator with certain assurances of
correctness -- it generates code that one would not write by hand. The
recent example of generating code for stencil computations in
high-performance computing

	http://okmij.org/ftp/meta-programming/HPC.html#stencil

was quite illustrative: MetaOCaml has generated code for edge
cases that human really did not want to write (and did not actually
write). Edge cases are really tedious and really difficult to get
right; automation is of great help. The generated code can be
printed-out and used in a project as any other code. MetaOCaml can be
used to generate libraries of specialized high-performance code.

  > Personally, I prefer common lisp's macros to scheme. It's painful
  > to play with racket's macros as well

Well, it is painful for some people to use typed languages
too. Untyped languages offer so much freedom, some think. What is
often forgotten is that the benefit of the type system is not in what
code you can write, but what code you cannot. Likewise, a syntax
extension should be evaluated not only on what it allows but also on
what it prevents.


As I general remark: it is not clear why so much fixation on
camlp4. Other people have tried extensible syntaxes. For example:

	- Felix
	- Notation of Coq
	- Stratego
	http://strategoxt.org/
In particular, see
	http://strategoxt.org/Sdf/SdfLanguage

Stratego could do program transformations in concrete syntax. Maybe
that would be a better approach? Maybe Stratego would satisfy all our
needs?