[Caml-list] Call for collaboration on the future of camlp4

[Caml-list] Call for collaboration on the future of camlp4

[oleg]

Hongbo Zhang wrote:
  > Meta-ocaml is orthogonal to camlp4 as well, but I did not see any hope
  > that it will be pushed into ocaml compiler,
  > since the main purpose of meta-ocaml is to do some run-time
  > optimization, without a native jit compiler support, It does
  > not make sense.

I believe you might be short-changing MetaOCaml. First of all, why
MetaOCaml needs JIT? It is already possible in MetaOCaml to generate
code at run-time, to compile it and link-in. The new code is generated
by the OCaml compiler itself, and works as if it were part of the
original code. Such run-time optimization can indeed be useful (e.g.,
profile-based optimization). I don't see why we need JIT if we already
have the original compiler itself.

However viewing MetaOCaml as a run-time optimizer misses a
lot. MetaOCaml is a code generator with certain assurances of
correctness -- it generates code that one would not write by hand. The
recent example of generating code for stencil computations in
high-performance computing

	http://okmij.org/ftp/meta-programming/HPC.html#stencil

was quite illustrative: MetaOCaml has generated code for edge
cases that human really did not want to write (and did not actually
write). Edge cases are really tedious and really difficult to get
right; automation is of great help. The generated code can be
printed-out and used in a project as any other code. MetaOCaml can be
used to generate libraries of specialized high-performance code.

  > Personally, I prefer common lisp's macros to scheme. It's painful
  > to play with racket's macros as well

Well, it is painful for some people to use typed languages
too. Untyped languages offer so much freedom, some think. What is
often forgotten is that the benefit of the type system is not in what
code you can write, but what code you cannot. Likewise, a syntax
extension should be evaluated not only on what it allows but also on
what it prevents.


As I general remark: it is not clear why so much fixation on
camlp4. Other people have tried extensible syntaxes. For example:

	- Felix
	- Notation of Coq
	- Stratego
	http://strategoxt.org/
In particular, see
	http://strategoxt.org/Sdf/SdfLanguage

Stratego could do program transformations in concrete syntax. Maybe
that would be a better approach? Maybe Stratego would satisfy all our
needs?

[Caml-list] Re: Call for collaboration on the future of camlp4

[Hongbo Zhang]

[-- Attachment #1: Type: text/plain, Size: 4415 bytes --]

On 9/20/12 5:36 AM, oleg@okmij.org wrote:
> Hongbo Zhang wrote:
>    > Meta-ocaml is orthogonal to camlp4 as well, but I did not see any hope
>    > that it will be pushed into ocaml compiler,
>    > since the main purpose of meta-ocaml is to do some run-time
>    > optimization, without a native jit compiler support, It does
>    > not make sense.
>
> I believe you might be short-changing MetaOCaml. First of all, why
> MetaOCaml needs JIT? It is already possible in MetaOCaml to generate
> code at run-time, to compile it and link-in. The new code is generated
> by the OCaml compiler itself, and works as if it were part of the
> original code. Such run-time optimization can indeed be useful (e.g.,
> profile-based optimization). I don't see why we need JIT if we already
> have the original compiler itself.
>
> However viewing MetaOCaml as a run-time optimizer misses a
> lot. MetaOCaml is a code generator with certain assurances of
> correctness -- it generates code that one would not write by hand. The
> recent example of generating code for stencil computations in
> high-performance computing
Yes, it's a run-time optimizer with type safe assurance. It can do
partial evaluation to generate some optimized code. With the help
of compiler-library, I can do byte-code eval(not native eval), I agree
it would be useful to have a native eval, but this requires non-trivial
changes to the compiler which I don't expect it will be realized in a
short term.
> 	http://okmij.org/ftp/meta-programming/HPC.html#stencil
>
> was quite illustrative: MetaOCaml has generated code for edge
> cases that human really did not want to write (and did not actually
> write). Edge cases are really tedious and really difficult to get
> right; automation is of great help. The generated code can be
> printed-out and used in a project as any other code. MetaOCaml can be
> used to generate libraries of specialized high-performance code.
>
>    > Personally, I prefer common lisp's macros to scheme. It's painful
>    > to play with racket's macros as well
>
> Well, it is painful for some people to use typed languages
> too. Untyped languages offer so much freedom, some think. What is
> often forgotten is that the benefit of the type system is not in what
> code you can write, but what code you cannot. Likewise, a syntax
> extension should be evaluated not only on what it allows but also on
> what it prevents.
Well, as I said, it's a personal taste. People are /*too cautiou*/s about
the safety the code generated, if you take a look at other languages,
like D, they *even use "string concatenation" to generate code*,*/we 
manipulate/**/
/**/the Ast instead of strings, and finally it's type checked, in 
practice, it works/**/
/**/pretty well/* and I wrote tens of thousands of code on top of 
camlp4, none of the bugs
can be detected by type checking. If you take a look at the
history of Template Haskell, they finally step back from type checking 
everything to
give up type checking some quasi-quotations. We don't want to sacrifice 
too much
experssivity for type safety, this is especially important in macros. In 
common lisp,
there is also a kind of macros called "Anaphoric macros" which you will 
find painful to
do in Scheme. To overcome the un-hygenic problem, if you have a look at 
the book "let over lambda",
there's best practice to avoid such problems. And you take a look at the 
compiler of opa,
  they override the module Stream to make use of the syntax extension 
what camlp4 can provide,
that's the beauty of syntax meta-programming, instead of a bug.

I really appreciate that ocaml's type checker can finally type check the 
generated code, but do
we really need to type check the macros? Would the cost be too high?
>
> As I general remark: it is not clear why so much fixation on
> camlp4. Other people have tried extensible syntaxes. For example:
>
> 	- Felix
Some ideas of Felix are neat that we should borrow from.
> 	- Notation of Coq
This is built on top of Camlp5

	- Stratego
	http://strategoxt.org/
In particular, see
	http://strategoxt.org/Sdf/SdfLanguage

Stratego could do program transformations in concrete syntax. Maybe
that would be a better approach? Maybe Stratego would satisfy all our
needs?

Well, this is definitely not true. The beauty of Camlp4 is that it is a 
growing macro system instead of
only a program transformation tool.

Thanks for your message.
>
>


[-- Attachment #2: Type: text/html, Size: 5974 bytes --]

[Caml-list] Re: Call for collaboration on the future of camlp4

[oleg]

> Yes, it's [MetaOCaml] a run-time optimizer with type safe assurance. It can do
> partial evaluation to generate some optimized code. 

Let me stress once again how narrow this view is. MetaOCaml goes well
beyond partial evaluation. For example, MetaOCaml, as a general code
generation framework, was used to derive optimal (in the number of
multiplications) FFT kernels. Partial evaluation will not give you
that.

Code generation is a very promising technique in High-Performance
computing. Most of the tools used in practice -- FFTW, ATLAS, SPIRAL
-- are all off-line tools. They generate a large number of candidate
codes and choose the best performing. What's important is to quickly
generate a large number of very tedious programs. Assurance of
correctness are important: a programmer, especially a domain expert,
will not want to even look at the generated code let alone debug it.
I see MetaOCaml target the same area.

> I agree it would be useful to have a native eval, but this requires
> non-trivial changes to the compiler which I don't expect it will be
> realized in a short term.
The assessment is mistaken. MetaOCaml v3.09 did have a native
back-end. I know quite well what changes were required. Those changes
are no longer needed since dynamic linking has since become part of
OCaml proper.

> If you take a look at the history of Template Haskell, they finally
> step back from type checking everything to give up type checking some
> quasi-quotations. 

This is a mistaken impression. While Template Haskell as a whole will
remain untyped for a long time -- after all, Template Haskell can
generate data and type class _declarations_, whose typing is far from
clear -- there is a definite push towards MetaOCaml-like type safety
for expressions. 
  http://hackage.haskell.org/trac/ghc/blog/Template%20Haskell%20Proposal

see especially
  Part B: Add new MetaML-style constructs for strongly-typed metaprogramming.

If this is implemented, TH becomes quite like MetaML.

> We don't want to sacrifice too much experssivity for type safety, this
> is especially important in macros. In common lisp, there is also a
> kind of macros called "Anaphoric macros" which you will find painful
> to do in Scheme.

That is not a very good argument since R5RS macros in Scheme were
intentionally limited in their expressivity. The macro system was
designed to be just enough expressive for the special forms
introduced in the Report. (Later on the system was found to be quite
more expressive than its designers have anticipated.)

The anaphoric macros are easily expressible in the system of our JFP
2011 paper (staging with a very limited delimited control). No
subversions of hygiene are needed.

[Caml-list] Re: Call for collaboration on the future of camlp4

[Hongbo Zhang]

[-- Attachment #1: Type: text/plain, Size: 3958 bytes --]

On 9/22/12 3:50 AM, oleg@okmij.org wrote:
>> Yes, it's [MetaOCaml] a run-time optimizer with type safe assurance. It can do
>> partial evaluation to generate some optimized code.
> Let me stress once again how narrow this view is. MetaOCaml goes well
> beyond partial evaluation. For example, MetaOCaml, as a general code
> generation framework, was used to derive optimal (in the number of
> multiplications) FFT kernels. Partial evaluation will not give you
> that.
>
> Code generation is a very promising technique in High-Performance
> computing. Most of the tools used in practice -- FFTW, ATLAS, SPIRAL
> -- are all off-line tools. They generate a large number of candidate
> codes and choose the best performing. What's important is to quickly
> generate a large number of very tedious programs. Assurance of
> correctness are important: a programmer, especially a domain expert,
> will not want to even look at the generated code let alone debug it.
> I see MetaOCaml target the same area.
>
To my limited knowledge, FFTW used techniques more like Camlp4.
Again: type safety(without dependent types) is far from correctness,
actually, the type error bugs are the easiest to fix. In practice,
you always need to debug the generated code if something goes wrong.
One thing people complains about template c++ code is that it's really
hard to debug.

>> I agree it would be useful to have a native eval, but this requires
>> non-trivial changes to the compiler which I don't expect it will be
>> realized in a short term.
> The assessment is mistaken. MetaOCaml v3.09 did have a native
> back-end. I know quite well what changes were required. Those changes
> are no longer needed since dynamic linking has since become part of
> OCaml proper.
Actually, I am one of the curious programmers who tried MetaOCaml and 
played
with it. Don't get me wrong, I would be happy to see MetaOCaml pushed 
into OCaml.
As I said before, it's orthogonal to P4, it's great if we could 
dynamically generate
type safe code, and we can make use of meta-ocaml as well ;-)
But the assumption is /*that we have a native meta-ocaml without 
patching the compiler */
>> If you take a look at the history of Template Haskell, they finally
>> step back from type checking everything to give up type checking some
>> quasi-quotations.
> This is a mistaken impression. While Template Haskell as a whole will
> remain untyped for a long time -- after all, Template Haskell can
> generate data and type class _declarations_, whose typing is far from
> clear -- there is a definite push towards MetaOCaml-like type safety
> for expressions.
>    http://hackage.haskell.org/trac/ghc/blog/Template%20Haskell%20Proposal
>
> see especially
>    Part B: Add new MetaML-style constructs for strongly-typed metaprogramming.
>
> If this is implemented, TH becomes quite like MetaML.
There are two problems here, first you object language is not only ocaml,
second, yes, we want /*everything to be first class, everything*/, the most
expressive system.
>> We don't want to sacrifice too much experssivity for type safety, this
>> is especially important in macros. In common lisp, there is also a
>> kind of macros called "Anaphoric macros" which you will find painful
>> to do in Scheme.
> That is not a very good argument since R5RS macros in Scheme were
> intentionally limited in their expressivity. The macro system was
> designed to be just enough expressive for the special forms
> introduced in the Report. (Later on the system was found to be quite
> more expressive than its designers have anticipated.)
>
> The anaphoric macros are easily expressible in the system of our JFP
> 2011 paper (staging with a very limited delimited control). No
> subversions of hygiene are needed.
That depends on how you define 'easily expressible' ;-)
Let's discuss in private, btw, it's a bit unfair that you just picked
some points and ignored others without context.

Thanks for your message.
>


[-- Attachment #2: Type: text/html, Size: 5533 bytes --]

Re: [Caml-list] Re: Call for collaboration on the future of camlp4

[Jacques Carette]

I emphatically agree with Oleg.  I'll reinforce on point regarding the 
competition:

On 22/09/2012 3:50 AM, oleg@okmij.org wrote:
>   [...] there is a definite push towards MetaOCaml-like type safety
> for expressions.
>    http://hackage.haskell.org/trac/ghc/blog/Template%20Haskell%20Proposal
>
> see especially
>    Part B: Add new MetaML-style constructs for strongly-typed metaprogramming.

This is being implemented NOW.  Simon Peyton Jones is doing the ghc 
work, and I am porting the metaocaml codes I have to serve as test 
cases.  Simon announced that this 'typed quotes and splices' will be in 
GHC 7.8 at the Haskell Workshop.

Jacques

Re: [Caml-list] Re: Call for collaboration on the future of camlp4

[Hongbo Zhang]

On 9/22/12 8:53 AM, Jacques Carette wrote:
> I emphatically agree with Oleg.  I'll reinforce on point regarding the 
> competition:
>
> On 22/09/2012 3:50 AM, oleg@okmij.org wrote:
>>   [...] there is a definite push towards MetaOCaml-like type safety
>> for expressions.
>> http://hackage.haskell.org/trac/ghc/blog/Template%20Haskell%20Proposal
>>
>> see especially
>>    Part B: Add new MetaML-style constructs for strongly-typed 
>> metaprogramming.
>
> This is being implemented NOW.  Simon Peyton Jones is doing the ghc 
> work, and I am porting the metaocaml codes I have to serve as test 
> cases.  Simon announced that this 'typed quotes and splices' will be 
> in GHC 7.8 at the Haskell Workshop.
>
> Jacques
Yes, I agree that it would much nicer we have such facility in ocaml, no 
competition..

[Caml-list] Call for collaboration on the future of camlp4

[bob zhang]

[-- Attachment #1: Type: text/plain, Size: 2773 bytes --]

Dear OCaml programmers,
   Last week, I give a talk about the future of Camlp4 in the ML workshop,
the slides are here
http://www.lexifi.com/ml2012/slides_panel_hongbo.pdf(some are already
done, some are work in progress).
   For a long time, in the caml community, thanks to the talented work of
Daniel and Michel,  we know there is a very very powerful tool called
*camlp4*, if there are some bolierpolate code you write here and there,
someone will tell you "hey, you can do it in camlp4" though he may not know
how to do it in camlp4.
   But it's a bit embarassing that camlp4 did not evolve very well(partly
due to the fragmentation of camlp[4,5]), another fact is that camlp4 is not
*designed*, it's like a prototype that works but not carefully designed,
and it does not provide anything out of the box  and itself was written in
a verbose way.
   It's time to bring the powerful tool back, my advisor Steve and I
started a new project Fan, which is mainly to evolve the camlp4 macro
system to be more expressive and more powerful, push the Camlp4 to the next
level. For me, I am a long-term Lisp programmer, I appreciated the value of
macros, I would really be happy to see we could make a such powerful macro
systems.
   Here is my repo https://bitbucket.org/HongboZhang/camlp4
   I already finished some cool staff,  to mention just a few:
   1. A very robust bootstrapping system, previously it takes me 20 minutes
to ver*ify my patches to camlp4 can reach a fixpoint or not, but now you
can compile your modification within seconds, and reach a fixpoint under 2
minutes, this accelerate the development cycles immediately.*
*   2. Now you can customize your lexer now, previously it's impossible(
and a number of bug fixes) you can do deep anti-quotation like *
*  <:expr< <:expr< $($(deepantiquot)) >> >> *
   3. Linking the compiler and a number of cool features (see the slides)
   4. A macro which write macros to scrap all the bolierpolate code for
generic programming, now you can customize your deriving stuff  in ten
lines (previously thousands of lines of code to write camlp4 plugin)
    5. A number of mini-DSLs  and more to be expected
   I love macros, I would commit to the project for a long time(probably my
Ph.D term), I promise that I would write the documentation to make users
happy( I am also writing a book about macros, it's un-readable though). And
I am really happy to co-laborate with anyone who love camlp4.
   Syntax matters, if you see that coffescript is now already a success,
camlp4 or Fan is actually more expressive than that, Dear ocaml
programmers, let us find a way to make ocaml more beautiful. I am also open
minded to any discussion about the future of camlp4
   Many thanks (sorry for my bad English)
-- 
Regards
-- Bob

[-- Attachment #2: Type: text/html, Size: 3165 bytes --]

Re: [Caml-list] Call for collaboration on the future of camlp4

[Wojciech Meyer]

Hi Hongboz,

Thanks for the slides, it's good. Indeed it makes sense now to focus on
extending Camlp4, however as usually there are some gotchas, the process
of extending syntax with Camlp4 is a bit not well known.  Camlp5 has an
excellent documentation, and great support for some of the things. Yet,
Camlp4 is more modular, and is just easier to write application on top
of Camlp4.

On OUD we had open ended discussions, and many people share (like me)
the same impression - -ppx that exist on trunk is powerful and that's
the way to implement most of the meta programming facilities (deriving
the code from type definitions, or using quotations to embed DSLs in
OCaml), but I believe that the direct syntax extensions (opening the
OCaml syntax) certainly has some benefits. How to do this in a clean way
myself - I don't know. I posted just some time ago what would be a macro
system of my dreams:

https://sympa.inria.fr/sympa/arc/caml-list/2012-05/msg00184.html

If the current Camlp4 could be used to do this, it would be great but
that's need to be somewhat kept up to date with all up-coming compiler
meta programming facilities.

I will have a closer look at the Fan project today.

Thanks,

--
Wojciech Meyer
http://danmey.org


bob zhang <bobzhang1988@gmail.com> writes:

> Dear OCaml programmers,
>
>    Last week, I give a talk about the future of Camlp4 in the ML
> workshop, the slides are here
> http://www.lexifi.com/ml2012/slides_panel_hongbo.pdf (some are already
> done, some are work in progress).
>    For a long time, in the caml community, thanks to the talented work
> of Daniel and Michel,  we know there is a very very powerful tool
> called *camlp4*, if there are some bolierpolate code you write here
> and there, someone will tell you "hey, you can do it in camlp4" though
> he may not know how to do it in camlp4. 
>    But it's a bit embarassing that camlp4 did not evolve very well
> (partly due to the fragmentation of camlp[4,5]), another fact is that
> camlp4 is not *designed*, it's like a prototype that works but not
> carefully designed, and it does not provide anything out of the box
>  and itself was written in a verbose way.
>    It's time to bring the powerful tool back, my advisor Steve and I
> started a new project Fan, which is mainly to evolve the camlp4 macro
> system to be more expressive and more powerful, push the Camlp4 to the
> next level. For me, I am a long-term Lisp programmer, I appreciated
> the value of macros, I would really be happy to see we could make a
> such powerful macro systems.
>    Here is my repo https://bitbucket.org/HongboZhang/camlp4
>    I already finished some cool staff,  to mention just a few:
>    1. A very robust bootstrapping system, previously it takes me 20
> minutes to verify my patches to camlp4 can reach a fixpoint or not,
> but now you can compile your modification within seconds, and reach a
> fixpoint under 2 minutes, this accelerate the development cycles
> immediately.
>    2. Now you can customize your lexer now, previously it's impossible
> ( and a number of bug fixes) you can do deep anti-quotation like 
>   <:expr< <:expr< $($(deepantiquot)) >> >> 
>    3. Linking the compiler and a number of cool features (see the
> slides)
>    4. A macro which write macros to scrap all the bolierpolate code
> for generic programming, now you can customize your deriving stuff  in
> ten lines (previously thousands of lines of code to write camlp4
> plugin)
>     5. A number of mini-DSLs  and more to be expected
>    I love macros, I would commit to the project for a long time
> (probably my Ph.D term), I promise that I would write the
> documentation to make users happy( I am also writing a book about
> macros, it's un-readable though). And I am really happy to co-laborate
> with anyone who love camlp4. 
>    Syntax matters, if you see that coffescript is now already a
> success, camlp4 or Fan is actually more expressive than that, Dear
> ocaml programmers, let us find a way to make ocaml more beautiful. I
> am also open minded to any discussion about the future of camlp4
>    Many thanks (sorry for my bad English)

[Caml-list] Re: Call for collaboration on the future of camlp4

[Hongbo Zhang]

On 9/19/12 3:55 PM, Wojciech Meyer wrote:
> Hi Hongboz,
>
Greetings,
> Thanks for the slides, it's good. Indeed it makes sense now to focus on
> extending Camlp4, however as usually there are some gotchas, the process
> of extending syntax with Camlp4 is a bit not well known.  Camlp5 has an
> excellent documentation, and great support for some of the things. Yet,
> Camlp4 is more modular, and is just easier to write application on top
> of Camlp4.
>
Yes, camlp5 documentation is much more comprehensive for the users.
But the underlying mechanism is not explained either, I did not see any
difference for low-level users, another problem is that camlp5 is
  not heavily tested.

> On OUD we had open ended discussions, and many people share (like me)
> the same impression - -ppx that exist on trunk is powerful and that's
> the way to implement most of the meta programming facilities (deriving
> the code from type definitions, or using quotations to embed DSLs in
> OCaml), but I believe that the direct syntax extensions (opening the
> OCaml syntax) certainly has some benefits. How to do this in a clean way
> myself - I don't know. I posted just some time ago what would be a macro
> system of my dreams:
-ppx is orthogonal to what camlp4 did, it's pretty easy to integrate
'ast rewriter' in Fan or camlp4. There are 3 problems with Ast Rewriter

1. No Quasiquotaion support
    It's ok to construct the Ast by hand for the first order macros,
    as I pointed out before, you can not write macros which write macros
    without quasi-quotation support. Remember that macros are itself 
programs,
    and there are dupliated code in macros as well, so you still need to
    scrap those dupliated code. So the expressivity to Ast Rewriter is like
    higher-order programming to first-order programming. It's a shame 
that so far
    there are not too many camlp4 plugins which generate macros to 
remove the duplicated
    code for camlp4 plugins. The camlp4 source tree itself is also 
written in a verbose way.
    Adding Quasiquotation support directly to  current parsetree is not 
an easy way,
    for macro programming, you really want to quote, and antiquote 
everywhere, and
    create some illegal Ast. I have programmed  a lot in Template 
Haskell, and I
    really appreciate that camlp4 have a much better quote-antiquot support.

2. No Delimited Syntax Extension support
    The Ast Rewriter can only override existing syntax(which is a bit 
fragile, IMO). To make
    things worse, if you want to link the compiler to your 
pre-processor, you really want to
    mark which piece of code should be eval during pre-processing  or 
delayed at run-time. So,
    even the minimal delimited syntax extension will increase the 
expressivety a lot

3. General Syntax Extension
    General Syntax Extension is definitely very useful when I finished 
the functional parser part.
    A problem with delimited syntax extension support is that sometimes 
you really want to mix the
    existing grammars with your new grammars, currently there's no easy 
way to do that in delimited
    syntax extension. When we finished functional parser, we can 
restrict the syntax extension scope
    to the minimal expression level

Btw, all the problems that camlp4 or Fan came across will not disappear 
in Ast Rewriter, you don't get
any benefit from them, hygenic, no, the same problem.
The beauty of camlp4 is that it is bootstrapped itself, when you evolve 
camlp4, you are making a more and more
expressive system, part of my work is to re-write camlp4 to make it much 
more succinct. :-)
>
> https://sympa.inria.fr/sympa/arc/caml-list/2012-05/msg00184.html
About your proposals:

Runtime meta-programming is a generalisation of static meta
programming. MetaOCaml has a nice set of abstraction to generate
typechecking code - yes - either at runtime or during compile time.

Meta-ocaml is orthogonal to camlp4 as well, but I did not see any hope 
that it will be pushed into ocaml compiler,
since the main purpose of meta-ocaml is to do some run-time 
optimization, without a native jit compiler support, It does
not make sense.

" - first of all non destructive updates to the grammar e.g: "let open
lang Sexp in ..." should open the Sexp syntax extension, install the
grammar, but when it goes out of scope it should vanish. Currently
Camlp4 can install, delete the rules after the functor is applied, and
no way of saying OK - let's go back."
Yes!

- Composable - in particular one language should behave like a module,
or functor, should have an interface consisting of grammar rules, AST,
AST transforms etc. So one could parametrise one syntax extension
over another, and possibly reuse the language grammar or AST in
other. Currently Camlp4 syntax extension is just a single separate
module which when loaded possibly expects some existing grammar rules
to be in place and mutates them as it's needed.

Yes!

- should be type safe and as mentioned before obey scoping rules. We
should be able to propagate type information even when the syntax
changes. This is difficult part - but I've seen it can be done with some
extra annotations - not talking about Camlp4
Personally, I prefer common lisp's macros to scheme. It's painful
to play with racket's macros as well

- Recursive - it should be able to apply the grammar rules not only
once but expand until it reached the fixpoint.

Yes!
- Reflective - it should be possible after each successful expansion have
the type information available for the next expansion.
Possibly with the compiler library help

- Grammar itself should be lexer-less - memoizing PEG with left
recursion - it's hurdle to define new grammar in terms of old lexer,
or having a stateful lexer that depends on context.

Yes!
- It should not be external tool - like previously observed - it's
difficult to support for code highlighters or refactoring (tools in
general) - if it depends on a build step or command line options.
Yes!

[Caml-list] Re: Call for collaboration on the future of camlp4

[Hongbo Zhang]

On 9/19/12 5:21 PM, Hongbo Zhang wrote:
> On 9/19/12 3:55 PM, Wojciech Meyer wrote:

> - It should not be external tool - like previously observed - it's
> difficult to support for code highlighters or refactoring (tools in
> general) - if it depends on a build step or command line options.
> Yes!
Sorry, a bit typo. IMHO, making it an external tool gains a lot of 
flexibility. The main purpose of macros is to define your own language 
without the support of compiler. If it's tightly coupled with compiler, 
any minimal change will be delayed. Given the compiler library was 
exported, I don't see any sense that P4 still stays in the compiler.
Regards, Bob
>
>
>

[Caml-list] Re: Call for collaboration on the future of camlp4

[bobzhang]

Dear all,
For those who are concerned:
   The latest version of building camlp4 is under 8s.
real	0m7.653s
user	0m5.595s
sys	0m1.736s
On 9/18/12 3:11 PM, bob zhang wrote:
> Dear OCaml programmers,
>     Last week, I give a talk about the future of Camlp4 in the ML
> workshop, the slides are here
> http://www.lexifi.com/ml2012/slides_panel_hongbo.pdf (some are already
> done, some are work in progress).
>     For a long time, in the caml community, thanks to the talented work
> of Daniel and Michel,  we know there is a very very powerful tool called
> *camlp4*, if there are some bolierpolate code you write here and there,
> someone will tell you "hey, you can do it in camlp4" though he may not
> know how to do it in camlp4.
>     But it's a bit embarassing that camlp4 did not evolve very
> well(partly due to the fragmentation of camlp[4,5]), another fact is
> that camlp4 is not *designed*, it's like a prototype that works but not
> carefully designed, and it does not provide anything out of the box  and
> itself was written in a verbose way.
>     It's time to bring the powerful tool back, my advisor Steve and I
> started a new project Fan, which is mainly to evolve the camlp4 macro
> system to be more expressive and more powerful, push the Camlp4 to the
> next level. For me, I am a long-term Lisp programmer, I appreciated the
> value of macros, I would really be happy to see we could make a such
> powerful macro systems.
>     Here is my repo https://bitbucket.org/HongboZhang/camlp4
>     I already finished some cool staff,  to mention just a few:
>     1. A very robust bootstrapping system, previously it takes me 20
> minutes to ver/ify my patches to camlp4 can reach a fixpoint or not, but
> now you can compile your modification within seconds, and reach a
> fixpoint under 2 minutes, this accelerate the development cycles
> immediately./
> /   2. Now you can customize your lexer now, previously it's impossible(
> and a number of bug fixes) you can do deep anti-quotation like /
> /  <:expr< <:expr< $($(deepantiquot)) >> >> /
>     3. Linking the compiler and a number of cool features (see the slides)
>     4. A macro which write macros to scrap all the bolierpolate code for
> generic programming, now you can customize your deriving stuff  in ten
> lines (previously thousands of lines of code to write camlp4 plugin)
>      5. A number of mini-DSLs  and more to be expected
>     I love macros, I would commit to the project for a long
> time(probably my Ph.D term), I promise that I would write the
> documentation to make users happy( I am also writing a book about
> macros, it's un-readable though). And I am really happy to co-laborate
> with anyone who love camlp4.
>     Syntax matters, if you see that coffescript is now already a
> success, camlp4 or Fan is actually more expressive than that, Dear ocaml
> programmers, let us find a way to make ocaml more beautiful. I am also
> open minded to any discussion about the future of camlp4
>     Many thanks (sorry for my bad English)
> --
> Regards
> -- Bob