[Caml-list] Call for collaboration on the future of camlp4

[Caml-list] Call for collaboration on the future of camlp4

[oleg]

Hongbo Zhang wrote:
  > Meta-ocaml is orthogonal to camlp4 as well, but I did not see any hope
  > that it will be pushed into ocaml compiler,
  > since the main purpose of meta-ocaml is to do some run-time
  > optimization, without a native jit compiler support, It does
  > not make sense.

I believe you might be short-changing MetaOCaml. First of all, why
MetaOCaml needs JIT? It is already possible in MetaOCaml to generate
code at run-time, to compile it and link-in. The new code is generated
by the OCaml compiler itself, and works as if it were part of the
original code. Such run-time optimization can indeed be useful (e.g.,
profile-based optimization). I don't see why we need JIT if we already
have the original compiler itself.

However viewing MetaOCaml as a run-time optimizer misses a
lot. MetaOCaml is a code generator with certain assurances of
correctness -- it generates code that one would not write by hand. The
recent example of generating code for stencil computations in
high-performance computing

	http://okmij.org/ftp/meta-programming/HPC.html#stencil

was quite illustrative: MetaOCaml has generated code for edge
cases that human really did not want to write (and did not actually
write). Edge cases are really tedious and really difficult to get
right; automation is of great help. The generated code can be
printed-out and used in a project as any other code. MetaOCaml can be
used to generate libraries of specialized high-performance code.

  > Personally, I prefer common lisp's macros to scheme. It's painful
  > to play with racket's macros as well

Well, it is painful for some people to use typed languages
too. Untyped languages offer so much freedom, some think. What is
often forgotten is that the benefit of the type system is not in what
code you can write, but what code you cannot. Likewise, a syntax
extension should be evaluated not only on what it allows but also on
what it prevents.


As I general remark: it is not clear why so much fixation on
camlp4. Other people have tried extensible syntaxes. For example:

	- Felix
	- Notation of Coq
	- Stratego
	http://strategoxt.org/
In particular, see
	http://strategoxt.org/Sdf/SdfLanguage

Stratego could do program transformations in concrete syntax. Maybe
that would be a better approach? Maybe Stratego would satisfy all our
needs?

[Caml-list] Re: Call for collaboration on the future of camlp4

[Hongbo Zhang]

[-- Attachment #1: Type: text/plain, Size: 4415 bytes --]

On 9/20/12 5:36 AM, oleg@okmij.org wrote:
> Hongbo Zhang wrote:
>    > Meta-ocaml is orthogonal to camlp4 as well, but I did not see any hope
>    > that it will be pushed into ocaml compiler,
>    > since the main purpose of meta-ocaml is to do some run-time
>    > optimization, without a native jit compiler support, It does
>    > not make sense.
>
> I believe you might be short-changing MetaOCaml. First of all, why
> MetaOCaml needs JIT? It is already possible in MetaOCaml to generate
> code at run-time, to compile it and link-in. The new code is generated
> by the OCaml compiler itself, and works as if it were part of the
> original code. Such run-time optimization can indeed be useful (e.g.,
> profile-based optimization). I don't see why we need JIT if we already
> have the original compiler itself.
>
> However viewing MetaOCaml as a run-time optimizer misses a
> lot. MetaOCaml is a code generator with certain assurances of
> correctness -- it generates code that one would not write by hand. The
> recent example of generating code for stencil computations in
> high-performance computing
Yes, it's a run-time optimizer with type safe assurance. It can do
partial evaluation to generate some optimized code. With the help
of compiler-library, I can do byte-code eval(not native eval), I agree
it would be useful to have a native eval, but this requires non-trivial
changes to the compiler which I don't expect it will be realized in a
short term.
> 	http://okmij.org/ftp/meta-programming/HPC.html#stencil
>
> was quite illustrative: MetaOCaml has generated code for edge
> cases that human really did not want to write (and did not actually
> write). Edge cases are really tedious and really difficult to get
> right; automation is of great help. The generated code can be
> printed-out and used in a project as any other code. MetaOCaml can be
> used to generate libraries of specialized high-performance code.
>
>    > Personally, I prefer common lisp's macros to scheme. It's painful
>    > to play with racket's macros as well
>
> Well, it is painful for some people to use typed languages
> too. Untyped languages offer so much freedom, some think. What is
> often forgotten is that the benefit of the type system is not in what
> code you can write, but what code you cannot. Likewise, a syntax
> extension should be evaluated not only on what it allows but also on
> what it prevents.
Well, as I said, it's a personal taste. People are /*too cautiou*/s about
the safety the code generated, if you take a look at other languages,
like D, they *even use "string concatenation" to generate code*,*/we 
manipulate/**/
/**/the Ast instead of strings, and finally it's type checked, in 
practice, it works/**/
/**/pretty well/* and I wrote tens of thousands of code on top of 
camlp4, none of the bugs
can be detected by type checking. If you take a look at the
history of Template Haskell, they finally step back from type checking 
everything to
give up type checking some quasi-quotations. We don't want to sacrifice 
too much
experssivity for type safety, this is especially important in macros. In 
common lisp,
there is also a kind of macros called "Anaphoric macros" which you will 
find painful to
do in Scheme. To overcome the un-hygenic problem, if you have a look at 
the book "let over lambda",
there's best practice to avoid such problems. And you take a look at the 
compiler of opa,
  they override the module Stream to make use of the syntax extension 
what camlp4 can provide,
that's the beauty of syntax meta-programming, instead of a bug.

I really appreciate that ocaml's type checker can finally type check the 
generated code, but do
we really need to type check the macros? Would the cost be too high?
>
> As I general remark: it is not clear why so much fixation on
> camlp4. Other people have tried extensible syntaxes. For example:
>
> 	- Felix
Some ideas of Felix are neat that we should borrow from.
> 	- Notation of Coq
This is built on top of Camlp5

	- Stratego
	http://strategoxt.org/
In particular, see
	http://strategoxt.org/Sdf/SdfLanguage

Stratego could do program transformations in concrete syntax. Maybe
that would be a better approach? Maybe Stratego would satisfy all our
needs?

Well, this is definitely not true. The beauty of Camlp4 is that it is a 
growing macro system instead of
only a program transformation tool.

Thanks for your message.
>
>


[-- Attachment #2: Type: text/html, Size: 5974 bytes --]

[Caml-list] Re: Call for collaboration on the future of camlp4

[oleg]

> Yes, it's [MetaOCaml] a run-time optimizer with type safe assurance. It can do
> partial evaluation to generate some optimized code. 

Let me stress once again how narrow this view is. MetaOCaml goes well
beyond partial evaluation. For example, MetaOCaml, as a general code
generation framework, was used to derive optimal (in the number of
multiplications) FFT kernels. Partial evaluation will not give you
that.

Code generation is a very promising technique in High-Performance
computing. Most of the tools used in practice -- FFTW, ATLAS, SPIRAL
-- are all off-line tools. They generate a large number of candidate
codes and choose the best performing. What's important is to quickly
generate a large number of very tedious programs. Assurance of
correctness are important: a programmer, especially a domain expert,
will not want to even look at the generated code let alone debug it.
I see MetaOCaml target the same area.

> I agree it would be useful to have a native eval, but this requires
> non-trivial changes to the compiler which I don't expect it will be
> realized in a short term.
The assessment is mistaken. MetaOCaml v3.09 did have a native
back-end. I know quite well what changes were required. Those changes
are no longer needed since dynamic linking has since become part of
OCaml proper.

> If you take a look at the history of Template Haskell, they finally
> step back from type checking everything to give up type checking some
> quasi-quotations. 

This is a mistaken impression. While Template Haskell as a whole will
remain untyped for a long time -- after all, Template Haskell can
generate data and type class _declarations_, whose typing is far from
clear -- there is a definite push towards MetaOCaml-like type safety
for expressions. 
  http://hackage.haskell.org/trac/ghc/blog/Template%20Haskell%20Proposal

see especially
  Part B: Add new MetaML-style constructs for strongly-typed metaprogramming.

If this is implemented, TH becomes quite like MetaML.

> We don't want to sacrifice too much experssivity for type safety, this
> is especially important in macros. In common lisp, there is also a
> kind of macros called "Anaphoric macros" which you will find painful
> to do in Scheme.

That is not a very good argument since R5RS macros in Scheme were
intentionally limited in their expressivity. The macro system was
designed to be just enough expressive for the special forms
introduced in the Report. (Later on the system was found to be quite
more expressive than its designers have anticipated.)

The anaphoric macros are easily expressible in the system of our JFP
2011 paper (staging with a very limited delimited control). No
subversions of hygiene are needed.

[Caml-list] Re: Call for collaboration on the future of camlp4

[Hongbo Zhang]

[-- Attachment #1: Type: text/plain, Size: 3958 bytes --]

On 9/22/12 3:50 AM, oleg@okmij.org wrote:
>> Yes, it's [MetaOCaml] a run-time optimizer with type safe assurance. It can do
>> partial evaluation to generate some optimized code.
> Let me stress once again how narrow this view is. MetaOCaml goes well
> beyond partial evaluation. For example, MetaOCaml, as a general code
> generation framework, was used to derive optimal (in the number of
> multiplications) FFT kernels. Partial evaluation will not give you
> that.
>
> Code generation is a very promising technique in High-Performance
> computing. Most of the tools used in practice -- FFTW, ATLAS, SPIRAL
> -- are all off-line tools. They generate a large number of candidate
> codes and choose the best performing. What's important is to quickly
> generate a large number of very tedious programs. Assurance of
> correctness are important: a programmer, especially a domain expert,
> will not want to even look at the generated code let alone debug it.
> I see MetaOCaml target the same area.
>
To my limited knowledge, FFTW used techniques more like Camlp4.
Again: type safety(without dependent types) is far from correctness,
actually, the type error bugs are the easiest to fix. In practice,
you always need to debug the generated code if something goes wrong.
One thing people complains about template c++ code is that it's really
hard to debug.

>> I agree it would be useful to have a native eval, but this requires
>> non-trivial changes to the compiler which I don't expect it will be
>> realized in a short term.
> The assessment is mistaken. MetaOCaml v3.09 did have a native
> back-end. I know quite well what changes were required. Those changes
> are no longer needed since dynamic linking has since become part of
> OCaml proper.
Actually, I am one of the curious programmers who tried MetaOCaml and 
played
with it. Don't get me wrong, I would be happy to see MetaOCaml pushed 
into OCaml.
As I said before, it's orthogonal to P4, it's great if we could 
dynamically generate
type safe code, and we can make use of meta-ocaml as well ;-)
But the assumption is /*that we have a native meta-ocaml without 
patching the compiler */
>> If you take a look at the history of Template Haskell, they finally
>> step back from type checking everything to give up type checking some
>> quasi-quotations.
> This is a mistaken impression. While Template Haskell as a whole will
> remain untyped for a long time -- after all, Template Haskell can
> generate data and type class _declarations_, whose typing is far from
> clear -- there is a definite push towards MetaOCaml-like type safety
> for expressions.
>    http://hackage.haskell.org/trac/ghc/blog/Template%20Haskell%20Proposal
>
> see especially
>    Part B: Add new MetaML-style constructs for strongly-typed metaprogramming.
>
> If this is implemented, TH becomes quite like MetaML.
There are two problems here, first you object language is not only ocaml,
second, yes, we want /*everything to be first class, everything*/, the most
expressive system.
>> We don't want to sacrifice too much experssivity for type safety, this
>> is especially important in macros. In common lisp, there is also a
>> kind of macros called "Anaphoric macros" which you will find painful
>> to do in Scheme.
> That is not a very good argument since R5RS macros in Scheme were
> intentionally limited in their expressivity. The macro system was
> designed to be just enough expressive for the special forms
> introduced in the Report. (Later on the system was found to be quite
> more expressive than its designers have anticipated.)
>
> The anaphoric macros are easily expressible in the system of our JFP
> 2011 paper (staging with a very limited delimited control). No
> subversions of hygiene are needed.
That depends on how you define 'easily expressible' ;-)
Let's discuss in private, btw, it's a bit unfair that you just picked
some points and ignored others without context.

Thanks for your message.
>


[-- Attachment #2: Type: text/html, Size: 5533 bytes --]

Re: [Caml-list] Re: Call for collaboration on the future of camlp4

[Jacques Carette]

I emphatically agree with Oleg.  I'll reinforce on point regarding the 
competition:

On 22/09/2012 3:50 AM, oleg@okmij.org wrote:
>   [...] there is a definite push towards MetaOCaml-like type safety
> for expressions.
>    http://hackage.haskell.org/trac/ghc/blog/Template%20Haskell%20Proposal
>
> see especially
>    Part B: Add new MetaML-style constructs for strongly-typed metaprogramming.

This is being implemented NOW.  Simon Peyton Jones is doing the ghc 
work, and I am porting the metaocaml codes I have to serve as test 
cases.  Simon announced that this 'typed quotes and splices' will be in 
GHC 7.8 at the Haskell Workshop.

Jacques

Re: [Caml-list] Re: Call for collaboration on the future of camlp4

[Hongbo Zhang]

On 9/22/12 8:53 AM, Jacques Carette wrote:
> I emphatically agree with Oleg.  I'll reinforce on point regarding the 
> competition:
>
> On 22/09/2012 3:50 AM, oleg@okmij.org wrote:
>>   [...] there is a definite push towards MetaOCaml-like type safety
>> for expressions.
>> http://hackage.haskell.org/trac/ghc/blog/Template%20Haskell%20Proposal
>>
>> see especially
>>    Part B: Add new MetaML-style constructs for strongly-typed 
>> metaprogramming.
>
> This is being implemented NOW.  Simon Peyton Jones is doing the ghc 
> work, and I am porting the metaocaml codes I have to serve as test 
> cases.  Simon announced that this 'typed quotes and splices' will be 
> in GHC 7.8 at the Haskell Workshop.
>
> Jacques
Yes, I agree that it would much nicer we have such facility in ocaml, no 
competition..

Re: [Caml-list] Call for collaboration on the future of camlp4

[Wojciech Meyer]

Hi Hongboz,

Thanks for the slides, it's good. Indeed it makes sense now to focus on
extending Camlp4, however as usually there are some gotchas, the process
of extending syntax with Camlp4 is a bit not well known.  Camlp5 has an
excellent documentation, and great support for some of the things. Yet,
Camlp4 is more modular, and is just easier to write application on top
of Camlp4.

On OUD we had open ended discussions, and many people share (like me)
the same impression - -ppx that exist on trunk is powerful and that's
the way to implement most of the meta programming facilities (deriving
the code from type definitions, or using quotations to embed DSLs in
OCaml), but I believe that the direct syntax extensions (opening the
OCaml syntax) certainly has some benefits. How to do this in a clean way
myself - I don't know. I posted just some time ago what would be a macro
system of my dreams:

https://sympa.inria.fr/sympa/arc/caml-list/2012-05/msg00184.html

If the current Camlp4 could be used to do this, it would be great but
that's need to be somewhat kept up to date with all up-coming compiler
meta programming facilities.

I will have a closer look at the Fan project today.

Thanks,

--
Wojciech Meyer
http://danmey.org


bob zhang <bobzhang1988@gmail.com> writes:

> Dear OCaml programmers,
>
>    Last week, I give a talk about the future of Camlp4 in the ML
> workshop, the slides are here
> http://www.lexifi.com/ml2012/slides_panel_hongbo.pdf (some are already
> done, some are work in progress).
>    For a long time, in the caml community, thanks to the talented work
> of Daniel and Michel,  we know there is a very very powerful tool
> called *camlp4*, if there are some bolierpolate code you write here
> and there, someone will tell you "hey, you can do it in camlp4" though
> he may not know how to do it in camlp4. 
>    But it's a bit embarassing that camlp4 did not evolve very well
> (partly due to the fragmentation of camlp[4,5]), another fact is that
> camlp4 is not *designed*, it's like a prototype that works but not
> carefully designed, and it does not provide anything out of the box
>  and itself was written in a verbose way.
>    It's time to bring the powerful tool back, my advisor Steve and I
> started a new project Fan, which is mainly to evolve the camlp4 macro
> system to be more expressive and more powerful, push the Camlp4 to the
> next level. For me, I am a long-term Lisp programmer, I appreciated
> the value of macros, I would really be happy to see we could make a
> such powerful macro systems.
>    Here is my repo https://bitbucket.org/HongboZhang/camlp4
>    I already finished some cool staff,  to mention just a few:
>    1. A very robust bootstrapping system, previously it takes me 20
> minutes to verify my patches to camlp4 can reach a fixpoint or not,
> but now you can compile your modification within seconds, and reach a
> fixpoint under 2 minutes, this accelerate the development cycles
> immediately.
>    2. Now you can customize your lexer now, previously it's impossible
> ( and a number of bug fixes) you can do deep anti-quotation like 
>   <:expr< <:expr< $($(deepantiquot)) >> >> 
>    3. Linking the compiler and a number of cool features (see the
> slides)
>    4. A macro which write macros to scrap all the bolierpolate code
> for generic programming, now you can customize your deriving stuff  in
> ten lines (previously thousands of lines of code to write camlp4
> plugin)
>     5. A number of mini-DSLs  and more to be expected
>    I love macros, I would commit to the project for a long time
> (probably my Ph.D term), I promise that I would write the
> documentation to make users happy( I am also writing a book about
> macros, it's un-readable though). And I am really happy to co-laborate
> with anyone who love camlp4. 
>    Syntax matters, if you see that coffescript is now already a
> success, camlp4 or Fan is actually more expressive than that, Dear
> ocaml programmers, let us find a way to make ocaml more beautiful. I
> am also open minded to any discussion about the future of camlp4
>    Many thanks (sorry for my bad English)

[Caml-list] Call for collaboration on the future of camlp4

[bob zhang]

[-- Attachment #1: Type: text/plain, Size: 2773 bytes --]

Dear OCaml programmers,
   Last week, I give a talk about the future of Camlp4 in the ML workshop,
the slides are here
http://www.lexifi.com/ml2012/slides_panel_hongbo.pdf(some are already
done, some are work in progress).
   For a long time, in the caml community, thanks to the talented work of
Daniel and Michel,  we know there is a very very powerful tool called
*camlp4*, if there are some bolierpolate code you write here and there,
someone will tell you "hey, you can do it in camlp4" though he may not know
how to do it in camlp4.
   But it's a bit embarassing that camlp4 did not evolve very well(partly
due to the fragmentation of camlp[4,5]), another fact is that camlp4 is not
*designed*, it's like a prototype that works but not carefully designed,
and it does not provide anything out of the box  and itself was written in
a verbose way.
   It's time to bring the powerful tool back, my advisor Steve and I
started a new project Fan, which is mainly to evolve the camlp4 macro
system to be more expressive and more powerful, push the Camlp4 to the next
level. For me, I am a long-term Lisp programmer, I appreciated the value of
macros, I would really be happy to see we could make a such powerful macro
systems.
   Here is my repo https://bitbucket.org/HongboZhang/camlp4
   I already finished some cool staff,  to mention just a few:
   1. A very robust bootstrapping system, previously it takes me 20 minutes
to ver*ify my patches to camlp4 can reach a fixpoint or not, but now you
can compile your modification within seconds, and reach a fixpoint under 2
minutes, this accelerate the development cycles immediately.*
*   2. Now you can customize your lexer now, previously it's impossible(
and a number of bug fixes) you can do deep anti-quotation like *
*  <:expr< <:expr< $($(deepantiquot)) >> >> *
   3. Linking the compiler and a number of cool features (see the slides)
   4. A macro which write macros to scrap all the bolierpolate code for
generic programming, now you can customize your deriving stuff  in ten
lines (previously thousands of lines of code to write camlp4 plugin)
    5. A number of mini-DSLs  and more to be expected
   I love macros, I would commit to the project for a long time(probably my
Ph.D term), I promise that I would write the documentation to make users
happy( I am also writing a book about macros, it's un-readable though). And
I am really happy to co-laborate with anyone who love camlp4.
   Syntax matters, if you see that coffescript is now already a success,
camlp4 or Fan is actually more expressive than that, Dear ocaml
programmers, let us find a way to make ocaml more beautiful. I am also open
minded to any discussion about the future of camlp4
   Many thanks (sorry for my bad English)
-- 
Regards
-- Bob

[-- Attachment #2: Type: text/html, Size: 3165 bytes --]