From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: caml-list@sympa.inria.fr Delivered-To: caml-list@sympa.inria.fr Received: from mail1-relais-roc.national.inria.fr (mail1-relais-roc.national.inria.fr [192.134.164.82]) by sympa.inria.fr (Postfix) with ESMTPS id 931587ED26 for ; Mon, 28 May 2012 12:00:00 +0200 (CEST) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AiECAINLw0/RVaC2kWdsb2JhbABEoxwziCwBiS4IIgEBAQEJCwsHFAQjghcBAQECAQESAiwBGxILAQMBCwYFCxohIgERAQUBChIGExIQh1oBAwYFC5l+CQOMK4JwhCsKGScDCleIcQEFDIp3G4UkA5UXgQ+NBj2EAQ X-IronPort-AV: E=Sophos;i="4.75,670,1330902000"; d="scan'208";a="160233859" Received: from mail-gh0-f182.google.com ([209.85.160.182]) by mail1-smtp-roc.national.inria.fr with ESMTP/TLS/RC4-MD5; 28 May 2012 11:59:59 +0200 Received: by ghbz22 with SMTP id z22so1662966ghb.27 for ; Mon, 28 May 2012 02:59:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=fu7B3ksfeWvBoleHezkqutjfp4h0pH4gh4A9CeeqcUc=; b=bJAu/0uCeteoesPdoT0FxHl5/IXfOXh1cEiFQXzPCQ2JZ438MKFeMXequYfwsQsRdB ta3JOAJkUfLspHAmpWWtX4y7HHvkfBfbrzUgKirQujXyxRGLgrXp6gSbLOEmh8WV27TP YLyNHXRTngYkU61vHMVOTHA/2nqo4RhdybZ3gifZKAVXyF9LzrjP8V2KSD8VqPK9EeL2 B86VRuf7b2qcahsHq59f1ZvYdTt2GuVQamf2G+GwSWbEbw08ATihSrjNUjGkoad/tXCX iQdU5jHTUeP3FMYeEucgNhtsi9Y6sS8H1gyZ9/B85yCiKyogirXAJG7hecp3e+uQE6PP K0Hg== Received: by 10.50.12.199 with SMTP id a7mr3924482igc.71.1338199198156; Mon, 28 May 2012 02:59:58 -0700 (PDT) MIME-Version: 1.0 Received: by 10.50.94.39 with HTTP; Mon, 28 May 2012 02:59:17 -0700 (PDT) In-Reply-To: <9B6D7FCED63545E2BB3F0DDD7B337AA1@erratique.ch> References: <4FC26FDD.9010407@gmail.com> <71FFF6A2CF1E4E6DA0FFF397B2084B89@erratique.ch> <9B6D7FCED63545E2BB3F0DDD7B337AA1@erratique.ch> From: Gabriel Scherer Date: Mon, 28 May 2012 11:59:17 +0200 Message-ID: To: =?ISO-8859-1?Q?Daniel_B=FCnzli?= Cc: Wojciech Meyer , Hongbo Zhang , caml-list@inria.fr Content-Type: multipart/alternative; boundary=14dae9341147d8492104c115c737 Subject: Re: [Caml-list] Re: Syntax extensions without Camlp4 --14dae9341147d8492104c115c737 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable As this seems to be the kind of "christmas shopping list" discussion where everyone throws something in, here is my opinion on camlp4 evolution: - we should forget about the general problem of extending syntax rules; it's too complex, doesn't compose well, and is only of arguable benefit - we should focus on blessing specific *fixed* syntactical extensibility points and making them enter the standard syntax for the language; there are two elephants in the rooms: quotations and type-conv-like annotations. Quotations are useful, used in practice (not always under the camlp4-blessed form), simple to define, understand and use, and compose extremely well. One problem with current camlp4 quotations is that they are a bit lexically heavy (<:name< ... >>) and inflexible (you basically can't use >> inside the quotation); users have tried to workaround this by re-coding quotations, eg. for smart string literals u"blah" or regexp syntactic sugar s/foo/bar/. I think we could consider having lexing-level extensibility (have a tool to modify lexing rule instead of grammar rules like Camlp4; hopefully lexing rules are simpler and compose better). Type-conv like annotations are a bit less well-defined, and would need some design work: what exactly is their intended scope/expressivity? Are they a general annotation mechanism, a rigid way to just add new phrases after each annotation phrase, or something in between? Other point of extensibility could be considered (eg. Jeremy Yallop's work on pattern-matching extensibility), but only added if they are simple enough, useful enough, and compose well. I would still welcome having central, well-supported libraries to manipulate Ocaml syntactic AST and, why not, typed AST (typedtree). Among its many features, Camlp4 currently provides the former (in a relatively convenient way thanks to its AST quotations), and this is quite useful. On Mon, May 28, 2012 at 11:35 AM, Daniel B=FCnzli wrote: > > > Le lundi, 28 mai 2012 =E0 00:43, Wojciech Meyer a =E9crit : > > > Runtime meta-programming is a generalisation of static meta > > programming. MetaOCaml has a nice set of abstraction to generate > > typechecking code - yes - either at runtime or during compile time. > > > You meant 'typechecked' (?). It's obviously a generalization but I didn't > know there was support to use it at compile time. > > > The > > problem is that it's purely for partial evaluation and not extending the > > syntax. > > > Then it's perfect ! I think it's wrong to try to extend the language per > se. Most of the time, except for very particular things (e.g. introducing= a > monad notation), the dsl approach is perfectly sufficient. Don't think you > absolutely need to extend the OCaml grammar, embed your dsl directly into > OCaml, using OCaml language binders if you need variables. > > Make libraries, not pet syntactic constructs. > > Many things that can be done with camlp4, can be done with that approach. > Not only is it very elegant, it's much easier to maintain w.r.t. the > evolution of the OCaml language itself. The techniques in these papers [1] > should be more known and used. > > > - It should not be external tool - like previously observed - it's > > difficult to support for code highlighters or refactoring (tools in > > general) - if it depends on a build step or command line options. > > > > > If you extend the grammar itself, code highlighters or any tool expecting > an OCaml expression is going to break whether the tool is external or not. > But for the rest of your comments I agree wholeheartedly (even though I'm > not sure all that power is needed, but at least it would make the tool > non-ugly to me). > > Best, > > Daniel > > [1] > > http://citeseerx.ist.psu.edu/viewdoc/download?doi=3D10.1.1.31.9782&rep=3D= rep1&type=3Dpdf > http://research.microsoft.com/apps/pubs/default.aspx?id=3D67334 > > > > > > > > Therefore the main purpose of syntactical abstraction is missing > > - but that's not a problem - MetaOCaml wasn't designed for it. > > > > Things that I would like to see in future "incarnation" or integrating = of > > meta programming facilities to the language would be (beware that's my > > blue dreams!): > > > > - first of all non destructive updates to the grammar e.g: "let open > > lang Sexp in ..." should open the Sexp syntax extension, install the > > grammar, but when it goes out of scope it should vanish. Currently > > Camlp4 can install, delete the rules after the functor is applied, and > > no way of saying OK - let's go back. > > > > - Composable - in particular one language should behave like a module, > > or functor, should have an interface consisting of grammar rules, AST, > > AST transforms etc. So one could parametrise one syntax extension > > over another, and possibly reuse the language grammar or AST in > > other. Currently Camlp4 syntax extension is just a single separate > > module which when loaded possibly expects some existing grammar rules > > to be in place and mutates them as it's needed. > > > > - should be type safe and as mentioned before obey scoping rules. We > > should be able to propagate type information even when the syntax > > changes. This is difficult part - but I've seen it can be done with some > > extra annotations - not talking about Camlp4 > > > > - Recursive - it should be able to apply the grammar rules not only > > once but expand until it reached the fixpoint. > > > > - Reflective - it should be possible after each successful expansion ha= ve > > the type information available for the next expansion. > > > > - Grammar itself should be lexer-less - memoizing PEG with left > > recursion - it's hurdle to define new grammar in terms of old lexer, > > or having a stateful lexer that depends on context. > > > > - It should not be external tool - like previously observed - it's > > difficult to support for code highlighters or refactoring (tools in > > general) - if it depends on a build step or command line options. > > > > That's said I find Camlp4 extremely useful for code generation purposes > > - when I need to generate some ML code through quotations. Also, some > > very important projects depend on Camlp4 (or Camlp5) like Coq. I don't > > see that ML can live without some meta programming facilities out of the > > box. > > > > -- > > Wojciech Meyer > > http://danmey.org > > > > -- > > Caml-list mailing list. Subscription management and archives: > > https://sympa-roc.inria.fr/wws/info/caml-list > > Beginner's list: http://groups.yahoo.com/group/ocaml_beginners > > Bug reports: http://caml.inria.fr/bin/caml-bugs > > > > > -- > Caml-list mailing list. Subscription management and archives: > https://sympa-roc.inria.fr/wws/info/caml-list > Beginner's list: http://groups.yahoo.com/group/ocaml_beginners > Bug reports: http://caml.inria.fr/bin/caml-bugs > > --14dae9341147d8492104c115c737 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable As this seems to be the kind of "christmas shopping list" discuss= ion where everyone throws something in, here is my opinion on camlp4 evolut= ion:
- we should forget about the general problem of extending syntax ru= les; it's too complex, doesn't compose well, and is only of arguabl= e benefit
- we should focus on blessing specific *fixed* syntactical extensibility po= ints and making them enter the standard syntax for the language; there are = two elephants in the rooms: quotations and type-conv-like annotations.

Quotations are useful, used in practice (not always under the camlp4-bl= essed form), simple to define, understand and use, and compose extremely we= ll. One problem with current camlp4 quotations is that they are a bit lexic= ally heavy (<:name< ... >>) and inflexible (you basically can&#= 39;t use >> inside the quotation); users have tried to workaround thi= s by re-coding quotations, eg. for smart string literals u"blah"= or regexp syntactic sugar s/foo/bar/. I think we could consider having lex= ing-level extensibility (have a tool to modify lexing rule instead of gramm= ar rules like Camlp4; hopefully lexing rules are simpler and compose better= ).

Type-conv like annotations are a bit less well-defined, and would need = some design work: what exactly is their intended scope/expressivity? Are th= ey a general annotation mechanism, a rigid way to just add new phrases afte= r each annotation phrase, or something in between?

Other point of extensibility could be considered (eg. Jeremy Yallop'= ;s work on pattern-matching extensibility), but only added if they are simp= le enough, useful enough, and compose well.

I would still welcome ha= ving central, well-supported libraries to manipulate Ocaml syntactic AST an= d, why not, typed AST (typedtree). Among its many features, Camlp4 currentl= y provides the former (in a relatively convenient way thanks to its AST quo= tations), and this is quite useful.

On Mon, May 28, 2012 at 11:35 AM, Daniel B= =FCnzli <daniel.buenzli@erratique.ch> wrote:


Le lundi, 28 mai 2012 =E0 00:43, Wojciech Meyer a =E9crit :

> Runtime meta-programming is a generalisation of static meta
> programming. MetaOCaml has a nice set of abstraction to generate
> typechecking code - yes - either at runtime or during compile time.


You meant 'typechecked' (?). It's obviously a generalizat= ion but I didn't know there was support to use it at compile time.

> The
> problem is that it's purely for partial evaluation and not extendi= ng the
> syntax.


Then it's perfect ! I think it's wrong to try to extend the l= anguage per se. Most of the time, except for very particular things (e.g. i= ntroducing a monad notation), the dsl approach is perfectly sufficient. Don= 't think you absolutely need to extend the OCaml grammar, embed your ds= l directly into OCaml, using OCaml language binders if you need variables.<= br>
Make libraries, not pet syntactic constructs.

Many things that can be done with camlp4, can be done with that approach. N= ot only is it very elegant, it's much easier to maintain w.r.t. the evo= lution of the OCaml language itself. The techniques in these papers [1] sho= uld be more known and used.

> - It should not be external tool - like previously observed - it's=
> difficult to support for code highlighters or refactoring (tools in
> general) - if it depends on a build step or command line options.




If you extend the grammar itself, code highlighters or any tool expec= ting an OCaml expression is going to break whether the tool is external or = not. But for the rest of your comments I agree wholeheartedly (even though = I'm not sure all that power is needed, but at least it would make the t= ool non-ugly to me).

Best,

Daniel

[1]
http://citeseerx.ist.ps= u.edu/viewdoc/download?doi=3D10.1.1.31.9782&rep=3Drep1&type=3Dpdf
http://research.microsoft.com/apps/pubs/default.aspx?id= =3D67334






> Therefore the main purpose of syntactical abstraction is missing
> - but that's not a problem - MetaOCaml wasn't designed for it.=
>
> Things that I would like to see in future "incarnation" or i= ntegrating of
> meta programming facilities to the language would be (beware that'= s my
> blue dreams!):
>
> - first of all non destructive updates to the grammar e.g: "let o= pen
> lang Sexp in ..." should open the Sexp syntax extension, install = the
> grammar, but when it goes out of scope it should vanish. Currently
> Camlp4 can install, delete the rules after the functor is applied, and=
> no way of saying OK - let's go back.
>
> - Composable - in particular one language should behave like a module,=
> or functor, should have an interface consisting of grammar rules, AST,=
> AST transforms etc. So one could parametrise one syntax extension
> over another, and possibly reuse the language grammar or AST in
> other. Currently Camlp4 syntax extension is just a single separate
> module which when loaded possibly expects some existing grammar rules<= br> > to be in place and mutates them as it's needed.
>
> - should be type safe and as mentioned before obey scoping rules. We > should be able to propagate type information even when the syntax
> changes. This is difficult part - but I've seen it can be done wit= h some
> extra annotations - not talking about Camlp4
>
> - Recursive - it should be able to apply the grammar rules not only
> once but expand until it reached the fixpoint.
>
> - Reflective - it should be possible after each successful expansion h= ave
> the type information available for the next expansion.
>
> - Grammar itself should be lexer-less - memoizing PEG with left
> recursion - it's hurdle to define new grammar in terms of old lexe= r,
> or having a stateful lexer that depends on context.
>
> - It should not be external tool - like previously observed - it's=
> difficult to support for code highlighters or refactoring (tools in
> general) - if it depends on a build step or command line options.
>
> That's said I find Camlp4 extremely useful for code generation pur= poses
> - when I need to generate some ML code through quotations. Also, some<= br> > very important projects depend on Camlp4 (or Camlp5) like Coq. I don&#= 39;t
> see that ML can live without some meta programming facilities out of t= he
> box.
>
> --
> Wojciech Meyer
> http://danmey.org<= br> >
> --
> Caml-list mailing list. Subscription management and archives:
> https://sympa-roc.inria.fr/wws/info/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs




--
Caml-list mailing list. =A0Subscription management and archives:
https://sympa-roc.inria.fr/wws/info/caml-list
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs


--14dae9341147d8492104c115c737--