Re: [Caml-list] Is it possible to extend OCaml lexer rules via Camlp4?

[Gabriel Scherer <gabriel.scherer@gmail.com>]

> I have tried to override whole the syntax as follows, but it seems
> that it changes nothing...:

Camlp4 is designed around mutable state. That you Make module produces
a new grammar doesn't make it the current grammar used by the
preprocessor. What need to happen is that the current state of the
lexer/parser is *mutated* by your Make module (whose evaluation is
then delayed and controlled by Camlp4 itself thanks to registration
with Register). This is what EXTEND does at the grammar level.

If it suits your need, you can define your modification as a filter
that will post-process the output of the original lexer. The Token
module expose a define_filter function to imperatively update the set
of such active stream transformers.

This idea was suggested to me by Jérémie Dimino, and works well. It is
used for example in pa_comprehension to define "[?" and "?]" as new
OCaml keywords (asking for "["; "?" at the camlp4 grammar level would
allow spaces in between):
  https://github.com/ocaml-batteries-team/batteries-included/blob/master/src/syntax/pa_comprehension/pa_comprehension.ml

Below is the relevant code:

  module Make (Syntax : Sig.Camlp4Syntax) = struct
    open Sig;
    include Syntax;


    (* "[?" and "?]" are not recognized as delimiters by the Camlp4
      lexer; This token parser will spot "["; "?" and "?"; "]" token
      and insert "[?" and "?]" instead.
     Thanks to Jérémie Dimino for the idea. *)
    value rec delim_filter older_filter stream =
      let rec filter = parser
      [ [: `(KEYWORD "[", loc); rest :] ->
          match rest with parser
          [ [: `(KEYWORD "?", _) :] -> [: `(KEYWORD "[?", loc); filter rest :]
          | [: :] -> [: `(KEYWORD "[", loc); filter rest :] ]
      | [: `(KEYWORD "?", loc); rest :] ->
          match rest with parser
          [ [: `(KEYWORD "]", loc) :] -> [: `(KEYWORD "?]", loc); filter rest :]
          | [: :] -> [: `(KEYWORD "?", loc); filter rest :] ]
      | [: `other; rest :] -> [: `other; filter rest :] ] in
      older_filter (filter stream);

    value _ = Token.Filter.define_filter (Gram.get_filter ()) delim_filter;

    (* REST OF THE CAMLP4 EXTENSION ... *)
  end;


On Wed, Nov 2, 2011 at 9:34 PM, Jun Furuse <jun.furuse@gmail.com> wrote:
> Hi,
>
> Is it possible for Camlp4 to implement an OCaml syntax extension (i.e.
> pa_*) which modifies the lexer of OCaml syntax?
>
> I have tried to override whole the syntax as follows, but it seems
> that it changes nothing...:
>
> -----------------------------------------------------------
> open Camlp4
>
> module Id : Sig.Id = struct
>  let name = "pa_extlex"
>  let version = "1.0"
> end
>
> module XLexer = Xlexer.Make(PreCast.Token)        (* XLexer
> reimplements OCaml lexer with some extra rules *)
> module XGram = PreCast.MakeGram(XLexer)
>
> module Make (Syntax : Sig.Camlp4Syntax) = struct
>  let _ = prerr_endline "Creating OCaml syntax with lexer extension"
>  module M1 = OCamlInitSyntax.Make(PreCast.Ast)(XGram)(PreCast.Quotation)
>  module M2 = Camlp4OCamlRevisedParser.Make(M1)
>  module M3 = Camlp4OCamlParser.Make(M2)
>  include M3
> end
>
> let module M = Register.OCamlSyntaxExtension(Id)(Make) in ()
> -----------------------------------
>
> Jun
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa-roc.inria.fr/wws/info/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>
>