[Caml-list] camlp4 extensions and lexer replacements

[Caml-list] camlp4 extensions and lexer replacements

[John Goerzen]

Hello,

I am working on developing a syntax extension that:

 1. Can use most of the rules in pa_o/pa_r, replaces some, and adds some

 2. Requires a new lexer.

Now, it's easy enough to just accomplish #1 with the EXTEND syntax.  I
can just place my .cmo after pa_o.cmo on the camlp4 command line and
that's good.

But #2 is trickier.  pa_o.ml contains this:

  Grammar.Unsafe.gram_reinit gram (Plexer.gmake ());

So I can't define my lexer first -- pa_o.ml will just overwrite it.
Worse, I can't define my lexer second, either.  Plexer relies on the
(strange, imho) notion of deciding whether something is a LIDENT or
keyword based on the list of keywords-as-tokens that were encountered in
a grammar.  But the grammar reinit functions that I would have to use to
put in my lexer don't preserve that information, so my lexer would have
no way to differentiate a LIDENT from a keyword.  Also, Plexer provides
no way to get a list of entries in its internal hashtable.

It seems that there should be an easy way to do this, but I just can't
figure it out.

I would really avoid having to hack up pa_o.ml.

Thanks,
John

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] camlp4 extensions and lexer replacements

[Virgile Prevosto]

Le lundi 10 mai, à 14h19 -0500,
John Goerzen a écrit:

> 
> But #2 is trickier.  pa_o.ml contains this:
> 
>   Grammar.Unsafe.gram_reinit gram (Plexer.gmake ());
> 
> So I can't define my lexer first -- pa_o.ml will just overwrite it.
> Worse, I can't define my lexer second, either.  Plexer relies on the
> (strange, imho) notion of deciding whether something is a LIDENT or
> keyword based on the list of keywords-as-tokens that were encountered
> in a grammar.  But the grammar reinit functions that I would have to
> use to put in my lexer don't preserve that information, so my lexer
> would have no way to differentiate a LIDENT from a keyword.  Also,
> Plexer provides no way to get a list of entries in its internal
> hashtable.
> 

Hello,
I'm not sure wether this will help you or not, but according to camlp4
documentation
(http://caml.inria.fr/camlp4/manual/lib/Grammar.html#TYPEg),
Grammar.tokens provides a way to get the list of the keywords used so
far:
"val tokens : g -> string -> (string * int) list

Given a grammar and a token pattern constructor, returns the list of the
corresponding values currently used in all entries of this grammar. The
integer is the number of times this pattern value is used.

Examples:

* If the associated lexer uses ("", xxx) to represent a keyword (what is
represented by then simple string xxx in an EXTEND statement rule), the
call Grammar.token g "" returns the keywords list."

Note that keywords are not always LIDENT:
~ [502]$ ocaml
        Objective Caml version 3.07+2

# #load "camlp4o.cma";;
	Camlp4 Parsing version 3.07+2

# let x = fst (List.split (Grammar.tokens Pcaml.gram ""));;
val x : string list =
  ["in"; "<>"; "-."; "<="; "struct"; "asr"; "if"; "{<"; "[<"; "<-";
";;";   "lor"; "external"; "sig"; "virtual"; ":>"; ":="; "type"; "::";
"method";   "then"; "**"; "true"; "match"; "parser"; "try"; "for";
"new"; "to"; "do";   "else"; "land"; "and"; "private"; "&&";
"initializer"; "as"; "lazy";   "open"; "false"; "rec"; "~"; "}"; "|";
"end"; "{"; "or"; "constraint";   "of"; "lxor"; ">}"; "`"; "_"; "^";
"]"; "["; "with"; "inherit"; "let";   "!="; "object"; "||"; "include";
"while"; "module"; ">]"; "@"; "?"; ">";   "="; "assert"; "<"; ";"; ":";
"??"; "when"; "exception"; "lsr"; "mutable"   "/"; "[|"; "."; "-"; ",";
"lsl"; "done"; "+"; "begin"; "downto"; "*"; ")"   "|]"; "("; "function";
"'"; "&"; "functor"; "%"; "$"; ">="; "#"; "~-.";   "!"; "~-"; "fun";
"mod"; "class"; ".."; "=="; "->"; "val"]

-- 
E tutto per oggi, a la prossima volta
Virgile

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners