RE: [Caml-list] The lexer hack

[Dario Teixeira <darioteixeira@yahoo.com>]

Hi,

> Out of interest, how LaTeX-ish do you mean? I would hazard
> a guess that it's impossible to parse an unrestricted TeX
> file using an LR grammar (or at least no more clear than a
> hand-coded automaton) because you have to execute the macro
> expander in order to parse the file *completely* correctly.
> However, if you only mean LaTeX-ish in syntax (i.e. the
> files aren't actually TeX files) then you don't have to
> worry about TeX's elegant (by which I mean terrifying)
> \catcode mechanism and macro language!

I developed the language's syntax in tandem with the parser/lexer
so I made sure it was LR-friendly and Ulex-friendly (the verbatim
environments are the only parsing-unfriendly features).  The language
looks and feels like LaTeX, but without the hairy stuff...

Incidentally, the dummy token/action trick seems to be working
fine with Menhir.  Since the parser will look ahead one token,
I just have a tokenizer sitting between the lexer and the parser,
and inserting a DUMMY token into the stream after any token that
precedes a dummy action:

inline:
  | (...)
  | BEGIN_VERBATIM enter_verb DUMMY RAW exit_verb END_VERBATIM {...}
  | (...)

enter_verb: /*empty*/ {Global.context := Global.Verbatim}
exit_verb: /*empty*/  {Global.context := Global.General}


It's not the prettiest thing in the world (and I suspect I might
still find some problem with it), but as far as lexer hacks go
it's not bad and a lot better than building a state machine.

Cheers,
Dario Teixeira