The lexer hack

[Dario Teixeira <darioteixeira@yahoo.com>]

Hi,

I'm creating a parser for a LaTeX-ish language that features verbatim blocks.
To handle them I want to switch lexers on-the-fly, depending on the parsing
context.  Therefore, I need the state from the (Menhir generated) parser
to influence the lexing process (I believe this is called the "lexer hack"
in compiler lore).

Presently I am doing this by placing a module between the lexer and the
parser, listening in on the flow of tokens, and using a crude state machine
to figure out the parsing context.  This solution is however error-prone
and a bit wasteful, since I'm reimplementing by hand stuff that should be
the sole competence of the parser generator.

Anyway, since I'm sure this problem pops up often, does someone have any
alternative suggestions?  I would preferably keep Menhir, but I'll switch
if some other generator offers a better approach(*).

Thanks + best regards,
Dario Teixeira


(*) I've looked into Dypgen, and its partial actions may offer a way out.
    Does someone have any experience with those and with real-world usage
    of Dypgen in general?  (In other words, is it stable enough for
    production use?)