Re: The lexer hack

Re: The lexer hack

[Jeff Shaw]

Dario,
You could write your lexers in Menhir and make them part of your 
grammar. I know this isn't a terribly easy solution but it would be 
elegant IMO.

Re: [Caml-list] Re: The lexer hack

[Dario Teixeira]

Hi,

> You could write your lexers in Menhir and make them part of
> your grammar. I know this isn't a terribly easy solution but
> it would be elegant IMO.

I thought of scannerless parsing, but the input is UTF8 encoded,
which makes using Ulex all the more convenient and less error-prone.

Anyway, I was looking at Dypgen's early actions, and an idea
occurred to me: I can create dummy empty actions that simply
change a global "parsing context" variable:

inline:
  | (...)
  | BEGIN_VERB enter_verb RAW END_VERB exit_verb {Ast.Verbatim $3}
  | (...)

enter_verb:  /* empty */  {Global.context := Global.Verbatim}
exit_verb:   /* empty */  {Global.context := Global.General}


Still hackish, but better than creating a state machine...

Cheers,
Dario Teixeira

Re: [Caml-list] Re: The lexer hack

[Francois Pottier]

Hello,

On Tue, Nov 10, 2009 at 08:26:23AM -0800, Dario Teixeira wrote:
> Anyway, I was looking at Dypgen's early actions, and an idea
> occurred to me: I can create dummy empty actions that simply
> change a global "parsing context" variable:
> 
> inline:
>   | (...)
>   | BEGIN_VERB enter_verb RAW END_VERB exit_verb {Ast.Verbatim $3}
>   | (...)
> 
> enter_verb:  /* empty */  {Global.context := Global.Verbatim}
> exit_verb:   /* empty */  {Global.context := Global.General}
> 
> 
> Still hackish, but better than creating a state machine...

Interesting. Have you confirmed that this works? I am slightly worried by the
fact that an LR parser reads one token ahead, i.e. one token past BEGIN_VERB
might already have been read before the enter_verb semantic action is
executed. If that is so, then this token would be read while the lexer is
still in the wrong mode.

-- 
François Pottier
Francois.Pottier@inria.fr
http://gallium.inria.fr/~fpottier/

Re: [Caml-list] Re: The lexer hack

[Dario Teixeira]

Hi,

> Interesting. Have you confirmed that this works? I am slightly
> worried by the fact that an LR parser reads one token ahead,
> i.e. one token past BEGIN_VERB might already have been read
> before the enter_verb semantic action is executed. If that is
> so, then this token would be read while the lexer is still in
> the wrong mode.

Yes, I was just thinking about that as well... :-)
I think I can pile another hack on top of the dummy action:
dummy tokens to take care of the readahead issue.  Though
this has the potential to get comically silly pretty quickly!

I'll report later...

cheers,
Dario Teixeira

Re: [Caml-list] Re: The lexer hack

[Martin Jambon]

Dario Teixeira wrote:
> Hi,
> 
>> Interesting. Have you confirmed that this works? I am slightly
>> worried by the fact that an LR parser reads one token ahead,
>> i.e. one token past BEGIN_VERB might already have been read
>> before the enter_verb semantic action is executed. If that is
>> so, then this token would be read while the lexer is still in
>> the wrong mode.
> 
> Yes, I was just thinking about that as well... :-)
> I think I can pile another hack on top of the dummy action:
> dummy tokens to take care of the readahead issue.  Though
> this has the potential to get comically silly pretty quickly!
> 
> I'll report later...

If the lexer to use can be determined by only one token (BEGIN_VERB), I think
you can change the state in the lexer like this:

rule token state = parse
 ""   { match !state with
             `Normal -> normal_token state lexbuf
           | `Verbatim -> verbatim_token state lexbuf
      }

and normal_token state = parse
  ...
| "\\begin{verbatim}"   { state := `Verbatim; BEGIN_VERB }

and verbatim_token state = parse
  ...                  { RAW (...) }
| "\\end{verbatim}"    { state := `Normal; END_VERB }



An even simpler option, if possible in your case, is to use a single token for
the whole verbatim section:

rule token = parse
  ...
| "\\begin{verbatim}"   { finish_verbatim lexbuf }

and finish_verbatim = shortest
  _* as s "\\end{verbatim}"   { RAW s }




Martin

-- 
http://mjambon.com/

Re: [Caml-list] Re: The lexer hack

[Dario Teixeira]

Hi,

> If the lexer to use can be determined by only one token
> (BEGIN_VERB), I think you can change the state in the lexer
> like this:

Unfortunately the language features some verbatim-like environments
where choosing the right scanner entails knowing more about the
context than what is available to the lexer.  As an example, in
the command \link{url}{text}, the "url" should be interpreted
verbatim, whereas the "text" portion uses the general scanner.
The parser is fully aware of the context, of course, which is
why the "dummy action" solution worked fine.

(And yes, I thought of having different delimiters for different
scanning contexts, but in many ways it would make the language
more cumbersome for the user).

Cheers,
Dario Teixeira