Re: [Caml-list] I don t get the lexer

[Nicolas Pouillard <nicolas.pouillard@inria.fr>]

On 2/12/06, jean-david hsu <jhsu1@email.sjsu.edu> wrote:
> Hello everyone
Hi,

> how come my lexer does not break "?!" both defined as keywords but puts
> "." aside?

Because the lexer of Genlex is not really generated but parametrized
by a keyword table.

Concerned rules are:
 | Some
        ('!' | '%' | '&' | '$' | '#' | '+' | '/' | ':' | '<' | '=' | '>' |
         '?' | '@' | '\\' | '~' | '^' | '|' | '*' as c) ->
        Stream.junk strm__;
        let s = strm__ in reset_buffer (); store c; ident2 s
[...]
| Some c -> Stream.junk strm__; Some (keyword_or_error c)
[...]
  and ident2 (strm__ : _ Stream.t) =
    match Stream.peek strm__ with
      Some
        ('!' | '%' | '&' | '$' | '#' | '+' | '-' | '/' | ':' | '<' | '=' |
         '>' | '?' | '@' | '\\' | '~' | '^' | '|' | '*' as c) ->
        Stream.junk strm__; let s = strm__ in store c; ident2 s

In short '?' and '!' are in the same character class but '.' is
treated by the default case. Since the lexer search the longest token
that matches, "?!" are packed together but not with a '.' .

--
Nicolas Pouillard