Re: [Caml-list] ocamllex question

[Martin Jambon <martin.jambon@ens-lyon.org>]

Robert Muller wrote:
> I am attempting to use ocamllex together with ocamlyacc to parse a
> subset of python. Python uses indentation to denote
> statement blocks so a lexer is sometimes required to return a sequence
> of tokens without advancing the input pointer. In
> particular, a lexer for python should return a sequence of so-called
> DEDENT tokens when indented fragments
> end. E.g.,
> 
> def f(x):
>     statement1;
>     statement2;
>         statement3;
>         statement4;
> A
> 
> the lexer should return two consecutive DEDENT tokens between the '\n'
> at the end of statement4 and the token for A.

What I would do is:

1. pass an argument to each "rule" function, containing the stack of
indentation information (current block and parent blocks, with first line
number and indentation).

2. let each rule produce as many tokens as necessary and return lists of tokens

3. create a token stream for ocamlyacc/menhir that would call the
ocamllex-generated functions as needed; these would put the tokens into a
queue. Refill when the queue is empty.

4. Figure how make good error reports :-)



Martin


> Looking at the documentation and examples, it isn't clear how to
> convince the generated lexer to not advance the input pointer
> so that two consecutive DEDENT tokens can be returned before the token
> for A is returned.
> 
> Any ocamllex perts out there?
> 
> Thanks,
> Bob Muller
> 
> _______________________________________________
> Caml-list mailing list. Subscription management:
> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
> Archives: http://caml.inria.fr
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
> 


-- 
http://mjambon.com/