ocamllex question

ocamllex question

[Robert Muller]

I am attempting to use ocamllex together with ocamlyacc to parse a  
subset of python. Python uses indentation to denote
statement blocks so a lexer is sometimes required to return a sequence  
of tokens without advancing the input pointer. In
particular, a lexer for python should return a sequence of so-called  
DEDENT tokens when indented fragments
end. E.g.,

def f(x):
	statement1;
	statement2;
		statement3;
		statement4;
A

the lexer should return two consecutive DEDENT tokens between the '\n'  
at the end of statement4 and the token for A.

Looking at the documentation and examples, it isn't clear how to  
convince the generated lexer to not advance the input pointer
so that two consecutive DEDENT tokens can be returned before the token  
for A is returned.

Any ocamllex perts out there?

Thanks,
Bob Muller

Re: [Caml-list] ocamllex question

[Martin Jambon]

Robert Muller wrote:
> I am attempting to use ocamllex together with ocamlyacc to parse a
> subset of python. Python uses indentation to denote
> statement blocks so a lexer is sometimes required to return a sequence
> of tokens without advancing the input pointer. In
> particular, a lexer for python should return a sequence of so-called
> DEDENT tokens when indented fragments
> end. E.g.,
> 
> def f(x):
>     statement1;
>     statement2;
>         statement3;
>         statement4;
> A
> 
> the lexer should return two consecutive DEDENT tokens between the '\n'
> at the end of statement4 and the token for A.

What I would do is:

1. pass an argument to each "rule" function, containing the stack of
indentation information (current block and parent blocks, with first line
number and indentation).

2. let each rule produce as many tokens as necessary and return lists of tokens

3. create a token stream for ocamlyacc/menhir that would call the
ocamllex-generated functions as needed; these would put the tokens into a
queue. Refill when the queue is empty.

4. Figure how make good error reports :-)



Martin


> Looking at the documentation and examples, it isn't clear how to
> convince the generated lexer to not advance the input pointer
> so that two consecutive DEDENT tokens can be returned before the token
> for A is returned.
> 
> Any ocamllex perts out there?
> 
> Thanks,
> Bob Muller
> 
> _______________________________________________
> Caml-list mailing list. Subscription management:
> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
> Archives: http://caml.inria.fr
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
> 


-- 
http://mjambon.com/

Re: [Caml-list] Ocamllex question

[Michael Wohlwend]

Hi,

On Sunday 23 October 2005 20:02, Matt Gushee wrote:
>
>   and comment = parse
>       [ ^ '\n' ]                        { comment lexbuf }
>
>     | '\n'                              { SEP; dict lexbuf }

I'm not quite sure whether I understood your problem, but... doesn' it help to 
just return a SEP after a comment:

rule dict = parse
 ...
 | ',' | '\n'   { SEP }
 | '#' [^ '\n']* '\n'  { SEP }
 ...


 Michael

Re: [Caml-list] ocamllex question

[Alex Baretta]

skaller wrote:
> Can eof be read from a lexbuf more than once by an ocamllex lexer?
> In particular is a recursive lexer matches an eof and
> returns to its caller, can the parent caller still read
> another eof?
> 
> In other words, is the character stream postpended by one eof
> or an infinite stream of them?

"eof" in ocamllex is a "condition" not a token. It's like "\b" in emacs,
which matches the empty string but only at the beginning or end of a
word. In ocamllex "eof" matches the empty string at the end of a lexbuf,
thus matching eof is non-destructive lexbuf-wise and can repeated any
number of times.

Alex

-- 
*********************************************************************
http://www.barettadeit.com/
Baretta DE&IT
A division of Baretta SRL

tel. +39 02 370 111 55
fax. +39 02 370 111 54

Our technology:

The Application System/Xcaml (AS/Xcaml)
<http://www.asxcaml.org/>

The FreerP Project
<http://www.freerp.org/>