ocamllex question

ocamllex question

[Robert Muller]

I am attempting to use ocamllex together with ocamlyacc to parse a  
subset of python. Python uses indentation to denote
statement blocks so a lexer is sometimes required to return a sequence  
of tokens without advancing the input pointer. In
particular, a lexer for python should return a sequence of so-called  
DEDENT tokens when indented fragments
end. E.g.,

def f(x):
	statement1;
	statement2;
		statement3;
		statement4;
A

the lexer should return two consecutive DEDENT tokens between the '\n'  
at the end of statement4 and the token for A.

Looking at the documentation and examples, it isn't clear how to  
convince the generated lexer to not advance the input pointer
so that two consecutive DEDENT tokens can be returned before the token  
for A is returned.

Any ocamllex perts out there?

Thanks,
Bob Muller

Re: [Caml-list] ocamllex question

[Martin Jambon]

Robert Muller wrote:
> I am attempting to use ocamllex together with ocamlyacc to parse a
> subset of python. Python uses indentation to denote
> statement blocks so a lexer is sometimes required to return a sequence
> of tokens without advancing the input pointer. In
> particular, a lexer for python should return a sequence of so-called
> DEDENT tokens when indented fragments
> end. E.g.,
> 
> def f(x):
>     statement1;
>     statement2;
>         statement3;
>         statement4;
> A
> 
> the lexer should return two consecutive DEDENT tokens between the '\n'
> at the end of statement4 and the token for A.

What I would do is:

1. pass an argument to each "rule" function, containing the stack of
indentation information (current block and parent blocks, with first line
number and indentation).

2. let each rule produce as many tokens as necessary and return lists of tokens

3. create a token stream for ocamlyacc/menhir that would call the
ocamllex-generated functions as needed; these would put the tokens into a
queue. Refill when the queue is empty.

4. Figure how make good error reports :-)



Martin


> Looking at the documentation and examples, it isn't clear how to
> convince the generated lexer to not advance the input pointer
> so that two consecutive DEDENT tokens can be returned before the token
> for A is returned.
> 
> Any ocamllex perts out there?
> 
> Thanks,
> Bob Muller
> 
> _______________________________________________
> Caml-list mailing list. Subscription management:
> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
> Archives: http://caml.inria.fr
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
> 


-- 
http://mjambon.com/

Ocamllex question

[Matt Gushee]

Hello, people--

In a lexer definition with two or more entry points, is there a way to
emit a lexeme and pass control to another entrypoint in one action?

The specific problem I am trying to deal with is a configuration file
format that includes comments denoted with an initial '#' character. I
would like to support the typical usage of '#', where a comment may
begin either at the beginning of the line, or after a declaration that I
want to capture, and in either case it extends to the end of the line.

So in general, anything after '#' up to the end of a line should be
ignored, which I think requires a separate 'comment' entrypoint. At the
end of the line, control returns to the main entry point. So my first
cut looks like this:

  rule dict = parse
      [' ']                             { dict lexbuf }
    | '#'                               { comment lexbuf }
    | word                              { WORD (Lexing.lexeme lexbuf) }
    | ':'                               { COLON }
    | '{'                               { DS }
    | '}'                               { DE }
    | ',' | '\n'                        { SEP }
    | eof                               { EOF }
  and comment = parse
      [ ^ '\n' ]                        { comment lexbuf }
    | '\n'                              { dict lexbuf }

So far so good. BUT, for the sake of simplicity (for users, not for me
;-)), my syntax has line endings as separators, and in order to support
comments following non-comments on the same line, a line ending after a
comment should be interpreted as a separator. So what I want to do is
something like:

  and comment = parse
      [ ^ '\n' ]                        { comment lexbuf }
    | '\n'                              { SEP; dict lexbuf }

But that doesn't work, of course. Maybe the solution is to push SEP back
onto the head of the buffer, but I don't see a way to do that.

Or would it be better to simply tag the comment text with, say, a
COMMENT symbol and pass it through to the parser?

--
Matt Gushee
Englewood, CO, USA

ocamllex question

[skaller]

Can eof be read from a lexbuf more than once by an ocamllex lexer?
In particular is a recursive lexer matches an eof and
returns to its caller, can the parent caller still read
another eof?

In other words, is the character stream postpended by one eof
or an infinite stream of them?

-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net