Ocamllex newline counting...

Ocamllex newline counting...

[Till Varoquaux]

The title is pretty self explanetory: Ocamllex is able to keep track
of positions automatically but it needs help with new lines ( you need
to register new lines with a function like:

 let newline lexbuf =
    let pos = lexbuf.lex_curr_p in
    lexbuf.lex_curr_p <-
      { pos with pos_lnum = pos.pos_lnum + 1; pos_bol = pos.pos_cnum }
).
This tends to pollute the code and require you to add additional rules
and underlying machinery. I can see one easy workaround: pipe the
function you build your lexer from trough an additional function that
registers newlines. This seems a bit costly at run time but should be
just fine in most cases.

Is there any fundamental reason I am missing why newlines are not
handled natively in the generated automaton?

Cheers,

Till

Re: [Caml-list] Ocamllex newline counting...

[Berke Durak]

Till Varoquaux a écrit :
> The title is pretty self explanetory: Ocamllex is able to keep track
> of positions automatically but it needs help with new lines ( you need
> to register new lines with a function like:
> 
>  let newline lexbuf =
>     let pos = lexbuf.lex_curr_p in
>     lexbuf.lex_curr_p <-
>       { pos with pos_lnum = pos.pos_lnum + 1; pos_bol = pos.pos_cnum }
> ).
> This tends to pollute the code and require you to add additional rules
> and underlying machinery. I can see one easy workaround: pipe the
> function you build your lexer from trough an additional function that
> registers newlines. This seems a bit costly at run time but should be
> just fine in most cases.
> 
> Is there any fundamental reason I am missing why newlines are not
> handled natively in the generated automaton?

Well, as you said, wrapping using Lexing.from_function to count newlines
or whatever you want is "just fine" in most cases; especially given that
you'll often want to build a byte-offset-to-line-number table while parsing.
Putting specialized logic in lexing doesn't seem very elegant...

My wild guess would be that the lexer_position record probably got those
extra lnum fields because they were needed in the compiler and they didn't
want to change the interface to insert a type parameter for the positions.
Am I right?

-- 
Berke DURAK

Re: [Caml-list] Ocamllex newline counting...

[Nathaniel Gray]

On Fri, Mar 7, 2008 at 2:57 AM, Till Varoquaux <till.varoquaux@gmail.com> wrote:
>
>  Is there any fundamental reason I am missing why newlines are not
>  handled natively in the generated automaton?

My guess is that this is for the times you're working on the output of
a preprocessor.  It's common for preprocessors to add extra lines that
shouldn't be counted for error reporting purposes.  I agree that there
should be better support for the common case, though.  That's why I
reported this feature request for a helper function to factor out some
of the boilerplate code, which was granted:

http://caml.inria.fr/mantis/view.php?id=3930

Cheers,
-n8

-- 
>>>-- Nathaniel Gray -- Caltech Computer Science ------>
>>>-- Mojave Project -- http://mojave.cs.caltech.edu -->