[Caml-list] updating position in an ocamllex lexer

["Sébastien Hinderer" <Sebastien.Hinderer@inria.fr>]

Dear all,

I have to maintain a lexer which did not, so far, compute token
positions during the lexing step. The positions were computed
later, which was not very eficient. So I am now trying to compute
positions during the lexing and have a problem to figure out what
to do for one token:

  | ['\n'] [' ' '\t' '\r' '\011' '\012' ]* { ... }

When this token is met and contains just "\n", calling new_line on the
lexbuf is enough to keep further position information accurate.

However, if the recognized token is, say, "\n\t", then it seems that the
column for further tokens will be incorrect.

I assumed that I had to manually update the current position and added
code like so:

  let s = Lexing.lexeme lexbuf in
  let l = String.length s in
  let t = TCommentNewline (tokinfo lexbuf) in
  (* Adjust the column manually *)
  Lexing.new_line lexbuf;
  let lcp = lexbuf.lex_curr_p in
  lexbuf.lex_curr_p <- { lcp with
    pos_cnum = lcp.pos_bol + l - 1;
  };
  t

But that does not seem to work.

Does somebody know how such tokens should be handled, please?

Many thanks in advance for any help,

Sébastien.