From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: caml-list@sympa.inria.fr Delivered-To: caml-list@sympa.inria.fr Received: from mail2-relais-roc.national.inria.fr (mail2-relais-roc.national.inria.fr [192.134.164.83]) by sympa.inria.fr (Postfix) with ESMTPS id 8B1687EEF8 for ; Thu, 30 Jul 2015 10:21:09 +0200 (CEST) Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of leo@lpw25.net) identity=pra; client-ip=66.111.4.27; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="leo@lpw25.net"; x-sender="leo@lpw25.net"; x-conformance=sidf_compatible Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of leo@lpw25.net) identity=mailfrom; client-ip=66.111.4.27; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="leo@lpw25.net"; x-sender="leo@lpw25.net"; x-conformance=sidf_compatible Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of postmaster@out3-smtp.messagingengine.com) identity=helo; client-ip=66.111.4.27; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="leo@lpw25.net"; x-sender="postmaster@out3-smtp.messagingengine.com"; x-conformance=sidf_compatible X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A0BOAQAB3rlVnBsEb0Jbg25prCuRc4V5AoFLTAEBAQEBARIBAQEBAQgLCQkhLoQkAQEEQC4MDwsYDRARLBkSK4gHAxINuAyFW4tjA4U5AQEBAQYBAQEBAQEBFQaLToUOF4MBgRSFagyHEodzhHuJE0aDWosxhD2DZII0HIFvIjGCTAEBAQ X-IPAS-Result: A0BOAQAB3rlVnBsEb0Jbg25prCuRc4V5AoFLTAEBAQEBARIBAQEBAQgLCQkhLoQkAQEEQC4MDwsYDRARLBkSK4gHAxINuAyFW4tjA4U5AQEBAQYBAQEBAQEBFQaLToUOF4MBgRSFagyHEodzhHuJE0aDWosxhD2DZII0HIFvIjGCTAEBAQ X-IronPort-AV: E=Sophos;i="5.15,576,1432591200"; d="scan'208";a="172102219" Received: from out3-smtp.messagingengine.com ([66.111.4.27]) by mail2-smtp-roc.national.inria.fr with ESMTP/TLS/DHE-RSA-AES256-SHA; 30 Jul 2015 10:21:08 +0200 Received: from compute2.internal (compute2.nyi.internal [10.202.2.42]) by mailout.nyi.internal (Postfix) with ESMTP id 1859020A37 for ; Thu, 30 Jul 2015 04:21:07 -0400 (EDT) Received: from web3 ([10.202.2.213]) by compute2.internal (MEProxy); Thu, 30 Jul 2015 04:21:07 -0400 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=lpw25.net; h= content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-sasl-enc :x-sasl-enc; s=mesmtp; bh=SJk8G3h9XfrLsvf6Lzy2Tfw67Bw=; b=Gi20So AT8eHxuOtqJ5IPW0pBOQUG5+P9G85f1WjlCsNERihdrLtSfUBzfKnwFkGh0w7n1N YRc8LjQrcUc8DzcnouFxqjYY/jqDAjHsfCkviourgBvQSDy6akZwluYqIA+iriIV DeWRQPuLiT1IKo8zPgV2SryKw6vnIc9zAnWgg= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d= messagingengine.com; h=content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-sasl-enc:x-sasl-enc; s=smtpout; bh=SJk8G3h9XfrLsvf 6Lzy2Tfw67Bw=; b=D+VyXV35H7oAIPcNE4nwLzvmkiIAtXnDPn8zRpw4vIlK6pD jJ5V73koqsmaJgte4ucsbdjY+Qb7OMLmx2iIPfvzMb6EPlO/FVK1WYczWNd2nIbn 2Aaar3aa1SYg2BPN7wHjS7ed8m9JguHOTH2vERQnt+22BsusAPd8TTF/Vm/w= Received: by web3.nyi.internal (Postfix, from userid 99) id D2BA3111CF6; Thu, 30 Jul 2015 04:21:06 -0400 (EDT) Message-Id: <1438244466.807779.337036689.55800F88@webmail.messagingengine.com> X-Sasl-Enc: hJ0lhEWSOBUhCb0WXxVZGZCuI/H5Vqsq+a5Npws5CtMB 1438244466 From: Leo White To: caml-list@inria.fr MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="ISO-8859-1" X-Mailer: MessagingEngine.com Webmail Interface - ajax-63a5d8c6 Date: Thu, 30 Jul 2015 04:21:06 -0400 In-Reply-To: <20150728155638.GA27761@pl-59055.rocqadm.inria.fr> References: <20150728155638.GA27761@pl-59055.rocqadm.inria.fr> Subject: Re: [Caml-list] updating position in an ocamllex lexer I think the problem with your attempt to manually update the current position is that you are adjusting pos_cnum, which is the number of characters since the beginning of the whole file, not the number of characters since the beginning of the line. Instead you should be adjusting pos_bol. For example, if you look at the source for Lexing.new_line: let new_line lexbuf =3D let lcp =3D lexbuf.lex_curr_p in lexbuf.lex_curr_p <- { lcp with pos_lnum =3D lcp.pos_lnum + 1; pos_bol =3D lcp.pos_cnum; } you can see that it adds one to the line number and sets pos_bol to be the current position. Regards, Leo On Tue, 28 Jul 2015, at 11:56 AM, S=E9bastien Hinderer wrote: > Dear all, >=20 > I have to maintain a lexer which did not, so far, compute token > positions during the lexing step. The positions were computed > later, which was not very eficient. So I am now trying to compute > positions during the lexing and have a problem to figure out what > to do for one token: >=20 > | ['\n'] [' ' '\t' '\r' '\011' '\012' ]* { ... } >=20 > When this token is met and contains just "\n", calling new_line on the > lexbuf is enough to keep further position information accurate. >=20 > However, if the recognized token is, say, "\n\t", then it seems that the > column for further tokens will be incorrect. >=20 > I assumed that I had to manually update the current position and added > code like so: >=20 > let s =3D Lexing.lexeme lexbuf in > let l =3D String.length s in > let t =3D TCommentNewline (tokinfo lexbuf) in > (* Adjust the column manually *) > Lexing.new_line lexbuf; > let lcp =3D lexbuf.lex_curr_p in > lexbuf.lex_curr_p <- { lcp with > pos_cnum =3D lcp.pos_bol + l - 1; > }; > t >=20 > But that does not seem to work. >=20 > Does somebody know how such tokens should be handled, please? >=20 > Many thanks in advance for any help, >=20 > S=E9bastien. >=20 > --=20 > Caml-list mailing list. Subscription management and archives: > https://sympa.inria.fr/sympa/arc/caml-list > Beginner's list: http://groups.yahoo.com/group/ocaml_beginners > Bug reports: http://caml.inria.fr/bin/caml-bugs