From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: caml-list@sympa.inria.fr Delivered-To: caml-list@sympa.inria.fr Received: from mail3-relais-sop.national.inria.fr (mail3-relais-sop.national.inria.fr [192.134.164.104]) by sympa.inria.fr (Postfix) with ESMTPS id 55C5B7F164 for ; Sun, 6 Mar 2016 23:59:36 +0100 (CET) IronPort-PHdr: 9a23:98NFghUljE1dKYN0x92dRwdrI/7V8LGtZVwlr6E/grcLSJyIuqrYZhyCt8tkgFKBZ4jH8fUM07OQ6PC/HzxYqs7c+Fk5M7VyFDY9wf0MmAIhBMPXQWbaF9XNKxIAIcJZSVV+9Gu6O0UGUOz3ZlnVv2HgpWVKQka3CwN5K6zPF5LIiIzvjqbpq8KVM10D3Gf1SIgxBSv1hD2ZjtMRj4pmJ/R54TryiVwMRd5rw3h1L0mYhRf265T41pdi9yNNp6BprJYYAu3SNp41Rr1ADTkgL3t9pIiy7UGCHkOz4S4zW28MkxdMSzPO7BzgU4255iTzvPB81S3cJsb2QKo5Qxyt6q5qTFnjjyJRZBAj92SCo8psi6QTihuqpx15i9rdZp2SMvB3daPce/sLTGpGWYBaUCkXUdD0VJcGE+dUZbUQlIL6vVZb6ELmXQQ= Authentication-Results: mail3-smtp-sop.national.inria.fr; spf=None smtp.pra=antonbachin@yahoo.com; spf=Pass smtp.mailfrom=antonbachin@yahoo.com; spf=None smtp.helo=postmaster@nm8-vm0.bullet.mail.ne1.yahoo.com Received-SPF: None (mail3-smtp-sop.national.inria.fr: no sender authenticity information available from domain of antonbachin@yahoo.com) identity=pra; client-ip=98.138.91.23; receiver=mail3-smtp-sop.national.inria.fr; envelope-from="antonbachin@yahoo.com"; x-sender="antonbachin@yahoo.com"; x-conformance=sidf_compatible Received-SPF: Pass (mail3-smtp-sop.national.inria.fr: domain of antonbachin@yahoo.com designates 98.138.91.23 as permitted sender) identity=mailfrom; client-ip=98.138.91.23; receiver=mail3-smtp-sop.national.inria.fr; envelope-from="antonbachin@yahoo.com"; x-sender="antonbachin@yahoo.com"; x-conformance=sidf_compatible; x-record-type="v=spf1" Received-SPF: None (mail3-smtp-sop.national.inria.fr: no sender authenticity information available from domain of postmaster@nm8-vm0.bullet.mail.ne1.yahoo.com) identity=helo; client-ip=98.138.91.23; receiver=mail3-smtp-sop.national.inria.fr; envelope-from="antonbachin@yahoo.com"; x-sender="postmaster@nm8-vm0.bullet.mail.ne1.yahoo.com"; x-conformance=sidf_compatible X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A0BHBAD0tdxWlhdbimJdFoIYgV5tEKl/KoUlim6BaSGFbgKBHTkTAQEBAQEBAQEQAQEBAQcNCQkhL4ItghQBAQEDASMdASsKAwEECwYFEQMBAgEnAwICISQBCQgGExuHcQEDCggOsESBaoJXhG0BIycDCoRHAQEBAQEBAQEBAQEBAQEBAQEBAQEBDwYMAgGGCIFvCIFJfYI9gh2CYCuBDwWOJIkGhWOGFYF1G4IThnsghTGHDIdJDxMBgjUegW9LAYk8AQEB X-IPAS-Result: A0BHBAD0tdxWlhdbimJdFoIYgV5tEKl/KoUlim6BaSGFbgKBHTkTAQEBAQEBAQEQAQEBAQcNCQkhL4ItghQBAQEDASMdASsKAwEECwYFEQMBAgEnAwICISQBCQgGExuHcQEDCggOsESBaoJXhG0BIycDCoRHAQEBAQEBAQEBAQEBAQEBAQEBAQEBDwYMAgGGCIFvCIFJfYI9gh2CYCuBDwWOJIkGhWOGFYF1G4IThnsghTGHDIdJDxMBgjUegW9LAYk8AQEB X-IronPort-AV: E=Sophos;i="5.22,548,1449529200"; d="scan'208,217";a="167339880" Received: from nm8-vm0.bullet.mail.ne1.yahoo.com ([98.138.91.23]) by mail3-smtp-sop.national.inria.fr with ESMTP/TLS/AES128-GCM-SHA256; 06 Mar 2016 23:59:34 +0100 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1457305172; bh=6V0XxZrVSt4Y3KiAm+0rMtTisgIyRO/1cY9JUhhF34Y=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject; b=JKxTkwM+vFyUii+DRo8ru84l1JfJ/4f3Si0JpdCkYto7kcpTSTA0A4R0RWB6niv79u4oUcEhSzECbSjSq9kbOb+dr4WFZUB7wkpwsnrjE48cBoqAY0WP64PslOqkSKz5uP8Mfv8k4MVJW5wveWd99yXzNOtDwjJma0IAQ+VRikeSh5sOSNyKbPQ6MTl0OB5T6RIat/TVgLtMx03v6SLnkHNt7HKq9qORiCgIwoBUJ9uLC0mRQyyUvhR8iadoNcE5EnZENCkH03XO/BssCb0YtEjyPt9g+KLKLytjY4x9uyOCGgD6zXME2jiDm9yLsLwUJBT/RCU3IRLdI4nqoZ5Cvw== Received: from [98.138.226.178] by nm8.bullet.mail.ne1.yahoo.com with NNFMP; 06 Mar 2016 22:59:32 -0000 Received: from [98.138.226.124] by tm13.bullet.mail.ne1.yahoo.com with NNFMP; 06 Mar 2016 22:59:32 -0000 Received: from [127.0.0.1] by smtp203.mail.ne1.yahoo.com with NNFMP; 06 Mar 2016 22:59:32 -0000 X-Yahoo-Newman-Id: 209594.97966.bm@smtp203.mail.ne1.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: sxt78cUVM1k9nqlF54OdBvoBTJ11ELLGomzF9pb821ZZoXZ RAULj_S357X0supGp4RP2bNCHIGQEPWzmjknh7MX8lov6YUwzbbJJySSz3tE Jqo4DWEhlgl3ALGC8d.B6MTlT_HmLLDPJBkLCnZprzScfH4SGheM2FbAfQCQ B2BKkS0mEsY0mtSpQJo25HxldORkzfYhj9KdPwA6nBko_.5J2hAi9nbwrmkT bjZjQy6AmfRZPFSUm3ab9hr.q2mDX0oL44NvgOxU9zj6Wc5knUkZ2bAH2E57 8BIimmjwb22rk6W597H4i5Kewlh2keaZC.LBVZMakqNEz.iFSD0BaShKG6Jn ZCjKjFxAniiaQUSOT98QPgvkR6QzoSGNoJjkmEU1zgyOcuT5hsVkzbOzU755 oNyMEK2_BWScq8dWFg0GCrU4z4xOgVocn5yNeRaIXdHFmcBJNILdFTVvzpJR S1ZtPjiXvs7PhlE9yfiZDqXBKY8dNSmdeqetuFR.lNgqbP2bxTAe786G8hzL Rzk8PxsARkQbFzeoJ5JCMHFyUgVnQmkz1QPbe7Ib1Hn._sd6klpmT6yIihlw fTA-- X-Yahoo-SMTP: ddtAESaswBBsjSthz_dVP91gr2NDfymF Content-Type: multipart/alternative; boundary="Apple-Mail=_1D761AAA-612C-45A6-9FEB-3857F12B0C4C" Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2070.6\)) From: Anton Bachin In-Reply-To: Date: Sun, 6 Mar 2016 16:59:29 -0600 Cc: Vincent Jacques , caml users , Francois Pottier X-Mao-Original-Outgoing-Id: 478997969.378683-3de6df930bb5c3820481597a3ab39950 Message-Id: <6080D67A-8238-4F27-AEA6-C2B00704F2AA@yahoo.com> References: To: Gabriel Scherer X-Mailer: Apple Mail (2.2070.6) Subject: Re: [Caml-list] Test coverage of generated lexers/parsers --Apple-Mail=_1D761AAA-612C-45A6-9FEB-3857F12B0C4C Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 If there is a good, general, alternative approach for this, we can support = it in Bisect_ppx. Unfortunately, I don=E2=80=99t know enough about Menhir t= o be able to propose anything specific at this point. Best, Anton > On Mar 6, 2016, at 16:53, Gabriel Scherer wro= te: >=20 > This is an interesting question and, as far as I know, there is no good s= olution using existing versions of the interacting tools. >=20 > Below very simple patch that will add (*BISECT-IGNORE*) in front of every= line of code generated by Menhir, except those written by the programmer (= the "strecthes" in Menhir-speak). It applies cleanly on top of the latest r= eleased Menhir archive, > http://gallium.inria.fr/~fpottier/menhir/menhir-20160303.tar.gz >=20 > The patch as-is is obviously a hack: it would need to be a configuration = option when running menhir, and hard-coding Bisect (or bisect_ppx)'s syntax= into Menhir is not elegant. One could try to have a configuration option t= o let users write a fixed string (or comment) at the beginning of each gene= rated code line, but I'm not sure whether Fran=C3=A7ois Pottier (in cc:) wo= uld consider this is elegant enough. Fran=C3=A7ois, would you comment on wh= ether this is a direction that seems acceptable to you? >=20 > (Bisect support ignoring entire regions at once by using (*BISECT-IGNORE-= BEGIN*) and (*BISECT-IGNORE-END*); we could try to implement that instead o= f a per-line change, but I suspect that it would be slightly harder to impl= ement (you have to hook the beginning of input, end of input, and around ea= ch user-code insertion) for no real gain.) >=20 > Toggling code-coverage semantics by inserting comments is not a very nice= interface (although rather logical when you think of the level of generali= ty required), so it's a bit frustrating that parser generators would have t= o play at this level. It would be better to have a more structured, unified= interface supported by all the code-coverage tools, but to my knowledge no= such thing exists. >=20 >=20 > From d595ba5149a314c56623e1735af7678f5f62d525 Mon Sep 17 00:00:00 2001 > From: Gabriel Scherer > > Date: Sun, 6 Mar 2016 17:43:14 -0500 > Subject: [PATCH] output (*BISECT-IGNORE*) in front of each > non-programmer-written line >=20 > EXPERIMENTAL PATCH: this should of course be turned into an explicit opti= on > --- > src/printer.ml | 1 + > 1 file changed, 1 insertion(+) >=20 > diff --git a/src/printer.ml b/src/printer.ml > index ea978bc..714bb08 100644 > --- a/src/printer.ml > +++ b/src/printer.ml > @@ -46,6 +46,7 @@ let rawnl f =3D >=20=20 > let nl f =3D > rawnl f; > + output_string f "(*BISECT-IGNORE*)"; > output_substring f whitespace 0 !indentation >=20=20 > let indent ofs producer f x =3D > --=20 > 2.5.0 >=20 >=20 >=20 > On Sun, Mar 6, 2016 at 2:53 PM, Vincent Jacques > wrote: > Hello, >=20 > Does somebody have experience measuring test coverage of generated lexers= /parsers? >=20 > I'm using ocamllex/ocamlyacc [1] (but I can switch to Menhir [2]) to gene= rate a lexer/parser. In my tests, I simply check that some input strings gi= ve the ASTs I expect. >=20 > I usually use Bisect [3] to make sure that my tests cover the code I inte= nded to cover, but in that configuration, Bisect is lost between the .mll/.= mly files and the generated .ml files and produces useless reports. >=20 > How would you measure test coverage in that case? >=20 > Thanks, >=20 > [1] http://caml.inria.fr/pub/docs/manual-ocaml/lexyacc.html > [2] http://gallium.inria.fr/~fpottier/menhir/ > [3] http://bisect.x9c.fr/ > --=20 > Vincent Jacques > http://vincent-jacques.net >=20 > "S'il n'y a pas de solution, c'est qu'il n'y a pas de probl=C3=A8me" > Devise Shadock >=20 > <0001-output-BISECT-IGNORE-in-front-of-each-non-programmer.patch> --Apple-Mail=_1D761AAA-612C-45A6-9FEB-3857F12B0C4C Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8
If= there is a good, general, alternative approach for this, we can support it= in Bisect_ppx. Unfortunately, I don=E2=80=99t know enough about Menhir to = be able to propose anything specific at this point.
Best,
Anton
On M= ar 6, 2016, at 16:53, Gabriel Scherer <gabriel.scherer@gmail.com> wrote:

This is = an interesting question and, as far as I know, there is no good solution us= ing existing versions of the interacting tools.

Below very simple patch that will add (*BISECT-IGNORE*) in front o= f every line of code generated by Menhir, except those written by the progr= ammer (the "strecthes" in Menhir-speak). It applies cleanly on top of the l= atest released Menhir archive,
  http://gall= ium.inria.fr/~fpottier/menhir/menhir-20160303.tar.gz

The patch as-is is obviously a hack: it would need to be a= configuration option when running menhir, and hard-coding Bisect (or bisec= t_ppx)'s syntax into Menhir is not elegant. One could try to have a configu= ration option to let users write a fixed string (or comment) at the beginni= ng of each generated code line, but I'm not sure whether Fran=C3=A7ois Pott= ier (in cc:) would consider this is elegant enough. Fran=C3=A7ois, would yo= u comment on whether this is a direction that seems acceptable to you?

(Bisect support ignoring entire regions at o= nce by using (*BISECT-IGNORE-BEGIN*) and (*BISECT-IGNORE-END*); we could tr= y to implement that instead of a per-line change, but I suspect that it wou= ld be slightly harder to implement (you have to hook the beginning of input= , end of input, and around each user-code insertion) for no real gain.)

Toggling code-coverage semantics by inserti= ng comments is not a very nice interface (although rather logical when you = think of the level of generality required), so it's a bit frustrating that = parser generators would have to play at this level. It would be better to h= ave a more structured, unified interface supported by all the code-coverage= tools, but to my knowledge no such thing exists.


From d595ba5149a314c56623e1735af7678f5f62d525 Mon Sep = 17 00:00:00 2001
From: Gabriel Scherer <gabriel.scherer@gmail.com>
Date: Sun, 6 Mar 2016 17:43:14 -0500
Subject: [PAT= CH] output (*BISECT-IGNORE*) in front of each
 non-progr= ammer-written line

EXPERIMENTAL PATCH: this sh= ould of course be turned into an explicit option
---
 src/printer.ml | = 1 +
 1 file changed, 1 insertion(+)

diff --git a/src/printer= .ml b/src/printer.ml
index ea978bc..714bb08 100644
--- a/src/printer.ml
+++ b/src/printer.ml
@@ -46,6 = +46,7 @@ let rawnl f =3D
 
 let nl f = =3D
   rawnl f;
+  output_string= f "(*BISECT-IGNORE*)";
   output_substring f white= space 0 !indentation
 
 let indent of= s producer f x =3D
--
2.5.0



<= div class=3D"gmail_quote">On Sun, Mar 6, 2016 at 2:53 PM, Vincent Jacques <= span dir=3D"ltr" class=3D""><vincent@vincent-jacques.net>= wrote:
Hello,

Does somebo= dy have experience measuring test coverage of generated lexers/parsers?

I'm using ocamllex/ocamlyacc [1] (but I can switc= h to Menhir [2]) to generate a lexer/parser. In my tests, I simply check th= at some input strings give the ASTs I expect.

I usually use Bisect = [3] to make sure that my tests cover the code I intended to cover, but in t= hat configuration, Bisect is lost between the .mll/.mly files and the gener= ated .ml files and produces useless reports.

<= /div>
How would you measure test coverage in that case?

Thanks= ,

[1] http://caml.inria.fr/pub/docs/manual-ocaml/lexyacc.html
[2] http://gallium.inria.fr/~fpottier/menhir/
[3] http:= //bisect.x9c.fr/
--
Vincent Jacques
http://vincent-jacques.net

"S'il n'y a pa= s de solution, c'est qu'il n'y a pas de probl=C3=A8me"
 =           Devise Shadock

<0001-output-BISEC= T-IGNORE-in-front-of-each-non-programmer.patch>

= --Apple-Mail=_1D761AAA-612C-45A6-9FEB-3857F12B0C4C--