From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: caml-list@sympa.inria.fr Delivered-To: caml-list@sympa.inria.fr Received: from mail2-relais-roc.national.inria.fr (mail2-relais-roc.national.inria.fr [192.134.164.83]) by sympa.inria.fr (Postfix) with ESMTPS id 2AE417F164 for ; Sun, 6 Mar 2016 23:54:15 +0100 (CET) IronPort-PHdr: 9a23:4puD8Rxb2qLKr9/XCy+O+j09IxM/srCxBDY+r6Qd0eMSIJqq85mqBkHD//Il1AaPBtWEraIcwLKM+4nbGkU+or+5+EgYd5JNUxJXwe43pCcHRPC/NEvgMfTxZDY7FskRHHVs/nW8LFQHUJ2mPw6anHS+4HYoFwnlMkItf6KuStGU35z8j7r60qaQSjsLrQL1Wal1IhSyoFeZnegtqqwmFJwMzADUqGBDYeVcyDAgD1uSmxHh+pX4p8Y7oGx48sgs/M9YUKj8Y79wDfkBVGxnYCgJ45jVtB/IQA2TrlkVWXwLnwEAVxbE6hr3WIvZrCr8ved7xGyYNMbvUL0vHzKv8/E4ZgXvjXIoPjQj8WzTwvd7jK9BrQjp8xN2yZTVbYXTL/F+c7nQZ/sVQGNAWoBaUCkXUdD0VJcGE+dUZbUQlIL6vVZb6ELmXQQ= Authentication-Results: mail2-smtp-roc.national.inria.fr; spf=None smtp.pra=gabriel.scherer@gmail.com; spf=Pass smtp.mailfrom=gabriel.scherer@gmail.com; spf=None smtp.helo=postmaster@mail-ig0-f173.google.com Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of gabriel.scherer@gmail.com) identity=pra; client-ip=209.85.213.173; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="gabriel.scherer@gmail.com"; x-sender="gabriel.scherer@gmail.com"; x-conformance=sidf_compatible Received-SPF: Pass (mail2-smtp-roc.national.inria.fr: domain of gabriel.scherer@gmail.com designates 209.85.213.173 as permitted sender) identity=mailfrom; client-ip=209.85.213.173; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="gabriel.scherer@gmail.com"; x-sender="gabriel.scherer@gmail.com"; x-conformance=sidf_compatible; x-record-type="v=spf1" Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of postmaster@mail-ig0-f173.google.com) identity=helo; client-ip=209.85.213.173; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="gabriel.scherer@gmail.com"; x-sender="postmaster@mail-ig0-f173.google.com"; x-conformance=sidf_compatible X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A0BUAQBLtNxWiK3VVdFdFoN2bQaqM5ATgWkhhW4CgRYHORMBAQEBAQEBARABAQEICwsJEg0xgi2CFAEBAQMBEhEdARsQCgMBAwELBgMCCwYDAQIrAgICHwEBEQEFARQIBhMbB4dqAQMKCA6TLY9BgTE+MYs2gWqCV4RvChknDVGDdgEBAQEBAQEBAQEBAQEBAQEBAQEBDgIBBQoEhgmEPYI9ghMKgmCBOgWOJIkGgUqBSIJRhhWBdYIujEyHDAiGAxEegQ8PEgGCNh6Bbh4uAYk8AQEB X-IPAS-Result: A0BUAQBLtNxWiK3VVdFdFoN2bQaqM5ATgWkhhW4CgRYHORMBAQEBAQEBARABAQEICwsJEg0xgi2CFAEBAQMBEhEdARsQCgMBAwELBgMCCwYDAQIrAgICHwEBEQEFARQIBhMbB4dqAQMKCA6TLY9BgTE+MYs2gWqCV4RvChknDVGDdgEBAQEBAQEBAQEBAQEBAQEBAQEBDgIBBQoEhgmEPYI9ghMKgmCBOgWOJIkGgUqBSIJRhhWBdYIujEyHDAiGAxEegQ8PEgGCNh6Bbh4uAYk8AQEB X-IronPort-AV: E=Sophos;i="5.22,548,1449529200"; d="scan'208,217,223";a="206243838" Received: from mail-ig0-f173.google.com ([209.85.213.173]) by mail2-smtp-roc.national.inria.fr with ESMTP/TLS/AES128-GCM-SHA256; 06 Mar 2016 23:54:13 +0100 Received: by mail-ig0-f173.google.com with SMTP id hb3so30213951igb.0; Sun, 06 Mar 2016 14:54:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=LSekCJbBimy229W8HFPplR031Aigr/E1/i1joO2Vfsw=; b=S7l2LjOyWpoF9dyeIyOBZXLmzA6QFscx8a79Qay4idSyWo3HmFiA5YmYJoetlWCRhs BPp5B1dNkxKTvamgETI1Oy8PLcA8reG1cLiZAVYZTmuCTBGqSQVi2pWs6O7VWHz1Ik7C 2CXma2E0qvWVx8eiymbyWSrcP693OLLPpOL/My8lROiib4IQsYfNnVP95MRt0R5M6HL0 Oln0vpdkzEgLQakai+uzQm8IKzBAqyZeDMd4/ffWZ/qrYPh5lESz9/QaXuo5NGF5o1dH xS+EoQNPlJAsmYpATxxOqLFr9KZdl0YyLANMDSy0sgRzWjxbGqdnXkoEjtN079Qe7JBq PykQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=LSekCJbBimy229W8HFPplR031Aigr/E1/i1joO2Vfsw=; b=iUAlLEpgUCyHpATuqS2rOXp0nDGNwCl3lkkbXIqa+uN7jp8/UNZiZ66W8IsXLJUmu5 OGtIee2z6L/+jlfXgu3o6ChHE2gfd8aJRvcc1HsM8dRHNxiogk1D4lOyRgMNhaCTvVZU BBUZnLzsXjmNDpsCrq8cG4PL6pogynkyVkjErx1zE6Exa7unTQBOk5u3LKkfyYuzjYi2 OM9MylJs+OSHvaLXl13a8+BMdVWgbicuTdPuSHV8rtuM/BaO80EAicDJgieZsGQJhOcS JnvoitJ/JWaAeuLU8cQjzAB4aD5cWaQ8CVSIOFRTJieVzHx2ZTXh8b9kk2YlEHTGDC9K e1Cg== X-Gm-Message-State: AD7BkJIyw+ws8y6P53udiBdv+t1ODgOVGstXv6lrU4DfITUylLz/IcLmHsXXqC5fbefo8lIAgf2nO154QmzD3A== X-Received: by 10.50.221.8 with SMTP id qa8mr8765861igc.42.1457304852884; Sun, 06 Mar 2016 14:54:12 -0800 (PST) MIME-Version: 1.0 Received: by 10.79.17.4 with HTTP; Sun, 6 Mar 2016 14:53:33 -0800 (PST) In-Reply-To: References: From: Gabriel Scherer Date: Sun, 6 Mar 2016 17:53:33 -0500 Message-ID: To: Vincent Jacques Cc: caml users , Francois Pottier Content-Type: multipart/mixed; boundary=001a11345022160669052d693aac Subject: Re: [Caml-list] Test coverage of generated lexers/parsers --001a11345022160669052d693aac Content-Type: multipart/alternative; boundary=001a11345022160664052d693aaa --001a11345022160664052d693aaa Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable This is an interesting question and, as far as I know, there is no good solution using existing versions of the interacting tools. Below very simple patch that will add (*BISECT-IGNORE*) in front of every line of code generated by Menhir, except those written by the programmer (the "strecthes" in Menhir-speak). It applies cleanly on top of the latest released Menhir archive, http://gallium.inria.fr/~fpottier/menhir/menhir-20160303.tar.gz The patch as-is is obviously a hack: it would need to be a configuration option when running menhir, and hard-coding Bisect (or bisect_ppx)'s syntax into Menhir is not elegant. One could try to have a configuration option to let users write a fixed string (or comment) at the beginning of each generated code line, but I'm not sure whether Fran=C3=A7ois Pottier (in cc:) would consider this is elegant enough. Fran=C3=A7ois, would you comment on whether this is a direction that seems acceptable to you? (Bisect support ignoring entire regions at once by using (*BISECT-IGNORE-BEGIN*) and (*BISECT-IGNORE-END*); we could try to implement that instead of a per-line change, but I suspect that it would be slightly harder to implement (you have to hook the beginning of input, end of input, and around each user-code insertion) for no real gain.) Toggling code-coverage semantics by inserting comments is not a very nice interface (although rather logical when you think of the level of generality required), so it's a bit frustrating that parser generators would have to play at this level. It would be better to have a more structured, unified interface supported by all the code-coverage tools, but to my knowledge no such thing exists. =46rom d595ba5149a314c56623e1735af7678f5f62d525 Mon Sep 17 00:00:00 2001 From: Gabriel Scherer Date: Sun, 6 Mar 2016 17:43:14 -0500 Subject: [PATCH] output (*BISECT-IGNORE*) in front of each non-programmer-written line EXPERIMENTAL PATCH: this should of course be turned into an explicit option --- src/printer.ml | 1 + 1 file changed, 1 insertion(+) diff --git a/src/printer.ml b/src/printer.ml index ea978bc..714bb08 100644 --- a/src/printer.ml +++ b/src/printer.ml @@ -46,6 +46,7 @@ let rawnl f =3D let nl f =3D rawnl f; + output_string f "(*BISECT-IGNORE*)"; output_substring f whitespace 0 !indentation let indent ofs producer f x =3D --=20 2.5.0 On Sun, Mar 6, 2016 at 2:53 PM, Vincent Jacques wrote: > Hello, > > Does somebody have experience measuring test coverage of generated > lexers/parsers? > > I'm using ocamllex/ocamlyacc [1] (but I can switch to Menhir [2]) to > generate a lexer/parser. In my tests, I simply check that some input > strings give the ASTs I expect. > > I usually use Bisect [3] to make sure that my tests cover the code I > intended to cover, but in that configuration, Bisect is lost between the > .mll/.mly files and the generated .ml files and produces useless reports. > > How would you measure test coverage in that case? > > Thanks, > > [1] http://caml.inria.fr/pub/docs/manual-ocaml/lexyacc.html > [2] http://gallium.inria.fr/~fpottier/menhir/ > [3] http://bisect.x9c.fr/ > -- > Vincent Jacques > http://vincent-jacques.net > > "S'il n'y a pas de solution, c'est qu'il n'y a pas de probl=C3=A8me" > Devise Shadock > --001a11345022160664052d693aaa Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
This is an interesting question and, a= s far as I know, there is no good solution using existing versions of the i= nteracting tools.

Below very simple patch that will add (*BIS= ECT-IGNORE*) in front of every line of code generated by Menhir, except tho= se written by the programmer (the "strecthes" in Menhir-speak). I= t applies cleanly on top of the latest released Menhir archive,
=C2=A0 <= a href=3D"http://gallium.inria.fr/~fpottier/menhir/menhir-20160303.tar.gz">= http://gallium.inria.fr/~fpottier/menhir/menhir-20160303.tar.gz

=
The patch as-is is obviously a hack: it would need to be a configurat= ion option when running menhir, and hard-coding Bisect (or bisect_ppx)'= s syntax into Menhir is not elegant. One could try to have a configuration = option to let users write a fixed string (or comment) at the beginning of e= ach generated code line, but I'm not sure whether Fran=C3=A7ois Pottier= (in cc:) would consider this is elegant enough. Fran=C3=A7ois, would you c= omment on whether this is a direction that seems acceptable to you?

=
(Bisect support ignoring entire regions at once by using (*BISECT-IGN= ORE-BEGIN*) and (*BISECT-IGNORE-END*); we could try to implement that inste= ad of a per-line change, but I suspect that it would be slightly harder to = implement (you have to hook the beginning of input, end of input, and aroun= d each user-code insertion) for no real gain.)

Toggling code-c= overage semantics by inserting comments is not a very nice interface (altho= ugh rather logical when you think of the level of generality required), so = it's a bit frustrating that parser generators would have to play at thi= s level. It would be better to have a more structured, unified interface su= pported by all the code-coverage tools, but to my knowledge no such thing e= xists.


From d595ba5149a314c56623e1735af7678f5f62d525 Mon Sep 17 = 00:00:00 2001
From: Gabriel Scherer <gabriel.scherer@gmail.com>
Date: Sun, 6 Mar 2016 17:= 43:14 -0500
Subject: [PATCH] output (*BISECT-IGNORE*) in front of each=C2=A0non-programmer-written line

EXPERIMENTAL PATCH: this should = of course be turned into an explicit option
---
=C2=A0src/printer.ml | 1 +
=C2=A01 file changed, 1 insertio= n(+)

diff --git a/src/printer.ml b= /src/printer.ml
index ea978bc..714bb08= 100644
--- a/src/printer.ml
+++ b/= src/printer.ml
@@ -46,6 +46,7 @@ let r= awnl f =3D
=C2=A0
=C2=A0let nl f =3D
=C2=A0=C2=A0 rawnl f;
+=C2= =A0 output_string f "(*BISECT-IGNORE*)";
=C2=A0=C2=A0 output_s= ubstring f whitespace 0 !indentation
=C2=A0
=C2=A0let indent ofs prod= ucer f x =3D
--
2.5.0


On Sun, Mar 6, 2016 at 2:53 PM, Vincent Jacque= s <vincent@vincent-jacques.net> wrote:
Hello,

Does someb= ody have experience measuring test coverage of generated lexers/parsers?
I'm using ocamllex/ocamlyacc [1] (but I can switch to Menhir [2]) = to generate a lexer/parser. In my tests, I simply check that some input str= ings give the ASTs I expect.

I usually use Bise= ct [3] to make sure that my tests cover the code I intended to cover, but i= n that configuration, Bisect is lost between the .mll/.mly files and the ge= nerated .ml files and produces useless reports.

How would= you measure test coverage in that case?

Thank= s,

[1] http://caml.inria.fr/pub/docs/manual-oc= aml/lexyacc.html
[2] http://gallium.inria.fr/~fpottier/menhir/
[= 3] http://bisect.x9c.fr= /
--
Vincent= Jacques
http:/= /vincent-jacques.net

"S'il n'y a pas de solution, c= 'est qu'il n'y a pas de probl=C3=A8me"
=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 Devise Shadock

--001a11345022160664052d693aaa-- --001a11345022160669052d693aac Content-Type: text/x-patch; charset=US-ASCII; name="0001-output-BISECT-IGNORE-in-front-of-each-non-programmer.patch" Content-Disposition: attachment; filename="0001-output-BISECT-IGNORE-in-front-of-each-non-programmer.patch" Content-Transfer-Encoding: base64 X-Attachment-Id: f_ilh5nbms0 RnJvbSBkNTk1YmE1MTQ5YTMxNGM1NjYyM2UxNzM1YWY3Njc4ZjVmNjJkNTI1 IE1vbiBTZXAgMTcgMDA6MDA6MDAgMjAwMQpGcm9tOiBHYWJyaWVsIFNjaGVy ZXIgPGdhYnJpZWwuc2NoZXJlckBnbWFpbC5jb20+CkRhdGU6IFN1biwgNiBN YXIgMjAxNiAxNzo0MzoxNCAtMDUwMApTdWJqZWN0OiBbUEFUQ0hdIG91dHB1 dCAoKkJJU0VDVC1JR05PUkUqKSBpbiBmcm9udCBvZiBlYWNoCiBub24tcHJv Z3JhbW1lci13cml0dGVuIGxpbmUKCkVYUEVSSU1FTlRBTCBQQVRDSDogdGhp cyBzaG91bGQgb2YgY291cnNlIGJlIHR1cm5lZCBpbnRvIGFuIGV4cGxpY2l0 IG9wdGlvbgotLS0KIHNyYy9wcmludGVyLm1sIHwgMSArCiAxIGZpbGUgY2hh bmdlZCwgMSBpbnNlcnRpb24oKykKCmRpZmYgLS1naXQgYS9zcmMvcHJpbnRl ci5tbCBiL3NyYy9wcmludGVyLm1sCmluZGV4IGVhOTc4YmMuLjcxNGJiMDgg MTAwNjQ0Ci0tLSBhL3NyYy9wcmludGVyLm1sCisrKyBiL3NyYy9wcmludGVy Lm1sCkBAIC00Niw2ICs0Niw3IEBAIGxldCByYXdubCBmID0KIAogbGV0IG5s IGYgPQogICByYXdubCBmOworICBvdXRwdXRfc3RyaW5nIGYgIigqQklTRUNU LUlHTk9SRSopIjsKICAgb3V0cHV0X3N1YnN0cmluZyBmIHdoaXRlc3BhY2Ug MCAhaW5kZW50YXRpb24KIAogbGV0IGluZGVudCBvZnMgcHJvZHVjZXIgZiB4 ID0KLS0gCjIuNS4wCgo= --001a11345022160669052d693aac--