From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: caml-list@sympa.inria.fr Delivered-To: caml-list@sympa.inria.fr Received: from mail2-relais-roc.national.inria.fr (mail2-relais-roc.national.inria.fr [192.134.164.83]) by sympa.inria.fr (Postfix) with ESMTPS id DEDF17FDF3 for ; Mon, 7 Mar 2016 09:20:26 +0100 (CET) IronPort-PHdr: 9a23:ty52RxExSMaVuGQPqoRY6p1GYnF86YWxBRYc798ds5kLTJ75oc6wAkXT6L1XgUPTWs2DsrQf27WQ4/irADRfqb+681k8M7V0HycfjssXmwFySOWkMmbcaMDQUiohAc5ZX0Vk9XzoeWJcGcL5ekGA6ibqtW1aJBzzOEJPK/jvHcaK1oLsh7/0q8WYOl0XzBOGIppMbzyO5T3LsccXhYYwYo0Q8TDu5kVyRuJN2GlzLkiSlRuvru25/Zpk7jgC86l5r50IAu3GePEzRLlcRCk9PnourJngvBzHCA+O/Wc0U2MMkxMODRKTvz/gWZKkjCr8ved7xGG4NMbvUL0vEWC87qFgRRn0oDkGMTU09n2SiMV7lb9Wu1SnqgApkN2cW52cKPcrJvCVRtgdX2cUBss= Authentication-Results: mail2-smtp-roc.national.inria.fr; spf=None smtp.pra=jacquev6@gmail.com; spf=Pass smtp.mailfrom=jacquev6@gmail.com; spf=None smtp.helo=postmaster@mail-qg0-f48.google.com Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of jacquev6@gmail.com) identity=pra; client-ip=209.85.192.48; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="jacquev6@gmail.com"; x-sender="jacquev6@gmail.com"; x-conformance=sidf_compatible Received-SPF: Pass (mail2-smtp-roc.national.inria.fr: domain of jacquev6@gmail.com designates 209.85.192.48 as permitted sender) identity=mailfrom; client-ip=209.85.192.48; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="jacquev6@gmail.com"; x-sender="jacquev6@gmail.com"; x-conformance=sidf_compatible; x-record-type="v=spf1" Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of postmaster@mail-qg0-f48.google.com) identity=helo; client-ip=209.85.192.48; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="jacquev6@gmail.com"; x-sender="postmaster@mail-qg0-f48.google.com"; x-conformance=sidf_compatible X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A0CPAABzOd1WjjDAVdFdFoN2bQaEOaVSKpAFAQ2CCoVugR0HOBQBAQEBAQEBARABAQEBBwsLCR8xQRIBgVmCFAEBAQMBEhEdASsKBAMBCwEFAwILBgMBAgEqAgIhARIBBQEKCggGExsHh2sDCggOkxOPQYExPjGLNoRBhSonDYRrAQUKBIURhDh9gj2CHYJggToFhhwMkQKFY4YVBoFvY4FLjEyHDIYLER6BDw8PAQF3gUGBbzsuAYECiDoBAQE X-IPAS-Result: A0CPAABzOd1WjjDAVdFdFoN2bQaEOaVSKpAFAQ2CCoVugR0HOBQBAQEBAQEBARABAQEBBwsLCR8xQRIBgVmCFAEBAQMBEhEdASsKBAMBCwEFAwILBgMBAgEqAgIhARIBBQEKCggGExsHh2sDCggOkxOPQYExPjGLNoRBhSonDYRrAQUKBIURhDh9gj2CHYJggToFhhwMkQKFY4YVBoFvY4FLjEyHDIYLER6BDw8PAQF3gUGBbzsuAYECiDoBAQE X-IronPort-AV: E=Sophos;i="5.22,550,1449529200"; d="scan'208,217";a="206301550" Received: from mail-qg0-f48.google.com ([209.85.192.48]) by mail2-smtp-roc.national.inria.fr with ESMTP/TLS/AES128-GCM-SHA256; 07 Mar 2016 09:20:25 +0100 Received: by mail-qg0-f48.google.com with SMTP id w104so90575466qge.1 for ; Mon, 07 Mar 2016 00:20:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to; bh=wCUHVdGsP/SYG2pqQ+CzITCt2KeUbDHUR2/5wvfxOjU=; b=rbgXUFXOBlDyc6Zybm/G3R1vF6LJ5m2ViiVa9RqtVzWNuMxW371WCpZxVvoHc32E3Q llI5AJBhpKAF0jCBtNGmbza1SVH4nEaXd8Nq4PohQ04IIq567UcFzBin2UUD+2Who8zi n6Wa9+U7Re+ZsVUourt3PjxfjcJjlBWGZK3RQvrImTmVY5YReJlMHAIYzvxF0s4bM9By b0+qK5AcJ5XONM2O6Zb55hNKUZq2tDzq/o55W20XiXcKVomuHaUbfMggTFFVlFnmOaIc /hz2GRh9t3X+Lrxubdgzhk88RQgJH3cmh+aL3SoFH+AE/vDbg9y5SlmyFtKHwL7CkJ9H 4N/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to; bh=wCUHVdGsP/SYG2pqQ+CzITCt2KeUbDHUR2/5wvfxOjU=; b=cuRGkD5BkuClFUshpTc1erPD1s/8rFK9sdX7Re8aLHYxRbxqTuRptfPomNGXr7ivGM Bs6vIaWZrjIYhn0YjMJvQnAO5Klh6ZZuQjCmdBKsukRLMp72atcGBFcqKStfJpxMkSuT t4PfHWciCXPjzSVF8pH8n57kbaG/3VVjWoeYCOSoyCntDX7/XuTQhJB4GzAu3NiiWDY8 g7F8ryafZsRMaGtOHA7M7dFC9GgTNufdK0diW+SG9ZEGsSNS7/1cS8LiU2YbGtEPuwrc OIn54KneqwCWSY7G6VmlfMCwTxvhtYDgVl9GAlnIW+o4f/YOf4CYhah59gP3VQ48uisE K/IQ== X-Gm-Message-State: AD7BkJKsBXpSQfVTaZ1z7BA4+T2Xq6ZJQXbH4QF6vGmC63CRW1bPE88sPxxnqzmJFPIyEgLLm0675dYiUld83g== MIME-Version: 1.0 X-Received: by 10.140.151.194 with SMTP id 185mr15207088qhx.92.1457338824212; Mon, 07 Mar 2016 00:20:24 -0800 (PST) Sender: jacquev6@gmail.com Received: by 10.55.36.133 with HTTP; Mon, 7 Mar 2016 00:20:24 -0800 (PST) In-Reply-To: <6080D67A-8238-4F27-AEA6-C2B00704F2AA@yahoo.com> References: <6080D67A-8238-4F27-AEA6-C2B00704F2AA@yahoo.com> Date: Mon, 7 Mar 2016 09:20:24 +0100 X-Google-Sender-Auth: f_i5GMGy4s64hVdxMWH_vyDEL3k Message-ID: From: Vincent Jacques To: caml users Content-Type: multipart/alternative; boundary=001a113547fcef4a6a052d7122f5 X-Validation-by: vincent@vincent-jacques.net Subject: Re: [Caml-list] Test coverage of generated lexers/parsers --001a113547fcef4a6a052d7122f5 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Thanks for your interest in this question! I suspected there was no obvious solution. I'll switch to Menhir (with Gabriel's experimental patch) and tell you if it improved my situation. 2016-03-06 23:59 GMT+01:00 Anton Bachin : > If there is a good, general, alternative approach for this, we can support > it in Bisect_ppx. Unfortunately, I don=E2=80=99t know enough about Menhir= to be > able to propose anything specific at this point. > > Best, > Anton > > On Mar 6, 2016, at 16:53, Gabriel Scherer > wrote: > > This is an interesting question and, as far as I know, there is no good > solution using existing versions of the interacting tools. > > Below very simple patch that will add (*BISECT-IGNORE*) in front of every > line of code generated by Menhir, except those written by the programmer > (the "strecthes" in Menhir-speak). It applies cleanly on top of the latest > released Menhir archive, > http://gallium.inria.fr/~fpottier/menhir/menhir-20160303.tar.gz > > The patch as-is is obviously a hack: it would need to be a configuration > option when running menhir, and hard-coding Bisect (or bisect_ppx)'s synt= ax > into Menhir is not elegant. One could try to have a configuration option = to > let users write a fixed string (or comment) at the beginning of each > generated code line, but I'm not sure whether Fran=C3=A7ois Pottier (in c= c:) > would consider this is elegant enough. Fran=C3=A7ois, would you comment on > whether this is a direction that seems acceptable to you? > > (Bisect support ignoring entire regions at once by using > (*BISECT-IGNORE-BEGIN*) and (*BISECT-IGNORE-END*); we could try to > implement that instead of a per-line change, but I suspect that it would = be > slightly harder to implement (you have to hook the beginning of input, end > of input, and around each user-code insertion) for no real gain.) > > Toggling code-coverage semantics by inserting comments is not a very nice > interface (although rather logical when you think of the level of > generality required), so it's a bit frustrating that parser generators > would have to play at this level. It would be better to have a more > structured, unified interface supported by all the code-coverage tools, b= ut > to my knowledge no such thing exists. > > > From d595ba5149a314c56623e1735af7678f5f62d525 Mon Sep 17 00:00:00 2001 > From: Gabriel Scherer > Date: Sun, 6 Mar 2016 17:43:14 -0500 > Subject: [PATCH] output (*BISECT-IGNORE*) in front of each > non-programmer-written line > > EXPERIMENTAL PATCH: this should of course be turned into an explicit opti= on > --- > src/printer.ml | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/src/printer.ml b/src/printer.ml > index ea978bc..714bb08 100644 > --- a/src/printer.ml > +++ b/src/printer.ml > @@ -46,6 +46,7 @@ let rawnl f =3D > > let nl f =3D > rawnl f; > + output_string f "(*BISECT-IGNORE*)"; > output_substring f whitespace 0 !indentation > > let indent ofs producer f x =3D > -- > 2.5.0 > > > > On Sun, Mar 6, 2016 at 2:53 PM, Vincent Jacques < > vincent@vincent-jacques.net> wrote: > >> Hello, >> >> Does somebody have experience measuring test coverage of generated >> lexers/parsers? >> >> I'm using ocamllex/ocamlyacc [1] (but I can switch to Menhir [2]) to >> generate a lexer/parser. In my tests, I simply check that some input >> strings give the ASTs I expect. >> >> I usually use Bisect [3] to make sure that my tests cover the code I >> intended to cover, but in that configuration, Bisect is lost between the >> .mll/.mly files and the generated .ml files and produces useless reports. >> >> How would you measure test coverage in that case? >> >> Thanks, >> >> [1] http://caml.inria.fr/pub/docs/manual-ocaml/lexyacc.html >> [2] http://gallium.inria.fr/~fpottier/menhir/ >> [3] http://bisect.x9c.fr/ >> -- >> Vincent Jacques >> http://vincent-jacques.net >> >> "S'il n'y a pas de solution, c'est qu'il n'y a pas de probl=C3=A8me" >> Devise Shadock >> > > <0001-output-BISECT-IGNORE-in-front-of-each-non-programmer.patch> > > > --=20 Vincent Jacques http://vincent-jacques.net "S'il n'y a pas de solution, c'est qu'il n'y a pas de probl=C3=A8me" Devise Shadock --001a113547fcef4a6a052d7122f5 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Thanks for your interest in this question! I suspecte= d there was no obvious solution.

I'll switch to Menhir (wi= th Gabriel's experimental patch) and tell you if it improved my situati= on.

2016-03-06= 23:59 GMT+01:00 Anton Bachin <antonbachin@yahoo.com>:
I= f there is a good, general, alternative approach for this, we can support i= t in Bisect_ppx. Unfortunately, I don=E2=80=99t know enough about Menhir to= be able to propose anything specific at this point.

Best,
Anton

=
On Mar 6, 2016, at 16:53, Gabriel Scherer <gabriel.scherer@gmail.com> wr= ote:

This is an interesting question and, as far as I know, there is no good = solution using existing versions of the interacting tools.

Bel= ow very simple patch that will add (*BISECT-IGNORE*) in front of every lin= e of code generated by Menhir, except those written by the programmer (the = "strecthes" in Menhir-speak). It applies cleanly on top of the la= test released Menhir archive,
=C2=A0 http://gallium.i= nria.fr/~fpottier/menhir/menhir-20160303.tar.gz

The patch = as-is is obviously a hack: it would need to be a configuration option when = running menhir, and hard-coding Bisect (or bisect_ppx)'s syntax into Me= nhir is not elegant. One could try to have a configuration option to let us= ers write a fixed string (or comment) at the beginning of each generated co= de line, but I'm not sure whether Fran=C3=A7ois Pottier (in cc:) would = consider this is elegant enough. Fran=C3=A7ois, would you comment on whethe= r this is a direction that seems acceptable to you?

(Bisect su= pport ignoring entire regions at once by using (*BISECT-IGNORE-BEGIN*) and = (*BISECT-IGNORE-END*); we could try to implement that instead of a per-line= change, but I suspect that it would be slightly harder to implement (you h= ave to hook the beginning of input, end of input, and around each user-code= insertion) for no real gain.)

Toggling code-coverage semantic= s by inserting comments is not a very nice interface (although rather logic= al when you think of the level of generality required), so it's a bit f= rustrating that parser generators would have to play at this level. It woul= d be better to have a more structured, unified interface supported by all t= he code-coverage tools, but to my knowledge no such thing exists.

From d595ba5149a314c56623e1735af7678f5f62d525 Mon Sep 17 00:00:00 2001From: Gabriel Scherer <gabriel.scherer@gmail.com>
Date: Sun, 6 Mar 2016 1= 7:43:14 -0500
Subject: [PATCH] output (*BISECT-IGNORE*) in front of each=
=C2=A0non-programmer-written line

EXPERIMENTAL PATCH: this shoul= d of course be turned into an explicit option
---
=C2=A0src/printer.ml | 1 +
=C2=A01 f= ile changed, 1 insertion(+)

diff --git a/src/printer.ml b/src/printer.ml
index ea978bc..714bb08 100644
---= a/src/printer.ml
+= ++ b/src/printer.ml@@ -46,6 +46,7 @@ let rawnl f =3D
=C2=A0
=C2=A0let nl f =3D
=C2= =A0=C2=A0 rawnl f;
+=C2=A0 output_string f "(*BISECT-IGNORE*)"= ;
=C2=A0=C2=A0 output_substring f whitespace 0 !indentation
=C2=A0=C2=A0let indent ofs producer f x =3D
--
2.5.0



On Sun, Mar 6, 2016 = at 2:53 PM, Vincent Jacques <vincent@vincent-jacques.net>= wrote:
Hell= o,

Does somebody have experience measuring test coverage of ge= nerated lexers/parsers?

I'm using ocamllex/ocamlyacc [1] (but I = can switch to Menhir [2]) to generate a lexer/parser. In my tests, I simply= check that some input strings give the ASTs I expect.

I usually use Bisect [3] to make sure that my tests cover the code = I intended to cover, but in that configuration, Bisect is lost between the = .mll/.mly files and the generated .ml files and produces useless reports.
How would you measure test coverage in that case?

Thanks,

[1] http://caml.i= nria.fr/pub/docs/manual-ocaml/lexyacc.html
[2] http://gallium.inria.fr/= ~fpottier/menhir/
[3] http://bisect.x9c.fr/
--
Vincent Jacques
http://vincent-jacques.net

"S'il n'y a pas de s= olution, c'est qu'il n'y a pas de probl=C3=A8me"
=C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Devise Shadock

<0001-output-BISECT-IGNORE-in-front-of-each-non-progra= mmer.patch>

=


--
Vincent Jacques
http://vincent-jacques.net
"S'il n'y a pas de solution, c'est qu'il n'y a p= as de probl=C3=A8me"
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Devi= se Shadock
--001a113547fcef4a6a052d7122f5--