[Caml-list] Test coverage of generated lexers/parsers

[Caml-list] Test coverage of generated lexers/parsers

[Vincent Jacques]

[-- Attachment #1: Type: text/plain, Size: 838 bytes --]

Hello,

Does somebody have experience measuring test coverage of generated
lexers/parsers?

I'm using ocamllex/ocamlyacc [1] (but I can switch to Menhir [2]) to
generate a lexer/parser. In my tests, I simply check that some input
strings give the ASTs I expect.

I usually use Bisect [3] to make sure that my tests cover the code I
intended to cover, but in that configuration, Bisect is lost between the
.mll/.mly files and the generated .ml files and produces useless reports.

How would you measure test coverage in that case?

Thanks,

[1] http://caml.inria.fr/pub/docs/manual-ocaml/lexyacc.html
[2] http://gallium.inria.fr/~fpottier/menhir/
[3] http://bisect.x9c.fr/
-- 
Vincent Jacques
http://vincent-jacques.net

"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème"
            Devise Shadock

[-- Attachment #2: Type: text/html, Size: 1278 bytes --]

Re: [Caml-list] Test coverage of generated lexers/parsers

[Gabriel Scherer]

[-- Attachment #1.1: Type: text/plain, Size: 3403 bytes --]

This is an interesting question and, as far as I know, there is no good
solution using existing versions of the interacting tools.

Below very simple patch that will add (*BISECT-IGNORE*) in front of every
line of code generated by Menhir, except those written by the programmer
(the "strecthes" in Menhir-speak). It applies cleanly on top of the latest
released Menhir archive,
  http://gallium.inria.fr/~fpottier/menhir/menhir-20160303.tar.gz

The patch as-is is obviously a hack: it would need to be a configuration
option when running menhir, and hard-coding Bisect (or bisect_ppx)'s syntax
into Menhir is not elegant. One could try to have a configuration option to
let users write a fixed string (or comment) at the beginning of each
generated code line, but I'm not sure whether François Pottier (in cc:)
would consider this is elegant enough. François, would you comment on
whether this is a direction that seems acceptable to you?

(Bisect support ignoring entire regions at once by using
(*BISECT-IGNORE-BEGIN*) and (*BISECT-IGNORE-END*); we could try to
implement that instead of a per-line change, but I suspect that it would be
slightly harder to implement (you have to hook the beginning of input, end
of input, and around each user-code insertion) for no real gain.)

Toggling code-coverage semantics by inserting comments is not a very nice
interface (although rather logical when you think of the level of
generality required), so it's a bit frustrating that parser generators
would have to play at this level. It would be better to have a more
structured, unified interface supported by all the code-coverage tools, but
to my knowledge no such thing exists.


From d595ba5149a314c56623e1735af7678f5f62d525 Mon Sep 17 00:00:00 2001
From: Gabriel Scherer <gabriel.scherer@gmail.com>
Date: Sun, 6 Mar 2016 17:43:14 -0500
Subject: [PATCH] output (*BISECT-IGNORE*) in front of each
 non-programmer-written line

EXPERIMENTAL PATCH: this should of course be turned into an explicit option
---
 src/printer.ml | 1 +
 1 file changed, 1 insertion(+)

diff --git a/src/printer.ml b/src/printer.ml
index ea978bc..714bb08 100644
--- a/src/printer.ml
+++ b/src/printer.ml
@@ -46,6 +46,7 @@ let rawnl f =

 let nl f =
   rawnl f;
+  output_string f "(*BISECT-IGNORE*)";
   output_substring f whitespace 0 !indentation

 let indent ofs producer f x =
-- 
2.5.0



On Sun, Mar 6, 2016 at 2:53 PM, Vincent Jacques <vincent@vincent-jacques.net
> wrote:

> Hello,
>
> Does somebody have experience measuring test coverage of generated
> lexers/parsers?
>
> I'm using ocamllex/ocamlyacc [1] (but I can switch to Menhir [2]) to
> generate a lexer/parser. In my tests, I simply check that some input
> strings give the ASTs I expect.
>
> I usually use Bisect [3] to make sure that my tests cover the code I
> intended to cover, but in that configuration, Bisect is lost between the
> .mll/.mly files and the generated .ml files and produces useless reports.
>
> How would you measure test coverage in that case?
>
> Thanks,
>
> [1] http://caml.inria.fr/pub/docs/manual-ocaml/lexyacc.html
> [2] http://gallium.inria.fr/~fpottier/menhir/
> [3] http://bisect.x9c.fr/
> --
> Vincent Jacques
> http://vincent-jacques.net
>
> "S'il n'y a pas de solution, c'est qu'il n'y a pas de problème"
>             Devise Shadock
>

[-- Attachment #1.2: Type: text/html, Size: 4617 bytes --]

[-- Attachment #2: 0001-output-BISECT-IGNORE-in-front-of-each-non-programmer.patch --]
[-- Type: text/x-patch, Size: 689 bytes --]

From d595ba5149a314c56623e1735af7678f5f62d525 Mon Sep 17 00:00:00 2001
From: Gabriel Scherer <gabriel.scherer@gmail.com>
Date: Sun, 6 Mar 2016 17:43:14 -0500
Subject: [PATCH] output (*BISECT-IGNORE*) in front of each
 non-programmer-written line

EXPERIMENTAL PATCH: this should of course be turned into an explicit option
---
 src/printer.ml | 1 +
 1 file changed, 1 insertion(+)

diff --git a/src/printer.ml b/src/printer.ml
index ea978bc..714bb08 100644
--- a/src/printer.ml
+++ b/src/printer.ml
@@ -46,6 +46,7 @@ let rawnl f =
 
 let nl f =
   rawnl f;
+  output_string f "(*BISECT-IGNORE*)";
   output_substring f whitespace 0 !indentation
 
 let indent ofs producer f x =
-- 
2.5.0

Re: [Caml-list] Test coverage of generated lexers/parsers

[Anton Bachin]

[-- Attachment #1: Type: text/plain, Size: 4229 bytes --]

If there is a good, general, alternative approach for this, we can support it in Bisect_ppx. Unfortunately, I don’t know enough about Menhir to be able to propose anything specific at this point.

Best,
Anton

> On Mar 6, 2016, at 16:53, Gabriel Scherer <gabriel.scherer@gmail.com> wrote:
> 
> This is an interesting question and, as far as I know, there is no good solution using existing versions of the interacting tools.
> 
> Below very simple patch that will add (*BISECT-IGNORE*) in front of every line of code generated by Menhir, except those written by the programmer (the "strecthes" in Menhir-speak). It applies cleanly on top of the latest released Menhir archive,
>   http://gallium.inria.fr/~fpottier/menhir/menhir-20160303.tar.gz <http://gallium.inria.fr/~fpottier/menhir/menhir-20160303.tar.gz>
> 
> The patch as-is is obviously a hack: it would need to be a configuration option when running menhir, and hard-coding Bisect (or bisect_ppx)'s syntax into Menhir is not elegant. One could try to have a configuration option to let users write a fixed string (or comment) at the beginning of each generated code line, but I'm not sure whether François Pottier (in cc:) would consider this is elegant enough. François, would you comment on whether this is a direction that seems acceptable to you?
> 
> (Bisect support ignoring entire regions at once by using (*BISECT-IGNORE-BEGIN*) and (*BISECT-IGNORE-END*); we could try to implement that instead of a per-line change, but I suspect that it would be slightly harder to implement (you have to hook the beginning of input, end of input, and around each user-code insertion) for no real gain.)
> 
> Toggling code-coverage semantics by inserting comments is not a very nice interface (although rather logical when you think of the level of generality required), so it's a bit frustrating that parser generators would have to play at this level. It would be better to have a more structured, unified interface supported by all the code-coverage tools, but to my knowledge no such thing exists.
> 
> 
> From d595ba5149a314c56623e1735af7678f5f62d525 Mon Sep 17 00:00:00 2001
> From: Gabriel Scherer <gabriel.scherer@gmail.com <mailto:gabriel.scherer@gmail.com>>
> Date: Sun, 6 Mar 2016 17:43:14 -0500
> Subject: [PATCH] output (*BISECT-IGNORE*) in front of each
>  non-programmer-written line
> 
> EXPERIMENTAL PATCH: this should of course be turned into an explicit option
> ---
>  src/printer.ml <http://printer.ml/> | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/src/printer.ml <http://printer.ml/> b/src/printer.ml <http://printer.ml/>
> index ea978bc..714bb08 100644
> --- a/src/printer.ml <http://printer.ml/>
> +++ b/src/printer.ml <http://printer.ml/>
> @@ -46,6 +46,7 @@ let rawnl f =
>  
>  let nl f =
>    rawnl f;
> +  output_string f "(*BISECT-IGNORE*)";
>    output_substring f whitespace 0 !indentation
>  
>  let indent ofs producer f x =
> -- 
> 2.5.0
> 
> 
> 
> On Sun, Mar 6, 2016 at 2:53 PM, Vincent Jacques <vincent@vincent-jacques.net <mailto:vincent@vincent-jacques.net>> wrote:
> Hello,
> 
> Does somebody have experience measuring test coverage of generated lexers/parsers?
> 
> I'm using ocamllex/ocamlyacc [1] (but I can switch to Menhir [2]) to generate a lexer/parser. In my tests, I simply check that some input strings give the ASTs I expect.
> 
> I usually use Bisect [3] to make sure that my tests cover the code I intended to cover, but in that configuration, Bisect is lost between the .mll/.mly files and the generated .ml files and produces useless reports.
> 
> How would you measure test coverage in that case?
> 
> Thanks,
> 
> [1] http://caml.inria.fr/pub/docs/manual-ocaml/lexyacc.html <http://caml.inria.fr/pub/docs/manual-ocaml/lexyacc.html>
> [2] http://gallium.inria.fr/~fpottier/menhir/ <http://gallium.inria.fr/~fpottier/menhir/>
> [3] http://bisect.x9c.fr/ <http://bisect.x9c.fr/>
> -- 
> Vincent Jacques
> http://vincent-jacques.net <http://vincent-jacques.net/>
> 
> "S'il n'y a pas de solution, c'est qu'il n'y a pas de problème"
>             Devise Shadock
> 
> <0001-output-BISECT-IGNORE-in-front-of-each-non-programmer.patch>


[-- Attachment #2: Type: text/html, Size: 6391 bytes --]

Re: [Caml-list] Test coverage of generated lexers/parsers

[Vincent Jacques]

[-- Attachment #1: Type: text/plain, Size: 4349 bytes --]

Thanks for your interest in this question! I suspected there was no obvious
solution.

I'll switch to Menhir (with Gabriel's experimental patch) and tell you if
it improved my situation.

2016-03-06 23:59 GMT+01:00 Anton Bachin <antonbachin@yahoo.com>:

> If there is a good, general, alternative approach for this, we can support
> it in Bisect_ppx. Unfortunately, I don’t know enough about Menhir to be
> able to propose anything specific at this point.
>
> Best,
> Anton
>
> On Mar 6, 2016, at 16:53, Gabriel Scherer <gabriel.scherer@gmail.com>
> wrote:
>
> This is an interesting question and, as far as I know, there is no good
> solution using existing versions of the interacting tools.
>
> Below very simple patch that will add (*BISECT-IGNORE*) in front of every
> line of code generated by Menhir, except those written by the programmer
> (the "strecthes" in Menhir-speak). It applies cleanly on top of the latest
> released Menhir archive,
>   http://gallium.inria.fr/~fpottier/menhir/menhir-20160303.tar.gz
>
> The patch as-is is obviously a hack: it would need to be a configuration
> option when running menhir, and hard-coding Bisect (or bisect_ppx)'s syntax
> into Menhir is not elegant. One could try to have a configuration option to
> let users write a fixed string (or comment) at the beginning of each
> generated code line, but I'm not sure whether François Pottier (in cc:)
> would consider this is elegant enough. François, would you comment on
> whether this is a direction that seems acceptable to you?
>
> (Bisect support ignoring entire regions at once by using
> (*BISECT-IGNORE-BEGIN*) and (*BISECT-IGNORE-END*); we could try to
> implement that instead of a per-line change, but I suspect that it would be
> slightly harder to implement (you have to hook the beginning of input, end
> of input, and around each user-code insertion) for no real gain.)
>
> Toggling code-coverage semantics by inserting comments is not a very nice
> interface (although rather logical when you think of the level of
> generality required), so it's a bit frustrating that parser generators
> would have to play at this level. It would be better to have a more
> structured, unified interface supported by all the code-coverage tools, but
> to my knowledge no such thing exists.
>
>
> From d595ba5149a314c56623e1735af7678f5f62d525 Mon Sep 17 00:00:00 2001
> From: Gabriel Scherer <gabriel.scherer@gmail.com>
> Date: Sun, 6 Mar 2016 17:43:14 -0500
> Subject: [PATCH] output (*BISECT-IGNORE*) in front of each
>  non-programmer-written line
>
> EXPERIMENTAL PATCH: this should of course be turned into an explicit option
> ---
>  src/printer.ml | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/src/printer.ml b/src/printer.ml
> index ea978bc..714bb08 100644
> --- a/src/printer.ml
> +++ b/src/printer.ml
> @@ -46,6 +46,7 @@ let rawnl f =
>
>  let nl f =
>    rawnl f;
> +  output_string f "(*BISECT-IGNORE*)";
>    output_substring f whitespace 0 !indentation
>
>  let indent ofs producer f x =
> --
> 2.5.0
>
>
>
> On Sun, Mar 6, 2016 at 2:53 PM, Vincent Jacques <
> vincent@vincent-jacques.net> wrote:
>
>> Hello,
>>
>> Does somebody have experience measuring test coverage of generated
>> lexers/parsers?
>>
>> I'm using ocamllex/ocamlyacc [1] (but I can switch to Menhir [2]) to
>> generate a lexer/parser. In my tests, I simply check that some input
>> strings give the ASTs I expect.
>>
>> I usually use Bisect [3] to make sure that my tests cover the code I
>> intended to cover, but in that configuration, Bisect is lost between the
>> .mll/.mly files and the generated .ml files and produces useless reports.
>>
>> How would you measure test coverage in that case?
>>
>> Thanks,
>>
>> [1] http://caml.inria.fr/pub/docs/manual-ocaml/lexyacc.html
>> [2] http://gallium.inria.fr/~fpottier/menhir/
>> [3] http://bisect.x9c.fr/
>> --
>> Vincent Jacques
>> http://vincent-jacques.net
>>
>> "S'il n'y a pas de solution, c'est qu'il n'y a pas de problème"
>>             Devise Shadock
>>
>
> <0001-output-BISECT-IGNORE-in-front-of-each-non-programmer.patch>
>
>
>


-- 
Vincent Jacques
http://vincent-jacques.net

"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème"
            Devise Shadock

[-- Attachment #2: Type: text/html, Size: 6229 bytes --]

Re: [Caml-list] Test coverage of generated lexers/parsers

[François Pottier]

Hello,

Le 03/06/2016 11:53 PM, Gabriel Scherer a écrit :
> Below very simple patch that will add (*BISECT-IGNORE*) in front of every
> line of code generated by Menhir, except those written by the programmer
> [...] One could try to have a configuration option to let users write a
> fixed string (or comment) at the beginning of each generated code line,

This seems to be such a simple change in Menhir that it would be acceptable.

-- 
François Pottier
francois.pottier@inria.fr
http://gallium.inria.fr/~fpottier/