[Caml-list] [ANN] Bisect_ppx 1.0.0 – Modernized code coverage for OCaml

[Caml-list] [ANN] Bisect_ppx 1.0.0 – Modernized code coverage for OCaml

[Anton Bachin]

Hello,

We would like to announce the release of Bisect_ppx 1.0.0, a code coverage tool
for OCaml with appealing reports:

  https://github.com/rleonid/bisect_ppx


You can see a live coverage report here:

  http://rleonid.github.io/bisect_ppx/coverage/

Reports can also be submitted to Coveralls.io using ocveralls [1]. See an
example here [2].


Bisect_ppx is a fork of the original Bisect by Xavier Clerc, with extensive
further development. Differences from Bisect, and from earlier versions of
Bisect_ppx, include:

- the nicer and more legible HTML reports,
- more thorough instrumentation, now including nested functions and or-patterns,
- improved compatibility with other PPX rewriters,
- an Ocamlbuild plugin,
- many bugfixes, and
- usage, performance, and documentation improvements.


Bisect_ppx was originally forked to update and maintain Bisect's PPX mode, with
the OCaml community moving to PPX. Bisect_ppx does not have Bisect's Camlp4
dependency. We do not believe that the original Bisect is being actively
maintained.


Regards,
Anton & Leonid


P.S. If you are working on a project that uses Bisect_ppx, please let us know!


[1]: https://github.com/sagotch/ocveralls
[2]: https://coveralls.io/builds/4913198/source?filename=src%2Fsyntax%2FinstrumentPpx.ml

Re: [Caml-list] [ANN] Bisect_ppx 1.0.0 – Modernized code coverage for OCaml

[David MENTRE]

Hello,

Interesting tool.

Le 10/02/2016 16:33, Anton Bachin a écrit :
> You can see a live coverage report here:
>
>    http://rleonid.github.io/bisect_ppx/coverage/

I quickly looked at the report output (so I might have not understood 
the working of your tool), but in 
https://rleonid.github.io/bisect_ppx/coverage/file0004.html , why:

  * l. 107, 'let try_finally x f h =' is not green. The function seems 
to be called as l. 108 is green. (Same remark for all defined functions)
    https://rleonid.github.io/bisect_ppx/coverage/file0004.html#L107

  * l. 110, 'f x' is not green? 'f' seems to be called if the previous 
"try" is green.
    https://rleonid.github.io/bisect_ppx/coverage/file0004.html#L110

  * l. 139, '^ ".cmp"' is not green? Once again, this expression seems 
to executed by your tests.
    https://rleonid.github.io/bisect_ppx/coverage/file0004.html#L139

  * l. 168 and l. 169, 'else' is never coloured? For me, it should be 
green l. 168 and red l. 170.
    https://rleonid.github.io/bisect_ppx/coverage/file0004.html#L168


Overall, I would expect all code lines to be either green, red or 
yellow, not without colour (except for comments and type and exception 
definitions).

Sincerely yours,
david

Re: [Caml-list] [ANN] Bisect_ppx 1.0.0 – Modernized code coverage for OCaml

[Gerd Stolpmann]

[-- Attachment #1: Type: text/plain, Size: 2296 bytes --]

That's interesting news. In my last job I used (the old) bisect
frequently.

What I really would love to see is negative coverage: Mark code sections
where you expect that they are never executed. That could be a simple
"assert false" but also more extensive error handling. Ideally, the tool
would automatically recognize certain patterns.

That's especially useful when you have a management that is after high
coverage numbers...

Gerd


Am Mittwoch, den 10.02.2016, 09:33 -0600 schrieb Anton Bachin:
> Hello,
> 
> We would like to announce the release of Bisect_ppx 1.0.0, a code coverage tool
> for OCaml with appealing reports:
> 
>   https://github.com/rleonid/bisect_ppx
> 
> 
> You can see a live coverage report here:
> 
>   http://rleonid.github.io/bisect_ppx/coverage/
> 
> Reports can also be submitted to Coveralls.io using ocveralls [1]. See an
> example here [2].
> 
> 
> Bisect_ppx is a fork of the original Bisect by Xavier Clerc, with extensive
> further development. Differences from Bisect, and from earlier versions of
> Bisect_ppx, include:
> 
> - the nicer and more legible HTML reports,
> - more thorough instrumentation, now including nested functions and or-patterns,
> - improved compatibility with other PPX rewriters,
> - an Ocamlbuild plugin,
> - many bugfixes, and
> - usage, performance, and documentation improvements.
> 
> 
> Bisect_ppx was originally forked to update and maintain Bisect's PPX mode, with
> the OCaml community moving to PPX. Bisect_ppx does not have Bisect's Camlp4
> dependency. We do not believe that the original Bisect is being actively
> maintained.
> 
> 
> Regards,
> Anton & Leonid
> 
> 
> P.S. If you are working on a project that uses Bisect_ppx, please let us know!
> 
> 
> [1]: https://github.com/sagotch/ocveralls
> [2]: https://coveralls.io/builds/4913198/source?filename=src%2Fsyntax%2FinstrumentPpx.ml
> 
> 

-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
My OCaml site:          http://www.camlcity.org
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Caml-list] [ANN] Bisect_ppx 1.0.0 – Modernized code coverage for OCaml

[Gabriel Scherer]

[-- Attachment #1: Type: text/plain, Size: 2567 bytes --]

On Wed, Feb 10, 2016 at 2:00 PM, Gerd Stolpmann <info@gerd-stolpmann.de>
wrote:

> That's interesting news. In my last job I used (the old) bisect
> frequently.
>
> What I really would love to see is negative coverage: Mark code sections
> where you expect that they are never executed. That could be a simple
> "assert false" but also more extensive error handling. Ideally, the tool
> would automatically recognize certain patterns.
>

The old bisect supports special (*BISECT-MARK*) comment to mark dead code
for this reason. Do you have something else in mind?


> That's especially useful when you have a management that is after high
> coverage numbers...
>
> Gerd
>
>
> Am Mittwoch, den 10.02.2016, 09:33 -0600 schrieb Anton Bachin:
> > Hello,
> >
> > We would like to announce the release of Bisect_ppx 1.0.0, a code
> coverage tool
> > for OCaml with appealing reports:
> >
> >   https://github.com/rleonid/bisect_ppx
> >
> >
> > You can see a live coverage report here:
> >
> >   http://rleonid.github.io/bisect_ppx/coverage/
> >
> > Reports can also be submitted to Coveralls.io using ocveralls [1]. See an
> > example here [2].
> >
> >
> > Bisect_ppx is a fork of the original Bisect by Xavier Clerc, with
> extensive
> > further development. Differences from Bisect, and from earlier versions
> of
> > Bisect_ppx, include:
> >
> > - the nicer and more legible HTML reports,
> > - more thorough instrumentation, now including nested functions and
> or-patterns,
> > - improved compatibility with other PPX rewriters,
> > - an Ocamlbuild plugin,
> > - many bugfixes, and
> > - usage, performance, and documentation improvements.
> >
> >
> > Bisect_ppx was originally forked to update and maintain Bisect's PPX
> mode, with
> > the OCaml community moving to PPX. Bisect_ppx does not have Bisect's
> Camlp4
> > dependency. We do not believe that the original Bisect is being actively
> > maintained.
> >
> >
> > Regards,
> > Anton & Leonid
> >
> >
> > P.S. If you are working on a project that uses Bisect_ppx, please let us
> know!
> >
> >
> > [1]: https://github.com/sagotch/ocveralls
> > [2]:
> https://coveralls.io/builds/4913198/source?filename=src%2Fsyntax%2FinstrumentPpx.ml
> >
> >
>
> --
> ------------------------------------------------------------
> Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
> My OCaml site:          http://www.camlcity.org
> Contact details:        http://www.camlcity.org/contact.html
> Company homepage:       http://www.gerd-stolpmann.de
> ------------------------------------------------------------
>
>

[-- Attachment #2: Type: text/html, Size: 4126 bytes --]

Re: [Caml-list] [ANN] Bisect_ppx 1.0.0 – Modernized code coverage for OCaml

[Anton Bachin]

Hi David,

Thanks for your comments.

> * l. 107, 'let try_finally x f h =' is not green. The function seems to be called as l. 108 is green. (Same remark for all defined functions)
>   https://rleonid.github.io/bisect_ppx/coverage/file0004.html#L107

This is an oversight on our part. We mark expressions in module-level
let-bindings, and not the let-bindings themselves. The idea is that the
expression is reached if and only if the let-binding is reached. That is
why you see the mark on the next line. However, in case the let binding
has parameters, our marking is wrong. Right now we handle

  let x = (* mark *) 1

correctly and

  let f x = (* mark *) 1

incorrectly, because that desugars to

  let f = (* no mark! *) fun x -> (* mark *) 1

The actual problem is very likely marks occupying the same location. In any
case, I’ve opened an issue for dealing with this:

https://github.com/rleonid/bisect_ppx/issues/81

> * l. 110, 'f x' is not green? 'f' seems to be called if the previous "try" is green.
>   https://rleonid.github.io/bisect_ppx/coverage/file0004.html#L110

The “f x” in try is evaluated if and only if the whole try-expression is
evaluated – unless, perhaps, the *implementation* of exception handling can
cause a memory or stack overflow exception to occur between these two points –
can someone knowledgeable comment? However, assuming my previous statement is
correct, it’s not necessary to have a mark in both places. We chose to put the
mark on “try” because it is technically very slightly easier to do so. However,
we could be convinced to move it to the expression instead.

> * l. 139, '^ ".cmp"' is not green? Once again, this expression seems to executed by your tests.
>   https://rleonid.github.io/bisect_ppx/coverage/file0004.html#L139

Thanks. We are now looking into putting marks on function arguments. It will
require some investigation to do correctly and legibly for all cases (e.g. infix
operators). Not sure why this wasn’t done in original Bisect. If we find a good
reason for *not* doing it, we will document it.

https://github.com/rleonid/bisect_ppx/issues/82

> * l. 168 and l. 169, 'else' is never coloured? For me, it should be green l. 168 and red l. 170.
>   https://rleonid.github.io/bisect_ppx/coverage/file0004.html#L168

To the best of my knowledge, we can’t get the location of the else keyword in an
if-then-else in PPX. That’s why we mark at the start of the subexpressions.

> Overall, I would expect all code lines to be either green, red or yellow, not without colour (except for comments and type and exception definitions).

Bisect_ppx colors expressions and not lines – the line colors just indicate
whether there is a colored expression on each line. So, yes, while many lines in
expressions should be colored, some won’t be – for example, from above, lines
containing only delimiters such as “then”, “else”, and “end”.


We are looking to extend coverage analysis as much as reasonably possible, so
thanks again, and do let us know if you find any other gaps :)


Regards,
Anton

Re: [Caml-list] [ANN] Bisect_ppx 1.0.0 – Modernized code coverage for OCaml

[Anton Bachin]

[-- Attachment #1: Type: text/plain, Size: 3560 bytes --]

Assuming this is what Gerd is indeed looking for, I can confirm that Bisect_ppx
still supports it. See

https://github.com/rleonid/bisect_ppx/blob/master/doc/advanced.md#unreachable-code <https://github.com/rleonid/bisect_ppx/blob/master/doc/advanced.md#unreachable-code>

We preferred to document only BISECT_VISIT (which does the same thing), in order
to have only option. But both Bisect_ppx and Bisect support BISECT_MARK and
BISECT_VISIT.

> On Feb 10, 2016, at 13:23, Gabriel Scherer <gabriel.scherer@gmail.com> wrote:
> 
> 
> 
> On Wed, Feb 10, 2016 at 2:00 PM, Gerd Stolpmann <info@gerd-stolpmann.de <mailto:info@gerd-stolpmann.de>> wrote:
> That's interesting news. In my last job I used (the old) bisect
> frequently.
> 
> What I really would love to see is negative coverage: Mark code sections
> where you expect that they are never executed. That could be a simple
> "assert false" but also more extensive error handling. Ideally, the tool
> would automatically recognize certain patterns.
> 
> The old bisect supports special (*BISECT-MARK*) comment to mark dead code for this reason. Do you have something else in mind?
>  
> That's especially useful when you have a management that is after high
> coverage numbers...
> 
> Gerd
> 
> 
> Am Mittwoch, den 10.02.2016, 09:33 -0600 schrieb Anton Bachin:
> > Hello,
> >
> > We would like to announce the release of Bisect_ppx 1.0.0, a code coverage tool
> > for OCaml with appealing reports:
> >
> >   https://github.com/rleonid/bisect_ppx <https://github.com/rleonid/bisect_ppx>
> >
> >
> > You can see a live coverage report here:
> >
> >   http://rleonid.github.io/bisect_ppx/coverage/ <http://rleonid.github.io/bisect_ppx/coverage/>
> >
> > Reports can also be submitted to Coveralls.io using ocveralls [1]. See an
> > example here [2].
> >
> >
> > Bisect_ppx is a fork of the original Bisect by Xavier Clerc, with extensive
> > further development. Differences from Bisect, and from earlier versions of
> > Bisect_ppx, include:
> >
> > - the nicer and more legible HTML reports,
> > - more thorough instrumentation, now including nested functions and or-patterns,
> > - improved compatibility with other PPX rewriters,
> > - an Ocamlbuild plugin,
> > - many bugfixes, and
> > - usage, performance, and documentation improvements.
> >
> >
> > Bisect_ppx was originally forked to update and maintain Bisect's PPX mode, with
> > the OCaml community moving to PPX. Bisect_ppx does not have Bisect's Camlp4
> > dependency. We do not believe that the original Bisect is being actively
> > maintained.
> >
> >
> > Regards,
> > Anton & Leonid
> >
> >
> > P.S. If you are working on a project that uses Bisect_ppx, please let us know!
> >
> >
> > [1]: https://github.com/sagotch/ocveralls <https://github.com/sagotch/ocveralls>
> > [2]: https://coveralls.io/builds/4913198/source?filename=src%2Fsyntax%2FinstrumentPpx.ml <https://coveralls.io/builds/4913198/source?filename=src%2Fsyntax%2FinstrumentPpx.ml>
> >
> >
> 
> --
> ------------------------------------------------------------
> Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de <mailto:gerd@gerd-stolpmann.de>
> My OCaml site:          http://www.camlcity.org <http://www.camlcity.org/>
> Contact details:        http://www.camlcity.org/contact.html <http://www.camlcity.org/contact.html>
> Company homepage:       http://www.gerd-stolpmann.de <http://www.gerd-stolpmann.de/>
> ------------------------------------------------------------
> 
> 


[-- Attachment #2: Type: text/html, Size: 6188 bytes --]

Re: [Caml-list] [ANN] Bisect_ppx 1.0.0 – Modernized code coverage for OCaml

[Gerd Stolpmann]

[-- Attachment #1: Type: text/plain, Size: 5940 bytes --]

Yes, we used BISECT_MARK a lot to resolve this, but it is a cumbersome
and incomplete solution. Well, error paths are special: some errors must
not occur at all, and some only in certain configurations, and some
errors can be tested out, and others cannot.

What I have in mind, is something like:

 - a new tag NEVER - it is an error if the code is executed.
   NEVER automatically extends to inner blocks. E.g.

   if error then begin 
     (* NEVER *)
     if debug_enabled then debug_log "error";
     failwith "error"
   end

   An "assert false" automatically counts as NEVER.

   In contrast, a MARK/VISIT only ignores code and AFAIK it doesn't
   extend to inner blocks.

 - conditional sections that can be turned off and on via
   command-line switches, and that either count as positive
   (normal) sections or negative sections like NEVER.

   Something like: (* BEGIN(name) *) ... (* END(name) *)

   and then a command-line switch (or env variable)

   -section name=(ignore|expect|never)

The background for conditional sections is that (1) sometimes certain
features are only testable in certain configurations, and (2) you want
to have some control for managing testing - let's first focus on feature
X but ignore Y for a while. In particular for larger code bases this is
very helpful.

But anyway, thanks for listening. Bisect was very helpful for testing
code.

Gerd


Am Mittwoch, den 10.02.2016, 13:33 -0600 schrieb Anton Bachin:
> Assuming this is what Gerd is indeed looking for, I can confirm that
> Bisect_ppx
> still supports it. See
> 
> 
> https://github.com/rleonid/bisect_ppx/blob/master/doc/advanced.md#unreachable-code
> 
> 
> We preferred to document only BISECT_VISIT (which does the same
> thing), in order
> to have only option. But both Bisect_ppx and Bisect support
> BISECT_MARK and
> BISECT_VISIT.
> 
> > On Feb 10, 2016, at 13:23, Gabriel Scherer
> > <gabriel.scherer@gmail.com> wrote:
> > 
> > 
> > 
> > On Wed, Feb 10, 2016 at 2:00 PM, Gerd Stolpmann
> > <info@gerd-stolpmann.de> wrote:
> >         That's interesting news. In my last job I used (the old)
> >         bisect
> >         frequently.
> >         
> >         What I really would love to see is negative coverage: Mark
> >         code sections
> >         where you expect that they are never executed. That could be
> >         a simple
> >         "assert false" but also more extensive error handling.
> >         Ideally, the tool
> >         would automatically recognize certain patterns.
> > 
> > 
> > The old bisect supports special (*BISECT-MARK*) comment to mark dead
> > code for this reason. Do you have something else in mind?
> >  
> >         That's especially useful when you have a management that is
> >         after high
> >         coverage numbers...
> >         
> >         Gerd
> >         
> >         
> >         Am Mittwoch, den 10.02.2016, 09:33 -0600 schrieb Anton
> >         Bachin:
> >         > Hello,
> >         >
> >         > We would like to announce the release of Bisect_ppx 1.0.0,
> >         a code coverage tool
> >         > for OCaml with appealing reports:
> >         >
> >         >   https://github.com/rleonid/bisect_ppx
> >         >
> >         >
> >         > You can see a live coverage report here:
> >         >
> >         >   http://rleonid.github.io/bisect_ppx/coverage/
> >         >
> >         > Reports can also be submitted to Coveralls.io using
> >         ocveralls [1]. See an
> >         > example here [2].
> >         >
> >         >
> >         > Bisect_ppx is a fork of the original Bisect by Xavier
> >         Clerc, with extensive
> >         > further development. Differences from Bisect, and from
> >         earlier versions of
> >         > Bisect_ppx, include:
> >         >
> >         > - the nicer and more legible HTML reports,
> >         > - more thorough instrumentation, now including nested
> >         functions and or-patterns,
> >         > - improved compatibility with other PPX rewriters,
> >         > - an Ocamlbuild plugin,
> >         > - many bugfixes, and
> >         > - usage, performance, and documentation improvements.
> >         >
> >         >
> >         > Bisect_ppx was originally forked to update and maintain
> >         Bisect's PPX mode, with
> >         > the OCaml community moving to PPX. Bisect_ppx does not
> >         have Bisect's Camlp4
> >         > dependency. We do not believe that the original Bisect is
> >         being actively
> >         > maintained.
> >         >
> >         >
> >         > Regards,
> >         > Anton & Leonid
> >         >
> >         >
> >         > P.S. If you are working on a project that uses Bisect_ppx,
> >         please let us know!
> >         >
> >         >
> >         > [1]: https://github.com/sagotch/ocveralls
> >         > [2]:
> >         https://coveralls.io/builds/4913198/source?filename=src%
> >         2Fsyntax%2FinstrumentPpx.ml
> >         >
> >         >
> >         
> >         --
> >         ------------------------------------------------------------
> >         Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
> >         My OCaml site:          http://www.camlcity.org
> >         Contact details:        http://www.camlcity.org/contact.html
> >         Company homepage:       http://www.gerd-stolpmann.de
> >         ------------------------------------------------------------
> >         
> > 
> > 
> 

-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
My OCaml site:          http://www.camlcity.org
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Caml-list] [ANN] Bisect_ppx 1.0.0 – Modernized code coverage for OCaml

[Alain Frisch]

On 10/02/2016 20:33, Anton Bachin wrote:
> Assuming this is what Gerd is indeed looking for, I can confirm that
> Bisect_ppx
> still supports it. See
>
> https://github.com/rleonid/bisect_ppx/blob/master/doc/advanced.md#unreachable-code
>
> We preferred to document only BISECT_VISIT (which does the same thing),
> in order
> to have only option. But both Bisect_ppx and Bisect support BISECT_MARK and
> BISECT_VISIT.


It's great to see the development of Bisect resurrecting!


Did you consider switching to attributes to mark code that must be 
excluded from the analysis, instead of special comments?  This seems 
simpler than having to scan again the source files, and it could be more 
robust when interacting with other ppx tools.

For instance:


- Marking a sub-expression, often using the shorthand syntax:

   begin[@bisect.visit]
     ...
   end

- Marking a complete function:


   let f x =
      ...
   [@@bisect.visit]

   or (in OCaml 4.03):

   let[@bisect.visit] f x =
      ...

- Marking sections:


     [@@@bisect.visit.begin]
     ...
     [@@@bisect.visit.end]



Alain

Re: [Caml-list] [ANN] Bisect_ppx 1.0.0 – Modernized code coverage for OCaml

[Anton Bachin]

Alain, that is a pretty nice suggestion – thanks! I’ve opened an issue for it:

https://github.com/rleonid/bisect_ppx/issues/83

Gerd, are you still working on such projects, as would need these features? We
would consider adding them, but ideally, to remain grounded in reality, it would
be for ongoing, routine usage. It's just easier to get feedback that way and
avoid getting ahead of ourselves.

Actually, Alain's suggestion may make some such features easier to use.


Regards,
Anton




> On Feb 10, 2016, at 15:49, Alain Frisch <alain.frisch@lexifi.com> wrote:
> 
> On 10/02/2016 20:33, Anton Bachin wrote:
>> Assuming this is what Gerd is indeed looking for, I can confirm that
>> Bisect_ppx
>> still supports it. See
>> 
>> https://github.com/rleonid/bisect_ppx/blob/master/doc/advanced.md#unreachable-code
>> 
>> We preferred to document only BISECT_VISIT (which does the same thing),
>> in order
>> to have only option. But both Bisect_ppx and Bisect support BISECT_MARK and
>> BISECT_VISIT.
> 
> 
> It's great to see the development of Bisect resurrecting!
> 
> 
> Did you consider switching to attributes to mark code that must be excluded from the analysis, instead of special comments?  This seems simpler than having to scan again the source files, and it could be more robust when interacting with other ppx tools.
> 
> For instance:
> 
> 
> - Marking a sub-expression, often using the shorthand syntax:
> 
>  begin[@bisect.visit]
>    ...
>  end
> 
> - Marking a complete function:
> 
> 
>  let f x =
>     ...
>  [@@bisect.visit]
> 
>  or (in OCaml 4.03):
> 
>  let[@bisect.visit] f x =
>     ...
> 
> - Marking sections:
> 
> 
>    [@@@bisect.visit.begin]
>    ...
>    [@@@bisect.visit.end]
> 
> 
> 
> Alain
> 
> -- 
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs

Re: [Caml-list] [ANN] Bisect_ppx 1.0.0 – Modernized code coverage for OCaml

[Gerd Stolpmann]

[-- Attachment #1: Type: text/plain, Size: 2749 bytes --]

Am Mittwoch, den 10.02.2016, 19:16 -0600 schrieb Anton Bachin:
> Alain, that is a pretty nice suggestion – thanks! I’ve opened an issue for it:
> 
> https://github.com/rleonid/bisect_ppx/issues/83
> 
> Gerd, are you still working on such projects, as would need these features? We
> would consider adding them, but ideally, to remain grounded in reality, it would
> be for ongoing, routine usage. It's just easier to get feedback that way and
> avoid getting ahead of ourselves.

No, currently I'm not working in this area anymore. It's low priority. I
just wanted to give you some feedback.

Gerd


> Actually, Alain's suggestion may make some such features easier to use.
> 
> 
> Regards,
> Anton
> 
> 
> 
> 
> > On Feb 10, 2016, at 15:49, Alain Frisch <alain.frisch@lexifi.com> wrote:
> > 
> > On 10/02/2016 20:33, Anton Bachin wrote:
> >> Assuming this is what Gerd is indeed looking for, I can confirm that
> >> Bisect_ppx
> >> still supports it. See
> >> 
> >> https://github.com/rleonid/bisect_ppx/blob/master/doc/advanced.md#unreachable-code
> >> 
> >> We preferred to document only BISECT_VISIT (which does the same thing),
> >> in order
> >> to have only option. But both Bisect_ppx and Bisect support BISECT_MARK and
> >> BISECT_VISIT.
> > 
> > 
> > It's great to see the development of Bisect resurrecting!
> > 
> > 
> > Did you consider switching to attributes to mark code that must be excluded from the analysis, instead of special comments?  This seems simpler than having to scan again the source files, and it could be more robust when interacting with other ppx tools.
> > 
> > For instance:
> > 
> > 
> > - Marking a sub-expression, often using the shorthand syntax:
> > 
> >  begin[@bisect.visit]
> >    ...
> >  end
> > 
> > - Marking a complete function:
> > 
> > 
> >  let f x =
> >     ...
> >  [@@bisect.visit]
> > 
> >  or (in OCaml 4.03):
> > 
> >  let[@bisect.visit] f x =
> >     ...
> > 
> > - Marking sections:
> > 
> > 
> >    [@@@bisect.visit.begin]
> >    ...
> >    [@@@bisect.visit.end]
> > 
> > 
> > 
> > Alain
> > 
> > -- 
> > Caml-list mailing list.  Subscription management and archives:
> > https://sympa.inria.fr/sympa/arc/caml-list
> > Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> > Bug reports: http://caml.inria.fr/bin/caml-bugs
> 
> 

-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
My OCaml site:          http://www.camlcity.org
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Caml-list] [ANN] Bisect_ppx 1.0.0 – Modernized code coverage for OCaml

[David MENTRÉ]

Hello Anton,

Thank you for your detailed explanations.

Le 2016-02-10 20:27, Anton Bachin a écrit :
>> * l. 110, 'f x' is not green? 'f' seems to be called if the previous "try" is green.
>> >   https://rleonid.github.io/bisect_ppx/coverage/file0004.html#L110
> The “f x” in try is evaluated if and only if the whole try-expression is
> evaluated – unless, perhaps, the *implementation* of exception handling can
> cause a memory or stack overflow exception to occur between these two points –
> can someone knowledgeable comment? However, assuming my previous statement is
> correct, it’s not necessary to have a mark in both places. We chose to put the
> mark on “try” because it is technically very slightly easier to do so. However,
> we could be convinced to move it to the expression instead.

Well I would have marked the "f x". My reasoning is simple, if f is
executed, it should be green. And f is more important than try regarding
coverage.


>> > Overall, I would expect all code lines to be either green, red or yellow, not without colour (except for comments and type and exception definitions).
> Bisect_ppx colors expressions and not lines – the line colors just indicate
> whether there is a colored expression on each line. So, yes, while many lines in
> expressions should be colored, some won’t be – for example, from above, lines
> containing only delimiters such as “then”, “else”, and “end”.

Two remarks:

 * If expressions are marked (which I suspected), then the HTML output
should color at expression granularity level and not line granularity
level. Therefore you would have only green or red and never orange[1]
(which would be simpler and clearer);

 * You could do some post-processing to look for such keywords as
“then”, “else”, and “end” and color them accordingly. It might be more a
hack than a clean approach, but I think it would make the output more
readable.

All that said, I'm not using your tool (for now), so I would understand
that you not take my remarks into account. ;-)

Best regards,
david

[1] Or maybe only in the summary on the left side of file display.

Re: [Caml-list] [ANN] Bisect_ppx 1.0.0 – Modernized code coverage for OCaml

[Anton Bachin]

> Well I would have marked the "f x". My reasoning is simple, if f is
> executed, it should be green. And f is more important than try regarding
> coverage.

It certainly would be more readable, and we will keep this in mind.
However, I think we would delay addressing this for now, because there are
even more “bad” examples from the point of view of readability, that have
higher priority. For example, the first line of every case is typically not
marked (the pattern is marked instead):

http://rleonid.github.io/bisect_ppx/coverage/file0026.html#L41

So if the same try-expression was on the first line of a case, neither
the try nor the subexpression would end up marked by the current procedure.

While it would be nice to have a solution to this, short of marking every
subexpression, we are not sure what to do. There is some tradeoff between
marking every subexpression and trying to mark only “relevant" branching
points, and so far we have tried to stick to the latter option.

Perhaps it would be good to hear arguments for and against both options.

> Two remarks:
> 
> * If expressions are marked (which I suspected), then the HTML output
> should color at expression granularity level and not line granularity
> level. Therefore you would have only green or red and never orange[1]
> (which would be simpler and clearer);

The output is colored at the expression granularity level. The letters with
the bolder background are the real locations of each mark. Lines are then
colored according to whether they have visited marks, unvisited marks,
neither, or both.

What I think you are suggesting is that each line be subdivided into
subexpression spans, but I don’t think that will work – because an unvisited
expression might have a visited subexpression, for instance.

> * You could do some post-processing to look for such keywords as
> “then”, “else”, and “end” and color them accordingly. It might be more a
> hack than a clean approach, but I think it would make the output more
> readable.

Leonid briefly looked at doing this yesterday, and we will probably look
at it again, but we would expect it to be a bit of a kludge so we would
probably hesitate to implement it. It would make things more intuitively
readable, however, so we are keeping it in mind.


Regards,
Anton