[Caml-list] Coverage profiling with bisect_ppx, Oasis, and OCamlbuild - best practice?

[Caml-list] Coverage profiling with bisect_ppx, Oasis, and OCamlbuild - best practice?

[Christian Lindig]

Bisect_ppx (https://github.com/aantron/bisect_ppx) is a great tool for instrument code for coverage profiling. I’d like to explore the best way to use this in a bigger project that is built with Oasis/OCamlbuild. In particular, I’d like to find a high-level way to specify that I want variants of binaries produced, analogous to 

	binary.native
	binary.byte
	binary.coverage (say, for native binary with coverage instrumentation)

without duplicating too much code in the _oasis file. It is my goal to produce such binaries for installation and hence would like to avoid doing this sequentially like in 

	make opt; make clean; make byte; make clean 

but rather have one target that produces all variants. If this is impossible, I’d revert to building variants sequentially. The reason I expect this to be difficult is that an object file produced with and without instrumentation looks the same and thus it might be difficult for the build system to understand that these need the be rebuilt. 

Code instrumented for coverage needs to be compiled with the corresponding PPX. OCamlfind and OCamlbuild abstracts this nicely. For compiling example.ml it’s enough to say:

	ocamlbuild —use-ocamlfind -pkg bisect_ppx example.native

It is less obvious how to organise this in a bigger project. (For now I add rules 

	<**/*.ml{,i,y}>: pkg_bisect_ppx 
	<**/*.native>: pkg_bisect_ppx 

to the _tags file generated by Oasis.) I believe the problem already arises with profiling and debugging builds and I would be happy to learn how OCamlbuild can be directed via myocamlbuild.ml to do this. 

— Christian

Re: [Caml-list] Coverage profiling with bisect_ppx, Oasis, and OCamlbuild - best practice?

[Gabriel Scherer]

[-- Attachment #1: Type: text/plain, Size: 3455 bytes --]

Inside the OCaml compiler codebase, profiling and debugging builds are
handled by using a different extension suffix: .p.cmx and .d.cmx, for
example. In my experience, this way of doing things is rather fragile,
because it is easy to get in a situation where we would expect the compiler
to go read foo.bar.cmi or foo.bar.cmx but it instead reads foo.cmi or
foo.cmx (or doesn't find them and fails or continues silently without
cross-module information). In my experience, using "-tag debug" instead is
less convenient but more robust, and this is what I recommend.

(This problem could be solved by having a richer language for telling the
compiler where to read files from and where to produce output files. I
think this would be of independent interest, have other benefits, but it
would also be a lot of work.)

Note that if you go the "separate ocamlbuild calls" route, you do not
necessarily need to clean between each build, you can use different build
directories instead:

  ocamlbuild -tag debug -build-dir _build_debug targets.otarget
  ocamlbuild -tag "package(bisect-ppx)" -build-dir _build_coverage
targets.otarget
  ...

In particular, this approach gives you incremental rebuilds during
development -- the clean-heavy solution does not.

Note that to appease ocamlbuild hygiene checker you then need to explicitly
mention that the *other* build directories should be ignored. (It used to
be ignored by default in older OCaml versions, but 0.9.2 will complain
about them.) You can use a generic rule in your _tags file for that:

   <_build*>: -traverse

On Thu, May 19, 2016 at 2:46 PM, Christian Lindig <lindig@gmail.com> wrote:

>
> Bisect_ppx (https://github.com/aantron/bisect_ppx) is a great tool for
> instrument code for coverage profiling. I’d like to explore the best way to
> use this in a bigger project that is built with Oasis/OCamlbuild. In
> particular, I’d like to find a high-level way to specify that I want
> variants of binaries produced, analogous to
>
>         binary.native
>         binary.byte
>         binary.coverage (say, for native binary with coverage
> instrumentation)
>
> without duplicating too much code in the _oasis file. It is my goal to
> produce such binaries for installation and hence would like to avoid doing
> this sequentially like in
>
>         make opt; make clean; make byte; make clean
>
> but rather have one target that produces all variants. If this is
> impossible, I’d revert to building variants sequentially. The reason I
> expect this to be difficult is that an object file produced with and
> without instrumentation looks the same and thus it might be difficult for
> the build system to understand that these need the be rebuilt.
>
> Code instrumented for coverage needs to be compiled with the corresponding
> PPX. OCamlfind and OCamlbuild abstracts this nicely. For compiling
> example.ml it’s enough to say:
>
>         ocamlbuild —use-ocamlfind -pkg bisect_ppx example.native
>
> It is less obvious how to organise this in a bigger project. (For now I
> add rules
>
>         <**/*.ml{,i,y}>: pkg_bisect_ppx
>         <**/*.native>: pkg_bisect_ppx
>
> to the _tags file generated by Oasis.) I believe the problem already
> arises with profiling and debugging builds and I would be happy to learn
> how OCamlbuild can be directed via myocamlbuild.ml to do this.
>
> — Christian
>
>

[-- Attachment #2: Type: text/html, Size: 4243 bytes --]