Re: [Caml-list] OCaml projects with tests

[Gabriel Scherer <gabriel.scherer@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 5838 bytes --]

Batteries uses a lot of tests, and we couldn't live without them.

In particularly, they make maintainer's life much easier (it's easier to
make the decision to integrate code when, besides being nice to read, it is
also well-tested), and that is key in a project with many small
contributions, so a lot of integration work. It is mandatory for new
functions proposed for inclusions to come with unit tests, which means that
new code has more tests than old code (inherited from Extlib, with no
tests). I would say that it is the most noticeable impact of tests:
smoothing the code contribution and reviewing workflow.

I suspect the benefits of testing may be easier to reap for library-style
projects (adapted to unit-testing) rather than final-executable-project
(with more complex specification and more integration testing to do). But,
in any case, all projects have a library part.

# Batteries' example: qtest/iTeml

A representative of "new code" would be batSubString.ml for example:

https://github.com/ocaml-batteries-team/batteries-included/blob/master/src/batSubstring.ml

As you can see, the tests are placed inside the modules, close to the
implementation. The tool we use for that was initially created by Ilmari
Heikkinen (for his Prelude.ml), lived inside Batteries for a long time, and
was extracted as a separate project by Vincent Hugo, it is now called iTeML
and found at
  https://github.com/vincent-hugot/iTeML

(It is completely independent from Batteries and everyone is encouraged to
have a look.)

The particular way it works is by extracting tests found in comments with a
special marking (typically "(*$T", other modifiers also exist). Vincent
Hugo wrote an excellent documentation:
  https://github.com/vincent-hugot/iTeML#introduction

# Test extraction tools, and testing libraries

Other projects have their own preprocessing tools to manipulate tests. I
don't have exhaustive knowledge of what is out there, but there is Jun
Furuse's ppx_test (on top of the oUnit library). there is pa_ounit (
https://github.com/janestreet/pa_ounit ), and Janestreet also has pa_test
and a PPX successor (also named ppx-test, I would guess). pa_ounit is the
best documented of these projects, as it describes the execution model.

I strongly believe that "text extraction" preprocessors and
"{random,unit,mock,...}testing libraries" are orthogonal components that
should be developped separately and combined together by user.
In practice Batteries uses oUnit for unit testing, and the "quickcheck"
library embedded inside qtest/iteml for random testing, but I've also had
good experiences with Xavier Clerc's Kaputt on other projects (
http://kaputt.x9c.fr/ ).

In practice I would say that, in my experience, for unit testing, the
unit-testing library doesn't actually matter that much: you need some kind
of "test : bool -> ..." function and decent text reporting capabilities
(telling you which tests failed). Simple boolean tests already provide a
lot of benefits to library projects, and the other bells and whistles
(support for automatically printing values, HTML reporting and what not)
may help in particular scenarios but there is a diminishing return much.
(Random-testing is harder to do really well and you will be more heavily
dependent on the library used.)

# Test extraction actually matters

I think that the ability to write *inline* tests, which I initially thought
to be mere syntactic sugar, is actually really important. They help giving
the programmer the (right) idea that the function *comes with* its test,
and improves locality of code modification. When refactoring a function or
whatever: you may already be annoyed by having to modify the .mli
separately, don't have tests cost you a third file to update. Plus reading
the .mli in isolation brings some value, tests not so much.

There are two broad families of "test extraction techniques":
[internal]: compile the module with or without tests; during test run, just
load the compiled-with-tests module
[external]: put the extracted tests in a separate file; the module is
always compiled without tests, and the separate test part is compiled for
test runs

qtest/iTeML is currently using the external model. On paper the internal
one is more powerful, as it allows you to test internal functions not
exported through the interface (external tests have to respect the
abstraction boundary), and facilitates testing functors -- I would assume
that some of the other extraction tools use the internal model. We have
discussed moving iTeML to the internal model in the past, but nobody has
committed to doing the work for now, and I think the benefits for Batteries
specifically wouldn't actually be that big -- but it may be specific to the
fact that the project is *a* library, and thus has much more exposed than
private functions.
(The internal model also raises the not-quite-trivial issues of either
interleaving test-registration side-effects throughout your (otherwise
pure, right?) code, urgh, or trying to have a value-driven approach which
implies non-trivial data-passing plumbing.)

On Thu, Oct 8, 2015 at 9:43 AM, Sébastien Hinderer <
Sebastien.Hinderer@inria.fr> wrote:

> Dear all,
>
> Recently the topic was discussed here and several testing frameworks were
> mentionned.
>
> I am wondering whether there are well established projects that actually
> use tests (unit tests and others) and that could be used as sources of
> inspiration?
>
> Thanks,
>
> Sébastien.
>
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>

[-- Attachment #2: Type: text/html, Size: 7440 bytes --]