Batteries uses a lot of tests, and we couldn't live without them.

In particularly, they make maintainer's life much easier (it's easier to make the decision to integrate code when, besides being nice to read, it is also well-tested), and that is key in a project with many small contributions, so a lot of integration work. It is mandatory for new functions proposed for inclusions to come with unit tests, which means that new code has more tests than old code (inherited from Extlib, with no tests). I would say that it is the most noticeable impact of tests: smoothing the code contribution and reviewing workflow.

I suspect the benefits of testing may be easier to reap for library-style projects (adapted to unit-testing) rather than final-executable-project (with more complex specification and more integration testing to do). But, in any case, all projects have a library part.

# Batteries' example: qtest/iTeml

A representative of "new code" would be batSubString.ml for example:
https://github.com/ocaml-batteries-team/batteries-included/blob/master/src/batSubstring.ml

As you can see, the tests are placed inside the modules, close to the implementation. The tool we use for that was initially created by Ilmari Heikkinen (for his Prelude.ml), lived inside Batteries for a long time, and was extracted as a separate project by Vincent Hugo, it is now called iTeML and found at
https://github.com/vincent-hugot/iTeML

(It is completely independent from Batteries and everyone is encouraged to have a look.)

The particular way it works is by extracting tests found in comments with a special marking (typically "(*$T", other modifiers also exist). Vincent Hugo wrote an excellent documentation:
https://github.com/vincent-hugot/iTeML#introduction

# Test extraction tools, and testing libraries

Other projects have their own preprocessing tools to manipulate tests. I don't have exhaustive knowledge of what is out there, but there is Jun Furuse's ppx_test (on top of the oUnit library). there is pa_ounit ( https://github.com/janestreet/pa_ounit ), and Janestreet also has pa_test and a PPX successor (also named ppx-test, I would guess). pa_ounit is the best documented of these projects, as it describes the execution model.

I strongly believe that "text extraction" preprocessors and "{random,unit,mock,...}testing libraries" are orthogonal components that should be developped separately and combined together by user.
In practice Batteries uses oUnit for unit testing, and the "quickcheck" library embedded inside qtest/iteml for random testing, but I've also had good experiences with Xavier Clerc's Kaputt on other projects ( http://kaputt.x9c.fr/ ).

In practice I would say that, in my experience, for unit testing, the unit-testing library doesn't actually matter that much: you need some kind of "test : bool -> ..." function and decent text reporting capabilities (telling you which tests failed). Simple boolean tests already provide a lot of benefits to library projects, and the other bells and whistles (support for automatically printing values, HTML reporting and what not) may help in particular scenarios but there is a diminishing return much. (Random-testing is harder to do really well and you will be more heavily dependent on the library used.)

# Test extraction actually matters

I think that the ability to write *inline* tests, which I initially thought to be mere syntactic sugar, is actually really important. They help giving the programmer the (right) idea that the function *comes with* its test, and improves locality of code modification. When refactoring a function or whatever: you may already be annoyed by having to modify the .mli separately, don't have tests cost you a third file to update. Plus reading the .mli in isolation brings some value, tests not so much.

There are two broad families of "test extraction techniques":

[internal]: compile the module with or without tests; during test run, just load the compiled-with-tests module

[external]: put the extracted tests in a separate file; the module is always compiled without tests, and the separate test part is compiled for test runs

qtest/iTeML is currently using the external model. On paper the internal one is more powerful, as it allows you to test internal functions not exported through the interface (external tests have to respect the abstraction boundary), and facilitates testing functors -- I would assume that some of the other extraction tools use the internal model. We have discussed moving iTeML to the internal model in the past, but nobody has committed to doing the work for now, and I think the benefits for Batteries specifically wouldn't actually be that big -- but it may be specific to the fact that the project is *a* library, and thus has much more exposed than private functions.
(The internal model also raises the not-quite-trivial issues of either interleaving test-registration side-effects throughout your (otherwise pure, right?) code, urgh, or trying to have a value-driven approach which implies non-trivial data-passing plumbing.)

On Thu, Oct 8, 2015 at 9:43 AM, Sébastien Hinderer <Sebastien.Hinderer@inria.fr> wrote:

Dear all,

Recently the topic was discussed here and several testing frameworks were
mentionned.

I am wondering whether there are well established projects that actually
use tests (unit tests and others) and that could be used as sources of
inspiration?

Thanks,

Sébastien.

--
Caml-list mailing list. Subscription management and archives:
https://sympa.inria.fr/sympa/arc/caml-list
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs