[Caml-list] OCaml projects with tests

[Caml-list] OCaml projects with tests

[Sébastien Hinderer]

Dear all,

Recently the topic was discussed here and several testing frameworks were
mentionned.

I am wondering whether there are well established projects that actually
use tests (unit tests and others) and that could be used as sources of
inspiration?

Thanks,

Sébastien.

Re: [Caml-list] OCaml projects with tests

[Francois Berenger]

On 10/08/2015 09:43 AM, Sébastien Hinderer wrote:
> Dear all,
>
> Recently the topic was discussed here and several testing frameworks were
> mentionned.
>
> I am wondering whether there are well established projects that actually
> use tests (unit tests and others) and that could be used as sources of
> inspiration?

In batteries there are:

https://raw.githubusercontent.com/ocaml-batteries-team/\
batteries-included/master/src/batList.mlv

Look for $T in the file for unit tests.
Or $Q for tests a la Haskell quickcheck.

> Thanks,
>
> Sébastien.
>
>

-- 
Regards,
Francois.
"When in doubt, use more types"

Re: [Caml-list] OCaml projects with tests

[Gabriel Scherer]

[-- Attachment #1: Type: text/plain, Size: 5838 bytes --]

Batteries uses a lot of tests, and we couldn't live without them.

In particularly, they make maintainer's life much easier (it's easier to
make the decision to integrate code when, besides being nice to read, it is
also well-tested), and that is key in a project with many small
contributions, so a lot of integration work. It is mandatory for new
functions proposed for inclusions to come with unit tests, which means that
new code has more tests than old code (inherited from Extlib, with no
tests). I would say that it is the most noticeable impact of tests:
smoothing the code contribution and reviewing workflow.

I suspect the benefits of testing may be easier to reap for library-style
projects (adapted to unit-testing) rather than final-executable-project
(with more complex specification and more integration testing to do). But,
in any case, all projects have a library part.

# Batteries' example: qtest/iTeml

A representative of "new code" would be batSubString.ml for example:

https://github.com/ocaml-batteries-team/batteries-included/blob/master/src/batSubstring.ml

As you can see, the tests are placed inside the modules, close to the
implementation. The tool we use for that was initially created by Ilmari
Heikkinen (for his Prelude.ml), lived inside Batteries for a long time, and
was extracted as a separate project by Vincent Hugo, it is now called iTeML
and found at
  https://github.com/vincent-hugot/iTeML

(It is completely independent from Batteries and everyone is encouraged to
have a look.)

The particular way it works is by extracting tests found in comments with a
special marking (typically "(*$T", other modifiers also exist). Vincent
Hugo wrote an excellent documentation:
  https://github.com/vincent-hugot/iTeML#introduction

# Test extraction tools, and testing libraries

Other projects have their own preprocessing tools to manipulate tests. I
don't have exhaustive knowledge of what is out there, but there is Jun
Furuse's ppx_test (on top of the oUnit library). there is pa_ounit (
https://github.com/janestreet/pa_ounit ), and Janestreet also has pa_test
and a PPX successor (also named ppx-test, I would guess). pa_ounit is the
best documented of these projects, as it describes the execution model.

I strongly believe that "text extraction" preprocessors and
"{random,unit,mock,...}testing libraries" are orthogonal components that
should be developped separately and combined together by user.
In practice Batteries uses oUnit for unit testing, and the "quickcheck"
library embedded inside qtest/iteml for random testing, but I've also had
good experiences with Xavier Clerc's Kaputt on other projects (
http://kaputt.x9c.fr/ ).

In practice I would say that, in my experience, for unit testing, the
unit-testing library doesn't actually matter that much: you need some kind
of "test : bool -> ..." function and decent text reporting capabilities
(telling you which tests failed). Simple boolean tests already provide a
lot of benefits to library projects, and the other bells and whistles
(support for automatically printing values, HTML reporting and what not)
may help in particular scenarios but there is a diminishing return much.
(Random-testing is harder to do really well and you will be more heavily
dependent on the library used.)

# Test extraction actually matters

I think that the ability to write *inline* tests, which I initially thought
to be mere syntactic sugar, is actually really important. They help giving
the programmer the (right) idea that the function *comes with* its test,
and improves locality of code modification. When refactoring a function or
whatever: you may already be annoyed by having to modify the .mli
separately, don't have tests cost you a third file to update. Plus reading
the .mli in isolation brings some value, tests not so much.

There are two broad families of "test extraction techniques":
[internal]: compile the module with or without tests; during test run, just
load the compiled-with-tests module
[external]: put the extracted tests in a separate file; the module is
always compiled without tests, and the separate test part is compiled for
test runs

qtest/iTeML is currently using the external model. On paper the internal
one is more powerful, as it allows you to test internal functions not
exported through the interface (external tests have to respect the
abstraction boundary), and facilitates testing functors -- I would assume
that some of the other extraction tools use the internal model. We have
discussed moving iTeML to the internal model in the past, but nobody has
committed to doing the work for now, and I think the benefits for Batteries
specifically wouldn't actually be that big -- but it may be specific to the
fact that the project is *a* library, and thus has much more exposed than
private functions.
(The internal model also raises the not-quite-trivial issues of either
interleaving test-registration side-effects throughout your (otherwise
pure, right?) code, urgh, or trying to have a value-driven approach which
implies non-trivial data-passing plumbing.)

On Thu, Oct 8, 2015 at 9:43 AM, Sébastien Hinderer <
Sebastien.Hinderer@inria.fr> wrote:

> Dear all,
>
> Recently the topic was discussed here and several testing frameworks were
> mentionned.
>
> I am wondering whether there are well established projects that actually
> use tests (unit tests and others) and that could be used as sources of
> inspiration?
>
> Thanks,
>
> Sébastien.
>
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>

[-- Attachment #2: Type: text/html, Size: 7440 bytes --]

Re: [Caml-list] OCaml projects with tests

[Hendrik Boom]

On Thu, Oct 08, 2015 at 10:21:47AM +0200, Gabriel Scherer wrote:

> qtest/iTeML is currently using the external model. On paper the internal
> one is more powerful, as it allows you to test internal functions not
> exported through the interface (external tests have to respect the
> abstraction boundary), and facilitates testing functors -- I would assume
> that some of the other extraction tools use the internal model. We have
> discussed moving iTeML to the internal model in the past, but nobody has
> committed to doing the work for now, and I think the benefits for Batteries
> specifically wouldn't actually be that big -- but it may be specific to the
> fact that the project is *a* library, and thus has much more exposed than
> private functions.
> (The internal model also raises the not-quite-trivial issues of either
> interleaving test-registration side-effects throughout your (otherwise
> pure, right?) code, urgh, or trying to have a value-driven approach which
> implies non-trivial data-passing plumbing.)

You need both internal and external testing -- internal testing cannot 
test whether you have exported the right names.  Both need to be 
interleaved with the code for maintainability, prefereably using the 
same code extraction tool.

Ideally the test notations should be part of the programming language.  
The test driver should interface with the test code in a well-defined 
way, because different projects will require different test tracking and 
reporting.

-- hendrik
k

Re: [Caml-list] OCaml projects with tests

[Jeremie Dimino]

On Thu, Oct 8, 2015 at 9:21 AM, Gabriel Scherer
<gabriel.scherer@gmail.com> wrote:
> Other projects have their own preprocessing tools to manipulate tests. I
> don't have exhaustive knowledge of what is out there, but there is Jun
> Furuse's ppx_test (on top of the oUnit library). there is pa_ounit (
> https://github.com/janestreet/pa_ounit ), and Janestreet also has pa_test
> and a PPX successor (also named ppx-test, I would guess). pa_ounit is the
> best documented of these projects, as it describes the execution model.

For the pointers:

- the ppx replacement of pa_ounit is ppx_inline_test [1]. We dropped
the "ounit" as it's not using ounit at all
- the ppx replacement of pa_test is ppx_assert [2]. ppx_test was
already taked and was a bit too generic anyway

 [1] https://github.com/janestreet/ppx_inline_test
 [2] https://github.com/janestreet/ppx_assert

-- 
Jeremie

Re: [Caml-list] OCaml projects with tests

[Hendrik Boom]

On Thu, Oct 08, 2015 at 09:43:15AM +0200, Sébastien Hinderer wrote:
> Dear all,
> 
> Recently the topic was discussed here and several testing frameworks were
> mentionned.
> 
> I am wondering whether there are well established projects that actually
> use tests (unit tests and others) and that could be used as sources of
> inspiration?

There's the "awe" compiler for Algol W.  It's written in OCaml, and is 
available as a Debian package, so it should be possible to get the 
source package.  I don't know where the upstrream source is right now, 
as I haven't been able to reach the author in years.

-- hendrik

> 
> Thanks,
> 
> Sébastien.
> 
> 
> -- 
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs

Re: [Caml-list] OCaml projects with tests

[Leonid Rozenberg]

[-- Attachment #1: Type: text/plain, Size: 1009 bytes --]

I've just written about it here
http://www.hammerlab.org/2015/09/30/testing-oml/
I think my review of the testing frameworks might be a bit dated, since I
made the decision
pre "ppx-testing" revolution ala Jane Street. But I've used Xavier Clerc's
Kaputt library extensively.

This is the link to the library https://github.com/hammerlab/oml

Leonid

On Thu, Oct 8, 2015 at 3:43 AM, Sébastien Hinderer <
Sebastien.Hinderer@inria.fr> wrote:

> Dear all,
>
> Recently the topic was discussed here and several testing frameworks were
> mentionned.
>
> I am wondering whether there are well established projects that actually
> use tests (unit tests and others) and that could be used as sources of
> inspiration?
>
> Thanks,
>
> Sébastien.
>
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>

[-- Attachment #2: Type: text/html, Size: 1851 bytes --]

Re: [Caml-list] OCaml projects with tests

[Thomas Gazagnaire]

[-- Attachment #1: Type: text/plain, Size: 1571 bytes --]

Hi Sebastien!

If you like functors, you can also check:

https://github.com/mirage/irmin/blob/master/lib_test/test_store.ml <https://github.com/mirage/irmin/blob/master/lib_test/test_store.ml>

Thomas


> On 8 Oct 2015, at 16:04, Leonid Rozenberg <leonidr@gmail.com> wrote:
> 
> I've just written about it here http://www.hammerlab.org/2015/09/30/testing-oml/ <http://www.hammerlab.org/2015/09/30/testing-oml/>
> I think my review of the testing frameworks might be a bit dated, since I made the decision
> pre "ppx-testing" revolution ala Jane Street. But I've used Xavier Clerc's Kaputt library extensively.
> 
> This is the link to the library https://github.com/hammerlab/oml <https://github.com/hammerlab/oml>
> 
> Leonid
> 
> On Thu, Oct 8, 2015 at 3:43 AM, Sébastien Hinderer <Sebastien.Hinderer@inria.fr <mailto:Sebastien.Hinderer@inria.fr>> wrote:
> Dear all,
> 
> Recently the topic was discussed here and several testing frameworks were
> mentionned.
> 
> I am wondering whether there are well established projects that actually
> use tests (unit tests and others) and that could be used as sources of
> inspiration?
> 
> Thanks,
> 
> Sébastien.
> 
> 
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list <https://sympa.inria.fr/sympa/arc/caml-list>
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners <http://groups.yahoo.com/group/ocaml_beginners>
> Bug reports: http://caml.inria.fr/bin/caml-bugs <http://caml.inria.fr/bin/caml-bugs>
> 


[-- Attachment #2: Type: text/html, Size: 3139 bytes --]

Re: [Caml-list] OCaml projects with tests

[Richard W.M. Jones]

On Thu, Oct 08, 2015 at 09:43:15AM +0200, Sébastien Hinderer wrote:
> Dear all,
> 
> Recently the topic was discussed here and several testing frameworks were
> mentionned.
> 
> I am wondering whether there are well established projects that actually
> use tests (unit tests and others) and that could be used as sources of
> inspiration?

libguestfs tools have extensive tests.

We have a few unit tests, using OUnit2.  An example:

https://github.com/libguestfs/libguestfs/blob/master/v2v/v2v_unit_tests.ml

But the vast majority of the upstream tests use automake's (terrible)
test framework, eg:

https://github.com/libguestfs/libguestfs/blob/master/v2v/Makefile.am#L239
https://github.com/libguestfs/libguestfs/blob/master/builder/Makefile.am#L224
https://github.com/libguestfs/libguestfs/blob/master/resize/Makefile.am#L130

We also have external tests that are non-free so can't be distributed
with the upstream sources:

http://git.annexia.org/?p=virt-v2v-test-cases-nonfree.git;a=summary

We also have a public CI system:

https://ci.centos.org/view/libguestfs/

And finally, we use some proprietary software called Polarion to track
a massive set of tests (many run manually) that control what gets into RHEL.
Those tests aren't published at all.

Rich.

-- 
Richard Jones
Red Hat

Re: [Caml-list] OCaml projects with tests

[Richard W.M. Jones]

How could I forget the deployment system, also written in OCaml ?!

When a release is made, I use 'goaljobs':

  http://git.annexia.org/?p=goaljobs.git;a=summary

to ensure the tarball gets built, tested, uploaded to the website and
built in Fedora.  The goals for this are here:

  http://git.annexia.org/?p=goals.git;a=blob;f=libguestfs_upstream.ml
  http://git.annexia.org/?p=goals.git;a=blob;f=libguestfs_fedora.ml

Rich.

-- 
Richard Jones
Red Hat