[Caml-list] Use of unit testing frameworks in OCaml

[Caml-list] Use of unit testing frameworks in OCaml

[John Christopher McAlpine]

 From what I can tell, the tests for the OCaml compiler and standard 
libraries are all done with a fairly small framework designed 
specifically for testing the compiler and standard libraries. Is there 
any particular reason not to use a tool such as OUnit or Broken for 
these tests? I am aware that adding dependencies to the compiler is 
generally frowned upon, but these libraries could significantly improve 
the maintainability of the core language and libraries.

Re: [Caml-list] Use of unit testing frameworks in OCaml

[Sébastien Hinderer]

Hi,

John Christopher McAlpine (2015/09/09 14:35 -0600):
> From what I can tell, the tests for the OCaml compiler and standard
> libraries are all done with a fairly small framework designed specifically
> for testing the compiler and standard libraries. Is there any particular
> reason not to use a tool such as OUnit or Broken for these tests?

Could you please provide an URL for Broken?
I couldn't find one...

I'm also wondering whether test frameworks exist for other kinds of
tests?

Thanks,

Sébastien.

Re: [Caml-list] Use of unit testing frameworks in OCaml

[Gabriel Scherer]

Could you be more specific in what kind of benefits you expect such a
migration would bring? I'm possibly not familiar enough with these
frameworks, but it is rather unclear to me what "significant
improvement to the maintenability" we should expect.

Note that many tests in the compiler testsuite are in fact rather
integration tests than unit tests: they test a complete run of the
compiler (or toplevel) on an input program, rather that a single
function of the compiler codebase.

If I had time to devote to improve the compiler testsuite, here are
the two changes I would be interested in implementing (or getting
implemented for me):
- A way to collect coverage checking information (which parts of the
compiler codebase are exercized by the tests?). I started working on
it once by just adding Bisect somewhere in the compiler build system,
but lost motivation before getting into a a work-out-of-the-box state
(it generated many independent coverage files, one for each test, that
would have to be aggregated). I think that should not be too hard.
- A separation between the tests that are expected to be long-running
and the others. Currently running the testsuite takes a few minutes on
my machine. If I could get a subset of the tests that has the same
coverage for changes that do not affect the code generation backend
running in under one minute, that would improve the hack-build-test
feedback loops.
- A way to run in parallel the tests that are not sensitive to
parallel execution (those that do not test performances or thread
scheduling).

All of these might be easier with a more elegant/declarative/structure
test framework, but it is not immediately obvious to me that switching
to oUnit would do the job.

On Wed, Sep 9, 2015 at 10:35 PM, John Christopher McAlpine
<jmcalpin@nmt.edu> wrote:
> From what I can tell, the tests for the OCaml compiler and standard
> libraries are all done with a fairly small framework designed specifically
> for testing the compiler and standard libraries. Is there any particular
> reason not to use a tool such as OUnit or Broken for these tests? I am aware
> that adding dependencies to the compiler is generally frowned upon, but
> these libraries could significantly improve the maintainability of the core
> language and libraries.
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs

Re: [Caml-list] Use of unit testing frameworks in OCaml

[John Christopher McAlpine]

Here you go: https://github.com/michipili/broken

On 09/09/2015 02:55 PM, Sébastien Hinderer wrote:
> Hi,
>
> John Christopher McAlpine (2015/09/09 14:35 -0600):
>>  From what I can tell, the tests for the OCaml compiler and standard
>> libraries are all done with a fairly small framework designed specifically
>> for testing the compiler and standard libraries. Is there any particular
>> reason not to use a tool such as OUnit or Broken for these tests?
> Could you please provide an URL for Broken?
> I couldn't find one...
>
> I'm also wondering whether test frameworks exist for other kinds of
> tests?
>
> Thanks,
>
> Sébastien.
>

Re: [Caml-list] Use of unit testing frameworks in OCaml

[Gerd Stolpmann]

[-- Attachment #1: Type: text/plain, Size: 2187 bytes --]

Am Mittwoch, den 09.09.2015, 14:35 -0600 schrieb John Christopher
McAlpine:
>  From what I can tell, the tests for the OCaml compiler and standard 
> libraries are all done with a fairly small framework designed 
> specifically for testing the compiler and standard libraries. Is there 
> any particular reason not to use a tool such as OUnit or Broken for 
> these tests? I am aware that adding dependencies to the compiler is 
> generally frowned upon, but these libraries could significantly improve 
> the maintainability of the core language and libraries.

The last sentence is a quite strong thesis. Recently, I was in a similar
situation as the OCaml developers, namely the need arose to create unit
tests for a fairly large project (a commercial one, so I cannot point to
it). I was considering oUnit. Finally, I dropped it, because it is not
that useful at all.

The point is that the most important function for unit-testing is
polymorphic equality, and this is a built-in. All the other stuff is
absolutely optional, and if you need it, just a matter of 10-20 lines
(e.g. for selecting unit tests from the command-line - IF this is the
applicable environment). It turns out the most useful are utility
functions like

let ( &&& ) x y = x && y
  (* logical AND without short-circuit eval *)

or

let named_check name p =
  if not p then printf "[FAILED: %s]" name;
  p

which are typically missing in unit testing frameworks because these
were once ported from imperative languages. E.g. a test could now look
like

let test0001() =
  named_check "subexpr1" (f 3 4 5 = 8)
  &&& named_check "subexpr2" (g 6 7 8 = 9)

My experience is that all the other stuff you need is highly special to
the project.

What is the gain of maintainability you get by using a framework?

Gerd
-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
My OCaml site:          http://www.camlcity.org
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Caml-list] Use of unit testing frameworks in OCaml

[Daniel Bünzli]

Le mercredi, 9 septembre 2015 à 23:29, John Christopher McAlpine a écrit :
> Here you go: https://github.com/michipili/broken

Looking at the examples it really seems to be.

Names are already hard to come by, why bother me with having to invent one for *each* assertion ? What you want is to subvert the backtrace mechanism so that failing tests just output a location your IDE can jump to and highlight.  

Also for tests that are more complex than simple assertions I'd like to have a print out of the function arguments that lead to the assertion failure. See for example [1] for a toy testing framework that does that and that I c&p around because I don't have the time to spend a month to give the problem a good solution. This simple thing is already, from a programmer perspective, much better than anything you mentioned — but it's missing the quickcheck and random value generation bits that any self-respecting solution should bring to the table.  

The problem (beyond Gabriel's more sensitive answer) with these all these testing framework is that they seem to be too busy to uncreatively copy the useless crap oo testing framework and terminology rather than provide actual useful tools for programmers.

Best,

Daniel

[1]
https://github.com/dbuenzli/astring/blob/master/test/testing.mli
https://github.com/dbuenzli/astring/blob/master/test/testing.ml

Re: [Caml-list] Use of unit testing frameworks in OCaml

[Raoul Duke]

> The problem (beyond Gabriel's more sensitive answer) with these all these testing framework is that they seem to be too busy to uncreatively copy the useless crap oo testing framework and terminology rather than provide actual useful tools for programmers.

hell yes. amen. always made/makes me mad, myself. testify! +1! i'd
vote for you. wish i had a successful company so i could try to hire
you away. w00t. thank got i'm not the only crazy one. are you married?

Re: [Caml-list] Use of unit testing frameworks in OCaml

[Hendrik Boom]

On Wed, Sep 09, 2015 at 10:55:48PM +0200, Sébastien Hinderer wrote:
> Hi,
> 
> John Christopher McAlpine (2015/09/09 14:35 -0600):
> > From what I can tell, the tests for the OCaml compiler and standard
> > libraries are all done with a fairly small framework designed specifically
> > for testing the compiler and standard libraries. Is there any particular
> > reason not to use a tool such as OUnit or Broken for these tests?
> 
> Could you please provide an URL for Broken?
> I couldn't find one...
> 
> I'm also wondering whether test frameworks exist for other kinds of
> tests?

The Algol W compiler was written in OCaml, and its test drivr was also 
written in OCaml. 

Of interest?

-- hendrik

Re: [Caml-list] Use of unit testing frameworks in OCaml

[Alan Schmitt]

[-- Attachment #1: Type: text/plain, Size: 935 bytes --]

Hi Gabriel,

On 2015-09-09 23:29, Gabriel Scherer <gabriel.scherer@gmail.com> writes:

> - A way to collect coverage checking information (which parts of the
> compiler codebase are exercized by the tests?). I started working on
> it once by just adding Bisect somewhere in the compiler build system,
> but lost motivation before getting into a a work-out-of-the-box state
> (it generated many independent coverage files, one for each test, that
> would have to be aggregated). I think that should not be too hard.

We use bisect to test coverage of the JavaScript test suite on our
interpreter, and aggregating the results is simply a matter of doing
  bisect-report -html report bisect*.out
(assuming you want to put the generated report in the report directory).

Best,

Alan

-- 
OpenPGP Key ID : 040D0A3B4ED2E5C7
Last week athmospheric CO₂ average (Updated September 6, 2015, Mauna Loa Obs.):
377.86 ppm

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 472 bytes --]

Re: [Caml-list] Use of unit testing frameworks in OCaml

[Fabrice Le Fessant]

One reason to avoid OCaml tests framworks for the compiler is probably
that, if the compiler is broken, we still want to be able to run the
testsuite, to break down the problem to the smallest failing test. If
the test framework is written in OCaml and compiled by the failing
compiler, it might not be able to run the tests at all. Maybe a
solution would be to use a tests framework written for OCaml... but in
another language :-) (which is currently the case, it is written in
Makefiles...)

--Fabrice

On Wed, Sep 9, 2015 at 10:35 PM, John Christopher McAlpine
<jmcalpin@nmt.edu> wrote:
> From what I can tell, the tests for the OCaml compiler and standard
> libraries are all done with a fairly small framework designed specifically
> for testing the compiler and standard libraries. Is there any particular
> reason not to use a tool such as OUnit or Broken for these tests? I am aware
> that adding dependencies to the compiler is generally frowned upon, but
> these libraries could significantly improve the maintainability of the core
> language and libraries.
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs



-- 
Fabrice LE FESSANT
Chercheur en Informatique
INRIA Paris Rocquencourt -- OCamlPro
Programming Languages and Distributed Systems