From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: caml-list@sympa.inria.fr Delivered-To: caml-list@sympa.inria.fr Received: from mail3-relais-sop.national.inria.fr (mail3-relais-sop.national.inria.fr [192.134.164.104]) by sympa.inria.fr (Postfix) with ESMTPS id 4560C7FC6B for ; Thu, 8 Oct 2015 10:23:00 +0200 (CEST) IronPort-PHdr: 9a23:b1It9BeX0Vu4jtRCi9oWUHcclGMj4u6mDksu8pMizoh2WeGdxc++Zx7h7PlgxGXEQZ/co6odzbGG7+a/BCQp2tWojjMrSNR0TRgLiMEbzUQLIfWuLgnFFsPsdDEwB89YVVVorDmROElRH9viNRWJ+iXhpQAbFhi3DwdpPOO9QteU1JTrkb/vs7ToICx2xxOFKYtoKxu3qQiD/uI3uqBFbpgL9x3Sv3FTcP5Xz247bXianhL7+9vitMU7q3cYk7sb+sVBSaT3ebgjBfwdVWx+cjMD39DwrRTIUSeI43IdVC1WzksJUED560TB3534qTf7u+xK+CicMcDsQKp8DQ+v5a5wVB7ljmEnNjg1/XvakORxirhaqVSvvUo7i4XdZYXdKeFzZLiVKdgTQG4EWsdKSwRABJm9Zs0BFbxSE/xfqtzSrlEUrBa6TTKnBO71xyUA0nD/17c73uBnCgrG0RYtBfoBtX3VqJP+M6JEArP997XB0TiWN6Ae4jz68oWdN014rA== Authentication-Results: mail3-smtp-sop.national.inria.fr; spf=None smtp.pra=gabriel.scherer@gmail.com; spf=Pass smtp.mailfrom=gabriel.scherer@gmail.com; spf=None smtp.helo=postmaster@mail-io0-f179.google.com Received-SPF: None (mail3-smtp-sop.national.inria.fr: no sender authenticity information available from domain of gabriel.scherer@gmail.com) identity=pra; client-ip=209.85.223.179; receiver=mail3-smtp-sop.national.inria.fr; envelope-from="gabriel.scherer@gmail.com"; x-sender="gabriel.scherer@gmail.com"; x-conformance=sidf_compatible Received-SPF: Pass (mail3-smtp-sop.national.inria.fr: domain of gabriel.scherer@gmail.com designates 209.85.223.179 as permitted sender) identity=mailfrom; client-ip=209.85.223.179; receiver=mail3-smtp-sop.national.inria.fr; envelope-from="gabriel.scherer@gmail.com"; x-sender="gabriel.scherer@gmail.com"; x-conformance=sidf_compatible; x-record-type="v=spf1" Received-SPF: None (mail3-smtp-sop.national.inria.fr: no sender authenticity information available from domain of postmaster@mail-io0-f179.google.com) identity=helo; client-ip=209.85.223.179; receiver=mail3-smtp-sop.national.inria.fr; envelope-from="gabriel.scherer@gmail.com"; x-sender="postmaster@mail-io0-f179.google.com"; x-conformance=sidf_compatible X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A0B/AwC3JxZWm7PfVdFdg0Y1bgasE4UjjXAjgnCBL1t/AoE7BzwQAQEBAQEBAQEQAQEBAQEGCwsJIS6CH4IIAQEDARIRBBkBGxIMAwELBgULGh0CAiIBEQEFAQoSBgESEgcJh3YBAwoIDaAwgTE+MYtJgWyCeYlQChknAwpWhE8BCgEBAQEXAQUOhmWEfoJugTZlC4JpgUUFhX8MgSqHBYdOhRiHf4FXSIcViXyEWYIjEiOBFxEnggskI4FdPDOHbAEBAQ X-IPAS-Result: A0B/AwC3JxZWm7PfVdFdg0Y1bgasE4UjjXAjgnCBL1t/AoE7BzwQAQEBAQEBAQEQAQEBAQEGCwsJIS6CH4IIAQEDARIRBBkBGxIMAwELBgULGh0CAiIBEQEFAQoSBgESEgcJh3YBAwoIDaAwgTE+MYtJgWyCeYlQChknAwpWhE8BCgEBAQEXAQUOhmWEfoJugTZlC4JpgUUFhX8MgSqHBYdOhRiHf4FXSIcViXyEWYIjEiOBFxEnggskI4FdPDOHbAEBAQ X-IronPort-AV: E=Sophos;i="5.17,654,1437429600"; d="scan'208";a="149955784" Received: from mail-io0-f179.google.com ([209.85.223.179]) by mail3-smtp-sop.national.inria.fr with ESMTP/TLS/AES128-GCM-SHA256; 08 Oct 2015 10:22:28 +0200 Received: by ioii196 with SMTP id i196so49706607ioi.3; Thu, 08 Oct 2015 01:22:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=diVT1JWXoqTjT0+iTu7UuHqO9wPM0P6mT9MoJZk/vG0=; b=S1FHK8enYtHSMUBANLmok21bPk6kNuA0qAbxuznI6D4oAoOzhR+e07RQlU4pOI3u50 S7r0SkvlN+3ZA+Mq7F9zKQyhKWGUCg6oGoir/qLvKM2Vr+h7yvb4lKjioft7zAOI3k+s dQlNpF+YBMgX3nPOu6siCn7UAw5YC1/1mpz83MdOHKtTwawzRcmqnerU1ezB+QImRx85 waxPEGZk31p/wIJhakaPE6FLhkeFqmyj7pCzqAw/ED+E33AdsQZkJd3xLn3hMqvbdtYz TahzK2HU1+Tit8yWLGr5oBYJq3QgTNZE6Bj2udrAlXl1eALfiAIG1XD8qe3NW7HHdV9q 7QRQ== X-Received: by 10.107.19.70 with SMTP id b67mr6610981ioj.144.1444292547061; Thu, 08 Oct 2015 01:22:27 -0700 (PDT) MIME-Version: 1.0 Received: by 10.79.19.68 with HTTP; Thu, 8 Oct 2015 01:21:47 -0700 (PDT) In-Reply-To: <20151008074315.GA1466@pl-59055.rocqadm.inria.fr> References: <20151008074315.GA1466@pl-59055.rocqadm.inria.fr> From: Gabriel Scherer Date: Thu, 8 Oct 2015 10:21:47 +0200 Message-ID: To: =?UTF-8?Q?S=C3=A9bastien_Hinderer?= , caml users Content-Type: multipart/alternative; boundary=001a113dea22382d19052193900c Subject: Re: [Caml-list] OCaml projects with tests --001a113dea22382d19052193900c Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Batteries uses a lot of tests, and we couldn't live without them. In particularly, they make maintainer's life much easier (it's easier to make the decision to integrate code when, besides being nice to read, it is also well-tested), and that is key in a project with many small contributions, so a lot of integration work. It is mandatory for new functions proposed for inclusions to come with unit tests, which means that new code has more tests than old code (inherited from Extlib, with no tests). I would say that it is the most noticeable impact of tests: smoothing the code contribution and reviewing workflow. I suspect the benefits of testing may be easier to reap for library-style projects (adapted to unit-testing) rather than final-executable-project (with more complex specification and more integration testing to do). But, in any case, all projects have a library part. # Batteries' example: qtest/iTeml A representative of "new code" would be batSubString.ml for example: https://github.com/ocaml-batteries-team/batteries-included/blob/master/src/= batSubstring.ml As you can see, the tests are placed inside the modules, close to the implementation. The tool we use for that was initially created by Ilmari Heikkinen (for his Prelude.ml), lived inside Batteries for a long time, and was extracted as a separate project by Vincent Hugo, it is now called iTeML and found at https://github.com/vincent-hugot/iTeML (It is completely independent from Batteries and everyone is encouraged to have a look.) The particular way it works is by extracting tests found in comments with a special marking (typically "(*$T", other modifiers also exist). Vincent Hugo wrote an excellent documentation: https://github.com/vincent-hugot/iTeML#introduction # Test extraction tools, and testing libraries Other projects have their own preprocessing tools to manipulate tests. I don't have exhaustive knowledge of what is out there, but there is Jun Furuse's ppx_test (on top of the oUnit library). there is pa_ounit ( https://github.com/janestreet/pa_ounit ), and Janestreet also has pa_test and a PPX successor (also named ppx-test, I would guess). pa_ounit is the best documented of these projects, as it describes the execution model. I strongly believe that "text extraction" preprocessors and "{random,unit,mock,...}testing libraries" are orthogonal components that should be developped separately and combined together by user. In practice Batteries uses oUnit for unit testing, and the "quickcheck" library embedded inside qtest/iteml for random testing, but I've also had good experiences with Xavier Clerc's Kaputt on other projects ( http://kaputt.x9c.fr/ ). In practice I would say that, in my experience, for unit testing, the unit-testing library doesn't actually matter that much: you need some kind of "test : bool -> ..." function and decent text reporting capabilities (telling you which tests failed). Simple boolean tests already provide a lot of benefits to library projects, and the other bells and whistles (support for automatically printing values, HTML reporting and what not) may help in particular scenarios but there is a diminishing return much. (Random-testing is harder to do really well and you will be more heavily dependent on the library used.) # Test extraction actually matters I think that the ability to write *inline* tests, which I initially thought to be mere syntactic sugar, is actually really important. They help giving the programmer the (right) idea that the function *comes with* its test, and improves locality of code modification. When refactoring a function or whatever: you may already be annoyed by having to modify the .mli separately, don't have tests cost you a third file to update. Plus reading the .mli in isolation brings some value, tests not so much. There are two broad families of "test extraction techniques": [internal]: compile the module with or without tests; during test run, just load the compiled-with-tests module [external]: put the extracted tests in a separate file; the module is always compiled without tests, and the separate test part is compiled for test runs qtest/iTeML is currently using the external model. On paper the internal one is more powerful, as it allows you to test internal functions not exported through the interface (external tests have to respect the abstraction boundary), and facilitates testing functors -- I would assume that some of the other extraction tools use the internal model. We have discussed moving iTeML to the internal model in the past, but nobody has committed to doing the work for now, and I think the benefits for Batteries specifically wouldn't actually be that big -- but it may be specific to the fact that the project is *a* library, and thus has much more exposed than private functions. (The internal model also raises the not-quite-trivial issues of either interleaving test-registration side-effects throughout your (otherwise pure, right?) code, urgh, or trying to have a value-driven approach which implies non-trivial data-passing plumbing.) On Thu, Oct 8, 2015 at 9:43 AM, S=C3=A9bastien Hinderer < Sebastien.Hinderer@inria.fr> wrote: > Dear all, > > Recently the topic was discussed here and several testing frameworks were > mentionned. > > I am wondering whether there are well established projects that actually > use tests (unit tests and others) and that could be used as sources of > inspiration? > > Thanks, > > S=C3=A9bastien. > > > -- > Caml-list mailing list. Subscription management and archives: > https://sympa.inria.fr/sympa/arc/caml-list > Beginner's list: http://groups.yahoo.com/group/ocaml_beginners > Bug reports: http://caml.inria.fr/bin/caml-bugs > --001a113dea22382d19052193900c Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Batteries uses a lot of tests, and we couldn't li= ve without them.

In particularly, they make maintainer's life mu= ch easier (it's easier to make the decision to integrate code when, bes= ides being nice to read, it is also well-tested), and that is key in a proj= ect with many small contributions, so a lot of integration work. It is mand= atory for new functions proposed for inclusions to come with unit tests, wh= ich means that new code has more tests than old code (inherited from Extlib= , with no tests). I would say that it is the most noticeable impact of test= s: smoothing the code contribution and reviewing workflow.

I suspect the benefits of testing may be easier to reap for library-style= projects (adapted to unit-testing) rather than final-executable-project (w= ith more complex specification and more integration testing to do). But, in= any case, all projects have a library part.

#= Batteries' example: qtest/iTeml

A representative of = "new code" would be batSubString.ml for example:
=C2=A0 https://github.com/ocaml-batteries-= team/batteries-included/blob/master/src/batSubstring.ml

As you can see, the tests are placed inside the modules, close to the im= plementation. The tool we use for that was initially created by Ilmari Heik= kinen (for his Prelude.ml), lived inside Batteries for a long time, and was= extracted as a separate project by Vincent Hugo, it is now called iTeML an= d found at
=C2=A0 https://github.com/vincent-hugot/iTeML

(It is completely independent from Batteries and everyone is encouraged to= have a look.)

The particular way it works is by extracti= ng tests found in comments with a special marking (typically "(*$T&quo= t;, other modifiers also exist). Vincent Hugo wrote an excellent documentat= ion:
=C2=A0 https://github.com/vincent-hugot/iTeML#introduction

# Test extraction tools, and testing libraries

Other projects have their own preprocessing tools to = manipulate tests. I don't have exhaustive knowledge of what is out ther= e, but there is Jun Furuse's ppx_test (on top of the oUnit library). th= ere is pa_ounit ( https://github.com/janestreet/pa_ounit ), and Janestreet al= so has pa_test and a PPX successor (also named ppx-test, I would guess). pa= _ounit is the best documented of these projects, as it describes the execut= ion model.

I strongly believe that "text extraction" preprocessors = and "{random,unit,mock,...}testing libraries" are orthogonal comp= onents that should be developped separately and combined together by user.<= br>In practice Batteries uses oUnit for unit testing, and the "quickch= eck" library embedded inside qtest/iteml for random testing, but I'= ;ve also had good experiences with Xavier Clerc's Kaputt on other proje= cts ( http://kaputt.x9c= .fr/ ).

In practice I would say that, in my experience, for unit= testing, the unit-testing library doesn't actually matter that much: y= ou need some kind of "test : bool -> ..." function and decent = text reporting capabilities (telling you which tests failed). Simple boolea= n tests already provide a lot of benefits to library projects, and the othe= r bells and whistles (support for automatically printing values, HTML repor= ting and what not) may help in particular scenarios but there is a diminish= ing return much. (Random-testing is harder to do really well and you will b= e more heavily dependent on the library used.)

# Test extraction actually matters

I think that the ability to wr= ite *inline* tests, which I initially thought to be mere syntactic sugar, i= s actually really important. They help giving the programmer the (right) id= ea that the function *comes with* its test, and improves locality of code m= odification. When refactoring a function or whatever: you may already be an= noyed by having to modify the .mli separately, don't have tests cost yo= u a third file to update. Plus reading the .mli in isolation brings some va= lue, tests not so much.

There are two bro= ad families of "test extraction techniques":
[internal]: compile the module with or without tests; duri= ng test run, just load the compiled-with-tests module
[external]: put the extracted tests in a separate file; the module is always=20 compiled without tests, and the separate test part is compiled for test=20 runs

qtest/iTeML is currently=20 using the external model. On paper the internal one is more powerful, as it allows you to test internal functions not exported through the=20 interface (external tests have to respect the abstraction boundary), and facilitates testing functors -- I would assume that some of the other=20 extraction tools use the internal model. We have discussed moving iTeML=20 to the internal model in the past, but nobody has committed to doing the work for now, and I think the benefits for Batteries specifically=20 wouldn't actually be that big -- but it may be specific to the fact that the project is *a* library, and thus has much more exposed than private functions.
(The internal model also raises the not-quite-trivial issues= of either interleaving test-registration side-effects throughout your (otherwise pure, right?) code, urgh, or trying to have a value-driven approach which implies non-trivial data-passing plumbing.)

On Thu, Oc= t 8, 2015 at 9:43 AM, S=C3=A9bastien Hinderer <Sebastien.Hindere= r@inria.fr> wrote:
Dear all,

Recently the topic was discussed here and several testing frameworks were mentionned.

I am wondering whether there are well established projects that actually
use tests (unit tests and others) and that could be used as sources of
inspiration?

Thanks,

S=C3=A9bastien.


--
Caml-list mailing list.=C2=A0 Subscription management and archives:
https://sympa.inria.fr/sympa/arc/caml-list
Beginner's list: http://groups.yahoo.com/group/ocam= l_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs

--001a113dea22382d19052193900c--