"make check" suggestion

"make check" suggestion

[Bart Schaefer]

I haven't messed around with ztst.zsh for some years, so I'm not
familiar with how the newer features like intentionally failed tests
are handled.  A suggestion perhaps for someone who has worked on it
more recently:

It might be nice to have a mode in which ALL the tests in each file
are run, rather than abort the whole test file when one of them fails
(or unexpectedly succeeds).

Re: "make check" suggestion

[Felipe Contreras]

On Sat, Nov 28, 2020 at 12:53 AM Bart Schaefer
<schaefer@brasslantern.com> wrote:
>
> I haven't messed around with ztst.zsh for some years, so I'm not
> familiar with how the newer features like intentionally failed tests
> are handled.  A suggestion perhaps for someone who has worked on it
> more recently:
>
> It might be nice to have a mode in which ALL the tests in each file
> are run, rather than abort the whole test file when one of them fails
> (or unexpectedly succeeds).

Coming from Git, I have to say their testing framework is the best
I've ever seen.

It is developed in shell. There may be some bashisms, but they try to
avoid them. Last time I tried it in zsh, it didn't work correctly, but
shouldn't be too hard to fix (and maybe it was a problem in zsh), and
I'm sure they would accept the patches.

The project that tried to extirpate the test system from git.git is
called sharness [1]. I've used it in all my projects, and it works
like a charm.

In this test system all you have to do is pass -i to the test, to stop
at a failure.

For example:

  ./completion-zsh.t -i

Additionally, the system supports Perl's Test Anything Protocol (TAP) [2].

So you can do:

  prove completion-zsh.t :: -i

That's how I developed the git-completion zsh specific tests, which
work great, and are integrated into Travis-CI [3].

If anyone is interested, I could give it a try to port some zsh test
to sharness.

In my personal opinion the testing system of zsh is far from ideal.
But that's just my opinion.

Cheers.

[1] https://github.com/chriscool/sharness
[2] https://en.wikipedia.org/wiki/Test_Anything_Protocol
[3] https://travis-ci.org/github/felipec/git-completion

-- 
Felipe Contreras

Re: "make check" suggestion

[Daniel Shahaf]

Felipe Contreras wrote on Sat, Nov 28, 2020 at 04:31:46 -0600:
> In my personal opinion the testing system of zsh is far from ideal.
> But that's just my opinion.

I agree that it'd be nice if we didn't have to maintain our own test harness
(as opposed to test cases).

No comment about sharness specifically.