Re: Re: Daily reports: Wednesday

[Szabolcs Nagy <nsz@port70.net>]

* Luka Mar??eti?? <paxcoder@gmail.com> [2011-07-06 22:28:14 +0200]:
> As for the future, I'm planing on making the cluts.c framework, and
> this is what I intend for it to do:
> 
> * find 'test collections' using dirent.h (haven't used it yet, but
> shouldn't be hard to learn I guess)
> * fork off for each test collection and execl() each one
> * simple statistics based on the status provided by wait()
> 

my opinion:

don't overengineer the test system

(i have ideas about it, i think i'll
implement them soon, but i'll be
offline for a week)

as a first round it's enough to
make it work from a makefile/shell script
(ie. writing a testsuit tool in c with
flags etc is not that important)

a simple solution is:
test collections are .c files in directories
each directory makes one executable
(several test_foo.c with some exported test_foo
functions and one main, the main can be even
generated if test_* convention is used but at first
hand writing it is probably simpler)
there is a shell script/makefile in the root that
builds and runs the tests in all directory
and gives a report at the end about the number
of failures

simplicity is important so that it is
easy to contribute tests or fix the system
later when there are more tests

extra features can come later:
> * advanced statistics(via what will be common/ipc.c): shared memory
> with two integers for each 'test collection' which will indicate:
>     a) number of tests that have begun executing (incremented before
> each test starts by the child)
>     b) number of successful tests
>     The pointer to shared memory will be passed as an argument to
> the child (test collection). From a) the framework will be able to
> know which test number has crashed the collection. If the collection
> returns, the framework will be able to calculate the success rate by
> dividing b) with a).
> * a few features which the user can invoke via some switches (using
> standard tokenizing functions - need to study them first):
>     -v --verbose without which collections will not print anything
> themselves
>     -t --test <test1>[,<test2>,...]  specific tests to execute
>     -x --exclude <test1>[,<test2>,...] exceptionally excluded tests
>     -h --help etc
> 
at some point (but probably not right now) it would
be nice if slow tests can be marked with some
macro oslt, by the test writer and it should be easy
to avoid those

i'm not sure about -t and -x
if long and descriptive testnames are used then a
regex filter is probably easier to use than -t -x
(but we don't know this yet so it should be decided
when there are many tests ready)

> Comments and suggestions are very welcome. Especially any advice
> about includes and .h files is welcome, I am new to structuring
> files in C.
if you are new to it then i really recommend to do
the absolute simplest/minimal thing you can imagine here,
then it will be easy to fix later..

of course ymmv