[PATCH 00/33] Test framework improvements

[Sebastian Huber]

On 23/07/2020 00:25, Chris Johns wrote:

> On 22/7/20 7:16 pm, Sebastian Huber wrote:
>> On 22/07/2020 07:04, Chris Johns wrote:
>>> On 22/7/20 1:04 am, Sebastian Huber wrote:
>>> b) As previously discussed there are a few more states a test result can be
>>> other than pass and fail and I thikn the wording here may need tightening. I am
>>> not sure what is needed. Also how do resource leaks effect the result?
>> Currently, the test framework supports only passed and failed at the level of a
>> test check. I am not sure if adding more states at this level is really helpful.
> The terms pass and failed are overloaded and relative to a specific context and
> I am wondering if this needs to be clearer.

Yes they are overloaded. For the test framework it is defined what the 
terms mean for a test check, a test case, and a test suite. If it is not 
clear from the context, you can say for example "the test check failed".

googletest uses these terms:

"When using googletest, you start by writing /assertions/, which are 
statements that check whether a condition is true. An assertion's result 
can be /success/, /nonfatal failure/, or /fatal failure/. If a fatal 
failure occurs, it aborts the current function; otherwise the program 
continues normally."

https://github.com/google/googletest/blob/master/googletest/docs/primer.md#basic-concepts

ISTQB uses pass/fail:

https://glossary.istqb.org/en/search/fail

I think we just have to clearly define what the terms mean in our scope 
and this is what I tried to do.

>
>>> c) Does the framework provide a standard way to export a dynamic test
>>> environment so the test and the results can be reviewed as a complete set of
>>> data?
>> Sorrry, I don't understand this question.
> If dynamic configuration effects the test and the test fails would reproducing
> the result be more difficult if you do not know what the dynamic state was? For
> example using this for a network test and the dynamic configuration can select
> between DHCP and a static set up.

You could write a log message which tells you the configuration used. 
The tests can be written in C, so you can do a lot of things. It is up 
to the writer of the tests to do it properly.

[...]
>>> a) What is the picture drawn in?
>> Libreoffice Draw.
>>> I will comment on the patches as well.
>> Thanks for the review.
> Sorry, another question, is libtest installed and available to other parts of
> RTEMS like libbsd and also available to users?

Yes, it is installed:

project_lib_LIBRARIES += librtemstest.a