[PATCH 2/3] Add the T Test Framework

[Sebastian Huber]

Hello Chris,

On 19/03/2019 06:17, Chris Johns wrote:
> Hi,
>
> I am not sure how this fits in with what we have so before I can support any
> changes of this scale and direction I need to understand:
>
> 1. Are all existing tests to be converted to this new framework?

the main purpose for the new framework is to write unit, integration and 
validation tests for the RTEMS SMP pre-qualification activity. I am not 
sure what we do with the existing test suite. One option is to pick up 
existing tests and covert them. How this is organized should be 
discussed in a separate thread.

>
> 2. How does this effect the existing eco-system support such as the `rtems-test`
> command and the documentation around that command?

The rtems-test command just looks at the begin and end of test messages. 
You can still use it with the new framework, see the test output of the 
example test:

*** BEGIN OF TEST TTEST 1 ***
*** TEST VERSION: 5.0.0.286e9354e008b08983e6390a68f8ecc5071de069
*** TEST STATE: EXPECTED-PASS
*** TEST BUILD: RTEMS_DEBUG RTEMS_NETWORKING RTEMS_POSIX_API RTEMS_SMP
*** TEST TOOLS: 7.4.0 20181206 (RTEMS 5, RSB 
e0aec65182449a4e22b820e773087636edaf5b32, Newlib 1d35a003f)
A:ttest01
S:Platform:RTEMS
S:Compiler:7.4.0 20181206 (RTEMS 5, RSB 
e0aec65182449a4e22b820e773087636edaf5b32, Newlib 1d35a003f)
S:Version:5.0.0.286e9354e008b08983e6390a68f8ecc5071de069
S:BSP:erc32
S:RTEMS_DEBUG:1
S:RTEMS_MULTIPROCESSING:0
S:RTEMS_POSIX_API:1
S:RTEMS_PROFILING:0
S:RTEMS_SMP:1
B:example
P:0:0:UI1:test-example.c:5
F:1:0:UI1:test-example.c:6:test fails
F:*:0:UI1:test-example.c:8:quiet test fails
P:2:0:UI1:test-example.c:9
F:3:0:UI1:test-example.c:10:step test fails
F:4:0:UI1:test-example.c:11:this is a format string
E:example:N:5:F:4:D:0.001000
Z:ttest01:C:1:N:5:F:4:D:0.003000

*** END OF TEST TTEST 1 ***

What you get in addition is a structured output in the middle. This can 
be used to generated more detailed reports. This is another topic. The 
test framework should just enable you to easily parse the test output 
and do something with it.

>
> 3. What does 'T' in THE_T_TEST_FRAMEWORK_H stand for? I prefer we prefix RTEMS_
> where it makes sense.

The 'T' is just a random name which associates with testing. I searched 
a bit for <t.h> and a T_ prefix and didn't found an existing project. 
So, there should be no name conflicts. It is short, so this is good for 
typing. The T Test Framework is portable. It runs also on Linux, 
FreeBSD, and MSYS2. I will probably add it also as a stand a lone 
project to github.

>
> 4. I see in another email you post a Sphinx generated report. What are those
> tests and what is used to capture and create that report and will this in time
> include all existing tests?

I wrote a very simple and stupid Python script to extract this 
information from the test output just to evaluate if the output format 
makes sense. The output is from some example tests I used to test the 
framework. For the pre-qualification we need test plans, tests, test 
reports, traceability to requirements and test verification. The 
framework enables you to efficiently write test code and generation of 
easy to parse output.

>
> Chris
>
> On 14/3/19 9:24 pm, Sebastian Huber wrote:
>> Update #3199.
>> ---
>>   cpukit/Makefile.am                    |   11 +
>>   cpukit/headers.am                     |    1 +
>>   cpukit/include/t.h                    | 2128 +++++++++++++++++++++++++++++++++
>>   cpukit/libtest/t-test-checks-eno.c    |  145 +++
>>   cpukit/libtest/t-test-checks-psx.c    |   17 +

-- 
Sebastian Huber, embedded brains GmbH

Address : Dornierstr. 4, D-82178 Puchheim, Germany
Phone   : +49 89 189 47 41-16
Fax     : +49 89 189 47 41-09
E-Mail  : sebastian.huber at embedded-brains.de
PGP     : Public key available on request.

Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: report.py
Type: text/x-python
Size: 7254 bytes
Desc: not available
URL: <http://lists.rtems.org/pipermail/devel/attachments/20190319/c9051591/attachment-0002.py>