[PATCH 2/3] Add the T Test Framework

[Sebastian Huber]

On 20/03/2019 01:44, Chris Johns wrote:
> On 19/3/19 7:10 pm, Sebastian Huber wrote:
>> Hello Chris,
>>
>> On 19/03/2019 06:17, Chris Johns wrote:
>>> Hi,
>>>
>>> I am not sure how this fits in with what we have so before I can support any
>>> changes of this scale and direction I need to understand:
>>>
>>> 1. Are all existing tests to be converted to this new framework?
>> the main purpose for the new framework is to write unit, integration and
>> validation tests for the RTEMS SMP pre-qualification activity.
> I do not get the "RTEMS SMP" thing and it is confusing to me. There is RTEMS and
> there is a feature called SMP that you can enable along with other features of
> RTEMS. Your project is a qualification or pre-qualification effort for RTEMS
> with a focus on SMP. I would like to see us stop using "RTEMS SMP" as it's use
> would mean a qual effort for a non-SMP kernel would not align with the results
> your efforts are building. I have also mentioned this to Edsoft in other thread
> which I think has wandered off list for some reason. I have a concern the
> excessive use of "RTEMS SMP" to mean this project will taint the search engines
> in way we will not want. I would prefer we all start to deal with this topic as
> RTEMS Pre-Qualification or some shortened version and if you want you can add
> "for SMP" when discussing the specific focus for ESA. Can we please start to do
> this?

Sorry, this is just a habit developed over two years or so. ESA has some 
RTEMS variants and we needed a name for the new activity. I will try to 
omit the SMP in the future.

>> I am not sure what we do with the existing test suite.
> Then I am not sure what to do with this patch and proposed path. Both are linked
> and have always been so.

I wrote a couple of tests in the past and I think this new test 
framework would have made this job easier. Actually, I regret that I 
didn't invest the time to write it 10 years before, since I am pretty 
sure it would have paid off quite quickly. Even without this 
pre-qualification activity it makes sense to think about how test code 
could be written more efficiently with better output.

>
>> One option is to pick up existing tests and covert them.
> Have you evaluated what is required to update and clean up the existing tests?
>
> Is there an understanding of which existing tests could be updated to meet your
> project goals?

There are a lot of open questions. The feature set we want to 
pre-qualify (RTEMS SMP space profile) is not yet fully defined and 
depends a bit on the survey outcome. We don't have a budget to 
pre-qualify everything which is in the RTEMS code base. Definitely out 
of scope are file systems and the network stack.

>
> Is there an understanding of what existing tests may cover requirements you
> develop? What happens to the existing tests that are not covered by requirements
> because they do not overlap with the profile you are tasked to complete?
>
> How will coverage be handled? Currently there are coverage report levels that
> need to be maintained or accounted for. I cannot see how one set of tests for
> coverage analysis and another to match requirements can be made to work.

This are all important questions, but I think they are unrelated to the 
test framework itself.

We should do this incrementally. At some point we have to write tests, 
e.g. for example for the Classic semaphores. We could then have a look 
at the existing tests and step by step convert and refactor them to use 
the new framework along with test plans. For each patch in the tests the 
impact on the coverage needs to be evaluated. We definitely should not 
make things worse in terms of coverage.

For code coverage in ECSS you can use all sorts of tests (unit, 
integration, validation, whatever).

>
> What happens if there is conflict in results for parts that are duplicated?
>
> I am confused by what I see is a pretty basic conflict. Either you need to
> accept some of the existing tests or you will need to repeat what is in some of
> those tests? If you are required to accept the existing tests as is then I am
> unsure what is being offered here and if you need to repeat pieces or fragments
> of tests then I would be concerned as the testsuite would become fragmented with
> the repeated pieces.

I don't want a test suite fragmentation which results from copy and 
paste with modifications. However, we should be careful not to reduce 
test coverage through a conversion. After the activity there will be 
still a mix of tests written in the new framework and existing tests, 
e.g. we will likely not touch the file system tests.

>
> What I am starting to see here is over 600 existing tests that may not be
> viewable by your "integration and validations tests" artifact generation process
> and think this is an issue that needs to be resolved to move forward. I have no
> technical issue with what is being offered here, I am however concerned about
> the long term project issues that arise. I cannot allow this change in and then
> the possibility of tests appearing where the project's needs to review each one
> to determine what overlaps and conflicts with the existing testsuite.

Yes, reviewing every test conversion patch at the detail level would be 
infeasible from my point of view. A typical patch will probably remove 
some test code in the existing test suite and add new test code 
somewhere else. There will be probably more than a hundred of these 
patches. We should make sure that the patches are self-contained and 
potentially easy to review. Quality goals should be defined, e.g. code 
coverage must not get worse through a test code patch.

>
>> How this is organized should be discussed in a separate thread
> I am not driving this. The parts are linked as I stated above. I cannot accept
> these changes in pieces without understanding and accepting the whole concept.

I think it is good to integrate things as early as possible. A mega 
patch after several months which covers test framework, test 
conversions, code changes, documentation changes, requirements, etc. is 
probably not helpful.

>
>>> 2. How does this effect the existing eco-system support such as the `rtems-test`
>>> command and the documentation around that command?
>> The rtems-test command just looks at the begin and end of test messages. You can
>> still use it with the new framework, see the test output of the example test:
>>
>> *** BEGIN OF TEST TTEST 1 ***
>> *** TEST VERSION: 5.0.0.286e9354e008b08983e6390a68f8ecc5071de069
>> *** TEST STATE: EXPECTED-PASS
>> *** TEST BUILD: RTEMS_DEBUG RTEMS_NETWORKING RTEMS_POSIX_API RTEMS_SMP
>> *** TEST TOOLS: 7.4.0 20181206 (RTEMS 5, RSB
>> e0aec65182449a4e22b820e773087636edaf5b32, Newlib 1d35a003f)
>> A:ttest01
>> S:Platform:RTEMS
>> S:Compiler:7.4.0 20181206 (RTEMS 5, RSB
>> e0aec65182449a4e22b820e773087636edaf5b32, Newlib 1d35a003f)
>> S:Version:5.0.0.286e9354e008b08983e6390a68f8ecc5071de069
>> S:BSP:erc32
>> S:RTEMS_DEBUG:1
>> S:RTEMS_MULTIPROCESSING:0
>> S:RTEMS_POSIX_API:1
>> S:RTEMS_PROFILING:0
>> S:RTEMS_SMP:1
>> B:example
>> P:0:0:UI1:test-example.c:5
>> F:1:0:UI1:test-example.c:6:test fails
>> F:*:0:UI1:test-example.c:8:quiet test fails
>> P:2:0:UI1:test-example.c:9
>> F:3:0:UI1:test-example.c:10:step test fails
>> F:4:0:UI1:test-example.c:11:this is a format string
>> E:example:N:5:F:4:D:0.001000
>> Z:ttest01:C:1:N:5:F:4:D:0.003000
>>
>> *** END OF TEST TTEST 1 ***
>>
>> What you get in addition is a structured output in the middle. This can be used
>> to generated more detailed reports. This is another topic. The test framework
>> should just enable you to easily parse the test output and do something with it.
> This is great and I welcome it. I was thinking we needed a way to capture the
> per test output as a sort of screen capture update for the existing tests.
>
>>> 3. What does 'T' in THE_T_TEST_FRAMEWORK_H stand for? I prefer we prefix RTEMS_
>>> where it makes sense.
>> The 'T' is just a random name which associates with testing. I searched a bit
>> for <t.h> and a T_ prefix and didn't found an existing project. So, there should
>> be no name conflicts. It is short, so this is good for typing.
> A number of pieces I have recently added can be standalone and I have added
> RTEMS as a prefix to help get the project's name out there.

Writing tests code is easier if the function names are short and have a 
specific prefix (this helps auto-completion through code editors, you 
just have to type T_... and not rtems_...). A specific prefix makes it 
easier to grasp what is test framework code and what are functions under 
test, e.g.

T_TEST_CASE(timer)
{
     rtems_status_code sc;
     rtems_id id;
     rtems_id task;
     rtems_event_set events;

     T_plan(8);
     T_step_true(0, T_is_runner(), "test body is not runner");

     sc = rtems_timer_create(rtems_build_name('T', 'E', 'S', 'T'), &id);
     T_step_rsc_success(1, sc);

     task = rtems_task_self();
     sc = rtems_timer_fire_after(id, 1, wakeup, &task);
     T_step_rsc_success(2, sc);

     events = 0;
     sc = rtems_event_receive(RTEMS_EVENT_0, RTEMS_WAIT | RTEMS_EVENT_ALL,
         RTEMS_NO_TIMEOUT, &events);
     T_step_rsc_success(5, sc);
     T_step_eq_u32(6, events, RTEMS_EVENT_0);

     sc = rtems_timer_delete(id);
     T_step_rsc_success(7, sc);
}

vs.

T_TEST_CASE(timer)
{
     rtems_status_code sc;
     rtems_id id;
     rtems_id task;
     rtems_event_set events;

     rtems_plan(8);
     rtems_step_true(0, rtems_is_runner(), "test body is not runner");

     sc = rtems_timer_create(rtems_build_name('T', 'E', 'S', 'T'), &id);
     rtems_step_rsc_success(1, sc);

     task = rtems_task_self();
     sc = rtems_timer_fire_after(id, 1, wakeup, &task);
     rtems_step_rsc_success(2, sc);

     events = 0;
     sc = rtems_event_receive(RTEMS_EVENT_0, RTEMS_WAIT | RTEMS_EVENT_ALL,
         RTEMS_NO_TIMEOUT, &events);
     rtems_step_rsc_success(5, sc);
     rtems_step_eq_u32(6, events, RTEMS_EVENT_0);

     sc = rtems_timer_delete(id);
     rtems_step_rsc_success(7, sc);
}

I think promoting the name RTEMS is not as important.

>
>> The T Test
>> Framework is portable. It runs also on Linux, FreeBSD, and MSYS2. I will
>> probably add it also as a stand a lone project to github.
> Which would be the master implementation?

The core code is portable. It needs a bit of work to keep two 
repositories in synchronization.

>
>>> 4. I see in another email you post a Sphinx generated report. What are those
>>> tests and what is used to capture and create that report and will this in time
>>> include all existing tests?
>> I wrote a very simple and stupid Python script to extract this information from
>> the test output just to evaluate if the output format makes sense. The output is
>> from some example tests I used to test the framework.
> I think the idea of generating ReST format to create documents is really nice.
>
>> For the pre-qualification
>> we need test plans, tests, test reports, traceability to requirements and test
>> verification.
> Yes, this is understood and why I am including the existing testsuite tests.
> There are complications around this area that need to be resolved.
>
>> The framework enables you to efficiently write test code and
>> generation of easy to parse output.
> I wish it could be used on all tests.

At some point you have to get started.

-- 
Sebastian Huber, embedded brains GmbH

Address : Dornierstr. 4, D-82178 Puchheim, Germany
Phone   : +49 89 189 47 41-16
Fax     : +49 89 189 47 41-09
E-Mail  : sebastian.huber at embedded-brains.de
PGP     : Public key available on request.

Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG.