[PATCH 1/1] eng: Add software test framework chapter

[Sebastian Huber]

On 19/03/2019 06:21, Chris Johns wrote:
> On 14/3/19 9:23 pm, Sebastian Huber wrote:
>> Update #3199.
>> ---
>>   eng/index.rst          |    3 +
>>   eng/test-framework.rst | 2077 ++++++++++++++++++++++++++++++++++++++++++++++++
>>   2 files changed, 2080 insertions(+)
>>   create mode 100644 eng/test-framework.rst
>>
>> diff --git a/eng/index.rst b/eng/index.rst
>> index fe51e91..f29d0f3 100644
>> --- a/eng/index.rst
>> +++ b/eng/index.rst
>> @@ -11,6 +11,8 @@ RTEMS Software Engineering (|version|)
>>   
>>   .. topic:: Copyrights and License
>>   
>> +    | |copy| 2018, 2019 embedded brains GmbH
>> +    | |copy| 2018, 2019 Sebastian Huber
>>       | |copy| 1988, 2015 On-Line Applications Research Corporation (OAR)
>>   
>>       .. include:: ../common/license.rst
>> @@ -26,6 +28,7 @@ RTEMS Software Engineering (|version|)
>>       stakeholders
>>       management
>>       test-plan
>> +    test-framework
>>       release-mgmt
>>       users-manuals
>>       license-requirements
>> diff --git a/eng/test-framework.rst b/eng/test-framework.rst
>> new file mode 100644
>> index 0000000..dda13bc
>> --- /dev/null
>> +++ b/eng/test-framework.rst
>> @@ -0,0 +1,2077 @@
>> +.. SPDX-License-Identifier: CC-BY-SA-4.0
>> +
>> +.. Copyright (C) 2018, 2019 embedded brains GmbH
>> +.. Copyright (C) 2018, 2019 Sebastian Huber
>> +
>> +Software Test Framework
>> +***********************
>> +
>> +.. _TheTTestFramework:
>> +
>> +The T Test Framework
>> +====================
>> +
>> +The `T Test Framework` helps to write test suites.  A `test suite` is a
>> +collection of test cases.  A `test case` consists of individual test actions
>> +and checks.  A `test check` determines if the outcome of a test action meets
>> +its expectation.  A `test action` is a program sequence with an observable
>> +outcome, for example a function invocation with a return status.  If the test
>> +action outcome is all right, then the test check passes, otherwise the test
>> +check fails.  The test check failures of a test case are summed up.  A test
>> +case passes, if the failure count of this test case is zero, otherwise the test
>> +case fails.  The test suite passes if all test cases pass, otherwise it fails.
> I think this needs a little more clarification. The density of that is being
> expressed is too compact and I think it requires some expansion. For example
> from this I cannot tell where an executable boundary is and given RTEMS has had
> 'testsuites' I feel expanding and explaining each of these is important in
> relationship to what RTEMS has and needs. For example:
>
>    The `T Test Framework` helps to write test suites.  A `test suite` is a
>    collection of test cases ......
>
> and then ...
>
>    ... where tests for specific functionality are grouped for example
>    `fstests`.

This chapter should just describe the test framework and not an RTEMS 
specific test suite. This is a different topic. The test framework 
should enable you to organize your tests one way or another.

>
> Is a test executable a collection of "test cases" or a single case? How is this
> decided?

A test executable can contain one or more test suites. Test cases using 
the T_TEST_CASE() macro are registered via a linker set. However, you 
can run test cases also individually. See the attached example which 
contains two test suites. It is a conversion of 
testsuites/sptests/spsysinit01/init.c. It demonstrates that you can test 
the system initialization procedure with this framework. How you 
organize the tests is up to you and a separate topic. The test framework 
is just a tool to write test cases.

>
> What is the expected number of tests once this SMP qual work ends?

I don't know. I guess it will be similar to the existing test suite. 
Probably a bit more performance tests.

> Is hardware
> testing a requirement?

Yes, although thanks to Jiri Gaisler's work we probably don't have to 
run the tests often on real hardware. We need it for the runtime 
measurements though.

>
> An outline of the problem domain being addressed would help?

What do you mean with problem domain? From my point of view the problem 
domain is writing test cases?

>
> Can there please be some explanation of what the issues this framework is
> attempt to solve are, what it does well, what it does not do well or is not,
> plus things like target requirements, test duration, etc?

Ok, good idea. I will put an overview/features section at the top of the 
chapter.

>
> I feel if this is done the detail that follows will be just that the detail.
>
> Chris

-- 
Sebastian Huber, embedded brains GmbH

Address : Dornierstr. 4, D-82178 Puchheim, Germany
Phone   : +49 89 189 47 41-16
Fax     : +49 89 189 47 41-09
E-Mail  : sebastian.huber at embedded-brains.de
PGP     : Public key available on request.

Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: test-sysinit.c
Type: text/x-csrc
Size: 21969 bytes
Desc: not available
URL: <http://lists.rtems.org/pipermail/devel/attachments/20190319/8f6b230b/attachment-0002.bin>