[PATCH 1/1] eng: Add software test framework chapter

[Chris Johns]

On 14/3/19 9:23 pm, Sebastian Huber wrote:
> Update #3199.
> ---
>  eng/index.rst          |    3 +
>  eng/test-framework.rst | 2077 ++++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 2080 insertions(+)
>  create mode 100644 eng/test-framework.rst
> 
> diff --git a/eng/index.rst b/eng/index.rst
> index fe51e91..f29d0f3 100644
> --- a/eng/index.rst
> +++ b/eng/index.rst
> @@ -11,6 +11,8 @@ RTEMS Software Engineering (|version|)
>  
>  .. topic:: Copyrights and License
>  
> +    | |copy| 2018, 2019 embedded brains GmbH
> +    | |copy| 2018, 2019 Sebastian Huber
>      | |copy| 1988, 2015 On-Line Applications Research Corporation (OAR)
>  
>      .. include:: ../common/license.rst
> @@ -26,6 +28,7 @@ RTEMS Software Engineering (|version|)
>      stakeholders
>      management
>      test-plan
> +    test-framework
>      release-mgmt
>      users-manuals
>      license-requirements
> diff --git a/eng/test-framework.rst b/eng/test-framework.rst
> new file mode 100644
> index 0000000..dda13bc
> --- /dev/null
> +++ b/eng/test-framework.rst
> @@ -0,0 +1,2077 @@
> +.. SPDX-License-Identifier: CC-BY-SA-4.0
> +
> +.. Copyright (C) 2018, 2019 embedded brains GmbH
> +.. Copyright (C) 2018, 2019 Sebastian Huber
> +
> +Software Test Framework
> +***********************
> +
> +.. _TheTTestFramework:
> +
> +The T Test Framework
> +====================
> +
> +The `T Test Framework` helps to write test suites.  A `test suite` is a
> +collection of test cases.  A `test case` consists of individual test actions
> +and checks.  A `test check` determines if the outcome of a test action meets
> +its expectation.  A `test action` is a program sequence with an observable
> +outcome, for example a function invocation with a return status.  If the test
> +action outcome is all right, then the test check passes, otherwise the test
> +check fails.  The test check failures of a test case are summed up.  A test
> +case passes, if the failure count of this test case is zero, otherwise the test
> +case fails.  The test suite passes if all test cases pass, otherwise it fails.

I think this needs a little more clarification. The density of that is being
expressed is too compact and I think it requires some expansion. For example
from this I cannot tell where an executable boundary is and given RTEMS has had
'testsuites' I feel expanding and explaining each of these is important in
relationship to what RTEMS has and needs. For example:

  The `T Test Framework` helps to write test suites.  A `test suite` is a
  collection of test cases ......

and then ...

  ... where tests for specific functionality are grouped for example
  `fstests`.

Is a test executable a collection of "test cases" or a single case? How is this
decided?

What is the expected number of tests once this SMP qual work ends? Is hardware
testing a requirement?

An outline of the problem domain being addressed would help?

Can there please be some explanation of what the issues this framework is
attempt to solve are, what it does well, what it does not do well or is not,
plus things like target requirements, test duration, etc?

I feel if this is done the detail that follows will be just that the detail.

Chris