[PATCH 1/1] eng: Add software test framework chapter
Sebastian Huber
sebastian.huber at embedded-brains.de
Thu Mar 14 10:23:27 UTC 2019
Update #3199.
---
eng/index.rst | 3 +
eng/test-framework.rst | 2077 ++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 2080 insertions(+)
create mode 100644 eng/test-framework.rst
diff --git a/eng/index.rst b/eng/index.rst
index fe51e91..f29d0f3 100644
--- a/eng/index.rst
+++ b/eng/index.rst
@@ -11,6 +11,8 @@ RTEMS Software Engineering (|version|)
.. topic:: Copyrights and License
+ | |copy| 2018, 2019 embedded brains GmbH
+ | |copy| 2018, 2019 Sebastian Huber
| |copy| 1988, 2015 On-Line Applications Research Corporation (OAR)
.. include:: ../common/license.rst
@@ -26,6 +28,7 @@ RTEMS Software Engineering (|version|)
stakeholders
management
test-plan
+ test-framework
release-mgmt
users-manuals
license-requirements
diff --git a/eng/test-framework.rst b/eng/test-framework.rst
new file mode 100644
index 0000000..dda13bc
--- /dev/null
+++ b/eng/test-framework.rst
@@ -0,0 +1,2077 @@
+.. SPDX-License-Identifier: CC-BY-SA-4.0
+
+.. Copyright (C) 2018, 2019 embedded brains GmbH
+.. Copyright (C) 2018, 2019 Sebastian Huber
+
+Software Test Framework
+***********************
+
+.. _TheTTestFramework:
+
+The T Test Framework
+====================
+
+The `T Test Framework` helps to write test suites. A `test suite` is a
+collection of test cases. A `test case` consists of individual test actions
+and checks. A `test check` determines if the outcome of a test action meets
+its expectation. A `test action` is a program sequence with an observable
+outcome, for example a function invocation with a return status. If the test
+action outcome is all right, then the test check passes, otherwise the test
+check fails. The test check failures of a test case are summed up. A test
+case passes, if the failure count of this test case is zero, otherwise the test
+case fails. The test suite passes if all test cases pass, otherwise it fails.
+
+Test Cases
+----------
+
+You can write a test case with the `T_TEST_CASE()` macro followed by a function
+body:
+
+.. code-block:: c
+
+ T_TEST_CASE(name)
+ {
+ /* Your test case code */
+ }
+
+The test case `name` must be a valid C designator. The test case names must be
+unique within the test suite. Just link modules with test cases to the test
+runner to form a test suite. The test cases are automatically registered via
+static constructors.
+
+.. code-block:: c
+ :caption: Test Case Example
+
+ #include <t.h>
+
+ static int add(int a, int b)
+ {
+ return a + b;
+ }
+
+ T_TEST_CASE(a_test_case)
+ {
+ int actual_value;
+
+ actual_value = add(1, 1);
+ T_eq_int(actual_value, 2);
+ T_true(false, "a test failure message");
+ }
+
+.. code-block:: none
+ :caption: Test Case Report
+
+ B:a_test_case
+ P:0:8:UI1:test-simple.c:13
+ F:1:8:UI1:test-simple.c:14:a test failure message
+ E:a_test_case:N:2:F:1:D:0.001657
+
+The `B` line indicates the begin of test case `a_test_case`. The `P` line
+shows that the test check in file `test-simple.c` at line 13 executed by task
+`UI1` on processor 0 as the test step 0 passed. The invocation of `add()` in
+line 12 is the test action of test step 0. The `F` lines shows that the test
+check in file `test-simple.c` at line 14 executed by task `UI1` on processor 0
+as the test step 1 failed with a message of `"a test failure message"`. The
+`E` line indicates the end of test case `a_test_case` resulting in a total of
+two test steps (`N`) and one test failure (`F`). The test case execution
+duration (`D`) was 0.001657 seconds. For test report details see:
+:ref:`Test Reporting <TheTTestFrameworkTestReporting>`.
+
+Test Fixture
+------------
+
+You can write a test case with a test fixture with the `T_TEST_CASE_FIXTURE()`
+macro followed by a function body:
+
+.. code-block:: c
+
+ T_TEST_CASE_FIXTURE(name, fixture)
+ {
+ /* Your test case code */
+ }
+
+The test case `name` must be a valid C designator. The test case names must be
+unique within the test suite. The `fixture` must point to a statically
+initialized read-only object of type `T_fixture`. The test fixture
+provides methods to setup, stop and tear down a test case. A context is passed
+to the methods. The initial context is defined by the read-only fixture
+object. The context can be obtained by the `T_fixture_context()`
+function. It can be set within the scope of one test case by the
+`T_set_fixture_context()` function. This can be used for example to
+dynamically allocate a test environment in the setup method.
+
+.. code-block:: c
+ :caption: Test Fixture Example
+
+ #include <t.h>
+
+ static int initial_value = 3;
+
+ static int counter;
+
+ static void
+ setup(void *ctx)
+ {
+ int *c;
+
+ T_log(T_QUIET, "setup begin");
+ T_eq_ptr(ctx, &initial_value);
+ T_eq_ptr(ctx, T_fixture_context());
+ c = ctx;
+ counter = *c;
+ T_set_fixture_context(&counter);
+ T_eq_ptr(&counter, T_fixture_context());
+ T_log(T_QUIET, "setup end");
+ }
+
+ static void
+ stop(void *ctx)
+ {
+ int *c;
+
+ T_log(T_QUIET, "stop begin");
+ T_eq_ptr(ctx, &counter);
+ c = ctx;
+ ++(*c);
+ T_log(T_QUIET, "stop end");
+ }
+
+ static void
+ teardown(void *ctx)
+ {
+ int *c;
+
+ T_log(T_QUIET, "teardown begin");
+ T_eq_ptr(ctx, &counter);
+ c = ctx;
+ T_eq_int(*c, 4);
+ T_log(T_QUIET, "teardown end");
+ }
+
+ static const T_fixture fixture = {
+ .setup = setup,
+ .stop = stop,
+ .teardown = teardown,
+ .initial_context = &initial_value
+ };
+
+ T_TEST_CASE_FIXTURE(fixture, &fixture)
+ {
+ T_assert_true(true, "all right");
+ T_assert_true(false, "test fails and we stop the test case");
+ T_log(T_QUIET, "not reached");
+ }
+
+.. code-block:: none
+ :caption: Test Fixture Report
+
+ B:fixture
+ L:setup begin
+ P:0:0:UI1:test-fixture.c:13
+ P:1:0:UI1:test-fixture.c:14
+ P:2:0:UI1:test-fixture.c:18
+ L:setup end
+ P:3:0:UI1:test-fixture.c:55
+ F:4:0:UI1:test-fixture.c:56:test fails and we stop the test case
+ L:stop begin
+ P:5:0:UI1:test-fixture.c:28
+ L:stop end
+ L:teardown begin
+ P:6:0:UI1:test-fixture.c:40
+ P:7:0:UI1:test-fixture.c:42
+ L:teardown end
+ E:fixture:N:8:F:1
+
+Test Case Planning
+------------------
+
+Each non-quiet test check fetches and increments the test step counter
+atomically. For each test case execution the planned steps can be specified
+with the `T_plan()` function.
+
+.. code-block:: c
+
+ void T_plan(unsigned int planned_steps);
+
+This function must be invoked at most once in each test case execution. If the
+planned test steps are set with this function, then the final test steps after
+the test case execution must be equal to the planned steps, otherwise the test
+case fails.
+
+Use the `T_step_*(step, ...)` test check variants to ensure that the test case
+execution follows exactly the planned steps.
+
+.. code-block:: c
+ :caption: Test Planning Example
+
+ #include <t.h>
+
+ T_TEST_CASE(wrong_step)
+ {
+ T_plan(2);
+ T_step_true(0, true, "all right");
+ T_step_true(2, true, "wrong step");
+ }
+
+ T_TEST_CASE(plan_ok)
+ {
+ T_plan(1);
+ T_step_true(0, true, "all right");
+ }
+
+ T_TEST_CASE(plan_failed)
+ {
+ T_plan(2);
+ T_step_true(0, true, "not enough steps");
+ T_quiet_true(true, "quiet test do not count");
+ }
+
+ T_TEST_CASE(double_plan)
+ {
+ T_plan(99);
+ T_plan(2);
+ }
+
+ T_TEST_CASE(steps)
+ {
+ T_step(0, "a");
+ T_plan(3);
+ T_step(1, "b");
+ T_step(2, "c");
+ }
+
+.. code-block:: none
+ :caption: Test Planning Report
+
+ B:wrong_step
+ P:0:0:UI1:test-plan.c:6
+ F:1:0:UI1:test-plan.c:7:planned step (2)
+ E:wrong_step:N:2:F:1
+ B:plan_ok
+ P:0:0:UI1:test-plan.c:13
+ E:plan_ok:N:1:F:0
+ B:plan_failed
+ P:0:0:UI1:test-plan.c:19
+ F:*:0:UI1:*:*:actual steps (1), planned steps (2)
+ E:plan_failed:N:1:F:1
+ B:double_plan
+ F:*:0:UI1:*:*:planned steps (99) already set
+ E:double_plan:N:0:F:1
+ B:steps
+ P:0:0:UI1:test-plan.c:31
+ P:1:0:UI1:test-plan.c:33
+ P:2:0:UI1:test-plan.c:34
+ E:steps:N:3:F:0
+
+Test Case Resource Accounting
+-----------------------------
+
+The framework can check if various resources are leaked during a test case
+execution. The resource checkers are specified by the test run configuration.
+On RTEMS, checks for the following resources are available
+
+* workspace and heap memory,
+* file descriptors,
+* POSIX keys and key value pairs,
+* RTEMS barriers,
+* RTEMS user extensions,
+* RTEMS message queues,
+* RTEMS partitions,
+* RTEMS periods,
+* RTEMS regions,
+* RTEMS semaphores,
+* RTEMS tasks, and
+* RTEMS timers.
+
+.. code-block:: c
+ :caption: Resource Accounting Example
+
+ #include <t.h>
+
+ #include <stdlib.h>
+
+ #include <rtems.h>
+
+ T_TEST_CASE(missing_sema_delete)
+ {
+ rtems_status_code sc;
+ rtems_id id;
+
+ sc = rtems_semaphore_create(rtems_build_name('S', 'E', 'M', 'A'), 0,
+ RTEMS_COUNTING_SEMAPHORE, 0, &id);
+ T_rsc_success(sc);
+ }
+
+ T_TEST_CASE(missing_free)
+ {
+ void *p;
+
+ p = malloc(1);
+ T_not_null(p);
+ }
+
+.. code-block:: none
+ :caption: Resource Accounting Report
+
+ B:missing_sema_delete
+ P:0:0:UI1:test-leak.c:14
+ F:*:0:UI1:*:*:RTEMS semaphore leak (1)
+ E:missing_sema_delete:N:1:F:1:D:0.004013
+ B:missing_free
+ P:0:0:UI1:test-leak.c:22
+ F:*:0:UI1:*:*:memory leak in workspace or heap
+ E:missing_free:N:1:F:1:D:0.003944
+
+Test Case Scoped Resources
+--------------------------
+
+You can allocate resources which are automatically destroyed after the test
+case execution.
+
+.. code-block:: c
+
+ void *T_malloc(size_t size);
+
+ void *T_calloc(size_t nelem, size_t elsize);
+
+ void *T_zalloc(size_t size);
+
+ void T_free(void *ptr);
+
+.. code-block:: c
+ :caption: Test Case Scoped Resource Example
+
+ #include <t.h>
+
+ T_TEST_CASE(malloc_free)
+ {
+ void *p;
+
+ p = T_malloc(1);
+ T_assert_not_null(p);
+ T_free(p);
+ }
+
+ T_TEST_CASE(malloc_auto)
+ {
+ void *p;
+
+ p = T_malloc(1);
+ T_assert_not_null(p);
+ }
+
+.. code-block:: none
+ :caption: Test Case Scoped Resource Report
+
+ B:malloc_free
+ P:0:0:UI1:test-malloc.c:8
+ E:malloc_free:N:1:F:0
+ B:malloc_auto
+ P:0:0:UI1:test-malloc.c:17
+ E:malloc_auto:N:1:F:0
+
+Test Case Destructors
+---------------------
+
+You can add test case destructors with `T_add_destructor()`. They are called
+automatically at the test case end before the resource accounting takes place.
+Optionally, a registered destructor can be removed before the test case end
+with `T_remove_destructor()`. The `T_destructor` structure of a destructor
+must exist after the return from the test case body. Do not use stack memory
+or dynamic memory obtained via `T_malloc()`, `T_calloc()` or `T_zalloc()` for
+the `T_destructor` structure.
+
+.. code-block:: c
+
+ void T_add_destructor(T_destructor *destructor,
+ void (*destroy)(T_destructor *));
+
+ void T_remove_destructor(T_destructor *destructor);
+
+.. code-block:: c
+ :caption: Test Case Destructor Example
+
+ #include <t.h>
+
+ static void
+ destroy(T_destructor *dtor)
+ {
+ (void)dtor;
+ T_step(0, "destroy");
+ }
+
+ T_TEST_CASE(destructor)
+ {
+ static T_destructor dtor;
+
+ T_plan(1);
+ T_add_destructor(&dtor, destroy);
+ }
+
+.. code-block:: none
+ :caption: Test Case Destructor Report
+
+ B:destructor
+ P:0:0:UI1:test-destructor.c:7
+ E:destructor:N:1:F:0:D:0.003714
+
+Test Checks
+-----------
+
+A `test check` determines if the actual value presented to the test check meets
+its expectation. The actual value should represent the outcome of a test
+action. If the actual value is all right, then the test check passes,
+otherwise the test check fails. A failed test check does not stop the test
+case execution immediately unless the `T_assert_*()` test variant is used.
+Each test check increments the test step counter unless the `T_quiet_*()` test
+variant is used. The test step counter is initialized to zero before the test
+case begins to execute. The `T_step_*(step, ...)` test check variants verify
+that the test step counter is equal to the planned test step value, otherwise
+the test check fails.
+
+Test Check Parameter Conventions
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The following names for test check parameters are used throughout the test
+checks:
+
+step
+ The planned test step for this test check.
+
+a
+ The actual value to check against an expected value. It is usually the
+ first parameter in all test checks, except in the `T_step_*(step, ...)`
+ test check variants, here it is the second parameter.
+
+e
+ The expected value of a test check. This parameter is optional. Some test
+ checks have an implicit expected value. If present, then this parameter is
+ directly after the actual value parameter of the test check.
+
+fmt
+ A printf()-like format string. Floating-point and exotic formats may be
+ not supported.
+
+Test Check Condition Conventions
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The following names for test check conditions are used:
+
+eq
+ The actual value must equal the expected value.
+
+ne
+ The actual value must not equal the value of the second parameter.
+
+ge
+ The actual value must be greater than or equal to the expected value.
+
+gt
+ The actual value must be greater than the expected value.
+
+le
+ The actual value must be less than or equal to the expected value.
+
+lt
+ The actual value must be less than the expected value.
+
+If the actual value satisfies the test check condition, then the test check
+passes, otherwise it fails.
+
+Test Check Variant Conventions
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The `T_assert_*()` test check variants stop the current test case execution if
+the test fails.
+
+The `T_quiet_*()` test check variants do not increment the test step counter
+and only print a message if the test check fails. This is helpful in case a
+test check appears in a tight loop.
+
+The `T_step_*(step, ...)` test check variants check that the test step counter
+is equal to the specified test step value, otherwise the test check fails.
+
+The following names for test check type variants are used:
+
+ptr
+ The test value must be a pointer (`void *`).
+
+mem
+ The test value must be a memory area with a specified length.
+
+str
+ The test value must be a null byte terminated string.
+
+nstr
+ The length of the test value string is limited to a specified maximum.
+
+char
+ The test value must be a character (`char`).
+
+schar
+ The test value must be a signed character (`signed char`).
+
+uchar
+ The test value must be an unsigned character (`unsigned char`).
+
+short
+ The test value must be a short integer (`short`).
+
+ushort
+ The test value must be an unsigned short integer (`unsigned short`).
+
+int
+ The test value must be an integer (`int`).
+
+uint
+ The test value must be an unsigned integer (`unsigned int`).
+
+long
+ The test value must be a long integer (`long`).
+
+ulong
+ The test value must be an unsigned long integer (`unsigned long`).
+
+ll
+ The test value must be a long long integer (`long long`).
+
+ull
+ The test value must be an unsigned long long integer (`unsigned long long`).
+
+i8
+ The test value must be a signed 8-bit integer (`int8_t`).
+
+u8
+ The test value must be an unsigned 8-bit integer (`uint8_t`).
+
+i16
+ The test value must be a signed 16-bit integer (`int16_t`).
+
+u16
+ The test value must be an unsigned 16-bit integer (`uint16_t`).
+
+i32
+ The test value must be a signed 32-bit integer (`int32_t`).
+
+u32
+ The test value must be an unsigned 32-bit integer (`uint32_t`).
+
+i64
+ The test value must be a signed 64-bit integer (`int64_t`).
+
+u64
+ The test value must be an unsigned 64-bit integer (`uint64_t`).
+
+iptr
+ The test value must be of type `intptr_t`.
+
+uptr
+ The test value must be of type `uintptr_t`.
+
+ssz
+ The test value must be of type `ssize_t`.
+
+sz
+ The test value must be of type `size_t`.
+
+Boolean Expressions
+~~~~~~~~~~~~~~~~~~~
+
+The following test checks for boolean expressions are available:
+
+.. code-block:: c
+
+ void T_true(bool a, const char *fmt, ...);
+ void T_assert_true(bool a, const char *fmt, ...);
+ void T_quiet_true(bool a, const char *fmt, ...);
+ void T_step_true(unsigned int step, bool a, const char *fmt, ...);
+
+ void T_false(bool a, const char *fmt, ...);
+ void T_assert_false(bool a, const char *fmt, ...);
+ void T_quiet_true(bool a, const char *fmt, ...);
+ void T_step_true(unsigned int step, bool a, const char *fmt, ...);
+
+The message is only printed in case the test check fails. The format parameter
+is mandatory.
+
+.. code-block:: c
+ :caption: Boolean Test Checks Example
+
+ #include <t.h>
+
+ T_TEST_CASE(example)
+ {
+ T_true(true, "test passes, no message output");
+ T_true(false, "test fails");
+ T_quiet_true(true, "quiet test passes, no output at all");
+ T_quiet_true(false, "quiet test fails");
+ T_step_true(2, true, "step test passes, no message output");
+ T_step_true(3, false, "step test fails");
+ T_assert_false(true, "this is a format %s", "string");
+ }
+
+.. code-block:: none
+ :caption: Boolean Test Checks Report
+
+ B:example
+ P:0:0:UI1:test-example.c:5
+ F:1:0:UI1:test-example.c:6:test fails
+ F:*:0:UI1:test-example.c:8:quiet test fails
+ P:2:0:UI1:test-example.c:9
+ F:3:0:UI1:test-example.c:10:step test fails
+ F:4:0:UI1:test-example.c:11:this is a format string
+ E:example:N:5:F:4
+
+Generic Types
+~~~~~~~~~~~~~
+
+The following test checks for data types with an equality (`==`) or inequality
+(`!=`) operator are available:
+
+.. code-block:: c
+
+ void T_eq(T a, T e, const char *fmt, ...);
+ void T_assert_eq(T a, T e, const char *fmt, ...);
+ void T_quiet_eq(T a, T e, const char *fmt, ...);
+ void T_step_eq(unsigned int step, T a, T e, const char *fmt, ...);
+
+ void T_ne(T a, T e, const char *fmt, ...);
+ void T_assert_ne(T a, T e, const char *fmt, ...);
+ void T_quiet_ne(T a, T e, const char *fmt, ...);
+ void T_step_ne(unsigned int step, T a, T e, const char *fmt, ...);
+
+The type name `T` specifies an arbitrary type which must support the
+corresponding operator. The message is only printed in case the test check
+fails. The format parameter is mandatory.
+
+Pointers
+~~~~~~~~
+
+The following test checks for pointers are available:
+
+.. code-block:: c
+
+ void T_eq_ptr(const void *a, const void *e);
+ void T_assert_eq_ptr(const void *a, const void *e);
+ void T_quiet_eq_ptr(const void *a, const void *e);
+ void T_step_eq_ptr(unsigned int step, const void *a, const void *e);
+
+ void T_ne_ptr(const void *a, const void *e);
+ void T_assert_ne_ptr(const void *a, const void *e);
+ void T_quiet_ne_ptr(const void *a, const void *e);
+ void T_step_ne_ptr(unsigned int step, const void *a, const void *e);
+
+ void T_null(const void *a);
+ void T_assert_null(const void *a);
+ void T_quiet_null(const void *a);
+ void T_step_null(unsigned int step, const void *a);
+
+ void T_not_null(const void *a);
+ void T_assert_not_null(const void *a);
+ void T_quiet_not_null(const void *a);
+ void T_step_not_null(unsigned int step, const void *a);
+
+An automatically generated message is printed in case the test check fails.
+
+Memory Areas
+~~~~~~~~~~~~
+
+The following test checks for memory areas are available:
+
+.. code-block:: c
+
+ void T_eq_mem(const void *a, const void *e, size_t n);
+ void T_assert_eq_mem(const void *a, const void *e, size_t n);
+ void T_quiet_eq_mem(const void *a, const void *e, size_t n);
+ void T_step_eq_mem(unsigned int step, const void *a, const void *e, size_t n);
+
+ void T_ne_mem(const void *a, const void *e, size_t n);
+ void T_assert_ne_mem(const void *a, const void *e, size_t n);
+ void T_quiet_ne_mem(const void *a, const void *e, size_t n);
+ void T_step_ne_mem(unsigned int step, const void *a, const void *e, size_t n);
+
+The `memcmp()` function is used to compare the memory areas. An automatically
+generated message is printed in case the test check fails.
+
+Strings
+~~~~~~~
+
+The following test checks for strings are available:
+
+.. code-block:: c
+
+ void T_eq_str(const char *a, const char *e);
+ void T_assert_eq_str(const char *a, const char *e);
+ void T_quiet_eq_str(const char *a, const char *e);
+ void T_step_eq_str(unsigned int step, const char *a, const char *e);
+
+ void T_ne_str(const char *a, const char *e);
+ void T_assert_ne_str(const char *a, const char *e);
+ void T_quiet_ne_str(const char *a, const char *e);
+ void T_step_ne_str(unsigned int step, const char *a, const char *e);
+
+ void T_eq_nstr(const char *a, const char *e, size_t n);
+ void T_assert_eq_nstr(const char *a, const char *e, size_t n);
+ void T_quiet_eq_nstr(const char *a, const char *e, size_t n);
+ void T_step_eq_nstr(unsigned int step, const char *a, const char *e, size_t n);
+
+ void T_ne_nstr(const char *a, const char *e, size_t n);
+ void T_assert_ne_nstr(const char *a, const char *e, size_t n);
+ void T_quiet_ne_nstr(const char *a, const char *e, size_t n);
+ void T_step_ne_nstr(unsigned int step, const char *a, const char *e, size_t n);
+
+The `strcmp()` and `strncmp()` functions are used to compare the strings. An
+automatically generated message is printed in case the test check fails.
+
+Characters
+~~~~~~~~~~
+
+The following test checks for characters (`char`) are available:
+
+.. code-block:: c
+
+ void T_eq_char(char a, char e);
+ void T_assert_eq_char(char a, char e);
+ void T_quiet_eq_char(char a, char e);
+ void T_step_eq_char(unsigned int step, char a, char e);
+
+ void T_ne_char(char a, char e);
+ void T_assert_ne_char(char a, char e);
+ void T_quiet_ne_char(char a, char e);
+ void T_step_ne_char(unsigned int step, char a, char e);
+
+An automatically generated message is printed in case the test check fails.
+
+Integers
+~~~~~~~~
+
+The following test checks for integers are available:
+
+.. code-block:: c
+
+ void T_eq_xyz(I a, I e);
+ void T_assert_eq_xyz(I a, I e);
+ void T_quiet_eq_xyz(I a, I e);
+ void T_step_eq_xyz(unsigned int step, I a, I e);
+
+ void T_ne_xyz(I a, I e);
+ void T_assert_ne_xyz(I a, I e);
+ void T_quiet_ne_xyz(I a, I e);
+ void T_step_ne_xyz(unsigned int step, I a, I e);
+
+ void T_ge_xyz(I a, I e);
+ void T_assert_ge_xyz(I a, I e);
+ void T_quiet_ge_xyz(I a, I e);
+ void T_step_ge_xyz(unsigned int step, I a, I e);
+
+ void T_gt_xyz(I a, I e);
+ void T_assert_gt_xyz(I a, I e);
+ void T_quiet_gt_xyz(I a, I e);
+ void T_step_gt_xyz(unsigned int step, I a, I e);
+
+ void T_le_xyz(I a, I e);
+ void T_assert_le_xyz(I a, I e);
+ void T_quiet_le_xyz(I a, I e);
+ void T_step_le_xyz(unsigned int step, I a, I e);
+
+ void T_lt_xyz(I a, I e);
+ void T_assert_lt_xyz(I a, I e);
+ void T_quiet_lt_xyz(I a, I e);
+ void T_step_lt_xyz(unsigned int step, I a, I e);
+
+The type variant `xyz` must be `schar`, `uchar`, `short`, `ushort`, `int`,
+`uint`, `long`, `ulong`, `ll`, `ull`, `i8`, `u8`, `i16`, `u16`, `i32`, `u32`,
+`i64`, `u64`, `iptr`, `uptr`, `ssz`, or `sz`.
+
+The type name `I` must be compatible to the type variant.
+
+An automatically generated message is printed in case the test check fails.
+
+RTEMS Status Codes
+~~~~~~~~~~~~~~~~~~
+
+The following test checks for RTEMS status codes are available:
+
+.. code-block:: c
+
+ void T_rsc(rtems_status_code a, rtems_status_code e);
+ void T_assert_rsc(rtems_status_code a, rtems_status_code e);
+ void T_quiet_rsc(rtems_status_code a, rtems_status_code e);
+ void T_step_rsc(unsigned int step, rtems_status_code a, rtems_status_code e);
+
+ void T_rsc_success(rtems_status_code a);
+ void T_assert_rsc_success(rtems_status_code a);
+ void T_quiet_rsc_success(rtems_status_code a);
+ void T_step_rsc_success(unsigned int step, rtems_status_code a);
+
+An automatically generated message is printed in case the test check fails.
+
+POSIX Error Numbers
+~~~~~~~~~~~~~~~~~~~
+
+The following test checks for POSIX error numbers are available:
+
+.. code-block:: c
+
+ void T_eno(int a, int e);
+ void T_assert_eno(int a, int e);
+ void T_quiet_eno(int a, int e);
+ void T_step_eno(unsigned int step, int a, int e);
+
+ void T_eno_success(int a);
+ void T_assert_eno_success(int a);
+ void T_quiet_eno_success(int a);
+ void T_step_eno_success(unsigned int step, int a);
+
+The actual and expected value must be a POSIX error number, e.g. EINVAL,
+ENOMEM, etc. An automatically generated message is printed in case the test
+check fails.
+
+POSIX Status Codes
+~~~~~~~~~~~~~~~~~~
+
+The following test checks for POSIX status codes are available:
+
+.. code-block:: c
+
+ void T_psx_error(int a, int eno);
+ void T_assert_psx_error(int a, int eno);
+ void T_quiet_psx_error(int a, int eno);
+ void T_step_psx_error(unsigned int step, int a, int eno);
+
+ void T_psx_success(int a);
+ void T_assert_psx_success(int a);
+ void T_quiet_psx_success(int a);
+ void T_step_psx_success(unsigned int step, int a);
+
+The `eno` value must be a POSIX error number, e.g. EINVAL, ENOMEM, etc. An
+actual value of zero indicates success. An actual value of minus one indicates
+an error. An automatically generated message is printed in case the test check
+fails.
+
+.. code-block:: c
+ :caption: POSIX Status Code Example
+
+ #include <t.h>
+
+ #include <sys/stat.h>
+ #include <errno.h>
+
+ T_TEST_CASE(stat)
+ {
+ struct stat st;
+ int status;
+
+ errno = 0;
+ status = stat("foobar", &st);
+ T_psx_error(status, ENOENT);
+ }
+
+.. code-block:: none
+ :caption: POSIX Status Code Report
+
+ B:stat
+ P:0:0:UI1:test-psx.c:13
+ E:stat:N:1:F:0
+
+Custom Log Messages
+-------------------
+
+You can print custom log messages with the `T_log()` function:
+
+.. code-block:: c
+
+ void T_log(T_verbosity verbosity, char const *fmt, ...);
+
+A newline is automatically added to terminate the log message line.
+
+.. code-block:: c
+ :caption: Custom Log Message Example
+
+ #include <t.h>
+
+ T_TEST_CASE(log)
+ {
+ T_log(T_NORMAL, "a custom message %i, %i, %i", 1, 2, 3);
+ T_set_verbosity(T_QUIET);
+ T_log(T_NORMAL, "not verbose enough");
+ }
+
+.. code-block:: none
+ :caption: Custom Log Message Report
+
+ B:log
+ L:a custom message 1, 2, 3
+ E:log:N:0:F:0
+
+Time Services
+-------------
+
+The test framework provides two unsigned integer types for time values. The
+`T_ticks` unsigned integer type is used by the `T_tick()` function which
+measures time using the highest frequency counter available on the platform.
+It should only be used to measure small time intervals. The `T_time` unsigned
+integer type is used by the `T_now()` function which returns the current
+monotonic clock value of the platform, e.g. `CLOCK_MONOTONIC`.
+
+.. code-block:: c
+
+ T_ticks T_tick(void);
+
+ T_time T_now(void);
+
+The reference time point for these two clocks is unspecified. You can obtain
+the test case begin time with the `T_case_begin_time()` function.
+
+.. code-block:: c
+
+ T_time T_case_begin_time(void);
+
+You can convert time into ticks with the `T_time_to_ticks()` function and vice
+versa with the `T_ticks_to_time()` function.
+
+.. code-block:: c
+
+ T_time T_ticks_to_time(T_ticks ticks);
+
+ T_ticks T_time_to_ticks(T_time time);
+
+You can convert seconds and nanoseconds values into a combined time value with
+the `T_seconds_and_nanoseconds_to_time()` function. You can convert a time
+value into separate seconds and nanoseconds values with the
+`T_time_to_seconds_and_nanoseconds()` function.
+
+.. code-block:: c
+
+ T_time T_seconds_and_nanoseconds_to_time(uint32_t s, uint32_t ns);
+
+ void T_time_to_seconds_and_nanoseconds(T_time time, uint32_t *s, uint32_t *ns);
+
+You can convert a time value into a string represention. The time unit of the
+string representation is seconds. The precision of the string represention may
+be nanoseconds, microseconds, milliseconds, or seconds. You have to provide a
+buffer for the string (`T_time_string`).
+
+.. code-block:: c
+
+ const char *T_time_to_string_ns(T_time time, T_time_string buffer);
+
+ const char *T_time_to_string_us(T_time time, T_time_string buffer);
+
+ const char *T_time_to_string_ms(T_time time, T_time_string buffer);
+
+ const char *T_time_to_string_s(T_time time, T_time_string buffer);
+
+.. code-block:: c
+ :caption: Time String Example
+
+ #include <t.h>
+
+ T_TEST_CASE(time_to_string)
+ {
+ T_time_string ts;
+ T_time t;
+ uint32_t s;
+ uint32_t ns;
+
+ t = T_seconds_and_nanoseconds_to_time(0, 123456789);
+ T_eq_str(T_time_to_string_ns(t, ts), "0.123456789");
+ T_eq_str(T_time_to_string_us(t, ts), "0.123456");
+ T_eq_str(T_time_to_string_ms(t, ts), "0.123");
+ T_eq_str(T_time_to_string_s(t, ts), "0");
+
+ T_time_to_seconds_and_nanoseconds(t, &s, &ns);
+ T_eq_u32(s, 0);
+ T_eq_u32(ns, 123456789);
+ }
+
+.. code-block:: none
+ :caption: Time String Report
+
+ B:time_to_string
+ P:0:0:UI1:test-time.c:11
+ P:1:0:UI1:test-time.c:12
+ P:2:0:UI1:test-time.c:13
+ P:3:0:UI1:test-time.c:14
+ P:4:0:UI1:test-time.c:17
+ P:5:0:UI1:test-time.c:18
+ E:time_to_string:N:6:F:0:D:0.005250
+
+You can convert a tick value into a string represention. The time unit of the
+string representation is seconds. The precision of the string represention may
+be nanoseconds, microseconds, milliseconds, or seconds. You have to provide a
+buffer for the string (`T_time_string`).
+
+.. code-block:: c
+
+ const char *T_ticks_to_string_ns(T_ticks ticks, T_time_string buffer);
+
+ const char *T_ticks_to_string_us(T_ticks ticks, T_time_string buffer);
+
+ const char *T_ticks_to_string_ms(T_ticks ticks, T_time_string buffer);
+
+ const char *T_ticks_to_string_s(T_ticks ticks, T_time_string buffer);
+
+Code Runtime Measurements
+-------------------------
+
+You can measure the runtime of code fragments in several execution environment
+variants with the `T_measure_runtime()` function. This function needs a
+context which must be created with the `T_measure_runtime_create()` function.
+The context is automatically destroyed after the test case execution.
+
+.. code-block:: c
+
+ typedef struct {
+ size_t sample_count;
+ } T_measure_runtime_config;
+
+ typedef struct {
+ const char *name;
+ int flags;
+ void (*setup)(void *arg);
+ void (*body)(void *arg);
+ void (*teardown)(void *arg);
+ void *arg;
+ } T_measure_runtime_request;
+
+ T_measure_runtime_context *T_measure_runtime_create(
+ const T_measure_runtime_config *config);
+
+ void T_measure_runtime(T_measure_runtime_context *ctx,
+ const T_measure_runtime_request *request);
+
+The runtime measurement is performed for the `body` request handler of the
+measurement request (`T_measure_runtime_request`). The optional `setup`
+request handler is called before each invocation of the `body` request handler.
+The optional `teardown` request handler is called after each invocation of the
+`body` request handler. The runtime of the operational `setup` and `teardown`
+request handlers is not measured.
+
+You can control some aspects of the measurement through the request flags (use
+zero for the default):
+
+T_MEASURE_RUNTIME_ALLOW_CLOCK_ISR
+ Allow clock interrupts during the measurement. By default, measurements
+ during which a clock interrupt happened are discarded unless it happens two
+ times in a row.
+
+T_MEASURE_RUNTIME_REPORT_SAMPLES
+ Report all measurement samples.
+
+T_MEASURE_RUNTIME_DISABLE_VALID_CACHE
+ Disable the `ValidCache` execution environment variant.
+
+T_MEASURE_RUNTIME_DISABLE_HOT_CACHE
+ Disable the `HotCache` execution environment variant.
+
+T_MEASURE_RUNTIME_DISABLE_DIRTY_CACHE
+ Disable the `DirtyCache` execution environment variant.
+
+T_MEASURE_RUNTIME_DISABLE_MINOR_LOAD
+ Disable the `Load` execution environment variants with a load worker count
+ less than the processor count.
+
+T_MEASURE_RUNTIME_DISABLE_MAX_LOAD
+ Disable the `Load` execution environment variant with a load worker count
+ equal to the processor count.
+
+The execution environment variants (`M:V`) are:
+
+ValidCache
+ Before the `body` request handler is invoked a memory area with twice the
+ size of the outer-most data cache is completely read. This fills the data
+ cache with valid cache lines which are unrelated to the `body` request
+ handler.
+
+ You can disable this variant with the
+ `T_MEASURE_RUNTIME_DISABLE_VALID_CACHE` request flag.
+
+HotCache
+ Before the `body` request handler is invoked the `body` request handler is
+ called without measuring the runtime. The aim is to load all data used by
+ the `body` request handler to the cache.
+
+ You can disable this variant with the
+ `T_MEASURE_RUNTIME_DISABLE_HOT_CACHE` request flag.
+
+DirtyCache
+ Before the `body` request handler is invoked a memory area with twice the
+ size of the outer-most data cache is completely written with new data.
+ This should produce a data cache with dirty cache lines which are unrelated
+ to the `body` request handler. In addition, the entire instruction cache
+ is invalidated.
+
+ You can disable this variant with the
+ `T_MEASURE_RUNTIME_DISABLE_DIRTY_CACHE` request flag.
+
+Load
+ This variant tries to get close to worst-case conditions. The cache is set
+ up according to the `DirtyCache` variant. In addition, other processors
+ try to fully load the memory system. The load is produced through writes
+ to a memory area with twice the size of the outer-most data cache. The
+ load variant is performed multiple times with a different set of active
+ load worker threads (`M:L`). The active workers range from one up to the
+ processor count.
+
+ You can disable these variants with the
+ `T_MEASURE_RUNTIME_DISABLE_MINOR_LOAD` and
+ `T_MEASURE_RUNTIME_DISABLE_MAX_LOAD` request flags.
+
+ On SPARC, the `body` request handler is called with a register window
+ setting so that window overflow traps will occur in the next level function
+ call.
+
+Each execution in an environment variant produces a sample set of `body`
+request handler runtime measurements. The minimum (`M:MI`), first quartile
+(`M:Q1`), median (`M:Q2`), third quartile (`M:Q3`), maximum (`M:MX`), median
+absolute deviation (`M:MAD`), and the sum of the sample values (`M:D`) is
+reported.
+
+.. code-block:: c
+ :caption: Code Runtime Measurement Example
+
+ #include <t.h>
+
+ static void
+ empty(void *arg)
+ {
+ (void)arg;
+ }
+
+ T_TEST_CASE(measure_empty)
+ {
+ static const T_measure_runtime_config config = {
+ .sample_count = 1024
+ };
+ T_measure_runtime_context *ctx;
+ T_measure_runtime_request req;
+
+ ctx = T_measure_runtime_create(&config);
+ T_assert_not_null(ctx);
+
+ memset(&req, 0, sizeof(req));
+ req.name = "Empty";
+ req.body = empty;
+ T_measure_runtime(ctx, &req);
+ }
+
+.. code-block:: none
+ :caption: Code Runtime Measurement Report
+
+ B:measure_empty
+ P:0:0:UI1:test-rtems-measure.c:18
+ M:B:Empty
+ M:V:ValidCache
+ M:N:1024
+ M:MI:0.000000000
+ M:Q1:0.000000000
+ M:Q2:0.000000000
+ M:Q3:0.000000000
+ M:MX:0.000000009
+ M:MAD:0.000000000
+ M:D:0.000000485
+ M:E:Empty:D:0.208984183
+ M:B:Empty
+ M:V:HotCache
+ M:N:1024
+ M:MI:0.000000003
+ M:Q1:0.000000003
+ M:Q2:0.000000003
+ M:Q3:0.000000003
+ M:MX:0.000000006
+ M:MAD:0.000000000
+ M:D:0.000002626
+ M:E:Empty:D:0.000017046
+ M:B:Empty
+ M:V:DirtyCache
+ M:N:1024
+ M:MI:0.000000007
+ M:Q1:0.000000007
+ M:Q2:0.000000007
+ M:Q3:0.000000008
+ M:MX:0.000000559
+ M:MAD:0.000000000
+ M:D:0.000033244
+ M:E:Empty:D:1.887834875
+ M:B:Empty
+ M:V:Load
+ M:L:1
+ M:N:1024
+ M:MI:0.000000000
+ M:Q1:0.000000002
+ M:Q2:0.000000002
+ M:Q3:0.000000003
+ M:MX:0.000000288
+ M:MAD:0.000000000
+ M:D:0.000002421
+ M:E:Empty:D:0.001798809
+ [... 22 more load variants ...]
+ M:E:Empty:D:0.021252583
+ M:B:Empty
+ M:V:Load
+ M:L:24
+ M:N:1024
+ M:MI:0.000000001
+ M:Q1:0.000000002
+ M:Q2:0.000000002
+ M:Q3:0.000000003
+ M:MX:0.000001183
+ M:MAD:0.000000000
+ M:D:0.000003406
+ M:E:Empty:D:0.015188063
+ E:measure_empty:N:1:F:0:D:14.284869
+
+
+Test Runner
+-----------
+
+You can call the `T_main()` function to run all registered test cases.
+
+.. code-block:: c
+
+ int T_main(const T_config *config);
+
+The `T_main()` function returns 0 if all test cases passed, otherwise it
+returns 1. Concurrent execution of the `T_main()` function is undefined
+behaviour.
+
+You can ask if you execute within the context of the test runner with the
+`T_is_runner()` function:
+
+.. code-block:: c
+
+ bool T_is_runner(void);
+
+It returns `true` if you execute within the context of the test runner (the
+context which executes for example `T_main()`). Otherwise it returns `false`,
+for example if you execute in another task, in interrupt context, nobody
+executes `T_main()`, or during system initialization on another processor.
+
+On RTEMS, you have to register the test cases with the `T_register()` function
+before you call `T_main()`. This makes it possible to run low level tests, for
+example without the operating system directly in `boot_card()` or during device
+driver initialization. On other platforms, the `T_register()` is a no
+operation.
+
+.. code-block:: c
+
+ void T_register(void);
+
+You can run test cases also individually. Use `T_run_initialize()` to
+initialize the test runner. Call `T_run_all()` to run all or `T_run_by_name()`
+to run specific registered test cases. Call `T_case_begin()` to begin a
+freestanding test case and call `T_case_end()` to finish it. Finally,
+call `T_run_finalize()`.
+
+.. code-block:: c
+
+ void T_run_initialize(const T_config *config);
+
+ void T_run_all(void);
+
+ void T_run_by_name(const char *name);
+
+ void T_case_begin(const char *name, const T_fixture *fixture);
+
+ void T_case_end(void);
+
+ bool T_run_finalize(void);
+
+The `T_run_finalize()` function returns `true` if all test cases passed,
+otherwise it returns `false`. Concurrent execution of the runner functions
+(including `T_main()`) is undefined behaviour. The configuration must be
+persistent throughout the test run.
+
+Test Verbosity
+--------------
+
+Three test verbosity levels are defined:
+
+T_QUIET
+ Only the test suite begin, system, test case end, and test suite end lines
+ are printed.
+
+T_NORMAL
+ Prints everything except passed test lines.
+
+T_VERBOSE
+ Prints everything.
+
+The test verbosity level can be set within the scope of one test case with the
+`T_set_verbosity()` function:
+
+.. code-block:: c
+
+ T_verbosity T_set_verbosity(T_verbosity new_verbosity);
+
+The function returns the previous verbosity. After the test case, the
+configured verbosity is automatically restored.
+
+An example with `T_QUIET` verbosity:
+
+ .. code-block:: none
+
+ A:xyz
+ S:Platform:RTEMS
+ [...]
+ E:a:N:2:F:1
+ E:b:N:0:F:1
+ E:c:N:1:F:1
+ E:d:N:6:F:0
+ Z:xyz:C:4:N:9:F:3
+
+The same example with `T_NORMAL` verbosity:
+
+ .. code-block:: none
+
+ A:xyz
+ S:Platform:RTEMS
+ [...]
+ B:a
+ F:1:0:UI1:test-verbosity.c:6:test fails
+ E:a:N:2:F:1
+ B:b
+ F:*:0:UI1:test-verbosity.c:12:quiet test fails
+ E:b:N:0:F:1
+ B:c
+ F:0:0:UI1:test-verbosity.c:17:this is a format string
+ E:c:N:1:F:1
+ B:d
+ E:d:N:6:F:0
+ Z:xyz:C:4:N:9:F:3
+
+The same example with `T_VERBOSE` verbosity:
+
+ .. code-block:: none
+
+ A:xyz
+ S:Platform:RTEMS
+ [...]
+ B:a
+ P:0:0:UI1:test-verbosity.c:5
+ F:1:0:UI1:test-verbosity.c:6:test fails
+ E:a:N:2:F:1
+ B:b
+ F:*:0:UI1:test-verbosity.c:12:quiet test fails
+ E:b:N:0:F:1
+ B:c
+ F:0:0:UI1:test-verbosity.c:17:this is a format string
+ E:c:N:1:F:1
+ B:d
+ P:0:0:UI1:test-verbosity.c:22
+ P:1:0:UI1:test-verbosity.c:23
+ P:2:0:UI1:test-verbosity.c:24
+ P:3:0:UI1:test-verbosity.c:25
+ P:4:0:UI1:test-verbosity.c:26
+ P:5:0:UI1:test-verbosity.c:27
+ E:d:N:6:F:0
+ Z:xyz:C:4:N:9:F:3
+
+.. _TheTTestFrameworkTestReporting:
+
+Test Reporting
+--------------
+
+The test reporting is line based which should be easy to parse with a simple
+state machine. Each line consists of a set of fields separated by colon
+characters (`:`). The first character of the line determines the line format:
+
+A
+ A test suite begin line. It has the format:
+
+ **A:<TestSuite>**
+
+ A description of the field follows:
+
+ <TestSuite>
+ The test suite name. Must not contain colon characters (`:`).
+
+S
+ A test suite system line. It has the format:
+
+ **S:<Key>:<Value>**
+
+ A description of the fields follows:
+
+ <Key>
+ A key string. Must not contain colon characters (`:`).
+
+ <Value>
+ An arbitrary key value string. May contain colon characters (`:`).
+
+B
+ A test case begin line. It has the format:
+
+ **B:<TestCase>**
+
+ A description of the field follows:
+
+ <TestCase>
+ A test case name. Must not contain colon characters (`:`).
+
+P
+ A test pass line. It has the format:
+
+ **P:<Step>:<Processor>:<Task>:<File>:<Line>**
+
+ A description of the fields follows:
+
+ <Step>
+ Each non-quiet test has a unique test step counter value in each test case
+ execution. The test step counter is set to zero before the test case
+ executes. For quiet test checks, there is no associated test step and the
+ character `*` instead of an integer is used to indicate this.
+
+ <Processor>
+ The processor index of the processor which executed at least one
+ instruction of the corresponding test.
+
+ <Task>
+ The name of the task which executed the corresponding test if the test
+ executed in task context. The name `ISR` indicates that the test executed
+ in interrupt context. The name `?` indicates that the test executed in an
+ arbitrary context with no valid executing task.
+
+ <File>
+ The name of the source file which contains the corresponding test. A
+ source file of `*` indicates that no test source file is associated
+ with the test, e.g. it was produced by the test framework itself.
+
+ <Line>
+ The line of the test statement in the source file which contains the
+ corresponding test. A line number of `*` indicates that no test source
+ file is associated with the test, e.g. it was produced by the test
+ framework itself.
+
+F
+ A test failure line. It has the format:
+
+ **F:<Step>:<Processor>:<Task>:<File>:<Line>:<Message>**
+
+ A description of the fields follows:
+
+ <Step> <Processor> <Task> <File> <Line>
+ See above **P** line.
+
+ <Message>
+ An arbitrary message string. May contain colon characters (`:`).
+
+L
+ A log message line. It has the format:
+
+ **L:<Message>**
+
+ A description of the field follows:
+
+ <Message>
+ An arbitrary message string. May contain colon characters (`:`).
+
+E
+ A test case end line. It has the format:
+
+ **E:<TestCase>:N:<Steps>:F:<Failures>:D:<Duration>**
+
+ A description of the fields follows:
+
+ <TestCase>
+ A test case name. Must not contain colon characters (`:`).
+
+ <Steps>
+ The final test step counter of a test case. Quiet test checks produce
+ no test steps.
+
+ <Failures>
+ The count of failed test checks of a test case.
+
+ <Duration>
+ The test case duration in seconds.
+
+Z
+ A test suite end line. It has the format:
+
+ **Z:<TestSuite>:C:<TestCases>:N:<OverallSteps>:F:<OverallFailures>:D:<Duration>**
+
+ A description of the fields follows:
+
+ <TestSuite>
+ The test suite name. Must not contain colon characters (`:`).
+
+ <TestCases>
+ The count of test cases in the test suite.
+
+ <OverallSteps>
+ The overall count of test steps in the test suite.
+
+ <OverallFailures>
+ The overall count of failed test cases in the test suite.
+
+ <Duration>
+ The test suite duration in seconds.
+
+M
+ A code runtime measurement line. It has the formats:
+
+ **M:B:<Name>**
+
+ **M:V:<Variant>**
+
+ **M:L:<Load>**
+
+ **M:N:<SampleCount>**
+
+ **M:S:<Count>:<Value>**
+
+ **M:MI:<Minimum>**
+
+ **M:Q1:<FirstQuartile>**
+
+ **M:Q2:<Median>**
+
+ **M:Q3:<ThirdQuartile>**
+
+ **M:MX:<Maximum>**
+
+ **M:MAD:<MedianAbsoluteDeviation>**
+
+ **M:D:<SumOfSampleValues>**
+
+ **M:E:<Name>:D:<Duration>**
+
+ A description of the fields follows:
+
+ <Name>
+ A code runtime measurement name. Must not contain colon characters
+ (`:`).
+
+ <Variant>
+ The execution variant which is one of **ValidCache**, **HotCache**,
+ **DirtyCache**, or **Load**.
+
+ <Load>
+ The active load workers count which ranges from one to the processor
+ count.
+
+ <SampleCount>
+ The sample count as defined by the runtime measurement configuration.
+
+ <Count>
+ The count of samples with the same value.
+
+ <Value>
+ A sample value in seconds.
+
+ <Minimum>
+ The minimum of the sample set in seconds.
+
+ <FirstQuartile>
+ The first quartile of the sample set in seconds.
+
+ <Median>
+ The median of the sample set in seconds.
+
+ <ThirdQuartile>
+ The third quartile of the sample set in seconds.
+
+ <Maximum>
+ The maximum of the sample set in seconds.
+
+ <MedianAbsoluteDeviation>
+ The median absolute deviation of the sample set in seconds.
+
+ <SumOfSampleValues>
+ The sum of all sample values of the sample set in seconds.
+
+ <Duration>
+ The runtime measurement duration in seconds. It includes time to set
+ up the execution environment variant.
+
+.. code-block:: none
+ :caption: Example Test Report
+
+ A:xyz
+ S:Platform:RTEMS
+ S:Compiler:7.3.0 20180125 (RTEMS 5, RSB 30da0c720b78eba16a3f5272206c07415368617b, Newlib 2ab57ad59bc35dafffa69cd4da5e228971de069f)
+ S:Version:5.0.0.af5840fbb3d5e5aae24af3da84c75a011098f086
+ S:BSP:erc32
+ S:RTEMS_DEBUG:0
+ S:RTEMS_MULTIPROCESSING:0
+ S:RTEMS_POSIX_API:1
+ S:RTEMS_PROFILING:0
+ S:RTEMS_SMP:0
+ B:rsc
+ P:0:0:UI1:test-rtems.c:33
+ F:1:0:UI1:test-rtems.c:34:RTEMS_INVALID_NUMBER == RTEMS_INVALID_ID
+ F:*:0:UI1:test-rtems.c:36:RTEMS_INVALID_NUMBER == RTEMS_INVALID_ID
+ P:2:0:UI1:test-rtems.c:37
+ F:3:0:UI1:test-rtems.c:38:RTEMS_INVALID_NUMBER == RTEMS_INVALID_ID
+ E:rsc:N:4:F:3
+ B:rsc_success
+ P:0:0:UI1:test-rtems.c:44
+ F:1:0:UI1:test-rtems.c:45:RTEMS_INVALID_NUMBER == RTEMS_SUCCESSFUL
+ F:*:0:UI1:test-rtems.c:47:RTEMS_INVALID_NUMBER == RTEMS_SUCCESSFUL
+ P:2:0:UI1:test-rtems.c:48
+ F:3:0:UI1:test-rtems.c:49:RTEMS_INVALID_NUMBER == RTEMS_SUCCESSFUL
+ E:rsc_success:N:4:F:3
+ B:timer
+ P:0:0:UI1:test-rtems.c:74
+ P:1:0:UI1:test-rtems.c:78
+ P:2:0:ISR:test-rtems.c:61
+ P:3:0:UI1:test-rtems.c:83
+ P:4:0:UI1:test-rtems.c:84
+ P:5:0:UI1:test-rtems.c:87
+ E:timer:N:6:F:0
+ Z:xyz:C:3:N:14:F:6
+
+Supported Platforms
+-------------------
+
+The framework runs on FreeBSD, MSYS2, Linux and RTEMS.
+
+Test Framework Requirements for RTEMS
+=====================================
+
+The requirements on a test framework suitable for RTEMS are:
+
+License Requirements
+--------------------
+
+TF.License.Permissive
+ The test framework shall have a permissive open source license such as
+ BSD-2-Clause.
+
+Portability Requirements
+------------------------
+
+TF.Portability
+ The test framework shall be portable.
+
+ TF.Portability.RTEMS
+ The test framework shall run on RTEMS.
+
+ TF.Portability.POSIX
+ The test framework shall be portable to POSIX compatible operating
+ systems. This allows to run test cases of standard C/POSIX/etc. APIs
+ on multiple platforms.
+
+ TF.Portability.POSIX.Linux
+ The test framework shall run on Linux.
+
+ TF.Portability.POSIX.FreeBSD
+ The test framework shall run on FreeBSD.
+
+ TF.Portability.C11
+ The test framework shall be written in C11.
+
+ TF.Portability.Static
+ Test framework shall not use dynamic memory for basic services.
+
+ TF.Portability.Small
+ The test framework shall be small enough to support low-end platforms
+ (e.g. 64KiB of RAM/ROM should be sufficient to test the architecture
+ port, e.g. no complex stuff such as file systems, etc.).
+
+ TF.Portability.Small.LinkTimeConfiguration
+ The test framework shall be configured at link-time.
+
+ TF.Portability.Small.Modular
+ The test framework shall be modular so that only necessary parts end up
+ in the final executable.
+
+ TF.Portability.Small.Memory
+ The test framework shall not aggregate data during test case executions.
+
+Reporting Requirements
+----------------------
+
+TF.Reporting
+ Test results shall be reported.
+
+ TF.Reporting.Verbosity
+ The test report verbosity shall be configurable. This allows different
+ test run scenarios, e.g. regression test runs, full test runs with test
+ report verification against the planned test output.
+
+ TF.Reporting.Verification
+ It shall be possible to use regular expressions to verify test reports
+ line by line.
+
+ TF.Reporting.Compact
+ Test output shall be compact to avoid long test runs on platforms with
+ a slow output device, e.g. 9600 Baud UART.
+
+ TF.Reporting.PutChar
+ A simple output one character function provided by the platform shall be
+ sufficient to report the test results.
+
+ TF.Reporting.NonBlocking
+ The ouptut functions shall be non-blocking.
+
+ TF.Reporting.Printf
+ The test framework shall provide printf()-like output functions.
+
+ TF.Reporting.Printf.WithFP
+ There shall be a printf()-like output function with floating point
+ support.
+
+ TF.Reporting.Printf.WithoutFP
+ There shall be a printf()-like output function without floating
+ point support on RTEMS.
+
+ TF.Reporting.Platform
+ The test platform shall be reported.
+
+ TF.Reporting.Platform.RTEMS.Git
+ The RTEMS source Git commit shall be reported.
+
+ TF.Reporting.Platform.RTEMS.Arch
+ The RTEMS architecture name shall be reported.
+
+ TF.Reporting.Platform.RTEMS.BSP
+ The RTEMS BSP name shall be reported.
+
+ TF.Reporting.Platform.RTEMS.Tools
+ The RTEMS tool chain version shall be reported.
+
+ TF.Reporting.Platform.RTEMS.Config.Debug
+ The shall be reported if RTEMS_DEBUG is defined.
+
+ TF.Reporting.Platform.RTEMS.Config.Multiprocessing
+ The shall be reported if RTEMS_MULTIPROCESSING is defined.
+
+ TF.Reporting.Platform.RTEMS.Config.POSIX
+ The shall be reported if RTEMS_POSIX_API is defined.
+
+ TF.Reporting.Platform.RTEMS.Config.Profiling
+ The shall be reported if RTEMS_PROFILING is defined.
+
+ TF.Reporting.Platform.RTEMS.Config.SMP
+ The shall be reported if RTEMS_SMP is defined.
+
+ TF.Reporting.TestCase
+ The test cases shall be reported.
+
+ TF.Reporting.TestCase.Begin
+ The test case begin shall be reported.
+
+ TF.Reporting.TestCase.End
+ The test case end shall be reported.
+
+ TF.Reporting.TestCase.Tests
+ The count of test checks of the test case shall be reported.
+
+ TF.Reporting.TestCase.Failures
+ The count of failed test checks of the test case shall be reported.
+
+ TF.Reporting.TestCase.Timing
+ Test case timing shall be reported.
+
+ TF.Reporting.TestCase.Tracing
+ Automatic tracing and reporting of thread context switches and
+ interrupt service routines shall be optionally performed.
+
+Environment Requirements
+------------------------
+
+TF.Environment
+ The test framework shall support all environment conditions of the platform.
+
+ TF.Environment.SystemStart
+ The test framework shall run during early stages of the system start,
+ e.g. valid stack pointer, initialized data and cleared BSS, nothing
+ more.
+
+ TF.Environment.BeforeDeviceDrivers
+ The test framework shall run before device drivers are initialized.
+
+ TF.Environment.InterruptContext
+ The test framework shall support test case code in interrupt context.
+
+Usability Requirements
+----------------------
+
+TF.Usability
+ The test framework shall be easy to use.
+
+ TF.Usability.TestCase
+ It shall be possible to write test cases.
+
+ TF.Usability.TestCase.Independence
+ It shall be possible to write test cases in modules independent of
+ the test runner.
+
+ TF.Usability.TestCase.AutomaticRegistration
+ Test cases shall be registered automatically, e.g. via constructors
+ or linker sets.
+
+ TF.Usability.TestCase.Order
+ It shall be possible to sort the registered test cases (e.g. random,
+ by name) before they are executed.
+
+ TF.Usability.TestCase.Resources
+ It shall be possible to use resources with a life time restricted to
+ the test case.
+
+ TF.Usability.TestCase.Resources.Memory
+ It shall be possible to dynamically allocate memory which is
+ automatically freed once the test case completed.
+
+ TF.Usability.TestCase.Resources.File
+ It shall be possible to create a file which is automatically
+ unlinked once the test case completed.
+
+ TF.Usability.TestCase.Resources.Directory
+ It shall be possible to create a directory which is automatically
+ removed once the test case completed.
+
+ TF.Usability.TestCase.Resources.FileDescriptor
+ It shall be possible to open a file descriptor which is
+ automatically closed once the test case completed.
+
+ TF.Usability.TestCase.Fixture
+ It shall be possible to use a text fixture for test cases.
+
+ TF.Usability.TestCase.Fixture.SetUp
+ It shall be possible to provide a set up handler for each test case.
+
+ TF.Usability.TestCase.Fixture.TearDown
+ It shall be possible to provide a tear down handler for each test
+ case.
+
+ TF.Usability.TestCase.Context
+ The test case context shall be verified a certain points.
+
+ TF.Usability.TestCase.Context.VerifyAtEnd
+ After a test case exection it shall be verified that the context
+ is equal to the context at the test case begin. This helps to
+ ensure that test cases are independent of each other.
+
+ TF.Usability.TestCase.Context.VerifyThread
+ The test framework shall provide a function to ensure that the
+ test case code executes in normal thread context. This helps
+ to ensure that operating system service calls return to a sane
+ context.
+
+ TF.Usability.TestCase.Context.Configurable
+ The context verified in test case shall be configurable at link-time.
+
+ TF.Usability.TestCase.Context.ThreadDispatchDisableLevel
+ It shall be possible to verify the thread dispatch disable level.
+
+ TF.Usability.TestCase.Context.ISRNestLevel
+ It shall be possible to verify the ISR nest level.
+
+ TF.Usability.TestCase.Context.InterruptLevel
+ It shall be possible to verify the interrupt level (interrupts
+ enabled/disabled).
+
+ TF.Usability.TestCase.Context.Workspace
+ It shall be possible to verify the workspace.
+
+ TF.Usability.TestCase.Context.Heap
+ It shall be possible to verify the heap.
+
+ TF.Usability.TestCase.Context.OpenFileDescriptors
+ It shall be possible to verify the open file descriptors.
+
+ TF.Usability.TestCase.Context.Classic
+ It shall be possible to verify Classic API objects.
+
+ TF.Usability.TestCase.Context.Classic.Barrier
+ It shall be possible to verify Classic API Barrier objects.
+
+ TF.Usability.TestCase.Context.Classic.Extensions
+ It shall be possible to verify Classic API User Extensions
+ objects.
+
+ TF.Usability.TestCase.Context.Classic.MessageQueues
+ It shall be possible to verify Classic API Message Queue
+ objects.
+
+ TF.Usability.TestCase.Context.Classic.Partitions
+ It shall be possible to verify Classic API Partition objects.
+
+ TF.Usability.TestCase.Context.Classic.Periods
+ It shall be possible to verify Classic API Rate Monotonic
+ Period objects.
+
+ TF.Usability.TestCase.Context.Classic.Regions
+ It shall be possible to verify Classic API Region objects.
+
+ TF.Usability.TestCase.Context.Classic.Semaphores
+ It shall be possible to verify Classic API Semaphore
+ objects.
+
+ TF.Usability.TestCase.Context.Classic.Tasks
+ It shall be possible to verify Classic API Task objects.
+
+ TF.Usability.TestCase.Context.Classic.Timers
+ It shall be possible to verify Classic API Timer objects.
+
+ TF.Usability.TestCase.Context.POSIX
+ It shall be possible to verify POSIX API objects.
+
+ TF.Usability.TestCase.Context.POSIX.Keys
+ It shall be possible to verify POSIX API Key objects.
+
+ TF.Usability.TestCase.Context.POSIX.KeyValuePairs
+ It shall be possible to verify POSIX API Key Value Pair
+ objects.
+
+ TF.Usability.TestCase.Context.POSIX.MessageQueues
+ It shall be possible to verify POSIX API Message Queue
+ objects.
+
+ TF.Usability.TestCase.Context.POSIX.Semaphores
+ It shall be possible to verify POSIX API Named Semaphores
+ objects.
+
+ TF.Usability.TestCase.Context.POSIX.Shms
+ It shall be possible to verify POSIX API Shared Memory
+ objects.
+
+ TF.Usability.TestCase.Context.POSIX.Threads
+ It shall be possible to verify POSIX API Thread objects.
+
+ TF.Usability.TestCase.Context.POSIX.Timers
+ It shall be possible to verify POSIX API Timer objects.
+
+ TF.Usability.Assert
+ There shall be functions to assert test objectives.
+
+ TF.Usability.Assert.Safe
+ Test assert functions shall be safe to use, e.g. assert(a == b) vs.
+ assert(a = b) vs. assert_eq(a, b).
+
+ TF.Usability.Assert.Continue
+ There shall be assert functions which allow the test case to
+ continue in case of an assertion failure.
+
+ TF.Usability.Assert.Abort
+ There shall be assert functions which abourt the test case in case
+ of an assertion failure.
+
+ TF.Usability.EasyToWrite
+ It shall be easy to write test code, e.g. avoid long namespace prefix
+ rtems_test_*.
+
+ TF.Usability.Threads
+ The test framework shall support multi-threading.
+
+ TF.Usability.Pattern
+ The test framework shall support test patterns.
+
+ TF.Usability.Pattern.Interrupts
+ The test framework shall support test cases which use interrupts,
+ e.g. spintrcritical*.
+
+ TF.Usability.Pattern.Parallel
+ The test framework shall support test cases which want to run code
+ in parallel on SMP machines.
+
+ TF.Usability.Pattern.Timing
+ The test framework shall support test cases which want to measure
+ the timing of code sections under various platform conditions, e.g.
+ dirty cache, empty cache, hot cache, with load from other
+ processors, etc..
+
+ TF.Usability.Configuration
+ The test framework shall be configurable.
+
+ TF.Usability.Configuration.Time
+ The timestamp function shall be configurable, e.g. to allow test
+ runs without a clock driver.
+
+Performance Requirements
+------------------------
+
+TF.Performance.RTEMS.No64BitDivision
+ The test framework shall not use 64-bit divisions on RTEMS.
+
+Off-the-shelf Test Frameworks
+=============================
+
+There are several
+`off-the-shelf test frameworks for C/C++ <https://en.wikipedia.org/wiki/List_of_unit_testing_frameworks#C>`_.
+The first obstacle for test frameworks is the license requirement
+(`TF.License.Permissive`).
+
+bdd-for-c
+---------
+
+In the `bdd-for-c <https://github.com/grassator/bdd-for-c>`_ framework the
+complete test suite must be contained in one file and the main function is
+generated. This violates `TF.Usability.TestCase.Independence`.
+
+CBDD
+----
+
+The `CBDD <https://github.com/nassersala/cbdd>`_ framework uses the
+`C blocks <https://clang.llvm.org/docs/BlockLanguageSpec.html>`_ extension from
+clang. This violates `TF.Portability.C11`.
+
+Google Test
+-----------
+
+`Google Test 1.8.1 <https://git.rtems.org/sebh/rtems-gtest.git/>`_
+is supported by RTEMS. Unfortunately, it is written in C++ and is to heavy
+weight for low-end platforms. Otherwise it is a nice framework.
+
+Unity
+-----
+
+The `Unity Test API <https://github.com/ThrowTheSwitch/Unity>`_ does not meet
+our requirements. There was a `discussion on the mailing list in 2013
+<https://lists.rtems.org/pipermail/devel/2013-September/004499.html>`_.
+
+Standard Test Report Formats
+============================
+
+JUnit XML
+---------
+
+A common test report format is `JUnit XML <http://llg.cubic.org/docs/junit/>`_.
+
+.. code-block:: xml
+
+ <?xml version="1.0" encoding="UTF-8" ?>
+ <testsuites id="xyz" name="abc" tests="225" failures="1262" time="0.001">
+ <testsuite id="def" name="ghi" tests="45" failures="17" time="0.001">
+ <testcase id="jkl" name="mno" time="0.001">
+ <failure message="pqr" type="stu"></failure>
+ <system-out>stdout</system-out>
+ <system-err>stderr</system-err>
+ </testcase>
+ </testsuite>
+ </testsuites>
+
+The major problem with this format is that you have to output the failure count
+of all test suites and the individual test suite before the test case output.
+You know the failure count only after a complete test run. This runs contrary
+to requirement `TF.Portability.Small.Memory`. It is also a bit verbose
+(`TF.Reporting.Compact`).
+
+It is easy to convert a full test report generated by :ref:`The T Test
+Framework <TheTTestFramework>` to the JUnit XML format.
+
+Test Anything Protocol
+----------------------
+
+The
+`Test Anything Protocol <http://testanything.org/>`_
+(TAP) is easy to consume and produce.
+
+.. code-block:: none
+
+ 1..4
+ ok 1 - Input file opened
+ not ok 2 - First line of the input valid
+ ok 3 - Read the rest of the file
+ not ok 4 - Summarized correctly # TODO Not written yet
+
+You have to know in advance how many test statements you want to execute in a
+test case. The problem with this format is that there is no standard way to
+provide auxiliary data such as test timing or a tracing report.
+
+It is easy to convert a full test report generated by :ref:`The T Test
+Framework <TheTTestFramework>` to the TAP format.
--
2.16.4
More information about the devel
mailing list