[PATCH] user/test: Update with the latest test configuration format.

[chrisj at rtems.org]

From: Chris Johns <chrisj at rtems.org>

---
 user/testing/tests.rst | 110 +++++++++++++++++++++++++++++++++++------
 1 file changed, 96 insertions(+), 14 deletions(-)

diff --git a/user/testing/tests.rst b/user/testing/tests.rst
index 0b44d05..557cbc0 100644
--- a/user/testing/tests.rst
+++ b/user/testing/tests.rst
@@ -6,9 +6,9 @@ Test Banners
 ------------
 
 All test output banners or strings are embedded in each test and the test
-outputs them to the BSP's console as it executes. The RTEMS Tester captures the
-BSP's console and uses this information to manage the state of the executing
-test. The banner strings are:
+outputs the banners to the BSP's console as it executes. The RTEMS Tester
+captures the BSP's console and uses this information to manage the state of
+the executing test. The banner strings are:
 
 .. _test-banner-begin:
 .. index:: test begin, TEST BEGIN
@@ -65,10 +65,14 @@ test. The banner strings are:
 .. _test-states:
 .. index:: Test states
 
-Test States
------------
+Test Controls
+-------------
+
+The tests in the RTEMS kernel testsuite can be configured for each BSP. The
+expected state of the test can be set as well as any configuration parameters
+a test needs to run on a BSP.
 
-The tests states are:
+The test states are:
 
 .. index:: test state passed
 
@@ -137,17 +141,46 @@ a tested is to ``pass``. If a test is known to fail it can have it's state set
 to ``expected-fail``. Setting tests that are known to fail to ``expected-fail``
 lets everyone know a failure is not to be countered and consider a regression.
 
-Expected test states are list in test configuration files that end with the
-file extension ``.tcfg``. The testsuite supports global test configurations in
-the ``testsuite/testdata`` directory. Global test states are applied to all
-BSPs. BSPs can provide a test configuration that applies to just that BSP.
+Expected test states are listed in test configuration files
+
+Test Configuration
+^^^^^^^^^^^^^^^^^^
+
+Tests can be configured for each BSP using test configuration files. These
+files have the file extension ``.tcfg``. The testsuite supports global test
+configurations in the ``testsuite/testdata`` directory. Global test states are
+applied to all BSPs. BSPs can provide a test configuration that applies to
+just that BSP and these files can include subsets of test configurations.
+
+The configuration supports:
+
+#. Including test configuration files to allow sharing of common
+   configurations.
+
+#. Excluding tests from being built that do not build for a BSP.
+
+#. Setting the test state if it is not ``passed``.
 
-The test configuration file format is::
+#. Specifing a BSP specific build configuration for a test.
 
-  state: test test test
+The test configuration file format is:
 
-where ``test test test`` is a list of tests the state applies too. The ``state`` is one
-of:
+.. code-block:: none
+
+  state: test
+
+where the ``state`` is state of the test and ``test`` is a comma separated
+list of tests the state applies too. The configuration format is:
+
+.. code-block:: none
+
+  flags: test: configuration
+
+where ``flags`` is the type of flags being set, the ``test`` is a comma
+separate list of regular expresions that match the test the configuration
+is applied too and the ``configuration`` is the string of flags.
+
+The ``state`` is one of:
 
 ``include``
   The test list is the name of a test configuration file to include
@@ -171,7 +204,56 @@ of:
   The tests listed are benchmarks. Benchmarks are flagged and not left to
   run to completion because they may take too long.
 
+Specialized filtering using regular expressions is supported using:
+
+``rexclude``
+  The test matching the regular expression are excluded.
+
+``rinclude``
+  The tests matching the regular expression are included.
+
+By default all tests are included, specific excluded tests using the
+``exclude`` state are excluded and cannot be included again. If a test
+matching a ``rexclude`` regular it is excluded unless it is included using a
+``rinclude`` regular expression. For example to build only the ``hello``
+sample application you can:
+
+.. code-block:: none
+
+  rexclude .*
+  rinclude hello
+
+Test configuration flags can be applied to a range of tests using
+flags. Currently only ``cflags`` is support. Tests need to support the
+configuration for it to work. For example to configure a test:
+
+.. code-block:: none
+
+  cflags: tm.*: -DTEST_CONFIG=42
+  cflags: sp0[456]: -DA_SET_OF_TESTS=1
+
+Flags setting are joined together and passed to the compiler's command
+line. For example:
+
+.. code-block:: none
+
+  cflags: tm.*: -DTEST_CONFIG=42
+  cflags: tm03: -DTEST_TM03_CONFIG=1
+
+would result in the command line to the test ``tm03`` being:
+
+.. code-block:: none
+
+  -DTEST_CONFIG=42 -DTEST_TM03_CONFIG=1
+
+Specific flags can be set for one test in a group. For example to set
+a configuration for all timer tests and a special configuraiton for
+one test:
+
+.. code-block:: none
 
+  cflags: (?!tm02)tm.*: -DTEST_CONFIG=one
+  cflags: tm02: -DTEST_CONFIG=two
 
 Test Builds
 -----------
-- 
2.19.1