[PATCH] eng: Add guidelines for validation tests

[Frank Kühndel]

On 6/15/23 09:33, Sebastian Huber wrote:
> Subject:
> [PATCH] eng: Add guidelines for validation tests
> From:
> Sebastian Huber <sebastian.huber at embedded-brains.de>
> Date:
> 6/15/23, 09:33
> 
> To:
> devel at rtems.org
> 
> 
> Update #3717.
> ---
>   eng/req/howto.rst | 59 +++++++++++++++++++++++++++++++++++++++++++++++
>   1 file changed, 59 insertions(+)
> 
> diff --git a/eng/req/howto.rst b/eng/req/howto.rst
> index 0de3c01..688f693 100644
> --- a/eng/req/howto.rst
> +++ b/eng/req/howto.rst
> @@ -1127,6 +1127,65 @@ the following post-condition states.
>             parameter in past calls to ${../if/directive:/name} shall not be
>             accessed by the ${../if/directive:/name} call.
>   
> +Validation Test Guidelines
> +--------------------------
> +
> +The validation test cases, test runners, and test suites are generated by the
> +``./spec2modules.py`` script from specification items.  For the placement and
> +naming of the generated sources use the following rules:
> +
> +* Place architecture-specific validation test sources and programs into the
> +  ``testsuites/validation/cpu`` directory.
> +
> +* Place BSP-specific validation test sources and programs into the
> +  ``testsuites/validation/bsps`` directory.
> +
> +* Place all other validation test sources and programs into the
> +  ``testsuites/validation`` directory.
> +
> +* Place architecture-specific unit test sources and programs into the
> +  ``testsuites/unit/cpu`` directory.
> +
> +* Place BSP-specific unit test sources and programs into the
> +  ``testsuites/unit/bsps`` directory.
> +
> +* Place all other unit test sources and programs into the
> +  ``testsuites/unit`` directory.
> +
> +* Use dashes (``-``) to separate parts of a file name.  Use only dashes, the
> +  digits ``0`` to ``9``, and the lower case characters ``a`` to ``z`` for file
> +  names.  In particular, do not use underscores (``_``).
> +
> +* Use the prefix ``tc-`` for test case files.
> +
> +* Use the prefix ``tr-`` for test runner files.
> +
> +* Use the prefix ``ts-`` for test suite files.
> +
> +* Use the prefix ``tx-`` for test extension files (test support code).
> +
> +* Use hierarchical parts to compose a file name.  The order of the parts from
> +  general to specific shall be left to right, for example
> +  ``tc-task-construct.c``.

When reading "hierarchical parts" it was not clear to me what they 
meant. Maybe the sentence could be dropped and the next sentence 
rephrased into something like: "The parts separated by dashes shall be 
ordered from most general (left) to more specific (right), for example"

> +
> +* Tests for fatal errors shall have ``fatal`` as the most general file part,
> +  for example ``ts-fatal-too-large-tls-size.c``.
> +
> +* Validation test suites shall have ``validation`` as the most general file
> +  part, for example ``ts-validation-no-clock-0.c``.
> +
> +* Unit test suites shall have ``unit`` as the most general file part, for
> +  example ``ts-unit-no-clock-0.c``.
> +
> +* Architecture-specific files shall have the architecture name as a file part,
> +  for example ``ts-fatal-sparc-leon3-clock-initialization.c``.

I wonder whether ``ts-sparc-leon3-fatal-clock-initialization.c`` would 
be more logical?

> +
> +* BSP-specific files shall have the BSP family or variant name as a file part,
> +  for example ``tc-sparc-gr712rc.c``.
> +
> +* Architecture-specific or BSP-specific tests shall use the ``enabled-by``
> +  attribute of the associated build item to make them conditional.

"for example ``enabled-by: sparc/gr712rc``"?

"specification item" instead of "build item" to be consistent with the 
first sentence below the headline of this new section?

Is an ``enabled-by`` clause also needed for the architecture specific files?

> +
>   Verify the Specification Items
>   ------------------------------
>   
> -- 2.35.3 _______________________________________________ devel mailing 
> list devel at rtems.org http://lists.rtems.org/mailman/listinfo/devel
> 

-- 
embedded brains GmbH & Co. KG
Herr Frank KÜHNDEL
Dornierstr. 4
82178 Puchheim
Germany
email: frank.kuehndel at embedded-brains.de
phone:  +49-89-18 94 741 - 23
mobile: +49-176-15 22 06 - 11

Registergericht: Amtsgericht München
Registernummer: HRA 117265
Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler
Unsere Datenschutzerklärung finden Sie hier:
https://embedded-brains.de/datenschutzerklaerung/