BSP-Specific Testing was: [PATCH 00/34] Integrate pre-qualified LEON3 BSP

[Gedare Bloom]

On Thu, Jun 1, 2023 at 2:00 PM Gedare Bloom <gedare at rtems.org> wrote:
>
> ---------- Forwarded message ---------
> From: Sebastian Huber <sebastian.huber at embedded-brains.de>
> Date: Wed, May 31, 2023 at 10:31 AM
>
> The existing tests in the RTEMS test suite are basically BSP
> independent. This patch set introduces BSP-specific validation tests.
> These tests are disabled for other BSPs through the build system, for
> example:
>
> spec/build/testsuites/validation/bsp-sparc-leon3-gr712rc.yml
> [...]
> cxxflags: []
> enabled-by: sparc/gr712rc
> features: c cprogram
> [...]

The use of the enabled-by: field to control the BSP-specific tests
looks reasonable. However, we should decide where/how any BSP-specific
tests should reside. It looks to me like the current approach is to
dump all test files in a single monolithic 'validation' directory, and
let the user/script sort it out. This results in a mix of tests
intended for all targets, and some for specific targets. This is
pretty much non-maintainable from my point-of-view without some
additional tool support. Correct me if I'm wrong.

I would like to discuss possible ways to manage the integration of
tests that are conditionally-built based on the arch/bsp tuple. We
should have clear guidance for others who want to add such tests in
the future, or who would modify existing tests.

Gedare