BSP-Specific Testing was: [PATCH 00/34] Integrate pre-qualified LEON3 BSP

[Sebastian Huber]

On 01.06.23 22:06, Gedare Bloom wrote:
> On Thu, Jun 1, 2023 at 2:00 PM Gedare Bloom<gedare at rtems.org>  wrote:
>> ---------- Forwarded message ---------
>> From: Sebastian Huber<sebastian.huber at embedded-brains.de>
>> Date: Wed, May 31, 2023 at 10:31 AM
>>
>> The existing tests in the RTEMS test suite are basically BSP
>> independent. This patch set introduces BSP-specific validation tests.
>> These tests are disabled for other BSPs through the build system, for
>> example:
>>
>> spec/build/testsuites/validation/bsp-sparc-leon3-gr712rc.yml
>> [...]
>> cxxflags: []
>> enabled-by: sparc/gr712rc
>> features: c cprogram
>> [...]
> The use of the enabled-by: field to control the BSP-specific tests
> looks reasonable. However, we should decide where/how any BSP-specific
> tests should reside. It looks to me like the current approach is to
> dump all test files in a single monolithic 'validation' directory, and
> let the user/script sort it out. This results in a mix of tests
> intended for all targets, and some for specific targets. This is
> pretty much non-maintainable from my point-of-view without some
> additional tool support. Correct me if I'm wrong.

Yes, this is the current approach. There is no strict one-to-one 
relationship of test cases and test suites. The file names are somewhat 
descriptive, for example:

ls -l *leon3*
-rw-r--r-- 1 sebastian_h domain users 3739 31. Mai 10:44 
tc-bsp-sparc-leon3-gr712rc.c
-rw-r--r-- 1 sebastian_h domain users 6138 31. Mai 10:44 
tc-fatal-bsp-sparc-leon3-shutdown.c
-rw-r--r-- 1 sebastian_h domain users 5135 31. Mai 10:44 
tr-fatal-bsp-sparc-leon3-cache-snooping-disabled-boot.c
-rw-r--r-- 1 sebastian_h domain users 2723 31. Mai 10:44 
tr-fatal-bsp-sparc-leon3-cache-snooping-disabled-boot.h
-rw-r--r-- 1 sebastian_h domain users 5406 31. Mai 10:44 
tr-fatal-bsp-sparc-leon3-cache-snooping-disabled-secondary.c
-rw-r--r-- 1 sebastian_h domain users 2753 31. Mai 10:44 
tr-fatal-bsp-sparc-leon3-cache-snooping-disabled-secondary.h
-rw-r--r-- 1 sebastian_h domain users 5270 31. Mai 10:44 
tr-fatal-bsp-sparc-leon3-clock-initialization.c
-rw-r--r-- 1 sebastian_h domain users 2681 31. Mai 10:44 
tr-fatal-bsp-sparc-leon3-clock-initialization.h
-rw-r--r-- 1 sebastian_h domain users 2488 31. Mai 10:44 
ts-bsp-sparc-leon3-gr712rc.c
-rw-r--r-- 1 sebastian_h domain users 2845 31. Mai 10:44 
ts-fatal-bsp-sparc-leon3-cache-snooping-disabled-boot.c
-rw-r--r-- 1 sebastian_h domain users 2919 31. Mai 10:44 
ts-fatal-bsp-sparc-leon3-cache-snooping-disabled-secondary.c
-rw-r--r-- 1 sebastian_h domain users 2797 31. Mai 10:44 
ts-fatal-bsp-sparc-leon3-clock-initialization.c
-rw-r--r-- 1 sebastian_h domain users 3148 31. Mai 10:45 
ts-fatal-bsp-sparc-leon3-shutdown-response.c
-rw-r--r-- 1 sebastian_h domain users 4909 31. Mai 10:44 
ts-fatal-bsp-sparc-leon3-shutdown.c

We could also introduce subdirectories to organize things. The test 
framework prints out the file name in messages, so it would be nice if 
they remain unique. With subdirectories this would lead to longer path 
names, for example

testsuites/validation/sparc/leon3/tc-bsp-sparc-leon3-gr712rc.c

> 
> I would like to discuss possible ways to manage the integration of
> tests that are conditionally-built based on the arch/bsp tuple. We
> should have clear guidance for others who want to add such tests in
> the future, or who would modify existing tests.

Yes, this makes sense. We could add a new section for BSP-specific tests to:

https://docs.rtems.org/branches/master/eng/req/howto.html

For a pre-qualified BSP you have to specify the fatal errors and write 
validation tests for it. Other BSP-specific specification and validation 
may be necessary for the kernel IO device driver, cache controller 
support, memory management unit initialization, memory protection unit 
initialization, etc.

Even for normal BSPs, some unit tests would be helpful. For example, I 
had to fix _AArch32_PMSA_Initialize() twice:

https://github.com/RTEMS/rtems/commits/master/cpukit/score/cpu/arm/aarch32-psma-init.c

Unit tests would have probably found the errors before applications hit 
the special cases.

-- 
embedded brains GmbH
Herr Sebastian HUBER
Dornierstr. 4
82178 Puchheim
Germany
email: sebastian.huber at embedded-brains.de
phone: +49-89-18 94 741 - 16
fax:   +49-89-18 94 741 - 08

Registergericht: Amtsgericht München
Registernummer: HRB 157899
Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler
Unsere Datenschutzerklärung finden Sie hier:
https://embedded-brains.de/datenschutzerklaerung/