BSP-Specific Testing was: [PATCH 00/34] Integrate pre-qualified LEON3 BSP

Wed Jun 14 16:13:58 UTC 2023

On Tue, Jun 13, 2023 at 8:56 AM Sebastian Huber
<sebastian.huber at embedded-brains.de> wrote:
>
>
>
> On 12.06.23 17:58, Gedare Bloom wrote:
> > On Mon, Jun 12, 2023 at 12:56 AM Sebastian Huber
> > <sebastian.huber at embedded-brains.de> wrote:
> >>
> >>
> >>
> >> On 01.06.23 22:06, Gedare Bloom wrote:
> >>> On Thu, Jun 1, 2023 at 2:00 PM Gedare Bloom<gedare at rtems.org>  wrote:
> >>>> ---------- Forwarded message ---------
> >>>> From: Sebastian Huber<sebastian.huber at embedded-brains.de>
> >>>> Date: Wed, May 31, 2023 at 10:31 AM
> >>>>
> >>>> The existing tests in the RTEMS test suite are basically BSP
> >>>> independent. This patch set introduces BSP-specific validation tests.
> >>>> These tests are disabled for other BSPs through the build system, for
> >>>> example:
> >>>>
> >>>> spec/build/testsuites/validation/bsp-sparc-leon3-gr712rc.yml
> >>>> [...]
> >>>> cxxflags: []
> >>>> enabled-by: sparc/gr712rc
> >>>> features: c cprogram
> >>>> [...]
> >>> The use of the enabled-by: field to control the BSP-specific tests
> >>> looks reasonable. However, we should decide where/how any BSP-specific
> >>> tests should reside. It looks to me like the current approach is to
> >>> dump all test files in a single monolithic 'validation' directory, and
> >>> let the user/script sort it out. This results in a mix of tests
> >>> intended for all targets, and some for specific targets. This is
> >>> pretty much non-maintainable from my point-of-view without some
> >>> additional tool support. Correct me if I'm wrong.
> >>
> >> Yes, this is the current approach. There is no strict one-to-one
> >> relationship of test cases and test suites. The file names are somewhat
> >> descriptive, for example:
> >>
> >> ls -l *leon3*
> >> -rw-r--r-- 1 sebastian_h domain users 3739 31. Mai 10:44
> >> tc-bsp-sparc-leon3-gr712rc.c
> >> -rw-r--r-- 1 sebastian_h domain users 6138 31. Mai 10:44
> >> tc-fatal-bsp-sparc-leon3-shutdown.c
> >> -rw-r--r-- 1 sebastian_h domain users 5135 31. Mai 10:44
> >> tr-fatal-bsp-sparc-leon3-cache-snooping-disabled-boot.c
> >> -rw-r--r-- 1 sebastian_h domain users 2723 31. Mai 10:44
> >> tr-fatal-bsp-sparc-leon3-cache-snooping-disabled-boot.h
> >> -rw-r--r-- 1 sebastian_h domain users 5406 31. Mai 10:44
> >> tr-fatal-bsp-sparc-leon3-cache-snooping-disabled-secondary.c
> >> -rw-r--r-- 1 sebastian_h domain users 2753 31. Mai 10:44
> >> tr-fatal-bsp-sparc-leon3-cache-snooping-disabled-secondary.h
> >> -rw-r--r-- 1 sebastian_h domain users 5270 31. Mai 10:44
> >> tr-fatal-bsp-sparc-leon3-clock-initialization.c
> >> -rw-r--r-- 1 sebastian_h domain users 2681 31. Mai 10:44
> >> tr-fatal-bsp-sparc-leon3-clock-initialization.h
> >> -rw-r--r-- 1 sebastian_h domain users 2488 31. Mai 10:44
> >> ts-bsp-sparc-leon3-gr712rc.c
> >> -rw-r--r-- 1 sebastian_h domain users 2845 31. Mai 10:44
> >> ts-fatal-bsp-sparc-leon3-cache-snooping-disabled-boot.c
> >> -rw-r--r-- 1 sebastian_h domain users 2919 31. Mai 10:44
> >> ts-fatal-bsp-sparc-leon3-cache-snooping-disabled-secondary.c
> >> -rw-r--r-- 1 sebastian_h domain users 2797 31. Mai 10:44
> >> ts-fatal-bsp-sparc-leon3-clock-initialization.c
> >> -rw-r--r-- 1 sebastian_h domain users 3148 31. Mai 10:45
> >> ts-fatal-bsp-sparc-leon3-shutdown-response.c
> >> -rw-r--r-- 1 sebastian_h domain users 4909 31. Mai 10:44
> >> ts-fatal-bsp-sparc-leon3-shutdown.c
> >>
> >> We could also introduce subdirectories to organize things. The test
> >> framework prints out the file name in messages, so it would be nice if
> >> they remain unique. With subdirectories this would lead to longer path
> >> names, for example
> >>
> >> testsuites/validation/sparc/leon3/tc-bsp-sparc-leon3-gr712rc.c
> >>
> > I see. Maybe it makes sense to have all BSP-specific tests in a bsps
> > subdirectory, with the unique names encoded to ensure the arch/bsp
> > combination appears in the filename, such as:
> > testsuites/validation/bsps/tc-bsp-sparc-leon3-gr712rc.c
> > If so, 'bsp-' can probably be omitted from the filename. This way,
> > architecture-specific testing may also be easily possible, like
> > tc-sparc-something.c
>
> This is a nicely balanced and easy approach. Place BSP-specific tests in
> a "bsps" directory. Place architecture-specific tests in a "cpu" directory.
>
> >
> >>>
> >>> I would like to discuss possible ways to manage the integration of
> >>> tests that are conditionally-built based on the arch/bsp tuple. We
> >>> should have clear guidance for others who want to add such tests in
> >>> the future, or who would modify existing tests.
> >>
> >> Yes, this makes sense. We could add a new section for BSP-specific tests to:
> >>
> >> https://docs.rtems.org/branches/master/eng/req/howto.html
> >>
> >> For a pre-qualified BSP you have to specify the fatal errors and write
> >> validation tests for it. Other BSP-specific specification and validation
> >> may be necessary for the kernel IO device driver, cache controller
> >> support, memory management unit initialization, memory protection unit
> >> initialization, etc.
> >>
> > Good, that would definitely be important to document.
>
> Ok, I will add something to this section.
>
> >
> > I think we should have some of the BSP-specific tests under some other
> > location in testsuites, while others would be under validation where
> > they are used for pre-qualification?
> >
> > Maybe it is sensible to introduce testsuites/bsps/ also.
>
> For the unit tests I would use testsuites/unit/bsps and testsuites/unit/cpu.
>
I like this.

> >
> > I'm not entirely clear about the difference between 'validation' and
> > any other kind of tests, such as where specific tests should be
> > located.
>
> The validation tests are linked to a specification item.
>
Ok

> >
> >> Even for normal BSPs, some unit tests would be helpful. For example, I
> >> had to fix _AArch32_PMSA_Initialize() twice:
> >>
> >> https://github.com/RTEMS/rtems/commits/master/cpukit/score/cpu/arm/aarch32-psma-init.c
> >>
> >> Unit tests would have probably found the errors before applications hit
> >> the special cases.
> >>
> > Something like testsuites/bsps/arm-aarch32-psma-init.c (or whatever
> > file naming convention we like to decide on. I forget the meaning of
> > tc/tr/ts/tx but recall they are about specific kinds of testing for
> > validation.)
>
> tc - Test case
>
> ts - Test suite
>
> tr - Test runner
>
> tx - Test extensions (support code)
>
Thanks

Gedare

> >
> > Gedare
> >
> >> --
> >> embedded brains GmbH
> >> Herr Sebastian HUBER
> >> Dornierstr. 4
> >> 82178 Puchheim
> >> Germany
> >> email: sebastian.huber at embedded-brains.de
> >> phone: +49-89-18 94 741 - 16
> >> fax:   +49-89-18 94 741 - 08
> >>
> >> Registergericht: Amtsgericht München
> >> Registernummer: HRB 157899
> >> Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler
> >> Unsere Datenschutzerklärung finden Sie hier:
> >> https://embedded-brains.de/datenschutzerklaerung/
>
> --
> embedded brains GmbH
> Herr Sebastian HUBER
> Dornierstr. 4
> 82178 Puchheim
> Germany
> email: sebastian.huber at embedded-brains.de
> phone: +49-89-18 94 741 - 16
> fax:   +49-89-18 94 741 - 08
>
> Registergericht: Amtsgericht München
> Registernummer: HRB 157899
> Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler
> Unsere Datenschutzerklärung finden Sie hier:
> https://embedded-brains.de/datenschutzerklaerung/