BSP-Specific Testing was: [PATCH 00/34] Integrate pre-qualified LEON3 BSP

Thu Jun 15 13:51:42 UTC 2023

On Mon, Jun 12, 2023 at 10:58 AM Gedare Bloom <gedare at rtems.org> wrote:

> On Mon, Jun 12, 2023 at 12:56 AM Sebastian Huber
> <sebastian.huber at embedded-brains.de> wrote:
> >
> >
> >
> > On 01.06.23 22:06, Gedare Bloom wrote:
> > > On Thu, Jun 1, 2023 at 2:00 PM Gedare Bloom<gedare at rtems.org>  wrote:
> > >> ---------- Forwarded message ---------
> > >> From: Sebastian Huber<sebastian.huber at embedded-brains.de>
> > >> Date: Wed, May 31, 2023 at 10:31 AM
> > >>
> > >> The existing tests in the RTEMS test suite are basically BSP
> > >> independent. This patch set introduces BSP-specific validation tests.
> > >> These tests are disabled for other BSPs through the build system, for
> > >> example:
> > >>
> > >> spec/build/testsuites/validation/bsp-sparc-leon3-gr712rc.yml
> > >> [...]
> > >> cxxflags: []
> > >> enabled-by: sparc/gr712rc
> > >> features: c cprogram
> > >> [...]
> > > The use of the enabled-by: field to control the BSP-specific tests
> > > looks reasonable. However, we should decide where/how any BSP-specific
> > > tests should reside. It looks to me like the current approach is to
> > > dump all test files in a single monolithic 'validation' directory, and
> > > let the user/script sort it out. This results in a mix of tests
> > > intended for all targets, and some for specific targets. This is
> > > pretty much non-maintainable from my point-of-view without some
> > > additional tool support. Correct me if I'm wrong.
> >
> > Yes, this is the current approach. There is no strict one-to-one
> > relationship of test cases and test suites. The file names are somewhat
> > descriptive, for example:
> >
> > ls -l *leon3*
> > -rw-r--r-- 1 sebastian_h domain users 3739 31. Mai 10:44
> > tc-bsp-sparc-leon3-gr712rc.c
> > -rw-r--r-- 1 sebastian_h domain users 6138 31. Mai 10:44
> > tc-fatal-bsp-sparc-leon3-shutdown.c
> > -rw-r--r-- 1 sebastian_h domain users 5135 31. Mai 10:44
> > tr-fatal-bsp-sparc-leon3-cache-snooping-disabled-boot.c
> > -rw-r--r-- 1 sebastian_h domain users 2723 31. Mai 10:44
> > tr-fatal-bsp-sparc-leon3-cache-snooping-disabled-boot.h
> > -rw-r--r-- 1 sebastian_h domain users 5406 31. Mai 10:44
> > tr-fatal-bsp-sparc-leon3-cache-snooping-disabled-secondary.c
> > -rw-r--r-- 1 sebastian_h domain users 2753 31. Mai 10:44
> > tr-fatal-bsp-sparc-leon3-cache-snooping-disabled-secondary.h
> > -rw-r--r-- 1 sebastian_h domain users 5270 31. Mai 10:44
> > tr-fatal-bsp-sparc-leon3-clock-initialization.c
> > -rw-r--r-- 1 sebastian_h domain users 2681 31. Mai 10:44
> > tr-fatal-bsp-sparc-leon3-clock-initialization.h
> > -rw-r--r-- 1 sebastian_h domain users 2488 31. Mai 10:44
> > ts-bsp-sparc-leon3-gr712rc.c
> > -rw-r--r-- 1 sebastian_h domain users 2845 31. Mai 10:44
> > ts-fatal-bsp-sparc-leon3-cache-snooping-disabled-boot.c
> > -rw-r--r-- 1 sebastian_h domain users 2919 31. Mai 10:44
> > ts-fatal-bsp-sparc-leon3-cache-snooping-disabled-secondary.c
> > -rw-r--r-- 1 sebastian_h domain users 2797 31. Mai 10:44
> > ts-fatal-bsp-sparc-leon3-clock-initialization.c
> > -rw-r--r-- 1 sebastian_h domain users 3148 31. Mai 10:45
> > ts-fatal-bsp-sparc-leon3-shutdown-response.c
> > -rw-r--r-- 1 sebastian_h domain users 4909 31. Mai 10:44
> > ts-fatal-bsp-sparc-leon3-shutdown.c
> >
> > We could also introduce subdirectories to organize things. The test
> > framework prints out the file name in messages, so it would be nice if
> > they remain unique. With subdirectories this would lead to longer path
> > names, for example
> >
> > testsuites/validation/sparc/leon3/tc-bsp-sparc-leon3-gr712rc.c
> >
> I see. Maybe it makes sense to have all BSP-specific tests in a bsps
> subdirectory, with the unique names encoded to ensure the arch/bsp
> combination appears in the filename, such as:
> testsuites/validation/bsps/tc-bsp-sparc-leon3-gr712rc.c
> If so, 'bsp-' can probably be omitted from the filename. This way,
> architecture-specific testing may also be easily possible, like
> tc-sparc-something.c
>

We also have to account for device driver/class type tests which
are not BSP specific but only work if a BSP supports a specific feature.
For example, Alex wrote some GPIO tests for the Microblaze work. Assuming
these were based on a common GPIO API, they would work on any BSP
which supported that API.

Similarly for other optional devices where the drivers are in rtems.git

>
> > >
> > > I would like to discuss possible ways to manage the integration of
> > > tests that are conditionally-built based on the arch/bsp tuple. We
> > > should have clear guidance for others who want to add such tests in
> > > the future, or who would modify existing tests.
> >
> > Yes, this makes sense. We could add a new section for BSP-specific tests
> to:
> >
> > https://docs.rtems.org/branches/master/eng/req/howto.html
> >
> > For a pre-qualified BSP you have to specify the fatal errors and write
> > validation tests for it. Other BSP-specific specification and validation
> > may be necessary for the kernel IO device driver, cache controller
> > support, memory management unit initialization, memory protection unit
> > initialization, etc.
> >
> Good, that would definitely be important to document.
>
> I think we should have some of the BSP-specific tests under some other
> location in testsuites, while others would be under validation where
> they are used for pre-qualification?
>
> Maybe it is sensible to introduce testsuites/bsps/ also.
>
> I'm not entirely clear about the difference between 'validation' and
> any other kind of tests, such as where specific tests should be
> located.
>
> > Even for normal BSPs, some unit tests would be helpful. For example, I
> > had to fix _AArch32_PMSA_Initialize() twice:
> >
> >
> https://github.com/RTEMS/rtems/commits/master/cpukit/score/cpu/arm/aarch32-psma-init.c
> >
> > Unit tests would have probably found the errors before applications hit
> > the special cases.
> >
> Something like testsuites/bsps/arm-aarch32-psma-init.c (or whatever
> file naming convention we like to decide on. I forget the meaning of
> tc/tr/ts/tx but recall they are about specific kinds of testing for
> validation.)
>

I was wondering it we more of less mirrored the bsps/ structure

shared/
ARCH/shared
ARCH/BSP

We have solved this problem elsewhere. We don't have to encode it in the
file names.

>
> Gedare
>
> > --
> > embedded brains GmbH
> > Herr Sebastian HUBER
> > Dornierstr. 4
> > 82178 Puchheim
> > Germany
> > email: sebastian.huber at embedded-brains.de
> > phone: +49-89-18 94 741 - 16
> > fax:   +49-89-18 94 741 - 08
> >
> > Registergericht: Amtsgericht München
> > Registernummer: HRB 157899
> > Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler
> > Unsere Datenschutzerklärung finden Sie hier:
> > https://embedded-brains.de/datenschutzerklaerung/
> _______________________________________________
> devel mailing list
> devel at rtems.org
> http://lists.rtems.org/mailman/listinfo/devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rtems.org/pipermail/devel/attachments/20230615/79368f2b/attachment-0001.htm>