BSP-Specific Testing was: [PATCH 00/34] Integrate pre-qualified LEON3 BSP
Sebastian Huber
sebastian.huber at embedded-brains.de
Thu Jun 15 14:54:39 UTC 2023
On 15.06.23 15:51, Joel Sherrill wrote:
>
> On Mon, Jun 12, 2023 at 10:58 AM Gedare Bloom <gedare at rtems.org
> <mailto:gedare at rtems.org>> wrote:
>
> On Mon, Jun 12, 2023 at 12:56 AM Sebastian Huber
> <sebastian.huber at embedded-brains.de
> <mailto:sebastian.huber at embedded-brains.de>> wrote:
> >
> >
> >
> > On 01.06.23 22:06, Gedare Bloom wrote:
> > > On Thu, Jun 1, 2023 at 2:00 PM Gedare Bloom<gedare at rtems.org
> <mailto:gedare at rtems.org>> wrote:
> > >> ---------- Forwarded message ---------
> > >> From: Sebastian Huber<sebastian.huber at embedded-brains.de
> <mailto:sebastian.huber at embedded-brains.de>>
> > >> Date: Wed, May 31, 2023 at 10:31 AM
> > >>
> > >> The existing tests in the RTEMS test suite are basically BSP
> > >> independent. This patch set introduces BSP-specific validation
> tests.
> > >> These tests are disabled for other BSPs through the build
> system, for
> > >> example:
> > >>
> > >> spec/build/testsuites/validation/bsp-sparc-leon3-gr712rc.yml
> > >> [...]
> > >> cxxflags: []
> > >> enabled-by: sparc/gr712rc
> > >> features: c cprogram
> > >> [...]
> > > The use of the enabled-by: field to control the BSP-specific tests
> > > looks reasonable. However, we should decide where/how any
> BSP-specific
> > > tests should reside. It looks to me like the current approach is to
> > > dump all test files in a single monolithic 'validation'
> directory, and
> > > let the user/script sort it out. This results in a mix of tests
> > > intended for all targets, and some for specific targets. This is
> > > pretty much non-maintainable from my point-of-view without some
> > > additional tool support. Correct me if I'm wrong.
> >
> > Yes, this is the current approach. There is no strict one-to-one
> > relationship of test cases and test suites. The file names are
> somewhat
> > descriptive, for example:
> >
> > ls -l *leon3*
> > -rw-r--r-- 1 sebastian_h domain users 3739 31. Mai 10:44
> > tc-bsp-sparc-leon3-gr712rc.c
> > -rw-r--r-- 1 sebastian_h domain users 6138 31. Mai 10:44
> > tc-fatal-bsp-sparc-leon3-shutdown.c
> > -rw-r--r-- 1 sebastian_h domain users 5135 31. Mai 10:44
> > tr-fatal-bsp-sparc-leon3-cache-snooping-disabled-boot.c
> > -rw-r--r-- 1 sebastian_h domain users 2723 31. Mai 10:44
> > tr-fatal-bsp-sparc-leon3-cache-snooping-disabled-boot.h
> > -rw-r--r-- 1 sebastian_h domain users 5406 31. Mai 10:44
> > tr-fatal-bsp-sparc-leon3-cache-snooping-disabled-secondary.c
> > -rw-r--r-- 1 sebastian_h domain users 2753 31. Mai 10:44
> > tr-fatal-bsp-sparc-leon3-cache-snooping-disabled-secondary.h
> > -rw-r--r-- 1 sebastian_h domain users 5270 31. Mai 10:44
> > tr-fatal-bsp-sparc-leon3-clock-initialization.c
> > -rw-r--r-- 1 sebastian_h domain users 2681 31. Mai 10:44
> > tr-fatal-bsp-sparc-leon3-clock-initialization.h
> > -rw-r--r-- 1 sebastian_h domain users 2488 31. Mai 10:44
> > ts-bsp-sparc-leon3-gr712rc.c
> > -rw-r--r-- 1 sebastian_h domain users 2845 31. Mai 10:44
> > ts-fatal-bsp-sparc-leon3-cache-snooping-disabled-boot.c
> > -rw-r--r-- 1 sebastian_h domain users 2919 31. Mai 10:44
> > ts-fatal-bsp-sparc-leon3-cache-snooping-disabled-secondary.c
> > -rw-r--r-- 1 sebastian_h domain users 2797 31. Mai 10:44
> > ts-fatal-bsp-sparc-leon3-clock-initialization.c
> > -rw-r--r-- 1 sebastian_h domain users 3148 31. Mai 10:45
> > ts-fatal-bsp-sparc-leon3-shutdown-response.c
> > -rw-r--r-- 1 sebastian_h domain users 4909 31. Mai 10:44
> > ts-fatal-bsp-sparc-leon3-shutdown.c
> >
> > We could also introduce subdirectories to organize things. The test
> > framework prints out the file name in messages, so it would be
> nice if
> > they remain unique. With subdirectories this would lead to longer
> path
> > names, for example
> >
> > testsuites/validation/sparc/leon3/tc-bsp-sparc-leon3-gr712rc.c
> >
> I see. Maybe it makes sense to have all BSP-specific tests in a bsps
> subdirectory, with the unique names encoded to ensure the arch/bsp
> combination appears in the filename, such as:
> testsuites/validation/bsps/tc-bsp-sparc-leon3-gr712rc.c
> If so, 'bsp-' can probably be omitted from the filename. This way,
> architecture-specific testing may also be easily possible, like
> tc-sparc-something.c
>
>
> We also have to account for device driver/class type tests which
> are not BSP specific but only work if a BSP supports a specific feature.
> For example, Alex wrote some GPIO tests for the Microblaze work. Assuming
> these were based on a common GPIO API, they would work on any BSP
> which supported that API.
>
> Similarly for other optional devices where the drivers are in rtems.git
The enabled-by expression is quite flexible. For the placement, if it is
BSP-specific just place it in testsuites/validation/bsps or
testsuites/unit/bsps.
>
>
> > >
> > > I would like to discuss possible ways to manage the integration of
> > > tests that are conditionally-built based on the arch/bsp tuple. We
> > > should have clear guidance for others who want to add such tests in
> > > the future, or who would modify existing tests.
> >
> > Yes, this makes sense. We could add a new section for
> BSP-specific tests to:
> >
> > https://docs.rtems.org/branches/master/eng/req/howto.html
> <https://docs.rtems.org/branches/master/eng/req/howto.html>
> >
> > For a pre-qualified BSP you have to specify the fatal errors and
> write
> > validation tests for it. Other BSP-specific specification and
> validation
> > may be necessary for the kernel IO device driver, cache controller
> > support, memory management unit initialization, memory protection
> unit
> > initialization, etc.
> >
> Good, that would definitely be important to document.
>
> I think we should have some of the BSP-specific tests under some other
> location in testsuites, while others would be under validation where
> they are used for pre-qualification?
>
> Maybe it is sensible to introduce testsuites/bsps/ also.
>
> I'm not entirely clear about the difference between 'validation' and
> any other kind of tests, such as where specific tests should be
> located.
>
> > Even for normal BSPs, some unit tests would be helpful. For
> example, I
> > had to fix _AArch32_PMSA_Initialize() twice:
> >
> >
> https://github.com/RTEMS/rtems/commits/master/cpukit/score/cpu/arm/aarch32-psma-init.c <https://github.com/RTEMS/rtems/commits/master/cpukit/score/cpu/arm/aarch32-psma-init.c>
> >
> > Unit tests would have probably found the errors before
> applications hit
> > the special cases.
> >
> Something like testsuites/bsps/arm-aarch32-psma-init.c (or whatever
> file naming convention we like to decide on. I forget the meaning of
> tc/tr/ts/tx but recall they are about specific kinds of testing for
> validation.)
>
>
> I was wondering it we more of less mirrored the bsps/ structure
>
> shared/
> ARCH/shared
> ARCH/BSP
>
> We have solved this problem elsewhere. We don't have to encode it in the
> file names.
The test framework prints out the location of events using the file name
and the line number. So, it would be good if the file names are unique
and descriptive. This is why I like Gedare's proposal to just use "bsps".
--
embedded brains GmbH
Herr Sebastian HUBER
Dornierstr. 4
82178 Puchheim
Germany
email: sebastian.huber at embedded-brains.de
phone: +49-89-18 94 741 - 16
fax: +49-89-18 94 741 - 08
Registergericht: Amtsgericht München
Registernummer: HRB 157899
Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler
Unsere Datenschutzerklärung finden Sie hier:
https://embedded-brains.de/datenschutzerklaerung/
More information about the devel
mailing list