Tiering, Expected Failures, and BSPs which Run on Multiple Sims/HW

[Joel Sherrill]

On Fri, Oct 25, 2019 at 9:18 AM Thomas Doerfler <
thomas.doerfler at embedded-brains.de> wrote:

> Hi,
>
>
> Am 25.10.19 um 16:08 schrieb Joel Sherrill:
> >
> > One question we need to discuss is what about BSPs which can run on
> > multiple simulators and possibly real hardware. For example, the sparc
> > BSPs can run on real hardware, sis, qemu, and tsim. How do we
> > distinguish the same BSP's expected results on > 1 platform?
>
> if I remember correctly, those BSPs targetting for multiple
> boards/simulators only have a limited number of target boards.
>

The set is larger than you might think. :) From the top of my head:

+ RISC-V has a lot of BSPs and can run on multiple simulators (sis, spike,
qemu)
   and maybe some on real HW
+ all SPARC BSPs (sis, some on qemu, tsim) and real hardware
+ PC variants
+ BeagleBone - one variant has qemu and HW
+ Zynq at least has a dedicated BSP variant for qemu so OK
+ Realview has qemu and real HW
+ some STM has a qemu support



>
> Could we consider each (reference) target board and/or simulator instead
> of a BSP? The tiering may then be for a board instead of a BSP. And the
> BSP tier is best tier of the target boards?
>

The tester makes a distinction (e.g. leon3-sis vs leon3-tsim or leon3-qemu)
but they
all test the same binaries with different results. The set of expected
failures on each
of those could be different.

I don't have a good answer. We currently think of three levels:

+ architecture
+ BSP Family
+ BSP Variant

and now we are adding the "what did we run that test on" variant for record
keeping purposes

AFAIK The current .tcfg files control only the set of tests that would be
considered permanent
failures across all "runner" variants.

This is just hard. :(

--joel


>
> Kind regards,
>
> Thomas.
>
> >
> > _______________________________________________
> > devel mailing list
> > devel at rtems.org
> > http://lists.rtems.org/mailman/listinfo/devel
> >
>
> --
> --------------------------------------------
> embedded brains GmbH
> Thomas Doerfler
> Dornierstr. 4
> D-82178 Puchheim
> Germany
> email: Thomas.Doerfler at embedded-brains.de
> Phone: +49-89-18 94 741-12
> Fax:   +49-89-18 94 741-09
> PGP: Public key available on request.
> For our privacy statement, see
> https://embedded-brains.de/en/data-privacy-statement/
> _______________________________________________
> devel mailing list
> devel at rtems.org
> http://lists.rtems.org/mailman/listinfo/devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rtems.org/pipermail/devel/attachments/20191025/b444d56e/attachment.html>