Tiering, Expected Failures, and BSPs which Run on Multiple Sims/HW

[Thomas Doerfler]

Joel,

it all comes down to "where was it tested". Knowing a certain test works
on ONE architecture, on ONE member of a BSP family never means it works
on all. Surely if it fails on one member it makes other members's
support suspicious, but the result is always for the hardware/simulator
it ran on.

If a BSP covers various variants and boards/simulators, you must test it
on all variants to find out whether they work. OTOH, if a BSP at least
succeeds on one supported HW, it is worth keeping it actively in the tree.

So I still think that we should track test results per
arch.bsp.variant.HW, not only per arch.bsp

And we should apply the tiering scheme to the HW, not only the BSP.

But, yes, it's hard.

Thomas.

> 
> The tester makes a distinction (e.g. leon3-sis vs leon3-tsim or
> leon3-qemu) but they
> all test the same binaries with different results. The set of expected
> failures on each
> of those could be different. 
> 
> I don't have a good answer. We currently think of three levels:
> 
> + architecture
> + BSP Family
> + BSP Variant
> 
> and now we are adding the "what did we run that test on" variant for
> record keeping purposes
> 
> AFAIK The current .tcfg files control only the set of tests that would
> be considered permanent
> failures across all "runner" variants.
> 
> This is just hard. :(
> 
> --joel
>  
> 
> 
>     Kind regards,
> 
>     Thomas.
> 
>     >
>     > _______________________________________________
>     > devel mailing list
>     > devel at rtems.org <mailto:devel at rtems.org>
>     > http://lists.rtems.org/mailman/listinfo/devel
>     >
> 
>     -- 
>     --------------------------------------------
>     embedded brains GmbH
>     Thomas Doerfler
>     Dornierstr. 4
>     D-82178 Puchheim
>     Germany
>     email: Thomas.Doerfler at embedded-brains.de
>     <mailto:Thomas.Doerfler at embedded-brains.de>
>     Phone: +49-89-18 94 741-12
>     Fax:   +49-89-18 94 741-09
>     PGP: Public key available on request.
>     For our privacy statement, see
>     https://embedded-brains.de/en/data-privacy-statement/
>     _______________________________________________
>     devel mailing list
>     devel at rtems.org <mailto:devel at rtems.org>
>     http://lists.rtems.org/mailman/listinfo/devel
> 

-- 
--------------------------------------------
embedded brains GmbH
Thomas Doerfler
Dornierstr. 4
D-82178 Puchheim
Germany
email: Thomas.Doerfler at embedded-brains.de
Phone: +49-89-18 94 741-12
Fax:   +49-89-18 94 741-09
PGP: Public key available on request.
For our privacy statement, see
https://embedded-brains.de/en/data-privacy-statement/