*.tcfg specification?

[Sebastian Huber]

On 13/11/2019 01:39, Chris Johns wrote:
>>      > Ok, it seems a test program has exactly one state for a given BSP.
>>
>>      Yes, this is driven by the ABI, hardware profile and similar constraints.
>>
>>
>> This is generally the case but we have a few BSPs where the same executable
>> runs on real hardware and multiple simulators. The SPARC BSPs are at the
>> top of this list. The pc386 is another example.
>>
>> We currently have no way to specify that a test could pass on real hardware,
>> fail on simulator 1 and pass on simulator 2. For the SPARC BSPs, we now
>> have real HW, sis, qemu and tsim.
>>
>> The .tcfg files don't begin to address this and I really don't have any good
>> ideas myself. It is just a very real life case that it would be nice to address
>> so we have 100% expected passes on all execution platforms.
> I think it is a difficult problem to try and solve in RTEMS with test states. I
> think this is best handled in the eco-system with rtems-test and anything that
> processes the results it generates. In the eco-system we have more flexibility
> to define a hardware profile and assign a BSP to it.
> 
> A hardware test result is of higher importance that a simulator result and a
> simulator test result is of higher importance than no test results. The
> expected-fail state is feedback driven and we cannot set this state without a
> suitable set of results. The tier a BSP resides in defines how you view the test
> state for a specific BSP.
> 
> Here are some observations ...
> 
> 1. By definition an accurate simulator will always match the hardware test results

This depends on the goals of the simulator. In Qemu for example there is 
by design no preservation of the timing properties of the simulated 
hardware. We see this in the spintrcritical* tests for example.

> 
> 2. We need to provide a workable way to capture the test results and feedback
> this back into the tiers
> 
> 3. The tiers need to be current and visible to be meaningful to our users
> 
> 4. We have no expected-fail states in any tests and we should ...
> 
>    $ grep 'expected-fail' `find . -name \*.tcfg`` | wc -l
>         0

One approach to capture the actual expectations on a particular target 
system (e.g. real hardware H, simulator A, simulator B) would be to 
split the state definitions. The build system would just build the test 
or not. A particular test runner which knows if the test runs on H, A, 
or B could use a test runner specific data set to get the runtime states 
(pass, expected-fail, user-input, indeterminate, benchmark).

-- 
Sebastian Huber, embedded brains GmbH

Address : Dornierstr. 4, D-82178 Puchheim, Germany
Phone   : +49 89 189 47 41-16
Fax     : +49 89 189 47 41-09
E-Mail  : sebastian.huber at embedded-brains.de
PGP     : Public key available on request.

Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG.