*.tcfg specification?

[Sebastian Huber]

On 14/11/2019 01:09, Chris Johns wrote:
> On 13/11/19 5:22 pm, Sebastian Huber wrote:
>> On 13/11/2019 01:39, Chris Johns wrote:
>>>>       > Ok, it seems a test program has exactly one state for a given BSP.
>>>>
>>>>       Yes, this is driven by the ABI, hardware profile and similar constraints.
>>>>
>>>>
>>>> This is generally the case but we have a few BSPs where the same executable
>>>> runs on real hardware and multiple simulators. The SPARC BSPs are at the
>>>> top of this list. The pc386 is another example.
>>>>
>>>> We currently have no way to specify that a test could pass on real hardware,
>>>> fail on simulator 1 and pass on simulator 2. For the SPARC BSPs, we now
>>>> have real HW, sis, qemu and tsim.
>>>>
>>>> The .tcfg files don't begin to address this and I really don't have any good
>>>> ideas myself. It is just a very real life case that it would be nice to address
>>>> so we have 100% expected passes on all execution platforms.
>>> I think it is a difficult problem to try and solve in RTEMS with test states. I
>>> think this is best handled in the eco-system with rtems-test and anything that
>>> processes the results it generates. In the eco-system we have more flexibility
>>> to define a hardware profile and assign a BSP to it.
>>>
>>> A hardware test result is of higher importance that a simulator result and a
>>> simulator test result is of higher importance than no test results. The
>>> expected-fail state is feedback driven and we cannot set this state without a
>>> suitable set of results. The tier a BSP resides in defines how you view the test
>>> state for a specific BSP.
>>>
>>> Here are some observations ...
>>>
>>> 1. By definition an accurate simulator will always match the hardware test
>>> results
>>
>> This depends on the goals of the simulator. In Qemu for example there is by
>> design no preservation of the timing properties of the simulated hardware. We
>> see this in the spintrcritical* tests for example.
> 
> No doubt however the assertion is still true. Having tests that pass on
> simulator and fail on hardware then flagging them as valid does not make sense.
> 
>>> 2. We need to provide a workable way to capture the test results and feedback
>>> this back into the tiers
>>>
>>> 3. The tiers need to be current and visible to be meaningful to our users
>>>
>>> 4. We have no expected-fail states in any tests and we should ...
>>>
>>>     $ grep 'expected-fail' `find . -name \*.tcfg`` | wc -l
>>>          0
>>
>> One approach to capture the actual expectations on a particular target system
>> (e.g. real hardware H, simulator A, simulator B) would be to split the state
>> definitions. The build system would just build the test or not.
> 
> I am not following why you would want this.

See some lines above from Joel:

"We currently have no way to specify that a test could pass on real 
hardware, fail on simulator 1 and pass on simulator 2. For the SPARC 
BSPs, we now have real HW, sis, qemu and tsim."

If we want to specify the test state at this level (I don't know if we 
want to do this, but Joel asked.), then the information should move from 
the BSPs to the test runner. Otherwise we have to pick up one target 
platform for the BSP and use the results on this platform to specify the 
states.

> 
>> A particular
>> test runner which knows if the test runs on H, A, or B could use a test runner
>> specific data set to get the runtime states (pass, expected-fail, user-input,
>> indeterminate, benchmark).
> 
> We should always build and run a test that can be executed. 

Yes.

> We need to know if a
> test flagged as expected-fail is now passing.

Yes.

-- 
Sebastian Huber, embedded brains GmbH

Address : Dornierstr. 4, D-82178 Puchheim, Germany
Phone   : +49 89 189 47 41-16
Fax     : +49 89 189 47 41-09
E-Mail  : sebastian.huber at embedded-brains.de
PGP     : Public key available on request.

Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG.