RISC-V Test Results

[Chris Johns]

On 9/8/19 9:47 pm, Sebastian Huber wrote:
> On 08/08/2019 15:33, Joel Sherrill wrote:
>> Hi
>>
>> If you are subscribed to the build@ mailing list, then you saw the flurry of test
>> results from over night. I built every variant and ran the test suite with RTEMS
>> debug on and off.  Here are some observations:
>>
>> + rv64imafd only has one test pass
>> + rv64_iamd_medany only has one test pass
>> + Generally speaking, 17-19 tests failed or timed out on every variant with
>>     551-553 passing. It would be great for someone to mark the tests in the
>>     tcfg files as expected fails.
> 
> The BSP runs on hardware and simulators.

Interesting.

> Some test may pass on real hardware.

Sure.

> So, marking them as expected fails is not right.

Agreed. I expect to see regressions in simulator from time to time so hardware
is a only way we can validate our test results.

If a BSP can run on both hardware and simulation the hardware results should
have precedent over simulation. This means the we tag against the hardware
results in this case.

Chris