BSP Test Results

[Chris Johns]

Hello,

I would like to discuss BSP Test results early in the release cycle in the hope
we avoid the last minute issues we encountered with RTEMS 5 and the "expected"
failure state ticket.

I would like to update this section ...

https://docs.rtems.org/branches/master/user/testing/tests.html#expected-test-states

to state there is to be a ticket for each `expected-fail` test state. I believe
this was the outcome of the discussion that took place. Please correct me if
this is not correct.

The purpose of the `expected-fail` is to aid the accounting of the test results
to let us know if there are any regressions. We need to account for tests that
fail so we can track if a recent commit results in a new failure, i.e. a
regression. To do this we need to capture the state in a way `rtems-test` can
indicate a regression.

I think the `indeterminate` state may need further explanation as it will help
in the cases a simulator passes a test but the test fails on some hardware. I am
currently seeing this with spcache01 on the PC BSP.

With the level of continuous building and testing we are currently doing being
able to easily determine a regression will become important. Check out the
example below.

I would like to avoid us sitting with failures that do not have tickets and are
not accounted for. I know there is a lump of work to account for the failures
and after that is done I think the effort needed to maintain the failure states
will drop.

As a result I have been pondering how I can encourage this work be done. I am
considering updating the tier-1 status to requiring there be 0 unaccounted for
failures. That is the `rtems-test`'s Failure count is 0 for a hardware test run.

Chris

An example using Joel's recent test run (thanks Joel :)). The sparc/leon2
results show no regressions:

Summary
=======

Passed:        580
Failed:          0
User Input:      6
Expected Fail:   1
Indeterminate:   0
Benchmark:       3
Timeout:         0
Invalid:         0
Wrong Version:   0
Wrong Build:     0
Wrong Tools:     0
------------------
Total:         590

[ https://lists.rtems.org/pipermail/build/2020-September/018089.html ]

while the sparc/erc32 has a single failure:

Summary
=======

Passed:        579
Failed:          1
User Input:      6
Expected Fail:   1
Indeterminate:   0
Benchmark:       3
Timeout:         0
Invalid:         0
Wrong Version:   0
Wrong Build:     0
Wrong Tools:     0
------------------
Total:         590

Failures:
 spintrcritical08.exe

[ https://lists.rtems.org/pipermail/build/2020-September/018088.html ]