BSP Test Results

[Gedare Bloom]

On Sun, Sep 6, 2020 at 8:55 PM Chris Johns <chrisj at rtems.org> wrote:
>
> Hello,
>
> I would like to discuss BSP Test results early in the release cycle in the hope
> we avoid the last minute issues we encountered with RTEMS 5 and the "expected"
> failure state ticket.
>
> I would like to update this section ...
>
> https://docs.rtems.org/branches/master/user/testing/tests.html#expected-test-states
>
> to state there is to be a ticket for each `expected-fail` test state. I believe
> this was the outcome of the discussion that took place. Please correct me if
> this is not correct.
>
> The purpose of the `expected-fail` is to aid the accounting of the test results
> to let us know if there are any regressions. We need to account for tests that
> fail so we can track if a recent commit results in a new failure, i.e. a
> regression. To do this we need to capture the state in a way `rtems-test` can
> indicate a regression.
>
> I think the `indeterminate` state may need further explanation as it will help
> in the cases a simulator passes a test but the test fails on some hardware. I am
> currently seeing this with spcache01 on the PC BSP.
>
> With the level of continuous building and testing we are currently doing being
> able to easily determine a regression will become important. Check out the
> example below.
>
> I would like to avoid us sitting with failures that do not have tickets and are
> not accounted for. I know there is a lump of work to account for the failures
> and after that is done I think the effort needed to maintain the failure states
> will drop.
>
> As a result I have been pondering how I can encourage this work be done. I am
> considering updating the tier-1 status to requiring there be 0 unaccounted for
> failures. That is the `rtems-test`'s Failure count is 0 for a hardware test run.
>
> Chris
>
> An example using Joel's recent test run (thanks Joel :)). The sparc/leon2
> results show no regressions:
>
> Summary
> =======
>
> Passed:        580
> Failed:          0
> User Input:      6
> Expected Fail:   1
Is there a ticket for this one? out of curiosity.

> Indeterminate:   0
> Benchmark:       3
> Timeout:         0
> Invalid:         0
> Wrong Version:   0
> Wrong Build:     0
> Wrong Tools:     0
> ------------------
> Total:         590
>
> [ https://lists.rtems.org/pipermail/build/2020-September/018089.html ]
>
> while the sparc/erc32 has a single failure:
>
> Summary
> =======
>
> Passed:        579
> Failed:          1
> User Input:      6
> Expected Fail:   1
> Indeterminate:   0
> Benchmark:       3
> Timeout:         0
> Invalid:         0
> Wrong Version:   0
> Wrong Build:     0
> Wrong Tools:     0
> ------------------
> Total:         590
>
> Failures:
>  spintrcritical08.exe
>
> [ https://lists.rtems.org/pipermail/build/2020-September/018088.html ]
>
> _______________________________________________
> devel mailing list
> devel at rtems.org
> http://lists.rtems.org/mailman/listinfo/devel