BSP Test Results

[Joel Sherrill]

On Sun, Sep 6, 2020, 9:55 PM Chris Johns <chrisj at rtems.org> wrote:

> Hello,
>
> I would like to discuss BSP Test results early in the release cycle in the
> hope
> we avoid the last minute issues we encountered with RTEMS 5 and the
> "expected"
> failure state ticket.
>
> I would like to update this section ...
>
>
> https://docs.rtems.org/branches/master/user/testing/tests.html#expected-test-states
>
> to state there is to be a ticket for each `expected-fail` test state. I
> believe
> this was the outcome of the discussion that took place. Please correct me
> if
> this is not correct.
>
> The purpose of the `expected-fail` is to aid the accounting of the test
> results
> to let us know if there are any regressions. We need to account for tests
> that
> fail so we can track if a recent commit results in a new failure, i.e. a
> regression. To do this we need to capture the state in a way `rtems-test`
> can
> indicate a regression.
>
> I think the `indeterminate` state may need further explanation as it will
> help
> in the cases a simulator passes a test but the test fails on some
> hardware. I am
> currently seeing this with spcache01 on the PC BSP.
>

I don't mind this as long as we have a rigid format to ensure these are
easy to find. Perhaps info at the main be of the line marking this state.
In the past, we have tended to our comments before.

>
> With the level of continuous building and testing we are currently doing
> being
> able to easily determine a regression will become important. Check out the
> example below.
>
> I would like to avoid us sitting with failures that do not have tickets
> and are
> not accounted for. I know there is a lump of work to account for the
> failures
> and after that is done I think the effort needed to maintain the failure
> states
> will drop.
>
> As a result I have been pondering how I can encourage this work be done. I
> am
> considering updating the tier-1 status to requiring there be 0 unaccounted
> for
> failures. That is the `rtems-test`'s Failure count is 0 for a hardware
> test run.
>
> Chris
>
> An example using Joel's recent test run (thanks Joel :)).


No problem. I do want to point out that you can't tell that the tests ran
on sis rather than Tsim or some specific hardware. For leon3, you can add
qemu to the list of potential "platforms". I've mentioned this before.

While typing this, I had the idea that maybe adding a note argument that
gets added to the report might be an easy solution. Whoever runs the test
could freeform annotate the board model, simulator and version, etc. This
would at least allow the logs to show which platform when we need to look
for an issue.

Also I am not sure but hopefully the test reports do accurately reflect
host OS.

The sparc/leon2
> results show no regressions:
>
> Summary
> =======
>
> Passed:        580
> Failed:          0
> User Input:      6
> Expected Fail:   1
> Indeterminate:   0
> Benchmark:       3
> Timeout:         0
> Invalid:         0
> Wrong Version:   0
> Wrong Build:     0
> Wrong Tools:     0
> ------------------
> Total:         590
>
> [ https://lists.rtems.org/pipermail/build/2020-September/018089.html ]
>
> while the sparc/erc32 has a single failure:
>
> Summary
> =======
>
> Passed:        579
> Failed:          1
> User Input:      6
> Expected Fail:   1
> Indeterminate:   0
> Benchmark:       3
> Timeout:         0
> Invalid:         0
> Wrong Version:   0
> Wrong Build:     0
> Wrong Tools:     0
> ------------------
> Total:         590
>
> Failures:
>  spintrcritical08.exe
>
> [ https://lists.rtems.org/pipermail/build/2020-September/018088.html ]
>
> _______________________________________________
> devel mailing list
> devel at rtems.org
> http://lists.rtems.org/mailman/listinfo/devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rtems.org/pipermail/devel/attachments/20200906/e802756c/attachment.html>