BSP Test Results

[Chris Johns]

On 7/9/20 2:16 pm, Joel Sherrill wrote:
> 
> 
> On Sun, Sep 6, 2020, 9:55 PM Chris Johns <chrisj at rtems.org
> <mailto:chrisj at rtems.org>> wrote:
> 
>     Hello,
> 
>     I would like to discuss BSP Test results early in the release cycle in the hope
>     we avoid the last minute issues we encountered with RTEMS 5 and the "expected"
>     failure state ticket.
> 
>     I would like to update this section ...
> 
>     https://docs.rtems.org/branches/master/user/testing/tests.html#expected-test-states
> 
>     to state there is to be a ticket for each `expected-fail` test state. I believe
>     this was the outcome of the discussion that took place. Please correct me if
>     this is not correct.
> 
>     The purpose of the `expected-fail` is to aid the accounting of the test results
>     to let us know if there are any regressions. We need to account for tests that
>     fail so we can track if a recent commit results in a new failure, i.e. a
>     regression. To do this we need to capture the state in a way `rtems-test` can
>     indicate a regression.
> 
>     I think the `indeterminate` state may need further explanation as it will help
>     in the cases a simulator passes a test but the test fails on some hardware. I am
>     currently seeing this with spcache01 on the PC BSP.
> 
> 
> I don't mind this as long as we have a rigid format to ensure these are easy to
> find. 

Do you mean find the location in the tcfg files?

> Perhaps info at the main be of the line marking this state. In the past,
> we have tended to our comments before.

If you mean the location in the tcfg files being printed out by the test, have
no idea how to do that. What about grep?

>     With the level of continuous building and testing we are currently doing being
>     able to easily determine a regression will become important. Check out the
>     example below.
> 
>     I would like to avoid us sitting with failures that do not have tickets and are
>     not accounted for. I know there is a lump of work to account for the failures
>     and after that is done I think the effort needed to maintain the failure states
>     will drop.
> 
>     As a result I have been pondering how I can encourage this work be done. I am
>     considering updating the tier-1 status to requiring there be 0 unaccounted for
>     failures. That is the `rtems-test`'s Failure count is 0 for a hardware test run.
> 
>     Chris
> 
>     An example using Joel's recent test run (thanks Joel :)).
> 
> 
> No problem. I do want to point out that you can't tell that the tests ran on sis
> rather than Tsim or some specific hardware. For leon3, you can add qemu to the
> list of potential "platforms". I've mentioned this before.

Maybe the `rtems-test` command line option name is misleading. It is
`--rtems-bsp=` but it is more like `--bsp-target=`. Some BSPs have more than one
way to be run, for example psim and psim-run. I see in one of the tests I linked
too the command line has `--rtems-bsp=leon2-sis`.

> While typing this, I had the idea that maybe adding a note argument that gets
> added to the report might be an easy solution. Whoever runs the test could
> freeform annotate the board model, simulator and version, etc. This would at
> least allow the logs to show which platform when we need to look for an issue.

If you think there should be a list of annotations a test run needs then please
create a ticket with a list.

> Also I am not sure but hopefully the test reports do accurately reflect host OS. 

There is a "Host" section at the top of the results log? It is just `uname -a`.

Chris