BSP Test Results
joel at rtems.org
Tue Sep 8 22:50:34 UTC 2020
On Mon, Sep 7, 2020 at 12:44 AM Chris Johns <chrisj at rtems.org> wrote:
> On 7/9/20 2:16 pm, Joel Sherrill wrote:
> > On Sun, Sep 6, 2020, 9:55 PM Chris Johns <chrisj at rtems.org
> > <mailto:chrisj at rtems.org>> wrote:
> > Hello,
> > I would like to discuss BSP Test results early in the release cycle
> in the hope
> > we avoid the last minute issues we encountered with RTEMS 5 and the
> > failure state ticket.
> > I would like to update this section ...
> > to state there is to be a ticket for each `expected-fail` test
> state. I believe
> > this was the outcome of the discussion that took place. Please
> correct me if
> > this is not correct.
> > The purpose of the `expected-fail` is to aid the accounting of the
> test results
> > to let us know if there are any regressions. We need to account for
> tests that
> > fail so we can track if a recent commit results in a new failure,
> i.e. a
> > regression. To do this we need to capture the state in a way
> `rtems-test` can
> > indicate a regression.
> > I think the `indeterminate` state may need further explanation as it
> will help
> > in the cases a simulator passes a test but the test fails on some
> hardware. I am
> > currently seeing this with spcache01 on the PC BSP.
> > I don't mind this as long as we have a rigid format to ensure these are
> easy to
> > find.
> Do you mean find the location in the tcfg files?
No. I just mean sometimes there is a comment directly above the l
exclude line, sometimes above a group, etc. I would want something
exclude: dl06 # #1234 dynamic loading broken on RISC-V
Maybe even not as a comment but a ticket number so it could be
checked that every exclude had a ticket number in the form of #NNNN[N]
(four or five digits).
> > Perhaps info at the main be of the line marking this state. In the past,
> > we have tended to our comments before.
> If you mean the location in the tcfg files being printed out by the test,
> no idea how to do that. What about grep?
grep works if on the same line. Hence the above suggestion. If we want to
add excludes, let's document which test and ticket in a way that
we could check if the ticket is open or closed.
No way to automate that it applies to this BSP's exclude I think.
> > With the level of continuous building and testing we are currently
> doing being
> > able to easily determine a regression will become important. Check
> out the
> > example below.
> > I would like to avoid us sitting with failures that do not have
> tickets and are
> > not accounted for. I know there is a lump of work to account for the
> > and after that is done I think the effort needed to maintain the
> failure states
> > will drop.
> > As a result I have been pondering how I can encourage this work be
> done. I am
> > considering updating the tier-1 status to requiring there be 0
> unaccounted for
> > failures. That is the `rtems-test`'s Failure count is 0 for a
> hardware test run.
> > Chris
> > An example using Joel's recent test run (thanks Joel :)).
> > No problem. I do want to point out that you can't tell that the tests
> ran on sis
> > rather than Tsim or some specific hardware. For leon3, you can add qemu
> to the
> > list of potential "platforms". I've mentioned this before.
> Maybe the `rtems-test` command line option name is misleading. It is
> `--rtems-bsp=` but it is more like `--bsp-target=`. Some BSPs have more
> than one
> way to be run, for example psim and psim-run. I see in one of the tests I
> too the command line has `--rtems-bsp=leon2-sis`.
> > While typing this, I had the idea that maybe adding a note argument that
> > added to the report might be an easy solution. Whoever runs the test
> > freeform annotate the board model, simulator and version, etc. This
> would at
> > least allow the logs to show which platform when we need to look for an
> If you think there should be a list of annotations a test run needs then
> create a ticket with a list.
This would be enough to address my "HW or which simulator" question.
> > Also I am not sure but hopefully the test reports do accurately reflect
> host OS.
> There is a "Host" section at the top of the results log? It is just `uname
I think that's sufficient as long as it can distinguish Linux
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the devel