BSP Test Results

[Chris Johns]

On 9/9/20 8:50 am, Joel Sherrill wrote:
> On Mon, Sep 7, 2020 at 12:44 AM Chris Johns <chrisj at rtems.org
> <mailto:chrisj at rtems.org>> wrote:
>     On 7/9/20 2:16 pm, Joel Sherrill wrote:
>     > On Sun, Sep 6, 2020, 9:55 PM Chris Johns <chrisj at rtems.org
>     <mailto:chrisj at rtems.org>
>     > <mailto:chrisj at rtems.org <mailto:chrisj at rtems.org>>> wrote:
>     >
>     >     Hello,
>     >
>     >     I would like to discuss BSP Test results early in the release cycle in
>     the hope
>     >     we avoid the last minute issues we encountered with RTEMS 5 and the
>     "expected"
>     >     failure state ticket.
>     >
>     >     I would like to update this section ...
>     >
>     >   
>      https://docs.rtems.org/branches/master/user/testing/tests.html#expected-test-states
>     >
>     >     to state there is to be a ticket for each `expected-fail` test state.
>     I believe
>     >     this was the outcome of the discussion that took place. Please correct
>     me if
>     >     this is not correct.
>     >
>     >     The purpose of the `expected-fail` is to aid the accounting of the
>     test results
>     >     to let us know if there are any regressions. We need to account for
>     tests that
>     >     fail so we can track if a recent commit results in a new failure, i.e. a
>     >     regression. To do this we need to capture the state in a way
>     `rtems-test` can
>     >     indicate a regression.
>     >
>     >     I think the `indeterminate` state may need further explanation as it
>     will help
>     >     in the cases a simulator passes a test but the test fails on some
>     hardware. I am
>     >     currently seeing this with spcache01 on the PC BSP.
>     >
>     >
>     > I don't mind this as long as we have a rigid format to ensure these are
>     easy to
>     > find.
> 
>     Do you mean find the location in the tcfg files?
> 
> No. I just mean sometimes there is a comment directly above the l
> exclude line, sometimes above a group, etc. I would want something
> like
> 
> exclude: dl06  # #1234 dynamic loading broken on RISC-V

I assume you mean `expected-fail`? Excluded tests are just that excluded, i.e.
not enough memory, so a ticket will not change that.

> Maybe even not as a comment but a ticket number so it could be
> checked that every exclude had a ticket number in the form of #NNNN[N]
> (four or five digits).

Oh of course. The format for expected-fail could be extended to include a ticket
number so it _has_ to be there? Comments tend to be fragile. A ticket number is
something I think can be implemented without too many issues.

Should the indeterminate state also have a ticket assigned?

I am fine to add this support but it does not mean I will be adding all the
tickets and the expected-fail states. It is a project wide task we all need to
help with. I just hope it happens rather than being left like last time. :)

>     > Perhaps info at the main be of the line marking this state. In the past,
>     > we have tended to our comments before.
> 
>     If you mean the location in the tcfg files being printed out by the test, have
>     no idea how to do that. What about grep?
> 
> 
> grep works if on the same line. Hence the above suggestion. If we want to
> add excludes, let's document which test and ticket in a way that theoretically
> we could check if the ticket is open or closed.  
> 
> No way to automate that it applies to this BSP's exclude I think.

Let me have a look to see what can be done now I understand it is a ticket
number you would like to have in the test.

>     >     With the level of continuous building and testing we are currently
>     doing being
>     >     able to easily determine a regression will become important. Check out the
>     >     example below.
>     >
>     >     I would like to avoid us sitting with failures that do not have
>     tickets and are
>     >     not accounted for. I know there is a lump of work to account for the
>     failures
>     >     and after that is done I think the effort needed to maintain the
>     failure states
>     >     will drop.
>     >
>     >     As a result I have been pondering how I can encourage this work be
>     done. I am
>     >     considering updating the tier-1 status to requiring there be 0
>     unaccounted for
>     >     failures. That is the `rtems-test`'s Failure count is 0 for a hardware
>     test run.
>     >
>     >     Chris
>     >
>     >     An example using Joel's recent test run (thanks Joel :)).
>     >
>     >
>     > No problem. I do want to point out that you can't tell that the tests ran
>     on sis
>     > rather than Tsim or some specific hardware. For leon3, you can add qemu to the
>     > list of potential "platforms". I've mentioned this before.
> 
>     Maybe the `rtems-test` command line option name is misleading. It is
>     `--rtems-bsp=` but it is more like `--bsp-target=`. Some BSPs have more than one
>     way to be run, for example psim and psim-run. I see in one of the tests I linked
>     too the command line has `--rtems-bsp=leon2-sis`.
> 
>     > While typing this, I had the idea that maybe adding a note argument that gets
>     > added to the report might be an easy solution. Whoever runs the test could
>     > freeform annotate the board model, simulator and version, etc. This would at
>     > least allow the logs to show which platform when we need to look for an issue.
> 
>     If you think there should be a list of annotations a test run needs then please
>     create a ticket with a list.
> 
> This would be enough to address my "HW or which simulator" question. 

I had created #4072, please update adding the items you would like. We can add
anything you like and they can be mandated. There already exist a number of
fields for various transports so adding this is not hard, creating the list is
harder. These can defined in the .cfg files or they can be part of the user .ini
configuration files and so specific to a user's set up. They would be
informational only.

>     > Also I am not sure but hopefully the test reports do accurately reflect
>     host OS.
> 
>     There is a "Host" section at the top of the results log? It is just `uname -a`.
> 
> I think that's sufficient as long as it can distinguish Linux distributions. 

Does Linux or Python on Linux provide a common means to find what distribution
you are on? Do distributions based on another distro make a suitable the
separation? A Linux distro expert will need to guide this. I have no idea.

Chris