<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Sep 7, 2020 at 12:44 AM Chris Johns <<a href="mailto:chrisj@rtems.org">chrisj@rtems.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On 7/9/20 2:16 pm, Joel Sherrill wrote:<br>
> <br>
> <br>
> On Sun, Sep 6, 2020, 9:55 PM Chris Johns <<a href="mailto:chrisj@rtems.org" target="_blank">chrisj@rtems.org</a><br>
> <mailto:<a href="mailto:chrisj@rtems.org" target="_blank">chrisj@rtems.org</a>>> wrote:<br>
> <br>
> Hello,<br>
> <br>
> I would like to discuss BSP Test results early in the release cycle in the hope<br>
> we avoid the last minute issues we encountered with RTEMS 5 and the "expected"<br>
> failure state ticket.<br>
> <br>
> I would like to update this section ...<br>
> <br>
> <a href="https://docs.rtems.org/branches/master/user/testing/tests.html#expected-test-states" rel="noreferrer" target="_blank">https://docs.rtems.org/branches/master/user/testing/tests.html#expected-test-states</a><br>
> <br>
> to state there is to be a ticket for each `expected-fail` test state. I believe<br>
> this was the outcome of the discussion that took place. Please correct me if<br>
> this is not correct.<br>
> <br>
> The purpose of the `expected-fail` is to aid the accounting of the test results<br>
> to let us know if there are any regressions. We need to account for tests that<br>
> fail so we can track if a recent commit results in a new failure, i.e. a<br>
> regression. To do this we need to capture the state in a way `rtems-test` can<br>
> indicate a regression.<br>
> <br>
> I think the `indeterminate` state may need further explanation as it will help<br>
> in the cases a simulator passes a test but the test fails on some hardware. I am<br>
> currently seeing this with spcache01 on the PC BSP.<br>
> <br>
> <br>
> I don't mind this as long as we have a rigid format to ensure these are easy to<br>
> find. <br>
<br>
Do you mean find the location in the tcfg files?<br></blockquote><div><br></div><div>No. I just mean sometimes there is a comment directly above the l</div><div>exclude line, sometimes above a group, etc. I would want something</div><div>like</div><div><br></div><div>exclude: dl06 # #1234 dynamic loading broken on RISC-V</div><div><br></div><div>Maybe even not as a comment but a ticket number so it could be</div><div>checked that every exclude had a ticket number in the form of #NNNN[N]</div><div>(four or five digits).</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
> Perhaps info at the main be of the line marking this state. In the past,<br>
> we have tended to our comments before.<br>
<br>
If you mean the location in the tcfg files being printed out by the test, have<br>
no idea how to do that. What about grep?<br></blockquote><div><br></div><div>grep works if on the same line. Hence the above suggestion. If we want to</div><div>add excludes, let's document which test and ticket in a way that theoretically</div><div>we could check if the ticket is open or closed. </div><div><br></div><div>No way to automate that it applies to this BSP's exclude I think.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
> With the level of continuous building and testing we are currently doing being<br>
> able to easily determine a regression will become important. Check out the<br>
> example below.<br>
> <br>
> I would like to avoid us sitting with failures that do not have tickets and are<br>
> not accounted for. I know there is a lump of work to account for the failures<br>
> and after that is done I think the effort needed to maintain the failure states<br>
> will drop.<br>
> <br>
> As a result I have been pondering how I can encourage this work be done. I am<br>
> considering updating the tier-1 status to requiring there be 0 unaccounted for<br>
> failures. That is the `rtems-test`'s Failure count is 0 for a hardware test run.<br>
> <br>
> Chris<br>
> <br>
> An example using Joel's recent test run (thanks Joel :)).<br>
> <br>
> <br>
> No problem. I do want to point out that you can't tell that the tests ran on sis<br>
> rather than Tsim or some specific hardware. For leon3, you can add qemu to the<br>
> list of potential "platforms". I've mentioned this before.<br>
<br>
Maybe the `rtems-test` command line option name is misleading. It is<br>
`--rtems-bsp=` but it is more like `--bsp-target=`. Some BSPs have more than one<br>
way to be run, for example psim and psim-run. I see in one of the tests I linked<br>
too the command line has `--rtems-bsp=leon2-sis`.<br>
<br>
> While typing this, I had the idea that maybe adding a note argument that gets<br>
> added to the report might be an easy solution. Whoever runs the test could<br>
> freeform annotate the board model, simulator and version, etc. This would at<br>
> least allow the logs to show which platform when we need to look for an issue.<br>
<br>
If you think there should be a list of annotations a test run needs then please<br>
create a ticket with a list.<br></blockquote><div><br></div><div>This would be enough to address my "HW or which simulator" question. </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
> Also I am not sure but hopefully the test reports do accurately reflect host OS. <br>
<br>
There is a "Host" section at the top of the results log? It is just `uname -a`.<br></blockquote><div><br></div><div>I think that's sufficient as long as it can distinguish Linux distributions. </div><div><br></div><div>--joel</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Chris<br>
</blockquote></div></div>