<div dir="auto"><div><br><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Sep 6, 2020, 9:55 PM Chris Johns <<a href="mailto:chrisj@rtems.org" target="_blank" rel="noreferrer">chrisj@rtems.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello,<br>
<br>
I would like to discuss BSP Test results early in the release cycle in the hope<br>
we avoid the last minute issues we encountered with RTEMS 5 and the "expected"<br>
failure state ticket.<br>
<br>
I would like to update this section ...<br>
<br>
<a href="https://docs.rtems.org/branches/master/user/testing/tests.html#expected-test-states" rel="noreferrer noreferrer noreferrer" target="_blank">https://docs.rtems.org/branches/master/user/testing/tests.html#expected-test-states</a><br>
<br>
to state there is to be a ticket for each `expected-fail` test state. I believe<br>
this was the outcome of the discussion that took place. Please correct me if<br>
this is not correct.<br>
<br>
The purpose of the `expected-fail` is to aid the accounting of the test results<br>
to let us know if there are any regressions. We need to account for tests that<br>
fail so we can track if a recent commit results in a new failure, i.e. a<br>
regression. To do this we need to capture the state in a way `rtems-test` can<br>
indicate a regression.<br>
<br>
I think the `indeterminate` state may need further explanation as it will help<br>
in the cases a simulator passes a test but the test fails on some hardware. I am<br>
currently seeing this with spcache01 on the PC BSP.<br></blockquote></div></div><div dir="auto"><br></div><div dir="auto">I don't mind this as long as we have a rigid format to ensure these are easy to find. Perhaps info at the main be of the line marking this state. In the past, we have tended to our comments before.</div><div dir="auto"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
With the level of continuous building and testing we are currently doing being<br>
able to easily determine a regression will become important. Check out the<br>
example below.<br>
<br>
I would like to avoid us sitting with failures that do not have tickets and are<br>
not accounted for. I know there is a lump of work to account for the failures<br>
and after that is done I think the effort needed to maintain the failure states<br>
will drop.<br>
<br>
As a result I have been pondering how I can encourage this work be done. I am<br>
considering updating the tier-1 status to requiring there be 0 unaccounted for<br>
failures. That is the `rtems-test`'s Failure count is 0 for a hardware test run.<br>
<br>
Chris<br>
<br>
An example using Joel's recent test run (thanks Joel :)).</blockquote></div></div><div dir="auto"><br></div><div dir="auto"></div><div dir="auto">No problem. I do want to point out that you can't tell that the tests ran on sis rather than Tsim or some specific hardware. For leon3, you can add qemu to the list of potential "platforms". I've mentioned this before. </div><div dir="auto"><br></div><div dir="auto">While typing this, I had the idea that maybe adding a note argument that gets added to the report might be an easy solution. Whoever runs the test could freeform annotate the board model, simulator and version, etc. This would at least allow the logs to show which platform when we need to look for an issue.</div><div dir="auto"><br></div><div dir="auto">Also I am not sure but hopefully the test reports do accurately reflect host OS. </div><div dir="auto"></div><div dir="auto"><br></div><div dir="auto"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> The sparc/leon2<br>
results show no regressions:<br>
<br>
Summary<br>
=======<br>
<br>
Passed: 580<br>
Failed: 0<br>
User Input: 6<br>
Expected Fail: 1<br>
Indeterminate: 0<br>
Benchmark: 3<br>
Timeout: 0<br>
Invalid: 0<br>
Wrong Version: 0<br>
Wrong Build: 0<br>
Wrong Tools: 0<br>
------------------<br>
Total: 590<br>
<br>
[ <a href="https://lists.rtems.org/pipermail/build/2020-September/018089.html" rel="noreferrer noreferrer noreferrer" target="_blank">https://lists.rtems.org/pipermail/build/2020-September/018089.html</a> ]<br>
<br>
while the sparc/erc32 has a single failure:<br>
<br>
Summary<br>
=======<br>
<br>
Passed: 579<br>
Failed: 1<br>
User Input: 6<br>
Expected Fail: 1<br>
Indeterminate: 0<br>
Benchmark: 3<br>
Timeout: 0<br>
Invalid: 0<br>
Wrong Version: 0<br>
Wrong Build: 0<br>
Wrong Tools: 0<br>
------------------<br>
Total: 590<br>
<br>
Failures:<br>
spintrcritical08.exe<br>
<br>
[ <a href="https://lists.rtems.org/pipermail/build/2020-September/018088.html" rel="noreferrer noreferrer noreferrer" target="_blank">https://lists.rtems.org/pipermail/build/2020-September/018088.html</a> ]<br>
<br>
_______________________________________________<br>
devel mailing list<br>
<a href="mailto:devel@rtems.org" rel="noreferrer noreferrer" target="_blank">devel@rtems.org</a><br>
<a href="http://lists.rtems.org/mailman/listinfo/devel" rel="noreferrer noreferrer noreferrer" target="_blank">http://lists.rtems.org/mailman/listinfo/devel</a><br>
</blockquote></div></div></div>