<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Dear Chris,<div class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On 7 Sep 2020, at 04:21, Chris Johns <<a href="mailto:chrisj@rtems.org" class="">chrisj@rtems.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class=""> ... stuff deleted ...<br class=""></div></div></blockquote><br class=""><blockquote type="cite" class=""><div class=""><div class=""><blockquote type="cite" class=""><blockquote type="cite" class="">How is the log from the test output used to validate the model?<br class=""></blockquote><br class="">The test output shows what the test did, so it can be used by someone familiar<br class="">with the<br class="">requirements to judge if the test expectations of correct behaviour are<br class="">themselves correct.<br class="">So, yes, they can help in that regard.<br class=""></blockquote><br class="">Just so I understand, if the model and the API do not match the test will report<br class="">a failure?<br class=""></div></div></blockquote><div><br class=""></div>Yes - I'd expect early test fails to expose hidden assumptions in our models,</div><div>rather than identifying true bugs. I would expect to get some developer feedback</div><div>to assist in fixing these. Hopefully we will converge to the truth fairly quickly !</div><div><br class=""><blockquote type="cite" class=""><div class=""><div class=""><br class=""><blockquote type="cite" class=""><blockquote type="cite" class="">I would like to see this work performed on a tier-1 architecture [1]. At this<br class="">point in time SPARC (and LEON) is not tier-1 because there are no test results<br class="">from hardware posted to build @ <a href="http://rtems.org" class="">rtems.org</a> <<a href="http://rtems.org" class="">http://rtems.org</a>>. This leaves the<br class="">ESA pre-qual project<br class="">with some choices. You could select a suitable target like a beagleboneblack and<br class="">post hardware test results or encourage other members of the pre-qual project to<br class="">run the testsuite on a LEON and post the results. We could then move SPARC and<br class="">LEON to tier-1, something I personally would love to see.<br class=""></blockquote><br class="">In one sense the ESA project has no choice - we have to work with<br class="">leon3/gr712rc/gr740.<br class=""></blockquote><br class="">That is fine. We understand few people often see the real hardware with flight<br class="">software development. We are relaxed on how an architecture can reach tier-1, we<br class="">need only one device from a family to run the tests with the results being<br class="">published. My desire is to automatically monitor the test results and age out<br class="">older results.<br class=""><br class=""><blockquote type="cite" class="">I should point out that a test generated by us has been run (standalone) on<br class="">gr740 hardware at ESTEC.<br class="">Would it help if the results of that was posted? <br class=""></blockquote><br class="">The test output needs to be posted using the public rtems-tools.git `rtems-test`<br class="">command. For public test results to be meaningful the tools used to generate the<br class="">results need to be public.<br class=""></div></div></blockquote><div><br class=""></div>That is certainly the plan. <br class=""><blockquote type="cite" class=""><div class=""><div class=""><br class=""><blockquote type="cite" class="">My understanding is that as our project rolls out we will be running hardware tests.<br class=""></blockquote><br class="">This is understandable and what I would expect. The role of tiers in RTEMS is to<br class="">bring the results out into the public so all users can view where each<br class="">architecture and device family sits. I personally see value for hardware vendors<br class="">having devices in tier-1 and I see value for large organisation like ESA<br class="">expecting to see their selected hardware in tier-1. You and I will not make this<br class="">happen.<br class=""><br class=""><blockquote type="cite" class="">However, the way I built the test executables was using waf configured for the<br class="">above three BSPs, and so it should be possible to do it for another tier-1 architecture.<br class=""></blockquote><br class="">If the tests end up in the testsuite the tests will be required to pass on all<br class="">supported architectures. We consider the range of architecture we support a real<br class="">feature because the code needs to be clean.<br class=""></div></div></blockquote><div><br class=""></div>I am currently looking at the Events Manager API top-down, to produce models and tests.</div><div>I see no reason why these would be biased towards any particular BSP, and would hope to</div><div>run these generated tests on a wide range of BSPs at some stage.</div><div><br class=""></div><div><br class=""></div><div>Andrew<br class=""></div><br class=""><div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">--------------------------------------------------------------------<br class="">Andrew Butterfield Tel: +353-1-896-2517 Fax: +353-1-677-2204<br class="">Lero@TCD, Head of Software Foundations & Verification Research Group<br class="">School of Computer Science and Statistics,<br class="">Room G.39, O'Reilly Institute, Trinity College, University of Dublin<br class=""> <a href="http://www.scss.tcd.ie/Andrew.Butterfield/" class="">http://www.scss.tcd.ie/Andrew.Butterfield/</a><br class="">--------------------------------------------------------------------</div>
</div>
<br class=""></div></body></html>