<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Feb 2, 2021 at 7:40 AM Joel Sherrill <<a href="mailto:joel@rtems.org">joel@rtems.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Feb 1, 2021 at 6:50 PM Chris Johns <<a href="mailto:chrisj@rtems.org" target="_blank">chrisj@rtems.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
<br>
On 2/2/21 9:12 am, Joel Sherrill wrote:<br>
> On Mon, Feb 1, 2021 at 3:50 PM Chris Johns <<a href="mailto:chrisj@rtems.org" target="_blank">chrisj@rtems.org</a><br>
> <mailto:<a href="mailto:chrisj@rtems.org" target="_blank">chrisj@rtems.org</a>>> wrote:<br>
> On 2/2/21 3:42 am, Joel Sherrill wrote:<br>
> > Hi<br>
> ><br>
> > On the aarch64 qemu testing, we are seeing some tests which seem to pass<br>
> most of<br>
> > the time but fail intermittently. It appears to be based somewhat on host load<br>
> > but there may be other factors. <br>
> ><br>
> > There does not appear to be a good test results state for these. Marking them<br>
> > expected pass or fail means they will get flagged incorrectly sometimes.<br>
> <br>
> We have the test state 'indeterminate' ...<br>
> <br>
> <a href="https://docs.rtems.org/branches/master/user/testing/tests.html#expected-test-states" rel="noreferrer" target="_blank">https://docs.rtems.org/branches/master/user/testing/tests.html#expected-test-states</a><br>
> <<a href="https://docs.rtems.org/branches/master/user/testing/tests.html#expected-test-states" rel="noreferrer" target="_blank">https://docs.rtems.org/branches/master/user/testing/tests.html#expected-test-states</a>><br>
> <br>
> It is for this type of test result.<br>
> <br>
> > I don't see not running them as a good option. Beyond adding a new state to<br>
> > reflect this oddity, any suggestions?<br>
> <br>
> I prefer we used the already defined and documented state.<br>
> <br>
> +1 <br>
> <br>
> Kinsey had already marked them as indeterminate and the guys were in the <br>
> process of documenting why. I interpreted the question of what to do more <br>
> broadly than it needed to be but the discussion was good.<br>
<br>
A discussion is needed and welcome. Handling these intermittent simulator<br>
failures is hard. I once looked into some gdb simulator cases when I first put<br>
rtems-test together and found myself quickly heading into a deep dark hole. I<br>
have not been back since.<br></blockquote><div><br></div><div>Agreed it is ugly.</div><div><br></div><div>If the BSP has a simulator variant, then using the test configuration is appropriate.</div><div><br></div><div>But for the PC and leon3, we don't have separate sim builds of the BSP so if </div><div>there are intermittent failures there, we would have to mark them in the </div><div>set shared with hardware test runs. That's bad.</div></div></div></blockquote><div>yeah, don't do that.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div><br></div><div>It's almost like we might need a conditional like "sp04: intermittent
sim=qemu"</div><div>or something. Which means build it but the tester ini could know the simulator</div><div>type and adjust its expectations. May have to account for multiple simulators</div><div>on the sim=XXX though. Just a thought. </div><div><br></div></div></div></blockquote><div>maybe pass a sim.tcfg file to tester that is different for the sim.cfg file than it is for the hw.cfg file?</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div></div><div>This is a dark hole.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Chris<br>
</blockquote></div></div>
_______________________________________________<br>
devel mailing list<br>
<a href="mailto:devel@rtems.org" target="_blank">devel@rtems.org</a><br>
<a href="http://lists.rtems.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.rtems.org/mailman/listinfo/devel</a></blockquote></div></div>