How to Classify Intermittent Test Failures

[Gedare Bloom]

On Mon, Feb 1, 2021 at 11:02 AM Joel Sherrill <joel at rtems.org> wrote:

>
>
> On Mon, Feb 1, 2021 at 11:56 AM Peter Dufault <dufault at hda.com> wrote:
>
>> Can the test be marked as a possible emulator failure?  For example (I
>> haven't looked at the report) use "?" (Huh?) and not "!" (Failed!) or "."
>> )(Passed), that is, the test failed as an emulated system and the RTEMS
>> project suspects the emulation itself is causing this.
>>
>
> That's not bad. The BSPs which might want to use this sometimes are qemu
> only but we have a lot of BSPs (pc, sparc, etc) that run on both simulator
> and real hardware. For ones like the Zynq where you have a qemu only
> variant, this would be perfect.  But for ones which can run on both, I
> don't know how to eliminate the use on hardware.
>

I'm fine with this. Unlike hardware, emulation/simulation bugs are
transient and can be safely ignored (to an extent) by us.


>
>> You shouldn't be allowed to use this on physical devices.
>>
>
> +1
>

Yes, I had initially thought perhaps this is like hardware bugs/errata, but
it is not. You would either fix/workaround the hardware (and want to know
it is broken/failing), or you would abandon that device :)


>
>> > On Feb 1, 2021, at 12:31 , Joel Sherrill <joel at rtems.org> wrote:
>> >
>> >
>> >
>> > On Mon, Feb 1, 2021 at 11:14 AM Gedare Bloom <gedare at rtems.org> wrote:
>> >
>> >
>> > On Mon, Feb 1, 2021 at 9:42 AM Joel Sherrill <joel at rtems.org> wrote:
>> > Hi
>> >
>> > On the aarch64 qemu testing, we are seeing some tests which seem to
>> pass most of the time but fail intermittently. It appears to be based
>> somewhat on host load but there may be other factors.
>> >
>> > There does not appear to be a good test results state for these.
>> Marking them expected pass or fail means they will get flagged incorrectly
>> sometimes.
>> >
>> >
>> > Are they expected to pass every time?
>> >
>> > Normally yes. But there appears to be some external factors
>> particularly load impacting qemu which lead to failures on target side code.
>> >
>> >
>> > Intermittent failures are suspicious, and there is limited value to
>> running a test that has an "intermittent failure" state, since it will
>> always be successful if you don't care if it passes or fails.
>> >
>> > Yeah. No matter what you mark them, it sucks.
>> >
>> >
>> > I don't see not running them as a good option. Beyond adding a new
>> state to reflect this oddity, any suggestions?
>> >
>> > --joel
>> > _______________________________________________
>> > devel mailing list
>> > devel at rtems.org
>> > http://lists.rtems.org/mailman/listinfo/devel
>> > _______________________________________________
>> > devel mailing list
>> > devel at rtems.org
>> > http://lists.rtems.org/mailman/listinfo/devel
>>
>> Peter
>> -----------------
>> Peter Dufault
>> HD Associates, Inc.      Software and System Engineering
>>
>> This email is delivered through the public internet using protocols
>> subject to interception and tampering.
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rtems.org/pipermail/devel/attachments/20210201/5868b4df/attachment.html>