How to Classify Intermittent Test Failures

[Joel Sherrill]

On Mon, Feb 1, 2021 at 11:56 AM Peter Dufault <dufault at hda.com> wrote:

> Can the test be marked as a possible emulator failure?  For example (I
> haven't looked at the report) use "?" (Huh?) and not "!" (Failed!) or "."
> )(Passed), that is, the test failed as an emulated system and the RTEMS
> project suspects the emulation itself is causing this.
>

That's not bad. The BSPs which might want to use this sometimes are qemu
only but we have a lot of BSPs (pc, sparc, etc) that run on both simulator
and real hardware. For ones like the Zynq where you have a qemu only
variant, this would be perfect.  But for ones which can run on both, I
don't know how to eliminate the use on hardware.

>
> You shouldn't be allowed to use this on physical devices.
>

+1

>
> > On Feb 1, 2021, at 12:31 , Joel Sherrill <joel at rtems.org> wrote:
> >
> >
> >
> > On Mon, Feb 1, 2021 at 11:14 AM Gedare Bloom <gedare at rtems.org> wrote:
> >
> >
> > On Mon, Feb 1, 2021 at 9:42 AM Joel Sherrill <joel at rtems.org> wrote:
> > Hi
> >
> > On the aarch64 qemu testing, we are seeing some tests which seem to pass
> most of the time but fail intermittently. It appears to be based somewhat
> on host load but there may be other factors.
> >
> > There does not appear to be a good test results state for these. Marking
> them expected pass or fail means they will get flagged incorrectly
> sometimes.
> >
> >
> > Are they expected to pass every time?
> >
> > Normally yes. But there appears to be some external factors particularly
> load impacting qemu which lead to failures on target side code.
> >
> >
> > Intermittent failures are suspicious, and there is limited value to
> running a test that has an "intermittent failure" state, since it will
> always be successful if you don't care if it passes or fails.
> >
> > Yeah. No matter what you mark them, it sucks.
> >
> >
> > I don't see not running them as a good option. Beyond adding a new state
> to reflect this oddity, any suggestions?
> >
> > --joel
> > _______________________________________________
> > devel mailing list
> > devel at rtems.org
> > http://lists.rtems.org/mailman/listinfo/devel
> > _______________________________________________
> > devel mailing list
> > devel at rtems.org
> > http://lists.rtems.org/mailman/listinfo/devel
>
> Peter
> -----------------
> Peter Dufault
> HD Associates, Inc.      Software and System Engineering
>
> This email is delivered through the public internet using protocols
> subject to interception and tampering.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rtems.org/pipermail/devel/attachments/20210201/6ea9ddb9/attachment-0001.html>