How to Classify Intermittent Test Failures

[Gedare Bloom]

On Mon, Feb 1, 2021 at 9:42 AM Joel Sherrill <joel at rtems.org> wrote:

> Hi
>
> On the aarch64 qemu testing, we are seeing some tests which seem to pass
> most of the time but fail intermittently. It appears to be based somewhat
> on host load but there may be other factors.
>
>
There does not appear to be a good test results state for these. Marking
> them expected pass or fail means they will get flagged incorrectly
> sometimes.
>
>
Are they expected to pass every time?

Intermittent failures are suspicious, and there is limited value to running
a test that has an "intermittent failure" state, since it will always be
successful if you don't care if it passes or fails.



> I don't see not running them as a good option. Beyond adding a new state
> to reflect this oddity, any suggestions?
>
> --joel
> _______________________________________________
> devel mailing list
> devel at rtems.org
> http://lists.rtems.org/mailman/listinfo/devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rtems.org/pipermail/devel/attachments/20210201/6fdf2963/attachment.html>