How to Classify Intermittent Test Failures

[Joel Sherrill]

On Mon, Feb 1, 2021 at 11:14 AM Gedare Bloom <gedare at rtems.org> wrote:

>
>
> On Mon, Feb 1, 2021 at 9:42 AM Joel Sherrill <joel at rtems.org> wrote:
>
>> Hi
>>
>> On the aarch64 qemu testing, we are seeing some tests which seem to pass
>> most of the time but fail intermittently. It appears to be based somewhat
>> on host load but there may be other factors.
>>
>>
> There does not appear to be a good test results state for these. Marking
>> them expected pass or fail means they will get flagged incorrectly
>> sometimes.
>>
>>
> Are they expected to pass every time?
>

Normally yes. But there appears to be some external factors particularly
load impacting qemu which lead to failures on target side code.


> Intermittent failures are suspicious, and there is limited value to
> running a test that has an "intermittent failure" state, since it will
> always be successful if you don't care if it passes or fails.
>

Yeah. No matter what you mark them, it sucks.

>
>
>
>> I don't see not running them as a good option. Beyond adding a new state
>> to reflect this oddity, any suggestions?
>>
>> --joel
>> _______________________________________________
>> devel mailing list
>> devel at rtems.org
>> http://lists.rtems.org/mailman/listinfo/devel
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rtems.org/pipermail/devel/attachments/20210201/522749d6/attachment-0001.html>