How to Classify Intermittent Test Failures

[Joel Sherrill]

On Mon, Feb 1, 2021, 11:45 AM Gedare Bloom <gedare at rtems.org> wrote:

>
>
> On Mon, Feb 1, 2021 at 10:31 AM Joel Sherrill <joel at rtems.org> wrote:
>
>>
>>
>> On Mon, Feb 1, 2021 at 11:14 AM Gedare Bloom <gedare at rtems.org> wrote:
>>
>>>
>>>
>>> On Mon, Feb 1, 2021 at 9:42 AM Joel Sherrill <joel at rtems.org> wrote:
>>>
>>>> Hi
>>>>
>>>> On the aarch64 qemu testing, we are seeing some tests which seem to
>>>> pass most of the time but fail intermittently. It appears to be based
>>>> somewhat on host load but there may be other factors.
>>>>
>>>>
>>> There does not appear to be a good test results state for these. Marking
>>>> them expected pass or fail means they will get flagged incorrectly
>>>> sometimes.
>>>>
>>>>
>>> Are they expected to pass every time?
>>>
>>
>> Normally yes. But there appears to be some external factors particularly
>> load impacting qemu which lead to failures on target side code.
>>
>>
> Since they can be made to pass by reducing the load (I assume), then you
> should document this in the target documentation and not change the testing
> expectations. The test should pass. If too much load cause qemu to fail
> randomly, this should be known and avoided or else your test results are
> much less useful.
>

I suggested a short paragraph in the qemu section of the BSP section in the
users guide and comments in the waf equivalent of a tcfg file. But there
isn't any other obvious place where someone would.look.


> Does anyone know this happens on other architectures with qemu?
>

Kinsey has noted the same on arm.

>
>
>>
>>> Intermittent failures are suspicious, and there is limited value to
>>> running a test that has an "intermittent failure" state, since it will
>>> always be successful if you don't care if it passes or fails.
>>>
>>
>> Yeah. No matter what you mark them, it sucks.
>>
>>>
>>>
>>>
>>>> I don't see not running them as a good option. Beyond adding a new
>>>> state to reflect this oddity, any suggestions?
>>>>
>>>> --joel
>>>> _______________________________________________
>>>> devel mailing list
>>>> devel at rtems.org
>>>> http://lists.rtems.org/mailman/listinfo/devel
>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rtems.org/pipermail/devel/attachments/20210201/17719fc1/attachment.html>