How to Classify Intermittent Test Failures

[Joel Sherrill]

On Mon, Feb 1, 2021 at 3:50 PM Chris Johns <chrisj at rtems.org> wrote:

> On 2/2/21 3:42 am, Joel Sherrill wrote:
> > Hi
> >
> > On the aarch64 qemu testing, we are seeing some tests which seem to pass
> most of
> > the time but fail intermittently. It appears to be based somewhat on
> host load
> > but there may be other factors.
> >
> > There does not appear to be a good test results state for these. Marking
> them
> > expected pass or fail means they will get flagged incorrectly sometimes.
>
> We have the test state 'indeterminate' ...
>
>
> https://docs.rtems.org/branches/master/user/testing/tests.html#expected-test-states
>
> It is for this type of test result.
>
> > I don't see not running them as a good option. Beyond adding a new state
> to
> > reflect this oddity, any suggestions?
>
> I prefer we used the already defined and documented state.


+1

Kinsey had already marked them as indeterminate and the guys were in the
process of documenting why. I interpreted the question of what to do more
broadly than it needed to be but the discussion was good.

--joel

>
> Chris
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rtems.org/pipermail/devel/attachments/20210201/8777d6e5/attachment-0001.html>