How to Classify Intermittent Test Failures

[Chris Johns]

On 2/2/21 3:42 am, Joel Sherrill wrote:
> Hi
> 
> On the aarch64 qemu testing, we are seeing some tests which seem to pass most of
> the time but fail intermittently. It appears to be based somewhat on host load
> but there may be other factors. 
> 
> There does not appear to be a good test results state for these. Marking them
> expected pass or fail means they will get flagged incorrectly sometimes.

We have the test state 'indeterminate' ...

https://docs.rtems.org/branches/master/user/testing/tests.html#expected-test-states

It is for this type of test result.

> I don't see not running them as a good option. Beyond adding a new state to
> reflect this oddity, any suggestions?

I prefer we used the already defined and documented state.

Chris