How to Classify Intermittent Test Failures

[Chris Johns]

On 2/2/21 9:12 am, Joel Sherrill wrote:
>  On Mon, Feb 1, 2021 at 3:50 PM Chris Johns <chrisj at rtems.org
> <mailto:chrisj at rtems.org>> wrote:
>     On 2/2/21 3:42 am, Joel Sherrill wrote:
>     > Hi
>     >
>     > On the aarch64 qemu testing, we are seeing some tests which seem to pass
>     most of
>     > the time but fail intermittently. It appears to be based somewhat on host load
>     > but there may be other factors. 
>     >
>     > There does not appear to be a good test results state for these. Marking them
>     > expected pass or fail means they will get flagged incorrectly sometimes.
> 
>     We have the test state 'indeterminate' ...
> 
>     https://docs.rtems.org/branches/master/user/testing/tests.html#expected-test-states
>     <https://docs.rtems.org/branches/master/user/testing/tests.html#expected-test-states>
> 
>     It is for this type of test result.
> 
>     > I don't see not running them as a good option. Beyond adding a new state to
>     > reflect this oddity, any suggestions?
> 
>     I prefer we used the already defined and documented state.
> 
> +1 
> 
> Kinsey had already marked them as indeterminate and the guys were in the 
> process of documenting why. I interpreted the question of what to do more 
> broadly than it needed to be but the discussion was good.

A discussion is needed and welcome. Handling these intermittent simulator
failures is hard. I once looked into some gdb simulator cases when I first put
rtems-test together and found myself quickly heading into a deep dark hole. I
have not been back since.

Chris