How to Classify Intermittent Test Failures
gedare at rtems.org
Tue Feb 2 16:13:54 UTC 2021
On Tue, Feb 2, 2021 at 7:40 AM Joel Sherrill <joel at rtems.org> wrote:
> On Mon, Feb 1, 2021 at 6:50 PM Chris Johns <chrisj at rtems.org> wrote:
>> On 2/2/21 9:12 am, Joel Sherrill wrote:
>> > On Mon, Feb 1, 2021 at 3:50 PM Chris Johns <chrisj at rtems.org
>> > <mailto:chrisj at rtems.org>> wrote:
>> > On 2/2/21 3:42 am, Joel Sherrill wrote:
>> > > Hi
>> > >
>> > > On the aarch64 qemu testing, we are seeing some tests which seem
>> to pass
>> > most of
>> > > the time but fail intermittently. It appears to be based somewhat
>> on host load
>> > > but there may be other factors.
>> > >
>> > > There does not appear to be a good test results state for these.
>> Marking them
>> > > expected pass or fail means they will get flagged incorrectly
>> > We have the test state 'indeterminate' ...
>> > <
>> > It is for this type of test result.
>> > > I don't see not running them as a good option. Beyond adding a
>> new state to
>> > > reflect this oddity, any suggestions?
>> > I prefer we used the already defined and documented state.
>> > +1
>> > Kinsey had already marked them as indeterminate and the guys were in
>> > process of documenting why. I interpreted the question of what to do
>> > broadly than it needed to be but the discussion was good.
>> A discussion is needed and welcome. Handling these intermittent simulator
>> failures is hard. I once looked into some gdb simulator cases when I
>> first put
>> rtems-test together and found myself quickly heading into a deep dark
>> hole. I
>> have not been back since.
> Agreed it is ugly.
> If the BSP has a simulator variant, then using the test configuration is
> But for the PC and leon3, we don't have separate sim builds of the BSP so
> there are intermittent failures there, we would have to mark them in the
> set shared with hardware test runs. That's bad.
yeah, don't do that.
> It's almost like we might need a conditional like "sp04: intermittent
> or something. Which means build it but the tester ini could know the
> type and adjust its expectations. May have to account for multiple
> on the sim=XXX though. Just a thought.
> maybe pass a sim.tcfg file to tester that is different for the sim.cfg
file than it is for the hw.cfg file?
> This is a dark hole.
> devel mailing list
> devel at rtems.org
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the devel