[PATCH] aarch64: Add tests that are failing intermittently
Kinsey Moore
kinsey.moore at oarcorp.com
Thu Aug 19 18:55:01 UTC 2021
On 8/19/2021 13:32, Gedare Bloom wrote:
> On Thu, Aug 19, 2021 at 11:43 AM Kinsey Moore <kinsey.moore at oarcorp.com> wrote:
>> I've seen these failures on my local system, in our CI, and on a build
>> server that I sometimes
>> use for development/testing so if it's a configuration issue we're being
>> pretty consistent about
>> misconfiguration across some pretty different environments (docker,
>> bare-metal, VM, different
>> OSs, different QEMU versions). I've seen enough of the spintrcritical
>> tests fail sporadically on
>> QEMU to lump them all into this category. These are also tests that I
>> have seen behave badly
>> on ARMv7 QEMU on my local system (which doesn't rule out
>> misconfiguration, but it's another
>> data point).
>>
> Yes, for example, it may be a matter of qemu process counts spawned by
> rtems-test, and the order in which tests get invoked could be a cause
> for which ones don't work. I could easily see this happening, since
> each test runtime will be fairly consistent, so you'll often see the
> same tests running concurrently with each other. But, if you change
> the order (e.g., by adding new tests), then we may see a new set of
> sporadically failing testcases, will we just add those, or do we need
> to re-examine this indetermine set periodically? Who will maintain
> this list? That's kind of the root of my concern here.
I understand your concern about maintenance of the failure list and I don't
have a good answer for you. I imagine going forward it would be a
combination
of the current stake-holders for a given BSP and anyone who watches the
automated build output from Joel's runs for these kinds of issues.
On the other hand if we don't mark those tests, people will get fatigued
looking at the spurious failures and assume any new ones just fall into the
same category as others. At that point is it even worth running the
automated tests for that platform?
>
>> As far as your worry about marking these indeterminate, they're only
>> being marked as such for
>> QEMU BSPs. The ZynqMP hardware BSP doesn't have these testing carve-outs
>> and runs all
>> these tests flawlessly.
>>
>> These failures become much more common when there is otherwise load on
>> the system and a
>> lot of them disappear when you limit the tester to a single QEMU
>> instance at a time.
>>
> I'm wondering if we should sacrifice testing speed for
> coverage/quality. If throttling rtems-test leads to more reliable test
> results, then it may be a better option than basically ignoring a
> swath of our testsuite.
That would certainly mitigate some of the failures, but you'd also have to
guarantee nothing else is running on the system which could cause the same
problem. I know at least some of the current automated runs operate on a
shared system which can and does often have other intensive processes
running on it. There are also the tests that are sporadic on QEMU even
without additional load.
>
>> Kinsey
>>
>>
>> On 8/19/2021 11:58, Gedare Bloom wrote:
>>> Can you explain the process for generating the lists of indeterminate
>>> test results?
>>>
>>> I hate to circle this subject so many times, but is labeling sporadic
>>> simulator failures as indeterminate results really the right thing to
>>> do? Are these indeterminate tests reproducible on different
>>> systems/qemus/loads? Or is it just what you observe locally when
>>> running rtems-test on one specific system? I don't think I see nearly
>>> so many spurious failures when I run rtems-test for example. I really
>>> need to believe we're not just hiding a system configuration problem.
>>>
>>> I know I OK'd looking at the versal, but on second thought, I'd rather
>>> leave the xilinx-versal/tstqemu.yml alone until the BSP is finished,
>>> so revert that part of your patch. Sorry about that.
>>>
>>> Gedare
>>>
>>> On Thu, Aug 19, 2021 at 9:53 AM Ryan Long <ryan.long at oarcorp.com> wrote:
>>>> - Change status of all spintrcritical tests to indeterminate, expanded upon
>>>> comments.
>>>> - Add indeterminate tests to xilinx-versal
>>>> ---
>>>> spec/build/bsps/aarch64/a53/tsta53.yml | 40 ++++++++++++++---
>>>> spec/build/bsps/aarch64/xilinx-versal/tstqemu.yml | 54 ++++++++++++++++++++++-
>>>> spec/build/bsps/aarch64/xilinx-zynqmp/tstqemu.yml | 40 ++++++++++++++---
>>>> 3 files changed, 120 insertions(+), 14 deletions(-)
>>>>
>>>> diff --git a/spec/build/bsps/aarch64/a53/tsta53.yml b/spec/build/bsps/aarch64/a53/tsta53.yml
>>>> index f263557..6e8f348 100644
>>>> --- a/spec/build/bsps/aarch64/a53/tsta53.yml
>>>> +++ b/spec/build/bsps/aarch64/a53/tsta53.yml
>>>> @@ -1,20 +1,26 @@
>>>> SPDX-License-Identifier: CC-BY-SA-4.0 OR BSD-2-Clause
>>>> actions:
>>>> - set-test-state:
>>>> - # expected to fail, don't compile these
>>>> + # This test fails when ran through rtems-tester because it does not
>>>> + # produce any output.
>>>> minimum: exclude
>>>>
>>>> - # don't compile due to toolchain issues
>>>> + # These tests do not compile due to an issue with the GNU Assembler.
>>>> + # The issue has been filed(https://devel.rtems.org/ticket/4218).
>>>> + # Once the issue has been fixed, these tests will be turned back on.
>>>> spconfig01: exclude
>>>> spmisc01: exclude
>>>>
>>>> - # tests that are passing intermittently
>>>> + # These tests may or may not fail, however, they do pass on real hardware.
>>>> + # It seems to be an issue with QEMU.
>>>> spcpucounter01: indeterminate
>>>> + sptimecounter01: indeterminate
>>>> rtmonuse: indeterminate
>>>> - sp68: indeterminate
>>>> sp04: indeterminate
>>>> sp20: indeterminate
>>>> + sp68: indeterminate
>>>> sp69: indeterminate
>>>> + sp71: indeterminate
>>>> rtmonusxtimes01: indeterminate
>>>> spedfsched02: indeterminate
>>>> spedfsched04: indeterminate
>>>> @@ -24,12 +30,34 @@ actions:
>>>> sptimecounter04: indeterminate
>>>> ttest02: indeterminate
>>>>
>>>> - # tests that pass nominally, but fail under Qemu when the host is under
>>>> - # heavy load
>>>> + # These tests may or may not fail, however, they do pass on real hardware.
>>>> + # It seems to be an issue with Qemu, and that this only occurs when the
>>>> + # host machine is under a heavy load.
>>>> psx12: indeterminate
>>>> + spintrcritical01: indeterminate
>>>> + spintrcritical02: indeterminate
>>>> spintrcritical03: indeterminate
>>>> spintrcritical04: indeterminate
>>>> spintrcritical05: indeterminate
>>>> + spintrcritical06: indeterminate
>>>> + spintrcritical07: indeterminate
>>>> + spintrcritical08: indeterminate
>>>> + spintrcritical09: indeterminate
>>>> + spintrcritical10: indeterminate
>>>> + spintrcritical11: indeterminate
>>>> + spintrcritical12: indeterminate
>>>> + spintrcritical13: indeterminate
>>>> + spintrcritical14: indeterminate
>>>> + spintrcritical15: indeterminate
>>>> + spintrcritical16: indeterminate
>>>> + spintrcritical17: indeterminate
>>>> + spintrcritical18: indeterminate
>>>> + spintrcritical19: indeterminate
>>>> + spintrcritical20: indeterminate
>>>> + spintrcritical21: indeterminate
>>>> + spintrcritical22: indeterminate
>>>> + spintrcritical23: indeterminate
>>>> + spintrcritical24: indeterminate
>>>> build-type: option
>>>> copyrights:
>>>> - Copyright (C) 2020 On-Line Applications Research (OAR)
>>>> diff --git a/spec/build/bsps/aarch64/xilinx-versal/tstqemu.yml b/spec/build/bsps/aarch64/xilinx-versal/tstqemu.yml
>>>> index 43f6b2e..884effc 100644
>>>> --- a/spec/build/bsps/aarch64/xilinx-versal/tstqemu.yml
>>>> +++ b/spec/build/bsps/aarch64/xilinx-versal/tstqemu.yml
>>>> @@ -1,13 +1,63 @@
>>>> SPDX-License-Identifier: CC-BY-SA-4.0 OR BSD-2-Clause
>>>> actions:
>>>> - set-test-state:
>>>> - # expected to fail
>>>> + # This test fails when ran through rtems-tester because it does not
>>>> + # produce any output.
>>>> minimum: exclude
>>>>
>>>> - # don't compile due to toolchain issues, see RTEMS issue #4218
>>>> + # These tests do not compile due to an issue with the GNU Assembler.
>>>> + # The issue has been filed(https://devel.rtems.org/ticket/4218).
>>>> + # Once the issue has been fixed, these tests will be turned back on.
>>>> spconfig01: exclude
>>>> spmisc01: exclude
>>>>
>>>> + # These tests may or may not fail, however, they do pass on real hardware.
>>>> + # It seems to be an issue with Qemu.
>>>> + spcpucounter01: indeterminate
>>>> + sptimecounter01: indeterminate
>>>> + rtmonuse: indeterminate
>>>> + sp04: indeterminate
>>>> + sp20: indeterminate
>>>> + sp68: indeterminate
>>>> + sp69: indeterminate
>>>> + sp71: indeterminate
>>>> + rtmonusxtimes01: indeterminate
>>>> + spedfsched02: indeterminate
>>>> + spedfsched04: indeterminate
>>>> + psxtimes01: indeterminate
>>>> + sprmsched01: indeterminate
>>>> + sptimecounter02: indeterminate
>>>> + sptimecounter04: indeterminate
>>>> + ttest02: indeterminate
>>>> +
>>>> + # These tests may or may not fail, however, they do pass on real hardware.
>>>> + # It seems to be an issue with Qemu, and that this only occurs when the
>>>> + # host machine is under a heavy load.
>>>> + psx12: indeterminate
>>>> + spintrcritical01: indeterminate
>>>> + spintrcritical02: indeterminate
>>>> + spintrcritical03: indeterminate
>>>> + spintrcritical04: indeterminate
>>>> + spintrcritical05: indeterminate
>>>> + spintrcritical06: indeterminate
>>>> + spintrcritical07: indeterminate
>>>> + spintrcritical08: indeterminate
>>>> + spintrcritical09: indeterminate
>>>> + spintrcritical10: indeterminate
>>>> + spintrcritical11: indeterminate
>>>> + spintrcritical12: indeterminate
>>>> + spintrcritical13: indeterminate
>>>> + spintrcritical14: indeterminate
>>>> + spintrcritical15: indeterminate
>>>> + spintrcritical16: indeterminate
>>>> + spintrcritical17: indeterminate
>>>> + spintrcritical18: indeterminate
>>>> + spintrcritical19: indeterminate
>>>> + spintrcritical20: indeterminate
>>>> + spintrcritical21: indeterminate
>>>> + spintrcritical22: indeterminate
>>>> + spintrcritical23: indeterminate
>>>> + spintrcritical24: indeterminate
>>>> build-type: option
>>>> copyrights:
>>>> - Copyright (C) 2021 Gedare Bloom <gedare at rtems.org>
>>>> diff --git a/spec/build/bsps/aarch64/xilinx-zynqmp/tstqemu.yml b/spec/build/bsps/aarch64/xilinx-zynqmp/tstqemu.yml
>>>> index efe0b82..06929ed 100644
>>>> --- a/spec/build/bsps/aarch64/xilinx-zynqmp/tstqemu.yml
>>>> +++ b/spec/build/bsps/aarch64/xilinx-zynqmp/tstqemu.yml
>>>> @@ -1,20 +1,26 @@
>>>> SPDX-License-Identifier: CC-BY-SA-4.0 OR BSD-2-Clause
>>>> actions:
>>>> - set-test-state:
>>>> - # expected to fail
>>>> + # This test fails when ran through rtems-tester because it does not
>>>> + # produce any output.
>>>> minimum: exclude
>>>>
>>>> - # don't compile due to toolchain issues
>>>> + # These tests do not compile due to an issue with the GNU Assembler.
>>>> + # The issue has been filed(https://devel.rtems.org/ticket/4218).
>>>> + # Once the issue has been fixed, these tests will be turned back on.
>>>> spconfig01: exclude
>>>> spmisc01: exclude
>>>>
>>>> - # tests that are passing intermittently
>>>> + # These tests may or may not fail, however, they do pass on real hardware.
>>>> + # It seems to be an issue with Qemu.
>>>> spcpucounter01: indeterminate
>>>> + sptimecounter01: indeterminate
>>>> rtmonuse: indeterminate
>>>> - sp68: indeterminate
>>>> sp04: indeterminate
>>>> sp20: indeterminate
>>>> + sp68: indeterminate
>>>> sp69: indeterminate
>>>> + sp71: indeterminate
>>>> rtmonusxtimes01: indeterminate
>>>> spedfsched02: indeterminate
>>>> spedfsched04: indeterminate
>>>> @@ -24,12 +30,34 @@ actions:
>>>> sptimecounter04: indeterminate
>>>> ttest02: indeterminate
>>>>
>>>> - # tests that pass nominally, but fail under Qemu when the host is under
>>>> - # heavy load
>>>> + # These tests may or may not fail, however, they do pass on real hardware.
>>>> + # It seems to be an issue with Qemu, and that this only occurs when the
>>>> + # host machine is under a heavy load.
>>>> psx12: indeterminate
>>>> + spintrcritical01: indeterminate
>>>> + spintrcritical02: indeterminate
>>>> spintrcritical03: indeterminate
>>>> spintrcritical04: indeterminate
>>>> spintrcritical05: indeterminate
>>>> + spintrcritical06: indeterminate
>>>> + spintrcritical07: indeterminate
>>>> + spintrcritical08: indeterminate
>>>> + spintrcritical09: indeterminate
>>>> + spintrcritical10: indeterminate
>>>> + spintrcritical11: indeterminate
>>>> + spintrcritical12: indeterminate
>>>> + spintrcritical13: indeterminate
>>>> + spintrcritical14: indeterminate
>>>> + spintrcritical15: indeterminate
>>>> + spintrcritical16: indeterminate
>>>> + spintrcritical17: indeterminate
>>>> + spintrcritical18: indeterminate
>>>> + spintrcritical19: indeterminate
>>>> + spintrcritical20: indeterminate
>>>> + spintrcritical21: indeterminate
>>>> + spintrcritical22: indeterminate
>>>> + spintrcritical23: indeterminate
>>>> + spintrcritical24: indeterminate
>>>> build-type: option
>>>> copyrights:
>>>> - Copyright (C) 2020 On-Line Applications Research (OAR)
>>>> --
>>>> 1.8.3.1
>>>>
>>>> _______________________________________________
>>>> devel mailing list
>>>> devel at rtems.org
>>>> http://lists.rtems.org/mailman/listinfo/devel
>>> _______________________________________________
>>> devel mailing list
>>> devel at rtems.org
>>> http://lists.rtems.org/mailman/listinfo/devel
>> _______________________________________________
>> devel mailing list
>> devel at rtems.org
>> http://lists.rtems.org/mailman/listinfo/devel
More information about the devel
mailing list