[PATCH] aarch64: Add tests that are failing intermittently

Thu Aug 19 16:58:21 UTC 2021

Can you explain the process for generating the lists of indeterminate
test results?

I hate to circle this subject so many times, but is labeling sporadic
simulator failures as indeterminate results really the right thing to
do? Are these indeterminate tests reproducible on different
systems/qemus/loads? Or is it just what you observe locally when
running rtems-test on one specific system? I don't think I see nearly
so many spurious failures when I run rtems-test for example. I really
need to believe we're not just hiding a system configuration problem.

I know I OK'd looking at the versal, but on second thought, I'd rather
leave the xilinx-versal/tstqemu.yml alone until the BSP is finished,
so revert that part of your patch. Sorry about that.

Gedare

On Thu, Aug 19, 2021 at 9:53 AM Ryan Long <ryan.long at oarcorp.com> wrote:
>
> - Change status of all spintrcritical tests to indeterminate, expanded upon
>   comments.
> - Add indeterminate tests to xilinx-versal
> ---
>  spec/build/bsps/aarch64/a53/tsta53.yml            | 40 ++++++++++++++---
>  spec/build/bsps/aarch64/xilinx-versal/tstqemu.yml | 54 ++++++++++++++++++++++-
>  spec/build/bsps/aarch64/xilinx-zynqmp/tstqemu.yml | 40 ++++++++++++++---
>  3 files changed, 120 insertions(+), 14 deletions(-)
>
> diff --git a/spec/build/bsps/aarch64/a53/tsta53.yml b/spec/build/bsps/aarch64/a53/tsta53.yml
> index f263557..6e8f348 100644
> --- a/spec/build/bsps/aarch64/a53/tsta53.yml
> +++ b/spec/build/bsps/aarch64/a53/tsta53.yml
> @@ -1,20 +1,26 @@
>  SPDX-License-Identifier: CC-BY-SA-4.0 OR BSD-2-Clause
>  actions:
>  - set-test-state:
> -    # expected to fail, don't compile these
> +    # This test fails when ran through rtems-tester because it does not
> +    # produce any output.
>      minimum: exclude
>
> -    # don't compile due to toolchain issues
> +    # These tests do not compile due to an issue with the GNU Assembler.
> +    # The issue has been filed(https://devel.rtems.org/ticket/4218).
> +    # Once the issue has been fixed, these tests will be turned back on.
>      spconfig01: exclude
>      spmisc01: exclude
>
> -    # tests that are passing intermittently
> +    # These tests may or may not fail, however, they do pass on real hardware.
> +    # It seems to be an issue with QEMU.
>      spcpucounter01: indeterminate
> +    sptimecounter01: indeterminate
>      rtmonuse: indeterminate
> -    sp68: indeterminate
>      sp04: indeterminate
>      sp20: indeterminate
> +    sp68: indeterminate
>      sp69: indeterminate
> +    sp71: indeterminate
>      rtmonusxtimes01: indeterminate
>      spedfsched02: indeterminate
>      spedfsched04: indeterminate
> @@ -24,12 +30,34 @@ actions:
>      sptimecounter04: indeterminate
>      ttest02: indeterminate
>
> -    # tests that pass nominally, but fail under Qemu when the host is under
> -    # heavy load
> +    # These tests may or may not fail, however, they do pass on real hardware.
> +    # It seems to be an issue with Qemu, and that this only occurs when the
> +    # host machine is under a heavy load.
>      psx12: indeterminate
> +    spintrcritical01: indeterminate
> +    spintrcritical02: indeterminate
>      spintrcritical03: indeterminate
>      spintrcritical04: indeterminate
>      spintrcritical05: indeterminate
> +    spintrcritical06: indeterminate
> +    spintrcritical07: indeterminate
> +    spintrcritical08: indeterminate
> +    spintrcritical09: indeterminate
> +    spintrcritical10: indeterminate
> +    spintrcritical11: indeterminate
> +    spintrcritical12: indeterminate
> +    spintrcritical13: indeterminate
> +    spintrcritical14: indeterminate
> +    spintrcritical15: indeterminate
> +    spintrcritical16: indeterminate
> +    spintrcritical17: indeterminate
> +    spintrcritical18: indeterminate
> +    spintrcritical19: indeterminate
> +    spintrcritical20: indeterminate
> +    spintrcritical21: indeterminate
> +    spintrcritical22: indeterminate
> +    spintrcritical23: indeterminate
> +    spintrcritical24: indeterminate
>  build-type: option
>  copyrights:
>  - Copyright (C) 2020 On-Line Applications Research (OAR)
> diff --git a/spec/build/bsps/aarch64/xilinx-versal/tstqemu.yml b/spec/build/bsps/aarch64/xilinx-versal/tstqemu.yml
> index 43f6b2e..884effc 100644
> --- a/spec/build/bsps/aarch64/xilinx-versal/tstqemu.yml
> +++ b/spec/build/bsps/aarch64/xilinx-versal/tstqemu.yml
> @@ -1,13 +1,63 @@
>  SPDX-License-Identifier: CC-BY-SA-4.0 OR BSD-2-Clause
>  actions:
>  - set-test-state:
> -    # expected to fail
> +    # This test fails when ran through rtems-tester because it does not
> +    # produce any output.
>      minimum: exclude
>
> -    # don't compile due to toolchain issues, see RTEMS issue #4218
> +    # These tests do not compile due to an issue with the GNU Assembler.
> +    # The issue has been filed(https://devel.rtems.org/ticket/4218).
> +    # Once the issue has been fixed, these tests will be turned back on.
>      spconfig01: exclude
>      spmisc01: exclude
>
> +    # These tests may or may not fail, however, they do pass on real hardware.
> +    # It seems to be an issue with Qemu.
> +    spcpucounter01: indeterminate
> +    sptimecounter01: indeterminate
> +    rtmonuse: indeterminate
> +    sp04: indeterminate
> +    sp20: indeterminate
> +    sp68: indeterminate
> +    sp69: indeterminate
> +    sp71: indeterminate
> +    rtmonusxtimes01: indeterminate
> +    spedfsched02: indeterminate
> +    spedfsched04: indeterminate
> +    psxtimes01: indeterminate
> +    sprmsched01: indeterminate
> +    sptimecounter02: indeterminate
> +    sptimecounter04: indeterminate
> +    ttest02: indeterminate
> +
> +    # These tests may or may not fail, however, they do pass on real hardware.
> +    # It seems to be an issue with Qemu, and that this only occurs when the
> +    # host machine is under a heavy load.
> +    psx12: indeterminate
> +    spintrcritical01: indeterminate
> +    spintrcritical02: indeterminate
> +    spintrcritical03: indeterminate
> +    spintrcritical04: indeterminate
> +    spintrcritical05: indeterminate
> +    spintrcritical06: indeterminate
> +    spintrcritical07: indeterminate
> +    spintrcritical08: indeterminate
> +    spintrcritical09: indeterminate
> +    spintrcritical10: indeterminate
> +    spintrcritical11: indeterminate
> +    spintrcritical12: indeterminate
> +    spintrcritical13: indeterminate
> +    spintrcritical14: indeterminate
> +    spintrcritical15: indeterminate
> +    spintrcritical16: indeterminate
> +    spintrcritical17: indeterminate
> +    spintrcritical18: indeterminate
> +    spintrcritical19: indeterminate
> +    spintrcritical20: indeterminate
> +    spintrcritical21: indeterminate
> +    spintrcritical22: indeterminate
> +    spintrcritical23: indeterminate
> +    spintrcritical24: indeterminate
>  build-type: option
>  copyrights:
>  - Copyright (C) 2021 Gedare Bloom <gedare at rtems.org>
> diff --git a/spec/build/bsps/aarch64/xilinx-zynqmp/tstqemu.yml b/spec/build/bsps/aarch64/xilinx-zynqmp/tstqemu.yml
> index efe0b82..06929ed 100644
> --- a/spec/build/bsps/aarch64/xilinx-zynqmp/tstqemu.yml
> +++ b/spec/build/bsps/aarch64/xilinx-zynqmp/tstqemu.yml
> @@ -1,20 +1,26 @@
>  SPDX-License-Identifier: CC-BY-SA-4.0 OR BSD-2-Clause
>  actions:
>  - set-test-state:
> -    # expected to fail
> +    # This test fails when ran through rtems-tester because it does not
> +    # produce any output.
>      minimum: exclude
>
> -    # don't compile due to toolchain issues
> +    # These tests do not compile due to an issue with the GNU Assembler.
> +    # The issue has been filed(https://devel.rtems.org/ticket/4218).
> +    # Once the issue has been fixed, these tests will be turned back on.
>      spconfig01: exclude
>      spmisc01: exclude
>
> -    # tests that are passing intermittently
> +    # These tests may or may not fail, however, they do pass on real hardware.
> +    # It seems to be an issue with Qemu.
>      spcpucounter01: indeterminate
> +    sptimecounter01: indeterminate
>      rtmonuse: indeterminate
> -    sp68: indeterminate
>      sp04: indeterminate
>      sp20: indeterminate
> +    sp68: indeterminate
>      sp69: indeterminate
> +    sp71: indeterminate
>      rtmonusxtimes01: indeterminate
>      spedfsched02: indeterminate
>      spedfsched04: indeterminate
> @@ -24,12 +30,34 @@ actions:
>      sptimecounter04: indeterminate
>      ttest02: indeterminate
>
> -    # tests that pass nominally, but fail under Qemu when the host is under
> -    # heavy load
> +    # These tests may or may not fail, however, they do pass on real hardware.
> +    # It seems to be an issue with Qemu, and that this only occurs when the
> +    # host machine is under a heavy load.
>      psx12: indeterminate
> +    spintrcritical01: indeterminate
> +    spintrcritical02: indeterminate
>      spintrcritical03: indeterminate
>      spintrcritical04: indeterminate
>      spintrcritical05: indeterminate
> +    spintrcritical06: indeterminate
> +    spintrcritical07: indeterminate
> +    spintrcritical08: indeterminate
> +    spintrcritical09: indeterminate
> +    spintrcritical10: indeterminate
> +    spintrcritical11: indeterminate
> +    spintrcritical12: indeterminate
> +    spintrcritical13: indeterminate
> +    spintrcritical14: indeterminate
> +    spintrcritical15: indeterminate
> +    spintrcritical16: indeterminate
> +    spintrcritical17: indeterminate
> +    spintrcritical18: indeterminate
> +    spintrcritical19: indeterminate
> +    spintrcritical20: indeterminate
> +    spintrcritical21: indeterminate
> +    spintrcritical22: indeterminate
> +    spintrcritical23: indeterminate
> +    spintrcritical24: indeterminate
>  build-type: option
>  copyrights:
>  - Copyright (C) 2020 On-Line Applications Research (OAR)
> --
> 1.8.3.1
>
> _______________________________________________
> devel mailing list
> devel at rtems.org
> http://lists.rtems.org/mailman/listinfo/devel