[PATCH 3/3] bsps/aarch64: changed comments and test statuses

[Chris Johns]

On 19/3/21 7:27 am, Ryan Long wrote:
> Expanded upon the commments that were present previously.
> Added more tests that may finish as 'indeterminate'.
> ---
>  spec/build/bsps/aarch64/a53/tsta53.yml            | 39 +++++++++++++++++++----
>  spec/build/bsps/aarch64/xilinx-zynqmp/tstqemu.yml | 39 +++++++++++++++++++----
>  2 files changed, 66 insertions(+), 12 deletions(-)
> 
> diff --git a/spec/build/bsps/aarch64/a53/tsta53.yml b/spec/build/bsps/aarch64/a53/tsta53.yml
> index 6876d23..5b60f52 100644
> --- a/spec/build/bsps/aarch64/a53/tsta53.yml
> +++ b/spec/build/bsps/aarch64/a53/tsta53.yml
> @@ -1,20 +1,25 @@
>  SPDX-License-Identifier: CC-BY-SA-4.0 OR BSD-2-Clause
>  actions:
>  - set-test-state:
> -    # expected to fail, don't compile these
> +    # This test fails when ran through rtems-tester because it does not

Should be `run through` and `rtems-test`. There is no tool called rtems-tester.

> +    # produce any output.
>      minimum: exclude

Does this test do the same thing on other BSPs?

If so may be this test needs to be globally tagged as no-run.

> -    # don't compile due to toolchain issues
> +    # These tests do not compile due to an issue with the GNU Assembler.
> +    # The issue has been filed(https://devel.rtems.org/ticket/4218).
> +    # Once the issue has been fixed, these tests will be turned back on.
>      spconfig01: exclude
>      spmisc01: exclude
>  
> -    # tests that are passing intermittently
> +    # These tests may or may not fail, however, they do pass on real hardware.
> +    # It seems to be an issue with Qemu. and that this only occurs when the
>      spcpucounter01: indeterminate
>      rtmonuse: indeterminate
> -    sp68: indeterminate
>      sp04: indeterminate
>      sp20: indeterminate
> +    sp68: indeterminate
>      sp69: indeterminate
> +    sp71: indeterminate

Hmmm indeterminate test states to manage hardware and qemu ... this was
discussed in February and I do not think it was ever resolved ...

https://lists.rtems.org/pipermail/devel/2021-February/064159.html

It is unclear what the agreed path so I suggest we sort that out.

>      rtmonusxtimes01: indeterminate
>      spedfsched02: indeterminate
>      spedfsched04: indeterminate
> @@ -24,12 +29,34 @@ actions:
>      sptimecounter04: indeterminate
>      ttest02: indeterminate
>  
> -    # tests that pass nominally, but fail under Qemu when the host is under
> -    # heavy load
> +    # These tests may or may not fail, however, they do pass on real hardware.
> +    # It seems to be an issue with Qemu, and that this only occurs when the
> +    # host machine is under a heavy load.

What happens in the test in qemu? Does the test fail or time out?

Do the tests pass in other simulators for other archs, ie sis? Is there
something in our test code that causes some in qemu to not simulate well?

>      psx12: indeterminate
> +    spintrcritical01: indeterminate
> +    spintrcritical02: indeterminate
>      spintrcritical03: indeterminate
>      spintrcritical04: indeterminate
>      spintrcritical05: indeterminate
> +    spintrcritical06: indeterminate
> +    spintrcritical07: indeterminate
> +    spintrcritical08: indeterminate
> +    spintrcritical09: indeterminate
> +    spintrcritical10: indeterminate
> +    spintrcritical11: indeterminate
> +    spintrcritical12: indeterminate
> +    spintrcritical13: indeterminate
> +    spintrcritical14: indeterminate
> +    spintrcritical15: indeterminate
> +    spintrcritical16: indeterminate
> +    spintrcritical17: indeterminate
> +    spintrcritical18: indeterminate
> +    spintrcritical19: indeterminate
> +    spintrcritical20: indeterminate
> +    spintrcritical21: indeterminate
> +    spintrcritical22: indeterminate
> +    spintrcritical23: indeterminate
> +    spintrcritical24: indeterminate

This is a long list. We need a way to separate qemu from hardware of we need to
decide on preferencing tier 1 over lower tiers.

>  build-type: option
>  copyrights:
>  - Copyright (C) 2020 On-Line Applications Research (OAR)
> diff --git a/spec/build/bsps/aarch64/xilinx-zynqmp/tstqemu.yml b/spec/build/bsps/aarch64/xilinx-zynqmp/tstqemu.yml
> index 80d338f..1862815 100644
> --- a/spec/build/bsps/aarch64/xilinx-zynqmp/tstqemu.yml
> +++ b/spec/build/bsps/aarch64/xilinx-zynqmp/tstqemu.yml
> @@ -1,20 +1,25 @@
>  SPDX-License-Identifier: CC-BY-SA-4.0 OR BSD-2-Clause
>  actions:
>  - set-test-state:
> -    # expected to fail
> +    # This test fails when ran through rtems-tester because it does not
> +    # produce any output.

Copy paste.

Chris

>      minimum: exclude
>  
> -    # don't compile due to toolchain issues
> +    # These tests do not compile due to an issue with the GNU Assembler.
> +    # The issue has been filed(https://devel.rtems.org/ticket/4218).
> +    # Once the issue has been fixed, these tests will be turned back on.
>      spconfig01: exclude
>      spmisc01: exclude
>  
> -    # tests that are passing intermittently
> +    # These tests may or may not fail, however, they do pass on real hardware.
> +    # It seems to be an issue with Qemu.
>      spcpucounter01: indeterminate
>      rtmonuse: indeterminate
> -    sp68: indeterminate
>      sp04: indeterminate
>      sp20: indeterminate
> +    sp68: indeterminate
>      sp69: indeterminate
> +    sp71: indeterminate
>      rtmonusxtimes01: indeterminate
>      spedfsched02: indeterminate
>      spedfsched04: indeterminate
> @@ -24,12 +29,34 @@ actions:
>      sptimecounter04: indeterminate
>      ttest02: indeterminate
>  
> -    # tests that pass nominally, but fail under Qemu when the host is under
> -    # heavy load
> +    # These tests may or may not fail, however, they do pass on real hardware.
> +    # It seems to be an issue with Qemu, and that this only occurs when the
> +    # host machine is under a heavy load.
>      psx12: indeterminate
> +    spintrcritical01: indeterminate
> +    spintrcritical02: indeterminate
>      spintrcritical03: indeterminate
>      spintrcritical04: indeterminate
>      spintrcritical05: indeterminate
> +    spintrcritical06: indeterminate
> +    spintrcritical07: indeterminate
> +    spintrcritical08: indeterminate
> +    spintrcritical09: indeterminate
> +    spintrcritical10: indeterminate
> +    spintrcritical11: indeterminate
> +    spintrcritical12: indeterminate
> +    spintrcritical13: indeterminate
> +    spintrcritical14: indeterminate
> +    spintrcritical15: indeterminate
> +    spintrcritical16: indeterminate
> +    spintrcritical17: indeterminate
> +    spintrcritical18: indeterminate
> +    spintrcritical19: indeterminate
> +    spintrcritical20: indeterminate
> +    spintrcritical21: indeterminate
> +    spintrcritical22: indeterminate
> +    spintrcritical23: indeterminate
> +    spintrcritical24: indeterminate
>  build-type: option
>  copyrights:
>  - Copyright (C) 2020 On-Line Applications Research (OAR)
>