<meta http-equiv="Content-Type" content="text/html; charset=utf-8"><div dir="ltr"><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jul 7, 2023 at 12:04 PM Gedare Bloom <<a href="mailto:gedare@rtems.org">gedare@rtems.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Thu, Jul 6, 2023 at 12:48 PM Kinsey Moore <<a href="mailto:kinsey.moore@oarcorp.com" target="_blank">kinsey.moore@oarcorp.com</a>> wrote:<br>
><br>
> QEMU is known to fail certain tests intermittently due to clock tick<br>
> delivery issues. This defines those tests as intermittent for BSPs<br>
> intended to run on QEMU alone.<br>
><br>
> Updates #4922<br>
> Updates #4072<br>
> ---<br>
>  spec/build/bsps/aarch64/a53/tsta53.yml        | 33 +--------<br>
>  spec/build/bsps/aarch64/a72/tsta72.yml        | 33 +--------<br>
>  .../bsps/aarch64/xilinx-versal/tstqemu.yml    |  4 +-<br>
>  .../bsps/aarch64/xilinx-zynqmp/tstqemu.yml    | 33 +--------<br>
>  .../arm/realview-pbx-a9/bsprealviewpbxa9.yml  |  2 +<br>
>  spec/build/bsps/arm/xilinx-zynq/bspqemu.yml   |  2 +<br>
>  spec/build/bsps/tstqemu.yml                   | 72 +++++++++++++++++++<br>
>  7 files changed, 88 insertions(+), 91 deletions(-)<br>
>  create mode 100644 spec/build/bsps/tstqemu.yml<br>
><br>
> diff --git a/spec/build/bsps/aarch64/a53/tsta53.yml b/spec/build/bsps/aarch64/a53/tsta53.yml<br>
> index a3d8c9810d..adf6d85ae1 100644<br>
> --- a/spec/build/bsps/aarch64/a53/tsta53.yml<br>
> +++ b/spec/build/bsps/aarch64/a53/tsta53.yml<br>
> @@ -6,40 +6,13 @@ actions:<br>
>      state: exclude<br>
>      tests:<br>
>      - minimum<br>
> -- set-test-state:<br>
> -    reason: |<br>
> -      Tests that are passing intermittently.<br>
> -    state: indeterminate<br>
> -    tests:<br>
> -    - psx12<br>
> -    - psxtimes01<br>
> -    - rtmonuse<br>
> -    - rtmonusxtimes01<br>
> -    - sp04<br>
> -    - sp20<br>
> -    - sp68<br>
> -    - sp69<br>
> -    - spcpucounter01<br>
> -    - spedfsched02<br>
> -    - spedfsched04<br>
> -    - sprmsched01<br>
> -    - sptimecounter02<br>
> -    - sptimecounter04<br>
> -    - ttest02<br>
> -- set-test-state:<br>
> -    reason: |<br>
> -      Tests that pass nominally, but fail under Qemu when the host is under<br>
> -      heavy load.<br>
> -    state: indeterminate<br>
> -    tests:<br>
> -    - spintrcritical03<br>
> -    - spintrcritical04<br>
> -    - spintrcritical05<br>
>  build-type: option<br>
>  copyrights:<br>
>  - Copyright (C) 2020 On-Line Applications Research (OAR)<br>
>  default: []<br>
>  description: ''<br>
>  enabled-by: true<br>
> -links: []<br>
> +links:<br>
> +- role: build-dependency<br>
> +  uid: ../../tstqemu<br>
>  type: build<br>
> diff --git a/spec/build/bsps/aarch64/a72/tsta72.yml b/spec/build/bsps/aarch64/a72/tsta72.yml<br>
> index a3d8c9810d..adf6d85ae1 100644<br>
> --- a/spec/build/bsps/aarch64/a72/tsta72.yml<br>
> +++ b/spec/build/bsps/aarch64/a72/tsta72.yml<br>
> @@ -6,40 +6,13 @@ actions:<br>
>      state: exclude<br>
>      tests:<br>
>      - minimum<br>
> -- set-test-state:<br>
> -    reason: |<br>
> -      Tests that are passing intermittently.<br>
> -    state: indeterminate<br>
> -    tests:<br>
> -    - psx12<br>
> -    - psxtimes01<br>
> -    - rtmonuse<br>
> -    - rtmonusxtimes01<br>
> -    - sp04<br>
> -    - sp20<br>
> -    - sp68<br>
> -    - sp69<br>
> -    - spcpucounter01<br>
> -    - spedfsched02<br>
> -    - spedfsched04<br>
> -    - sprmsched01<br>
> -    - sptimecounter02<br>
> -    - sptimecounter04<br>
> -    - ttest02<br>
> -- set-test-state:<br>
> -    reason: |<br>
> -      Tests that pass nominally, but fail under Qemu when the host is under<br>
> -      heavy load.<br>
> -    state: indeterminate<br>
> -    tests:<br>
> -    - spintrcritical03<br>
> -    - spintrcritical04<br>
> -    - spintrcritical05<br>
>  build-type: option<br>
>  copyrights:<br>
>  - Copyright (C) 2020 On-Line Applications Research (OAR)<br>
>  default: []<br>
>  description: ''<br>
>  enabled-by: true<br>
> -links: []<br>
> +links:<br>
> +- role: build-dependency<br>
> +  uid: ../../tstqemu<br>
>  type: build<br>
> diff --git a/spec/build/bsps/aarch64/xilinx-versal/tstqemu.yml b/spec/build/bsps/aarch64/xilinx-versal/tstqemu.yml<br>
> index 0a0ab619af..69b85cadbd 100644<br>
> --- a/spec/build/bsps/aarch64/xilinx-versal/tstqemu.yml<br>
> +++ b/spec/build/bsps/aarch64/xilinx-versal/tstqemu.yml<br>
> @@ -12,5 +12,7 @@ copyrights:<br>
>  default: []<br>
>  description: ''<br>
>  enabled-by: true<br>
> -links: []<br>
> +links:<br>
> +- role: build-dependency<br>
> +  uid: ../../tstqemu<br>
>  type: build<br>
> diff --git a/spec/build/bsps/aarch64/xilinx-zynqmp/tstqemu.yml b/spec/build/bsps/aarch64/xilinx-zynqmp/tstqemu.yml<br>
> index a3d8c9810d..adf6d85ae1 100644<br>
> --- a/spec/build/bsps/aarch64/xilinx-zynqmp/tstqemu.yml<br>
> +++ b/spec/build/bsps/aarch64/xilinx-zynqmp/tstqemu.yml<br>
> @@ -6,40 +6,13 @@ actions:<br>
>      state: exclude<br>
>      tests:<br>
>      - minimum<br>
> -- set-test-state:<br>
> -    reason: |<br>
> -      Tests that are passing intermittently.<br>
> -    state: indeterminate<br>
> -    tests:<br>
> -    - psx12<br>
> -    - psxtimes01<br>
> -    - rtmonuse<br>
> -    - rtmonusxtimes01<br>
> -    - sp04<br>
> -    - sp20<br>
> -    - sp68<br>
> -    - sp69<br>
> -    - spcpucounter01<br>
> -    - spedfsched02<br>
> -    - spedfsched04<br>
> -    - sprmsched01<br>
> -    - sptimecounter02<br>
> -    - sptimecounter04<br>
> -    - ttest02<br>
> -- set-test-state:<br>
> -    reason: |<br>
> -      Tests that pass nominally, but fail under Qemu when the host is under<br>
> -      heavy load.<br>
> -    state: indeterminate<br>
> -    tests:<br>
> -    - spintrcritical03<br>
> -    - spintrcritical04<br>
> -    - spintrcritical05<br>
>  build-type: option<br>
>  copyrights:<br>
>  - Copyright (C) 2020 On-Line Applications Research (OAR)<br>
>  default: []<br>
>  description: ''<br>
>  enabled-by: true<br>
> -links: []<br>
> +links:<br>
> +- role: build-dependency<br>
> +  uid: ../../tstqemu<br>
>  type: build<br>
> diff --git a/spec/build/bsps/arm/realview-pbx-a9/bsprealviewpbxa9.yml b/spec/build/bsps/arm/realview-pbx-a9/bsprealviewpbxa9.yml<br>
> index a6e4d5b1a9..4111830ce9 100644<br>
> --- a/spec/build/bsps/arm/realview-pbx-a9/bsprealviewpbxa9.yml<br>
> +++ b/spec/build/bsps/arm/realview-pbx-a9/bsprealviewpbxa9.yml<br>
> @@ -53,6 +53,8 @@ links:<br>
>    uid: ../../objmem<br>
>  - role: build-dependency<br>
>    uid: ../../opto0<br>
> +- role: build-dependency<br>
> +  uid: ../../tstqemu<br>
>  - role: build-dependency<br>
>    uid: ../../bspopts<br>
>  source:<br>
> diff --git a/spec/build/bsps/arm/xilinx-zynq/bspqemu.yml b/spec/build/bsps/arm/xilinx-zynq/bspqemu.yml<br>
> index 4780a9d64b..8068284d82 100644<br>
> --- a/spec/build/bsps/arm/xilinx-zynq/bspqemu.yml<br>
> +++ b/spec/build/bsps/arm/xilinx-zynq/bspqemu.yml<br>
> @@ -13,6 +13,8 @@ install: []<br>
>  links:<br>
>  - role: build-dependency<br>
>    uid: ../../opto2<br>
> +- role: build-dependency<br>
> +  uid: ../../tstqemu<br>
>  - role: build-dependency<br>
>    uid: grp<br>
>  source: []<br>
> diff --git a/spec/build/bsps/tstqemu.yml b/spec/build/bsps/tstqemu.yml<br>
> new file mode 100644<br>
> index 0000000000..3956cc4828<br>
> --- /dev/null<br>
> +++ b/spec/build/bsps/tstqemu.yml<br>
> @@ -0,0 +1,72 @@<br>
> +SPDX-License-Identifier: CC-BY-SA-4.0 OR BSD-2-Clause<br>
> +actions:<br>
> +- set-test-state:<br>
> +    reason: |<br>
> +      Tests that are passing intermittently due to #4922<br>
> +    state: indeterminate<br>
> +    tests:<br>
> +    - psx12<br>
> +    - psxclock<br>
> +    - psxtimes01<br>
> +    - psxualarm<br>
> +    - rtmonuse<br>
> +    - rtmonusxtimes01<br>
> +    - smpclock01<br>
> +    - smpfatal01<br>
> +    - smpfatal03<br>
> +    - smpmrsp01<br>
> +    - smpmutex01<br>
> +    - smppsxmutex01<br>
> +    - smpschedaffinity01<br>
> +    - smpschedaffinity02<br>
> +    - smpschededf01<br>
> +    - smpschededf03<br>
> +    - smpscheduler04<br>
> +    - smpthreadpin01<br>
> +    - sp04<br>
> +    - sp20<br>
> +    - sp68<br>
> +    - sp69<br>
> +    - sp71<br>
> +    - spcpucounter01<br>
> +    - spedfsched02<br>
> +    - spedfsched04<br>
> +    - spintrcritical01<br>
> +    - spintrcritical02<br>
> +    - spintrcritical03<br>
> +    - spintrcritical04<br>
> +    - spintrcritical05<br>
> +    - spintrcritical06<br>
> +    - spintrcritical07<br>
> +    - spintrcritical08<br>
> +    - spintrcritical09<br>
> +    - spintrcritical10<br>
> +    - spintrcritical11<br>
> +    - spintrcritical12<br>
> +    - spintrcritical13<br>
> +    - spintrcritical14<br>
> +    - spintrcritical15<br>
> +    - spintrcritical16<br>
> +    - spintrcritical17<br>
> +    - spintrcritical18<br>
> +    - spintrcritical19<br>
> +    - spintrcritical20<br>
> +    - spintrcritical21<br>
> +    - spintrcritical22<br>
> +    - spintrcritical23<br>
> +    - spintrcritical24<br>
> +    - sprmsched01<br>
> +    - sptimecounter01<br>
> +    - sptimecounter02<br>
> +    - sptimecounter04<br>
> +    - ttest02<br>
> +    - validation-0<br>
> +    - validation-smp-only-0<br>
<br>
This looks OK to me, but it looks like you've added a lot more tests<br>
than just the union of the existing ones in the individual BSPs. I<br>
take it that you have found more variance in these tests also?<br></blockquote><div><br></div><div>The list of tests presented here was provided in a previous update to the list which was rejected due to not having appropriate documentation as to the reason behind the failures and associated ticket to reference. This list is slightly expanded from the previous patch, but all tests here have been observed to produce spurious failures under QEMU (escalating with system load) with no such reproduction on hardware even after >100 repeat tests.</div><div><br></div><div>Thanks,</div><div>Kinsey<br></div></div></div>