[PATCH] spec: Add QEMU test annotations
Kinsey Moore
kinsey.moore at oarcorp.com
Fri Jul 7 17:08:15 UTC 2023
On Fri, Jul 7, 2023 at 12:04 PM Gedare Bloom <gedare at rtems.org> wrote:
> On Thu, Jul 6, 2023 at 12:48 PM Kinsey Moore <kinsey.moore at oarcorp.com>
> wrote:
> >
> > QEMU is known to fail certain tests intermittently due to clock tick
> > delivery issues. This defines those tests as intermittent for BSPs
> > intended to run on QEMU alone.
> >
> > Updates #4922
> > Updates #4072
> > ---
> > spec/build/bsps/aarch64/a53/tsta53.yml | 33 +--------
> > spec/build/bsps/aarch64/a72/tsta72.yml | 33 +--------
> > .../bsps/aarch64/xilinx-versal/tstqemu.yml | 4 +-
> > .../bsps/aarch64/xilinx-zynqmp/tstqemu.yml | 33 +--------
> > .../arm/realview-pbx-a9/bsprealviewpbxa9.yml | 2 +
> > spec/build/bsps/arm/xilinx-zynq/bspqemu.yml | 2 +
> > spec/build/bsps/tstqemu.yml | 72 +++++++++++++++++++
> > 7 files changed, 88 insertions(+), 91 deletions(-)
> > create mode 100644 spec/build/bsps/tstqemu.yml
> >
> > diff --git a/spec/build/bsps/aarch64/a53/tsta53.yml
> b/spec/build/bsps/aarch64/a53/tsta53.yml
> > index a3d8c9810d..adf6d85ae1 100644
> > --- a/spec/build/bsps/aarch64/a53/tsta53.yml
> > +++ b/spec/build/bsps/aarch64/a53/tsta53.yml
> > @@ -6,40 +6,13 @@ actions:
> > state: exclude
> > tests:
> > - minimum
> > -- set-test-state:
> > - reason: |
> > - Tests that are passing intermittently.
> > - state: indeterminate
> > - tests:
> > - - psx12
> > - - psxtimes01
> > - - rtmonuse
> > - - rtmonusxtimes01
> > - - sp04
> > - - sp20
> > - - sp68
> > - - sp69
> > - - spcpucounter01
> > - - spedfsched02
> > - - spedfsched04
> > - - sprmsched01
> > - - sptimecounter02
> > - - sptimecounter04
> > - - ttest02
> > -- set-test-state:
> > - reason: |
> > - Tests that pass nominally, but fail under Qemu when the host is
> under
> > - heavy load.
> > - state: indeterminate
> > - tests:
> > - - spintrcritical03
> > - - spintrcritical04
> > - - spintrcritical05
> > build-type: option
> > copyrights:
> > - Copyright (C) 2020 On-Line Applications Research (OAR)
> > default: []
> > description: ''
> > enabled-by: true
> > -links: []
> > +links:
> > +- role: build-dependency
> > + uid: ../../tstqemu
> > type: build
> > diff --git a/spec/build/bsps/aarch64/a72/tsta72.yml
> b/spec/build/bsps/aarch64/a72/tsta72.yml
> > index a3d8c9810d..adf6d85ae1 100644
> > --- a/spec/build/bsps/aarch64/a72/tsta72.yml
> > +++ b/spec/build/bsps/aarch64/a72/tsta72.yml
> > @@ -6,40 +6,13 @@ actions:
> > state: exclude
> > tests:
> > - minimum
> > -- set-test-state:
> > - reason: |
> > - Tests that are passing intermittently.
> > - state: indeterminate
> > - tests:
> > - - psx12
> > - - psxtimes01
> > - - rtmonuse
> > - - rtmonusxtimes01
> > - - sp04
> > - - sp20
> > - - sp68
> > - - sp69
> > - - spcpucounter01
> > - - spedfsched02
> > - - spedfsched04
> > - - sprmsched01
> > - - sptimecounter02
> > - - sptimecounter04
> > - - ttest02
> > -- set-test-state:
> > - reason: |
> > - Tests that pass nominally, but fail under Qemu when the host is
> under
> > - heavy load.
> > - state: indeterminate
> > - tests:
> > - - spintrcritical03
> > - - spintrcritical04
> > - - spintrcritical05
> > build-type: option
> > copyrights:
> > - Copyright (C) 2020 On-Line Applications Research (OAR)
> > default: []
> > description: ''
> > enabled-by: true
> > -links: []
> > +links:
> > +- role: build-dependency
> > + uid: ../../tstqemu
> > type: build
> > diff --git a/spec/build/bsps/aarch64/xilinx-versal/tstqemu.yml
> b/spec/build/bsps/aarch64/xilinx-versal/tstqemu.yml
> > index 0a0ab619af..69b85cadbd 100644
> > --- a/spec/build/bsps/aarch64/xilinx-versal/tstqemu.yml
> > +++ b/spec/build/bsps/aarch64/xilinx-versal/tstqemu.yml
> > @@ -12,5 +12,7 @@ copyrights:
> > default: []
> > description: ''
> > enabled-by: true
> > -links: []
> > +links:
> > +- role: build-dependency
> > + uid: ../../tstqemu
> > type: build
> > diff --git a/spec/build/bsps/aarch64/xilinx-zynqmp/tstqemu.yml
> b/spec/build/bsps/aarch64/xilinx-zynqmp/tstqemu.yml
> > index a3d8c9810d..adf6d85ae1 100644
> > --- a/spec/build/bsps/aarch64/xilinx-zynqmp/tstqemu.yml
> > +++ b/spec/build/bsps/aarch64/xilinx-zynqmp/tstqemu.yml
> > @@ -6,40 +6,13 @@ actions:
> > state: exclude
> > tests:
> > - minimum
> > -- set-test-state:
> > - reason: |
> > - Tests that are passing intermittently.
> > - state: indeterminate
> > - tests:
> > - - psx12
> > - - psxtimes01
> > - - rtmonuse
> > - - rtmonusxtimes01
> > - - sp04
> > - - sp20
> > - - sp68
> > - - sp69
> > - - spcpucounter01
> > - - spedfsched02
> > - - spedfsched04
> > - - sprmsched01
> > - - sptimecounter02
> > - - sptimecounter04
> > - - ttest02
> > -- set-test-state:
> > - reason: |
> > - Tests that pass nominally, but fail under Qemu when the host is
> under
> > - heavy load.
> > - state: indeterminate
> > - tests:
> > - - spintrcritical03
> > - - spintrcritical04
> > - - spintrcritical05
> > build-type: option
> > copyrights:
> > - Copyright (C) 2020 On-Line Applications Research (OAR)
> > default: []
> > description: ''
> > enabled-by: true
> > -links: []
> > +links:
> > +- role: build-dependency
> > + uid: ../../tstqemu
> > type: build
> > diff --git a/spec/build/bsps/arm/realview-pbx-a9/bsprealviewpbxa9.yml
> b/spec/build/bsps/arm/realview-pbx-a9/bsprealviewpbxa9.yml
> > index a6e4d5b1a9..4111830ce9 100644
> > --- a/spec/build/bsps/arm/realview-pbx-a9/bsprealviewpbxa9.yml
> > +++ b/spec/build/bsps/arm/realview-pbx-a9/bsprealviewpbxa9.yml
> > @@ -53,6 +53,8 @@ links:
> > uid: ../../objmem
> > - role: build-dependency
> > uid: ../../opto0
> > +- role: build-dependency
> > + uid: ../../tstqemu
> > - role: build-dependency
> > uid: ../../bspopts
> > source:
> > diff --git a/spec/build/bsps/arm/xilinx-zynq/bspqemu.yml
> b/spec/build/bsps/arm/xilinx-zynq/bspqemu.yml
> > index 4780a9d64b..8068284d82 100644
> > --- a/spec/build/bsps/arm/xilinx-zynq/bspqemu.yml
> > +++ b/spec/build/bsps/arm/xilinx-zynq/bspqemu.yml
> > @@ -13,6 +13,8 @@ install: []
> > links:
> > - role: build-dependency
> > uid: ../../opto2
> > +- role: build-dependency
> > + uid: ../../tstqemu
> > - role: build-dependency
> > uid: grp
> > source: []
> > diff --git a/spec/build/bsps/tstqemu.yml b/spec/build/bsps/tstqemu.yml
> > new file mode 100644
> > index 0000000000..3956cc4828
> > --- /dev/null
> > +++ b/spec/build/bsps/tstqemu.yml
> > @@ -0,0 +1,72 @@
> > +SPDX-License-Identifier: CC-BY-SA-4.0 OR BSD-2-Clause
> > +actions:
> > +- set-test-state:
> > + reason: |
> > + Tests that are passing intermittently due to #4922
> > + state: indeterminate
> > + tests:
> > + - psx12
> > + - psxclock
> > + - psxtimes01
> > + - psxualarm
> > + - rtmonuse
> > + - rtmonusxtimes01
> > + - smpclock01
> > + - smpfatal01
> > + - smpfatal03
> > + - smpmrsp01
> > + - smpmutex01
> > + - smppsxmutex01
> > + - smpschedaffinity01
> > + - smpschedaffinity02
> > + - smpschededf01
> > + - smpschededf03
> > + - smpscheduler04
> > + - smpthreadpin01
> > + - sp04
> > + - sp20
> > + - sp68
> > + - sp69
> > + - sp71
> > + - spcpucounter01
> > + - spedfsched02
> > + - spedfsched04
> > + - spintrcritical01
> > + - spintrcritical02
> > + - spintrcritical03
> > + - spintrcritical04
> > + - spintrcritical05
> > + - spintrcritical06
> > + - spintrcritical07
> > + - spintrcritical08
> > + - spintrcritical09
> > + - spintrcritical10
> > + - spintrcritical11
> > + - spintrcritical12
> > + - spintrcritical13
> > + - spintrcritical14
> > + - spintrcritical15
> > + - spintrcritical16
> > + - spintrcritical17
> > + - spintrcritical18
> > + - spintrcritical19
> > + - spintrcritical20
> > + - spintrcritical21
> > + - spintrcritical22
> > + - spintrcritical23
> > + - spintrcritical24
> > + - sprmsched01
> > + - sptimecounter01
> > + - sptimecounter02
> > + - sptimecounter04
> > + - ttest02
> > + - validation-0
> > + - validation-smp-only-0
>
> This looks OK to me, but it looks like you've added a lot more tests
> than just the union of the existing ones in the individual BSPs. I
> take it that you have found more variance in these tests also?
>
The list of tests presented here was provided in a previous update to the
list which was rejected due to not having appropriate documentation as to
the reason behind the failures and associated ticket to reference. This
list is slightly expanded from the previous patch, but all tests here have
been observed to produce spurious failures under QEMU (escalating with
system load) with no such reproduction on hardware even after >100 repeat
tests.
Thanks,
Kinsey
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rtems.org/pipermail/devel/attachments/20230707/16293a50/attachment-0001.htm>
More information about the devel
mailing list