[PATCH] aarch64: Add tests that are failing intermittently

[Kinsey Moore]

On 8/19/2021 18:03, Chris Johns wrote:
> On 20/8/21 4:55 am, Kinsey Moore wrote:
>> On 8/19/2021 13:32, Gedare Bloom wrote:
>>> On Thu, Aug 19, 2021 at 11:43 AM Kinsey Moore <kinsey.moore at oarcorp.com> wrote:
>>>> I've seen these failures on my local system, in our CI, and on a build
>>>> server that I sometimes
>>>> use for development/testing so if it's a configuration issue we're being
>>>> pretty consistent about
>>>> misconfiguration across some pretty different environments (docker,
>>>> bare-metal, VM, different
>>>> OSs, different QEMU versions). I've seen enough of the spintrcritical
>>>> tests fail sporadically on
>>>> QEMU to lump them all into this category. These are also tests that I
>>>> have seen behave badly
>>>> on ARMv7 QEMU on my local system (which doesn't rule out
>>>> misconfiguration, but it's another
>>>> data point).
>>>>
>>> Yes, for example, it may be a matter of qemu process counts spawned by
>>> rtems-test, and the order in which tests get invoked could be a cause
>>> for which ones don't work. I could easily see this happening, since
>>> each test runtime will be fairly consistent, so you'll often see the
>>> same tests running concurrently with each other. But, if you change
>>> the order (e.g., by adding new tests), then we may see a new set of
>>> sporadically failing testcases, will we just add those, or do we need
>>> to re-examine this indetermine set periodically? Who will maintain
>>> this list? That's kind of the root of my concern here.
>> I understand your concern about maintenance of the failure list and I don't
>> have a good answer for you. I imagine going forward it would be a combination
>> of the current stake-holders for a given BSP and anyone who watches the
>> automated build output from Joel's runs for these kinds of issues.
>>
>> On the other hand if we don't mark those tests, people will get fatigued
>> looking at the spurious failures and assume any new ones just fall into the
>> same category as others. At that point is it even worth running the
>> automated tests for that platform?
>>
>>>> As far as your worry about marking these indeterminate, they're only
>>>> being marked as such for
>>>> QEMU BSPs. The ZynqMP hardware BSP doesn't have these testing carve-outs
>>>> and runs all these tests flawlessly.
> Great, this is important.
>
>>>> These failures become much more common when there is otherwise load on
>>>> the system and a
>>>> lot of them disappear when you limit the tester to a single QEMU
>>>> instance at a time.
>>>>
>>> I'm wondering if we should sacrifice testing speed for
>>> coverage/quality. If throttling rtems-test leads to more reliable test
>>> results, then it may be a better option than basically ignoring a
>>> swath of our testsuite.
>> That would certainly mitigate some of the failures, but you'd also have to
>> guarantee nothing else is running on the system which could cause the same
>> problem. I know at least some of the current automated runs operate on a
>> shared system which can and does often have other intensive processes
>> running on it. There are also the tests that are sporadic on QEMU even
>> without additional load.
> What is it in these tests when combined with qemu that causes the tests to fail?
> Is there some relation to a real clock, some shared host resource or a bug in
> qemu? I am concerned a simulator can vary like this based on the host's load and
> it makes me wonder how people use it on machines to host a number VMs.
I experienced very similar results on an ARMv7 BSP (not Zynq) and 
assumed that this
was a known/accepted problem with QEMU when the same issues popped up on
AArch64. My local system under no other load produces these failures for 
the Zynq A9 QEMU
BSP:

         "failed": [
             "spcpucounter01.exe",
             "psxtimes01.exe",
             "sp69.exe",
             "psx12.exe",
             "minimum.exe",
             "dl06.exe",
             "sptimecounter02.exe"
         ],

minimum.exe and dl06.exe are probably unrelated, but the remainder are in my
problem set for AArch64 on QEMU.

A run of the AArch64 ZynqMP ILP32 BSP produced these failures under the same
conditions with all the test carve-outs removed:

         "failed": [
             "psx12.exe",
             "spcpucounter01.exe",
             "sptimecounter01.exe",
             "sptimecounter02.exe",
             "sp04.exe"
         ],

Because of my experience with the aforementioned ARMv7 BSP and the lack of
failures on hardware, I chose not to weed out the root cause of the 
failures under
QEMU. This patch is documentation of our observations across multiple
architectures and BSPs running on QEMU more than anything else.
> I feel with this volume of tests being tagged this way we should have a better
> understanding of the problem and so a means to track or not track how to resolve
> it. As Gedare has kindly stated once pushed this change disappears into a dark
> corner and we have no means to track it.
>
> The other solution is to set `jobs` to `1` in this BSP's tester config, again
> something Gedare has raised. It means we get better or even valid results. What
> is more important, valid results or running the testsuite as fast as possible?
I fully support dropping the number of jobs to "half" or 1 for better 
results on QEMU
runs that display these problems. My comment in that regard was that 
other system
loading (or multiple simultaneous test runs) can also cause the same 
problem and so
this is only a partial solution. Barring a fix for RTEMS or QEMU for 
these load-
dependent and sporadic failures, this at least still needs to be 
documented in some
form.


Kinsey