[PATCH] tester: Limit simultaneous QEMU jobs to 1

[Chris Johns]

On 31/8/21 3:20 pm, Sebastian Huber wrote:
> On 30/08/2021 20:32, Kinsey Moore wrote:
>> On 8/30/2021 12:12, Sebastian Huber wrote:
>>> On 24/08/2021 20:45, Kinsey Moore wrote:
>>>> diff --git a/tester/rtems/testing/bsps/a53_ilp32_qemu.ini
>>>> b/tester/rtems/testing/bsps/a53_ilp32_qemu.ini
>>>> index 3beba06..581c59c 100644
>>>> --- a/tester/rtems/testing/bsps/a53_ilp32_qemu.ini
>>>> +++ b/tester/rtems/testing/bsps/a53_ilp32_qemu.ini
>>>> @@ -36,3 +36,4 @@ bsp           = a53_ilp32_qemu
>>>>   arch          = aarch64
>>>>   tester        = %{_rtscripts}/qemu.cfg
>>>>   bsp_qemu_opts = %{qemu_opts_base} -serial mon:stdio -machine
>>>> virt,gic-version=3 -cpu cortex-a53 -m 4096
>>>> +jobs          = 1
>>>
>>> Does this overwrite the command line option or is this a default value?
>>>
>> When this is set in the tester configuration, the command line switch has no
>> effect but it can be overridden in the user-config.
> 
> Overruling the command line option is not that great. I have a vastly different
> test run duration with --jobs=1 vs. --jobs=48 with more or less the same test
> results. 

What does more or less mean?

I appreciate the efforts Kinsey has gone to looking into why we have this
happening and I also believe we need to keep pushing towards repeatable result.
If limiting to 1 gives us repeatable results on qemu then I prefer this over
tainted test results with intermittent tags.

> I think this option should be split into a "force-jobs" and
> "default-jobs" option.

I am sorry I do not understand these options?

The command line is ignored because and the value is fixed on purpose and I am
not seeing a reason to change this.

When specified in a config it is a physical limit. A user being able to change
the number of TFTP jobs on the command line does not make sense.

This tool's focus is testing on hardware and I see that as more important. And
as I have said before if we have problematic tests maybe the test or the tool
generating the results needs to be investigated.

I see this issue as something specific to the design of qemu and a few of our
tests. I can guess at some of the reasons qemu does this but also being able to
have the tick timer's clock be sync'ed with the CPU clock is important in some
types of simulation, ie our case and these problematic test. We are a real-time
operating system so needing this to be right to closer in simulation does not
seem unreasonable.

This discussion send a clear message, tier 1 archs and BSPs are very important
to this project.

Chris