[PATCH] tester: Make the SIS time limit user configurable

[Sebastian Huber]

On 06/07/2022 09:38, Chris Johns wrote:
> On 6/7/2022 4:26 pm, Sebastian Huber wrote:
>> On 06/07/2022 01:51, chrisj at rtems.org wrote:
>>> +#
>>> +# Timeout option. This is the default for timeout for the CPU realtime
>>> +# clock
>>> +
>>> +%ifn %{defined sis_time_limit}
>>> + %define sis_time_limit -tlim 400 s
>>> +%endif
>>
>> Making this configurable is good, but why do you impose a limit by default? Why
>> can't the simulator run forever in the standard configuration?
> 
> I think I understand your issue so I will explain my understanding and then we
> can see if we align. :)
> 
> The SIS has the ability to set a time limit which I understand is a simulated
> CPU realtime clock period. A simulated realtime clock should not be effected by
> the host hardware specs, the host's current load or the number of parallel
> simulations it has running. The timeout fails a test that does not complete in
> the set period of time and that time is scaled against the host's realtime clock
> as the host schedules time to each simulator.

The SIS -tlim option specifies the maximum simulation time.

> 
> The tester has a timeout that time's out against the host's realtime clock and
> that clock is not adjusted based on the execution time given to a simulator.
> With hardware targets the host's realtime clock and the hardware target's
> realtime clock are aligned because Einstein says they are.
> 
> We need timeouts to catch tests that fail and to catch test equipment that
> fails. In the case of simulators it could be a bug that locks the simulation up.
> The tester would like to be able to distinguish between a test failure and
> equipment failure. In the case of hardware targets using TFTP there is
> considerable effort put into determining if a target has failed to start a test
> so it can retry starting the test verses a test starting and failing to end.
> TFTP and networks do fail and in the case of the BeagleBoard it has a network
> device with no reset that can lock up after a software reset to restarts include
> power cycling.
> 
> This change has a default because the original logic had a default and I did not
> want to changed what we had along with the config change. I am not sure the
> default I selected is right but it is now easier to change.

Yes, if tests go wrong the tester can kill a test execution after the 
specified timeout.

Why do we need this arbitrary SIS -tlim of 400 s? Which problem does 
this solve? Why can't we let the tests run in SIS without a limit just 
like we do it for Qemu? Normally, if a test is done, it terminates the 
SIS execution.

> 
> This change does not cater for tests that have varying and valid long test times
> and I suspect this is the issue you are facing with the validation tests. This
> will also be the case for tests that are not a single test per test executable.
> I see this as a valid but separate issue to the SIS time limit parameter config
> change.
> 
> I have thought for a while tests should output as test metsdata a time limit.
> The time limit can be maximum test period specified relative to the target's
> realtime clock. The time limit could arch or BSP specific to match the
> performance of the hardware being tested.
> 
> A per test time limit means we do not stall the tester with really long timeouts
> that are needed on loaded simulation hosts. Hardware targets and simulators that
> have time limit options can be adjusted to a test's valid time limit. For qemu
> we could scale the per test timeout depending on the number of jobs plus a scale
> factor.
> 
> The time limit metadata couls also privde an error factor so the tester could
> start to flag performance regression.

The new performance tests can be used to catch performance regressions. 
In the long run I think we need a more modular approach. For example, 
one component which runs tests and reports the test output. Another 
component which analyses the test output. The test outputs can be archived.

-- 
embedded brains GmbH
Herr Sebastian HUBER
Dornierstr. 4
82178 Puchheim
Germany
email: sebastian.huber at embedded-brains.de
phone: +49-89-18 94 741 - 16
fax:   +49-89-18 94 741 - 08

Registergericht: Amtsgericht München
Registernummer: HRB 157899
Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler
Unsere Datenschutzerklärung finden Sie hier:
https://embedded-brains.de/datenschutzerklaerung/