[PATCH] tester: Make the SIS time limit user configurable

Sebastian Huber sebastian.huber at embedded-brains.de
Wed Jul 6 08:00:34 UTC 2022

On 06/07/2022 09:38, Chris Johns wrote:
> On 6/7/2022 4:26 pm, Sebastian Huber wrote:
>> On 06/07/2022 01:51, chrisj at rtems.org wrote:
>>> +#
>>> +# Timeout option. This is the default for timeout for the CPU realtime
>>> +# clock
>>> +
>>> +%ifn %{defined sis_time_limit}
>>> + %define sis_time_limit -tlim 400 s
>>> +%endif
>> Making this configurable is good, but why do you impose a limit by default? Why
>> can't the simulator run forever in the standard configuration?
> I think I understand your issue so I will explain my understanding and then we
> can see if we align. :)
> The SIS has the ability to set a time limit which I understand is a simulated
> CPU realtime clock period. A simulated realtime clock should not be effected by
> the host hardware specs, the host's current load or the number of parallel
> simulations it has running. The timeout fails a test that does not complete in
> the set period of time and that time is scaled against the host's realtime clock
> as the host schedules time to each simulator.

The SIS -tlim option specifies the maximum simulation time.

> The tester has a timeout that time's out against the host's realtime clock and
> that clock is not adjusted based on the execution time given to a simulator.
> With hardware targets the host's realtime clock and the hardware target's
> realtime clock are aligned because Einstein says they are.
> We need timeouts to catch tests that fail and to catch test equipment that
> fails. In the case of simulators it could be a bug that locks the simulation up.
> The tester would like to be able to distinguish between a test failure and
> equipment failure. In the case of hardware targets using TFTP there is
> considerable effort put into determining if a target has failed to start a test
> so it can retry starting the test verses a test starting and failing to end.
> TFTP and networks do fail and in the case of the BeagleBoard it has a network
> device with no reset that can lock up after a software reset to restarts include
> power cycling.
> This change has a default because the original logic had a default and I did not
> want to changed what we had along with the config change. I am not sure the
> default I selected is right but it is now easier to change.

Yes, if tests go wrong the tester can kill a test execution after the 
specified timeout.

Why do we need this arbitrary SIS -tlim of 400 s? Which problem does 
this solve? Why can't we let the tests run in SIS without a limit just 
like we do it for Qemu? Normally, if a test is done, it terminates the 
SIS execution.

> This change does not cater for tests that have varying and valid long test times
> and I suspect this is the issue you are facing with the validation tests. This
> will also be the case for tests that are not a single test per test executable.
> I see this as a valid but separate issue to the SIS time limit parameter config
> change.
> I have thought for a while tests should output as test metsdata a time limit.
> The time limit can be maximum test period specified relative to the target's
> realtime clock. The time limit could arch or BSP specific to match the
> performance of the hardware being tested.
> A per test time limit means we do not stall the tester with really long timeouts
> that are needed on loaded simulation hosts. Hardware targets and simulators that
> have time limit options can be adjusted to a test's valid time limit. For qemu
> we could scale the per test timeout depending on the number of jobs plus a scale
> factor.
> The time limit metadata couls also privde an error factor so the tester could
> start to flag performance regression.

The new performance tests can be used to catch performance regressions. 
In the long run I think we need a more modular approach. For example, 
one component which runs tests and reports the test output. Another 
component which analyses the test output. The test outputs can be archived.

embedded brains GmbH
Herr Sebastian HUBER
Dornierstr. 4
82178 Puchheim
email: sebastian.huber at embedded-brains.de
phone: +49-89-18 94 741 - 16
fax:   +49-89-18 94 741 - 08

Registergericht: Amtsgericht München
Registernummer: HRB 157899
Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler
Unsere Datenschutzerklärung finden Sie hier:

More information about the devel mailing list