[PATCH] tester: Make the SIS time limit user configurable

[Chris Johns]

On 6/7/2022 4:26 pm, Sebastian Huber wrote:
> On 06/07/2022 01:51, chrisj at rtems.org wrote:
>> +#
>> +# Timeout option. This is the default for timeout for the CPU realtime
>> +# clock
>> +
>> +%ifn %{defined sis_time_limit}
>> + %define sis_time_limit -tlim 400 s
>> +%endif
> 
> Making this configurable is good, but why do you impose a limit by default? Why
> can't the simulator run forever in the standard configuration?

I think I understand your issue so I will explain my understanding and then we
can see if we align. :)

The SIS has the ability to set a time limit which I understand is a simulated
CPU realtime clock period. A simulated realtime clock should not be effected by
the host hardware specs, the host's current load or the number of parallel
simulations it has running. The timeout fails a test that does not complete in
the set period of time and that time is scaled against the host's realtime clock
as the host schedules time to each simulator.

The tester has a timeout that time's out against the host's realtime clock and
that clock is not adjusted based on the execution time given to a simulator.
With hardware targets the host's realtime clock and the hardware target's
realtime clock are aligned because Einstein says they are.

We need timeouts to catch tests that fail and to catch test equipment that
fails. In the case of simulators it could be a bug that locks the simulation up.
The tester would like to be able to distinguish between a test failure and
equipment failure. In the case of hardware targets using TFTP there is
considerable effort put into determining if a target has failed to start a test
so it can retry starting the test verses a test starting and failing to end.
TFTP and networks do fail and in the case of the BeagleBoard it has a network
device with no reset that can lock up after a software reset to restarts include
power cycling.

This change has a default because the original logic had a default and I did not
want to changed what we had along with the config change. I am not sure the
default I selected is right but it is now easier to change.

This change does not cater for tests that have varying and valid long test times
and I suspect this is the issue you are facing with the validation tests. This
will also be the case for tests that are not a single test per test executable.
I see this as a valid but separate issue to the SIS time limit parameter config
change.

I have thought for a while tests should output as test metsdata a time limit.
The time limit can be maximum test period specified relative to the target's
realtime clock. The time limit could arch or BSP specific to match the
performance of the hardware being tested.

A per test time limit means we do not stall the tester with really long timeouts
that are needed on loaded simulation hosts. Hardware targets and simulators that
have time limit options can be adjusted to a test's valid time limit. For qemu
we could scale the per test timeout depending on the number of jobs plus a scale
factor.

The time limit metadata couls also privde an error factor so the tester could
start to flag performance regression.

Chris