[PATCH] tester: Make the SIS time limit user configurable

[Chris Johns]

On 6/7/2022 6:00 pm, Sebastian Huber wrote:
> Yes, if tests go wrong the tester can kill a test execution after the specified
> timeout.

Killing should be taken as a sign something in the test equipment is broken.

> Why do we need this arbitrary SIS -tlim of 400 s?

There was a few values and I select this one based on those. If you have a
better value please say.

I do not recommend removing the time limit option from testing. If you want to
operate that way create a user config with:

[erc32-sis]
sis_time_limit =

> Which problem does this solve?

Repeatable test results across wide ranging hosts and host operating systems.

> Why can't we let the tests run in SIS without a limit just like we do it for
> Qemu? 

Qemu is painful to make work in a consistent and reliable way.

> Normally, if a test is done, it terminates the SIS execution.

The time outs are for the cases that end abnormally.

> The new performance tests can be used to catch performance regressions. 

We need to consider the existing benchmarks.

> In the
> long run I think we need a more modular approach. For example, one component
> which runs tests and reports the test output. Another component which analyses
> the test output. The test outputs can be archived.

The rtems-test command is required to report regressions and as simply as
possible. A user wants to build, test then know what they have matches what we
released.

Chris