[tools] tester: Remove hard coded time limits for SIS

[Chris Johns]

On 5/7/2022 8:50 pm, Jiri Gaisler wrote:
> 
> On 7/5/22 12:45, Sebastian Huber wrote:
>>
>>
>> On 05/07/2022 12:31, Chris Johns wrote:
>>>
>>>> On 5 Jul 2022, at 6:23 pm, Sebastian
>>>> Huber<sebastian.huber at embedded-brains.de>  wrote:
>>>>
>>>> On 05/07/2022 10:21, Chris Johns wrote:
>>>>>> On 5/7/2022 4:29 pm, Sebastian Huber wrote:
>>>>>> On 05/07/2022 08:23, Chris Johns wrote:
>>>>>>> On 5/7/2022 4:02 pm, Sebastian Huber wrote:
>>>>>>>> On 05/07/2022 07:14, Chris Johns wrote:
>>>>>>>>> On 5/7/2022 2:58 pm, Sebastian Huber wrote:
>>>>>>>>>> On 05/07/2022 03:08, Chris Johns wrote:
>>>>>>>>>>> On 5/7/2022 9:44 am, Joel Sherrill wrote:
>>>>>>>>>>>> The limit removed in sis and tsim is the simulated cpu time used. If
>>>>>>>>>>>> not
>>>>>>>>>>>> using
>>>>>>>>>>>> that, the behavior of the tester is to let the simulator run for so
>>>>>>>>>>>> much
>>>>>>>>>>>> real
>>>>>>>>>>>> processor time.
>>>>>>>>>>>>
>>>>>>>>>>>> Replacing these with a command line argument is probably good but just
>>>>>>>>>>>> removing
>>>>>>>>>>>> these mean these simulators will just run much longer before being
>>>>>>>>>>>> killed.
>>>>>>>>>>>>
>>>>>>>>>>>> How best to capture the distinction between target run time and host
>>>>>>>>>>>> run
>>>>>>>>>>>> time?
>>>>>>>>>>> Thank you for the explanation. I was not sure how the option effected
>>>>>>>>>>> things
>>>>>>>>>>> and
>>>>>>>>>>> yes it does matter we have this set correctly.
>>>>>>>>>>>
>>>>>>>>>>> Options can be set in the $HOME/.rtemstesterrc is via the --user-config
>>>>>>>>>>> option.
>>>>>>>>>>> Maybe this can be used to control the time out for specific user tests?

This is the only way we currently have and it works ok for hardware type testing
with site specific overrides. I think we need an sis.cfg to test for a user
defined time limit or use a default.

>>>>>>>>>> I would not make this more complicated than necessary. We have a
>>>>>>>>>> --timeout
>>>>>>>>>> command line option and the default timeout value can be set by *.ini
>>>>>>>>>> files. The
>>>>>>>>>> simulator speed is just a detail similar to running a target at 100MHz or
>>>>>>>>>> 1GHz.
>>>>>>>>> It is actually simpler to have this option and to measure time against
>>>>>>>>> the cpu
>>>>>>>>> time. The work loads on SMP hosts with qemu shows simulation timeouts are
>>>>>>>>> difficult to get right.
>>>>>>>> I don't know what is wrong with the patch. Overruling command line
>>>>>>>> options is
>>>>>>>> just bad.
>>>>>>> It does not work that way. When simulating the timeout in the tester is a
>>>>>>> catch
>>>>>>> all. It may triggered if the simulator locks up. With real hardware it is
>>>>>>> the
>>>>>>> timeout but that is a different use case. A simulator timeout is
>>>>>>> preferred when
>>>>>>> available.
>>>>>> Ok, good. Who will fix this?
>>>>> I am sorry I am not following. The tests have valid times for the default
>>>>> optimisation. What is broken?
>>>> What is broken is that the --timeout command line option doesn't work with
>>>> SIS because it uses hard coded values.
>>> The timeout option is correct and your understanding of it’s purpose is
>>> wrong. Joining them as you would like would break it.
>>
>> It would be nice if someone could offer me a way to run tests which exceed the
>> hard coded SIS timeout values?

I will post a patch. I have only tested the erc32-sis bsp.

I think it is important we do not join the host's realtime clock to the
simulated CPU's realtime clock if we can. The problem with the host's realtime
clock controlling the simulated test's time limit is the performance of hosts
users differ and what works for one person may not work for someone else. Using
extra long timeouts on the host tends to skew the testing time by having the
tester wait for the stuck tests to time out and be killed.

Note, testing on hardware does not have this problem because the host's and
hardware's realtime clocks are the same.

> sis accepts several -tlim options. The last one will be the active one. So
> rtems-test could add an extra -tlim option at the end of the sis parameters
> which would override the default one.

I think adding a `sis.cfg` and then letting the user pass the option via the
$HOME/.rtemstesterrc or the --user-config option is the best approach. It
provides an easy to manage site specific configuration.

Chris