Proposal for hardware configuration dependent performance limits

[Sebastian Huber]

On 22/11/2020 22:45, Chris Johns wrote:

>>>>>> My point is that we need a key reported by the BSP and then some performance
>>>>>> limits which can be found by arch/bsp/key to check if there are performance
>>>>>> regressions.
>>>>> I am missing the place where the performance limits are held. Do the tests
>>>>> report timing values and the checks against the limits happen on a host?
>>>> Yes, this is what I proposed.
>>> Thanks and sorry for not picking up on this before now. It makes sense to do it
>>> this way.
>>>
>> I chimed in on the idea of not using a hash, because of the opaqueness
>> of the specification and difficulty to derive what should be
>> reasonable performance based on small configuration changes from a
>> standard set. In that case, we do punt some responsibility to the end
>> user to start from a configuration with a known hash and performance
>> bounds before defining their own. Otherwise, the best they can do is
>> like what we do: run it, record the measurements, and use those as the
>> bounds moving forward.
>>
>> When a user sends us a report saying my configuration
>> a/lwjeVQ:H#TIHFOAH doesn't match the performance of
>> z./hleg.khEHWIWEHFWHFE then we can have this conversation again. :)
> If the user is basing their figures on a set of results we publish would
> providing a description in the YAML be sufficient? This moves the burden of
> maintenance from being internal to RTEMS to outside. And I am fine if there are
> mandatory informational fields.

In the proposal the performance limits are optional in the 
specification. The specifications only enables you to check limits in 
test runs. We should probably not integrate limits for custom hardware 
in the RTEMS Project. We could add performance limits for some 
evaluation boards or simulators which we use for regular test runs to 
notice performance regressions. The configuration hashes included in the 
RTEMS Project need to be documented (in the BSP build item) to have a 
reverse mapping, e.g. XrY7u+Ae7tCTyyK7j1rNww== is board X revision Y 
running at 123MHz with SMP enabled, etc. Users which want to monitor 
performance requirements have to use also this approach:

1. You select a board to use for long term performance tests.

2. You define a set of configurations you want to test.

3. You do an initial run of the test suite for each configuration. The 
RTEMS Tester provides you with a machine readable output (test data) of 
the test run with the raw test output per test executable and some meta 
information (TODO).

4. A tool reads the  test data and the RTEMS specification and updates 
the specification with the performance limits obtained from the test run 
(maybe with some simple transformation, for example increase maximum by 
10% and round to microseconds).

5. You review the performance limits and then commit them to your 
private branch.

6. Later you run the tests with a new RTEMS commit, get the performance 
values, compare them against the limits stored in the specification, and 
generate a report.

Maybe a configuration option for the RTEMS Tester should be added which 
allows you to set the performance hash and ignore the hash provided by 
the test output. This could be used to compare a custom board with 
values obtain from an evaluation board.

-- 
embedded brains GmbH
Sebastian HUBER
Dornierstr. 4
82178 Puchheim
Germany
email: sebastian.huber at embedded-brains.de
Phone: +49-89-18 94 741 - 16
Fax:   +49-89-18 94 741 - 08
PGP: Public key available on request.

embedded brains GmbH
Registergericht: Amtsgericht München
Registernummer: HRB 157899
Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler
Unsere Datenschutzerklärung finden Sie hier: https://embedded-brains.de/datenschutzerklaerung/