Proposal for hardware configuration dependent performance limits

[Sebastian Huber]

On 13/11/2020 20:01, Gedare Bloom wrote:

> On Fri, Nov 13, 2020 at 3:48 AM Sebastian Huber
> <sebastian.huber at embedded-brains.de>  wrote:
>> Hello,
>>
>> there is one aspect with respect to performance limits which is
>> currently not considered in this proposal:
>>
>> https://lists.rtems.org/pipermail/devel/2020-November/063213.html
>>
>> You can run the some BSPs such as sparc/gr712rc on several boards.
>> However, each board may have different settings which affect the system
>> performance, for example the CPU frequency, memory controller settings,
>> memory chips, etc. In the proposal, limits are specified like this:
>>
>>
>> limits:
>>     sparc/gr712rc:
>>       DirtyCache:
>>         max-upper-bound: 0.000005
>>         mean-upper-bound: 0.000005
>>       FullCache:
>>         max-upper-bound: 0.000005
>>         mean-upper-bound: 0.000005
>>       HotCache:
>>         max-upper-bound: 0.000005
>>         mean-upper-bound: 0.000005
>>       Load/1:
>>         max-upper-bound: 0.00001
>>         mean-upper-bound: 0.00001
>>       Load/2:
>>         max-upper-bound: 0.00001
>>         mean-upper-bound: 0.00001
>>       Load/3:
>>         max-upper-bound: 0.00001
>>         mean-upper-bound: 0.00001
>>       Load/4:
>>         max-upper-bound: 0.00001
>>         mean-upper-bound: 0.00001
>>
>> This neglects that the limits are subject to a board configuration. One
>> approach to cover this is the addition of a new BSP provided function:
>>
>> const char *rtems_get_hardware_performance_hash();
>>
>> The BSP feeds all performance related data into a hash function and
> "data" here means configuration?
Yes, hardware configuration.
>
>> returns a string encoding (for example a MD5 digest in base64 encoding).
>> The example from above with the performance hash:
>>
>> limits:
>>     sparc/gr712rc/XrY7u+Ae7tCTyyK7j1rNww==:
>>       DirtyCache:
>>         max-upper-bound: 0.000005
>>         mean-upper-bound: 0.000005
>>
>> The test suite could report the performance has and a test output analyser coudl check that the reported values are in the specified bounds.
>>
> This will work for fixed sets of configurations. I wonder if it is
> reasonable instead to define ranges over configuration values? Or to
> use board variant mnemonic names instead?  How many possible
> configurations are we talking about?
>
> The hash output is fairly opaque, and really small configuration
> changes would result in totally different hashes, so if I change a
> board from 2 MiB RAM to 3 MiB RAM, I might like to know which
> configuration this is closest to in case there is no matching hash.
>
> To invert the hash we will need to keep the table of all the
> configurations mapped to hash values, which is fine I suppose.
>
> I can see that the advantage of a hash is that we don't need to create
> a namespace for configuration options that may influence performance.
> This can give some flexibility for boards that have different sets of
> configuration options that matter.
Maybe we should name it rtems_get_performance_magic_value(). On some 
BSPs it might be possible to return something with less entropy, but I 
think this stuff can get very quickly very complicated if you want to 
use explicit hardware configuration parameters to approximate expected 
limits for a new hardware configuration. The hash is a crude 
simplification, but it works well for known hardware configurations. 
Just using arch/bsp is definitely not enough to record performance 
limits for later regression testing.

-- 
embedded brains GmbH
Sebastian HUBER
Dornierstr. 4
82178 Puchheim
Germany
email: sebastian.huber at embedded-brains.de
Phone: +49-89-18 94 741 - 16
Fax:   +49-89-18 94 741 - 08
PGP: Public key available on request.

embedded brains GmbH
Registergericht: Amtsgericht München
Registernummer: HRB 157899
Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler
Unsere Datenschutzerklärung finden Sie hier: https://embedded-brains.de/datenschutzerklaerung/