[PATCH 0/8] Add test suite to validate performance requirements

[Sebastian Huber]

On 13/11/2020 20:21, Gedare Bloom wrote:

> I didn't really raise this in your other threads related to
> performance, but how are we (RTEMS Project) defining performance
> requirements? Are these simply the performance values we get by
> running the tests on our release? Do we aim to hit certain targets and
> reject patches if they don't? Or do we have to debate when a patch
> does change a performance value? Are we going to need to run these
> performance tests as part of regular testing before committing?
>
> I'd like to get some clarity on the idea of performance requirements
> as a community burden before we bake them in to RTEMS.

Right now we have no systematic indication if there are performance 
regressions or improvements. Which influence have compiler updates, we 
don't know. I think for a supposed to be real-time operating system, 
this is not really good and we should step by step improve the 
situation. One way to do this is to add performance requirements that 
state that the runtime of a certain operation in a particular 
environment and system condition should less than or equal to a certain 
limit. This limit is obviously hardware dependent. For a start we can 
run the performance tests and record the data in the specification data. 
In a later test run we can check the values and notice if there are some 
changes in the wrong direction. The new test framework was specifically 
written to get this data easily. See for example (1.5 Test Suite - 
ClockTick):

https://ftp.rtems.org/pub/rtems/people/sebh/test-report.pdf

Not every target is suitable to get stable performance data (Qemu is 
really bad at this). I think the SIS (or other GDB simulators) would be 
a good candidate to do this on a regular basis.

What can we do with the data? It can give some aid to judge changes.

-- 
embedded brains GmbH
Sebastian HUBER
Dornierstr. 4
82178 Puchheim
Germany
email: sebastian.huber at embedded-brains.de
Phone: +49-89-18 94 741 - 16
Fax:   +49-89-18 94 741 - 08
PGP: Public key available on request.

embedded brains GmbH
Registergericht: Amtsgericht München
Registernummer: HRB 157899
Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler
Unsere Datenschutzerklärung finden Sie hier: https://embedded-brains.de/datenschutzerklaerung/