[PATCH] membench: Add memory benchmark programs

[Sebastian Huber]

On 21.07.23 09:43, Chris Johns wrote:
> On 21/7/2023 3:28 pm, Sebastian Huber wrote:
>> On 21.07.23 03:27, Chris Johns wrote:
>>> On 21/7/2023 3:51 am, Sebastian Huber wrote:
>>>> On 20.07.23 18:58, Gedare Bloom wrote:
>>>>> On Thu, Jul 20, 2023 at 7:42 AM Sebastian Huber
>>>>> <sebastian.huber at embedded-brains.de>  wrote:
>>>>>> These memory benchmark programs are not supposed to run.  Instead, they
>>>>>> can be analysed on the host system to measure the memory usage of
>>>>>> features.  See the membench module of rtems-central.
>>>>>>
>>>>> This needs some kind of documentation and probably a README inside of
>>>>> membench with that information.
>>>>
>>>> Ok, I can add a README.md.
>>>>
>>>>>
>>>>> This appears to be about benchmarking the program size (static memory
>>>>> usage) only? If so, make that clear in the README / log note. I think
>>>>> it's in the doxygen already so that's helpful.
>>>>
>>>> Yes, it measures only the static memory size required for certain operating
>>>> system services. See 4.7 Memory Usage Benchmarks in:
>>>>
>>>> https://ftp.rtems.org/pub/rtems/people/sebh/rtems-6-sparc-gr740-uni-6-scf.pdf
>>>
>>> Should `static` be part of naming?
>>
>> Yes, good idea.
>>
>>>
>>>>> What happens when the membench gets built, and then someone runs
>>>>> $> rtems-test build/${ARCH}/${BSP}/testsuites
>>>>>
>>>>> Because I don't see anything that is filtering these executables.
>>>>
>>>> They are filtered out due to the *.norun.* pattern:>
>>>> target: testsuites/membench/mem-scheduler-add-cpu.norun.exe
>>>>
>>>
>>> Currently tests with `norun` assume the build fails if there is an issue with a
>>> test. This is why we allow these tests and they are tagged `norun`.
>>
>> We already have a couple of norun tests in libtests. This filtering is simple
>> and works fine why would you want to change it?
> 
> I am not asking for that to change. After a build we will have a set of norun
> tests and in that set are some that are be used for other purposes, eg memory
> analysis, but that information is not available in the project. The norun could
> be extended to be .norun.memstatic.exe and so the executables that form a
> specific subset can be found and analysed.

The modules used to analyze static memory benchmarks know how to find 
them. The pattern is currently f"{path}/mem-{module}-{name}.norun.exe". 
We could of course use also a different pattern or no pattern at all, 
since the specification knows exactly which executable is associated 
with which item.

> 
> The tests have been self contained for a long time and I would like that to
> continue. ELF notes has been discussed in the past however we do not yet support
> that so we need to find other ways to handling things.

The static memory benchmarks are not useful individually. You have to 
know the relationship between them to get the results. For example, what 
is the cost of using API call X in terms of static memory usage?

> 
>>> Are they suppose to be checked or are they informational? Is something going to
>>> be added to the project, for example in rtems-tools.git, to allow these tests to
>>> be checked?
>>
>> Currently, they are just informational,
> 
> I do not understand. What information, for what purpose and for whom?

I would have a look at the section 4.7 Memory Usage Benchmarks in to get 
an idea:

https://ftp.rtems.org/pub/rtems/people/sebh/rtems-6-sparc-gr740-uni-6-scf.pdf

This is just one application. You could also use the static memory 
benchmarks in a CI runner to catch memory usage regressions.

> 
>> All the stuff to analyze this and work with
>> the specification is in rtems-central.git. If you think this needs to be
>> changed, then I am happy to discuss this.
> 
> Lets first understand the role these tests have. Adding them slows the build
> down but that is OK if there is value in them for everyone.

The are controlled by the build option BUILD_MEMBENCH which is set to 
false by default. So, by default it doesn't slow down the build.

> 
>> My preference is still to get rid of
>> all the separate repositories and move everything back to rtems.git.
> 
> The qual work was separated for specific reasons. Those reasons are still valid.
Maybe this needs to be reevaluated. The tool box in rtems-central is 
quite capable. It could be also used for example to create an RTEMS release.

> 
>> What is the plan for the CI flows?
> 
> I believe it is with Joel and Amar. I am waiting like everyone else.


-- 
embedded brains GmbH
Herr Sebastian HUBER
Dornierstr. 4
82178 Puchheim
Germany
email: sebastian.huber at embedded-brains.de
phone: +49-89-18 94 741 - 16
fax:   +49-89-18 94 741 - 08

Registergericht: Amtsgericht München
Registernummer: HRB 157899
Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler
Unsere Datenschutzerklärung finden Sie hier:
https://embedded-brains.de/datenschutzerklaerung/