Benchmark Apps for RTEMS

[Rempel, Cynthia]

>________________________________________
>From: gedare at gwmail.gwu.edu [gedare at gwmail.gwu.edu] on behalf of Gedare Bloom [gedare at rtems.org]
>Sent: Thursday, April 18, 2013 6:12 AM
>To: Rempel, Cynthia
>Cc: Vivek Krishnamurthy; rtems-devel at rtems.org
>Subject: Re: Benchmark Apps for RTEMS
>
>I would like to see a build system approach for the benchkit that is
>easily maintained and flexible, probably something Waf-based. 
I agree it should be maintainable, and the current version of examples-v2 is maintainable...

I think we need to pause and think about the implications of pushing something Waf-based on students...

My understanding is we have been trying for months to get examples-v2 to be Waf-based, and because of the header problems, we have been unsuccessful so far.  Waf is a step up in complexity over a standard Makefile; if we go that route, we need the examples-v2 with a Waf build system publicly available as a reference first...

Only a select few RTEMS developers know how to use Waf and we don't want this to be an elitist open-source project.  There needs to be a publically available tutorial for how to maintain a Waf-based build-system (I think its inadvisable that only a few people should have keys to the build-system kingdom)...

I will go along with doing a Waf-based build system, I would like to hear your refutations, but I think it's inadvisable to do so until the above issues are addressed.

>I have
>ported a few benchmarks, e.g. mibench [1], in the past but not in a
>flexible way--I integrated them directly to the RTEMS build for my
>ease of use. I think this benchkit can be a benefit by providing an
>external (to rtems.git) repository of benchmark packages that can be
>individually maintained, perhaps in a git submodule, but uniformly
>compiled and executed with the same build system.

I agree it should be a separate git, I would suggest it remains being built independently, so we can build the benchkit with older versions of RTEMS as well. We could always tie the benchkit into RSB.

>The work here is to define the "kit" framework, which is of general
>interest for porting other software to RTEMS, and to apply the kit
>framework to some benchmark suites. 
There is a separate "make package" open project to address this issue... but the student didn't select that one.

>Aside from the build system, there
>may be other issues to handle such as dealing with IO and
>filing/fixing bugs that arise when running these benchmarks.
>[1] http://code.google.com/p/rtemssparc64/source/browse/#svn%2Ftrunk%2Frtems%2Frtemscvs%2Ftestsuites%2Fmibench
After selecting benchmarks this is what the majority of coding will entail...