stress program in RTEMS

Gedare Bloom gedare at rtems.org
Thu Oct 24 13:15:45 UTC 2013


I've identified some potential real-time benchmarking suites:
http://wiki.rtems.org/wiki/index.php/RTEMS_BenchKit

I have not had the time to try them out though.

I have integrated some benchmarks into RTEMS testsuite directory for
personal use, see
http://code.google.com/p/rtemssparc64/source/browse/# and look under:
trunk/rtems/rtemscvs/testsuites/ for the disbench [1], heapbench, and
mibench subdirectories. disbench are "data intensive system"
benchmarks [1], heapbench is a collection of benchmarks many of which
can be found in the Octane benchmark suite, and mibench is the MIBench
suite [2] that is an open-source alternative to the EEMBC. They can't
be contributed to RTEMS directly due to licensing issues, but they are
all open-source. One could use them as I have done, or repackage them
as an external suite that can be built and executed.

I have other stress/benchmark projects in the wings, but they need
some maturing before I can bring them to light.

[1] www.ai.mit.edu/projects/aries/Documents/DIS_Benchmarks_v1.pdf
[2] http://www.eecs.umich.edu/mibench/

-Gedare

On Thu, Oct 24, 2013 at 3:43 AM, Bornet Romain <romain.bornet at heig-vd.ch> wrote:
> Hi,
>
>> -----Original Message-----
>> From: Joel Sherrill [mailto:joel.sherrill at oarcorp.com]
>> Sent: mercredi 23 octobre 2013 19:39
>> To: Bornet Romain; Pierre Ficheux; rtems-users at rtems.org
>> Subject: Re: stress program in RTEMS
>>
>> Sorry for a long email. I think a button got pushed. :)
> No problem, always good to have feedback from insiders and experts (I'm rather new in RTEMS and don't have you background and experience on it).
>
>>
>> On 10/23/2013 2:31 AM, Bornet Romain wrote:
>> > Hi!
>> >
>> > I'm not aware of any standard stress test but I had a look at the ones
>> presented here:
>> ftp://ftp.rtems.com/pub/rtems/esa_validation_report_450/RTEMS_Depend
>> abilityTesting/index.html
>> >
>> ftp://ftp.rtems.com/pub/rtems/esa_validation_report_450/RTEMS_Depend
>> abilityTesting/
>> >
>> > There are stress and robustness tests explained in details and whose
>> implementation is available...
>>
>> Where is this code available? I thought it was dependent
>> on a proprietary tool. I don't know of anything that was
>> in that report that is completely publicly available.
>> I personally never saw any code on that and all opinions
>> from here down are from the reports and my > 10 year old
>> recollection. Hopefully someone at ESA/ESTEC can comment.
>> If the code/tests are still useful, then it would be
>> nice to have them.
>>
>> I haven't looked at those > 10 years but recall that
>> not much came out of it. The folks doing the work uncovered
>> a few things that were useful and wrote up a lot of cases
>> that were pointless. They pushed arbitrary values for each
>> data type through the API calls they were interested in.
>> This is useful but doesn't say anything about real applications.
>
> As stated, I never found time to dig into this more in detail and only saw there were "Implementation" folders in the tree with sources which looked like valid test cases. If I had took a bit more time I would have tried to compile all the stuff and according to your feedback, I would have had much trouble with it :-)
>
>>
>> This is the technical list of what they found:
>>
>> + the Classic API did not check for NULL parameters. This
>> was per the original specification and was by design. 0x0000
>> was a valid address on some targets at that point in history.
>> We added NULL checks in response.
>>
>> + A few range checks on parameters were incorrect. These
>> were the real bugs found. It wasn't many.
>>
>> + A sentence in the manual was inverted. It had a not or
>> was missing a not. I don't recall which. The code was correct.
>>
>> Most of the failure cases documented were for do something
>> like configuring a system with 1MB RAM with 1000 tasks and being
>> surprised to get a fatal error early in the BSP. At the time
>> on the BSP they used, this happened on the second line of code
>> executed in bsp_start(). It failed silently. Not perfect but not
>> worth writing up > 1000 times.
>>
>> As a matter of perspective, I asked for a period of off-contract
>> paid time from OAR to investigate this report. Every issue was
>> easy enough that I fixed them while reading the report.
>> The "estimation" phase resulted in all the issues being
>> fixed. No further engineering was done. 4.6 has a gap in the
>> numbering because a release was being staged and I stopped it.
>> We bumped the last digit and that one went out a few days later
>> with all issues fixed.
>
> Nice to get the whole story about it from an insider. In fact, technical issues reported do not seem really relevant and would have been fixed more quickly by standard means (mailing-list,...) than with a huge report...
>
>> This report is also my prime example of "please ask for help".
>> The work was done, reports written and contract closed before
>> we ever heard about it. We didn't get to review, comment, etc.
>> I use this as an example of please let a core developer review
>> any reports. If I hadn't felt pride in RTEMS, this report would
>> have been ignored and had zero impact. If we had talked with the
>> people doing the work, we could have addressed the issues as
>> they went and possibly improved their work with our insights.
>
> Totally agree... There is often a big gap between QA and developers which just adds overhead and delays instead of just fixing things.
>
> Now that we spoke about this not so useful "test and benchmarking" tool, would you have a recommendation for the original poster who is looking for "stress" or benchmarking tools ?
>
> I know about the 'paranoia' floating point test: http://git.rtems.org/joel/rtems.git/tree/testsuites/samples/paranoia
> Any other pointer ?
>
> Regards,
>     Romain
>
>> > I didn't run them on my system until now but kept the link "in case I have
>> time"...
>> >
>> >>> I'm currently writing a new article about RTEMS.
>> > Looking forward to read it ,hopefully in
>> GLMF[http://www.unixgarden.com/index.php/category/gnu-linux-
>> magazine] or
>> > Open Silicium[http://www.unixgarden.com/index.php/category/open-
>> silicium] :-)
>> >
>> > Hope this helps,
>> >     Romain Bornet
>> >
>> >
>> > -----Original Message-----
>> > From: rtems-users-bounces at rtems.org [mailto:rtems-users-
>> bounces at rtems.org] On Behalf Of Pierre Ficheux
>> > Sent: mercredi 23 octobre 2013 08:18
>> > To: rtems-users at rtems.org
>> > Subject: stress program in RTEMS
>> >
>> > Hi,
>> >
>> > I'm currently writing a new article about RTEMS. I'm looking for an example
>> of stress task such as "stress" or "hackbench" on Linux.
>> >
>> > What's the best way to generate stress for RTEMS ? using floating floating
>> point operation?
>> >
>> > Thx
>> >
>>
>>
>>
>> --
>> Joel Sherrill, Ph.D.             Director of Research & Development
>> joel.sherrill at OARcorp.com        On-Line Applications Research
>> Ask me about RTEMS: a free RTOS  Huntsville AL 35805
>> Support Available                (256) 722-9985
> _______________________________________________
> rtems-users mailing list
> rtems-users at rtems.org
> http://www.rtems.org/mailman/listinfo/rtems-users




More information about the users mailing list