stress program in RTEMS

[Bornet Romain]

Hi,

> -----Original Message-----
> From: Joel Sherrill [mailto:joel.sherrill at oarcorp.com]
> Sent: mercredi 23 octobre 2013 19:39
> To: Bornet Romain; Pierre Ficheux; rtems-users at rtems.org
> Subject: Re: stress program in RTEMS
> 
> Sorry for a long email. I think a button got pushed. :)
No problem, always good to have feedback from insiders and experts (I'm rather new in RTEMS and don't have you background and experience on it).

> 
> On 10/23/2013 2:31 AM, Bornet Romain wrote:
> > Hi!
> >
> > I'm not aware of any standard stress test but I had a look at the ones
> presented here:
> ftp://ftp.rtems.com/pub/rtems/esa_validation_report_450/RTEMS_Depend
> abilityTesting/index.html
> >
> ftp://ftp.rtems.com/pub/rtems/esa_validation_report_450/RTEMS_Depend
> abilityTesting/
> >
> > There are stress and robustness tests explained in details and whose
> implementation is available...
> 
> Where is this code available? I thought it was dependent
> on a proprietary tool. I don't know of anything that was
> in that report that is completely publicly available.
> I personally never saw any code on that and all opinions
> from here down are from the reports and my > 10 year old
> recollection. Hopefully someone at ESA/ESTEC can comment.
> If the code/tests are still useful, then it would be
> nice to have them.
> 
> I haven't looked at those > 10 years but recall that
> not much came out of it. The folks doing the work uncovered
> a few things that were useful and wrote up a lot of cases
> that were pointless. They pushed arbitrary values for each
> data type through the API calls they were interested in.
> This is useful but doesn't say anything about real applications.

As stated, I never found time to dig into this more in detail and only saw there were "Implementation" folders in the tree with sources which looked like valid test cases. If I had took a bit more time I would have tried to compile all the stuff and according to your feedback, I would have had much trouble with it :-)

> 
> This is the technical list of what they found:
> 
> + the Classic API did not check for NULL parameters. This
> was per the original specification and was by design. 0x0000
> was a valid address on some targets at that point in history.
> We added NULL checks in response.
> 
> + A few range checks on parameters were incorrect. These
> were the real bugs found. It wasn't many.
> 
> + A sentence in the manual was inverted. It had a not or
> was missing a not. I don't recall which. The code was correct.
> 
> Most of the failure cases documented were for do something
> like configuring a system with 1MB RAM with 1000 tasks and being
> surprised to get a fatal error early in the BSP. At the time
> on the BSP they used, this happened on the second line of code
> executed in bsp_start(). It failed silently. Not perfect but not
> worth writing up > 1000 times.
> 
> As a matter of perspective, I asked for a period of off-contract
> paid time from OAR to investigate this report. Every issue was
> easy enough that I fixed them while reading the report.
> The "estimation" phase resulted in all the issues being
> fixed. No further engineering was done. 4.6 has a gap in the
> numbering because a release was being staged and I stopped it.
> We bumped the last digit and that one went out a few days later
> with all issues fixed.

Nice to get the whole story about it from an insider. In fact, technical issues reported do not seem really relevant and would have been fixed more quickly by standard means (mailing-list,...) than with a huge report...

> This report is also my prime example of "please ask for help".
> The work was done, reports written and contract closed before
> we ever heard about it. We didn't get to review, comment, etc.
> I use this as an example of please let a core developer review
> any reports. If I hadn't felt pride in RTEMS, this report would
> have been ignored and had zero impact. If we had talked with the
> people doing the work, we could have addressed the issues as
> they went and possibly improved their work with our insights.

Totally agree... There is often a big gap between QA and developers which just adds overhead and delays instead of just fixing things.

Now that we spoke about this not so useful "test and benchmarking" tool, would you have a recommendation for the original poster who is looking for "stress" or benchmarking tools ?

I know about the 'paranoia' floating point test: http://git.rtems.org/joel/rtems.git/tree/testsuites/samples/paranoia
Any other pointer ?

Regards,
    Romain

> > I didn't run them on my system until now but kept the link "in case I have
> time"...
> >
> >>> I'm currently writing a new article about RTEMS.
> > Looking forward to read it ,hopefully in
> GLMF[http://www.unixgarden.com/index.php/category/gnu-linux-
> magazine] or
> > Open Silicium[http://www.unixgarden.com/index.php/category/open-
> silicium] :-)
> >
> > Hope this helps,
> >     Romain Bornet
> >
> >
> > -----Original Message-----
> > From: rtems-users-bounces at rtems.org [mailto:rtems-users-
> bounces at rtems.org] On Behalf Of Pierre Ficheux
> > Sent: mercredi 23 octobre 2013 08:18
> > To: rtems-users at rtems.org
> > Subject: stress program in RTEMS
> >
> > Hi,
> >
> > I'm currently writing a new article about RTEMS. I'm looking for an example
> of stress task such as "stress" or "hackbench" on Linux.
> >
> > What's the best way to generate stress for RTEMS ? using floating floating
> point operation?
> >
> > Thx
> >
> 
> 
> 
> --
> Joel Sherrill, Ph.D.             Director of Research & Development
> joel.sherrill at OARcorp.com        On-Line Applications Research
> Ask me about RTEMS: a free RTOS  Huntsville AL 35805
> Support Available                (256) 722-9985