Critical Software Analysis on RTEMS 4.5.0

[Joel Sherrill <joel@OARcorp.com>]

Fabrício de Novaes Kucinskis wrote:
> Hi all,
> 
> 
> I heard about an extensive analysis that Critical Software did on the RTEMS
> 4.5.0 port to ERC32 under a contract with ESA.
> 
> Do you know about it? There were 49 errors reported (34 on the Classic API,
> 15 on Posix) - it's important to notice that this analysis has occurred
> focusing space applications, so the requirements are higher than the normal.

I have seen the report but it was only after it was generated and no 
person (volunteer or paid consultant) associated with the RTEMS Project 
was given an opportunity to review the tests before the report was
published.  In addition, if it was so critical to have RTEMS analyzed, 
why wasn't it critical to contact the project to report the problems
and maybe sponsor someone to fix them?

The analysis that has been done and the fixes committed (see PRs in
the PR650-660 range fixed in 4.6.2) were done by me in purely volunteer
capacity. If someone wants a more thorough analysis and explanation of
the problems with RTEMS or the tests themselves, they need to pay for it.

Moreover, most of the problems were not as "critical" as you would think
based upon the title.  A large portion of the tests, pushed arbitrary 
values into API calls to see if the error checking caught them.  So
most of the errors were like a missing NULL check on argument or a range 
check not right or missing on argument.  A few tests tripped some
math overflow errors in address calculations and those were fixed.  One
test just pointed out a spot in the documentation that was not clear.

Another class of errors were configuring too many objects for the 
limited memory on their target hardware.  Sometimes the BSP called
fatal error in bspstart.c since it couldn't allocate that much memory. 
Other times RTEMS noticed that there was not enough memory and
called fatal error.  Since there was no printing on these system
initialization error paths, I recall a lot of these were classified
as failures.

Sorry for the ranting.  I was just hurt that someone paid a lot of
money to analyse RTEMS and then didn't bother to make any effort to
get the issues looked at.

I am very thankful that the people at Critical Software were kind enough
to even let me know of the report even if it was already completed and
published on an ESA ftp site.  Otherwise, we never would have even
known about it.   They even answered some questions which aided in
analyzing the report.

If anyone cares, I really would like the opportunity to properly analyze
the test report and generate proper responses to each anomaly.  That is
what testing is supposed to be about.

Again I apologize for the rant.  And I want to repeat that many of the
issues were addressed in RTEMS 4.6.2.


> Best regards,
> 
> 
> 
> Fabrício de Novaes Kucinskis - DEA / INPE
> -----------------------------------------
> Divisão de Eletrônica Aeroespacial
> Instituto Nacional de Pesquisas Espaciais
> 


-- 
Joel Sherrill, Ph.D.             Director of Research & Development
joel at OARcorp.com                 On-Line Applications Research
Ask me about RTEMS: a free RTOS  Huntsville AL 35805
    Support Available             (256) 722-9985