Space Qualified RTEMS

Joel Sherrill joel.sherrill at OARcorp.com
Wed Jan 25 17:05:22 UTC 2012 

This thread has touched on an issue which is really important
to the RTEMS Project.  Forking old versions isn't a good
long-term solution for users or the project.  The RTEMS Project
exists because of its users and finding the best technical
solution for RTEMS releases to feed into an organization's
validation and qualification process is important.

The RTEMS Project produces releases with a particular
set of supporting artifacts in arbitrary formats. The set of
artifacts, the format, and the technical information in them
can and should be driven by user requirements.

For example, RTEMS Tests are designed to be run on any user
hardware. Combined with the coverage analysis tools, you can
reproduce our test and coverage results.  If the results are
insufficient, provide improvements so they meet your
requirements. You use those improvements locally and they
get merged into the main project. Next time, the test reports
and results should meet your requirements or at least be closer.

Without feedback to the RTEMS Project and a closed loop,
this will never happen.  Nothing personal, this is just what
needs to happen.

Cynically, I believe it is not in the business interest of the large
contractors doing the space work to provide feedback. If it keeps
their fork alive or keeps it costly to validate a new RTEMS version for
the next mission, then it is good for their business. They sell hours
and bodies.

On 01/25/2012 09:38 AM, Zandin Johan RUAG N wrote:
> Lee: My experience is that it's very dependent on the criticality of the software:
> * If it's just part of an instrument, where the mission can handle that the instrument
>    goes down and has to be rebooted, the requirements from ESA are usually less rigorous.
> * But if you build mission-critical software, e.g. for attitude- and orbit-control or
>    for the overall management of the spacecraft (which is my main tasks), they will
>    usually require a validated version. The same goes for safety-critical software
>    (e.g. manned space missions and launcher systems).
>
>
I expect this to always be true. But the distance between "official" RTEMS
and what meets a project's validation requirements doesn't have to be
far apart.

The fundamental problem is that there is no desire, intention, effort, etc.
to provide any feedback from the "space qualified" version back into the
main source base. This means that it will be no easier to feed a new version
of RTEMS into the validation machinery than it was to do 4.8.
> Thomas: Very good points! I think they should be discussed with ESA and with both
> minor an major RTEMS users in the European space industry.
>
Thomas and I have talked a lot about this over the past few years and he
is spot on.  I agree with everything he said.  I would just like to add that
the ultimate goal of the test suite improvements and coverage analysis
is for the open RTEMS Project to provide releases which can be easily
incorporated into projects with high requirements.

There has been no significant feedback from anyone in a qualification
path on what coverage or information in reports they would like to see.
This is even less input than providing test cases or code improvements
which would improve testability, coverage, etc.

We want the RTEMS Project releases to be of high quality and feed into
ESA and NASA qualification efforts. But so far, this has been a one-sided
conversation.
> In the numerous RTEMS projects I have worked, our intention has definitely been to
> report all detected problems upstream and some of the differences between the
> "space qualified" versions and the RTEMS main branch can probably be solved by
> properly committing patches upstream.
>
Sorry to say but there really hasn't been any significant input from the ESA
community.  Maybe a handful of PRs but I don't remember any off the
top of my head.

In contrast, MMS is using a release tarball (or something very close). They
provided feedback and we worked with them so a standard release met
their requirements.
> But other ones would probably have a larger impact on the RTEMS main branch if
> they should be integrated there, e.g. adding configuration options to strip away
> unused code (to save memory space, to avoid accidental execution and to minimize
> the amount of code to qualify). That of course has to be discussed in the RTEMS
> community too.
>
It wouldn't take any more effort than it would to implement on an old
version of RTEMS. It isn't a matter of money or engineering effort, it is
simply a lack of intention.

I have no idea what code space reduction options have been added/hacked
into the ESA 4.8 but many code space reduction options have been added
to the real source since then.  Implementing these type of configuration 
options
is tricky and I would have my personal doubts that if they have done 
anything
significant, they have likely broken something.

But no attempt has even been made to feed these back into the community.
So that leaves ESA with a hacked fork that bears less and less 
resemblance to
the main source that continues to evolve and improve with virtually no 
input
from that community.
> But once such issues have been resolved, it should be perfectly possible to space
> qualify an official RTEMS version instead. To minimize the qualification work, you
> would probably only qualify a limited set of configurations on a particular HW
> platform. But it would be the same code base as anyone else used.
>
I stand by my assertion that it is possible for the RTEMS Project to provide
test suites, coverage reports and tools, and configuration options that 
would
make your in-house efforts easier. Certainly the distance between RTEMS
release tarball and what passes an organization's test and validation 
efforts
can shrink. But that takes feedback on what is missing and what of that
makes sense to even be provided by the RTEMS Project.

Ultimately, RTEMS is what the users make of it.
> Best regards
> Johan Zandin
>
> -----------------------------------------------------------
> Johan Zandin                      Software Engineer
> RUAG Space AB                     Phone: +46-31-735 41 47
> SE-405 15 Gothenburg, Sweden      Fax:   +46-31-735 40 00
> -----------------------------------------------------------
>
>
>> -----Original Message-----
>> From: rtems-users-bounces at rtems.org
>> [mailto:rtems-users-bounces at rtems.org] On Behalf Of Matthews, Lee
>> Sent: den 25 januari 2012 10:52
>> To: 'Thomas Doerfler'; rtems-users at rtems.org
>> Subject: RE: Space Qualified RTEMS
>>
>> Thomas, Cláudio,
>>
>> Thank you for your feedback. Thomas you note that the RTEMS
>> community has contributed to many space missions. Could you
>> please give me some examples of any previous (or upcoming)
>> missions that use RTEMS, particularly version 4.10. This will
>> give me an idea of heritage and provide me with some leverage
>> if ever ESA decides to try and force version 4.8 upon us.
>>
>> Thanks again.
>>
>> Best wishes,
>> Lee
>>
>> -----Original Message-----
>> From: rtems-users-bounces at rtems.org
>> [mailto:rtems-users-bounces at rtems.org] On Behalf Of Thomas Doerfler
>> Sent: 24 January 2012 17:13
>> To: rtems-users at rtems.org
>> Subject: Re: Space Qualified RTEMS
>>
>> Hi Lee, hi all,
>>
>> First of all: the RTEMS commmunity is proud to have contributed to so
>> many space missions. AFAIK at least the NASA projects had
>> returned their
>> share of code improvement to RTEMS, which is a benefit to all future
>> users (including future NASA missions).
>>
>> Your set of questions significantly emphasises the problem with those
>> "pull everything, commit nothing" projects like the ESA "space
>> qualified" RTEMS kernel.
>>
>> It is nice and vital for the ESA space community to have a qualified
>> RTOS, which also supports the space specific platforms like ERC32/LEON.
>> But those "read-only" activities are all prone to severe bit rot.
>>
>> Since RTEMS 4.8, there were severe improvements to the system:
>> - many bugs were identified and eliminated
>> - new features were added
>> - support for up-to-date hardware was added (and this is what you ask
>> for in your mail)
>> - siginificant activities were launched to improve code quality, like
>> extensions to the test sets, automated code coverage analysis
>> and others.
>>
>> You will not have access to these improvements, because the "space
>> qualified" RTEMS has not been merged back into the main RTEMS
>> repository.
>>
>> -------------------------
>>
>> I have a scenario in mind:
>>
>> - Let's say that during a certain test configuration late in the
>> development cycle, my system will fail
>>
>> - the reason for the failure can be tracked down to an OS bug (which is
>> unlikely, but may happen in every piece of software)
>>
>> - I or my colleagues will find out that the bug has long been
>> identified
>> and fixed in RTEMS
>>
>> - but I am using a "space qualified" version of RTEMS, which
>> has the bug
>> still in it...
>>
>> ... this would really bother me. Almost every software has minor bugs.
>> But using an old release of a software, which, due to the develpoment
>> model selected, does not care about improvements and fixes in the main
>> source tree may be regarded rather careless...
>> -------------------------
>>
>> What I really long to see is to have these space qualification related
>> modifications to RTEMS be integrated back into the main RTEMS
>> repository, because that's how Open Source software works:
>>
>> If all users share their OS software knowledge, every user can partake
>> in the whole collected knowledge.
>>
>> Just my two cents,
>>
>> Thomas.
>>
>>
>> Am 24.01.2012 16:41, schrieb Matthews, Lee:
>>> Hi,
>>>
>>>
>>>
>>> I'm developing software that uses RTEMS on an Aeroflex
>> Gaisler LEON3FT
>>> processor that is running on a Pender GR-CPCI-AX2000
>> development board.
>>> I am using version 4.10 of RTEMS and RCC 1.1.99.19.
>>>
>>>
>>>
>>> We are working on producing a Magnetometer instrument that will be
>>> integrated into ESA's upcoming Solar Orbiter mission to the Sun. We
>>> shall be using RTEMS on our Leon3FT CPU. Having just gone
>> through ESA's
>>> Preliminary Design Review process, it has been highlighted
>> by ESA that
>>> there is a space qualified version of RTEMS available, though this is
>>> currently based on version 4.8 of the kernel. I believe this is being
>>> developed by the company Edisoft (http://rtemscentre.edisoft.pt).**
>>>
>>>
>>>
>>> I have a few questions about this :
>>>
>>>
>>>
>>> 1)  Does anyone know what the difference is exactly between
>> a "standard"
>>> and a "space qualified" version of RTEMS ?
>>>
>>> 2)  Assuming that we were to use the space qualified RTEMS
>> 4.8 kernel,
>>> would we still be able to use Aeroflex Gaisler's BSP (RCC
>> 1.1.99.19) ?
>>> 3)  Would Aeroflex Gaisler's BSP also need to be space qualified ?
>>>
>>>
>>>
>>> Thanks in advance.
>>>
>>>
>>>
>>> Best wishes,
>>>
>>> Lee Matthews
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> rtems-users mailing list
>>> rtems-users at rtems.org
>>> http://www.rtems.org/mailman/listinfo/rtems-users
>>
>> -- 
>> --------------------------------------------
>> Embedded Brains GmbH
>> Thomas Doerfler           Obere Lagerstr. 30
>> D-82178 Puchheim          Germany
>> email: Thomas.Doerfler at embedded-brains.de
>> Phone: +49-89-18908079-2
>> Fax:   +49-89-18908079-9
>> _______________________________________________
>> rtems-users mailing list
>> rtems-users at rtems.org
>> http://www.rtems.org/mailman/listinfo/rtems-users
>>
>> _______________________________________________
>> rtems-users mailing list
>> rtems-users at rtems.org
>> http://www.rtems.org/mailman/listinfo/rtems-users
>>
> _______________________________________________
> rtems-users mailing list
> rtems-users at rtems.org
> http://www.rtems.org/mailman/listinfo/rtems-users


-- 
Joel Sherrill, Ph.D.             Director of Research&   Development
joel.sherrill at OARcorp.com        On-Line Applications Research
Ask me about RTEMS: a free RTOS  Huntsville AL 35805
     Support Available             (256) 722-9985

More information about the users mailing list