Pre-Qualification Master Plan

[Sebastian Huber]

Hello,

I will try to present the "master plan" for the RTEMS pre-qualification project
done in the scope of an ESA activity. Since I am not a master in software
qualification activities, this is probably only a novice plan. 

There is a master ticket for this activity here:

https://devel.rtems.org/ticket/3701

We analysed the relevant ECSS standards with respect to software development
(ECSS-E-ST-40C and ECSS-Q-ST-80C Rev.1) and other standards such as IEC 61508,
ISO 26262, DO-178, DO-330, DO-333, and Galileo Software Standard (GSWS). We
also had a look at previous RTEMS pre-qualification projects. We ended up with
roughly the same conclusions as presented in the RTEMS Software Engineering
manual (content probably from Joel Sherill or Scott Zemerick).

https://docs.rtems.org/branches/master/eng/prequalification.html

ECSS-E-ST-40C requires that you deliver about 70 documents. For some documents
the scope, content, formatting, structure, etc. is defined by the standard. For
each document we have a proposal how it can be delivered. Several documents
were discarded since they correspond to project life-cycles which are not
included in the ESA activity, e.g. operational phase, maintenance. RTEMS is
operational and maintained. Some documents are specific to the ESA activity and
have no value for the RTEMS Project, e.g. the Software Development Plan for the
overall ESA activity.

At least the ECSS standards demand from you that the software is developed
according to procedures defined by the standards. ECSS is very detailed here.
Obviously, it makes no sense to apply these clauses to an open source project
one-to-one. What we try to do is document the procedures used by the RTEMS
community, so that an auditor not familiar with RTEMS can get an overview about
the project organization. New contributors may also benefit from this
documentation.

Another big chunk in the standards is how the software product is derived from
and in line with requirements. This is also outlined in the RTEMS Software
Engineering manual:

https://docs.rtems.org/branches/master/eng/prequalification.html

"The second area is the creation and maintenance of master technical data that
has traditionally not been owned or maintained by the RTEMS Project. The most
obvious example of this is a requirements set with proper infrastructure for
tracing requirements through code to test and documentation. It is expected
that these will be maintained by the RTEMS Qualification Project. They will be
evaluated for adoption by the main RTEMS Project but the additional maintenance
burden imposed will be a strong factor in this consideration. It behooves the
RTEMS Qualification Project to limit dependence on manual checks and ensure
that automation and ongoing support for that automation is contributed to the
RTEMS Project."

The missing requirements set and the corresponding traceability is the biggest
work package we have in this activity. To manage requirements a tool support is
necessary. We evaluated some open source requirements management tools:

https://docs.rtems.org/branches/master/eng/req-eng.html#tooling

We ended up with Doorstop. I am very happy about this tool and I think it
perfectly fits to what I have in mind with the specification items:

https://docs.rtems.org/branches/master/eng/req-eng.html#

I wrote a new build system for RTEMS based on specification items:

https://lists.rtems.org/pipermail/devel/2019-November/056267.html

This turned out to be a good proof of concept.

The specification items (YAML files) are machine and human readable (at least
for the subset of humans which got used to read plain text files). Between the
specification items and the documents mandated by standards is a gap. To fill
this gap tools are necessary. These tools run on the host (not the RTEMS
target). They will use the following inputs:

* specification items (in the RTEMS main repository "spec" directory)

* the RTEMS sources

* the RTEMS documentation sources

* output from test suite runs

* output from other tools (e.g. static analysers)

* databases (e.g. a ticket system)

Some tools will generate documentation sources (*.rst files) and configuration
files for other tools (e.g. Doxygen, static code analysers) for documents such
as:

* user manuals

* software architecture and design documentation

* Software Requirements Specification (SRS; ECSS-specific name)

* Interface Control Document (ICD; ECSS-specific name)

* test reports

* static analysis reports

* software problem reports

Some documents may be standard-specific. The goal is to have all meta-data
available in the specification items to satisfy different standards. The
generator tools should handle the standard specializations.

Other tools will execute tests on a specific target and collect the output
(e.g. the RTEMS Tester). We need tools to collect everything and produce a
package.

To implement the tools we need a programming language. From the set of
languages currently used by the RTEMS project for tools running on the host C++
and Python are an option. We think that Python is the better choice. Someone in
favour of Java?

Developing these tools is a team work activity. This gives raise to coding
guidelines, continuous integration builds, use of tools in a state-of-the art
development work flow (static analysis, type checkers, code formatters,
unit/integration tests with code coverage). Python has good tools available for
all of this and there are Python projects which use them successfully (e.g.
Doorstop). In the RTEMS Project we currently have only the Python code and some
high level documentation. This works fine if only a very small set of
experienced developers work in this area. To allow team work and integrate new
contributors we have to improve this.

To get started with the specification items and generator tools I would like to
begin with the RTEMS glossary:

https://lists.rtems.org/pipermail/devel/2020-January/056728.html

The glossary categories and terms will be added as specification items. A tool
will read these items and produce the glossary.rst for documents (e.g.  c-user
and eng). The automatically generated glossary.rst should be version controlled
in the rtems-docs repository. The rational for this is that the glossary terms
change only infrequently. Having the generated files under version control
makes it possible to build the documentation without access to the generator
tools. Changes can be easily reviewed.

The next step will be to specify the RTEMS application configuration options. A
tool can then generate the documentation (RTEMS Classic API Guide) and parts of
the ICD.

The next step will be to specify the Classic API. A tool can then generate the
API header files with Doxygen markup, the API documentation in the RTEMS
Classic API Guide and parts of the ICD.

The next step will be the functional specification of a subset of the Classic
API.

https://ftp.rtems.org/pub/rtems/people/sebh/tn-space-profile-r2-18072019.pdf

The aim is to start with simple things and then introduce step by step the more
complex specification items.

The specification items will reduce redundancy in the project at the expense of
non-obvious data flows. For example, let us look at glossary.rst. With a hand
written file you can easily edit it and for example add a new term or fix a
typo. With a generated glossary.rst you have to do changes in another
repository and use a tool to generate the file. However, this will fix the typo
also in a couple of other documents and not just the one you looked at.

To document standard activities carried out by RTEMS maintainers I suggest to
add a Howto chapter to the RTEMS Software Engineering manual:

https://lists.rtems.org/pipermail/devel/2020-January/056849.html

I hope this helps a bit to get some background of the ongoing work.

-- 
Sebastian Huber, embedded brains GmbH

Address : Dornierstr. 4, D-82178 Puchheim, Germany
Phone   : +49 89 189 47 41-16
Fax     : +49 89 189 47 41-09
E-Mail  : sebastian.huber at embedded-brains.de
PGP     : Public key available on request.

Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG.