Tool Roadmap for the RTEMS Pre-Qualification

Tue Feb 11 10:28:53 UTC 2020

Hello,

this email tries to give an overview of the tool roadmap for the RTEMS
pre-qualification activity and things to decide for the RTEMS Project.

The tools used for the RTEMS pre-qualification will be command line 
tools only.
We will not use GUIs. New tools will be written in Python. The aim is to 
reuse
and improve existing tools of the RTEMS Project. The tools will be 
configured
via command line options and configuration files. The tool inputs will be
command line options, configuration files, source files, output from other
tools, and bug tracker databases.

The tools fall roughly in three categories. Firstly, there will be
compiler-like tools which generate output files from input files. For 
example,
one tool could generate document-specific glossaries from a project-wide
glossary. Secondly, there will be client/server tools. For example, 
there could
be a tool which receives test programs, runs them on a target system and 
sends
back the result. Thirdly, there will be builder tools. For example, the
creation of an RTEMS pre-qualification data package for a particular target
which can be handed over to an end user.

=== Important Decisions to Make ===

* Where to place the specification items?

* What should be in the specification (master data set)?

* Which content is generated?

* Which generated content is version controlled?

* How is the content generation integrated in the build system?

* How is the Python development organized?

=== Repository Overview ===

We have currently four main repositories in the RTEMS Project:

* rtems

* rtems-tools

* rtems-docs

* rtems-source-builder (RSB)

The RSB is a self-contained component and we will use it as is for the
pre-qualification activity.

There are dependencies between the rtems, rtems-tools, and rtems-docs
repositories. Inconsistencies must be currently resolved manually 
without any
tool support.

For example, we have documentation of API functions in Doxygen and the 
Classic API Guide:

https://docs.rtems.org/doxygen/branches/master/group__ClassicTasks.html#gabffda1c2301962f0ae5af042ac0bba62

https://docs.rtems.org/branches/master/c-user/task_manager.html#task-create-create-a-task

One goal of the pre-qualification activity is to introduce a master data set
(specification) and use tools to generate content in the right format 
from it.

For example, currently a human needs to edit

https://git.rtems.org/rtems/tree/cpukit/include/rtems/rtems/tasks.h

and

https://git.rtems.org/rtems-docs/tree/c-user/task_manager.rst

to declare and document the Classic Tasks API. This should be replaced by an
API specification which includes documentation elements and a generator tool
which produces the header file with Doxygen markup and the reST files 
for the
Classic API Guide. Specifically for the pre-qualification activity, the API
specification can be used to also generate an interface specification 
document
(Interface Control Document in ECSS terms).

For the generated files, we have two options:

1. They are version controlled. Specification changes and the generated 
changes
    are sent for review to the mailing list and then checked in or not. This
    means a normal user of the repository has no need to install the 
generator
    tools.  Also in case the generator tools stop to work, we are back 
to the
    current situation (with a dead specification in the repository).

2. They are not version controlled and the build system generates them on
    demand.

=== Location of the Specification ===

This seems to be a hot topic. The specification could be located in the 
rtems,
the rtems-docs, or a separate repository. The new build system is based on
specification items which are located in the "spec/build" directory in the
RTEMS sources. It is desirable to not split up the specification.

We could split up the specification and add specification items more 
related to
the documentation to the rtems-docs repository. This may end up in 
discussions
if a particular item goes into rtems or rtems-docs. I don't think this helps
and makes things easier.

We could add a new repository which contains the specification and the other
repositories as Git submodules along with tools and scripts and whatever. I
think we already have more than enough repositories in the RTEMS Project 
and we
should only introduce new repositories then there is a real need. However, I
also acknowledge that the impact of the pre-qualification activity should be
manageable. Splitting up the specification into a build related part 
which is
contained in the RTEMS sources repository and the rest could be a way 
forward
to get started. Thanks to Chris for bringing up this approach.

What do you think about this:

1. The new build system will remain as is and uses the build specification
    items located in "spec/build" in the RTEMS sources.

2. We add a new empty rtems-qual repository.

2.1. In this repository we add a "spec" directory for the non-build
      specification items.

2.2. In this repository we add all the generator tools.

2.3. In this repository we add things closely related to pre-qualification.

2.4. In this repository we add Git submodules for the other RTEMS 
repositories
      touched by the generator tools. Changes in generated files in the 
standard
      RTEMS repositories go through the normal patch review process.

2.5. This repository may use a simplified review policy during the initial
      pre-qualification activity.

Once the pre-qualification activity produced in a mature and usable
infrastructure we can re-evaluate the repository organization and the 
location
of the specification.

=== Python Development ===

The tool development in Python for the pre-qualification is a team work
activity. Therefore we should introduce a coding standard, automatic code
formatters (black, yapf), static analysis tools (mypy, pylint, flake8),
documentation checkers (pydocstyle), and unit/integration tests 
(unittest and
unittest.mock modules). The mentioned Python development tools are just
examples. There use and configuration should be discussed on the RTEMS 
mailing
list.

If we place the specification along with corresponding tools into a separate
repository we can use it to set up a prototype Python development work flow.

=== Build System Integration ===

This section is only relevant if the specification is not located in a
dedicated repository.

It is tedious to know which tools, input files, and output files are 
used in a
particular repository and how they are invoked. In each repository, a build
system is present. The build system can be used to update the generated 
content
of a particular repository on demand (e.g. if the specification 
changed). For
example, we could add an --enable-maintainer-mode option to the waf 
configure
command.

./waf configure --enable-maintainer-mode --rtems-tools=X --rtems-spec=Y

This could check that the tools and specification are available and enable
rules to re-generate content based on changes in the specification. The 
tools
could report a version and the build system could check that the right tool
version is used to avoid a re-generation with the wrong tools.

Having the tool configuration, invocation, and dependencies in the build 
system
is also an accurate documentation how the things are set up and makes sure
everyone is using them in a defined way.

=== Howtos ===

We will add howtos for common RTEMS maintenance task, e.g. how to add a 
new API
function, how to add a glossary term, etc. See also "8.6 Howtos" section 
of the
new build system:

https://ftp.rtems.org/pub/rtems/people/sebh/eng.pdf

It may make sense to collect all howtos in a dedicated chapter:

https://lists.rtems.org/pipermail/devel/2020-January/056849.html

=== Specification-to-X Tool ===

The specification-to-X tool could generate the following content from the
specification (controlled by command line options or a configuration file):

* document-specific glossaries (VC)

* API documentation reST files for the Classic API Guide (VC)

* API header files with Doxygen markup (VC)

* interface specification document (Interface Control Document in ECSS 
terms)

* software requirements specification document

* test plans

* configuration files for static analysis tools

* configuration file for Doxygen

Generated content which is independent of a particular target system and 
has a
low rate of change should be version controlled (VC).

=== Traceability of Specification Items ===

For the traceability of specification items, please have a look at:

https://docs.rtems.org/branches/master/eng/req-eng.html#traceability-of-specification-items

There are some options available to provide a traceable history of 
specification items.

Standards demand forward and backward traceability between specification 
items
(e.g. requirements). This is achieved through standard Doorstop features.

The traceability between software requirements, architecture and design will
probably need some iterations to find to a good solution.

=== Additional Tools ===

José Valdez from EDISOFT will give you shortly an overview of additional 
tools
which cover the following topics:

* static code analysis

* test execution

* code coverage

* test output analysis

* test report generation

* test plan generation