Tool Roadmap for the RTEMS Pre-Qualification
Jose Valdez
Jose.Valdez at edisoft.pt
Wed Feb 19 13:05:23 UTC 2020
Hello,
Following the e-mail sent by Sebastian, please find here the missing information, regarding the tools for RTEMS pre qualification.
Best regards
José
=== Test Executor ===
The Test Executor will be the software that manages the execution of the RTEMS Tests.
It will set the necessary hardware, run the executable tests, control the execution and gather the output.
The following capabilities will be available:
* Send commands to the target platform.
A subset of commands will be made available, but it shall be possible to add more commands.
* Send commands to auxiliary test programs, which run in external devices.
This will be done by a "command interface" that will allow the Test Executor to communicate with external components in a generic way.
This "command interface" is intended to be generic to fit any functionality need.
Note that this capability is still to be evaluated if its implementation complexity fits the the RTEMS SMP project budget.
* Load and execute RTEMS executables to the target platform.
* Get and store the output (log) from the running executable.
* Wait for end of execution of the RTEMS executable.
This includes both cases of test termination (Successfully/Unsuccessfully) and to stop execution with a timeout mechanism.
Currently RTEMS Tester offers part of these functionalities.
An evaluation in undergoing to assess how RTEMS Tester fits these capabilities and the possibility to add the missing features.
The functionalities described above will be subject to validation to make sure that the Test Executor is suitable to pre qualify RTEMS for critical missions.
In the scope of RTEMS SMP project, the following specific functionalities are required from the above generic functionality:
* Send a board reset command (to allow to set the board to boot-up conditions).
* Load and execute tests in the tsim-leon2, tsim-leon3, gr712rc and gr740 targets.
* Use an auxiliary PC to test MIL-STD-1553 interface.
The need for this PC is due that the gr712rc and gr740 targets only support a single MIL-STD-1553 interface, which makes impossible loopback tests.
This auxiliary PC will contain a MIL-STD-1553 interface and will run an application which will translate the "command interface" messages into command to this auxiliary hardware.
Note that this activity depends on the assessment of the possibility to have this command interface.
* Receive test output either by UART or the standard output.
=== Test Report Analyser and Report Generator ===
The Test Analyser will receive the reports produced by the Test Executor, which will follow the `Test Framework <https://lists.rtems.org/pipermail/devel/2019-March/025178.html>`_ structure and assess the status of the tests.
The possible status for each test are presented `here <https://docs.rtems.org/branches/master/user/testing/tests.html#test-controls>`_.
For each test report, the Test Report Analyser will scan the requirements targeted by the test and, depending on the result of the test, will set the requirement as OK or Not OK.
Note also the result of a requirement tested by several tests will be the logical conjunction of the result of the associated tests (that is, a requirement is considered passed only if all the tests result is the expected for that requirement).
The Report Generator will gather the information from the Test Report Analyser and will also check for `other validation <https://docs.rtems.org/branches/master/eng/req-eng.html#requirement-validation>`_ performed to the requirements.
These validation items are written in doorstop format for each requirement, containing the result of the validation activity performed to the requirement, including the final assessment (OK or Not OK, as for testing and respective justification.
The Report Generator will read each of these validattion and, as for the requirements validated by test, set the requirement as OK or Not OK, depending on the result of the validation.
=== Test Plan Generator ===
The Test Plan will read the following information written in doorstop format:
* `Test Suite Specifications <https://docs.rtems.org/branches/master/eng/req-eng.html#test-suite>`_, which will contain the testsuite general description.
* `Test Procedure Specifications <https://docs.rtems.org/branches/master/eng/req-eng.html#test-procedure>`_, which will contain the set-up for each test configuration.
* `Test Case Specifications <https://docs.rtems.org/branches/master/eng/req-eng.html#test-case>`_, which will contain the steps for each test.
The information present in the above specifications will be printed in the Test Plan document, which will contain all the tests specifications and the necessary set-ups in order to run the complete test suite for RTEMS SMP.
Note that the contents of the above specifications are still to be refined.
The Test Procedure Specifications may also have the following additional information:
* configurations - software configurations in which the test case shall be executed (eg: coverage flag set)
The Test Case Specifications may also have the following additional information:
* title - title of the validation test case (summary description)
* criteria - the criteria to decide whether the test has passed or failed
* environment - the exact configuration and the set of the facility used to execute the test case and the configuration of the software utilized to support the test conduction
* constraints - any special constraints on the used test procedures
* dependencies - list of all test cases that must be executed before this test case
* type - type of test (e.g. Functional, Performance, etc)
* test procedure - list of all test procedures in which the test case should be executed in (see next paragraph's explanation)
* reset - boolean value to indicate a power reset to the target board should be made before running the test (some tests may require boot up conditions)
* auxiliary software - indicates the auxiliary software test code that should be run in auxiliary PC (if needed).
* timeout - timeout of the test case
* executable - the executable file name where the test case belongs to (a single test executable can contain several test cases)
=== Code Coverage Analysis ===
The C code coverage will be performed by using the GCOV to determine the C RTEMS code exercised by each test.
The coverage information will be put in the test log, after the end of test marker (in the end of the test).
Note that this is done automatically by the instrumented GCOV code, added to a test executable, which means that there is no interference of the Test Executor itself in gathering coverage (an executable with coverage instrumented code is seen as a normal executable, which means that the Test Executor is "coverage agnostic").
This GCOV information at each executable will be interpreted by the QualificationManager tool and the coverage results of all tests will be gathered to obtain the overall RTEMS source code coverage.
Note also, for user reference, the LCOV tool will be used to transform the information from GCOV to human readable format: source code files, with embedded coverage information.
The assembly coverage approach is not yet fully defined.
One of the approachs could be to use simulators and/or hardware with a trace unit and do the following:
* execute the tests that exercise specific assembly code.
* trace the assembly code execution (insert breakpoint in the first assembly instruction and trace untill the last asssembly instruction).
* combine the coverage of all tests to get the overall assembly coverage.
=== Static Code Analysis ===
The static code analysis is an open topic (see https://lists.rtems.org/pipermail/devel/2019-July/026805.html and https://lists.rtems.org/pipermail/devel/2019-July/026796.html).
The currently proposed approach is to use Coverity tool, with an European Space Agency license, (currently RTEMS community uses the free version, `Coverity Scan <https://scan.coverity.com/projects/rtems>`_), which supports `these <https://www.synopsys.com/content/dam/synopsys/sig-assets/datasheets/coverity-misra-standards-ds-ul.pdf>`_ rules.
Alternatively, the project could consider to use open-source tools for the static analysis.
The open-source tools that were considered as a possibility were `Clang Static Analyzer <https://clang-analyzer.llvm.org/>`_ and `Cppcheck <http://cppcheck.sourceforge.net/>`_.
Both of these tools are active projects.
The rules covered by the Clang Static Analyzer are available `here <https://clang.llvm.org/docs/ClangStaticAnalyzer.html>`_.
The rules covered by the Cppcheck are available `here <https://sourceforge.net/p/cppcheck/wiki/ListOfChecks/>`_ and the Cppcheck compliance with MISRA C 2012 rules are available `here <http://cppcheck.sourceforge.net/misra.php>`_.
The results obtained by the chosen tools will be read by the Qualification Manager and presented in the Software Verification Report document.
In addition, the justifications for the violations, written directly inside the source code, will be also read by the tool and will be presented in the Software Verification Report document.
-----Original Message-----
From: devel [mailto:devel-bounces at rtems.org] On Behalf Of Sebastian Huber
Sent: terça-feira, 11 de fevereiro de 2020 10:29
To: rtems-devel at rtems.org
Subject: Tool Roadmap for the RTEMS Pre-Qualification
Hello,
this email tries to give an overview of the tool roadmap for the RTEMS
pre-qualification activity and things to decide for the RTEMS Project.
The tools used for the RTEMS pre-qualification will be command line
tools only.
We will not use GUIs. New tools will be written in Python. The aim is to
reuse
and improve existing tools of the RTEMS Project. The tools will be
configured
via command line options and configuration files. The tool inputs will be
command line options, configuration files, source files, output from other
tools, and bug tracker databases.
The tools fall roughly in three categories. Firstly, there will be
compiler-like tools which generate output files from input files. For
example,
one tool could generate document-specific glossaries from a project-wide
glossary. Secondly, there will be client/server tools. For example,
there could
be a tool which receives test programs, runs them on a target system and
sends
back the result. Thirdly, there will be builder tools. For example, the
creation of an RTEMS pre-qualification data package for a particular target
which can be handed over to an end user.
=== Important Decisions to Make ===
* Where to place the specification items?
* What should be in the specification (master data set)?
* Which content is generated?
* Which generated content is version controlled?
* How is the content generation integrated in the build system?
* How is the Python development organized?
=== Repository Overview ===
We have currently four main repositories in the RTEMS Project:
* rtems
* rtems-tools
* rtems-docs
* rtems-source-builder (RSB)
The RSB is a self-contained component and we will use it as is for the
pre-qualification activity.
There are dependencies between the rtems, rtems-tools, and rtems-docs
repositories. Inconsistencies must be currently resolved manually
without any
tool support.
For example, we have documentation of API functions in Doxygen and the
Classic API Guide:
https://docs.rtems.org/doxygen/branches/master/group__ClassicTasks.html#gabffda1c2301962f0ae5af042ac0bba62
https://docs.rtems.org/branches/master/c-user/task_manager.html#task-create-create-a-task
One goal of the pre-qualification activity is to introduce a master data set
(specification) and use tools to generate content in the right format
from it.
For example, currently a human needs to edit
https://git.rtems.org/rtems/tree/cpukit/include/rtems/rtems/tasks.h
and
https://git.rtems.org/rtems-docs/tree/c-user/task_manager.rst
to declare and document the Classic Tasks API. This should be replaced by an
API specification which includes documentation elements and a generator tool
which produces the header file with Doxygen markup and the reST files
for the
Classic API Guide. Specifically for the pre-qualification activity, the API
specification can be used to also generate an interface specification
document
(Interface Control Document in ECSS terms).
For the generated files, we have two options:
1. They are version controlled. Specification changes and the generated
changes
are sent for review to the mailing list and then checked in or not. This
means a normal user of the repository has no need to install the
generator
tools. Also in case the generator tools stop to work, we are back
to the
current situation (with a dead specification in the repository).
2. They are not version controlled and the build system generates them on
demand.
=== Location of the Specification ===
This seems to be a hot topic. The specification could be located in the
rtems,
the rtems-docs, or a separate repository. The new build system is based on
specification items which are located in the "spec/build" directory in the
RTEMS sources. It is desirable to not split up the specification.
We could split up the specification and add specification items more
related to
the documentation to the rtems-docs repository. This may end up in
discussions
if a particular item goes into rtems or rtems-docs. I don't think this helps
and makes things easier.
We could add a new repository which contains the specification and the other
repositories as Git submodules along with tools and scripts and whatever. I
think we already have more than enough repositories in the RTEMS Project
and we
should only introduce new repositories then there is a real need. However, I
also acknowledge that the impact of the pre-qualification activity should be
manageable. Splitting up the specification into a build related part
which is
contained in the RTEMS sources repository and the rest could be a way
forward
to get started. Thanks to Chris for bringing up this approach.
What do you think about this:
1. The new build system will remain as is and uses the build specification
items located in "spec/build" in the RTEMS sources.
2. We add a new empty rtems-qual repository.
2.1. In this repository we add a "spec" directory for the non-build
specification items.
2.2. In this repository we add all the generator tools.
2.3. In this repository we add things closely related to pre-qualification.
2.4. In this repository we add Git submodules for the other RTEMS
repositories
touched by the generator tools. Changes in generated files in the
standard
RTEMS repositories go through the normal patch review process.
2.5. This repository may use a simplified review policy during the initial
pre-qualification activity.
Once the pre-qualification activity produced in a mature and usable
infrastructure we can re-evaluate the repository organization and the
location
of the specification.
=== Python Development ===
The tool development in Python for the pre-qualification is a team work
activity. Therefore we should introduce a coding standard, automatic code
formatters (black, yapf), static analysis tools (mypy, pylint, flake8),
documentation checkers (pydocstyle), and unit/integration tests
(unittest and
unittest.mock modules). The mentioned Python development tools are just
examples. There use and configuration should be discussed on the RTEMS
mailing
list.
If we place the specification along with corresponding tools into a separate
repository we can use it to set up a prototype Python development work flow.
=== Build System Integration ===
This section is only relevant if the specification is not located in a
dedicated repository.
It is tedious to know which tools, input files, and output files are
used in a
particular repository and how they are invoked. In each repository, a build
system is present. The build system can be used to update the generated
content
of a particular repository on demand (e.g. if the specification
changed). For
example, we could add an --enable-maintainer-mode option to the waf
configure
command.
./waf configure --enable-maintainer-mode --rtems-tools=X --rtems-spec=Y
This could check that the tools and specification are available and enable
rules to re-generate content based on changes in the specification. The
tools
could report a version and the build system could check that the right tool
version is used to avoid a re-generation with the wrong tools.
Having the tool configuration, invocation, and dependencies in the build
system
is also an accurate documentation how the things are set up and makes sure
everyone is using them in a defined way.
=== Howtos ===
We will add howtos for common RTEMS maintenance task, e.g. how to add a
new API
function, how to add a glossary term, etc. See also "8.6 Howtos" section
of the
new build system:
https://ftp.rtems.org/pub/rtems/people/sebh/eng.pdf
It may make sense to collect all howtos in a dedicated chapter:
https://lists.rtems.org/pipermail/devel/2020-January/056849.html
=== Specification-to-X Tool ===
The specification-to-X tool could generate the following content from the
specification (controlled by command line options or a configuration file):
* document-specific glossaries (VC)
* API documentation reST files for the Classic API Guide (VC)
* API header files with Doxygen markup (VC)
* interface specification document (Interface Control Document in ECSS
terms)
* software requirements specification document
* test plans
* configuration files for static analysis tools
* configuration file for Doxygen
Generated content which is independent of a particular target system and
has a
low rate of change should be version controlled (VC).
=== Traceability of Specification Items ===
For the traceability of specification items, please have a look at:
https://docs.rtems.org/branches/master/eng/req-eng.html#traceability-of-specification-items
There are some options available to provide a traceable history of
specification items.
Standards demand forward and backward traceability between specification
items
(e.g. requirements). This is achieved through standard Doorstop features.
The traceability between software requirements, architecture and design will
probably need some iterations to find to a good solution.
=== Additional Tools ===
José Valdez from EDISOFT will give you shortly an overview of additional
tools
which cover the following topics:
* static code analysis
* test execution
* code coverage
* test output analysis
* test report generation
* test plan generation
_______________________________________________
devel mailing list
devel at rtems.org
http://lists.rtems.org/mailman/listinfo/devel
More information about the devel
mailing list