Tool Roadmap for the RTEMS Pre-Qualification

Wed Feb 19 15:04:53 UTC 2020

Hi Jose,

Thank you for the detailed information. I have a few questions:
1) Will the output of test analysis and report generator tools be
plaintext, and will it be structured or unstructured (flat file)?
2) Do you consider the covoar tool as viable for code coverage? That
is what we have been using in the community. Joel may like to
weigh-in.
3) I want to see Clang Static Analyzer working. If there is any
possibility you all can help move us toward that direction, I think it
would be a great positive to the community. Coverity is difficult and
the license we can get freely is limited. I also have plans to work on
both Coverity and clang-analyzer improvements soon.

Gedare

On Wed, Feb 19, 2020 at 6:05 AM Jose Valdez <Jose.Valdez at edisoft.pt> wrote:
>
> Hello,
>
> Following the e-mail sent by Sebastian, please find here the missing information, regarding the tools for RTEMS pre qualification.
>
> Best regards
>
> José
>
> === Test Executor ===
>
> The Test Executor will be the software that manages the execution of the RTEMS Tests.
> It will set the necessary hardware, run the executable tests, control the execution and gather the output.
> The following capabilities will be available:
>
> * Send commands to the target platform.
>   A subset of commands will be made available, but it shall be possible to add more commands.
> * Send commands to auxiliary test programs, which run in external devices.
>   This will be done by a "command interface" that will allow the Test Executor to communicate with external components in a generic way.
>   This "command interface" is intended to be generic to fit any functionality need.
>   Note that this capability is still to be evaluated if its implementation complexity fits the the RTEMS SMP project budget.
> * Load and execute RTEMS executables to the target platform.
> * Get and store the output (log) from the running executable.
> * Wait for end of execution of the RTEMS executable.
>   This includes both cases of test termination (Successfully/Unsuccessfully) and to stop execution with a timeout mechanism.
>
> Currently RTEMS Tester offers part of these functionalities.
> An evaluation in undergoing to assess how RTEMS Tester fits these capabilities and the possibility to add the missing features.
> The functionalities described above will be subject to validation to make sure that the Test Executor is suitable to pre qualify RTEMS for critical missions.
>
> In the scope of RTEMS SMP project, the following specific functionalities are required from the above generic functionality:
>
> * Send a board reset command (to allow to set the board to boot-up conditions).
> * Load and execute tests in the tsim-leon2, tsim-leon3, gr712rc and gr740 targets.
> * Use an auxiliary PC to test MIL-STD-1553 interface.
>   The need for this PC is due that the gr712rc and gr740 targets only support a single MIL-STD-1553 interface, which makes impossible loopback tests.
>   This auxiliary PC will contain a MIL-STD-1553 interface and will run an application which will translate the "command interface" messages into command to this auxiliary hardware.
>   Note that this activity depends on the assessment of the possibility to have this command interface.
> * Receive test output either by UART or the standard output.
>
>
> === Test Report Analyser and Report Generator ===
>
> The Test Analyser will receive the reports produced by the Test Executor, which will follow the `Test Framework <https://lists.rtems.org/pipermail/devel/2019-March/025178.html>`_ structure and assess the status of the tests.
> The possible status for each test are presented  `here <https://docs.rtems.org/branches/master/user/testing/tests.html#test-controls>`_.
> For each test report, the Test Report Analyser will scan the requirements targeted by the test and, depending on the result of the test, will set the requirement as OK or Not OK.
> Note also the result of a requirement tested by several tests will be the logical conjunction of the result of the associated tests (that is, a requirement is considered passed only if all the tests result is the expected for that requirement).
>
> The Report Generator will gather the information from the Test Report Analyser and will also check for `other validation <https://docs.rtems.org/branches/master/eng/req-eng.html#requirement-validation>`_ performed to the requirements.
> These validation items are written in doorstop format for each requirement, containing the result of the validation activity performed to the requirement, including the final assessment (OK or Not OK, as for testing and respective justification.
> The Report Generator will read each of these validattion and, as for the requirements validated by test, set the requirement as OK or Not OK, depending on the result of the validation.
>
>
> === Test Plan Generator ===
>
> The Test Plan will read the following information written in doorstop format:
>
> * `Test Suite Specifications <https://docs.rtems.org/branches/master/eng/req-eng.html#test-suite>`_, which will contain the testsuite general description.
> * `Test Procedure Specifications <https://docs.rtems.org/branches/master/eng/req-eng.html#test-procedure>`_, which will contain the set-up for each test configuration.
> * `Test Case Specifications <https://docs.rtems.org/branches/master/eng/req-eng.html#test-case>`_, which will contain the steps for each test.
>
> The information present in the above specifications will be printed in the Test Plan document, which will contain all the tests specifications and the necessary set-ups in order to run the complete test suite for RTEMS SMP.
> Note that the contents of the above specifications are still to be refined.
> The Test Procedure Specifications may also have the following additional information:
>
> * configurations - software configurations in which the test case shall be executed (eg: coverage flag set)
>
> The Test Case Specifications may also have the following additional information:
>
> * title - title of the validation test case (summary description)
> * criteria - the criteria to decide whether the test has passed or failed
> * environment - the exact configuration and the set of the facility used to execute the test case and the configuration of the software utilized to support the test conduction
> * constraints - any special constraints on the used test procedures
> * dependencies - list of all test cases that must be executed before this test case
> * type - type of test (e.g. Functional, Performance, etc)
> * test procedure - list of all test procedures in which the test case should be executed in (see next paragraph's explanation)
> * reset - boolean value to indicate a power reset to the target board should be made before running the test (some tests may require boot up conditions)
> * auxiliary software - indicates the auxiliary software test code that should be run in auxiliary PC (if needed).
> * timeout - timeout of the test case
> * executable - the executable file name where the test case belongs to (a single test executable can contain several test cases)
>
>
> === Code Coverage Analysis ===
>
> The C code coverage will be performed by using the GCOV to determine the C RTEMS code exercised by each test.
> The coverage information will be put in the test log, after the end of test marker (in the end of the test).
> Note that this is done automatically by the instrumented GCOV code, added to a test executable, which means that there is no interference of the Test Executor itself in gathering coverage (an executable with coverage instrumented code is seen as a normal executable, which means that the Test Executor is "coverage agnostic").
> This GCOV information at each executable will be interpreted by the QualificationManager tool and the coverage results of all tests will be gathered to obtain the overall RTEMS source code coverage.
> Note also, for user reference, the LCOV tool will be used to transform the information from GCOV to human readable format: source code files, with embedded coverage information.
>
> The assembly coverage approach is not yet fully defined.
> One of the approachs could be to use simulators and/or hardware with a trace unit and do the following:
>
> * execute the tests that exercise specific assembly code.
> * trace the assembly code execution (insert breakpoint in the first assembly instruction and trace untill the last asssembly instruction).
> * combine the coverage of all tests to get the overall assembly coverage.
>
>
> === Static Code Analysis ===
>
> The static code analysis is an open topic (see https://lists.rtems.org/pipermail/devel/2019-July/026805.html and https://lists.rtems.org/pipermail/devel/2019-July/026796.html).
> The currently proposed approach is to use Coverity tool, with an European Space Agency license, (currently RTEMS community uses the free version, `Coverity Scan <https://scan.coverity.com/projects/rtems>`_), which supports `these <https://www.synopsys.com/content/dam/synopsys/sig-assets/datasheets/coverity-misra-standards-ds-ul.pdf>`_ rules.
>
> Alternatively, the project could consider to use open-source tools for the static analysis.
> The open-source tools that were considered as a possibility were `Clang Static Analyzer <https://clang-analyzer.llvm.org/>`_ and `Cppcheck <http://cppcheck.sourceforge.net/>`_.
> Both of these tools are active projects.
> The rules covered by the Clang Static Analyzer are available `here <https://clang.llvm.org/docs/ClangStaticAnalyzer.html>`_.
> The rules covered by the Cppcheck are available `here <https://sourceforge.net/p/cppcheck/wiki/ListOfChecks/>`_ and the Cppcheck compliance with MISRA C 2012 rules are available `here <http://cppcheck.sourceforge.net/misra.php>`_.
>
> The results obtained by the chosen tools will be read by the Qualification Manager and presented in the Software Verification Report document.
> In addition, the justifications for the violations, written directly inside the source code, will be also read by the tool and will be presented in the Software Verification Report document.
>
>
> -----Original Message-----
> From: devel [mailto:devel-bounces at rtems.org] On Behalf Of Sebastian Huber
> Sent: terça-feira, 11 de fevereiro de 2020 10:29
> To: rtems-devel at rtems.org
> Subject: Tool Roadmap for the RTEMS Pre-Qualification
>
> Hello,
>
> this email tries to give an overview of the tool roadmap for the RTEMS
> pre-qualification activity and things to decide for the RTEMS Project.
>
> The tools used for the RTEMS pre-qualification will be command line
> tools only.
> We will not use GUIs. New tools will be written in Python. The aim is to
> reuse
> and improve existing tools of the RTEMS Project. The tools will be
> configured
> via command line options and configuration files. The tool inputs will be
> command line options, configuration files, source files, output from other
> tools, and bug tracker databases.
>
> The tools fall roughly in three categories. Firstly, there will be
> compiler-like tools which generate output files from input files. For
> example,
> one tool could generate document-specific glossaries from a project-wide
> glossary. Secondly, there will be client/server tools. For example,
> there could
> be a tool which receives test programs, runs them on a target system and
> sends
> back the result. Thirdly, there will be builder tools. For example, the
> creation of an RTEMS pre-qualification data package for a particular target
> which can be handed over to an end user.
>
> === Important Decisions to Make ===
>
> * Where to place the specification items?
>
> * What should be in the specification (master data set)?
>
> * Which content is generated?
>
> * Which generated content is version controlled?
>
> * How is the content generation integrated in the build system?
>
> * How is the Python development organized?
>
> === Repository Overview ===
>
> We have currently four main repositories in the RTEMS Project:
>
> * rtems
>
> * rtems-tools
>
> * rtems-docs
>
> * rtems-source-builder (RSB)
>
> The RSB is a self-contained component and we will use it as is for the
> pre-qualification activity.
>
> There are dependencies between the rtems, rtems-tools, and rtems-docs
> repositories. Inconsistencies must be currently resolved manually
> without any
> tool support.
>
> For example, we have documentation of API functions in Doxygen and the
> Classic API Guide:
>
> https://docs.rtems.org/doxygen/branches/master/group__ClassicTasks.html#gabffda1c2301962f0ae5af042ac0bba62
>
> https://docs.rtems.org/branches/master/c-user/task_manager.html#task-create-create-a-task
>
> One goal of the pre-qualification activity is to introduce a master data set
> (specification) and use tools to generate content in the right format
> from it.
>
> For example, currently a human needs to edit
>
> https://git.rtems.org/rtems/tree/cpukit/include/rtems/rtems/tasks.h
>
> and
>
> https://git.rtems.org/rtems-docs/tree/c-user/task_manager.rst
>
> to declare and document the Classic Tasks API. This should be replaced by an
> API specification which includes documentation elements and a generator tool
> which produces the header file with Doxygen markup and the reST files
> for the
> Classic API Guide. Specifically for the pre-qualification activity, the API
> specification can be used to also generate an interface specification
> document
> (Interface Control Document in ECSS terms).
>
> For the generated files, we have two options:
>
> 1. They are version controlled. Specification changes and the generated
> changes
>     are sent for review to the mailing list and then checked in or not. This
>     means a normal user of the repository has no need to install the
> generator
>     tools.  Also in case the generator tools stop to work, we are back
> to the
>     current situation (with a dead specification in the repository).
>
> 2. They are not version controlled and the build system generates them on
>     demand.
>
> === Location of the Specification ===
>
> This seems to be a hot topic. The specification could be located in the
> rtems,
> the rtems-docs, or a separate repository. The new build system is based on
> specification items which are located in the "spec/build" directory in the
> RTEMS sources. It is desirable to not split up the specification.
>
> We could split up the specification and add specification items more
> related to
> the documentation to the rtems-docs repository. This may end up in
> discussions
> if a particular item goes into rtems or rtems-docs. I don't think this helps
> and makes things easier.
>
> We could add a new repository which contains the specification and the other
> repositories as Git submodules along with tools and scripts and whatever. I
> think we already have more than enough repositories in the RTEMS Project
> and we
> should only introduce new repositories then there is a real need. However, I
> also acknowledge that the impact of the pre-qualification activity should be
> manageable. Splitting up the specification into a build related part
> which is
> contained in the RTEMS sources repository and the rest could be a way
> forward
> to get started. Thanks to Chris for bringing up this approach.
>
> What do you think about this:
>
> 1. The new build system will remain as is and uses the build specification
>     items located in "spec/build" in the RTEMS sources.
>
> 2. We add a new empty rtems-qual repository.
>
> 2.1. In this repository we add a "spec" directory for the non-build
>       specification items.
>
> 2.2. In this repository we add all the generator tools.
>
> 2.3. In this repository we add things closely related to pre-qualification.
>
> 2.4. In this repository we add Git submodules for the other RTEMS
> repositories
>       touched by the generator tools. Changes in generated files in the
> standard
>       RTEMS repositories go through the normal patch review process.
>
> 2.5. This repository may use a simplified review policy during the initial
>       pre-qualification activity.
>
> Once the pre-qualification activity produced in a mature and usable
> infrastructure we can re-evaluate the repository organization and the
> location
> of the specification.
>
> === Python Development ===
>
> The tool development in Python for the pre-qualification is a team work
> activity. Therefore we should introduce a coding standard, automatic code
> formatters (black, yapf), static analysis tools (mypy, pylint, flake8),
> documentation checkers (pydocstyle), and unit/integration tests
> (unittest and
> unittest.mock modules). The mentioned Python development tools are just
> examples. There use and configuration should be discussed on the RTEMS
> mailing
> list.
>
> If we place the specification along with corresponding tools into a separate
> repository we can use it to set up a prototype Python development work flow.
>
> === Build System Integration ===
>
> This section is only relevant if the specification is not located in a
> dedicated repository.
>
> It is tedious to know which tools, input files, and output files are
> used in a
> particular repository and how they are invoked. In each repository, a build
> system is present. The build system can be used to update the generated
> content
> of a particular repository on demand (e.g. if the specification
> changed). For
> example, we could add an --enable-maintainer-mode option to the waf
> configure
> command.
>
> ./waf configure --enable-maintainer-mode --rtems-tools=X --rtems-spec=Y
>
> This could check that the tools and specification are available and enable
> rules to re-generate content based on changes in the specification. The
> tools
> could report a version and the build system could check that the right tool
> version is used to avoid a re-generation with the wrong tools.
>
> Having the tool configuration, invocation, and dependencies in the build
> system
> is also an accurate documentation how the things are set up and makes sure
> everyone is using them in a defined way.
>
> === Howtos ===
>
> We will add howtos for common RTEMS maintenance task, e.g. how to add a
> new API
> function, how to add a glossary term, etc. See also "8.6 Howtos" section
> of the
> new build system:
>
> https://ftp.rtems.org/pub/rtems/people/sebh/eng.pdf
>
> It may make sense to collect all howtos in a dedicated chapter:
>
> https://lists.rtems.org/pipermail/devel/2020-January/056849.html
>
> === Specification-to-X Tool ===
>
> The specification-to-X tool could generate the following content from the
> specification (controlled by command line options or a configuration file):
>
> * document-specific glossaries (VC)
>
> * API documentation reST files for the Classic API Guide (VC)
>
> * API header files with Doxygen markup (VC)
>
> * interface specification document (Interface Control Document in ECSS
> terms)
>
> * software requirements specification document
>
> * test plans
>
> * configuration files for static analysis tools
>
> * configuration file for Doxygen
>
> Generated content which is independent of a particular target system and
> has a
> low rate of change should be version controlled (VC).
>
> === Traceability of Specification Items ===
>
> For the traceability of specification items, please have a look at:
>
> https://docs.rtems.org/branches/master/eng/req-eng.html#traceability-of-specification-items
>
> There are some options available to provide a traceable history of
> specification items.
>
> Standards demand forward and backward traceability between specification
> items
> (e.g. requirements). This is achieved through standard Doorstop features.
>
> The traceability between software requirements, architecture and design will
> probably need some iterations to find to a good solution.
>
> === Additional Tools ===
>
> José Valdez from EDISOFT will give you shortly an overview of additional
> tools
> which cover the following topics:
>
> * static code analysis
>
> * test execution
>
> * code coverage
>
> * test output analysis
>
> * test report generation
>
> * test plan generation
>
> _______________________________________________
> devel mailing list
> devel at rtems.org
> http://lists.rtems.org/mailman/listinfo/devel
> _______________________________________________
> devel mailing list
> devel at rtems.org
> http://lists.rtems.org/mailman/listinfo/devel