Tool Roadmap for the RTEMS Pre-Qualification

Mon Feb 24 18:49:35 UTC 2020

Hello Gedare,

In the sequence of your last e-mail, I spent some time to putting to work again the CLANG Static Analyzer.
About one year ago, I run the CLANG Static Analyzer version 7.0.0 in my Debian 9 machine. I followed the next steps to install it:

tar xvf llvm-7.0.0.src.tar.xz
mv llvm-7.0.0.src llvm
cd llvm/tools/
tar xvf ../../cfe-7.0.0.src.tar.xz
mv cfe-7.0.0.src/ clang
cd clang/tools/
tar xvf ../../../../clang-tools-extra-7.0.0.src.tar.xz
mv clang-tools-extra-7.0.0.src/ extra
cd ../../../../
mkdir build
cd build/
cmake -G "Unix Makefiles" ../llvm
make

Meanwhile, I upgraded my machine to Debian 10 and I tried to install the CLANG Static Analyzer head version (11).
This version did not work for me and I tried (successfully) the default CLANG package provided in the system (version 7.0.1-8):

sudo apt-get install clang-format clang-tidy clang-tools clang libc++-dev libc++1 libc++abi-dev libc++abi1 libclang-dev libclang1 liblldb-dev libllvm-ocaml-dev libomp-dev libomp5 lld lldb llvm-dev llvm-runtime llvm python-clang

Note that the "build from source" CLANG installation (steps above) takes about 10 hours! If your system provides a default CLANG package, I suggest to use it, to avoid the long-time build from source.

After installing the CLANG, you need to install the rtems source builder, clone and bootstrap RTEMS (as usual).
After this you shall follow the following steps, the example here is for gr740 which is a 4-core SPARC target:

1) Change the file rtems-master/cpukit/include/rtems/score/cpustdatomic.h, by removing the "const" qualifier on the parameter "*obj" of the header of the functions _CPU_atomic_Load_uint, _CPU_atomic_Load_ulong and _CPU_atomic_Load_uintptr.

-> I cannot explain you very well the rationale of this step, but there
-> is a conflict with the "const" qualifier in these variables when
-> CLANG scans RTEMS

2) Set the C_INCLUDE_PATH environment variable: export C_INCLUDE_PATH=<YOUR_TOOLCHAIN_PATH>/sparc-rtems5/include/:<YOUR_TOOLCHAIN_PATH>/lib/gcc/sparc-rtems5/7.3.0/include/
Note that the order of the two paths matter (do not revert the order of these two paths).

-> This is for CLANG to know the path of the RTEMS included files from the source builder tools.

3) Add the toolchain binary folder to the path variable:
export PATH=<YOUR_TOOLCHAIN_PATH>/bin:${PATH}

4) In the build directory (placed alongside RTEMS) execute the following commands:

scan-build --use-cc=<YOUR_TOOLCHAIN_PATH>/bin/sparc-rtems5-gcc --use-c++=<YOUR_TOOLCHAIN_PATH>/bin/sparc-rtems5-g++ --analyzer-target=sparc-rtems-unknown ../rtems/configure --target=sparc-rtems5 --disable-posix --enable-smp --enable-cxx --disable-networking --enable-drvmgr --disable-tests --prefix=<PLACE_FOR_THE_INSTALLATION> CFLAGS="-D__CLANG_ATOMICS -D__rtems__"  --enable-rtemsbsp="gr740"

scan-build --use-cc=<YOUR_TOOLCHAIN_PATH>/bin/sparc-rtems5-gcc --use-c++=<YOUR_TOOLCHAIN_PATH>/bin/sparc-rtems5-g++ --analyzer-target=sparc-rtems-unknown make all

-> Note that each normal RTEMS step is preceded by " scan-build --use-cc=<YOUR_TOOLCHAIN_PATH>/bin/sparc-rtems5-gcc --use-c++=<YOUR_TOOLCHAIN_PATH>/bin/sparc-rtems5-g++ --analyzer-target=sparc-rtems-unknown ". If you install the CLANG Static Analyzer from the source (i.e not use the sytem one), then in the scan-build command you shall precede it with its complete path. The flags CFLAGS="-D__CLANG_ATOMICS -D__rtems__" for the scan-build to run (they are not used in a normal RTEMS compilation).

The e-mail with the results of scan-build was blocked in your server.

Tell me if you are able to build it.

For the

"It might be better to have a parseable intermediate output for the results in a well-structured format (e.g., XML, yaml, etc.). The report in sphinx makes sense, but there should be a structured output file that could be used for example by other report generators."

I agree that it might be convenient, and probably not hard to do.

José

-----Original Message-----
From: Gedare Bloom [mailto:gedare at rtems.org]
Sent: quarta-feira, 19 de fevereiro de 2020 19:03
To: Jose Valdez
Cc: sebastian huber; rtems-devel at rtems.org
Subject: Re: Tool Roadmap for the RTEMS Pre-Qualification

On Wed, Feb 19, 2020 at 8:44 AM Jose Valdez <Jose.Valdez at edisoft.pt> wrote:
>
> Hello Gedare,
>
> Thank you for your review.
>
> Please find my answers below.
>
> Best regards
>
> José
>
> -----Original Message-----
> From: Gedare Bloom [mailto:gedare at rtems.org]
> Sent: quarta-feira, 19 de fevereiro de 2020 15:05
> To: Jose Valdez
> Cc: sebastian huber; rtems-devel at rtems.org
> Subject: Re: Tool Roadmap for the RTEMS Pre-Qualification
>
> Hi Jose,
>
> Thank you for the detailed information. I have a few questions:
> 1) Will the output of test analysis and report generator tools be
> plaintext, and will it be structured or unstructured (flat file)?
>
>
> [Jose] The final output will be in sphinx (rst) format.
>

It might be better to have a parseable intermediate output for the results in a well-structured format (e.g., XML, yaml, etc.). The report in sphinx makes sense, but there should be a structured output file that could be used for example by other report generators.

> 2) Do you consider the covoar tool as viable for code coverage? That
> is what we have been using in the community. Joel may like to
> weigh-in.
>
>
> [Jose] We don't know yet. Since we are currently analyzing the RTEMS Tester, we will also analyze the covoar and comeback with the conclusions.
> If the covoar presents good results I don't see why to not use it.
>
Great.

> 3) I want to see Clang Static Analyzer working. If there is any
> possibility you all can help move us toward that direction, I think it
> would be a great positive to the community. Coverity is difficult and
> the license we can get freely is limited. I also have plans to work on
> both Coverity and clang-analyzer improvements soon.
>
>
> [Jose] In the past (2018)we were able to run the CLANG static analyzer on RTEMS (we had to do a minor "dirty" modification) and we get interesting results.
> Maybe I could try to do it again and send you the steps to run the CLANG Static Analyzer, if you would like.
>
Yes, I would like that very much. It is something I'd like to work into our open-source workflow if possible.

Thanks,
Gedare

> Gedare
>
> On Wed, Feb 19, 2020 at 6:05 AM Jose Valdez <Jose.Valdez at edisoft.pt> wrote:
> >
> > Hello,
> >
> > Following the e-mail sent by Sebastian, please find here the missing information, regarding the tools for RTEMS pre qualification.
> >
> > Best regards
> >
> > José
> >
> > === Test Executor ===
> >
> > The Test Executor will be the software that manages the execution of the RTEMS Tests.
> > It will set the necessary hardware, run the executable tests, control the execution and gather the output.
> > The following capabilities will be available:
> >
> > * Send commands to the target platform.
> >   A subset of commands will be made available, but it shall be possible to add more commands.
> > * Send commands to auxiliary test programs, which run in external devices.
> >   This will be done by a "command interface" that will allow the Test Executor to communicate with external components in a generic way.
> >   This "command interface" is intended to be generic to fit any functionality need.
> >   Note that this capability is still to be evaluated if its implementation complexity fits the the RTEMS SMP project budget.
> > * Load and execute RTEMS executables to the target platform.
> > * Get and store the output (log) from the running executable.
> > * Wait for end of execution of the RTEMS executable.
> >   This includes both cases of test termination (Successfully/Unsuccessfully) and to stop execution with a timeout mechanism.
> >
> > Currently RTEMS Tester offers part of these functionalities.
> > An evaluation in undergoing to assess how RTEMS Tester fits these capabilities and the possibility to add the missing features.
> > The functionalities described above will be subject to validation to make sure that the Test Executor is suitable to pre qualify RTEMS for critical missions.
> >
> > In the scope of RTEMS SMP project, the following specific functionalities are required from the above generic functionality:
> >
> > * Send a board reset command (to allow to set the board to boot-up conditions).
> > * Load and execute tests in the tsim-leon2, tsim-leon3, gr712rc and gr740 targets.
> > * Use an auxiliary PC to test MIL-STD-1553 interface.
> >   The need for this PC is due that the gr712rc and gr740 targets only support a single MIL-STD-1553 interface, which makes impossible loopback tests.
> >   This auxiliary PC will contain a MIL-STD-1553 interface and will run an application which will translate the "command interface" messages into command to this auxiliary hardware.
> >   Note that this activity depends on the assessment of the possibility to have this command interface.
> > * Receive test output either by UART or the standard output.
> >
> >
> > === Test Report Analyser and Report Generator ===
> >
> > The Test Analyser will receive the reports produced by the Test Executor, which will follow the `Test Framework <https://lists.rtems.org/pipermail/devel/2019-March/025178.html>`_ structure and assess the status of the tests.
> > The possible status for each test are presented  `here <https://docs.rtems.org/branches/master/user/testing/tests.html#test-controls>`_.
> > For each test report, the Test Report Analyser will scan the requirements targeted by the test and, depending on the result of the test, will set the requirement as OK or Not OK.
> > Note also the result of a requirement tested by several tests will be the logical conjunction of the result of the associated tests (that is, a requirement is considered passed only if all the tests result is the expected for that requirement).
> >
> > The Report Generator will gather the information from the Test Report Analyser and will also check for `other validation <https://docs.rtems.org/branches/master/eng/req-eng.html#requirement-validation>`_ performed to the requirements.
> > These validation items are written in doorstop format for each requirement, containing the result of the validation activity performed to the requirement, including the final assessment (OK or Not OK, as for testing and respective justification.
> > The Report Generator will read each of these validattion and, as for the requirements validated by test, set the requirement as OK or Not OK, depending on the result of the validation.
> >
> >
> > === Test Plan Generator ===
> >
> > The Test Plan will read the following information written in doorstop format:
> >
> > * `Test Suite Specifications <https://docs.rtems.org/branches/master/eng/req-eng.html#test-suite>`_, which will contain the testsuite general description.
> > * `Test Procedure Specifications <https://docs.rtems.org/branches/master/eng/req-eng.html#test-procedure>`_, which will contain the set-up for each test configuration.
> > * `Test Case Specifications <https://docs.rtems.org/branches/master/eng/req-eng.html#test-case>`_, which will contain the steps for each test.
> >
> > The information present in the above specifications will be printed in the Test Plan document, which will contain all the tests specifications and the necessary set-ups in order to run the complete test suite for RTEMS SMP.
> > Note that the contents of the above specifications are still to be refined.
> > The Test Procedure Specifications may also have the following additional information:
> >
> > * configurations - software configurations in which the test case
> > shall be executed (eg: coverage flag set)
> >
> > The Test Case Specifications may also have the following additional information:
> >
> > * title - title of the validation test case (summary description)
> > * criteria - the criteria to decide whether the test has passed or
> > failed
> > * environment - the exact configuration and the set of the facility
> > used to execute the test case and the configuration of the software
> > utilized to support the test conduction
> > * constraints - any special constraints on the used test procedures
> > * dependencies - list of all test cases that must be executed before
> > this test case
> > * type - type of test (e.g. Functional, Performance, etc)
> > * test procedure - list of all test procedures in which the test
> > case should be executed in (see next paragraph's explanation)
> > * reset - boolean value to indicate a power reset to the target
> > board should be made before running the test (some tests may require
> > boot up conditions)
> > * auxiliary software - indicates the auxiliary software test code that should be run in auxiliary PC (if needed).
> > * timeout - timeout of the test case
> > * executable - the executable file name where the test case belongs
> > to (a single test executable can contain several test cases)
> >
> >
> > === Code Coverage Analysis ===
> >
> > The C code coverage will be performed by using the GCOV to determine the C RTEMS code exercised by each test.
> > The coverage information will be put in the test log, after the end of test marker (in the end of the test).
> > Note that this is done automatically by the instrumented GCOV code, added to a test executable, which means that there is no interference of the Test Executor itself in gathering coverage (an executable with coverage instrumented code is seen as a normal executable, which means that the Test Executor is "coverage agnostic").
> > This GCOV information at each executable will be interpreted by the QualificationManager tool and the coverage results of all tests will be gathered to obtain the overall RTEMS source code coverage.
> > Note also, for user reference, the LCOV tool will be used to transform the information from GCOV to human readable format: source code files, with embedded coverage information.
> >
> > The assembly coverage approach is not yet fully defined.
> > One of the approachs could be to use simulators and/or hardware with a trace unit and do the following:
> >
> > * execute the tests that exercise specific assembly code.
> > * trace the assembly code execution (insert breakpoint in the first assembly instruction and trace untill the last asssembly instruction).
> > * combine the coverage of all tests to get the overall assembly coverage.
> >
> >
> > === Static Code Analysis ===
> >
> > The static code analysis is an open topic (see https://lists.rtems.org/pipermail/devel/2019-July/026805.html and https://lists.rtems.org/pipermail/devel/2019-July/026796.html).
> > The currently proposed approach is to use Coverity tool, with an European Space Agency license, (currently RTEMS community uses the free version, `Coverity Scan <https://scan.coverity.com/projects/rtems>`_), which supports `these <https://www.synopsys.com/content/dam/synopsys/sig-assets/datasheets/coverity-misra-standards-ds-ul.pdf>`_ rules.
> >
> > Alternatively, the project could consider to use open-source tools for the static analysis.
> > The open-source tools that were considered as a possibility were `Clang Static Analyzer <https://clang-analyzer.llvm.org/>`_ and `Cppcheck <http://cppcheck.sourceforge.net/>`_.
> > Both of these tools are active projects.
> > The rules covered by the Clang Static Analyzer are available `here <https://clang.llvm.org/docs/ClangStaticAnalyzer.html>`_.
> > The rules covered by the Cppcheck are available `here <https://sourceforge.net/p/cppcheck/wiki/ListOfChecks/>`_ and the Cppcheck compliance with MISRA C 2012 rules are available `here <http://cppcheck.sourceforge.net/misra.php>`_.
> >
> > The results obtained by the chosen tools will be read by the Qualification Manager and presented in the Software Verification Report document.
> > In addition, the justifications for the violations, written directly inside the source code, will be also read by the tool and will be presented in the Software Verification Report document.
> >
> >
> > -----Original Message-----
> > From: devel [mailto:devel-bounces at rtems.org] On Behalf Of Sebastian
> > Huber
> > Sent: terça-feira, 11 de fevereiro de 2020 10:29
> > To: rtems-devel at rtems.org
> > Subject: Tool Roadmap for the RTEMS Pre-Qualification
> >
> > Hello,
> >
> > this email tries to give an overview of the tool roadmap for the
> > RTEMS pre-qualification activity and things to decide for the RTEMS Project.
> >
> > The tools used for the RTEMS pre-qualification will be command line
> > tools only.
> > We will not use GUIs. New tools will be written in Python. The aim
> > is to reuse and improve existing tools of the RTEMS Project. The
> > tools will be configured via command line options and configuration
> > files. The tool inputs will be command line options, configuration
> > files, source files, output from other tools, and bug tracker
> > databases.
> >
> > The tools fall roughly in three categories. Firstly, there will be
> > compiler-like tools which generate output files from input files.
> > For example, one tool could generate document-specific glossaries
> > from a project-wide glossary. Secondly, there will be client/server
> > tools. For example, there could be a tool which receives test
> > programs, runs them on a target system and sends back the result.
> > Thirdly, there will be builder tools. For example, the creation of
> > an RTEMS pre-qualification data package for a particular target
> > which can be handed over to an end user.
> >
> > === Important Decisions to Make ===
> >
> > * Where to place the specification items?
> >
> > * What should be in the specification (master data set)?
> >
> > * Which content is generated?
> >
> > * Which generated content is version controlled?
> >
> > * How is the content generation integrated in the build system?
> >
> > * How is the Python development organized?
> >
> > === Repository Overview ===
> >
> > We have currently four main repositories in the RTEMS Project:
> >
> > * rtems
> >
> > * rtems-tools
> >
> > * rtems-docs
> >
> > * rtems-source-builder (RSB)
> >
> > The RSB is a self-contained component and we will use it as is for
> > the pre-qualification activity.
> >
> > There are dependencies between the rtems, rtems-tools, and
> > rtems-docs repositories. Inconsistencies must be currently resolved
> > manually without any tool support.
> >
> > For example, we have documentation of API functions in Doxygen and
> > the Classic API Guide:
> >
> > https://docs.rtems.org/doxygen/branches/master/group__ClassicTasks.h
> > tml#gabffda1c2301962f0ae5af042ac0bba62
> >
> > https://docs.rtems.org/branches/master/c-user/task_manager.html#task
> > -create-create-a-task
> >
> > One goal of the pre-qualification activity is to introduce a master
> > data set
> > (specification) and use tools to generate content in the right
> > format from it.
> >
> > For example, currently a human needs to edit
> >
> > https://git.rtems.org/rtems/tree/cpukit/include/rtems/rtems/tasks.h
> >
> > and
> >
> > https://git.rtems.org/rtems-docs/tree/c-user/task_manager.rst
> >
> > to declare and document the Classic Tasks API. This should be
> > replaced by an API specification which includes documentation
> > elements and a generator tool which produces the header file with
> > Doxygen markup and the reST files for the Classic API Guide.
> > Specifically for the pre-qualification activity, the API
> > specification can be used to also generate an interface
> > specification document (Interface Control Document in ECSS terms).
> >
> > For the generated files, we have two options:
> >
> > 1. They are version controlled. Specification changes and the
> > generated changes
> >     are sent for review to the mailing list and then checked in or not. This
> >     means a normal user of the repository has no need to install the
> > generator
> >     tools.  Also in case the generator tools stop to work, we are
> > back to the
> >     current situation (with a dead specification in the repository).
> >
> > 2. They are not version controlled and the build system generates them on
> >     demand.
> >
> > === Location of the Specification ===
> >
> > This seems to be a hot topic. The specification could be located in
> > the rtems, the rtems-docs, or a separate repository. The new build
> > system is based on specification items which are located in the
> > "spec/build" directory in the RTEMS sources. It is desirable to not
> > split up the specification.
> >
> > We could split up the specification and add specification items more
> > related to the documentation to the rtems-docs repository. This may
> > end up in discussions if a particular item goes into rtems or
> > rtems-docs. I don't think this helps and makes things easier.
> >
> > We could add a new repository which contains the specification and
> > the other repositories as Git submodules along with tools and
> > scripts and whatever. I think we already have more than enough
> > repositories in the RTEMS Project and we should only introduce new
> > repositories then there is a real need. However, I also acknowledge
> > that the impact of the pre-qualification activity should be
> > manageable. Splitting up the specification into a build related part
> > which is contained in the RTEMS sources repository and the rest
> > could be a way forward to get started. Thanks to Chris for bringing
> > up this approach.
> >
> > What do you think about this:
> >
> > 1. The new build system will remain as is and uses the build specification
> >     items located in "spec/build" in the RTEMS sources.
> >
> > 2. We add a new empty rtems-qual repository.
> >
> > 2.1. In this repository we add a "spec" directory for the non-build
> >       specification items.
> >
> > 2.2. In this repository we add all the generator tools.
> >
> > 2.3. In this repository we add things closely related to pre-qualification.
> >
> > 2.4. In this repository we add Git submodules for the other RTEMS
> > repositories
> >       touched by the generator tools. Changes in generated files in
> > the standard
> >       RTEMS repositories go through the normal patch review process.
> >
> > 2.5. This repository may use a simplified review policy during the initial
> >       pre-qualification activity.
> >
> > Once the pre-qualification activity produced in a mature and usable
> > infrastructure we can re-evaluate the repository organization and
> > the location of the specification.
> >
> > === Python Development ===
> >
> > The tool development in Python for the pre-qualification is a team
> > work activity. Therefore we should introduce a coding standard,
> > automatic code formatters (black, yapf), static analysis tools
> > (mypy, pylint, flake8), documentation checkers (pydocstyle), and
> > unit/integration tests (unittest and unittest.mock modules). The
> > mentioned Python development tools are just examples. There use and
> > configuration should be discussed on the RTEMS mailing list.
> >
> > If we place the specification along with corresponding tools into a
> > separate repository we can use it to set up a prototype Python development work flow.
> >
> > === Build System Integration ===
> >
> > This section is only relevant if the specification is not located in
> > a dedicated repository.
> >
> > It is tedious to know which tools, input files, and output files are
> > used in a particular repository and how they are invoked. In each
> > repository, a build system is present. The build system can be used
> > to update the generated content of a particular repository on demand
> > (e.g. if the specification changed). For example, we could add an
> > --enable-maintainer-mode option to the waf configure command.
> >
> > ./waf configure --enable-maintainer-mode --rtems-tools=X
> > --rtems-spec=Y
> >
> > This could check that the tools and specification are available and
> > enable rules to re-generate content based on changes in the
> > specification. The tools could report a version and the build system
> > could check that the right tool version is used to avoid a
> > re-generation with the wrong tools.
> >
> > Having the tool configuration, invocation, and dependencies in the
> > build system is also an accurate documentation how the things are
> > set up and makes sure everyone is using them in a defined way.
> >
> > === Howtos ===
> >
> > We will add howtos for common RTEMS maintenance task, e.g. how to
> > add a new API function, how to add a glossary term, etc. See also
> > "8.6 Howtos" section of the new build system:
> >
> > https://ftp.rtems.org/pub/rtems/people/sebh/eng.pdf
> >
> > It may make sense to collect all howtos in a dedicated chapter:
> >
> > https://lists.rtems.org/pipermail/devel/2020-January/056849.html
> >
> > === Specification-to-X Tool ===
> >
> > The specification-to-X tool could generate the following content
> > from the specification (controlled by command line options or a configuration file):
> >
> > * document-specific glossaries (VC)
> >
> > * API documentation reST files for the Classic API Guide (VC)
> >
> > * API header files with Doxygen markup (VC)
> >
> > * interface specification document (Interface Control Document in
> > ECSS
> > terms)
> >
> > * software requirements specification document
> >
> > * test plans
> >
> > * configuration files for static analysis tools
> >
> > * configuration file for Doxygen
> >
> > Generated content which is independent of a particular target system
> > and has a low rate of change should be version controlled (VC).
> >
> > === Traceability of Specification Items ===
> >
> > For the traceability of specification items, please have a look at:
> >
> > https://docs.rtems.org/branches/master/eng/req-eng.html#traceability
> > -of-specification-items
> >
> > There are some options available to provide a traceable history of
> > specification items.
> >
> > Standards demand forward and backward traceability between
> > specification items (e.g. requirements). This is achieved through
> > standard Doorstop features.
> >
> > The traceability between software requirements, architecture and
> > design will probably need some iterations to find to a good solution.
> >
> > === Additional Tools ===
> >
> > José Valdez from EDISOFT will give you shortly an overview of
> > additional tools which cover the following topics:
> >
> > * static code analysis
> >
> > * test execution
> >
> > * code coverage
> >
> > * test output analysis
> >
> > * test report generation
> >
> > * test plan generation
> >
> > _______________________________________________
> > devel mailing list
> > devel at rtems.org
> > http://lists.rtems.org/mailman/listinfo/devel
> > _______________________________________________
> > devel mailing list
> > devel at rtems.org
> > http://lists.rtems.org/mailman/listinfo/devel