Tool Roadmap for the RTEMS Pre-Qualification

Wed Feb 12 16:19:48 UTC 2020

On Tue, Feb 11, 2020 at 3:29 AM Sebastian Huber
<sebastian.huber at embedded-brains.de> wrote:
>
> Hello,
>
> this email tries to give an overview of the tool roadmap for the RTEMS
> pre-qualification activity and things to decide for the RTEMS Project.
>
> The tools used for the RTEMS pre-qualification will be command line
> tools only.
I would like to see an explicit intent for tools to work on specific
command lines of our common development platforms. At least, they
should work in Ubuntu LTS/Centos/SUSE/MacOS default shells (bash).

Whether they need to work in Windows is debatable,  I don't know any
open-source RTEMS developers that work primarily in Windows. The
command line there is a moving target--with the Linux subsystem, it is
more or less possible to run any scripts that work in Ubuntu under
Windows. It would be nice to define a path that works with Cygwin,
simply from an open-source ecosystem perspective.

> We will not use GUIs. New tools will be written in Python. The aim is to
> reuse
> and improve existing tools of the RTEMS Project. The tools will be
> configured
> via command line options and configuration files. The tool inputs will be
> command line options, configuration files, source files, output from other
> tools, and bug tracker databases.
>
> The tools fall roughly in three categories. Firstly, there will be
> compiler-like tools which generate output files from input files. For
> example,
> one tool could generate document-specific glossaries from a project-wide
> glossary. Secondly, there will be client/server tools. For example,
> there could
> be a tool which receives test programs, runs them on a target system and
> sends
> back the result. Thirdly, there will be builder tools. For example, the
> creation of an RTEMS pre-qualification data package for a particular target
> which can be handed over to an end user.
>
> === Important Decisions to Make ===
>
> * Where to place the specification items?
>
> * What should be in the specification (master data set)?
>
> * Which content is generated?
>
> * Which generated content is version controlled?
>
> * How is the content generation integrated in the build system?
>
> * How is the Python development organized?
>
> === Repository Overview ===
>
> We have currently four main repositories in the RTEMS Project:
>
> * rtems
>
> * rtems-tools
>
> * rtems-docs
>
> * rtems-source-builder (RSB)
>
> The RSB is a self-contained component and we will use it as is for the
> pre-qualification activity.
>
> There are dependencies between the rtems, rtems-tools, and rtems-docs
> repositories. Inconsistencies must be currently resolved manually
> without any
> tool support.
>
> For example, we have documentation of API functions in Doxygen and the
> Classic API Guide:
>
> https://docs.rtems.org/doxygen/branches/master/group__ClassicTasks.html#gabffda1c2301962f0ae5af042ac0bba62
>
> https://docs.rtems.org/branches/master/c-user/task_manager.html#task-create-create-a-task
>
> One goal of the pre-qualification activity is to introduce a master data set
> (specification) and use tools to generate content in the right format
> from it.
>
> For example, currently a human needs to edit
>
> https://git.rtems.org/rtems/tree/cpukit/include/rtems/rtems/tasks.h
>
> and
>
> https://git.rtems.org/rtems-docs/tree/c-user/task_manager.rst
>
> to declare and document the Classic Tasks API. This should be replaced by an
> API specification which includes documentation elements and a generator tool
> which produces the header file with Doxygen markup and the reST files
> for the
> Classic API Guide. Specifically for the pre-qualification activity, the API
> specification can be used to also generate an interface specification
> document
> (Interface Control Document in ECSS terms).
>
> For the generated files, we have two options:
>
> 1. They are version controlled. Specification changes and the generated
> changes
>     are sent for review to the mailing list and then checked in or not. This
>     means a normal user of the repository has no need to install the
> generator
>     tools.  Also in case the generator tools stop to work, we are back
> to the
>     current situation (with a dead specification in the repository).
>
> 2. They are not version controlled and the build system generates them on
>     demand.
>
> === Location of the Specification ===
>
> This seems to be a hot topic. The specification could be located in the
> rtems,
> the rtems-docs, or a separate repository. The new build system is based on
> specification items which are located in the "spec/build" directory in the
> RTEMS sources. It is desirable to not split up the specification.
>
> We could split up the specification and add specification items more
> related to
> the documentation to the rtems-docs repository. This may end up in
> discussions
> if a particular item goes into rtems or rtems-docs. I don't think this helps
> and makes things easier.
>
> We could add a new repository which contains the specification and the other
> repositories as Git submodules along with tools and scripts and whatever. I
> think we already have more than enough repositories in the RTEMS Project
> and we
> should only introduce new repositories then there is a real need. However, I
> also acknowledge that the impact of the pre-qualification activity should be
> manageable. Splitting up the specification into a build related part
> which is
> contained in the RTEMS sources repository and the rest could be a way
> forward
> to get started. Thanks to Chris for bringing up this approach.
>
> What do you think about this:
>
> 1. The new build system will remain as is and uses the build specification
>     items located in "spec/build" in the RTEMS sources.
>
That's fine with me, integrating this sooner than later is good.

> 2. We add a new empty rtems-qual repository.
>
> 2.1. In this repository we add a "spec" directory for the non-build
>       specification items.
>
> 2.2. In this repository we add all the generator tools.
>
> 2.3. In this repository we add things closely related to pre-qualification.
>
> 2.4. In this repository we add Git submodules for the other RTEMS
> repositories
>       touched by the generator tools. Changes in generated files in the
> standard
>       RTEMS repositories go through the normal patch review process.
>
> 2.5. This repository may use a simplified review policy during the initial
>       pre-qualification activity.
>
> Once the pre-qualification activity produced in a mature and usable
> infrastructure we can re-evaluate the repository organization and the
> location
> of the specification.
>
This seems OK. Basically, you propose a "draft repository" for
developing the products that hopefully can get integrated to the "4
main repositories" of RTEMS Project. It can also keep community burden
and review off the critical path of the sponsored project's timeline.
The risk is that the effort does not merge anything, but I think there
is sufficient interest and motivation that the pieces of this that
seem useful will migrate into the main repos.

An organization like this also makes a lot of sense from a
Packaging/Release standpoint.

Will it be clear in the subrepos that certain source code should not
be modified without first checking the HOWTOs and specifications? Or
is this knowledge to be understood by the development community and
enforced by the maintainers? In other words, what mechanisms will be
in place to help us avoid "breaking" traceability/pre-qualification?

> === Python Development ===
>
> The tool development in Python for the pre-qualification is a team work
> activity. Therefore we should introduce a coding standard, automatic code
> formatters (black, yapf), static analysis tools (mypy, pylint, flake8),
> documentation checkers (pydocstyle), and unit/integration tests
> (unittest and
> unittest.mock modules). The mentioned Python development tools are just
> examples. There use and configuration should be discussed on the RTEMS
> mailing
> list.
>
> If we place the specification along with corresponding tools into a separate
> repository we can use it to set up a prototype Python development work flow.
>
I think the main concern here is that the Python tools for
qualification shouldn't impose a great burden on the community
maintainers. Developing them in a separate repo will help. The
qualification activity can use a more stringent (i.e., restrictive)
set of coding standards---such as Python versions and workflow
processes---than currently in use elsewhere (RSB, rtems-test).

> === Build System Integration ===
>
> This section is only relevant if the specification is not located in a
> dedicated repository.
>
> It is tedious to know which tools, input files, and output files are
> used in a
> particular repository and how they are invoked. In each repository, a build
> system is present. The build system can be used to update the generated
> content
> of a particular repository on demand (e.g. if the specification
> changed). For
> example, we could add an --enable-maintainer-mode option to the waf
> configure
> command.
>
> ./waf configure --enable-maintainer-mode --rtems-tools=X --rtems-spec=Y
>
> This could check that the tools and specification are available and enable
> rules to re-generate content based on changes in the specification. The
> tools
> could report a version and the build system could check that the right tool
> version is used to avoid a re-generation with the wrong tools.
>
> Having the tool configuration, invocation, and dependencies in the build
> system
> is also an accurate documentation how the things are set up and makes sure
> everyone is using them in a defined way.
>
> === Howtos ===
>
> We will add howtos for common RTEMS maintenance task, e.g. how to add a
> new API
> function, how to add a glossary term, etc. See also "8.6 Howtos" section
> of the
> new build system:
>
> https://ftp.rtems.org/pub/rtems/people/sebh/eng.pdf
>
> It may make sense to collect all howtos in a dedicated chapter:
>
> https://lists.rtems.org/pipermail/devel/2020-January/056849.html
>
> === Specification-to-X Tool ===
>
> The specification-to-X tool could generate the following content from the
> specification (controlled by command line options or a configuration file):
>
> * document-specific glossaries (VC)
>
> * API documentation reST files for the Classic API Guide (VC)
>
> * API header files with Doxygen markup (VC)
>
> * interface specification document (Interface Control Document in ECSS
> terms)
>
> * software requirements specification document
>
> * test plans
>
> * configuration files for static analysis tools
>
> * configuration file for Doxygen
>
> Generated content which is independent of a particular target system and
> has a
> low rate of change should be version controlled (VC).
>
> === Traceability of Specification Items ===
>
> For the traceability of specification items, please have a look at:
>
> https://docs.rtems.org/branches/master/eng/req-eng.html#traceability-of-specification-items
>
> There are some options available to provide a traceable history of
> specification items.
>
> Standards demand forward and backward traceability between specification
> items
> (e.g. requirements). This is achieved through standard Doorstop features.
>
> The traceability between software requirements, architecture and design will
> probably need some iterations to find to a good solution.
>
> === Additional Tools ===
>
> José Valdez from EDISOFT will give you shortly an overview of additional
> tools
> which cover the following topics:
>
> * static code analysis
>
> * test execution
>
> * code coverage
>
> * test output analysis
>
> * test report generation
>

The community's current tools in these areas should also be
considered/supported. Our open-source use of Coverity, rtems-test, and
covoar come to mind. I also suspect someone will get llvm analyzers to
work in the near future, even if not as part of qual those analyzers
are important.

> * test plan generation
>
> _______________________________________________
> devel mailing list
> devel at rtems.org
> http://lists.rtems.org/mailman/listinfo/devel