Tool Roadmap for the RTEMS Pre-Qualification

[Sebastian Huber]

On 12/02/2020 17:19, Gedare Bloom wrote:
> On Tue, Feb 11, 2020 at 3:29 AM Sebastian Huber
> <sebastian.huber at embedded-brains.de> wrote:
>> Hello,
>>
>> this email tries to give an overview of the tool roadmap for the RTEMS
>> pre-qualification activity and things to decide for the RTEMS Project.
>>
>> The tools used for the RTEMS pre-qualification will be command line
>> tools only.
> I would like to see an explicit intent for tools to work on specific
> command lines of our common development platforms. At least, they
> should work in Ubuntu LTS/Centos/SUSE/MacOS default shells (bash).
>
> Whether they need to work in Windows is debatable,  I don't know any
> open-source RTEMS developers that work primarily in Windows. The
> command line there is a moving target--with the Linux subsystem, it is
> more or less possible to run any scripts that work in Ubuntu under
> Windows. It would be nice to define a path that works with Cygwin,
> simply from an open-source ecosystem perspective.
Ok, my sentence was a bit unclear. What I meant is that the tools 
developed for the RTEMS pre-qualification will be command line tools 
only. For static analysis we may want to use Coverity for example. The 
formal methods stuff uses its own bunch of tools.

I think we have to differentiate here a bit. It is clear that the tools 
used also for daily RTEMS development should run on all our usual 
development hosts. The generator tools which read the specification and 
output header and documentation sources are an example for this. These 
tools are really simple. Apart from needing a module to read YAML files, 
they will only use the standard Python library (maybe Python 3.6+). For 
the pre-qualification we may need also complex third-party tools or web 
services which are not available on all development hosts. For these 
tools it is important to evaluate the impact on the RTEMS Project. Tool 
reports could for example result in a series of patches. The patches 
should make sense for the RTEMS Project. It is important to document 
what was used. It is also important to make the complex tools optional 
and check their availability on the host. If it is not present, you get 
a warning and reduced functionality, but you should still be able to use 
the basic tools.

> [...]
>> 2. We add a new empty rtems-qual repository.
>>
>> 2.1. In this repository we add a "spec" directory for the non-build
>>        specification items.
>>
>> 2.2. In this repository we add all the generator tools.
>>
>> 2.3. In this repository we add things closely related to pre-qualification.
>>
>> 2.4. In this repository we add Git submodules for the other RTEMS
>> repositories
>>        touched by the generator tools. Changes in generated files in the
>> standard
>>        RTEMS repositories go through the normal patch review process.
>>
>> 2.5. This repository may use a simplified review policy during the initial
>>        pre-qualification activity.
>>
>> Once the pre-qualification activity produced in a mature and usable
>> infrastructure we can re-evaluate the repository organization and the
>> location
>> of the specification.
>>
> This seems OK. Basically, you propose a "draft repository" for
> developing the products that hopefully can get integrated to the "4
> main repositories" of RTEMS Project. It can also keep community burden
> and review off the critical path of the sponsored project's timeline.
> The risk is that the effort does not merge anything, but I think there
> is sufficient interest and motivation that the pieces of this that
> seem useful will migrate into the main repos.
>
> An organization like this also makes a lot of sense from a
> Packaging/Release standpoint.
>
> Will it be clear in the subrepos that certain source code should not
> be modified without first checking the HOWTOs and specifications? Or
> is this knowledge to be understood by the development community and
> enforced by the maintainers? In other words, what mechanisms will be
> in place to help us avoid "breaking" traceability/pre-qualification?
Generated files will have a notice in the file header. However, who 
reads file headers if you want to change something in the middle? It 
will show up during patch review if something was not done properly 
provided the parties interested in the pre-qualification remain to be 
active. If not, then the normal bit rot will occur. If the stuff is 
difficult to use without a clear benefit, then it will also bit rot. It 
is our job to make it easy to use with clear benefits for the maintainers.
>
>> === Python Development ===
>>
>> The tool development in Python for the pre-qualification is a team work
>> activity. Therefore we should introduce a coding standard, automatic code
>> formatters (black, yapf), static analysis tools (mypy, pylint, flake8),
>> documentation checkers (pydocstyle), and unit/integration tests
>> (unittest and
>> unittest.mock modules). The mentioned Python development tools are just
>> examples. There use and configuration should be discussed on the RTEMS
>> mailing
>> list.
>>
>> If we place the specification along with corresponding tools into a separate
>> repository we can use it to set up a prototype Python development work flow.
>>
> I think the main concern here is that the Python tools for
> qualification shouldn't impose a great burden on the community
> maintainers. Developing them in a separate repo will help. The
> qualification activity can use a more stringent (i.e., restrictive)
> set of coding standards---such as Python versions and workflow
> processes---than currently in use elsewhere (RSB, rtems-test).
I did a bit of Python development for the Doorstop project last year. I 
am quite confident that the use of a Python development workflow with 
the aid of standard Python development tools will make the job a lot 
easier for maintainers. In Doorstop, if you make a pull request on 
Github your patch set is piped through code formatters, style checkers, 
static analysis, documentation style checkers, unit tests, integration 
tests, and code coverage analysis. The stuff is packaged and deployed in 
a test environment. If everything is all right, you see three green 
lights in the pull request. If there is one red light you know the 
maintainer will not look at this pull request.