Requirement Document generator tool

[Chris Johns]

On 14/1/20 7:18 pm, Christian Mauderer wrote:
> On 13/01/2020 18:03, Joel Sherrill wrote:
>>
>>
>> On Mon, Jan 13, 2020 at 10:04 AM Christian Mauderer
>> <christian.mauderer at embedded-brains.de
>> <mailto:christian.mauderer at embedded-brains.de>> wrote:
>>
>>     Hello Jose,
>>
>>     On 10/01/2020 17:33, Jose Valdez wrote:
>>     > Hello,
>>     >
>>     > Regarding this topic and to start to define the python tools that
>>     are more appropriate for the RTEMS community python developments, I
>>     would like to propose the following tools (to be placed in RTEMS
>>     Software Engineering manual):
>>     >
>>     > Python language version: 3.6 (minimum version for Doorstop 2.0.post2)
>>     > Python coding style/standard - Google (as suggested by Gedare)
>>     > Python documentation style - Google docstrings (to be in
>>     accordance with coding style. Note it is supported by sphinx)
>>     > Python test approach - unittest (seems to be the standard used by
>>     python)
>>     > Python static analysis - pylint (recommended by Google style, see
>>     http://google.github.io/styleguide/pyguide.html) and mypy (catch
>>     additional errors) Python code coverage - Coverage.py (seems to be
>>     the standard used by python)
>>     > Python third party packages - For now we are using: paramiko,
>>     pyserial, pexpect, gitpython, but the list is expected to be
>>     changed. I think here any third package is allowed, as long as the
>>     license of the package complies with RTEMS project license.
>>
>>     To be future-proof: python3 compatibility would be nearly a must-have
>>     for any new library.
>>
>>
>> We discussed this somewhere earlier and I thought we decided that if
>> it was a core tool required for a "normal user", then it would have to
>> be Python 2.x and 3.x compatible. Tools for "requirements user" or
>> "pre-qualification user" (pick a name for this role), can be 3.x only 
>> mainly because there was so much infrastructure required for the
>> "requirements user" that already needed Python 3.x.

Yes this is correct. Python is used in the rtems-tools repo and any program in
the rtems-tools repo needs to support Python 2 and Python 3 no matter what role
it performs. The rtems-tools repo is not the same as doorstop or similar tools
used to develop RTEMS, they are user tools used to develop application to run on
RTEMS.

It is not clear to me what role the original python specification Jose listed is
for. I will restrict my comments to rtems-tools or more generally any tool
installed into the user's tools prefix for use when developing RTEMS applications.

For a user focused tool:

- Python is 2 and 3
- No external third party packages, all support needs to be internal.
- Fully portable for all supported host platforms, that is Windows, MacOS,
FreeBSD and Linux.

The second item is important. If user installed tools start to depend on 3rd
party packages the deployment complexity for us rises and the process becomes
complex.

While on the topic of 3rd party packages I do not agree with Jose's view any
package can be used if the license it suitable. Package complexity, it's
dependencies and portability need to be considered, for example gitpython uses
gitdb and this is python code to directly read and write a git repo. I would not
like any of our repo's being modified by that code so I would prefer we do not
use it. If we need to extract data I prefer the solution in the rtems-tools
toolkit so we should also consider using existing support first.

> 
> Yes, sorry. I used the wrong wording. What I meant was that we should
> avoid libraries that support only python 2. But you have added a lot of
> relevant details to this.
> 
>>
>> I think this discussion should also include "documentation producer"
>> because that requires Python 3.x as well.
>>
>> We just need to be conscious of the user roles that the tools are
>> supporting and justify the minimum. This gives us at least 3 categories
>> of tools with potentially different minimum hosting requirements. For
>> example, we may not care if a certain set of tools runs on Mingw.
>>
>> Obviously documented somewhere.
>>  
>>
>>
>>     I don't think that there are heavy license problems. The python packages
>>     are most likely only used in tools. So as long as they are open enough
>>     to use them on all host platforms for all development situations that
>>     shouldn't be a problem. Exception: If the package is used to generate
>>     code with a specific license. That generated code would have to be
>>     compatible with the RTEMS license.
>>
>>
>> +1 and very much non-viral and no advertising for generated code.
>>
>>
>>     >
>>     > Please tell me your opinions for each python tool and then we can
>>     start to agree on what to use. Note that once this is defined, our
>>     Qualification software will comply with the RTEMS community chosen
>>     tools.
>>
>>     If no one objects I would suggest to start documenting them in the RTEMS
>>     Software Engineering manual. A small example project in the rtems-tools
>>     repository that uses the suggested code checkers (meaning: calls them
>>     during a waf build or a special 'waf check' target) would be great too.
>>     That could be used for all future python based tools that are added to
>>     rtems-tools (which includes the whole qualification tools).
>>
>>
>> Yes. It must be enforced. Whether it is a special step or not is up for
>> discussion.
>> My first thought is that a special step is easy to skip.
>>
>> Also can "check" be done as a pre-commit hook? 
> 
> Possible. But I'm not sure. We might can try it as soon as we have it
> running in an example.

The problem I see with git hooks is the catch is too late. A patch is tested
then reviewed only for a push to fail and the review process needs to start
again or be restarted.

>> And when the style is agreed to, should existing code be auto reformatted
>> to meet it?
> 
> I think this could be migrated on a per-tool basis. If we have one
> project that is a sample for all future tools the checkers can be back
> ported to existing tools when there is time or a good opportunity.

You need to convince me a Python reformatting tool does not break existing
python code. A misaligned indent changes the logic.

Chris