Requirement Document generator tool

Wed Jan 22 08:20:11 UTC 2020

On 22/01/2020 02:39, Chris Johns wrote:
> On 21/1/20 8:10 pm, Christian Mauderer wrote:
>> On 20/01/2020 22:37, Chris Johns wrote:
>>> On 14/1/20 7:18 pm, Christian Mauderer wrote:
>>>> On 13/01/2020 18:03, Joel Sherrill wrote:
>>>>> On Mon, Jan 13, 2020 at 10:04 AM Christian Mauderer
>>>>> <christian.mauderer at embedded-brains.de
>>>>> <mailto:christian.mauderer at embedded-brains.de>> wrote:
>>>>>
>>>>>     Hello Jose,
>>>>>
>>>>>     On 10/01/2020 17:33, Jose Valdez wrote:
>>>>>     > Hello,
>>>>>     >
>>>>>     > Regarding this topic and to start to define the python tools that
>>>>>     are more appropriate for the RTEMS community python developments, I
>>>>>     would like to propose the following tools (to be placed in RTEMS
>>>>>     Software Engineering manual):
>>>>>     >
>>>>>     > Python language version: 3.6 (minimum version for Doorstop 2.0.post2)
>>>>>     > Python coding style/standard - Google (as suggested by Gedare)
>>>>>     > Python documentation style - Google docstrings (to be in
>>>>>     accordance with coding style. Note it is supported by sphinx)
>>>>>     > Python test approach - unittest (seems to be the standard used by
>>>>>     python)
>>>>>     > Python static analysis - pylint (recommended by Google style, see
>>>>>     http://google.github.io/styleguide/pyguide.html) and mypy (catch
>>>>>     additional errors) Python code coverage - Coverage.py (seems to be
>>>>>     the standard used by python)
>>>>>     > Python third party packages - For now we are using: paramiko,
>>>>>     pyserial, pexpect, gitpython, but the list is expected to be
>>>>>     changed. I think here any third package is allowed, as long as the
>>>>>     license of the package complies with RTEMS project license.
>>>>>
>>>>>     To be future-proof: python3 compatibility would be nearly a must-have
>>>>>     for any new library.
>>>>>
>>>>>
>>>>> We discussed this somewhere earlier and I thought we decided that if
>>>>> it was a core tool required for a "normal user", then it would have to
>>>>> be Python 2.x and 3.x compatible. Tools for "requirements user" or
>>>>> "pre-qualification user" (pick a name for this role), can be 3.x only 
>>>>> mainly because there was so much infrastructure required for the
>>>>> "requirements user" that already needed Python 3.x.
>>>
>>> Yes this is correct. Python is used in the rtems-tools repo and any program in
>>> the rtems-tools repo needs to support Python 2 and Python 3 no matter what role
>>> it performs. The rtems-tools repo is not the same as doorstop or similar tools
>>> used to develop RTEMS, they are user tools used to develop application to run on
>>> RTEMS.
>>
>> Does that mean you don't want to have any pre-qualification tools in the
>> rtems-tools repo?
> 
> This is not what I am saying. All I am doing is listing what is currently needed
> when a tool is added to rtems-tools. If a pre-qual tool meets the requirements
> for inclusion in the tools repo then it can be added.
> 
>> If yes: Where else?
> 
> I do not know. I am not sure what these tools are to do, what interfaces they
> provide to their users, their complexity and who are the maintainers. Until
> these questions are answered I cannot say.
> 
>> I fully agree that every tool that is thought for every day use for 99%
>> of the RTEMS users should have Python 2 and 3 support. With no official
>> Python 2 support since this year I don't think that this would be a good
>> requirement for new specialized areas like pre-qualification. I think
>> you agree there?
> 
> I believe Joel has looked at Python 3 on long life operating systems like CentOS
> and it can be installed and made to work. I am sure there will be cases we have
> missed and this assumption may not hold.

I'm aware that CentOS 7 still has a python 2 as default version. I think
CentOS 8 changed that. So we will have to support python 2 at least till
EOL of CentOS 7 which is 30 June 2024.

> 
> My experience with Python 2 and 3 is most things we would use can be handled by
> future imports. With a little care programs can be portable. Doorstop has not
> been written this way so we have said Python 3 is required when developing
> RTEMS. It was never made a requirement for users.

OK.

> 
>> On the other hand I wouldn't know a better place then rtems-tools for
>> everything produced through the pre-qualification effort. So I'm not a
>> fan of saying: Everything in rtems-tools should be Python 2 + 3.
> 
> Let me put you in the position of the rtems-tools maintainer and Python 3 is
> mandated plus a number of new pre-qual tools with a variable number of packages
> are added.
> 
> How would you manage these new dependencies the RSB would encounter?
> 
> How would you manage the errors at runtime the python programs throw if a
> dependent package is not install?
> 
> How would you manage the deployment of tools as binary packages by those users
> who want to do this? We have users doing this now.
> 
> Is the increase in on-boarding complexity for new RTEMS users warranted for a
> very small and highly skill section of the user community?
> 
> The current rtems-tools model combined with the RSB currently handle these
> things. That is, the dependencies for the RSB are simple, small and known. The
> RTEMS tools once built can be copied to a new machine (the tar packaging options
> in the RSB have been present from the start).
> 
> The experience of getting every user to install the single python development
> package so GDB can be built has shown me this is hard a problem to solve and
> maintain. Distros change things all the time.
> 
> A possible solution is adding a packaging interfaces for a range of operating
> systems to the RSB. I am more than happy to discuss this offline. It is a large
> task. The RSB has been clearly documented from day one that is it not a
> packaging systems and it will never be one so adding packaging support would be
> in the form of interfacing to the host's packing system. The only issue here is
> MacOS does not have a packaging system. I know MacPorts would like to be more
> closely associated with RTEMS and I recently discovered MacPorts installs to a
> non-standard prefix so it could work. I should point out adding an interface
> will create a new set of problems, ie should the RSB exec "sudo" or "su" typoe
> calls?

OK. So from the current direction of the discussion we most likely will
get something into the rules that they should be avoided and if not
avoidable the points you just mentioned have to be discussed on a
per-package basis. If necessary a similar approach like for the few
packages like pygdb or tftpy _might_ could work (copying them into
rtems-tools). But again: A per package decission.

> 
>>>
>>> It is not clear to me what role the original python specification Jose listed is
>>> for. I will restrict my comments to rtems-tools or more generally any tool
>>> installed into the user's tools prefix for use when developing RTEMS applications.
>>>
>>> For a user focused tool:
>>>
>>> - Python is 2 and 3
>>> - No external third party packages, all support needs to be internal.
>>
>> What about stuff like pygdb (which is already used in rtems-tools)?
> 
> The source has been copied into the RTEMS tools repo and is installed everytime
> the build is installed. It does not rely on an "external" package. I have
> reviewed each package and considered the license, size and complexity of the code.
> 
> I should mention we have allowed the llvm DWARF support to be an external
> package. Before allowing this I added building llvm to the RSB to cover the case
> of it not being provided on a host.

OK.

> 
>>> - Fully portable for all supported host platforms, that is Windows, MacOS,
>>> FreeBSD and Linux.
>>>
>>> The second item is important. If user installed tools start to depend on 3rd
>>> party packages the deployment complexity for us rises and the process becomes
>>> complex.
>>>
>>> While on the topic of 3rd party packages I do not agree with Jose's view any
>>> package can be used if the license it suitable. Package complexity, it's
>>> dependencies and portability need to be considered, for example gitpython uses
>>> gitdb and this is python code to directly read and write a git repo. I would not
>>> like any of our repo's being modified by that code so I would prefer we do not
>>> use it. If we need to extract data I prefer the solution in the rtems-tools
>>> toolkit so we should also consider using existing support first.
>>
>> OK. So rule for third party is:
>>
>> For everything that is used by 99% of our users: Basically don't use any
>> third party. 
> 
> I think the answer is somewhere in the middle. We cannot openly allow any
> package to be included but we need to make use of suitable packages if they
> exist, LLVM DWARF and pygdb. I used gitpython because it is a special example as
> it allows repo edits and I am opposed to that happening.
> 
>> Exceptions for that user group have to be discussed (and
>> are most likely rejected) on the mailing list.
> 
> I would prefer you do not prejudge answers to questions that have not been asked. :)

Jose is currently working on writing up a suggestion for some rules for
python based tools that will be written during the pre-qualification
efforts. The list that started this thread was based on a similar
slightly less final list from Sebastian. These rules should end up in
the "RTEMS Software Engineering Manual". So if there are rules like "it
has to be discussed on the list how exceptions for very special cases
can be integrated" it should be in the manual too.

But you are right that I shouldn't have said that they are most likely
rejected. It was a bit mean and definitively shouldn't be in a manual.
Sorry for that.

> 
>> For special use cases:
>> - Should be avoided if a build-in can be used instead.
>> - Only if there isn't another package for the same purpose already in use.
>> - Open source license that is not putting restrictions on our code (like
>> a code generator that would put the generated code under GPL).
>> - Portable to all mayor host systems (for RTEMS that's Windows, Linux,
>> FreeBSD, MacOS).
>> - Libraries that touch critical infrastructure would have to discussed
>> thoroughly with the community before it can be added.>
>> About right?
> 
> Yeah this is a good summary of what we need.

OK. Thanks.

> 
> Chris
> 

Best regards

Christian
-- 
--------------------------------------------
embedded brains GmbH
Herr Christian Mauderer
Dornierstr. 4
D-82178 Puchheim
Germany
email: christian.mauderer at embedded-brains.de
Phone: +49-89-18 94 741 - 18
Fax:   +49-89-18 94 741 - 08
PGP: Public key available on request.

Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG.