Change Control Board (CCB) proposal for requirements (v2)

[Sebastian Huber]

On 25/07/2019 01:17, Joel Sherrill wrote:
> 
> 
> On Wed, Jul 24, 2019 at 1:39 AM Sebastian Huber 
> <sebastian.huber at embedded-brains.de 
> <mailto:sebastian.huber at embedded-brains.de>> wrote:
> 
>     Hello,
> 
>     I work currently on a requirements engineering section for RTEMS in the
>     RTEMS Software Engineering manual:
> 
>     https://docs.rtems.org/branches/master/eng/index.html
> 
>     One topic is the Change Control Board (CCB). The Doorstop maintainer
>     was
>     not so fond of my idea to add support for digital signatures:
> 
>     https://github.com/jacebrowning/doorstop/issues/375
> 
>     So, here is a new proposal without signatures. I think this makes the
>     process a bit easier.
> 
>     Change Control Board
>     --------------------
> 
>     Working with requirements usually involves a Change Control Board
>     (:term:`CCB`).  The CCB of the RTEMS project is the
>     `RTEMS developer mailing list
>     <https://lists.rtems.org/mailman/listinfo/devel>`_.
> 
>     There are the following actors involved:
> 
>     * *RTEMS users*: Everyone using the RTEMS real-time operating system to
>     design,
>         develop and build an application on top of it.
> 
>     * *RTEMS developers*: The persons developing and maintaining RTEMS.
>     They write
>         patches to add or modify code, requirements, tests and
>     documentation.
> 
>     * *RTEMS maintainers*: They are listed in the
>         `MAINTAINERS <https://git.rtems.org/rtems/tree/MAINTAINERS>`_ file
>     and have
>         write access to the project repositories.
> 
>     .. TODO: Make a reference to the "normal patch review process".
> 
>     Adding and changing requirements follows the normal patch review
>     process.
> 
> 
> The normal process should be defined and referenced.

Yes, I propose to add it to a "Contributing" section in the user manual. 
Is this all right for you, or do you prefer the RTEMS Software 
Engineering manual?

> 
>     Reviews and comments may be submitted by anyone, but a maintainer
>     review is
>     required to approve *significant* changes.  In addition for significant
>     changes, there should be at least one reviewer with a sufficient
>     independence
>     from the author which proposes a new requirement or a change of an
>     existing
>     requirement.  Working in another company on different projects is
>     sufficiently
>     independent. 
> 
> 
> Different companies and same funding source isn't independent.

The "another company on different projects" was my attempt to rule out 
people fed by the same money pot. Do yo have a suggestion to formulate 
this more clearly?

> 
>     RTEMS maintainers do not know all the details, so they
>     trust in
>     general people with experience on a certain platform.  Sometimes no
>     review
>     comments may appear in a reasonable time frame, then an implicit
>     agreement to
>     the proposed changes is assumed.  Patches can be sent at anytime, so
>     controlling changes in RTEMS requires a permanent involvement on the
>     RTEMS
>     developer mailing list.
> 
> 
> There is a problem with independence and reasonable time frame. You are
> relying on volunteers to review requirements (or anything else) and at 
> their time
> availability mercy. I don't know how you ensure you have proper second 
> reviewers
> from the volunteer community in a reasonable time frame. Lack of commentary
> could just mean personal issues or travel. For example, I spent 3 weeks 
> of June
> traveling and/or teaching and was sick enough for two rounds of 
> antibiotics. I
> care about reviewing but I didn't review much last month. And even when I do
> review, there is a practical limit.
> 
> Some of this going to have to be spoon fed to the community to have a chance
> to get it reviewed by volunteer. Better would be paid time from 
> alternative core
> developers to speed this up.But ultimately some core reviewers will be 
> volunteer
> and slow.
> 
> A normal timeout isn't appropriate for requirements. They really deserve 
> positive
> acknowledge. A true CCB waits for a majority from a quorum.

When you get no response from maintainers and you know they are on 
holidays or otherwise not available, then a reasonable timeout is not 
within this time frame.

I think we should not put all "requirements" into one bucket. There are 
high level ones and low level ones, e.g. "If the id parameter is NULL, 
then the X directive shall return the RTEMS_Y status.". Initially, we do 
a reverse engineering of the RTEMS code base to get the requirements. 
So, the code base will probably stay as is. The work I produce in the 
next year is also reviewed by ESA staff. We have a bit of funding to 
spend on review work by community members (for example Gedare if he is 
interested).

> A technical part of the solution is be something like Phabricator which 
> tracks
> getting multiple approvals and lets you do online reviews. A tool like 
> this also
> helps with the records for code reviews and approval beyond the final 
> git log.
Thinking about alternatives to mailing list based patch review process 
is a good idea. I would use the same approach for code and requirements. 
I am not sure if it is now the right time to introduce a new process. I 
would prefer to first get started with the requirements using the 
established work flow we used in the last couple of years and in 
parallel think about how the patch work flow can be improved in general 
with modern tools.

>     For a qualification of RTEMS according to certain standards, the
>     requirements
>     may be approved by an RTEMS user. 
> 
> 
> I think you mean tailoring the requirements and code.

I mean that someone reviews all the requirements and then writes a 
report which states if the are fine to use for the project or not. 
Ideally, the users should have no need to tailor requirements and code 
of RTEMS.

> 
>     The approval by RTEMS users is not the
>     concern of the RTEMS project, however, the RTEMS project should
>     enable RTEMS
>     users to manage the approval of requirements easily.  This information
>     may be
>     also used by a independent authority which comes into play with an
>     Independent
>     Software Verification and Validation (:term:`ISVV`).  It could be
>     used to
>     select a subset of requirements, e.g. look only at the ones approved
>     by a
>     certain user. 
> 
> 
> Careful with approved. Do you mean tailoring again?
> 
>     RTEMS users should be able to reference the determinative
>     content of requirements, test procedures, test cases and justification
>     reports
>     in their own documentation.  Changes in the determinative content should
>     invalidate all references to previous versions.
> 
> 
> Determinative doesn't appear to be a word.

It passed my spell checker ;-)

What I wanted to say is that the file in which the requirement is stored 
contains multiple attributes (Doorstop attributes). Some of these 
attributes are only informative and if this content changes, the 
references are not invalidated. Other attributes contain the binding 
content, e.g. the requirement text. If this content changes, then the 
references should be invalidated. I need names for these two sets of 
requirements. What about "informative" and "normative"? The problem with 
"normative" is that this is an overloaded term. So, I started with a 
German word, picked up a fancy sounding term and ended up with 
"determinative".

-- 
Sebastian Huber, embedded brains GmbH

Address : Dornierstr. 4, D-82178 Puchheim, Germany
Phone   : +49 89 189 47 41-16
Fax     : +49 89 189 47 41-09
E-Mail  : sebastian.huber at embedded-brains.de
PGP     : Public key available on request.

Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG.