Doorstop Issues

[Sebastian Huber]

On 20/04/2020 15:20, Joel Sherrill wrote:

>
>
> On Sun, Apr 19, 2020 at 10:14 AM Sebastian Huber 
> <sebastian.huber at embedded-brains.de 
> <mailto:sebastian.huber at embedded-brains.de>> wrote:
>
>     Hello,
>
>     I am about to add the next couple of hundred specification items
>     for the
>     RTEMS pre-qualification activity. Before I do this, I would like to
>     present some issues that popped up during my daily work with
>     Doorstop.
>     Doorstop is a requirements management tool which was selected for the
>     pre-qualification activity:
>
>     https://docs.rtems.org/branches/master/eng/req-eng.html#selected-tool-doorstop
>
>     I am quite happy that Jan Sommer suggested to use this tool. I has
>     some
>     pretty nice features. In particular, I think the file based
>     organization
>     in YAML format forming a directed acyclic graph is really the right
>     thing for this project. However, it turned out that it lacks the
>     required flexibility.
>
>     1. Its use case is requirements management. So it has some standard
>     attributes useful in this domain, like derived, header, level,
>     normative, ref, reviewed, and text. However, I want to use it more
>     generally for specification items and these attributes make not
>     always
>     sense. Having them in every item is just overhead and may cause
>     confusion.
>
>
> If they are documented well and their flow through the process is clear,
> there should not be huge confusion. Performance overhead maybe.
>
> I have seen project requirements that had schedule phase or HW/SW 
> associated
> with each requirement. On those projects, it seemed to me that the 
> primary
> problem was that the tooling couldn't check that parent requirement
> custom attributes had an relationship to those of the children. Schedule
> phases and HW/SW didn't match across the levels of requirements and
> there was no way to know if it was intentional or just 100s of mistakes.
>
>
>     2. The links cannot have custom attributes, e.g. role, enabled-by.
>
>
> i assume by enabled-by you are tracking to an RTEMS compile or condefs.h
> configure option.
Yes, the enabled-by stuff mostly relates to the RTEMS CPU options, e.g. 
RTEMS_SMP, RTEMS_POSIX_API, etc.
>
> The FACE Technical Standard uses the term conditional to describe 
> capabilities
> an implementation of the standard may provide. The term optional is 
> also used
> in the general requirements community but these. I recall that is what 
> POSIX
> calls them. Interestingly, neither standard uses will or may in those 
> requirements
> as the textbooks suggest but collects them under optional feature 
> categories.
>
>
>     3. Inside a document (directory) items are supposed to have a common
>     type. I would like to store at a hierarchy level also distinct
>     specializations.
>
>     4. The verification if the items is quite limited. We need
>     verification
>     with type-based rules.
>
>
> This sounds like you have found the problem I was referring to above.
> Parent and child should follow rules on custom attributes.
Yes, I think it is a must to verify basic things, e.g. proper 
relationships, the right set of attributes, proper values for the 
attributes, etc.
>
>
>     5. The UIDs in combination with the document hierarchy lead to
>     duplication, e.g. a/b/c/a-b-c-d.yml. You have the path (a/b/c)
>     also in
>     the file name (a-b-c). You cannot have relative UIDs in links (e.g.
>     ../parent-req) . The specification items may contain multiple
>     requirements, e.g. min/max attributes. There is no way to identify
>     them.
>
>
> This seems fixable.
>
>
>     I will probably stop using Doorstop and use a custom tooling. With
>     Python, this is quite easy. I already have the infrastructure for
>     this in:
>
>     https://git.rtems.org/sebh/rtems-qual.git/
>
>     I will try this out with the new build system.
>
>
> You are indeed pushing beyond what a requirements management tool
> is usually scoped to do. Which is funny because some of the commercial
> tools have capabilities not discussed so far. They are usually metrics,
> static analysis, and/or code coverage. We address two of the three with
> custom tools.
>
> I agree with Chris that you should discuss this with the Doorstop project.
> RTEMS has certainly grown with odd requirements thrown at it. It would
> be rude to not give another open source project the same opportunity.
> Of course, that may not change the end result.

https://github.com/doorstop-dev/doorstop/issues/471

>
> If we do end up beyond Doorstop's mission, then it is worth considering
> a sibling tool to do the generation. But I also know that Doorstop is not
> a huge body of Python either.

I converted the specification of the new build system using path-like 
UIDs. Python is really nice here, since you can simply use the os.path 
support. The transformation to the new format was pretty easy now that 
everything is contained in a machine readable format that ends up with a 
bunch of Python lists, dicts, and scalars.