[PATCH 0/1] Generate application configuration option documentation

Wed Jul 8 05:39:12 UTC 2020

[It seems Thunderbird misformatted the e-mail, maybe this one looks better.]

Hello Chris,

thanks for the detailed response.

On 08/07/2020 03:55, Chris Johns wrote:
> Hi Sebastian,
>
> Thank you for this patch. I am sorry but I would like to see this 
> central repo
> issue resolved before any generated files are added to any of the 
> project's repos.
>
> I understand to some level the path you are taking and moving along 
> but these
> generated files are coming from a personal repo that has changes, 
> processes and
> tools that are not being reviewed. I think it would be wise for us to 
> agree on
> what is behind all this before we agree to what is being generated.

I rescheduled some work to be able to provide this patch earlier. Richi 
Dubey asked a couple of questions with respect to the scheduler 
implementation and some of them are related to how application 
configuration options define internal scheduler data structures. So, It 
would be nice to be able to reference them in Doxygen. You can review 
this patch independent from the way it is produced.

>
> On 7/7/20 9:25 pm, Sebastian Huber wrote:
>> This patch adds the application configuration option documentation 
>> generated
>> from specification items:
>>
>> https://git.rtems.org/sebh/rtems-qual.git/tree/spec/if/acfg
> <gulp> That is a lot of files. I am not sure I am ready to accept 
> documentation
> as a lot of small files. I am happy to be persuaded otherwise.
>
> It is not clear to me if these YAML files generate documentation, 
> headers, are
> requirements or something else.

Each application configuration option and group is provided as a 
separate specification item (YAML file). The type of the specification 
items is documented here:

https://docs.rtems.org/branches/master/eng/req/items.html

https://docs.rtems.org/branches/master/eng/req/items.html#spectypeapplicationconfigurationoptionitemtype

Application configuration options are things on their own, so they have 
their own items. This enables other items to reference application 
configuration options. In the manual you have now a lot more cross 
references for example if you compare it with the RTEMS 4.11 
documentation. Items have their own history and can be reviewed 
individually. I see this as benefits.

> There is nothing I can see in them that defines
> their `role` or `roles`.

When you look at an item

https://git.rtems.org/sebh/rtems-qual.git/tree/spec/if/acfg/message-buffer-memory.yml

you see attributes like

type: interface

interface-type: appl-config-option

appl-config-option-type: integer

This defines the type of an item. I corresponds to the specification 
type refinement:

https://docs.rtems.org/branches/master/eng/req/items.html#spectyperootitemtype

https://docs.rtems.org/branches/master/eng/req/items.html#spectypeinterfaceitemtype

https://docs.rtems.org/branches/master/eng/req/items.html#spectypeapplicationconfigurationoptionitemtype

https://docs.rtems.org/branches/master/eng/req/items.html#spectypeapplicationconfigurationvalueoptionitemtype

All attributes and the value types are defined by specification item 
types. The specification item types are defined by specification items:

https://git.rtems.org/sebh/rtems-qual.git/tree/spec/spec

The type information is used to generate the documentation in the RTEMS 
Software Engineering manual and is used to verify that the items are in 
line with the type specification:

https://git.rtems.org/sebh/rtems-qual.git/tree/specverify.py

> If they do generate files how does that happen and how
> are those generated header files related to the C code they are an 
> interface
> for?

The Doxygen header file of this patch is not read by the C compiler. 
However, I would like to generate also the API header files from 
specification items. I am now able to generate <rtems.h> and all header 
files included by this header which define API elements. About 585 items 
describe the API defined by <rtems.h>:

https://git.rtems.org/sebh/rtems-qual.git/tree/spec/if/rtems

This item describes rtems_semaphore_create():

https://git.rtems.org/sebh/rtems-qual.git/tree/spec/if/rtems/sem/create.yml

It is worth to have a closer look at this item.

The interface specification items can be used to generate the header 
files with Doxygen markup, Sphinx sources for the manuals, and an 
Interface Control Document (ICD, a document required by ECSS).

> How do I keep each piece in sync?
The basic work flow would be to update an item, then call one or more 
scripts, run make or ./waf, or whatever, to generate the derived 
content. This will alter the state of Git submodules. Then you create 
patch sets from the changes in the submodules and send them for review. 
After acceptance you push the changes and update the submodules.
> A compile error seems a long way from one
> of these YAML files.
Yes, this is a bit harder to correlate. I don't think this will be a 
real issue. It is for example easy to figure out with "grep" which item 
is relevant.
>
> For example, if I was to add a YAML file to this directory I have no 
> idea where
> I look to find the naming, what roles are assumed, how it integrates 
> into the
> existing documentation, doxygen or header files? I also have no idea 
> how I would
> create a new API header file? What if I push a new header file to the 
> rtems.git
> repo with a new #include in rtems.h. Does a regenerate over write it?

Yes, this is a real issue. We have to think about how we want to 
continue with the specification stuff in general. I see clear benefits 
to generate content from a project-wide master data set to avoid the 
copy and paste stuff we have right now.

>
> It seems to me the key is the central repo and how it functions, how 
> easy or
> hard it is to learn about it and how well it will work in practice. A key
> concern is taking it from a single person repo, ie you, to a project 
> wide repo
> with concurrent updates ad lots of moving pieces. It is easy to ask these
> questions and I appreciate they are not all easy to answer but I think 
> we need
> to try.
My approach would be to document common tasks for the every day RTEMS 
maintenance:

https://docs.rtems.org/branches/master/eng/req/howto.html

You also have to consider the future rate of change. The API parts, 
application configuration options, and glossary terms for example are 
all things which don't change every week.

With respect to the code which generates the content. It is written in 
Python 3 with the new Python guidelines in mind:

https://docs.rtems.org/branches/master/eng/python-devel.html

It uses yapf for the code formatting. It uses the static analyzers 
flake8, mypy, and pylint. There are unit tests using pytest which yield 
100% branch coverage. It is not finished work. It was not reviewed, but 
I think it is a good start.

>
>> The header file is generated by the following script and module:
>>
>> https://git.rtems.org/sebh/rtems-qual.git/tree/spec2doc.py
>>
>> https://git.rtems.org/sebh/rtems-qual.git/tree/rtemsqual/applconfig.py
>>
>> The module uses currently a hack to resolve references external to 
>> specification
>> items, e.g. sections in the RTEMS Classic API Guide or URLs. I think 
>> we need
>> specialized specification items for these external references.
> I do not understand what this means?

Being able to generate Doxygen markup and Sphinx sources from the same 
data set was not a small task. We talk here about several weeks of work. 
You can't take it for granted that this works at all in an area with 
many cross references such as the application configuration options. For 
all external references I had to create a table in the source code:

https://git.rtems.org/sebh/rtems-qual.git/tree/rtemsqual/applconfig.py#n327

https://git.rtems.org/sebh/rtems-qual.git/tree/rtemsqual/applconfig.py#n381

This is not really elegant.

>
>> Sebastian Huber (1):
>> Document application configuration options
>>
>> cpukit/doxygen/appl-config.h | 4133 ++++++++++++++++++++++++++++++++++
>> 1 file changed, 4133 insertions(+)
>> create mode 100644 cpukit/doxygen/appl-config.h
> Is this information found in the user manual as well? I am struggling to
> understand the relationships and what is being presented where.

Yes, you find it here for example:

https://docs.rtems.org/branches/master/c-user/config/general.html