[PATCH 0/1] c-user: Generate I/O Manager documentation

Fri Oct 2 06:53:42 UTC 2020

On 02/10/2020 02:23, Chris Johns wrote:

> On 1/10/20 3:01 pm, Sebastian Huber wrote:
>> On 01/10/2020 06:11, Chris Johns wrote:
>>> On 30/9/20 12:59 am, Sebastian Huber wrote:
>>> Where is the source of this generated documentation? I would like to review that
>>> side of things and how it is generated before I am OK with this change.
>> The documentation source for the IO Manager is here:
>>
>> https://git.rtems.org/rtems-central/tree/spec/rtems/io/if
> Thanks.
>
> This file is complicated and I am concerned this has raised the bar to a level
> that inhibits it being maintained at a community level. The YAML file seems to
> be a programming language of it's own where I need to know other pieces, for
> example:
>
>      - ${device-major-number:/name} ${.:/params[0]/name}

The specification internal referencing is a key feature. The ${...} are 
variables which reference other parts of the specification. For a 
reference, you have to provide two things which are separated by the ":".

1. The item of the attribute identified by a relative or absolute UID.

2. The path to the attribute.

What you get is a context-dependent representation of the attribute 
value. These references are flexible addition to the explicit role-based 
item links.

For example, in the API specification the references to other types used 
to specify an item enables an automatic dependency ordering in header 
files. You just say, put interface X in header file Y. What you don't 
have to say explicitly is: place interface A before interface B in 
header file C.

The references reduce the probability to have typos since the references 
are verified by the generator tools. Using references follows the DRY 
principle. You change things only in one place.

>
> I have no idea if this is YAML or a preprocessor layer you have added.
This is not YAML. It is content stored in YAML attributes. There is no 
preprocessor involved here. It is a custom format for specification 
internal references used by tools processing the specification.
> In this bit:
>
>      value: ${../../status/if/successful:/name}
>
> the double dots look like a directory so now we have relative paths in the files
> and this is fragile because any internal movement of the files breaks these
> links.

This is only fragile if you move things by hand around. Moving items 
could be done by a tool. The specification items are specified by 
specification items:

https://git.rtems.org/rtems-central/tree/spec/spec

This information is used to generate the documentation of the items:

https://docs.rtems.org/branches/master/eng/req/items.html#

It is also use to verify the format and content (to some extent) of the 
items (specverify.py). It can also be used to enable support for item 
movements.

>   We do not use those sorts of path in C code for headers for good reason
> and I think the same rational applies here. The string `status/if/successful`
> should be unique enough to be a reliable key.
The item is uniquely identified by this reference.
>
> Another issue is crosstalk between the generated documentation, tests, API and
> anything else. I feel I would need to understand those parts so I could
> confidently make changes. I would not want to unintentionally break something else.
Yes, there is crosstalk, but is intentional and one of the primary 
reasons to write the specification. We need full traceability between 
the various parts the determine RTEMS (API, documentation, requirements, 
validation tests).
>
> On the topic of making changes the only way I could contribute once this is
> merged is to alter the generated files. This is not helpful and I also think not
> the intention. It raises a couple of issues. Does the generation process check
> to see if a generated file has been touched since last generated so downstream
> changes are not lost?

The rtems-central repository contains the destination repositories as 
submodules. You can notice upstream changes in two ways.

1. You can update the submodule to the latest commit and then run the 
generator tools. If this generates modified files in the submodules, 
then you have upstream changes in generated files.

2. You update the submodules and forget to do the check in 1. You modify 
the specification and re-generate the files. Then create a patch and 
send it for review. The review will probably notice that there are 
unrelated changes.

> Second, I am concern you end up marooning yourself with
> the sole maintenance effort beyond the qual project you are on and that also
> means we are dependent on you. This does not pass the bus test.
If it is too difficult to use the specification, then I did a bad job. 
This can always happen. In the worst case, you can continue to maintain 
the sources and documentation as is and throw away the specification stuff.
>
> I understand these are API interfaces that should not or do not change but this
> is documentation and we can always make that better. For example I have wanted
> to add code fragments to the calls, a sort of copy and paste template. I could
> add this with little or no extra effort to the sphinx source however I have no
> idea where to start with these YAML files.

I don't think it is very hard to fix issues in an existing item. For 
example, lets assume you want to fix a typo in the rtems_task_construct 
documentation. My approach is do document this step by step in an how-to:

https://docs.rtems.org/branches/master/eng/req/howto.html

First you need to know that rtems-central exists. Then you have to find 
the right item. You can do this with grep:

grep -r rtems_task_construct spec
spec/testsuites/validation/profile.yml:    sc = 
rtems_task_construct(&task_config, &id);
spec/testsuites/validation-0.yml:    sc = rtems_task_construct( 
&runner_task_config, &id );
spec/rtems/task/if/construct.yml:name: rtems_task_construct
spec/rtems/task/req/ident.yml:    sc = rtems_task_construct(
spec/rtems/task/req/construct-errors.yml:        sc = 
rtems_task_construct( &valid_task_config, &id );
spec/rtems/task/req/construct-errors.yml:  ctx->status = 
rtems_task_construct( &ctx->config, ctx->id );
spec/rtems/event/req/send-receive.yml:    sc = rtems_task_construct( 
&WorkerConfig, &ctx->worker_id );

I think you can guess, that the right file is:

https://git.rtems.org/rtems-central/tree/spec/rtems/task/if/construct.yml

If you only want to fix a typo or reformulate a sentence I think it is 
easy to edit this file even if you have no idea what YAML is and how the 
specification works.

When you are done you call the generator scripts and send the patches to 
the mailing list.

If you want to add a new API interface, then there needs to be a 
discussion on the RTEMS mailing list anyway and a ticket. This can be 
used to give hints to the how-to. I am quite confident that regular 
contributors will learn how to use it.

With respect to new documentation elements such as examples. For this we 
can add a new attribute to interface items, for example this is already 
used for build specification items:

https://docs.rtems.org/branches/master/eng/req/items.html#spectypebuildlibraryitemtype

https://git.rtems.org/rtems-central/tree/spec/spec/build-library.yml#n13

>
>>> The order the directives are in has changed. The IO Manager's directives are
>>> currently in order of use by a user. What defines the order now?
>> The order is now alphabetical. I think this is a bit easier for newcomers which
>> know the alphabet but not necessarily the usual use by a user order.
> Sorry I do not agree. I agree with Joel the current order is good.
Ok, no problem. I can add an explicit ordering for the documentation.
>
>>> The formatting around 'DIRECTIVE RETURN VALUES:' needs fixing. There is vertical
>>> space after the heading but not before. It looks to me like the heading type is
>>> broken as 'CALLING SEQUENCE:' etc also appears to have the same vertical spacing.
>>>
>>> Why change 'DIRECTIVE STATUS CODES:' to 'DIRECTIVE RETURN VALUES:'?
>> This is a more general term, for example here status code doesn't fit well:
>>
>> https://docs.rtems.org/branches/master/c-user/scheduling-concepts/directives.html#scheduler-get-processor-maximum-get-processor-maximum
> OK
>
>>> I do not know as a long
>>> description moves the call seq away from the top. Do the 'NOTES:' follow the
>>> description around?
>> What do you mean with "follow the description around"?
> If the description moves do notes always appear after it?
Moving sections is not an issue since it just needs minor changes in the 
generator script. Do you want the notes as the last paragraph or after 
the description?