[PATCH 0/1] Generate application configuration option documentation

[Richi Dubey]

>
> Ah OK. Can Richi hold the changes locally until we work through the
> process to
> bring this work into the project?

Yes, it'd be at least two weeks before I test my all changes and send a
patch.

On Mon, Jul 13, 2020 at 5:22 AM Chris Johns <chrisj at rtems.org> wrote:

> On 8/7/20 3:35 pm, Sebastian Huber wrote:
> > Hello Chris,
> >
> > thanks for the detailed response.
> >
> > On 08/07/2020 03:55, Chris Johns wrote:
> >> Hi Sebastian,
> >>
> >> Thank you for this patch. I am sorry but I would like to see this
> central repo
> >> issue resolved before any generated files are added to any of the
> project's
> >> repos.
> >>
> >> I understand to some level the path you are taking and moving along but
> these
> >> generated files are coming from a personal repo that has changes,
> processes and
> >> tools that are not being reviewed. I think it would be wise for us to
> agree on
> >> what is behind all this before we agree to what is being generated.
> >
> > I rescheduled some work to be able to provide this patch earlier. Richi
> Dubey
> > asked a couple of questions with respect to the scheduler implementation
> and
> > some of them are related to how application configuration options define
> > internal scheduler data structures. So, It would be nice to be able to
> reference
> > them in Doxygen. You can review this patch independent from the way it
> is produced.
>
> Ah OK. Can Richi hold the changes locally until we work through the
> process to
> bring this work into the project?
>
> >> On 7/7/20 9:25 pm, Sebastian Huber wrote:
> >>> This patch adds the application configuration option documentation
> generated
> >>> from specification items:
> >>>
> >>> https://git.rtems.org/sebh/rtems-qual.git/tree/spec/if/acfg
> >> <gulp> That is a lot of files. I am not sure I am ready to accept
> documentation
> >> as a lot of small files. I am happy to be persuaded otherwise.
> >>
> >> It is not clear to me if these YAML files generate documentation,
> headers, are
> >> requirements or something else.
> >
> > Each application configuration option and group is provided as a separate
> > specification item (YAML file). The type of the specification items is
> > documented here:
> >
> > https://docs.rtems.org/branches/master/eng/req/items.html
>
> Thanks. I have looked at this before and it is detailed. The documentation
> make
> sense and it is a really good starting place.
>
> >
> https://docs.rtems.org/branches/master/eng/req/items.html#spectypeapplicationconfigurationoptionitemtype
> >
> > Application configuration options are things on their own, so they have
> their
> > own items. This enables other items to reference application
> configuration
> > options.
>
> When you say `item` is this in relation to the qual tools and the `item` as
> documented?
>
> > In the manual you have now a lot more cross references for example if
> > you compare it with the RTEMS 4.11 documentation. Items have their own
> history
> > and can be reviewed individually. I see this as benefits.
>
> Yes there are benefits and I am not questioning these benefits. Once we
> depend
> on these generated files we also depend on what generates these so this
> should
> be our focus. Accepting the output into the repos means accepting the
> tools that
> generated these files.
>
> I see the process to get the repo accepted being related to the high level
> architectural and procedure things this change brings. There needs to be a
> review, consideration and discussion.
>
> >> There is nothing I can see in them that defines
> >> their `role` or `roles`.
> >
> > When you look at an item
> >
> >
> https://git.rtems.org/sebh/rtems-qual.git/tree/spec/if/acfg/message-buffer-memory.yml
> >
> >
> > you see attributes like
> >
> > |type: interface ||interface-type: appl-config-option
> ||appl-config-option-type:
> > integer This defines the type of an item. I corresponds to the
> specification
> > type refinement:
> >
> https://docs.rtems.org/branches/master/eng/req/items.html#spectyperootitemtype
> >
> https://docs.rtems.org/branches/master/eng/req/items.html#spectypeinterfaceitemtype
> >
> https://docs.rtems.org/branches/master/eng/req/items.html#spectypeapplicationconfigurationoptionitemtype
> >
> https://docs.rtems.org/branches/master/eng/req/items.html#spectypeapplicationconfigurationvalueoptionitemtype
> > All attributes and the value types are defined by specification item
> types. The
> > specification item types are defined by specification items:
> > https://git.rtems.org/sebh/rtems-qual.git/tree/spec/spec The type
> information is
> > used to generate the documentation in the RTEMS Software Engineering
> manual and
> > is used to verify that the items are in line with the type specification:
> > https://git.rtems.org/sebh/rtems-qual.git/tree/specverify.py |||||
> >
> >> If they do generate files how does that happen and how
> >> are those generated header files related to the C code they are an
> interface
> >> for?
> >
> > The Doxygen header file of this patch is not read by the C compiler.
> However, I
> > would like to generate also the API header files from specification
> items. I am
> > now able to generate <rtems.h> and all header files included by this
> header
> > which define API elements. About 585 items describe the API defined by
> <rtems.h>:
> >
> > https://git.rtems.org/sebh/rtems-qual.git/tree/spec/if/rtems
> >
> > This item describes rtems_semaphore_create():
> >
> >
> https://git.rtems.org/sebh/rtems-qual.git/tree/spec/if/rtems/sem/create.yml
> >
> > It is worth to have a closer look at this item.
> >
> > The interface specification items can be used to generate the header
> files with
> > Doxygen markup, Sphinx sources for the manuals, and an Interface Control
> > Document (ICD, a document required by ECSS).
>
> OK. It will require a bit of time to take in the detail and to understand
> the
> connections.
>
> >> How do I keep each piece in sync?
> > The basic work flow would be to update an item, then call one or more
> scripts,
> > run make or ./waf, or whatever, to generate the derived content. This
> will alter
> > the state of Git submodules. Then you create patch sets from the changes
> in the
> > submodules and send them for review. After acceptance you push the
> changes and
> > update the submodules.
>
> OK, so there is no pulling changes in the repos back, it is a one way
> process?
>
> >> A compile error seems a long way from one
> >> of these YAML files.
> > Yes, this is a bit harder to correlate. I don't think this will be a
> real issue.
> > It is for example easy to figure out with "grep" which item is relevant.
>
> I think you are right, coding interfaces is not done often and I think our
> users
> are please this is the case.
>
> >> For example, if I was to add a YAML file to this directory I have no
> idea where
> >> I look to find the naming, what roles are assumed, how it integrates
> into the
> >> existing documentation, doxygen or header files? I also have no idea
> how I would
> >> create a new API header file? What if I push a new header file to the
> rtems.git
> >> repo with a new #include in rtems.h. Does a regenerate over write it?
> >
> > Yes, this is a real issue. We have to think about how we want to
> continue with
> > the specification stuff in general. I see clear benefits to generate
> content
> > from a project-wide master data set to avoid the copy and paste stuff we
> have
> > right now.
>
> Yes this is true. We have evolved to have a number of places that share
> common
> data, the version is an example. A script to update an item is workable but
> fragile and I think it would not be long before we start to consider a
> master
> repo that lets us work across the various locations in a unified manner.
>
> >> It seems to me the key is the central repo and how it functions, how
> easy or
> >> hard it is to learn about it and how well it will work in practice. A
> key
> >> concern is taking it from a single person repo, ie you, to a project
> wide repo
> >> with concurrent updates ad lots of moving pieces. It is easy to ask
> these
> >> questions and I appreciate they are not all easy to answer but I think
> we need
> >> to try.
> > My approach would be to document common tasks for the every day RTEMS
> maintenance:
> >
> > https://docs.rtems.org/branches/master/eng/req/howto.html
> >
> > You also have to consider the future rate of change. The API parts,
> application
> > configuration options, and glossary terms for example are all things
> which don't
> > change every week.
>
> I think we will end up using something like nuggit ..
>
> https://github.com/monacca1/nuggit
>
> I have not looked into this tool yet. I suspect while it is just you
> working in
> the repo it is fine but I wonder how managing a few submodules will go
> once it
> is opened up and there are multiple people working in the files.
>
> > With respect to the code which generates the content. It is written in
> Python 3
> > with the new Python guidelines in mind:
> >
> > https://docs.rtems.org/branches/master/eng/python-devel.html
> >
> > It uses yapf for the code formatting. It uses the static analyzers
> flake8, mypy,
> > and pylint. There are unit tests using pytest which yield 100% branch
> coverage.
> > It is not finished work. It was not reviewed, but I think it is a good
> start.
>
> I have taken a quick look over the Python code and looks great. I can see
> it is
> all formatted to the new standard and the tests look good. Thank you for
> this.
>
> Following the standards we have and creating tests makes it easier for us
> to
> accept the code. As a result I do not think we need to review it is detail.
>
> >>> The header file is generated by the following script and module:
> >>>
> >>> https://git.rtems.org/sebh/rtems-qual.git/tree/spec2doc.py
> >>>
> >>> https://git.rtems.org/sebh/rtems-qual.git/tree/rtemsqual/applconfig.py
> >>>
> >>> The module uses currently a hack to resolve references external to
> specification
> >>> items, e.g. sections in the RTEMS Classic API Guide or URLs.  I think
> we need
> >>> specialized specification items for these external references.
> >> I do not understand what this means?
> >
> > Being able to generate Doxygen markup and Sphinx sources from the same
> data set
> > was not a small task. We talk here about several weeks of work. You
> can't take
> > it for granted that this works at all in an area with many cross
> references such
> > as the application configuration options. For all external references I
> had to
> > create a table in the source code:
> >
> >
> https://git.rtems.org/sebh/rtems-qual.git/tree/rtemsqual/applconfig.py#n327
> >
> >
> https://git.rtems.org/sebh/rtems-qual.git/tree/rtemsqual/applconfig.py#n381
> >
> > This is not really elegant.
>
> The table can be moved to an external file when ready.
>
> >
> >>
> >>> Sebastian Huber (1):
> >>>    Document application configuration options
> >>>
> >>>   cpukit/doxygen/appl-config.h | 4133
> ++++++++++++++++++++++++++++++++++
> >>>   1 file changed, 4133 insertions(+)
> >>>   create mode 100644 cpukit/doxygen/appl-config.h
> >> Is this information found in the user manual as well? I am struggling to
> >> understand the relationships and what is being presented where.
> >
> > Yes, you find it here for example:
> >
> > https://docs.rtems.org/branches/master/c-user/config/general.html
> >
>
> Thanks
> Chris
> _______________________________________________
> devel mailing list
> devel at rtems.org
> http://lists.rtems.org/mailman/listinfo/devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rtems.org/pipermail/devel/attachments/20200715/3720c897/attachment-0001.html>