Fwd: Identify 3rd party source in spec?

[Gedare Bloom]

On Fri, Nov 4, 2022 at 11:00 AM Gedare Bloom <gedare at rtems.org> wrote:
>
> On Fri, Nov 4, 2022 at 9:55 AM Gedare Bloom <gedare at rtems.org> wrote:
> >
> > On Fri, Nov 4, 2022 at 8:58 AM Sebastian Huber
> > <sebastian.huber at embedded-brains.de> wrote:
> > >
> > > On 04/11/2022 15:38, Gedare Bloom wrote:
> > > > On Fri, Nov 4, 2022 at 2:59 AM Sebastian Huber
> > > > <sebastian.huber at embedded-brains.de>  wrote:
> > > >> On 31/10/2022 20:01, Gedare Bloom wrote:
> > > >>> I would like to float the idea of managing 3rd party source tracking
> > > >>> through the build system spec files. I believe this would be the most
> > > >>> efficient way to maintain this information, and we can leverage the
> > > >>> existing build system code for tasks such as automatic format checks,
> > > >>> generating lists of third-party code, etc.
> > > >>>
> > > >>> This will require refactoring some spec files to pull 3rd party code
> > > >>> out to a separate .yml file that gets linked. Once that is done, then
> > > >>> we could add another attribute for this tracking purpose. I would like
> > > >>> to keep it simple as a boolean, maybe just "third-party: true/false"
> > > >>> Attached is an example patch showing how this might work for the
> > > >>> dtc/libfdt code as a build objects item type 'obj', and for zlib
> > > >>> library as a build library item type 'lib' with some proof-of-concept
> > > >>> code for generating a listing of third party source files.
> > > >> In the future we could expand this third-party attribute to a dictionary
> > > >> to provide more detailed information about the third-party sources, for
> > > >> example the upstream Git repository, the import commit, path translation
> > > >> mappings, post and pre import actions. This could be used to
> > > >> automatically generate updates from upstream changes.
> > > >>
> > > > Nice. I don't see an explicit "dictionary" type for attributes, but I
> > > > do see that dict-like behavior could be achieved using a 'list' of
> > > > attributes nested below an attribute. I have the time allocated that i
> > > > could try to learn how to do this with a simple example, for example
> > > > to add a "upstream" and an "import-commit" field. Then, the
> > > > third-party indicator will change from
> > > > third-party: false   ->
> > > >      third-party: []
> > > > third-party: true    ->
> > > >       third-party:
> > > >           - upstream: URL
> > > >           - import-commit: git-sha
> > > > I would guess. Would this kind of structure "just work" in the current
> > > > support of spec/build or will it require some integration with
> > > > rtems-central?  Or, am I following the wrong ideas,
> > >
> > > third-party: null
> > >
> > > would indicate that this object/library doesn't have an upstream project.
> > >
> > > If there is an upstream project, then we could describe this through
> > > additional attributes, for example:
> > >
> > > third-party:
> > >    type: git
> > >    commit: abc
> > >    repository: def
> > >    file-map:
> > >    - replace regex 1
> > >    - replace regex 2
> > >
> >
> > OK, that makes sense. I think I can make a first pass just to put in
> > the framework for extending this later. Taking DTC as an example,
> > maybe something like the following makes sense?
> >
> > third-party:
> >   - type: git
> >   - repository: git://git.kernel.org/pub/scm/utils/dtc/dtc.git
> >   - commits:
> >     - upstream: c0c2e115f82ed3bc5f9d3f9e5380f0f7e81a1c21
> >     - import: 175263ec91f00f6668ae7f28f385850a1c77bf6b
> >   - update: cherry-pick
> >
> >
> > Where we will have to define some update methods such as applying
> > patches from the upstream (cherry-pick).
> >
> > As POC I can process this with wscript and obtain
> > [{'type': 'git'}, {'repository':
> > 'git://git.kernel.org/pub/scm/utils/dtc/dtc.git'}, {'commits':
> > [{'upstream': 'c0c2e115f82ed3bc5f9d3f9e5380f0f7e81a1c21'}, {'import':
> > '175263ec91f00f6668ae7f28f385850a1c77bf6b'}]}, {'update':
> > 'cherry-pick'}, {'source': ['cpukit/dtc/libfdt/fdt.c',
> > 'cpukit/dtc/libfdt/fdt_addresses.c',
> > 'cpukit/dtc/libfdt/fdt_empty_tree.c', 'cpukit/dtc/libfdt/fdt_ro.c',
> > 'cpukit/dtc/libfdt/fdt_rw.c', 'cpukit/dtc/libfdt/fdt_strerror.c',
> > 'cpukit/dtc/libfdt/fdt_sw.c', 'cpukit/dtc/libfdt/fdt_wip.c']}]
> >
> > If this looks reasonable, I can try to add these fields as part of my
> > sweep through the cpukit tagging third-party sources.
> >
>
> Given the complexity of this tagging, I'm going to start with just the
> true/false approach to categorize third-party sources. We can do
> something like the above in the future.

I have come across one problem with my approach. Headers that aren't
installed (e.g., #include "fastlz.h") do not appear within the build
system. These might need to be identified and handled as some kind of
special case. There are 79 such files in cpukit.
rtems/cpukit$ find . -name "*.h" | grep -v include | wc
     79      79    2072