[PATCH] build: Add RTEMS_QUALIFIED

[Chris Johns]

On 10/11/2023 7:39 pm, Sebastian Huber wrote:
> On 09.11.23 06:28, Chris Johns wrote:
>> On 8/11/2023 6:44 pm, Sebastian Huber wrote:
>>> On 06.11.23 21:55, Chris Johns wrote:
>>>> On 6/11/2023 8:27 pm, Sebastian Huber wrote:
>>>>> On 06.11.23 01:14, Chris Johns wrote:
>>>>>> On 4/11/2023 1:31 am, Sebastian Huber wrote:
>>>>>>> On 03.11.23 15:08, Joel Sherrill wrote:
>>>>>>>> On Fri, Nov 3, 2023 at 3:58 AM Sebastian Huber
>>>>>>>> <sebastian.huber at embedded-brains.de
>>>>>>>> <mailto:sebastian.huber at embedded-brains.de>> wrote:
>>>>>>>>
>>>>>>>>        The goal of the RTEMS pre-qualification activity #3701 is a
>>>>>>>> specified
>>>>>>>>        and validated subset of RTEMS.  For users of the pre-qualified
>>>>>>>> subset of
>>>>>>>>        RTEMS it is important to not accidentally use not pre-qualified
>>>>>>>>        features.  One way to achieve this, is to build only the sources
>>>>>>>> of the
>>>>>>>>        pre-qualified feature set. This customized build is enabled by
>>>>>>>> the new
>>>>>>>>        build configuration option RTEMS_QUALIFIED.  If it is enabled, then
>>>>>>>> only
>>>>>>>>        the pre-qualified subset of RTEMS is built and installed.
>>>>>>>>
>>>>>>>>        Building with RTEMS_QUALIFIED enable is currently only supported for
>>>>>>>> the
>>>>>>>>        sparc/leon3 BSP family.  To support an RTEMS_QUALIFIED enabled
>>>>>>>> build,
>>>>>>>>        changes in the CPU port and the BSP are required to only use
>>>>>>>> features of
>>>>>>>>        the pre-qualified feature set.
>>>>>>>>
>>>>>>>>
>>>>>>>> Where is this documented?
>>>>>>> You mean a documentation of what needs to be done to create pre-qualified
>>>>>>> BSP? I
>>>>>>> don't have it available yet. A good place to add this would be the how-to
>>>>>>> section in the RTEMS Software Engineering manual.
>>>>>>>
>>>>>>>> This is a very large patch. Are you assuming that if "not qualified" is
>>>>>>>> specified,
>>>>>>>> then it is in the qualified set?
>>>>>>> No, the logic is reversed. Everything is built by default. Some parts are
>>>>>>> only
>>>>>>> enabled if RTEMS_QUALIFIED is not enabled, for example
>>>>>>> (spec/build/cpukit/objextra.yml):
>>>>>>>
>>>>>>> enabled-by:
>>>>>>>      not: RTEMS_QUALIFIED
>>>>>> It seems counter intuitive to me. I have no idea about qual work but my
>>>>>> limited
>>>>>> understanding is everything is controlled yet this is the inverse of that and
>>>>>> anything anyone adds will by default be qualified?
>>>>>
>>>>> The goal of this patch set is to place each source and header file of RTEMS
>>>>> into
>>>>> two buckets, the pre-qualified bucket and the extra bucket. For two buckets
>>>>> you
>>>>> need just one option with two values. In the current patch it is
>>>>> RTEMS_QUALIFIED
>>>>> enabled or disabled.
>>>>
>>>> I think we should also include in our discussion code contained within this
>>>> define (or defines) in shared files and how we manage changes to that code? In
>>>> an ideal world we would not have any need for conditional code however I
>>>> appreciate this may not be possible or initial achievable. We should however
>>>> look for approaches that try to avoid this and understand the constraints the
>>>> define brings. For example can I change code in a qualification or FACE define
>>>> when I do not know the standards they support?
>>>
>>> We should avoid using the C preprocessor for the profile customization. For the
>>> ECSS pre-qualification, there was no need to do this (with the exception of
>>> testbeginend.c, but this could be changed).
>>
>> +1
>>
>>>>> Since Joel already brought the FACE profile into play, an alternative could be
>>>>> this approach. Lets assume that we pre-qualified the FACE profile
>>>>> interfaces of
>>>>> RTEMS. Use RTEMS_EVERYTHING for the I want everything profile. Use
>>>>> RTEMS_QUALIFIED for the pre-qualified profile. Use RTEMS_FACE_PROFILE for
>>>>> interfaces of the FACE profile.
>>>>
>>>> If these are profiles I suggest RTEMS_PROFILE_EVERYTHING so RTEMS_PROFILE_.*
>>>> can
>>>> find all profiles?
>>>
>>> Yes, this make sense.
>>>
>>>>
>>>>> The objects which are not pre-qualified would use this:
>>>>>
>>>>> enabled-by: RTEMS_EVERYTHING
>>>>
>>>> RTEMS_PROFILE_COMMERCIAL ?
>>>> RTEMS_PROFILE_INDUSTRIAL ?
>>>> RTEMS_PROFILE_RTEMS ?
>>>>
>>>> ... or something that reflects what we have always shipped? I borrowed those
>>>> labels from the way parts are classed. I tend to keep away from labels like
>>>> new,
>>>> next, everything etc because they quickly date.
>>>>
>>>>>
>>>>> The objects which are pre-qualified would use this:
>>>>>
>>>>> enabled-by:
>>>>> - RTEMS_EVERYTHING
>>>>> - RTEMS_QUALIFIED
>>>>
>>>> Qualified to what? RTEMS_PROFILE_FACE defines something useful and concrete. Is
>>>> there an equivalent we could use for the work you have been doing?
>>>
>>> The selection of the feature set for the ECSS pre-qualification was based on a
>>> survey:
>>>
>>> https://ftp.rtems.org/pub/rtems/people/sebh/rtems-smp-userconsultation-27032019-release.pdf
>>>
>>> https://ftp.rtems.org/pub/rtems/people/sebh/tn-space-profile-r6-23062019.pdf
>>>
>>
>> RTEMS_PROFILE_SPACE ? I just used the name on page 1 of this document.
> 
> The profile in the patch set is by no means limited to the European space
> industry. We need a name for the subset of RTEMS which fulfills the 4 rules
> below. This subset may grow over time and include for example the POSIX threads
> API. We already have plans to work on this.

Yes I agree the set of files covered is not limited to the EU space program
however I am confused by a qualification file set that is not explicitly defined
for a specific profile. Are you able to define a suitable subset for all
possible profiles? Maybe I do not understand the problem but to me you either
need to maintain a union of overlaying files for all profiles and what ever
rules that brings or the profiles need to maintain their own set?

>>>> I do not like the use of QUALIFIED as mentioned in my other post in this thread
>>>> because it implies something the open project should not and that is the
>>>> code is
>>>> qualified. We will only provide pre-qualification data packs from the public
>>>> repos unless that has changed?
>>>
>>> In the context of this patch set and the Software Requirements Engineering
>>> chapter of the RTEMS Software Engineering manual
>>>
>>> https://docs.rtems.org/branches/master/eng/req/index.html
>>>
>>> "qualified" means that:
>>>
>>> 1. An interface specification exists for each qualified API element.
>>>
>>> 2. Each qualified interface has a functional specification.
>>>
>>> 3. The function is validated by a test.
>>>
>>> 4. You get 100% branch and code coverage from the validation tests.  For some
>>> justified exceptions you can use unit tests.
>>
>> Yes but there is no public proof so it is a promise until it is a fact and that
>> is achieved at the end of the process. What is written is correct but a repo is
>> in the present tense.
> 
> I continuously work on integrating the work into the RTEMS Project. This patch
> set is a part of this work. It would reduce the maintenance cost on my side and
> so help to free time to work on other parts of the integration TODO list.

I understand this and appreciate the work being done. My point is about the
language and the word being used. If you know nothing about RTEMS or qualifying
code can the use of RTEMS_QUALIFIED mislead someone into thinking that code is
qualified for them when it is not?

One thing this project has taught me over the last few decades is what I see and
think is not what other users see and think. The accuracy of language is a factor.

I suggested INDUSTRIAL and Amar suggested GENERAL. If we assume RTEMS_PROFILE_.*
is about a qualifable (?) profile does that help?

>>> If someone has a better name than "qualified" for this, I am happy to use it.
>>
>> Good question. If RTEMS_PROFILE_SPACE is used can RTEMS_QUALIFIED be avoided?
> 
> I think "space" is too specific.

Yes it could be and GENERAL may not be specific enough?

>>>>> The objects which are pre-qualified or in the FACE profile would use this:
>>>>>
>>>>> enabled-by:
>>>>> - RTEMS_EVERYTHING
>>>>> - RTEMS_QUALIFIED
>>>>> - RTEMS_FACE_PROFILE
>>>>
>>>> The addition of these defines explodes the builds and testing we need to
>>>> perform
>>>> for each commit. We need to discuss this some more and look for boundaries that
>>>> could help limit the builds and testing for each profile.
>>>
>>> Yes, all these options lead to a combinatorial explosion. We have to be
>>> pragmatic and focus on a useful subset for tests.
>>
>> I know for RTEMS_PROFILE_SPACE it will be a specific set of LEON archs? I see no
>> point building that profile for anything else. The point is a profile should
>> have a list of arches or BSPs associated to it as part of being accepted.
> 
> This profile is not specific to a particular BSP. However, for each
> pre-qualified BSP there is some work to do. Firstly, it needs to be restructured
> to use only features of the pre-qualified feature set. Secondly, you may have to
> specify BSP-specific functions and validate them. Thirdly, you have to review
> all chip errata, implement workarounds, justify why erratas are not applicable
> to the BSP.

Yes. Do you have a different list of BSPs in mind?

>> Chris
>>
>>>>> Of RTEMS_EVERYTHING, RTEMS_QUALIFIED, and RTEMS_FACE_PROFILE exactly one
>>>>> option
>>>>> shall be enabled.
>>>>
>>>> Yes this makes sense. Should a source file not in a profile be considered an
>>>> error?
> 
> Yes, this would be an error.

Great.

Chris