New validation test suites

[Chris Johns]

On 16/12/21 6:36 pm, Sebastian Huber wrote:
> On 16/12/2021 04:51, Chris Johns wrote:
>> On 16/12/21 3:27 am, Sebastian Huber wrote:
>>> On 15/12/2021 06:46, Chris Johns wrote:
>>>> On 14/12/21 6:24 pm, Sebastian Huber wrote:
>>>>> Hello Chris,
>>>>>
>>>>> On 13/12/2021 22:01, Chris Johns wrote:
>>>>>> On 14/12/21 1:53 am, Sebastian Huber wrote:
>>> [...]
>>>>>>> We finished the specification of the pre-qualified RTEMS feature set. The
>>>>>>> specification is available in an RTEMS Project repository:
>>>>>>>
>>>>>>> https://git.rtems.org/rtems-central/tree/spec
>>>>>>
>>>>>> I had a quick look. Is there a more user friendly view of this data?
>>>>>>
>>>>>> I think the term "specification" is a little bit misleading because the data
>>>>>> files are not easily read by a person. I understand this is the specification
>>>>>> data set however it is not what I am traditionally use to seeing.
>>>>>
>>>>> You can use the "./specview.py" script to get views of the specification.  For
>>>>> example, this command displays the transition map for the rtems_signal_send()
>>>>> directive:
>>>>
>>>> Is specview.py part of rtems.git?
>>>
>>> No, this script is in rtems-central.  This is also the location of the
>>> specification items.
>>
>> I am not sure linking a script from that repo like this is helpful.
>>
>>>> If not part of rtems.git how much data is there for all the output? That is it
>>>> is generated and held in the repo with the tests?
>>>
>>> In rtems.git, there are only the generated sources.
>>>
>>> [...]
>>
>> There should be no reach back to the upstream specs, scripts etc and for good
>> reasons. The information you posted is nice and useful and I do not wish to
>> release manage rtems-central to accommodate these tests in a release.
>>
>> Would capturing that information with the tests be something worth doing?
> 
> I don't think it would be useful. If you want to modify the tests you should
> work with the specification items and the corresponding scripts.

This is not about modifying the tests. As I previously stated the tests provide
little detail on the verification matrix being solved.

In relation to where is best to make changes, rtems-central may be the best
place however we will accept patches to the tests as they are in rtems.git. How
that is pushed back to rtems-central is not a focus here.

> Adding the
> tables as comments would blow up the sources considerably. Some tests have about
> 50000 table entries and the table entries depend on C preprocessor defines.

Ah yes I agree and what I was wanting to understand.

I think we need to understand what a release will contain because the specs used
are not captured with a release. I appreciate the efforts to make this available
as a workflow for development and for the pre-qual process but I am now
critically examining what this means for a release. For example lets say years
after a release someone questions a test, they will only have the test source
code in the release package and I think this is a short coming. Adding the
rtems-central repo as a package to the releases source may be a solution however
this creates further issues. How do I know the master of the rtems-central and
the committed sources match when creating the release as separate pieces of
rtems.git may have been updated at different commit points in rtems-central?

I suggest we get together when we can. Contact me off-line and lets see if we
can arrange a time :)

> 
> [...]
>>>>> In an earlier version of the header, we had a link which you didn't like:
>>>>
>>>> If I need to look at the formatting rules the heading "Software Development
>>>> Management" is easy to see and then a click on "Coding Standards" gives me what
>>>> I am looking for.
>>>>
>>>> To generate these headers I click on "Software Requirements Engineering" and
>>>> then do I just guess until I find it in the "How To" section? I am actually
>>>> asking this be sorted out so it is not left hanging and we are not left
>>>> guessing
>>>> what to do. If it can be rearrange into something meaningful it would help. :)
>>>
>>> Well, if you read the text in the header:
>>>
>>>   * For information on updating and regenerating please refer to the How-To
>>>   * section in the Software Requirements Engineering chapter of the
>>>   * RTEMS Software Engineering manual.  The manual is provided as a part of
>>>   * a release.  For development sources please refer to the online
>>>   * documentation at:
>>>   *
>>>   * https://docs.rtems.org
>>>
>>> You should read the How-to section or not?
>>
>> Yes I should have and thanks for pointing this out but I did not see this and
>> the manual as it stands did not help. I think it should change. It can be
>> performed post this patch set but I think the documentation would read better if
>> changed.
> 
> Could you please make a suggestions how the text should be changed?
> 

I am not sure. I have not looked at rtems-central, the scripts or commands it
provides. I think at a minimum a section each set of generated sources in
rtems.git needs to be covered.

>>>>>> What hardware have the validation tests been run on? Any tier 1 archs?
>>>>>
>>>>> I tested with the sparc/leon3 BSPs and the arm/realview_pbx_a9_qemu.
>>>>
>>>> Is the leon3 tested on hardware or simulation?
>>>>
>>>>> You need a
>>>>> full implementation of the new Interrupt Manager directives and a working
>>>>> Cache
>>>>> Manager implementation.
>>>>
>>>> Is this documented?
>>>>
>>>> I am sorry I do not know the list of archs and bsps that support the new
>>>> interrupt manager directives. Maybe it would be good to list them?
>>>
>>> All BSPs have at least a stub implementation of the new directives. The
>>> directives are tested in a dedicated test suite. You will notice failures in
>>> this test suite if the directives are not implemented.
>>
>> Are these expected failures?
> 
> Yes, they would be expected failures. I can add the test information. For which
> BSPs should I do this?

Any we know the tests will not run on?

>>>>> I noticed an issue with the thread restart on aarch64/a53_lp64_qemu.
>>>>>
>>>>> On powerpc/psim there is an issue in one test case, due to:
>>>>>
>>>>> #define CPU_ALL_TASKS_ARE_FP CPU_HARDWARE_FP
>>>>
>>>> Sorry, I am not following what the issue is? Does this effect all PPC BSPS?
>>>
>>> Not all, the newer BSPs have no separate floating-point context.
>>
>> Which ones have the issue, the newer BSPs or the older ones?
> 
> The older ones.
> 
>>
>>> This is something which needs to be fixed in the specification.
>>
>> Of?
>>
>> < From my point of view this is just a minor issue.
>>
>> As in fixing these tests?
>>
>>>>> Another issue is that the tm27 interrupt must be independent of the clock
>>>>> driver
>>>>> interrupt.  This is not the case for powerpc/psim.
>>>>>
>>>>> There is definitely some work left to cover all edge cases. Some tests are
>>>>> quite
>>>>> complicated.
>>>>
>>>> Sure. I would like to understand the effects this has?
>>>
>>> Maybe I can rearrange the test cases so that the tm27 support is only used if no
>>> clock driver is needed. The tm27 support is used to run handlers in interrupt
>>> context.
>>
>> OK.
> 
> I will try to fix these issues, but this will delay the integration for a couple
> of weeks.

I am fine with tickets that track the work to be done.

>>>>>> Is there anything that interprets the new test output format? It looks like
>>>>>> lots
>>>>>> of great info but a little difficult to read.
>>>>>
>>>>> EDISOFT worked on a test report generator, however, it is not yet in a
>>>>> reviewable state.
>>>>
>>>> OK. I think something that handles this data would be good to have.
>>>
>>> Yes, maybe we could let a student work on this. In theory, this is not
>>> difficult. Read the report.yaml generated by the RTEMS Tester and convert it to
>>> a Python objects representation. Then use this high-level representation to
>>> generate a report in format X.
>>
>> Sounds good.
>>
>> And we need to get all the BSPs baselined with 0 failures so we know where we
>> stand as changes are being made.
> 
> All BSPs is a bit too much.
> 

Yes sorry I meant all tier 1 BSPs.

Chris