New validation test suites

[Chris Johns]

On 17/1/22 8:51 pm, Sebastian Huber wrote:
> On 17/01/2022 09:47, Chris Johns wrote:
>> On 17/1/22 7:04 pm, Sebastian Huber wrote:
>>> On 17/01/2022 08:52, Chris Johns wrote:
>>>> My understanding of the status of these patches is the remaining topic is the
>>>> release dependencies. I have not had time to give this any consideration
>>>> however
>>>> I have a feeling it will not be easy or simple because of the inter-dependency
>>>> of the repos and sub-module relationship. I hope I am wrong about this.
>>>
>>> In order to work with the generated test programs you have to use the
>>> specification items which are in rtems-central.  Most of the validation tests
>>> are defined through transitions from pre-conditions to post-conditions. Working
>>> directly with the generated sources is too complicated. I don't know what
>>> complicates a release here, the rtems-central is just another repository which
>>> needs a release branch. In the release branch, the submodules track the
>>> corresponding release branch of the referenced repository.
>>
>> A release has tarballs of sources and not git repos so I am not sure how
>> branches help? How does the submodules in rtems-central get captured? The
>> current release scripts expand sub-modules which means they need to reconcile or
>> we will have different copies of the code in the release.
>>
>> At the point of release how does the release manager know the rtems-central
>> scripts match the generated sources in the down stream repos that are released?
> 
> The rtems-central contains nothing relevant to an user of RTEMS. There is no
> need to provide a release archive. The purpose of the branch is the maintenance
> of an RTEMS release branch.

This thread indicates my position about releases needing to be solved to move
forwards. This patch set is the first export from rtems-central that we cannot
maintain outside of rtems-central. It has been a requirement of the qual effort
that we can cleanly separate the two parts and accepting these patches changes
that. I still have real reservations about that happening.

>>> Please note, that the RTEMS sources used by rtems-central are currently not
>>> one-to-one a commit of the RTEMS master branch. There are about 90 additional
>>> patches. The patch set with the new validation sets is a part of it.
>>
>> Which is a concern and why a release needs to check and make sure everything is
>> in sync.
>>
>> Ideally the rtems-central submodule hashes match the release tarball sources for
>> each repo. This is the task as I see it.
>>
>> I am sure this can be resolved and I prefer it happens before being pushed so it
>> is not left as a task for the release manager.
> 
> The only way to reduce the gap between the RTEMS version referenced by
> rtems-central and the RTEMS master is to integrate patch sets step by step. Some
> patch sets are specific to the sparc/leon3 BSPs and the build system. 

If rtems.git is a submodule where are the changes being held?

And please understand, I think these tests are important and I would like to see
them merged.

> I am not sure if this should delay the RTEMS 6 release.

You could be right, I am not sure. I am still digging myself out of a pile of
things.

Chris