New validation test suites

[Sebastian Huber]

On 18/01/2022 02:20, Chris Johns wrote:
> On 17/1/22 8:51 pm, Sebastian Huber wrote:
>> On 17/01/2022 09:47, Chris Johns wrote:
>>> On 17/1/22 7:04 pm, Sebastian Huber wrote:
>>>> On 17/01/2022 08:52, Chris Johns wrote:
>>>>> My understanding of the status of these patches is the remaining topic is the
>>>>> release dependencies. I have not had time to give this any consideration
>>>>> however
>>>>> I have a feeling it will not be easy or simple because of the inter-dependency
>>>>> of the repos and sub-module relationship. I hope I am wrong about this.
>>>>
>>>> In order to work with the generated test programs you have to use the
>>>> specification items which are in rtems-central.  Most of the validation tests
>>>> are defined through transitions from pre-conditions to post-conditions. Working
>>>> directly with the generated sources is too complicated. I don't know what
>>>> complicates a release here, the rtems-central is just another repository which
>>>> needs a release branch. In the release branch, the submodules track the
>>>> corresponding release branch of the referenced repository.
>>>
>>> A release has tarballs of sources and not git repos so I am not sure how
>>> branches help? How does the submodules in rtems-central get captured? The
>>> current release scripts expand sub-modules which means they need to reconcile or
>>> we will have different copies of the code in the release.
>>>
>>> At the point of release how does the release manager know the rtems-central
>>> scripts match the generated sources in the down stream repos that are released?
>>
>> The rtems-central contains nothing relevant to an user of RTEMS. There is no
>> need to provide a release archive. The purpose of the branch is the maintenance
>> of an RTEMS release branch.
> 
> This thread indicates my position about releases needing to be solved to move
> forwards. This patch set is the first export from rtems-central that we cannot
> maintain outside of rtems-central. It has been a requirement of the qual effort
> that we can cleanly separate the two parts and accepting these patches changes
> that. I still have real reservations about that happening.

Most patches for release branches are back ports from the master (git 
cherry-pick). It is not likely that you have to work with rtems-central 
for release branches. In the worst case you just have to remove/disable 
a validation test.

The existing tests covered more or less a certain scenario. The new 
validation use pre-conditions and post-conditions relevant to a 
directive call. They usually cover the existing test scenarios as 
special cases, however, they execute and validate all valid variants of 
the pre-condition states. So, they cover a lot of corner cases. This 
approach discovered several small bugs and also some critical bugs in 
the SMP scheduler framework. The test coverage is quite high in the 
operating system core services. It is likely that bug fixes or 
improvements in this area will lead to changes in the specification.

> 
>>>> Please note, that the RTEMS sources used by rtems-central are currently not
>>>> one-to-one a commit of the RTEMS master branch. There are about 90 additional
>>>> patches. The patch set with the new validation sets is a part of it.
>>>
>>> Which is a concern and why a release needs to check and make sure everything is
>>> in sync.
>>>
>>> Ideally the rtems-central submodule hashes match the release tarball sources for
>>> each repo. This is the task as I see it.
>>>
>>> I am sure this can be resolved and I prefer it happens before being pushed so it
>>> is not left as a task for the release manager.
>>
>> The only way to reduce the gap between the RTEMS version referenced by
>> rtems-central and the RTEMS master is to integrate patch sets step by step. Some
>> patch sets are specific to the sparc/leon3 BSPs and the build system.
> 
> If rtems.git is a submodule where are the changes being held?

The pre-qualification branches are in my personal repository:

https://git.rtems.org/sebh/rtems.git/

-- 
embedded brains GmbH
Herr Sebastian HUBER
Dornierstr. 4
82178 Puchheim
Germany
email: sebastian.huber at embedded-brains.de
phone: +49-89-18 94 741 - 16
fax:   +49-89-18 94 741 - 08

Registergericht: Amtsgericht München
Registernummer: HRB 157899
Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler
Unsere Datenschutzerklärung finden Sie hier:
https://embedded-brains.de/datenschutzerklaerung/