New validation test suites

[Chris Johns]

On 15/2/22 7:48 pm, Sebastian Huber wrote:
> On 15/02/2022 02:16, Chris Johns wrote:
>> On 15/2/22 12:43 am, Sebastian Huber wrote:
>>> On 03/02/2022 09:09, Sebastian Huber wrote:
>>>> On 30/01/2022 23:26, Chris Johns wrote:
>>>>>>>> We just have to create a
>>>>>>>> release branch for RTEMS 6 and reference release branch commits in the
>>>>>>>> submodules.
>>>>>>> How do you validate the generated sources in the sub-modules match those in
>>>>>>> the
>>>>>>> branched submodules? I think this should be done as part of the release
>>>>>>> procedure to catch any mistakes.
>>>>>> I will write something about this in the release procedure documentation.
>>>>> Thank you, I appreciate this. This is a big help.
>>>>
>>>> I added a note to the post-branch procedure:
>>>>
>>>> https://docs.rtems.org/branches/master/eng/release-process.html#post-branch-procedure
>>>>
>>>
>>
>> Thank you.
>>
>> I noticed the documenting of branches in the procedure. Is it required they be
>> present to run the procedure? An important phase of making a release is the
>> release candidates and these are run off master. Branching before we know we are
>> close to working is an important aspect of the pre-release testing as it brings
>> a range of parts together. This should happen before branching.
> 
> At the moment there is a gap between the RTEMS sources used by rtems-central.git
> and the RTEMS sources in rtems.git. The goal is to close this gap so that
> rtems-central.git can point to a commit in rtems.git. However, this is a process
> which may need a couple of months. On the master branches this gap is not really
> an issue. On a release branch it would be an issue and this is covered by the
> post-branch procedure in the manual now>
>>> How do we want to proceed here?
>>
>> I am currently rather busy on something and will be for a while. As a result I
>> have been a little distracted and I apologise for this.
>>
>>> I have no idea what should I do next.
>>
>> As has been stated, making a release takes some time and effort and my
>> experience of doing it has taught me all the detail needs to be covered. Until I
>> can see rtems-central integrated into a release we cannot not depend on it.
>> Given the way the validation tests have been created merging makes rtems.git
>> depend on rtems-central if merged. In the past we have not taken care of these
>> things and it has fallen to me to sort out and I prefer that does not happen in
>> this case.
> 
> From my point of view, the release related activities for rtems-central.git are
> covered in the Software Release Management chapter.

You may be able to fill in the blanks that I have because you have done the
work. I do not have that experience or information so I am hesitant.

>> I should also state the rtems-central repo is on probation and even if it is
>> added to a release it will still remain like that for a reasonable period of
>> time. It is still not clear where the line of responsibility lies with that repo
>> and the long term maintenance of the qual effort. This may appear abrupt however
>> I need to be clear about this so there is no misunderstanding.
> 
> My ability to predict the future is also rather limited. I focus currently on
> the integration of the remaining work done for the pre-qualification activity.
> This is enough work to keep me busy for a while.
> 
>>
>>> The new validation tests are not the only work from the pre-qualification
>>> activity I would like to integrate. We also have work from Andrew Butterfields
>>> related to formal methods. There are changes in the GR712RC and GR740 BSPs, the
>>> gcov code coverage support, and build system changes to support the build of
>>> RTEMS libraries which contain only the pre-qualified feature set.
>>
>> Making a release is documented here:
>>
>> https://git.rtems.org/rtems-release/tree/README.txt
>>
>> I run a single command that creates a complete release. How will the procedure
>> outlined in the eng manual be implemented here? [1]
>>
>> What will an rtems-central tarball look like? I am specifically concerned the
>> export process to turn a git repo into a tarball will include a copy of the
>> sub-modules. This is the current way the release scripts work. If this is the
>> case we will have 2 copies of some repos in the release and I think that is a
>> bad idea.
> 
> We don't need a rtems-central tarball for releases. This repository contains
> nothing directly relevant to an RTEMS user. 

These tests are not readable by mortals and that changes things. The headers etc
are human readable and can be maintained without rtems-central. If these tests
are merged it will be the first release with them and there may be issues.
Without rtems-central there is no means to figure out any issue. Looking at
repos, checking inputs file, generations etc after a release is something I want
to avoid.

We would not release the documentation as PDFs without the documentation source.

> In theory the rtems-central.git
> repository could be used to make releases. It could make sense to move the stuff
> in rtems-release.git to rtems-central.git.

Actually I am leaning the other way and for other reasons.

>> [1] The release scripts assume an autoconf RTEMS and so I suspect what is there
>> is broken. This makes the rtems-central integration more complicated than it
>> need be. I have not looked at what is needed
> 
> I updated the scripts so that they support the new RTEMS build system. 

Thanks. I have only just got to this email after I had posted the other emails.

> I fixed also some FreeBSD/GNU compatibility issues.

I have posted this in another thread so lets discuss that there.

> It still doesn't run completely on
> Linux. One issue is the missing "sha512" command line tool.

On Linux? Can openssl can do it?

Chris