New validation test suites

[Chris Johns]

On 17/2/22 5:47 pm, Sebastian Huber wrote:
> On 17/02/2022 06:11, Chris Johns wrote:
>> On 15/2/22 7:48 pm, Sebastian Huber wrote:
>>> On 15/02/2022 02:16, Chris Johns wrote:
>>>> On 15/2/22 12:43 am, Sebastian Huber wrote:
>>>>> On 03/02/2022 09:09, Sebastian Huber wrote:
>>>>>> On 30/01/2022 23:26, Chris Johns wrote:
>>>>>>>>>> We just have to create a
>>>>>>>>>> release branch for RTEMS 6 and reference release branch commits in the
>>>>>>>>>> submodules.
>>>>>>>>> How do you validate the generated sources in the sub-modules match
>>>>>>>>> those in
>>>>>>>>> the
>>>>>>>>> branched submodules? I think this should be done as part of the release
>>>>>>>>> procedure to catch any mistakes.
>>>>>>>> I will write something about this in the release procedure documentation.
>>>>>>> Thank you, I appreciate this. This is a big help.
>>>>>>
>>>>>> I added a note to the post-branch procedure:
>>>>>>
>>>>>> https://docs.rtems.org/branches/master/eng/release-process.html#post-branch-procedure
>>>>>>
>>>>>>
>>>>>
>>>>
>>>> Thank you.
>>>>
>>>> I noticed the documenting of branches in the procedure. Is it required they be
>>>> present to run the procedure? An important phase of making a release is the
>>>> release candidates and these are run off master. Branching before we know we
>>>> are
>>>> close to working is an important aspect of the pre-release testing as it brings
>>>> a range of parts together. This should happen before branching.
>>>
>>> At the moment there is a gap between the RTEMS sources used by rtems-central.git
>>> and the RTEMS sources in rtems.git. The goal is to close this gap so that
>>> rtems-central.git can point to a commit in rtems.git. However, this is a process
>>> which may need a couple of months. On the master branches this gap is not really
>>> an issue. On a release branch it would be an issue and this is covered by the
>>> post-branch procedure in the manual now>
>>>>> How do we want to proceed here?
>>>>
>>>> I am currently rather busy on something and will be for a while. As a result I
>>>> have been a little distracted and I apologise for this.
>>>>
>>>>> I have no idea what should I do next.
>>>>
>>>> As has been stated, making a release takes some time and effort and my
>>>> experience of doing it has taught me all the detail needs to be covered.
>>>> Until I
>>>> can see rtems-central integrated into a release we cannot not depend on it.
>>>> Given the way the validation tests have been created merging makes rtems.git
>>>> depend on rtems-central if merged. In the past we have not taken care of these
>>>> things and it has fallen to me to sort out and I prefer that does not happen in
>>>> this case.
>>>
>>>  From my point of view, the release related activities for rtems-central.git are
>>> covered in the Software Release Management chapter.
>>
>> You may be able to fill in the blanks that I have because you have done the
>> work. I do not have that experience or information so I am hesitant.
> 
> Ok, and what should I do now, just sit and wait?
>

I have a couple of ideas that may make this happen but I need a little time to
play. And as said my time is limited at the moment.

> [...]
>>>> What will an rtems-central tarball look like? I am specifically concerned the
>>>> export process to turn a git repo into a tarball will include a copy of the
>>>> sub-modules. This is the current way the release scripts work. If this is the
>>>> case we will have 2 copies of some repos in the release and I think that is a
>>>> bad idea.
>>>
>>> We don't need a rtems-central tarball for releases. This repository contains
>>> nothing directly relevant to an RTEMS user.
>>
>> These tests are not readable by mortals and that changes things. The headers etc
>> are human readable and can be maintained without rtems-central. If these tests
>> are merged it will be the first release with them and there may be issues.
> 
> The worst case is that you have to remove the tests or mark them as expected
> fail if they turn out to be not maintainable. I really don't share your concerns
> with respect to the release. The validation tests are an essential part of the
> pre-qualification activity.
> 
>> Without rtems-central there is no means to figure out any issue. Looking at
>> repos, checking inputs file, generations etc after a release is something I want
>> to avoid.
> 
> The rtems-central.git should be used as a Git clone. Providing it as an archive
> may just invite users to do stupid things.
> 
>>
>> We would not release the documentation as PDFs without the documentation source.
> 
> Providing the documentation sources as an archive wouldn't be strictly necessary.

It is if you need to show a clear path from sources to any generated output.

>>> In theory the rtems-central.git
>>> repository could be used to make releases. It could make sense to move the stuff
>>> in rtems-release.git to rtems-central.git.
>>
>> Actually I am leaning the other way and for other reasons.
>>
>>>> [1] The release scripts assume an autoconf RTEMS and so I suspect what is there
>>>> is broken. This makes the rtems-central integration more complicated than it
>>>> need be. I have not looked at what is needed
>>>
>>> I updated the scripts so that they support the new RTEMS build system.
>>
>> Thanks. I have only just got to this email after I had posted the other emails.
>>
>>> I fixed also some FreeBSD/GNU compatibility issues.
>>
>> I have posted this in another thread so lets discuss that there.
>>
>>> It still doesn't run completely on
>>> Linux. One issue is the missing "sha512" command line tool.
>>
>> On Linux? Can openssl can do it?
> 
> On Linux, the command is sha512sum:
> 
> sha512sum wscript
> c7d699abcc9a8ae16261c5340cb57b151f3da5034a9d7fde21ecf0e02ad18036bfb102cc6d8e8603961f97e12d47ade9b04ed05a5eed1098cfe10eac27baf095
>  wscript
> 
> sha512sum --tag wscript
> SHA512 (wscript) =
> c7d699abcc9a8ae16261c5340cb57b151f3da5034a9d7fde21ecf0e02ad18036bfb102cc6d8e8603961f97e12d47ade9b04ed05a5eed1098cfe10eac27baf095
> 
> 
> Using openssl would be a portable option, but it is not installed by default on
> FreeBSD:
> 
> openssl dgst -sha512 wscript
> SHA512(wscript)=
> c7d699abcc9a8ae16261c5340cb57b151f3da5034a9d7fde21ecf0e02ad18036bfb102cc6d8e8603961f97e12d47ade9b04ed05a5eed1098cfe10eac27baf095
> 

Yes but you also need sphinx and other things but there is a simple shell hack
to check a command so we can check for either the Linux or FreeBSD versions.

Chris