Store test reports?

[Chris Johns]

On 24/3/2022 3:38 am, Sebastian Huber wrote:
> I plan to commit the new validation tests tomorrow. Then I will make some test
> runs. The test reports sent to build at rtems.org have only limited value to track
> performance regressions for example. 

The test reports sent to that list are archived and could be post processed as a
batch. A procmail filter could then be used to then maintain any batch generated
data. So there is some flexibility with its data.

A mailing list is nice because it is cheap and easy to create tools to send reports.

> Would it make sense to store the full test
> reports in YAML format in an RTEMS repository (for example
> rtems-central/test-reports/arch/bsp/yyyy-mm-dd-commit.yml)?

It would be ideal having in a release the YAML and human digestible reports for
all tier 1 archs. I am not sure that can be achieved because the release's hash
and the tested hash may differ and we do not have a central hardware lab we can
access and use to create the reports when creating the release.

I am not a fan of adding dated files like this to repos. Every time I have done
this I have come to regret it. What about a path on ftp.rtems.org, for example
pub/rtems/reports/validation/6/arch/bsp/yyyy-mm-dd-commit.[yml,pdf] ?

What archs will be supported?

Is there a PDF type report?

Chris