Store test reports?

[Sebastian Huber]

On 23/03/2022 23:47, Chris Johns wrote:> On 24/3/2022 3:38 am, Sebastian 
Huber wrote:
>> I plan to commit the new validation tests tomorrow. Then I will make some test
>> runs. The test reports sent to build at rtems.org have only limited value to track
>> performance regressions for example.
> 
> The test reports sent to that list are archived and could be post processed as a
> batch. A procmail filter could then be used to then maintain any batch generated
> data. So there is some flexibility with its data.
> 
> A mailing list is nice because it is cheap and easy to create tools to send reports.

You could also send the YAML report as a patch e-mail.

> 
>> Would it make sense to store the full test
>> reports in YAML format in an RTEMS repository (for example
>> rtems-central/test-reports/arch/bsp/yyyy-mm-dd-commit.yml)?
> 
> It would be ideal having in a release the YAML and human digestible reports for
> all tier 1 archs. I am not sure that can be achieved because the release's hash
> and the tested hash may differ and we do not have a central hardware lab we can
> access and use to create the reports when creating the release.

This is not only useful for releases. The time when you run tests is 
somewhat arbitrary. Having the full raw test data available can be used 
to figure out known good versions of a BSP in case something breaks. 
This helps during a git bisect. It can be also used to gather 
performance data which can be compared over time to notice regressions 
introduced by new compiler versions and the RTEMS development. It is 
important that only test reports are added which use commits of RTEMS 
repositories without any local modifications so that everyone is able to 
reproduce the data.

> 
> I am not a fan of adding dated files like this to repos. Every time I have done
> this I have come to regret it. What about a path on ftp.rtems.org, for example
> pub/rtems/reports/validation/6/arch/bsp/yyyy-mm-dd-commit.[yml,pdf] ?
> 
> What archs will be supported?

The YAML report is supported by rtems-test in general.

> 
> Is there a PDF type report?

For archiving a good human and machine readable format like YAML is 
better suited. You can even diff two reports easily.

-- 
embedded brains GmbH
Herr Sebastian HUBER
Dornierstr. 4
82178 Puchheim
Germany
email: sebastian.huber at embedded-brains.de
phone: +49-89-18 94 741 - 16
fax:   +49-89-18 94 741 - 08

Registergericht: Amtsgericht München
Registernummer: HRB 157899
Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler
Unsere Datenschutzerklärung finden Sie hier:
https://embedded-brains.de/datenschutzerklaerung/