[tools] tester: Normalize JSON and YAML reports

[Sebastian Huber]

On 29/06/2022 17:54, Kinsey Moore wrote:
> On 6/29/2022 04:34, Sebastian Huber wrote:
>> On 29/06/2022 11:20, Chris Johns wrote:
>>>
>>>> On 29 Jun 2022, at 4:42 pm, Sebastian Huber 
>>>> <sebastian.huber at embedded-brains.de> wrote:
>>>>
>>>> On 29/06/2022 08:40, Sebastian Huber wrote:
>>>>> Report the same data in JSON and YAML reports.  Do not report 
>>>>> redundant
>>>>> information.
>>>>> Update 4671.
>>>>
>>>> This patch changes the JSON reports. Are there already consumers for 
>>>> the JSON reports so that we have to be backward compatible?
>>>
>>> Could compatibility be added back in if this proves to be an issue?
>>>
>>> I am wondering if that could be considered if comparability becomes 
>>> an issue.
>>
>> The JSON report was added by:
>>
>> commit 5fcaf843bdb491e2da14dc1af61806ac6d20bae5
>> Author: Kinsey Moore <kinsey.moore at oarcorp.com>
>> Date:   Wed Aug 21 16:34:12 2019 +0000
>>
>>     Add JSON log generation
>>
>>     Add log formatter hooks and JSON log formatter to the test 
>> infrastructure
>>     for consumption by automated processes or report generators.
>>
>>
>> I am not sure if these automated processes or report generators 
>> already exist.
>>
>> The existing attribute names are quite inconsistent, for example 
>> "Command Line", "passed_count", "wrong-version_count". I would use 
>> lower case only with "-" as a separator. The JSON report should 
>> contain all information of a test run.
>>
>> The new report looks like this:
>>
>> {
>>     "command-line": [
>>         "/opt/rtems/6/bin/rtems-test",
>>         "--rtems-bsp=xilinx_zynq_a9_qemu",
>>         "--report-format=json",
>>         "--report-path=report",
>>         "build/arm/xilinx_zynq_a9_qemu/testsuites/samples/"
>>     ],
>>     "end-time": "2022-06-28T14:08:47.595131",
>>     "host": "Linux-5.3.18-150300.59.76-default-x86_64-with-glibc2.3.4 
>> (Linux lupus 5.3.18-150300.59.76-default #1 SMP Thu Jun 16 04:23:47 
>> UTC 2022 (2cc2ade) x86_64 x86_64)",
>>     "python": "3.6.15 (default, Sep 23 2021, 15:41:43) [GCC]",
>>     "reports": [
>>         {
>>             "arch": "arm",
>>             "bsp": "xilinx_zynq_a9_qemu",
>>             "command-line": "qemu-system-arm -no-reboot -nographic 
>> -net none -serial null -serial mon:stdio -M xilinx-zynq-a9 -m 256M 
>> -kernel build/arm/xilinx_zynq_a9_qemu/testsuites/samples/cdtest.exe",
>>             "end-time": "2022-06-28T12:08:48.161691+00:00",
>>             "executable": 
>> "build/arm/xilinx_zynq_a9_qemu/testsuites/samples/cdtest.exe",
>>             "executable-sha512": 
>> "413bc8a506053c95a0e1d93942cce9fd0d653f7b3cde42623085ce147aea324f4baeff7ce0102008322fb1341bc2b38defd3925a64144e47e13cc5f32185bf78", 
>>
>>             "output": [
>>                 "qemu-system-arm: warning: nic cadence_gem.0 has no 
>> peer",
>>                 "qemu-system-arm: warning: nic cadence_gem.1 has no 
>> peer",
>>                 "",
>>                 "",
>>                 "*** BEGIN OF TEST CONSTRUCTOR/DESTRUCTOR ***",
>>                 "*** TEST VERSION: 
>> 6.0.0.3302b72754df5f37214e86dd68522189857772c7",
>>                 "*** TEST STATE: EXPECTED_PASS",
>>                 "*** TEST BUILD: RTEMS_DEBUG RTEMS_POSIX_API",
>>                 "*** TEST TOOLS: 12.1.1 20220622 (RTEMS 6, RSB 
>> bf3f66c29a046ec1c5029e4f919b525947df6245, Newlib e7ffbdb)",
>>                 "GLOBAL: Hey I'm in base class constructor number 1 
>> for 0x214474.",
>>                 "GLOBAL: Hey I'm in base class constructor number 2 
>> for 0x214480.",
>>                 "GLOBAL: Hey I'm in derived class constructor number 3 
>> for 0x214480.",
>>                 "LOCAL: Hey I'm in base class constructor number 4 for 
>> 0x228864.",
>>         ...
>>
>>
>> "WRoZqtoO3A8AAAABDAAAAJkUSmRGNOBaaHW1UwAAoQGQ////AAAAAQwAAABck8Bx+K/znDWWTEcA", 
>>
>>
>> "AKEB2P///wAAAAEMAAAAVP6TRqQZY+4+srvAAAChAfD///8AAAABDAAAAMXOxS0Rhzqx6Old2wAA", 
>>
>>
>> "oQH4////AAAAAQwAAADJyx1ZCnZ4l+m/XkMAAKEB0P///wAAAAEMAAAAN3+9YAwMW8gTHIoPAACh", 
>>
>>
>> "Adj///8AAAABDAAAABtxsj3zXZF/+UqzfAAAoQHw////AAAAAQwAAAAwMDMeE7mphT6yu8AAAKEB", 
>>
>> "8P///wAAAAEMAAAAr2rLCcwzVnf5SrN8AAChAfD///8AAAAA",
>>                 "*** END OF GCOV INFO BASE64 ***",
>>                 ""
>>             ],
>>             "result": "passed",
>>             "start-time": "2022-06-28T12:08:47.721822+00:00"
>>         }
>>     ],
>>     "start-time": "2022-06-28T14:08:47.595131"
>> }
>>
> There are no publicly available report generators for the existing 
> format, but I do have one I use to generate STR documents for delivery 
> using mustache templates and further processing. The existing format was 
> designed to be easily consumable by that and other simple templating 
> mechanisms by providing structure (test subsets) and precalculating 
> values that would otherwise be implicit in the data. Changes to the 
> names of various fields should be easy to accommodate, but a more 
> significant restructure is going to present problems continuing with any 
> simple templating mechanism and would instead require a more 
> comprehensive document generator for all cases.
> 
> I'm not necessarily opposed to this change as it significantly 
> simplifies the report generation code, but it will definitely require 
> some rework in my processes. I can't speak for any users of the YAML 
> report format, but it was committed over a year and a half ago.

Ok, I will bring back the redundant data and use a consistent naming.

The YAML format was added for the RTEMS pre-qualification project, 
however, it is pretty useless since the YAML processing is very slow in 
Python. Also the test report generators need to be rewritten. This patch 
is a part of this work.

-- 
embedded brains GmbH
Herr Sebastian HUBER
Dornierstr. 4
82178 Puchheim
Germany
email: sebastian.huber at embedded-brains.de
phone: +49-89-18 94 741 - 16
fax:   +49-89-18 94 741 - 08

Registergericht: Amtsgericht München
Registernummer: HRB 157899
Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler
Unsere Datenschutzerklärung finden Sie hier:
https://embedded-brains.de/datenschutzerklaerung/