[tools] tester: Normalize JSON and YAML reports

[Kinsey Moore]

On 6/29/2022 04:34, Sebastian Huber wrote:
> On 29/06/2022 11:20, Chris Johns wrote:
>>
>>> On 29 Jun 2022, at 4:42 pm, Sebastian Huber 
>>> <sebastian.huber at embedded-brains.de> wrote:
>>>
>>> On 29/06/2022 08:40, Sebastian Huber wrote:
>>>> Report the same data in JSON and YAML reports.  Do not report 
>>>> redundant
>>>> information.
>>>> Update 4671.
>>>
>>> This patch changes the JSON reports. Are there already consumers for 
>>> the JSON reports so that we have to be backward compatible?
>>
>> Could compatibility be added back in if this proves to be an issue?
>>
>> I am wondering if that could be considered if comparability becomes 
>> an issue.
>
> The JSON report was added by:
>
> commit 5fcaf843bdb491e2da14dc1af61806ac6d20bae5
> Author: Kinsey Moore <kinsey.moore at oarcorp.com>
> Date:   Wed Aug 21 16:34:12 2019 +0000
>
>     Add JSON log generation
>
>     Add log formatter hooks and JSON log formatter to the test 
> infrastructure
>     for consumption by automated processes or report generators.
>
>
> I am not sure if these automated processes or report generators 
> already exist.
>
> The existing attribute names are quite inconsistent, for example 
> "Command Line", "passed_count", "wrong-version_count". I would use 
> lower case only with "-" as a separator. The JSON report should 
> contain all information of a test run.
>
> The new report looks like this:
>
> {
>     "command-line": [
>         "/opt/rtems/6/bin/rtems-test",
>         "--rtems-bsp=xilinx_zynq_a9_qemu",
>         "--report-format=json",
>         "--report-path=report",
>         "build/arm/xilinx_zynq_a9_qemu/testsuites/samples/"
>     ],
>     "end-time": "2022-06-28T14:08:47.595131",
>     "host": "Linux-5.3.18-150300.59.76-default-x86_64-with-glibc2.3.4 
> (Linux lupus 5.3.18-150300.59.76-default #1 SMP Thu Jun 16 04:23:47 
> UTC 2022 (2cc2ade) x86_64 x86_64)",
>     "python": "3.6.15 (default, Sep 23 2021, 15:41:43) [GCC]",
>     "reports": [
>         {
>             "arch": "arm",
>             "bsp": "xilinx_zynq_a9_qemu",
>             "command-line": "qemu-system-arm -no-reboot -nographic 
> -net none -serial null -serial mon:stdio -M xilinx-zynq-a9 -m 256M 
> -kernel build/arm/xilinx_zynq_a9_qemu/testsuites/samples/cdtest.exe",
>             "end-time": "2022-06-28T12:08:48.161691+00:00",
>             "executable": 
> "build/arm/xilinx_zynq_a9_qemu/testsuites/samples/cdtest.exe",
>             "executable-sha512": 
> "413bc8a506053c95a0e1d93942cce9fd0d653f7b3cde42623085ce147aea324f4baeff7ce0102008322fb1341bc2b38defd3925a64144e47e13cc5f32185bf78",
>             "output": [
>                 "qemu-system-arm: warning: nic cadence_gem.0 has no 
> peer",
>                 "qemu-system-arm: warning: nic cadence_gem.1 has no 
> peer",
>                 "",
>                 "",
>                 "*** BEGIN OF TEST CONSTRUCTOR/DESTRUCTOR ***",
>                 "*** TEST VERSION: 
> 6.0.0.3302b72754df5f37214e86dd68522189857772c7",
>                 "*** TEST STATE: EXPECTED_PASS",
>                 "*** TEST BUILD: RTEMS_DEBUG RTEMS_POSIX_API",
>                 "*** TEST TOOLS: 12.1.1 20220622 (RTEMS 6, RSB 
> bf3f66c29a046ec1c5029e4f919b525947df6245, Newlib e7ffbdb)",
>                 "GLOBAL: Hey I'm in base class constructor number 1 
> for 0x214474.",
>                 "GLOBAL: Hey I'm in base class constructor number 2 
> for 0x214480.",
>                 "GLOBAL: Hey I'm in derived class constructor number 3 
> for 0x214480.",
>                 "LOCAL: Hey I'm in base class constructor number 4 for 
> 0x228864.",
>         ...
>
>
> "WRoZqtoO3A8AAAABDAAAAJkUSmRGNOBaaHW1UwAAoQGQ////AAAAAQwAAABck8Bx+K/znDWWTEcA", 
>
>
> "AKEB2P///wAAAAEMAAAAVP6TRqQZY+4+srvAAAChAfD///8AAAABDAAAAMXOxS0Rhzqx6Old2wAA", 
>
>
> "oQH4////AAAAAQwAAADJyx1ZCnZ4l+m/XkMAAKEB0P///wAAAAEMAAAAN3+9YAwMW8gTHIoPAACh", 
>
>
> "Adj///8AAAABDAAAABtxsj3zXZF/+UqzfAAAoQHw////AAAAAQwAAAAwMDMeE7mphT6yu8AAAKEB", 
>
> "8P///wAAAAEMAAAAr2rLCcwzVnf5SrN8AAChAfD///8AAAAA",
>                 "*** END OF GCOV INFO BASE64 ***",
>                 ""
>             ],
>             "result": "passed",
>             "start-time": "2022-06-28T12:08:47.721822+00:00"
>         }
>     ],
>     "start-time": "2022-06-28T14:08:47.595131"
> }
>
There are no publicly available report generators for the existing 
format, but I do have one I use to generate STR documents for delivery 
using mustache templates and further processing. The existing format was 
designed to be easily consumable by that and other simple templating 
mechanisms by providing structure (test subsets) and precalculating 
values that would otherwise be implicit in the data. Changes to the 
names of various fields should be easy to accommodate, but a more 
significant restructure is going to present problems continuing with any 
simple templating mechanism and would instead require a more 
comprehensive document generator for all cases.

I'm not necessarily opposed to this change as it significantly 
simplifies the report generation code, but it will definitely require 
some rework in my processes. I can't speak for any users of the YAML 
report format, but it was committed over a year and a half ago.


Kinsey