[tools] tester: Normalize JSON and YAML reports

Thu Jun 30 14:08:44 UTC 2022

On 6/30/2022 01:34, Sebastian Huber wrote:
>
>
> On 30/06/2022 07:58, Sebastian Huber wrote:
>> On 29/06/2022 17:54, Kinsey Moore wrote:
>>> On 6/29/2022 04:34, Sebastian Huber wrote:
>>>> On 29/06/2022 11:20, Chris Johns wrote:
>>>>>
>>>>>> On 29 Jun 2022, at 4:42 pm, Sebastian Huber 
>>>>>> <sebastian.huber at embedded-brains.de> wrote:
>>>>>>
>>>>>> On 29/06/2022 08:40, Sebastian Huber wrote:
>>>>>>> Report the same data in JSON and YAML reports.  Do not report 
>>>>>>> redundant
>>>>>>> information.
>>>>>>> Update 4671.
>>>>>>
>>>>>> This patch changes the JSON reports. Are there already consumers 
>>>>>> for the JSON reports so that we have to be backward compatible?
>>>>>
>>>>> Could compatibility be added back in if this proves to be an issue?
>>>>>
>>>>> I am wondering if that could be considered if comparability 
>>>>> becomes an issue.
>>>>
>>>> The JSON report was added by:
>>>>
>>>> commit 5fcaf843bdb491e2da14dc1af61806ac6d20bae5
>>>> Author: Kinsey Moore <kinsey.moore at oarcorp.com>
>>>> Date:   Wed Aug 21 16:34:12 2019 +0000
>>>>
>>>>     Add JSON log generation
>>>>
>>>>     Add log formatter hooks and JSON log formatter to the test 
>>>> infrastructure
>>>>     for consumption by automated processes or report generators.
>>>>
>>>>
>>>> I am not sure if these automated processes or report generators 
>>>> already exist.
>>>>
>>>> The existing attribute names are quite inconsistent, for example 
>>>> "Command Line", "passed_count", "wrong-version_count". I would use 
>>>> lower case only with "-" as a separator. The JSON report should 
>>>> contain all information of a test run.
>>>>
>>>> The new report looks like this:
>>>>
>>>> {
>>>>     "command-line": [
>>>>         "/opt/rtems/6/bin/rtems-test",
>>>>         "--rtems-bsp=xilinx_zynq_a9_qemu",
>>>>         "--report-format=json",
>>>>         "--report-path=report",
>>>>         "build/arm/xilinx_zynq_a9_qemu/testsuites/samples/"
>>>>     ],
>>>>     "end-time": "2022-06-28T14:08:47.595131",
>>>>     "host": 
>>>> "Linux-5.3.18-150300.59.76-default-x86_64-with-glibc2.3.4 (Linux 
>>>> lupus 5.3.18-150300.59.76-default #1 SMP Thu Jun 16 04:23:47 UTC 
>>>> 2022 (2cc2ade) x86_64 x86_64)",
>>>>     "python": "3.6.15 (default, Sep 23 2021, 15:41:43) [GCC]",
>>>>     "reports": [
>>>>         {
>>>>             "arch": "arm",
>>>>             "bsp": "xilinx_zynq_a9_qemu",
>>>>             "command-line": "qemu-system-arm -no-reboot -nographic 
>>>> -net none -serial null -serial mon:stdio -M xilinx-zynq-a9 -m 256M 
>>>> -kernel build/arm/xilinx_zynq_a9_qemu/testsuites/samples/cdtest.exe",
>>>>             "end-time": "2022-06-28T12:08:48.161691+00:00",
>>>>             "executable": 
>>>> "build/arm/xilinx_zynq_a9_qemu/testsuites/samples/cdtest.exe",
>>>>             "executable-sha512": 
>>>> "413bc8a506053c95a0e1d93942cce9fd0d653f7b3cde42623085ce147aea324f4baeff7ce0102008322fb1341bc2b38defd3925a64144e47e13cc5f32185bf78", 
>>>>
>>>>             "output": [
>>>>                 "qemu-system-arm: warning: nic cadence_gem.0 has no 
>>>> peer",
>>>>                 "qemu-system-arm: warning: nic cadence_gem.1 has no 
>>>> peer",
>>>>                 "",
>>>>                 "",
>>>>                 "*** BEGIN OF TEST CONSTRUCTOR/DESTRUCTOR ***",
>>>>                 "*** TEST VERSION: 
>>>> 6.0.0.3302b72754df5f37214e86dd68522189857772c7",
>>>>                 "*** TEST STATE: EXPECTED_PASS",
>>>>                 "*** TEST BUILD: RTEMS_DEBUG RTEMS_POSIX_API",
>>>>                 "*** TEST TOOLS: 12.1.1 20220622 (RTEMS 6, RSB 
>>>> bf3f66c29a046ec1c5029e4f919b525947df6245, Newlib e7ffbdb)",
>>>>                 "GLOBAL: Hey I'm in base class constructor number 1 
>>>> for 0x214474.",
>>>>                 "GLOBAL: Hey I'm in base class constructor number 2 
>>>> for 0x214480.",
>>>>                 "GLOBAL: Hey I'm in derived class constructor 
>>>> number 3 for 0x214480.",
>>>>                 "LOCAL: Hey I'm in base class constructor number 4 
>>>> for 0x228864.",
>>>>         ...
>>>>
>>>>
>>>> "WRoZqtoO3A8AAAABDAAAAJkUSmRGNOBaaHW1UwAAoQGQ////AAAAAQwAAABck8Bx+K/znDWWTEcA", 
>>>>
>>>>
>>>> "AKEB2P///wAAAAEMAAAAVP6TRqQZY+4+srvAAAChAfD///8AAAABDAAAAMXOxS0Rhzqx6Old2wAA", 
>>>>
>>>>
>>>> "oQH4////AAAAAQwAAADJyx1ZCnZ4l+m/XkMAAKEB0P///wAAAAEMAAAAN3+9YAwMW8gTHIoPAACh", 
>>>>
>>>>
>>>> "Adj///8AAAABDAAAABtxsj3zXZF/+UqzfAAAoQHw////AAAAAQwAAAAwMDMeE7mphT6yu8AAAKEB", 
>>>>
>>>> "8P///wAAAAEMAAAAr2rLCcwzVnf5SrN8AAChAfD///8AAAAA",
>>>>                 "*** END OF GCOV INFO BASE64 ***",
>>>>                 ""
>>>>             ],
>>>>             "result": "passed",
>>>>             "start-time": "2022-06-28T12:08:47.721822+00:00"
>>>>         }
>>>>     ],
>>>>     "start-time": "2022-06-28T14:08:47.595131"
>>>> }
>>>>
>>> There are no publicly available report generators for the existing 
>>> format, but I do have one I use to generate STR documents for 
>>> delivery using mustache templates and further processing. The 
>>> existing format was designed to be easily consumable by that and 
>>> other simple templating mechanisms by providing structure (test 
>>> subsets) and precalculating values that would otherwise be implicit 
>>> in the data. Changes to the names of various fields should be easy 
>>> to accommodate, but a more significant restructure is going to 
>>> present problems continuing with any simple templating mechanism and 
>>> would instead require a more comprehensive document generator for 
>>> all cases.
>>>
>>> I'm not necessarily opposed to this change as it significantly 
>>> simplifies the report generation code, but it will definitely 
>>> require some rework in my processes. I can't speak for any users of 
>>> the YAML report format, but it was committed over a year and a half 
>>> ago.
>>
>> Ok, I will bring back the redundant data and use a consistent naming.
>>
>> The YAML format was added for the RTEMS pre-qualification project, 
>> however, it is pretty useless since the YAML processing is very slow 
>> in Python. Also the test report generators need to be rewritten. This 
>> patch is a part of this work.
>
> Do you really need this "test_groups" stuff?
>
>     "test_groups": [
>         {
>             "index": 1,
>             "name": "benchmarks",
>             "results": [
>                 {
>                     "name": "dhrystone",
>                     "result": "benchmark"
>                 },
>
> From my point of view the JSON/YAML reports should not satisfy every 
> possible consumer directly.
>
This structure was put in place because it's a logical way to divide the 
tests for reporting. Simple templating mechanisms can remove structure, 
but they can't add it. I never used "index", so that can be removed.

There is also an issue with the test groups in that they don't really 
apply to tests that exist outside of the RTEMS testsuite. If we're going 
to remove this kind of structure, your original patch is fine since I'll 
need to do non-trivial processing on the reports, anyway.

Kinsey