[tools] tester: Normalize JSON and YAML reports

Thu Jun 30 23:12:41 UTC 2022

On 30/6/2022 4:34 pm, Sebastian Huber wrote:
> On 30/06/2022 07:58, Sebastian Huber wrote:
>> On 29/06/2022 17:54, Kinsey Moore wrote:
>>> On 6/29/2022 04:34, Sebastian Huber wrote:
>>>> On 29/06/2022 11:20, Chris Johns wrote:
>>>>>
>>>>>> On 29 Jun 2022, at 4:42 pm, Sebastian Huber
>>>>>> <sebastian.huber at embedded-brains.de> wrote:
>>>>>>
>>>>>> On 29/06/2022 08:40, Sebastian Huber wrote:
>>>>>>> Report the same data in JSON and YAML reports.  Do not report redundant
>>>>>>> information.
>>>>>>> Update 4671.
>>>>>>
>>>>>> This patch changes the JSON reports. Are there already consumers for the
>>>>>> JSON reports so that we have to be backward compatible?
>>>>>
>>>>> Could compatibility be added back in if this proves to be an issue?
>>>>>
>>>>> I am wondering if that could be considered if comparability becomes an issue.
>>>>
>>>> The JSON report was added by:
>>>>
>>>> commit 5fcaf843bdb491e2da14dc1af61806ac6d20bae5
>>>> Author: Kinsey Moore <kinsey.moore at oarcorp.com>
>>>> Date:   Wed Aug 21 16:34:12 2019 +0000
>>>>
>>>>     Add JSON log generation
>>>>
>>>>     Add log formatter hooks and JSON log formatter to the test infrastructure
>>>>     for consumption by automated processes or report generators.
>>>>
>>>>
>>>> I am not sure if these automated processes or report generators already exist.
>>>>
>>>> The existing attribute names are quite inconsistent, for example "Command
>>>> Line", "passed_count", "wrong-version_count". I would use lower case only
>>>> with "-" as a separator. The JSON report should contain all information of a
>>>> test run.
>>>>
>>>> The new report looks like this:
>>>>
>>>> {
>>>>     "command-line": [
>>>>         "/opt/rtems/6/bin/rtems-test",
>>>>         "--rtems-bsp=xilinx_zynq_a9_qemu",
>>>>         "--report-format=json",
>>>>         "--report-path=report",
>>>>         "build/arm/xilinx_zynq_a9_qemu/testsuites/samples/"
>>>>     ],
>>>>     "end-time": "2022-06-28T14:08:47.595131",
>>>>     "host": "Linux-5.3.18-150300.59.76-default-x86_64-with-glibc2.3.4 (Linux
>>>> lupus 5.3.18-150300.59.76-default #1 SMP Thu Jun 16 04:23:47 UTC 2022
>>>> (2cc2ade) x86_64 x86_64)",
>>>>     "python": "3.6.15 (default, Sep 23 2021, 15:41:43) [GCC]",
>>>>     "reports": [
>>>>         {
>>>>             "arch": "arm",
>>>>             "bsp": "xilinx_zynq_a9_qemu",
>>>>             "command-line": "qemu-system-arm -no-reboot -nographic -net none
>>>> -serial null -serial mon:stdio -M xilinx-zynq-a9 -m 256M -kernel
>>>> build/arm/xilinx_zynq_a9_qemu/testsuites/samples/cdtest.exe",
>>>>             "end-time": "2022-06-28T12:08:48.161691+00:00",
>>>>             "executable":
>>>> "build/arm/xilinx_zynq_a9_qemu/testsuites/samples/cdtest.exe",
>>>>             "executable-sha512":
>>>> "413bc8a506053c95a0e1d93942cce9fd0d653f7b3cde42623085ce147aea324f4baeff7ce0102008322fb1341bc2b38defd3925a64144e47e13cc5f32185bf78",
>>>>
>>>>             "output": [
>>>>                 "qemu-system-arm: warning: nic cadence_gem.0 has no peer",
>>>>                 "qemu-system-arm: warning: nic cadence_gem.1 has no peer",
>>>>                 "",
>>>>                 "",
>>>>                 "*** BEGIN OF TEST CONSTRUCTOR/DESTRUCTOR ***",
>>>>                 "*** TEST VERSION:
>>>> 6.0.0.3302b72754df5f37214e86dd68522189857772c7",
>>>>                 "*** TEST STATE: EXPECTED_PASS",
>>>>                 "*** TEST BUILD: RTEMS_DEBUG RTEMS_POSIX_API",
>>>>                 "*** TEST TOOLS: 12.1.1 20220622 (RTEMS 6, RSB
>>>> bf3f66c29a046ec1c5029e4f919b525947df6245, Newlib e7ffbdb)",
>>>>                 "GLOBAL: Hey I'm in base class constructor number 1 for
>>>> 0x214474.",
>>>>                 "GLOBAL: Hey I'm in base class constructor number 2 for
>>>> 0x214480.",
>>>>                 "GLOBAL: Hey I'm in derived class constructor number 3 for
>>>> 0x214480.",
>>>>                 "LOCAL: Hey I'm in base class constructor number 4 for
>>>> 0x228864.",
>>>>         ...
>>>>
>>>>
>>>> "WRoZqtoO3A8AAAABDAAAAJkUSmRGNOBaaHW1UwAAoQGQ////AAAAAQwAAABck8Bx+K/znDWWTEcA",
>>>>
>>>> "AKEB2P///wAAAAEMAAAAVP6TRqQZY+4+srvAAAChAfD///8AAAABDAAAAMXOxS0Rhzqx6Old2wAA",
>>>>
>>>> "oQH4////AAAAAQwAAADJyx1ZCnZ4l+m/XkMAAKEB0P///wAAAAEMAAAAN3+9YAwMW8gTHIoPAACh",
>>>>
>>>> "Adj///8AAAABDAAAABtxsj3zXZF/+UqzfAAAoQHw////AAAAAQwAAAAwMDMeE7mphT6yu8AAAKEB",
>>>> "8P///wAAAAEMAAAAr2rLCcwzVnf5SrN8AAChAfD///8AAAAA",
>>>>                 "*** END OF GCOV INFO BASE64 ***",
>>>>                 ""
>>>>             ],
>>>>             "result": "passed",
>>>>             "start-time": "2022-06-28T12:08:47.721822+00:00"
>>>>         }
>>>>     ],
>>>>     "start-time": "2022-06-28T14:08:47.595131"
>>>> }
>>>>
>>> There are no publicly available report generators for the existing format,
>>> but I do have one I use to generate STR documents for delivery using mustache
>>> templates and further processing. The existing format was designed to be
>>> easily consumable by that and other simple templating mechanisms by providing
>>> structure (test subsets) and precalculating values that would otherwise be
>>> implicit in the data. Changes to the names of various fields should be easy
>>> to accommodate, but a more significant restructure is going to present
>>> problems continuing with any simple templating mechanism and would instead
>>> require a more comprehensive document generator for all cases.
>>>
>>> I'm not necessarily opposed to this change as it significantly simplifies the
>>> report generation code, but it will definitely require some rework in my
>>> processes. I can't speak for any users of the YAML report format, but it was
>>> committed over a year and a half ago.
>>
>> Ok, I will bring back the redundant data and use a consistent naming.
>>
>> The YAML format was added for the RTEMS pre-qualification project, however, it
>> is pretty useless since the YAML processing is very slow in Python. Also the
>> test report generators need to be rewritten. This patch is a part of this work.
> 
> Do you really need this "test_groups" stuff?
> 
>     "test_groups": [
>         {
>             "index": 1,
>             "name": "benchmarks",
>             "results": [
>                 {
>                     "name": "dhrystone",
>                     "result": "benchmark"
>                 },
> 
> From my point of view the JSON/YAML reports should not satisfy every possible
> consumer directly.

How do you determine the list you are satisfying? I can see this argument
working in two directions.

If there a technical issue with providing this data?

Does the output provide format metadata to determine the version of the output?

Chris