[tools] tester: Normalize JSON and YAML reports

Thu Jun 30 06:34:03 UTC 2022

On 30/06/2022 07:58, Sebastian Huber wrote:
> On 29/06/2022 17:54, Kinsey Moore wrote:
>> On 6/29/2022 04:34, Sebastian Huber wrote:
>>> On 29/06/2022 11:20, Chris Johns wrote:
>>>>
>>>>> On 29 Jun 2022, at 4:42 pm, Sebastian Huber 
>>>>> <sebastian.huber at embedded-brains.de> wrote:
>>>>>
>>>>> On 29/06/2022 08:40, Sebastian Huber wrote:
>>>>>> Report the same data in JSON and YAML reports.  Do not report 
>>>>>> redundant
>>>>>> information.
>>>>>> Update 4671.
>>>>>
>>>>> This patch changes the JSON reports. Are there already consumers 
>>>>> for the JSON reports so that we have to be backward compatible?
>>>>
>>>> Could compatibility be added back in if this proves to be an issue?
>>>>
>>>> I am wondering if that could be considered if comparability becomes 
>>>> an issue.
>>>
>>> The JSON report was added by:
>>>
>>> commit 5fcaf843bdb491e2da14dc1af61806ac6d20bae5
>>> Author: Kinsey Moore <kinsey.moore at oarcorp.com>
>>> Date:   Wed Aug 21 16:34:12 2019 +0000
>>>
>>>     Add JSON log generation
>>>
>>>     Add log formatter hooks and JSON log formatter to the test 
>>> infrastructure
>>>     for consumption by automated processes or report generators.
>>>
>>>
>>> I am not sure if these automated processes or report generators 
>>> already exist.
>>>
>>> The existing attribute names are quite inconsistent, for example 
>>> "Command Line", "passed_count", "wrong-version_count". I would use 
>>> lower case only with "-" as a separator. The JSON report should 
>>> contain all information of a test run.
>>>
>>> The new report looks like this:
>>>
>>> {
>>>     "command-line": [
>>>         "/opt/rtems/6/bin/rtems-test",
>>>         "--rtems-bsp=xilinx_zynq_a9_qemu",
>>>         "--report-format=json",
>>>         "--report-path=report",
>>>         "build/arm/xilinx_zynq_a9_qemu/testsuites/samples/"
>>>     ],
>>>     "end-time": "2022-06-28T14:08:47.595131",
>>>     "host": "Linux-5.3.18-150300.59.76-default-x86_64-with-glibc2.3.4 
>>> (Linux lupus 5.3.18-150300.59.76-default #1 SMP Thu Jun 16 04:23:47 
>>> UTC 2022 (2cc2ade) x86_64 x86_64)",
>>>     "python": "3.6.15 (default, Sep 23 2021, 15:41:43) [GCC]",
>>>     "reports": [
>>>         {
>>>             "arch": "arm",
>>>             "bsp": "xilinx_zynq_a9_qemu",
>>>             "command-line": "qemu-system-arm -no-reboot -nographic 
>>> -net none -serial null -serial mon:stdio -M xilinx-zynq-a9 -m 256M 
>>> -kernel build/arm/xilinx_zynq_a9_qemu/testsuites/samples/cdtest.exe",
>>>             "end-time": "2022-06-28T12:08:48.161691+00:00",
>>>             "executable": 
>>> "build/arm/xilinx_zynq_a9_qemu/testsuites/samples/cdtest.exe",
>>>             "executable-sha512": 
>>> "413bc8a506053c95a0e1d93942cce9fd0d653f7b3cde42623085ce147aea324f4baeff7ce0102008322fb1341bc2b38defd3925a64144e47e13cc5f32185bf78", 
>>>
>>>             "output": [
>>>                 "qemu-system-arm: warning: nic cadence_gem.0 has no 
>>> peer",
>>>                 "qemu-system-arm: warning: nic cadence_gem.1 has no 
>>> peer",
>>>                 "",
>>>                 "",
>>>                 "*** BEGIN OF TEST CONSTRUCTOR/DESTRUCTOR ***",
>>>                 "*** TEST VERSION: 
>>> 6.0.0.3302b72754df5f37214e86dd68522189857772c7",
>>>                 "*** TEST STATE: EXPECTED_PASS",
>>>                 "*** TEST BUILD: RTEMS_DEBUG RTEMS_POSIX_API",
>>>                 "*** TEST TOOLS: 12.1.1 20220622 (RTEMS 6, RSB 
>>> bf3f66c29a046ec1c5029e4f919b525947df6245, Newlib e7ffbdb)",
>>>                 "GLOBAL: Hey I'm in base class constructor number 1 
>>> for 0x214474.",
>>>                 "GLOBAL: Hey I'm in base class constructor number 2 
>>> for 0x214480.",
>>>                 "GLOBAL: Hey I'm in derived class constructor number 
>>> 3 for 0x214480.",
>>>                 "LOCAL: Hey I'm in base class constructor number 4 
>>> for 0x228864.",
>>>         ...
>>>
>>>
>>> "WRoZqtoO3A8AAAABDAAAAJkUSmRGNOBaaHW1UwAAoQGQ////AAAAAQwAAABck8Bx+K/znDWWTEcA", 
>>>
>>>
>>> "AKEB2P///wAAAAEMAAAAVP6TRqQZY+4+srvAAAChAfD///8AAAABDAAAAMXOxS0Rhzqx6Old2wAA", 
>>>
>>>
>>> "oQH4////AAAAAQwAAADJyx1ZCnZ4l+m/XkMAAKEB0P///wAAAAEMAAAAN3+9YAwMW8gTHIoPAACh", 
>>>
>>>
>>> "Adj///8AAAABDAAAABtxsj3zXZF/+UqzfAAAoQHw////AAAAAQwAAAAwMDMeE7mphT6yu8AAAKEB", 
>>>
>>> "8P///wAAAAEMAAAAr2rLCcwzVnf5SrN8AAChAfD///8AAAAA",
>>>                 "*** END OF GCOV INFO BASE64 ***",
>>>                 ""
>>>             ],
>>>             "result": "passed",
>>>             "start-time": "2022-06-28T12:08:47.721822+00:00"
>>>         }
>>>     ],
>>>     "start-time": "2022-06-28T14:08:47.595131"
>>> }
>>>
>> There are no publicly available report generators for the existing 
>> format, but I do have one I use to generate STR documents for delivery 
>> using mustache templates and further processing. The existing format 
>> was designed to be easily consumable by that and other simple 
>> templating mechanisms by providing structure (test subsets) and 
>> precalculating values that would otherwise be implicit in the data. 
>> Changes to the names of various fields should be easy to accommodate, 
>> but a more significant restructure is going to present problems 
>> continuing with any simple templating mechanism and would instead 
>> require a more comprehensive document generator for all cases.
>>
>> I'm not necessarily opposed to this change as it significantly 
>> simplifies the report generation code, but it will definitely require 
>> some rework in my processes. I can't speak for any users of the YAML 
>> report format, but it was committed over a year and a half ago.
> 
> Ok, I will bring back the redundant data and use a consistent naming.
> 
> The YAML format was added for the RTEMS pre-qualification project, 
> however, it is pretty useless since the YAML processing is very slow in 
> Python. Also the test report generators need to be rewritten. This patch 
> is a part of this work.

Do you really need this "test_groups" stuff?

     "test_groups": [
         {
             "index": 1,
             "name": "benchmarks",
             "results": [
                 {
                     "name": "dhrystone",
                     "result": "benchmark"
                 },

 From my point of view the JSON/YAML reports should not satisfy every 
possible consumer directly.

-- 
embedded brains GmbH
Herr Sebastian HUBER
Dornierstr. 4
82178 Puchheim
Germany
email: sebastian.huber at embedded-brains.de
phone: +49-89-18 94 741 - 16
fax:   +49-89-18 94 741 - 08

Registergericht: Amtsgericht München
Registernummer: HRB 157899
Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler
Unsere Datenschutzerklärung finden Sie hier:
https://embedded-brains.de/datenschutzerklaerung/