[PATCH v2] Add JSON log generation
Chris Johns
chrisj at rtems.org
Wed Aug 21 01:06:46 UTC 2019
On 20/8/19 11:22 pm, Kinsey Moore wrote:
>> -----Original Message-----
>> From: Chris Johns <chrisj at rtems.org>
>> Sent: Monday, August 19, 2019 17:33
>> To: Kinsey Moore <kinsey.moore at oarcorp.com>; devel at rtems.org
>> Subject: Re: [PATCH v2] Add JSON log generation
>>
>> On 20/8/19 2:13 am, Kinsey Moore wrote:
>>> Add log formatter hooks and JSON log formatter to the test
>>> infrastructure for consumption by automated processes or report generators.
>>> ---
>>> tester/rt/test.py | 84
>>> +++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> 1 file changed, 84 insertions(+)
>>>
>>> diff --git a/tester/rt/test.py b/tester/rt/test.py index
>>> da0a11e..0ed799a 100644
>>> --- a/tester/rt/test.py
>>> +++ b/tester/rt/test.py
>>> @@ -38,6 +38,7 @@ import re
>>> import sys
>>> import threading
>>> import time
>>> +import json
>>
>> The import list is sorted :)
>
> I'll move it into the formatter to reduce scope and sort it there.
Good idea.
>>
>>> from rtemstoolkit import configuration from rtemstoolkit import
>>> error @@ -217,6 +218,69 @@ def killall(tests):
>>> for test in tests:
>>> test.kill()
>>>
>>> +def generate_json_report(args, reports, start_time, end_time, total,
>> json_file):
>>> + import sys
>>> + json_log = {}
>>> + json_log['Command Line'] = " ".join(args)
>>> + json_log['Python'] = sys.version.replace('\n', '')
>>> + json_log['test_groups'] = []
>>> + json_log['Host'] = host.label(mode = 'all')
>>> + json_log['summary'] = {}
>>> + json_log['summary']['passed_count'] = reports.passed
>>> + json_log['summary']['failed_count'] = reports.failed
>>> + json_log['summary']['user-input_count'] = reports.user_input
>>> + json_log['summary']['expected-fail_count'] = reports.expected_fail
>>> + json_log['summary']['indeterminate_count'] = reports.indeterminate
>>> + json_log['summary']['benchmark_count'] = reports.benchmark
>>> + json_log['summary']['timeout_count'] = reports.timeouts
>>> + json_log['summary']['invalid_count'] = reports.invalids
>>> + json_log['summary']['wrong-version_count'] = reports.wrong_version
>>> + json_log['summary']['wrong-build_count'] = reports.wrong_build
>>> + json_log['summary']['wrong-tools_count'] = reports.wrong_tools
>>> + json_log['summary']['total_count'] = reports.total
>>> + json_log['summary']['average_test_time'] = str((end_time - start_time) /
>> total)
>>> + json_log['summary']['testing_time'] = str(end_time - start_time)
>>> +
>>> + result_types = ['failed', 'user-input', 'expected-fail',
>>> + 'indeterminate', 'benchmark', 'timeout', 'invalid', 'wrong-version',
>>> + 'wrong-build', 'wrong-tools']
>>
>> There is a soft'ish limit that attempts to use 80 columns in the python code.
>> This one is too long.
>>
>>> + json_results = {}
>>> + for result_type in result_types:
>>> + json_log['summary'][result_type] = []
>>> +
>>> + # collate results for JSON log
>>> + for name in reports.results:
>>> + result_type = reports.results[name]['result']
>>> + test_parts = name.split("/")
>>> + test_category = test_parts[-2]
>>> + test_name = test_parts[-1]
>>> + if result_type != 'passed':
>>> + json_log['summary'][result_type].append(test_name)
>>> + if test_category not in json_results:
>>> + json_results[test_category] = []
>>> + json_result = {}
>>> + # remove the file extension
>>> + json_result["name"] = test_name.split('.')[0]
>>> + json_result["result"] = result_type
>>> + if result_type == "failed" or result_type == "timeout":
>>> + json_result["output"] = reports.results[name]["output"]
>>> + json_results[test_category].append(json_result)
>>> +
>>> + # convert results to a better format for report generation
>>> + sorted_keys = sorted(json_results.keys())
>>> + for i in range(len(sorted_keys)):
>>> + results_log = {}
>>> + results_log["index"] = i + 1
>>> + results_log["name"] = sorted_keys[i]
>>> + results_log["results"] = json_results[sorted_keys[i]]
>>> + json_log["test_groups"].append(results_log)
>>> +
>>> + # write out JSON log
>>> + with open(json_file, 'w') as outfile:
>>> + json.dump(json_log, outfile, sort_keys=True, indent=4)
>>> +
>>> +report_formatters = {
>>> + 'json': generate_json_report
>>> +}
>>> +
>>> def run(args, command_path = None):
>>> import sys
>>> tests = []
>>> @@ -227,6 +291,8 @@ def run(args, command_path = None):
>>> optargs = { '--rtems-tools': 'The path to the RTEMS tools',
>>> '--rtems-bsp': 'The RTEMS BSP to run the test on',
>>> '--user-config': 'Path to your local user configuration INI file',
>>> + '--report-format': 'Formats in which to report test results in
>> addition to txt: json',
>>> + '--log': 'Log output location',
>>
>> Is this option already taken by the options.py module which imports the
>> rtemstoolkit's options? Would --report work?
>
> The thought here was to use the same log location that was already being provided for that option. The options framework in use wouldn't let me grab the value of --log unless I defined it there. It may be better to separate reports from logs a bit more.
There is no way to turn off the current log so the file will be txt and json or
json will wipe away the log neither I think work.
I think a log is useful when tracking down errors and you may still want the
json to export the results to your configuration management system.
>>> '--report-mode': 'Reporting modes, failures (default),all,none',
>>
>> I wonder if this is now looking a bit confusing?
>
> I agree. If we're differentiating between reports and logs, this is more of a log mode than a report mode. Unfortunately, the terminology is already mixed and they're both delivering the same content in different formats.
OK, then lets change the `--report-mode` option to `--log-mode` so reports can
be left as JSON and whatever we add. I think this change till trigger a doco
update as well. :)
Thanks
Chris
More information about the devel
mailing list