[GSoC] RTEMS Tester Improvements

[Chris Johns]

On 14/03/2017 03:30, Gedare Bloom wrote:
> On Mon, Mar 13, 2017 at 11:50 AM, Tanu Hari Dixit <tokencolour at gmail.com> wrote:
>> Hello all,
>>
>> I am interested in applying for GSoC under RTEMS. I am interested in the
>> idea RTEMS Tester Improvements (https://devel.rtems.org/ticket/2919). I have
>> used rtems-tester in the past and also have a working knowledge of python.
>>
> What platforms have you run tester on?
>
>> I have a few questions and will be grateful if they are answered.
>> 1) Which simulator recipes need to be added?
>
> The first place to look is to compare what is supported in tester
> versus what we have scripts for in the older rtems-testing.git repo
> under sim-scripts/
> https://git.rtems.org/rtems-testing/tree/sim-scripts
>
> I personally would also be interested to add simulator recipes for the
> gem5 (gem5.org) simulator. I made a start at this a long time ago and
> got it to work reasonably well, but appear to have misplaced the
> actual changes. Related to this would be adding complementary recipes
> for building simulators in the RSB. I have run rtems under gem5 for
> the sparc64/usiii and arm/realview_pbx_a9_qemu BSPs in the past and
> would be able to help with simulator setup to get you to the point of
> transitioning from manual/shell-scripted steps into python automation.
>
>> 2) What are the improvements that need to be done? I couldn't find the
>> appropriate tickets or pointers.
>
> I believe it is primarily to increase the range of simulators that are
> supported. Perhaps Chris has other ideas for infrastructure
> improvements, e.g. there is almost always improvements that can be
> made to report usability such as visualizing aggregated testing
> results, tracking down failed tests quicker, and so forth.

The follow is the work I see needing to be done.

The current format for the BSPs and scripting is crypt and I am 
wondering if moving to YAML would help make it simpler. On top of a YAML 
change we can add per site and/or per user specifics that allow local 
hardware to be used. For example line 58 and 59 in the script 
https://git.rtems.org/rtems-tools/tree/tester/rtems/testing/bsps/xilinx_zynq_zc706.mc#n58 
are only useful to me and it is wrong to have this data in the repo. We 
need a way to allow users to provide the needed values they can 
configuration control for their project.

Further to this there is a need to add support for testing state 
'expected-fail' I added to the testsuite in RTEMS. We also need to add 
to each BSP the expected test results so this tool can evaluate regressions.

Chris