[RTEMS Project] #2927: RTEMS Testing Tool Project

[RTEMS trac]

#2927: RTEMS Testing Tool Project
-------------------------+-----------------------------
 Reporter:  Chris Johns  |      Owner:  joel.sherrill@…
     Type:  project      |     Status:  new
 Priority:  normal       |  Milestone:  Indefinite
Component:  testing      |    Version:  4.12
 Severity:  normal       |   Keywords:  SoC, testing
-------------------------+-----------------------------
 = RTEMS Testing Tool Project =

 [[PageOutline(1-5, Contents, inline)]]

 == Mentors ==
 Chris Johns

 == Status ==

 Looking for funding.

 = Introduction =

 This project involves working on the RTEMS Testing Tool that resides in
 the RTEMS Tools package and is part of the RTEMS Tools Project. The RTEMS
 Testing Tool provides a consistent way to test BSPs on different hardware.
 The tool aims to integrate with a wide range of testing tools and methods
 generating a consistent report for the RTEMS test suite.

 = Project =

 The project is to enhance and improve some of the existing functionality
 and to complete needed parts.

 == Goal ==

 * Make the configuration file format easier to use.
 * Allows users to customise and change behaviour without changing
 configuration files.
 * To be able to determine if a change in RTEMS causes a regression.
 * To be able to verify simulator and real hardware results match for a
 BSP.
 * Improve console support.

 == Prerequisite ==

 * Knowledge of C and Python programming languages.
 * Knowledge of host software and building packages such as simulators.
 * Knowledge of debugging and debuggers.
 * Knowledge of the RTEMS Tool's Toolkit.
 * Requires Unix (Linux or FreeBSD) host.
 * Requires modern PC hardware. Building all tests and running takes.
 * Optionally a Zedboard or MicroZed with a JTAG debugging pod.
 * Optionally a WIndows 7 or Windows 10 machine.

 == Resources ==

 * Current RTEMS developers.
 * The existing RTEMS Testing Tool.

 = Tasks =

 The following are the tasks:

 == Tools ==

 1. Build RTEMS `erc32` and `xilinx_zynq_a9_qemu` with all tests.
 1. Build QEMU with required patches to run the Zynq BSP executables.
 1. Verify `rtems-test` runs all tests and generates a report.

 == Configuration Files ==

 The `rtems-test` command uses the RTEMS Tool's Toolkit macro support to
 read macro files. This file format is a simple text version of the macro
 fields and is loaded into the macro table. It is not easy to read and not
 really suitable. It was used as a configuration file format to get
 something working rather than a suitable configuration file format.

 The RTEMS Testing tool uses the macros internally for its data and this is
 not to change. It is working and has nice features.

 The task is to look at replacing the raw macro format files with a YAML
 file that lets us define and manage the BSP back end configurations we
 need to support.

 An important area to be considered and implemented is an interface for
 user specific data a BSP back end configuration needs in order to run. For
 example when running the tests via JTAG you need to initialise the
 hardware in a target specific way and this data is not part of RTEMS so a
 user needs to provide it to the tool to run. A command line option is one
 way we need implement however adding endless command line options is not
 reasonable so a configuration file is also needed.

 The configuration file format shall allow a user to define the console
 interface they have. This could be a direct serial connection or a telnet
 console.

 == Regression Analysis ==

 The RTEMS Testing tools is to provide regression analysis for BSPs that
 provides the expected test result data.

 This task requires adding support for the `expected-fail` result tests can
 now report and to track this value.

 The task also requires a YAML format file for each BSP that defines its
 expected test results. The `rtems-test` tool shall attempt to locate the
 file when run in regression mode inside a build tree and also when the BSP
 is installed and determine if the BSP has regressed or the test results in
 RTEMS need updating.

 == Generic Serial Console ==

 The console support in the back end is termios however this is not
 available on Windows. This task is to integrate a BSD licensed Python
 serial package that provides Unix and Windows support so consoles can be
 supported on Windows and Unix.

 == Telnet Console ==

 Add support for telnet as a console. The ''ser2net'' package is a nice way
 to provide a number of serial ports on a network and a telnet console
 provides a way for testers to use it.

 = Acknowledgements =

 None.

 = Miscellaneous Sections =

 As the project progresses, you will need to add to the RTEMS Testing
 section of the User manual. This section does not currently exist but it
 should be present even in a skeleton form.

 = References =

 * None

--
Ticket URL: <http://devel.rtems.org/ticket/2927>
RTEMS Project <http://www.rtems.org/>
RTEMS Project