[GSoC] RTEMS Tester Improvements

Wed Mar 15 22:57:27 UTC 2017

On 16/03/2017 08:44, Tanu Hari Dixit wrote:
> Hello Chris, Gedare, Joel,
> Thank you for answering my questions. Sorry for the late reply, it
> took me time to go through the tasks suggested.
>>
>> What platforms have you run tester on?
> 
> I was searching for a way to compile and execute a single test and the
> community suggetsed rtems-tester. So I ran the tester on sparc/erc32
> (and no simulator).

Great, this is what the tool is for.

> I am unable to find a relevant doc on how to run rtems-tester on qemu.
> Though there is this rtems-tools.git/tester/rtems/testing/qemu.cfg so
> I know that it is supported.

The `--list-bsps` gives some hints however not all BSPs that use qemu have this in the test BSP name. After a waf configure and build try:

ruru rtems-tools.git $ pwd
/opt/work/chris/rtems/tools/rtems-tools.git
ruru rtems-tools.git $ ./tester/rtems-test --list-bsps | grep qemu
  realview_pbx_a9_qemu
  xilinx_zynq_a9_qemu
  xilinx_zynq_a9_qemu_smp
  xilinx_zynq_zc706_qemu

The low level and simplest way to currently see is:

ruru rtems-tools.git $ grep -r qemu.cfg tester/rtems
tester/rtems/testing/bsps/xilinx_zynq_a9_qemu.mc:xilinx_zynq_a9_qemu:      none,    none,  '%{_rtscripts}/qemu.cfg'
tester/rtems/testing/bsps/generic_or1k.mc:generic_or1k:      none,    none,  '%{_rtscripts}/qemu.cfg'
tester/rtems/testing/bsps/xilinx_zynq_zc706_qemu.mc:xilinx_zynq_zc706:      none,    none,  '%{_rtscripts}/qemu.cfg'
tester/rtems/testing/bsps/xilinx_zynq_a9_qemu_smp.mc:xilinx_zynq_a9_qemu:      none,    none,  '%{_rtscripts}/qemu.cfg'
tester/rtems/testing/bsps/realview_pbx_a9_qemu.mc:realview_pbx_a9_qemu:      none,    none,  '%{_rtscripts}/qemu.cfg'

It is not the best way but it does work :)

> Also I couldn't find any other hint in
> the rtems-tools repository as to which simulators (except for qemu)
> are supported. rtems-testing/sim-scripts support gdbsim, skyeye and
> tsim (other than qemu). So we can add support for these and gem5 (as
> Gedare suggested). Please correct me if I am wrong.

You will need to review each .cfg file we have and see what there is. I see tsim is there.

> Furthermore, if you could please point me to a doc that explains using
> rtems-tester on qemu? I have tested qemu for running an application,
> the steps for which I have documented here
> (http://tokencolour.github.io/2017/03/16/how-to-run-applications-on-qemu-rtems.html).

I am not sure there is any doco. If you have built the RSB qemu which runs the zynq and you have built the xilinx_zynq_a9_qemu BSP you should be able to do:

ruru xilinx_zynq_a9_qemu $ pwd
/opt/work/chris/rtems/kernel/bsps/xilinx_zynq_a9_qemu
ruru xilinx_zynq_a9_qemu $ rtems-test --report-mode=all --rtems-bsp=xilinx_zynq_a9_qemu `find . -name hello.exe`
RTEMS Testing - Tester, 4.12.not_released
 Command Line: /opt/work/rtems/4.12/bin/rtems-test --report-mode=all --rtems-bsp=xilinx_zynq_a9_qemu ./arm-rtems4.12/c/xilinx_zynq_a9_qemu/testsuites/samples/hello/hello.exe
 Python: 2.7.13 (default, Jan 12 2017, 01:19:30) [GCC 4.2.1 Compatible FreeBSD Clang 3.8.0 (tags/RELEASE_380/final 262564)]
[1/1] p:0 f:0 t:0 i:0 | arm/xilinx_zynq_a9_qemu: hello.exe
[1/1] p:0 f:0 t:0 i:0 | arm/xilinx_zynq_a9_qemu: hello.exe
> run: qemu-system-arm -no-reboot -serial null -serial mon:stdio -nographic -net none -M xilinx-zynq-a9 -m 256M -kernel ./arm-rtems4.12/c/xilinx_zynq_a9_qemu/testsuites/samples/hello/hello.exe
] Warning: nic cadence_gem.0 has no peer
] Warning: nic cadence_gem.1 has no peer
]
]
] *** BEGIN OF TEST HELLO WORLD ***
] Hello World
] *** END OF TEST HELLO WORLD ***
Result: passed     Time: 0:00:01.069137 hello.exe

Passed:   1
Failed:   0
Timeouts: 0
Invalid:  0
-----------
Total:    1

Average test time: 0:00:01.526334
Testing time     : 0:00:01.526334

> I can play with visualizing aggregated testing results with plotting
> techniques (like matplotlib or maybe something that the devs
> recommend) :).

I would prefer this not be part of rtems-test. I am happy to have rtems-test generate report data in XML that can be used by other tools to generate visual data. I also happy to see these other tools being developed as part of this project if there is time and added to the RTEMS Tools Project if suitable, just not part of the rtems-test tool.

A couple of points ...

The separation is done to avoid extra dependencies being added to the rtems-test tool. Separating this type of functionality this way means the way this data is used is up to the user and not the rtems-test tool. It also brings the tool inline with the RSB which also exports XML reports.

Second, any tool to do this and added to the RTEMS Tool Project has to be fully cross platform, which means FreeBSD, OSX, Windows and even Linux. If the tool can be based on just standard Python libraries it will be cross platform and preferred.

> Also, where can I find more about "expected fail" state? Is it for
> those tests that fail nevertheless and need to be skipped? Can you
> please point me to an example? 

I do not think there currently are examples. I suspect a few tests which are expected fails maybe currently configured as exclude because that was all we had.

A test is tagged as 'expected-fail' in a .tcfg per BSP file using the newer tag format. To list the test config files try `find . -name \*.tcfg` from the top of the RTEMS source tree and take a look at a few of these files.

Setting a test as expected-fail causes the test to be built with compile command line directive of `-DTEST_STATE_EXPECTED_FAIL=1` and this results in the message:

 *** TEST STATE: EXPECTED-FAIL

being printed in the test. I have also added the test state `indeterminate` which prints:

 *** TEST STATE: INDETERMINATE

See testsuites/support/include/buffer_test_io.h for the details.

rtems-test needs to scan for this text like it currently scans for the begin and end messages and it needs to record the result for the test and have the result stats updated.

> Also, what kind of regression will one need to perform?

None, it is a report. If there are per BSP expected test results that are accessible to rtems-test you can easily report to the user if they have regressed or improved. The expected test results could simply be fail, expected-fail and indeterminate counts the remainder being pass. This could be enhanced to specify specific tests which would give a more accurate picture in case failures move around and maintain the same test result counts.

> Do the final states of the test need to maintain an
> overall state of the system so as to recommend/predict the state of
> affairs next time the tests are run (i.e. a predictor (which has
> memory) kind of thing)?

No it is just relative to the current BSP expected test results.

> I also think that converting macro files to yaml won't be very
> difficult. I might be very myopic here and it would be great if I get
> a heads up. What will be the challenge here?

Determining what we need to describe and creating and documenting the format. These types of task are easy to visualize but time consuming to get right and working. Also making the changes then verifying all the converted configurations are working is a lot of work and may require interactions with others who have the various simulators or hardware being used. For example tsim and ARM JTAG.

> For the generic serial console, will pySerial do the trick, on
> Windows? Do you have some other package in mind?

pySerial is the package (https://github.com/pyserial/pyserial). It has a suitable license. We would need to being the specific pieces of code into the rtemstoolkit, wrap and make available for the rtems-test tool.

> I have a problem here, though. I don't own a Zedboard and can't afford
> one. I can check with my department at college once I reach there, but
> I don't have my hopes too high. Will this pose a problem in the course
> of the project?

I do not think so. I have one and I know others do as well, Joel?. We can see if some form of remote access can be made available. Please indicate this in your proposal.

> I am yet to build for xilinx_zynq_zc706, so I'll ask more questions on
> the configuration control once I am done with it. Also, I'm trying out
> the old and new approach for Coverage reporting. I'll ask questions on
> it, once I try it.

I suggest you concentrate on the xilinx_zynq_a9_qemu BSP and then the RSB qemu under the bare config tree.

Chris