rtems and testing

[Chris Johns]

On 16/01/2017 05:35, Denis Obrezkov wrote:
> Hello,
> maybe I missed something in documentation, but
> could you tell me what is the best way to test RTEMS?

No you have not missed anything. I am adding something to the documentation but it is a low priority at the moment. The Tester needs lots of doco due to its back-end complexity.

> I found this link:
> https://devel.rtems.org/wiki/Developer/Testing/TestSuites
> but I can't found documentation for it. Could you help me with it?
> 
> Also, is there any testing tool which is recommended to use with
> user applications?

The RTEMS Tools installs a tool called rtems-test. The command line help is all that exists for documentation at the moment. To see the help run 'rtems-test --help;.

You specify a BSP and the tester will run each of tests provided on the command line and if the target is a simulator it runs the tests in parallel. To see the BSPs run 'rtems-test --list-bsps'.

Test the tester with a single test:

$ rtems-test --rtems-bsp=erc32 $(find . -name hello.exe)
RTEMS Testing - Tester, 4.12.not_released
 Command Line: rtems-test --rtems-bsp=erc32 ./sparc-rtems4.12/c/erc32/testsuites/samples/hello/hello.exe
 Python: 2.7.12 (default, Sep 27 2016, 01:49:05) [GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032)]
[1/1] p:0 f:0 t:0 i:0 | sparc/erc32: hello.exe
[1/1] p:0 f:0 t:0 i:0 | sparc/erc32: hello.exe

Passed:   1
Failed:   0
Timeouts: 0
Invalid:  0
-----------
Total:    1

Average test time: 0:00:01.333361
Testing time     : 0:00:01.333361

To all the tests:

$ rtems-test --rtems-bsp=erc32 $(find . -name \*..exe)
RTEMS Testing - Tester, 4.12.not_released
 Command Line: /opt/work/rtems/4.12/bin/rtems-test --rtems-bsp=erc32
 Python: 2.7.12 (default, Sep 27 2016, 01:49:05) [GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032)]
error: no executables supplied
ruru erc32 $ /opt/work/rtems/4.12/bin/rtems-test --rtems-bsp=erc32 $(find . -name \*.exe)
RTEMS Testing - Tester, 4.12.not_released
 Command Line: /opt/work/rtems/4.12/bin/rtems-test --rtems-bsp=erc32 ./sparc-rtems4.12/c/erc32 ..
 Python: 2.7.12 (default, Sep 27 2016, 01:49:05) [GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032)]
[  1/544] p:0   f:0   t:0   i:0   | sparc/erc32: fsbdpart01.exe
[  5/544] p:0   f:0   t:0   i:0   | sparc/erc32: fsdosfswrite01.exe
[  2/544] p:0   f:0   t:0   i:0   | sparc/erc32: fsdosfsformat01.exe
[  6/544] p:0   f:0   t:0   i:0   | sparc/erc32: fsfseeko01.exe
....
[541/544] p:529 f:1   t:3   i:0   | sparc/erc32: tmcontext01.exe
[542/544] p:530 f:1   t:3   i:0   | sparc/erc32: tmfine01.exe
[543/544] p:531 f:1   t:3   i:0   | sparc/erc32: tmoverhd.exe
[544/544] p:532 f:1   t:3   i:0   | sparc/erc32: tmtimer01.exe

Passed:   538
Failed:     1
Timeouts:   5
Invalid:    0
-------------
Total:    544

Failures:
 spcontext01.exe
Timeouts:
 fileio.exe
 top.exe
 termios.exe
 capture.exe
 monitor.exe
Average test time: 0:00:00.563533
Testing time     : 0:05:06.562379

The BSP backend support is defined in *.mc files under $PREFIX/share/rtems/tester/rtems/testing/bsps. Some of these are more complex than others, for exampling using GDB to run the tests on target hardware using JTAG, eg xilinx_zynq_zc706.mc.

Chris