RTEMS Tier Allocations.

[Chris Johns]

On 23/10/2017 17:12, Sebastian Huber wrote:
> I would like to try the RTEMS tester on a Altera Cyclone V board with U-Boot. I
> can turn on/off the power via a network controlled power switch.  The console is
> connected to a Raspberry Pi.  It can also run the RTEMS tester.  What do I need
> to add to get it running?

Excellent question and thank you for asking. I am working on moving the BSP .mc
files to INI format so I can provide documentation. I did not want to document
the .mc files, that is an internal format. Coming with the INI support is the
ability to have local options that match your specific hardware and test set up
that you keep locally. Currently the repo contains my settings and that is not
great.

I hope this helps you get going until I get to the documentation.

Chris

-----------------------------------------------------------------------------

The RTEMS Tester now supports a TFTP back-end or "tester". The TFTP tester runs
a single test or job at a time and each test creates a TFTP server configured to
send the current test executable ignoring the client's requested file name. The
client does not know or care this is happening and it means you do not need to
touch the target's configuration once set up. To achieve a repeatable test
process there are a number of target commands and console output filters you can
configure to let the tester know if your target is working. Real hardware can be
variable and not every boot works or a test can crash locking the target up. The
console output filters combined with the target commands to let the tester
recover and continue.

The most complete example is the Zedboard. The macro configuration file is:

 https://git.rtems.org/rtems-tools/tree/tester/rtems/testing/bsps/xilinx_zynq_zedboard.mc

Target U-Boot Configuration
===========================

Configure U-Boot's eEnv.txt file to network boot using TFTP from your test host.
The eEnv.txt file for the Zedboard is:

bootfile=zed/rtems.img
loadaddr=0x02000000
uenvcmd=echo Booting RTEMS Zed from net; set autoload no; dhcp; set serverip
10.10.5.2; tftpboot zed/rtems.img; bootm; reset;

My test host is 10.10.5.2 and the target is using DHCP. Make sure you add the
'reset;' command to the end so a failure will reboot the target rather than it
continuing on and booting what ever is held in it's storage.

Target Tester Configuration
===========================

Target support in the configuration file has 4 groups:

 - Test pre and post commands
 - Executable file name filter
 - Target control
 - Target console filters

The tester will perform the following substitutions on the target command strings:

 `@EXE@` : The test executable name.
 `@FEXE@`: The filtered test executable name or the executable name if
           there is no filter.
 `@ARCH@`: The test executable architecture

Test Pre-test and Post-test Commands
------------------------------------

The pre-test command is a shell command run before each test. The post-test
command is a shell command run after each test. These commands can be used to
prepare a custom test image and to clean up after the test removing any
generated files. For example the Zedboard's U-Boot only accepts a U-Boot image
format file and the 'target_pretest_command' for the Zedboard executes a shell
script to create the per test image file with a '.exe.img' extension.

Executable File Name Filter
---------------------------

The target pre-test command may prepare any files needed to run a test on the
target. The executable file name filter maps the executable name to a generated
file name. The filter is based on Python's 're.sub' call where the first
character in the filter string is the delimiter, for example '/'. The content
between the first set of delimiters is the pattern and the content between the
second set of delimiters is plain text, not a regular expression, substituted
where the pattern matches in the executable file name.

For example the Zedboard filter of `/\.exe/.exe.img/` converts the executable
file name `somewhere/hello.exe` to `somewhere/hello.exe.img`.

Target Control
--------------

There are 3 target control commands:

 - `target_on_command`
 - `target_off_command`
 - `target_reset_command`

Target ON Command
~~~~~~~~~~~~~~~~~

The `target_on_command` command is a shell command executed before the first
test is run. The shell commands needs to turn the target on, start a simulator
or perform what target specific action is required to make the target available.
If you are using a power switch do not assume the target is off. An on command
to an already on target may do nothing and if the target is locked up the first
test may failing.

Target OFF Command
~~~~~~~~~~~~~~~~~

The `target_off_command` command is a shell command executed after the last test
has finished. The shell command is required to turn the target off, shut down a
simulator or perform what ever target specific action is required to disable the
target. A target not being tested needs to be disabled so it does not interfere
with the testing of other targets by stealing a TFTP download.

Target RESET Command
~~~~~~~~~~~~~~~~~~~~

The `target_reset_command` command is a shell command executed after a test
fails, time's out or the reset console filter triggers. The shell command needs
to reset the target. This could be a power cycle where the target is powered
down then powered on or it could be a specific reset signal that is asserted to
reset the target. The period to wait between a power off and power on depends on
the target hardware, power supply and where the power is being controlled.

Target Console Filters
----------------------

Target console filters are Python regular expressions that trigger an event when
the target's console output matches a filter. The filters are:

 - `target_start_regex`
 - `target_reset_regex`

Target Start Filter
~~~~~~~~~~~~~~~~~~~

The target start filter matches when a target restarts. The tester uses this
trigger as an indication the target has reset when it was not expected. This can
happening if the target crashes in a test. Adding this filter speeds up the test
run because the tester detects the reset quickly.

Target Reset Filter
~~~~~~~~~~~~~~~~~~~

The target reset filter matches various boot loader errors requiring the target
to be reset. This filter is useful if your target has networking issues after a
reset or power up.

Telnet Console
==============

The telnet console is configured with the `bsp_tty_dev` value and is a standard
host and port address. Telnet provides a simple clean cross platform way to get
serial target console output to a tester host machine. Your tester host can be a
large service machine in a rack in a machine room away from the actual target
hardware controlling the testing via only a network.

We recommend a RPi board with a USB HUB running the `ser2net` daemon.

Debugging A Configuration
-------------------------

Debugging a target configuration is best done in stages.

- Test you can telnet to the target from the tester host.

- Test with a single executable such as the sample's hello world and make sure
it passes. Select a single test with:

  --filter=hello.exe .

where the current directory (`.`) is the top of the BSP build tree. In this
example the tester either executes the ELF executable directly or there is a
pre-test command to create a suitable file to load onto the target and an
executable file name filter.

- Do not add a log file option to the command line. The log will appear on your
console.

- Adding `--debug-trace=output` to the command line will output all target
console. The target's console output is prefixed with ']' and tester's console
output is prefixed with '=>'.

- Once hello world works run a few tests. For a Zedboard this is:

  --filter=*.exe ./arm-rtems4.12/c/xilinx_zynq_zedboard/testsuites/samples

These options will test all the sample executables.

- If these are working you can now run all the tests. This time add a log:

  --log=t.txt --filter=*.exe .

The tester will only capture to the log the target and tester console output if
a test fails, time's out or is invalid. If you want to capture all console
output for review add:

  --report-mode=all

- Mailing the test results. The options are:

  `--mail`          Mail results
  `--mail-to`       Mail to address
  `--mail-from`     Mail from address
  `--smtp-host`     SMTP server, no passwords or security supported

If no `--mail-to` is provided on the command line the  default of
`build at rtems.org` is used. If no SMTP host is provided the default of
`localhost` is used. You can add to `$HOME/.mailrc` the line:

  set from=chrisj at rtems.org

to set a default from address.

To test mailing results use:

  --mail --mail-to=chrisj at rtems.org --mail-from=chrisj at rtems.org \
  --filter=hello.exe .

To mail a full set of test results with a from address in your `.mailrc` file use:

  --log=t.txt --mail --filter=*.exe .