[GSoC] RTEMS Tester Improvements
Tanu Hari Dixit
tokencolour at gmail.com
Tue Mar 21 16:46:41 UTC 2017
Thank you so much Chris for the elaborate answers. I have a clearer
idea of the project now. Thank you Gedare for the tip, it definitely
comes in handy.
I'm sorry for the late reply. I have more questions and I will be
grateful if the devs can help me with these.
How does one use the console? I see extensive use of termios in
rtems-tools.git/tester/rt/stty.py which is in turn used in
tester/rt/console.py Can you please demonstrate one example so that I
get an idea? I believe PySerial will need to be configured in a
similar way i.e. making objects of class tty and assigning them
attributes with Pyserial instead of termios (like ignbrk and the
rest). Please correct me if I am wrong.
Moreover, about the telnet console, I went through the ser2net
package. It serves as a good way to convert (or maybe the right word
is "use") serial connections via telnet. I can make a simple bash
script to install ser2net and then change its conf file to fix the
appropriate port numbers and banners(which I can take as entry from
user). Then I shall use them via telnetlib to communicate to the hosts
and then I can run tests via the console on the target. Is this the
right direction or should I be thinking differently?
Also I am really confused as to how rtemstoolkit comes into the
picture. Can you please provide me with a rough workflow of the
console?
I also had questions regarding extending support for "expected-fail"
and "user-input" state as mentioned here
(https://devel.rtems.org/ticket/2946). What follows is my
understanding of what I need to do to extend support for
"user-input"(and "expected-fail"), please correct me if I am wrong. I
need to make a directive like DTEST_STATE_USER_INPUT and declare it to
be 1 in /rtems/src/rtems/tools/build/rtems-test-check and choose a
suitable TEST_STATE_STRING in buffer_test_io.h. Consequently, parse
for it in tester/rt/report.py. (probably this has to be done for
"expected-fail" state also). Also, I would need to extend the
attributes of class report that already has
self.passed = 0
self.failed = 0
self.timeouts = 0
self.invalids = 0
to also include self.expected_fails and self.user_inputs. Please
correct me if I am wrong. Can you please point me to the commit with
which you added support for "extended-fail"?
Furthermore, I was able to run the hello world example on
arm/xilinx_zync_a9_qemu with rtems-test as you demonstrated. Thank
you. Also, I understand why you emphasize on keeping a plotting tool
different from rtems-tester.
An aside question: Do you use winpdb to debug the python scripts in
rtems-tester? Do you recommend some other tool?
Thanking you,
Tanu Hari Dixit.
On Sat, Mar 18, 2017 at 1:53 AM, Gedare Bloom <gedare at rtems.org> wrote:
> On Wed, Mar 15, 2017 at 6:57 PM, Chris Johns <chrisj at rtems.org> wrote:
>> On 16/03/2017 08:44, Tanu Hari Dixit wrote:
>>> Hello Chris, Gedare, Joel,
>>> Thank you for answering my questions. Sorry for the late reply, it
>>> took me time to go through the tasks suggested.
>>>>
>>>> What platforms have you run tester on?
>>>
>>> I was searching for a way to compile and execute a single test and the
>>> community suggetsed rtems-tester. So I ran the tester on sparc/erc32
>>> (and no simulator).
>>
>> Great, this is what the tool is for.
>>
>>> I am unable to find a relevant doc on how to run rtems-tester on qemu.
>>> Though there is this rtems-tools.git/tester/rtems/testing/qemu.cfg so
>>> I know that it is supported.
>>
>> The `--list-bsps` gives some hints however not all BSPs that use qemu have this in the test BSP name. After a waf configure and build try:
>>
>> ruru rtems-tools.git $ pwd
>> /opt/work/chris/rtems/tools/rtems-tools.git
>> ruru rtems-tools.git $ ./tester/rtems-test --list-bsps | grep qemu
>> realview_pbx_a9_qemu
>> xilinx_zynq_a9_qemu
>> xilinx_zynq_a9_qemu_smp
>> xilinx_zynq_zc706_qemu
>>
>> The low level and simplest way to currently see is:
>>
>> ruru rtems-tools.git $ grep -r qemu.cfg tester/rtems
>> tester/rtems/testing/bsps/xilinx_zynq_a9_qemu.mc:xilinx_zynq_a9_qemu: none, none, '%{_rtscripts}/qemu.cfg'
>> tester/rtems/testing/bsps/generic_or1k.mc:generic_or1k: none, none, '%{_rtscripts}/qemu.cfg'
>> tester/rtems/testing/bsps/xilinx_zynq_zc706_qemu.mc:xilinx_zynq_zc706: none, none, '%{_rtscripts}/qemu.cfg'
>> tester/rtems/testing/bsps/xilinx_zynq_a9_qemu_smp.mc:xilinx_zynq_a9_qemu: none, none, '%{_rtscripts}/qemu.cfg'
>> tester/rtems/testing/bsps/realview_pbx_a9_qemu.mc:realview_pbx_a9_qemu: none, none, '%{_rtscripts}/qemu.cfg'
>>
>> It is not the best way but it does work :)
>>
>>> Also I couldn't find any other hint in
>>> the rtems-tools repository as to which simulators (except for qemu)
>>> are supported. rtems-testing/sim-scripts support gdbsim, skyeye and
>>> tsim (other than qemu). So we can add support for these and gem5 (as
>>> Gedare suggested). Please correct me if I am wrong.
>>
>> You will need to review each .cfg file we have and see what there is. I see tsim is there.
>>
>>> Furthermore, if you could please point me to a doc that explains using
>>> rtems-tester on qemu? I have tested qemu for running an application,
>>> the steps for which I have documented here
>>> (http://tokencolour.github.io/2017/03/16/how-to-run-applications-on-qemu-rtems.html).
>>
>> I am not sure there is any doco. If you have built the RSB qemu which runs the zynq and you have built the xilinx_zynq_a9_qemu BSP you should be able to do:
>>
>> ruru xilinx_zynq_a9_qemu $ pwd
>> /opt/work/chris/rtems/kernel/bsps/xilinx_zynq_a9_qemu
>> ruru xilinx_zynq_a9_qemu $ rtems-test --report-mode=all --rtems-bsp=xilinx_zynq_a9_qemu `find . -name hello.exe`
>> RTEMS Testing - Tester, 4.12.not_released
>> Command Line: /opt/work/rtems/4.12/bin/rtems-test --report-mode=all --rtems-bsp=xilinx_zynq_a9_qemu ./arm-rtems4.12/c/xilinx_zynq_a9_qemu/testsuites/samples/hello/hello.exe
>> Python: 2.7.13 (default, Jan 12 2017, 01:19:30) [GCC 4.2.1 Compatible FreeBSD Clang 3.8.0 (tags/RELEASE_380/final 262564)]
>> [1/1] p:0 f:0 t:0 i:0 | arm/xilinx_zynq_a9_qemu: hello.exe
>> [1/1] p:0 f:0 t:0 i:0 | arm/xilinx_zynq_a9_qemu: hello.exe
>>> run: qemu-system-arm -no-reboot -serial null -serial mon:stdio -nographic -net none -M xilinx-zynq-a9 -m 256M -kernel ./arm-rtems4.12/c/xilinx_zynq_a9_qemu/testsuites/samples/hello/hello.exe
>> ] Warning: nic cadence_gem.0 has no peer
>> ] Warning: nic cadence_gem.1 has no peer
>> ]
>> ]
>> ] *** BEGIN OF TEST HELLO WORLD ***
>> ] Hello World
>> ] *** END OF TEST HELLO WORLD ***
>> Result: passed Time: 0:00:01.069137 hello.exe
>>
>> Passed: 1
>> Failed: 0
>> Timeouts: 0
>> Invalid: 0
>> -----------
>> Total: 1
>>
>> Average test time: 0:00:01.526334
>> Testing time : 0:00:01.526334
>>
> Thanks Chris for the excellent answers. I want to just pile-on with my
> typical command line for running tester as another point in the
> conversation:
>
> Command Line: ./rtems-test --log=log_zynq_run
> --rtems-bsp=xilinx_zynq_a9_qemu
> --rtems-tools=/home/gedare/install/4.12
> /home/gedare/work/rtems/builds/b-xilinx_zynq_a9_qemu/arm-rtems4.12/c/xilinx_zynq_a9_qemu/testsuites
>
> The last argument points to the testsuites in my build tree, which
> results in running all of the testsuites I built.
>
> I almost always just do...
> $ head log_zynq_run
> and then copy-paste the command line into my terminal to run tester.
> It keeps me from having to remember/type anything more than once. :)
>
> (I do the same thing with RSB. It is a great feature of the log file!)
>
>>> I can play with visualizing aggregated testing results with plotting
>>> techniques (like matplotlib or maybe something that the devs
>>> recommend) :).
>>
>> I would prefer this not be part of rtems-test. I am happy to have rtems-test generate report data in XML that can be used by other tools to generate visual data. I also happy to see these other tools being developed as part of this project if there is time and added to the RTEMS Tools Project if suitable, just not part of the rtems-test tool.
>>
>> A couple of points ...
>>
>> The separation is done to avoid extra dependencies being added to the rtems-test tool. Separating this type of functionality this way means the way this data is used is up to the user and not the rtems-test tool. It also brings the tool inline with the RSB which also exports XML reports.
>>
>> Second, any tool to do this and added to the RTEMS Tool Project has to be fully cross platform, which means FreeBSD, OSX, Windows and even Linux. If the tool can be based on just standard Python libraries it will be cross platform and preferred.
>>
>>> Also, where can I find more about "expected fail" state? Is it for
>>> those tests that fail nevertheless and need to be skipped? Can you
>>> please point me to an example?
>>
>> I do not think there currently are examples. I suspect a few tests which are expected fails maybe currently configured as exclude because that was all we had.
>>
>> A test is tagged as 'expected-fail' in a .tcfg per BSP file using the newer tag format. To list the test config files try `find . -name \*.tcfg` from the top of the RTEMS source tree and take a look at a few of these files.
>>
>> Setting a test as expected-fail causes the test to be built with compile command line directive of `-DTEST_STATE_EXPECTED_FAIL=1` and this results in the message:
>>
>> *** TEST STATE: EXPECTED-FAIL
>>
>> being printed in the test. I have also added the test state `indeterminate` which prints:
>>
>> *** TEST STATE: INDETERMINATE
>>
>> See testsuites/support/include/buffer_test_io.h for the details.
>>
>> rtems-test needs to scan for this text like it currently scans for the begin and end messages and it needs to record the result for the test and have the result stats updated.
>>
>>> Also, what kind of regression will one need to perform?
>>
>> None, it is a report. If there are per BSP expected test results that are accessible to rtems-test you can easily report to the user if they have regressed or improved. The expected test results could simply be fail, expected-fail and indeterminate counts the remainder being pass. This could be enhanced to specify specific tests which would give a more accurate picture in case failures move around and maintain the same test result counts.
>>
>>> Do the final states of the test need to maintain an
>>> overall state of the system so as to recommend/predict the state of
>>> affairs next time the tests are run (i.e. a predictor (which has
>>> memory) kind of thing)?
>>
>> No it is just relative to the current BSP expected test results.
>>
>>> I also think that converting macro files to yaml won't be very
>>> difficult. I might be very myopic here and it would be great if I get
>>> a heads up. What will be the challenge here?
>>
>> Determining what we need to describe and creating and documenting the format. These types of task are easy to visualize but time consuming to get right and working. Also making the changes then verifying all the converted configurations are working is a lot of work and may require interactions with others who have the various simulators or hardware being used. For example tsim and ARM JTAG.
>>
>>> For the generic serial console, will pySerial do the trick, on
>>> Windows? Do you have some other package in mind?
>>
>> pySerial is the package (https://github.com/pyserial/pyserial). It has a suitable license. We would need to being the specific pieces of code into the rtemstoolkit, wrap and make available for the rtems-test tool.
>>
>>> I have a problem here, though. I don't own a Zedboard and can't afford
>>> one. I can check with my department at college once I reach there, but
>>> I don't have my hopes too high. Will this pose a problem in the course
>>> of the project?
>>
>> I do not think so. I have one and I know others do as well, Joel?. We can see if some form of remote access can be made available. Please indicate this in your proposal.
>>
>>> I am yet to build for xilinx_zynq_zc706, so I'll ask more questions on
>>> the configuration control once I am done with it. Also, I'm trying out
>>> the old and new approach for Coverage reporting. I'll ask questions on
>>> it, once I try it.
>>
>> I suggest you concentrate on the xilinx_zynq_a9_qemu BSP and then the RSB qemu under the bare config tree.
>>
>> Chris
More information about the devel
mailing list