Couverture-Qemu Testing Results
Joel Sherrill
joel at rtems.org
Sat Jun 10 18:35:05 UTC 2017
I would expect them to be the same. Couverture is a mild fork by AdaCore
focused on embedded targets and coverage. They periodically debase and
carefully apply patches. They are interested in those two targets. :)
Does pc386 look similar? Are there other qemu bsps?
I guess this means it is almost time to look at switching the rsb
On Jun 10, 2017 1:25 PM, "Cillian O'Donnell" <cpodonnell8 at gmail.com> wrote:
Hi,
I'm testing Couverture-Qemu against the RSB-Qemu to check if it is
working correctly. So far I have good results for ARM bsps
xilinx_zynq_a9_qemu and realview_pbx_a9_qemu and Sparc Leon3. They
show identical results for the samples and almost identical results
for the full testsuite. They just need a slight change in qemu options
to agree (results at the end).
****************************
COMMON BSPS
****************************
ARM:
xilinx-zynq-a9
realview-pbx-a9
lm3s6965evb
i386:
pc (default PC386)
LM32:
lm32-evr (default)
milkymist
MIPS:
malta (default)
POWERPC:
prep
SPARC:
leon3_generic
I created bsp_qemu.mc files for any missing bsps from RTEMS Tester.
However the other bsps mostly timeout, this is true for Qemu and
Couverture-qemu so they are reacting the same way. Exceptions are
PC386 runs on Couverture-qemu but not on regular Qemu. I haven't been
able to find the right qemu options to get rid of the timeouts and the
--report-mode=all feature of rtems-test doesn't turn up any other info
as to why except for powerpc/qemuprep.
So are the following results enough to verify Couverture-qemu or
should I keep going trying to get the others to work too before doing
the RSB support?
***********************
Leon 3
***********************
qemu-system-sparc -no-reboot -monitor null -serial stdio -nographic
-net none -M leon3_generic -kernel...
--------------------------
COUVERTURE-QEMU
--------------------------
---------------
SAMPLES
---------------
Passed: 11
Failed: 0
User Input: 1
Expected Fail: 0
Indeterminate: 0
Benchmark: 0
Timeout: 0
Invalid: 1
Total: 13
Average test time: 0:00:03.679456
Testing time : 0:00:47.832933
-------------
FULL
-------------
Passed: 543
Failed: 3
User Input: 4
Expected Fail: 0
Indeterminate: 0
Benchmark: 3
Timeout: 9
Invalid: 3
Total: 565
Average test time: 0:00:01.693401
Testing time : 0:15:56.771872
---------------------------
RSB-QEMU
---------------------------
---------------
SAMPLES
---------------
Passed: 11
Failed: 0
User Input: 1
Expected Fail: 0
Indeterminate: 0
Benchmark: 0
Timeout: 0
Invalid: 1
Total: 13
Average test time: 0:00:03.679446
Testing time : 0:00:47.832800
-------------
FULL
-------------
Passed: 544
Failed: 2
User Input: 4
Expected Fail: 0
Indeterminate: 0
Benchmark: 3
Timeout: 9
Invalid: 3
Total: 565
Average test time: 0:00:01.739428
Testing time : 0:16:22.777222
*****************************
xilinx_zynq_a9_qemu
*****************************
qemu-system-arm -no-reboot -monitor null -serial stdio -nographic -M
xilinx-zynq-a9 -kernel...
----------------------------
COUVERTURE-QEMU
----------------------------
---------------
SAMPLES
---------------
Passed: 11
Failed: 0
User Input: 1
Expected Fail: 0
Indeterminate: 0
Benchmark: 0
Timeout: 0
Invalid: 1
Total: 13
Average test time: 0:00:00.559658
Testing time : 0:00:07.275561
-------------
FULL
-------------
Passed: 551
Failed: 5
User Input: 4
Expected Fail: 0
Indeterminate: 0
Benchmark: 3
Timeout: 2
Invalid: 1
Total: 566
Average test time: 0:00:00.652736
Testing time : 0:06:09.448995
------------------
RSB-QEMU
------------------
qemu-system-arm -no-reboot -serial null -serial mon:stdio -nographic
-M xilinx-zynq-a9 -kernel...
---------------
SAMPLES
---------------
Passed: 11
Failed: 0
User Input: 1
Expected Fail: 0
Indeterminate: 0
Benchmark: 0
Timeout: 0
Invalid: 1
Total: 13
Average test time: 0:00:00.560756
Testing time : 0:00:07.289829
-------------
FULL
-------------
Passed: 545
Failed: 4
User Input: 4
Expected Fail: 0
Indeterminate: 0
Benchmark: 3
Timeout: 9
Invalid: 1
Total: 566
Average test time: 0:00:01.168632
Testing time : 0:11:01.445923
*********************************
realview_pbx_a9_qemu
*********************************
--------------------------
RSB-QEMU
--------------------------
qemu-system-arm -no-reboot -monitor null -serial stdio -nographic -M
realview-pbx-a9 -kernel...
---------------
SAMPLES
---------------
Passed: 11
Failed: 0
User Input: 1
Expected Fail: 0
Indeterminate: 0
Benchmark: 0
Timeout: 0
Invalid: 1
Total: 13
Average test time: 0:00:01.656712
Testing time : 0:00:21.537256
--------------
FULL
--------------
Passed: 547
Failed: 3
User Input: 4
Expected Fail: 0
Indeterminate: 0
Benchmark: 3
Timeout: 8
Invalid: 1
Total: 566
Average test time: 0:00:01.175699
Testing time : 0:11:05.445813
------------------------
COUVERTURE-QEMU
------------------------
qemu-system-arm -no-reboot -monitor null -serial stdio -nographic -M
realview-pbx-a9 -kernel...
---------------
SAMPLES
---------------
Passed: 11
Failed: 0
User Input: 1
Expected Fail: 0
Indeterminate: 0
Benchmark: 0
Timeout: 0
Invalid: 1
Total: 13
Average test time: 0:00:00.617387
Testing time : 0:00:08.026043
--------------
FULL
--------------
Passed: 553
Failed: 4
User Input: 4
Expected Fail: 0
Indeterminate: 0
Benchmark: 3
Timeout: 1
Invalid: 1
Total: 566
Average test time: 0:00:00.671176
Testing time : 0:06:19.886106
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rtems.org/pipermail/devel/attachments/20170610/dd6e8308/attachment.html>
More information about the devel
mailing list