Couverture-Qemu Testing Results
Joel Sherrill
joel at rtems.org
Sun Jun 11 17:12:18 UTC 2017
On Jun 11, 2017 4:13 AM, "Cillian O'Donnell" <cpodonnell8 at gmail.com> wrote:
> Does pc386 look similar?
I haven't been able to get a decent set of results for PC386. For
RSB-Qemu all the tests time out. Using command
qemu-system-i386 -m 128 -boot b -hda path_to/rtems-boot.img -no-reboot
-monitor null -serial stdio -nographic -append "--console=com1;boot;"
----------------
RSB-QEMU
-----------------
--------------
SAMPLES
--------------
Passed: 0
Failed: 0
User Input: 0
Expected Fail: 0
Indeterminate: 0
Benchmark: 0
Timeout: 13
Invalid: 0
Total: 13
Average test time: 0:01:09.359520
Testing time : 0:15:01.673760
------------------------------------------------------
For Couverture-Qemu I ran the whole test suite which took 7 hours and
it decided a lot of them passed, although they probably should have
timed out. For hello.exe they both take 3 minutes, RSB-Qemu times out
and Couverture-qemu passes it.
-------------------------------------
COUVERTURE-QEMU
-------------------------------------
--------------
SAMPLES
--------------
Passed: 9
Failed: 0
User Input: 1
Expected Fail: 0
Indeterminate: 0
Benchmark: 0
Timeout: 2
Invalid: 1
Total: 13
Average test time: 0:00:14.133522
Testing time : 0:03:03.735787
------------
FULL
------------
Passed: 485
Failed: 0
User Input: 4
Expected Fail: 0
Indeterminate: 0
Benchmark: 3
Timeout: 74
Invalid: 0
Total: 566
Average test time: 0:00:44.922802
Testing time : 7:03:46.305933
------------------------------------------------------
> Are there other qemu bsps?
There are a few other rtems BSPS that seem to have a corresponding
machine that is supported by both qemu varaints. I had been hoping to
test all of these. But again all tests time out for RSB-Qemu and
Couverture-Qemu. I can't seem to find any qemu options to prevent
this. Should I keep testing and see can I get all of these working?
ARM: lm3s6965evb
LM32: lm32-evr, milkymist
MIPS: malta
POWERPC: prep (qemuprep)
You might should hand check hello on all the bsps. I recall that some do
not have a way for the bsp to "reset" the board and exit back to Linux. The
tests run but since qemu doesn't exit at RTEMS shutdown, you have to
timeout.
The sim-scripts covered this up watching the log file for "END OF TEST" and
killing the simulator.
But if it turns out that not exiting is the issue, that's a separate
problem and you should file a ticket a few move along.
On 10 June 2017 at 19:35, Joel Sherrill <joel at rtems.org> wrote:
> I would expect them to be the same. Couverture is a mild fork by AdaCore
> focused on embedded targets and coverage. They periodically debase and
> carefully apply patches. They are interested in those two targets. :)
>
> Does pc386 look similar? Are there other qemu bsps?
>
> I guess this means it is almost time to look at switching the rsb
>
> On Jun 10, 2017 1:25 PM, "Cillian O'Donnell" <cpodonnell8 at gmail.com>
wrote:
>
> Hi,
>
> I'm testing Couverture-Qemu against the RSB-Qemu to check if it is
> working correctly. So far I have good results for ARM bsps
> xilinx_zynq_a9_qemu and realview_pbx_a9_qemu and Sparc Leon3. They
> show identical results for the samples and almost identical results
> for the full testsuite. They just need a slight change in qemu options
> to agree (results at the end).
>
> ****************************
> COMMON BSPS
> ****************************
>
> ARM:
> xilinx-zynq-a9
> realview-pbx-a9
> lm3s6965evb
>
> i386:
> pc (default PC386)
>
> LM32:
> lm32-evr (default)
> milkymist
>
> MIPS:
> malta (default)
>
> POWERPC:
> prep
>
> SPARC:
> leon3_generic
>
>
> I created bsp_qemu.mc files for any missing bsps from RTEMS Tester.
> However the other bsps mostly timeout, this is true for Qemu and
> Couverture-qemu so they are reacting the same way. Exceptions are
> PC386 runs on Couverture-qemu but not on regular Qemu. I haven't been
> able to find the right qemu options to get rid of the timeouts and the
> --report-mode=all feature of rtems-test doesn't turn up any other info
> as to why except for powerpc/qemuprep.
>
> So are the following results enough to verify Couverture-qemu or
> should I keep going trying to get the others to work too before doing
> the RSB support?
>
> ***********************
> Leon 3
> ***********************
>
> qemu-system-sparc -no-reboot -monitor null -serial stdio -nographic
> -net none -M leon3_generic -kernel...
>
> --------------------------
>
> COUVERTURE-QEMU
>
> --------------------------
>
> ---------------
> SAMPLES
> ---------------
>
>
> Passed: 11
> Failed: 0
> User Input: 1
> Expected Fail: 0
> Indeterminate: 0
> Benchmark: 0
> Timeout: 0
> Invalid: 1
> Total: 13
> Average test time: 0:00:03.679456
> Testing time : 0:00:47.832933
>
> -------------
> FULL
> -------------
>
> Passed: 543
> Failed: 3
> User Input: 4
> Expected Fail: 0
> Indeterminate: 0
> Benchmark: 3
> Timeout: 9
> Invalid: 3
> Total: 565
> Average test time: 0:00:01.693401
> Testing time : 0:15:56.771872
>
> ---------------------------
>
> RSB-QEMU
>
> ---------------------------
>
> ---------------
> SAMPLES
> ---------------
>
>
> Passed: 11
> Failed: 0
> User Input: 1
> Expected Fail: 0
> Indeterminate: 0
> Benchmark: 0
> Timeout: 0
> Invalid: 1
> Total: 13
> Average test time: 0:00:03.679446
> Testing time : 0:00:47.832800
>
> -------------
> FULL
> -------------
>
> Passed: 544
> Failed: 2
> User Input: 4
> Expected Fail: 0
> Indeterminate: 0
> Benchmark: 3
> Timeout: 9
> Invalid: 3
> Total: 565
> Average test time: 0:00:01.739428
> Testing time : 0:16:22.777222
>
> *****************************
>
> xilinx_zynq_a9_qemu
>
> *****************************
>
> qemu-system-arm -no-reboot -monitor null -serial stdio -nographic -M
> xilinx-zynq-a9 -kernel...
>
> ----------------------------
>
> COUVERTURE-QEMU
>
> ----------------------------
>
> ---------------
> SAMPLES
> ---------------
>
>
> Passed: 11
> Failed: 0
> User Input: 1
> Expected Fail: 0
> Indeterminate: 0
> Benchmark: 0
> Timeout: 0
> Invalid: 1
> Total: 13
> Average test time: 0:00:00.559658
> Testing time : 0:00:07.275561
>
> -------------
> FULL
> -------------
>
> Passed: 551
> Failed: 5
> User Input: 4
> Expected Fail: 0
> Indeterminate: 0
> Benchmark: 3
> Timeout: 2
> Invalid: 1
> Total: 566
> Average test time: 0:00:00.652736
> Testing time : 0:06:09.448995
>
>
> ------------------
>
> RSB-QEMU
>
> ------------------
>
> qemu-system-arm -no-reboot -serial null -serial mon:stdio -nographic
> -M xilinx-zynq-a9 -kernel...
>
> ---------------
> SAMPLES
> ---------------
>
> Passed: 11
> Failed: 0
> User Input: 1
> Expected Fail: 0
> Indeterminate: 0
> Benchmark: 0
> Timeout: 0
> Invalid: 1
> Total: 13
> Average test time: 0:00:00.560756
> Testing time : 0:00:07.289829
>
> -------------
> FULL
> -------------
>
> Passed: 545
> Failed: 4
> User Input: 4
> Expected Fail: 0
> Indeterminate: 0
> Benchmark: 3
> Timeout: 9
> Invalid: 1
> Total: 566
> Average test time: 0:00:01.168632
> Testing time : 0:11:01.445923
>
>
> *********************************
>
> realview_pbx_a9_qemu
>
> *********************************
> --------------------------
>
> RSB-QEMU
>
> --------------------------
>
> qemu-system-arm -no-reboot -monitor null -serial stdio -nographic -M
> realview-pbx-a9 -kernel...
>
> ---------------
> SAMPLES
> ---------------
>
> Passed: 11
> Failed: 0
> User Input: 1
> Expected Fail: 0
> Indeterminate: 0
> Benchmark: 0
> Timeout: 0
> Invalid: 1
> Total: 13
> Average test time: 0:00:01.656712
> Testing time : 0:00:21.537256
>
> --------------
> FULL
> --------------
>
> Passed: 547
> Failed: 3
> User Input: 4
> Expected Fail: 0
> Indeterminate: 0
> Benchmark: 3
> Timeout: 8
> Invalid: 1
> Total: 566
> Average test time: 0:00:01.175699
> Testing time : 0:11:05.445813
>
> ------------------------
>
> COUVERTURE-QEMU
>
> ------------------------
>
> qemu-system-arm -no-reboot -monitor null -serial stdio -nographic -M
> realview-pbx-a9 -kernel...
>
> ---------------
> SAMPLES
> ---------------
>
>
> Passed: 11
> Failed: 0
> User Input: 1
> Expected Fail: 0
> Indeterminate: 0
> Benchmark: 0
> Timeout: 0
> Invalid: 1
> Total: 13
> Average test time: 0:00:00.617387
> Testing time : 0:00:08.026043
>
> --------------
> FULL
> --------------
>
> Passed: 553
> Failed: 4
> User Input: 4
> Expected Fail: 0
> Indeterminate: 0
> Benchmark: 3
> Timeout: 1
> Invalid: 1
> Total: 566
> Average test time: 0:00:00.671176
> Testing time : 0:06:19.886106
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rtems.org/pipermail/devel/attachments/20170611/6218ac5a/attachment-0002.html>
More information about the devel
mailing list