Couverture-Qemu Testing Results

Joel Sherrill joel at rtems.org
Sun Jun 11 17:12:18 UTC 2017


On Jun 11, 2017 4:13 AM, "Cillian O'Donnell" <cpodonnell8 at gmail.com> wrote:

> Does pc386 look similar?


I haven't been able to get a decent set of results for PC386. For
RSB-Qemu all the tests time out. Using command

qemu-system-i386 -m 128 -boot b -hda path_to/rtems-boot.img -no-reboot
-monitor null -serial stdio -nographic -append "--console=com1;boot;"

----------------

    RSB-QEMU

-----------------
--------------
    SAMPLES
--------------
Passed:         0
Failed:         0
User Input:     0
Expected Fail:  0
Indeterminate:  0
Benchmark:      0
Timeout:       13
Invalid:        0
Total:         13
Average test time: 0:01:09.359520
Testing time     : 0:15:01.673760

------------------------------------------------------

For Couverture-Qemu  I ran the whole test suite which took 7 hours and
it decided a lot of them passed, although they probably should have
timed out. For hello.exe they both take 3 minutes, RSB-Qemu times out
and Couverture-qemu passes it.

-------------------------------------

COUVERTURE-QEMU

-------------------------------------

--------------
    SAMPLES
--------------

Passed:         9
Failed:         0
User Input:     1
Expected Fail:  0
Indeterminate:  0
Benchmark:      0
Timeout:        2
Invalid:        1
Total:         13
Average test time: 0:00:14.133522
Testing time     : 0:03:03.735787

------------
    FULL
------------

Passed:        485
Failed:          0
User Input:      4
Expected Fail:   0
Indeterminate:   0
Benchmark:       3
Timeout:        74
Invalid:         0
Total:         566
Average test time: 0:00:44.922802
Testing time     : 7:03:46.305933

------------------------------------------------------

> Are there other qemu bsps?

There are a few other rtems BSPS that seem to have a corresponding
machine that is supported by both qemu varaints. I had been hoping to
test all of these. But again all tests time out for RSB-Qemu and
Couverture-Qemu. I can't seem to find any qemu options to prevent
this. Should I keep testing and see can I get all of these working?

ARM: lm3s6965evb

LM32:  lm32-evr, milkymist

MIPS: malta

POWERPC: prep (qemuprep)


You might should hand check hello on all the bsps. I recall that some do
not have a way for the bsp to "reset" the board and exit back to Linux. The
tests run but since qemu doesn't exit at RTEMS shutdown, you have to
timeout.

The sim-scripts covered this up watching the log file for "END OF TEST" and
killing the simulator.

But if it turns out that not exiting is the issue, that's a separate
problem and you should file a ticket a few move along.



On 10 June 2017 at 19:35, Joel Sherrill <joel at rtems.org> wrote:
> I would expect them to be the same. Couverture is a mild fork by AdaCore
> focused on embedded targets and coverage. They periodically debase and
> carefully apply patches. They are interested in those two targets. :)
>
> Does pc386 look similar? Are there other qemu bsps?
>
> I guess this means it is almost time to look at switching the rsb
>
> On Jun 10, 2017 1:25 PM, "Cillian O'Donnell" <cpodonnell8 at gmail.com>
wrote:
>
> Hi,
>
> I'm testing Couverture-Qemu against the RSB-Qemu to check if it is
> working correctly. So far I have good results for ARM bsps
> xilinx_zynq_a9_qemu and realview_pbx_a9_qemu and Sparc Leon3. They
> show identical results for the samples and almost identical results
> for the full testsuite. They just need a slight change in qemu options
> to agree (results at the end).
>
> ****************************
>         COMMON BSPS
> ****************************
>
> ARM:
>     xilinx-zynq-a9
>     realview-pbx-a9
>     lm3s6965evb
>
> i386:
>     pc (default PC386)
>
> LM32:
>     lm32-evr (default)
>     milkymist
>
> MIPS:
>     malta (default)
>
> POWERPC:
>     prep
>
> SPARC:
>     leon3_generic
>
>
> I created bsp_qemu.mc files for any missing bsps from RTEMS Tester.
> However the other bsps mostly timeout, this is true for Qemu and
> Couverture-qemu so they are reacting the same way. Exceptions are
> PC386 runs on Couverture-qemu but not on regular Qemu. I haven't been
> able to find the right qemu options to get rid of the timeouts and the
> --report-mode=all feature of rtems-test doesn't turn up any other info
> as to why except for powerpc/qemuprep.
>
> So are the following results enough to verify Couverture-qemu or
> should I keep going trying to get the others to work too before doing
> the RSB support?
>
> ***********************
>         Leon 3
> ***********************
>
> qemu-system-sparc -no-reboot -monitor null -serial stdio -nographic
> -net none -M leon3_generic -kernel...
>
> --------------------------
>
>     COUVERTURE-QEMU
>
> --------------------------
>
> ---------------
>     SAMPLES
> ---------------
>
>
> Passed:        11
> Failed:         0
> User Input:     1
> Expected Fail:  0
> Indeterminate:  0
> Benchmark:      0
> Timeout:        0
> Invalid:        1
> Total:         13
> Average test time: 0:00:03.679456
> Testing time     : 0:00:47.832933
>
> -------------
>     FULL
> -------------
>
> Passed:        543
> Failed:          3
> User Input:      4
> Expected Fail:   0
> Indeterminate:   0
> Benchmark:       3
> Timeout:         9
> Invalid:         3
> Total:         565
> Average test time: 0:00:01.693401
> Testing time     : 0:15:56.771872
>
> ---------------------------
>
>         RSB-QEMU
>
> ---------------------------
>
> ---------------
>     SAMPLES
> ---------------
>
>
> Passed:        11
> Failed:         0
> User Input:     1
> Expected Fail:  0
> Indeterminate:  0
> Benchmark:      0
> Timeout:        0
> Invalid:        1
> Total:         13
> Average test time: 0:00:03.679446
> Testing time     : 0:00:47.832800
>
> -------------
>     FULL
> -------------
>
> Passed:        544
> Failed:          2
> User Input:      4
> Expected Fail:   0
> Indeterminate:   0
> Benchmark:       3
> Timeout:         9
> Invalid:         3
> Total:         565
> Average test time: 0:00:01.739428
> Testing time     : 0:16:22.777222
>
> *****************************
>
>     xilinx_zynq_a9_qemu
>
> *****************************
>
> qemu-system-arm -no-reboot -monitor null -serial stdio -nographic -M
> xilinx-zynq-a9 -kernel...
>
> ----------------------------
>
>     COUVERTURE-QEMU
>
> ----------------------------
>
> ---------------
>     SAMPLES
> ---------------
>
>
> Passed:        11
> Failed:         0
> User Input:     1
> Expected Fail:  0
> Indeterminate:  0
> Benchmark:      0
> Timeout:        0
> Invalid:        1
> Total:         13
> Average test time: 0:00:00.559658
> Testing time     : 0:00:07.275561
>
> -------------
>     FULL
> -------------
>
> Passed:        551
> Failed:          5
> User Input:      4
> Expected Fail:   0
> Indeterminate:   0
> Benchmark:       3
> Timeout:         2
> Invalid:         1
> Total:         566
> Average test time: 0:00:00.652736
> Testing time     : 0:06:09.448995
>
>
> ------------------
>
>     RSB-QEMU
>
> ------------------
>
> qemu-system-arm -no-reboot -serial null -serial mon:stdio -nographic
> -M xilinx-zynq-a9 -kernel...
>
> ---------------
>     SAMPLES
> ---------------
>
> Passed:        11
> Failed:         0
> User Input:     1
> Expected Fail:  0
> Indeterminate:  0
> Benchmark:      0
> Timeout:        0
> Invalid:        1
> Total:         13
> Average test time: 0:00:00.560756
> Testing time     : 0:00:07.289829
>
> -------------
>     FULL
> -------------
>
> Passed:        545
> Failed:          4
> User Input:      4
> Expected Fail:   0
> Indeterminate:   0
> Benchmark:       3
> Timeout:         9
> Invalid:         1
> Total:         566
> Average test time: 0:00:01.168632
> Testing time     : 0:11:01.445923
>
>
> *********************************
>
>     realview_pbx_a9_qemu
>
> *********************************
> --------------------------
>
>         RSB-QEMU
>
> --------------------------
>
> qemu-system-arm -no-reboot -monitor null -serial stdio -nographic -M
> realview-pbx-a9 -kernel...
>
> ---------------
>     SAMPLES
> ---------------
>
> Passed:        11
> Failed:         0
> User Input:     1
> Expected Fail:  0
> Indeterminate:  0
> Benchmark:      0
> Timeout:        0
> Invalid:        1
> Total:         13
> Average test time: 0:00:01.656712
> Testing time     : 0:00:21.537256
>
> --------------
>     FULL
> --------------
>
> Passed:        547
> Failed:          3
> User Input:      4
> Expected Fail:   0
> Indeterminate:   0
> Benchmark:       3
> Timeout:         8
> Invalid:         1
> Total:         566
> Average test time: 0:00:01.175699
> Testing time     : 0:11:05.445813
>
> ------------------------
>
>     COUVERTURE-QEMU
>
> ------------------------
>
> qemu-system-arm -no-reboot -monitor null -serial stdio -nographic -M
> realview-pbx-a9 -kernel...
>
> ---------------
>     SAMPLES
> ---------------
>
>
> Passed:        11
> Failed:         0
> User Input:     1
> Expected Fail:  0
> Indeterminate:  0
> Benchmark:      0
> Timeout:        0
> Invalid:        1
> Total:         13
> Average test time: 0:00:00.617387
> Testing time     : 0:00:08.026043
>
> --------------
>     FULL
> --------------
>
> Passed:        553
> Failed:          4
> User Input:      4
> Expected Fail:   0
> Indeterminate:   0
> Benchmark:       3
> Timeout:         1
> Invalid:         1
> Total:         566
> Average test time: 0:00:00.671176
> Testing time     : 0:06:19.886106
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rtems.org/pipermail/devel/attachments/20170611/6218ac5a/attachment-0002.html>


More information about the devel mailing list