Couverture-Qemu Testing Results

Joel Sherrill joel at rtems.org
Sat Jun 10 18:35:05 UTC 2017


I would expect them to be the same. Couverture is a mild fork by AdaCore
focused on embedded targets and coverage. They periodically debase and
carefully apply patches. They are interested in those two targets. :)

Does pc386 look similar? Are there other qemu bsps?

I guess this means it is almost time to look at switching the rsb

On Jun 10, 2017 1:25 PM, "Cillian O'Donnell" <cpodonnell8 at gmail.com> wrote:

Hi,

I'm testing Couverture-Qemu against the RSB-Qemu to check if it is
working correctly. So far I have good results for ARM bsps
xilinx_zynq_a9_qemu and realview_pbx_a9_qemu and Sparc Leon3. They
show identical results for the samples and almost identical results
for the full testsuite. They just need a slight change in qemu options
to agree (results at the end).

****************************
        COMMON BSPS
****************************

ARM:
    xilinx-zynq-a9
    realview-pbx-a9
    lm3s6965evb

i386:
    pc (default PC386)

LM32:
    lm32-evr (default)
    milkymist

MIPS:
    malta (default)

POWERPC:
    prep

SPARC:
    leon3_generic


I created bsp_qemu.mc files for any missing bsps from RTEMS Tester.
However the other bsps mostly timeout, this is true for Qemu and
Couverture-qemu so they are reacting the same way. Exceptions are
PC386 runs on Couverture-qemu but not on regular Qemu. I haven't been
able to find the right qemu options to get rid of the timeouts and the
--report-mode=all feature of rtems-test doesn't turn up any other info
as to why except for powerpc/qemuprep.

So are the following results enough to verify Couverture-qemu or
should I keep going trying to get the others to work too before doing
the RSB support?

***********************
        Leon 3
***********************

qemu-system-sparc -no-reboot -monitor null -serial stdio -nographic
-net none -M leon3_generic -kernel...

--------------------------

    COUVERTURE-QEMU

--------------------------

---------------
    SAMPLES
---------------


Passed:        11
Failed:         0
User Input:     1
Expected Fail:  0
Indeterminate:  0
Benchmark:      0
Timeout:        0
Invalid:        1
Total:         13
Average test time: 0:00:03.679456
Testing time     : 0:00:47.832933

-------------
    FULL
-------------

Passed:        543
Failed:          3
User Input:      4
Expected Fail:   0
Indeterminate:   0
Benchmark:       3
Timeout:         9
Invalid:         3
Total:         565
Average test time: 0:00:01.693401
Testing time     : 0:15:56.771872

---------------------------

        RSB-QEMU

---------------------------

---------------
    SAMPLES
---------------


Passed:        11
Failed:         0
User Input:     1
Expected Fail:  0
Indeterminate:  0
Benchmark:      0
Timeout:        0
Invalid:        1
Total:         13
Average test time: 0:00:03.679446
Testing time     : 0:00:47.832800

-------------
    FULL
-------------

Passed:        544
Failed:          2
User Input:      4
Expected Fail:   0
Indeterminate:   0
Benchmark:       3
Timeout:         9
Invalid:         3
Total:         565
Average test time: 0:00:01.739428
Testing time     : 0:16:22.777222

*****************************

    xilinx_zynq_a9_qemu

*****************************

qemu-system-arm -no-reboot -monitor null -serial stdio -nographic -M
xilinx-zynq-a9 -kernel...

----------------------------

    COUVERTURE-QEMU

----------------------------

---------------
    SAMPLES
---------------


Passed:        11
Failed:         0
User Input:     1
Expected Fail:  0
Indeterminate:  0
Benchmark:      0
Timeout:        0
Invalid:        1
Total:         13
Average test time: 0:00:00.559658
Testing time     : 0:00:07.275561

-------------
    FULL
-------------

Passed:        551
Failed:          5
User Input:      4
Expected Fail:   0
Indeterminate:   0
Benchmark:       3
Timeout:         2
Invalid:         1
Total:         566
Average test time: 0:00:00.652736
Testing time     : 0:06:09.448995


------------------

    RSB-QEMU

------------------

qemu-system-arm -no-reboot -serial null -serial mon:stdio -nographic
-M xilinx-zynq-a9 -kernel...

---------------
    SAMPLES
---------------

Passed:        11
Failed:         0
User Input:     1
Expected Fail:  0
Indeterminate:  0
Benchmark:      0
Timeout:        0
Invalid:        1
Total:         13
Average test time: 0:00:00.560756
Testing time     : 0:00:07.289829

-------------
    FULL
-------------

Passed:        545
Failed:          4
User Input:      4
Expected Fail:   0
Indeterminate:   0
Benchmark:       3
Timeout:         9
Invalid:         1
Total:         566
Average test time: 0:00:01.168632
Testing time     : 0:11:01.445923


*********************************

    realview_pbx_a9_qemu

*********************************
--------------------------

        RSB-QEMU

--------------------------

qemu-system-arm -no-reboot -monitor null -serial stdio -nographic -M
realview-pbx-a9 -kernel...

---------------
    SAMPLES
---------------

Passed:        11
Failed:         0
User Input:     1
Expected Fail:  0
Indeterminate:  0
Benchmark:      0
Timeout:        0
Invalid:        1
Total:         13
Average test time: 0:00:01.656712
Testing time     : 0:00:21.537256

--------------
    FULL
--------------

Passed:        547
Failed:          3
User Input:      4
Expected Fail:   0
Indeterminate:   0
Benchmark:       3
Timeout:         8
Invalid:         1
Total:         566
Average test time: 0:00:01.175699
Testing time     : 0:11:05.445813

------------------------

    COUVERTURE-QEMU

------------------------

qemu-system-arm -no-reboot -monitor null -serial stdio -nographic -M
realview-pbx-a9 -kernel...

---------------
    SAMPLES
---------------


Passed:        11
Failed:         0
User Input:     1
Expected Fail:  0
Indeterminate:  0
Benchmark:      0
Timeout:        0
Invalid:        1
Total:         13
Average test time: 0:00:00.617387
Testing time     : 0:00:08.026043

--------------
    FULL
--------------

Passed:        553
Failed:          4
User Input:      4
Expected Fail:   0
Indeterminate:   0
Benchmark:       3
Timeout:         1
Invalid:         1
Total:         566
Average test time: 0:00:00.671176
Testing time     : 0:06:19.886106
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rtems.org/pipermail/devel/attachments/20170610/dd6e8308/attachment-0002.html>


More information about the devel mailing list