Couverture-Qemu Testing Results
Joel Sherrill
joel at rtems.org
Sun Jun 11 18:53:07 UTC 2017
On Jun 11, 2017 1:14 PM, "Cillian O'Donnell" <cpodonnell8 at gmail.com> wrote:
>You might should hand check hello on all the bsps. I recall that some do
not have a way for the bsp to "reset" the board and exit back to Linux. The
tests run but since qemu doesn't exit at RTEMS shutdown, you have to
timeout.
Actually I've already done this and that is the exact problem I'm
running into, almost all of the bsps hang on 'end of test hello world'
and I have to exit it myself.
*** BEGIN OF TEST HELLO WORLD ***
Hello World
*** END OF TEST HELLO WORLD ***
qemu: terminating on signal 2
The only exceptions is lm32/milkymist bsp with:
qemu-system-lm32 -no-reboot -monitor null -serial stdio -nographic
-net none -M milkymist -kernel
$HOME/development/rtems/milkymist/lm32-rtems4.12/c/
milkymist/testsuites/samples/hello/hello.exe
audio: Could not init `oss' audio driver
Warning: nic milkymist-minimac2.0 has no peer
and PowerPC/qemuprep with:
qemu-system-ppc -no-reboot -monitor null -serial stdio -nographic -net
none -M prep -kernel
$HOME/development/rtems/qemuprep/powerpc-rtems4.12/c/
qemuprep/testsuites/samples/hello/hello.exe
register PCI host 'pci-bridge' 'pci' '<null>' 'PREP Host PCI Bridge -
Motorola Raven'
register 'pci-bridge' 'pci' '<null>' 'PREP Host PCI Bridge - Motorola
Raven' 0x80000000 in 'device-tree' 0xffffffff
Done 582b000 582b880
PCI device '<null>' 0 11 0 has no reg properties:
PCI device '<null>' 0 11 0 has no assigned addresses properties:
register pci device 'Qemu VGA' 0000000d 'display' 'VGA' 'Qemu VGA'
register 'Qemu VGA' 'display' 'VGA' 'Qemu VGA' 0x0000000d in
'pci-bridge' 0x80000000
Done 582b880 582b980
PCI device 'Qemu VGA' 0 13 0 reg properties:
addr: 82006810 00000000 f0000000 size: 00000000 01000000
addr: 82006818 00000000 f1000000 size: 00000000 00010000
PCI device 'Qemu VGA' 0 13 0 assigned addresses properties:
addr: 82006810 00000000 f0000000 size: 00000000 01000000
addr: 82006818 00000000 f1000000 size: 00000000 00010000
PPC Open Hack'Ware BIOS for qemu version 0.4.1
Build 2014-07-04 15:23:25
Copyright 2003-2005 Jocelyn Mayer
Memory size: 128 MB.
Booting from device m
ide0: drive 0: none
ide0: drive 1: none
ide1: drive 0: CD-ROM
ERROR: OF_property_copy cannot get property 'cd' for aliases
ERROR: ATAPI TEST_UNIT_READY : status 58 != 0x08
ERROR: ATAPI TEST_UNIT_READY : status 41 != 0x40
ide1: drive 1: none
Probe partitions for device m
ERROR: No MSDOS signature (38 0 0 0)
Use bloc device as raw partition
Boot partition: 0 9401fff8 9401fff8 0
ERROR: OF_property_copy cannot get property 'alias' for <null>
boot device: 5832000 image 1000000 size 4599176
ERROR: No MSDOS signature (7f 45 0 0)
Use bloc device as raw partition
Boot partition: 0 9401fff8 9401fff8 0
boot device: 5832000
ERROR: Found no boot partition!
ERROR: BUG caught...
BIOS execution exception
nip=0x05800000 msr=0x00002000 dar=0x00000000 dsisr=0x00000000
Stopping execution
qemu: terminating on signal 2
> But if it turns out that not exiting is the issue, that's a separate
problem and you should file a ticket a few move along.
Ok, so I should try and fix the above 2, file a ticket for the rest
and then move on to the RSB work?
File individual tickets on the bsps with unique failures on qemu. And one
to cover the "no exit" case. I know Chris has been hesitant to look at test
output and kill tests but I think it is necessary unless you can find a way
to make qemu automatically exit from the BSP.
Anyway, you have two BSPs which work well enough to continue on with the
framework. Think of it as a skyscraper with only some floors ready for
occupancy. We need the building finished before anyone can move in but we
can finish the interior on some floors later. :)
On 11 June 2017 at 18:12, Joel Sherrill <joel at rtems.org> wrote:
>
>
> On Jun 11, 2017 4:13 AM, "Cillian O'Donnell" <cpodonnell8 at gmail.com>
wrote:
>
>> Does pc386 look similar?
>
>
> I haven't been able to get a decent set of results for PC386. For
> RSB-Qemu all the tests time out. Using command
>
> qemu-system-i386 -m 128 -boot b -hda path_to/rtems-boot.img -no-reboot
> -monitor null -serial stdio -nographic -append "--console=com1;boot;"
>
> ----------------
>
> RSB-QEMU
>
> -----------------
> --------------
> SAMPLES
> --------------
> Passed: 0
> Failed: 0
> User Input: 0
> Expected Fail: 0
> Indeterminate: 0
> Benchmark: 0
> Timeout: 13
> Invalid: 0
> Total: 13
> Average test time: 0:01:09.359520
> Testing time : 0:15:01.673760
>
> ------------------------------------------------------
>
> For Couverture-Qemu I ran the whole test suite which took 7 hours and
> it decided a lot of them passed, although they probably should have
> timed out. For hello.exe they both take 3 minutes, RSB-Qemu times out
> and Couverture-qemu passes it.
>
> -------------------------------------
>
> COUVERTURE-QEMU
>
> -------------------------------------
>
> --------------
> SAMPLES
> --------------
>
> Passed: 9
> Failed: 0
> User Input: 1
> Expected Fail: 0
> Indeterminate: 0
> Benchmark: 0
> Timeout: 2
> Invalid: 1
> Total: 13
> Average test time: 0:00:14.133522
> Testing time : 0:03:03.735787
>
> ------------
> FULL
> ------------
>
> Passed: 485
> Failed: 0
> User Input: 4
> Expected Fail: 0
> Indeterminate: 0
> Benchmark: 3
> Timeout: 74
> Invalid: 0
> Total: 566
> Average test time: 0:00:44.922802
> Testing time : 7:03:46.305933
>
> ------------------------------------------------------
>
>> Are there other qemu bsps?
>
> There are a few other rtems BSPS that seem to have a corresponding
> machine that is supported by both qemu varaints. I had been hoping to
> test all of these. But again all tests time out for RSB-Qemu and
> Couverture-Qemu. I can't seem to find any qemu options to prevent
> this. Should I keep testing and see can I get all of these working?
>
> ARM: lm3s6965evb
>
> LM32: lm32-evr, milkymist
>
> MIPS: malta
>
> POWERPC: prep (qemuprep)
>
>
> You might should hand check hello on all the bsps. I recall that some do
not
> have a way for the bsp to "reset" the board and exit back to Linux. The
> tests run but since qemu doesn't exit at RTEMS shutdown, you have to
> timeout.
>
> The sim-scripts covered this up watching the log file for "END OF TEST"
and
> killing the simulator.
>
> But if it turns out that not exiting is the issue, that's a separate
problem
> and you should file a ticket a few move along.
>
>
>
> On 10 June 2017 at 19:35, Joel Sherrill <joel at rtems.org> wrote:
>> I would expect them to be the same. Couverture is a mild fork by AdaCore
>> focused on embedded targets and coverage. They periodically debase and
>> carefully apply patches. They are interested in those two targets. :)
>>
>> Does pc386 look similar? Are there other qemu bsps?
>>
>> I guess this means it is almost time to look at switching the rsb
>>
>> On Jun 10, 2017 1:25 PM, "Cillian O'Donnell" <cpodonnell8 at gmail.com>
>> wrote:
>>
>> Hi,
>>
>> I'm testing Couverture-Qemu against the RSB-Qemu to check if it is
>> working correctly. So far I have good results for ARM bsps
>> xilinx_zynq_a9_qemu and realview_pbx_a9_qemu and Sparc Leon3. They
>> show identical results for the samples and almost identical results
>> for the full testsuite. They just need a slight change in qemu options
>> to agree (results at the end).
>>
>> ****************************
>> COMMON BSPS
>> ****************************
>>
>> ARM:
>> xilinx-zynq-a9
>> realview-pbx-a9
>> lm3s6965evb
>>
>> i386:
>> pc (default PC386)
>>
>> LM32:
>> lm32-evr (default)
>> milkymist
>>
>> MIPS:
>> malta (default)
>>
>> POWERPC:
>> prep
>>
>> SPARC:
>> leon3_generic
>>
>>
>> I created bsp_qemu.mc files for any missing bsps from RTEMS Tester.
>> However the other bsps mostly timeout, this is true for Qemu and
>> Couverture-qemu so they are reacting the same way. Exceptions are
>> PC386 runs on Couverture-qemu but not on regular Qemu. I haven't been
>> able to find the right qemu options to get rid of the timeouts and the
>> --report-mode=all feature of rtems-test doesn't turn up any other info
>> as to why except for powerpc/qemuprep.
>>
>> So are the following results enough to verify Couverture-qemu or
>> should I keep going trying to get the others to work too before doing
>> the RSB support?
>>
>> ***********************
>> Leon 3
>> ***********************
>>
>> qemu-system-sparc -no-reboot -monitor null -serial stdio -nographic
>> -net none -M leon3_generic -kernel...
>>
>> --------------------------
>>
>> COUVERTURE-QEMU
>>
>> --------------------------
>>
>> ---------------
>> SAMPLES
>> ---------------
>>
>>
>> Passed: 11
>> Failed: 0
>> User Input: 1
>> Expected Fail: 0
>> Indeterminate: 0
>> Benchmark: 0
>> Timeout: 0
>> Invalid: 1
>> Total: 13
>> Average test time: 0:00:03.679456
>> Testing time : 0:00:47.832933
>>
>> -------------
>> FULL
>> -------------
>>
>> Passed: 543
>> Failed: 3
>> User Input: 4
>> Expected Fail: 0
>> Indeterminate: 0
>> Benchmark: 3
>> Timeout: 9
>> Invalid: 3
>> Total: 565
>> Average test time: 0:00:01.693401
>> Testing time : 0:15:56.771872
>>
>> ---------------------------
>>
>> RSB-QEMU
>>
>> ---------------------------
>>
>> ---------------
>> SAMPLES
>> ---------------
>>
>>
>> Passed: 11
>> Failed: 0
>> User Input: 1
>> Expected Fail: 0
>> Indeterminate: 0
>> Benchmark: 0
>> Timeout: 0
>> Invalid: 1
>> Total: 13
>> Average test time: 0:00:03.679446
>> Testing time : 0:00:47.832800
>>
>> -------------
>> FULL
>> -------------
>>
>> Passed: 544
>> Failed: 2
>> User Input: 4
>> Expected Fail: 0
>> Indeterminate: 0
>> Benchmark: 3
>> Timeout: 9
>> Invalid: 3
>> Total: 565
>> Average test time: 0:00:01.739428
>> Testing time : 0:16:22.777222
>>
>> *****************************
>>
>> xilinx_zynq_a9_qemu
>>
>> *****************************
>>
>> qemu-system-arm -no-reboot -monitor null -serial stdio -nographic -M
>> xilinx-zynq-a9 -kernel...
>>
>> ----------------------------
>>
>> COUVERTURE-QEMU
>>
>> ----------------------------
>>
>> ---------------
>> SAMPLES
>> ---------------
>>
>>
>> Passed: 11
>> Failed: 0
>> User Input: 1
>> Expected Fail: 0
>> Indeterminate: 0
>> Benchmark: 0
>> Timeout: 0
>> Invalid: 1
>> Total: 13
>> Average test time: 0:00:00.559658
>> Testing time : 0:00:07.275561
>>
>> -------------
>> FULL
>> -------------
>>
>> Passed: 551
>> Failed: 5
>> User Input: 4
>> Expected Fail: 0
>> Indeterminate: 0
>> Benchmark: 3
>> Timeout: 2
>> Invalid: 1
>> Total: 566
>> Average test time: 0:00:00.652736
>> Testing time : 0:06:09.448995
>>
>>
>> ------------------
>>
>> RSB-QEMU
>>
>> ------------------
>>
>> qemu-system-arm -no-reboot -serial null -serial mon:stdio -nographic
>> -M xilinx-zynq-a9 -kernel...
>>
>> ---------------
>> SAMPLES
>> ---------------
>>
>> Passed: 11
>> Failed: 0
>> User Input: 1
>> Expected Fail: 0
>> Indeterminate: 0
>> Benchmark: 0
>> Timeout: 0
>> Invalid: 1
>> Total: 13
>> Average test time: 0:00:00.560756
>> Testing time : 0:00:07.289829
>>
>> -------------
>> FULL
>> -------------
>>
>> Passed: 545
>> Failed: 4
>> User Input: 4
>> Expected Fail: 0
>> Indeterminate: 0
>> Benchmark: 3
>> Timeout: 9
>> Invalid: 1
>> Total: 566
>> Average test time: 0:00:01.168632
>> Testing time : 0:11:01.445923
>>
>>
>> *********************************
>>
>> realview_pbx_a9_qemu
>>
>> *********************************
>> --------------------------
>>
>> RSB-QEMU
>>
>> --------------------------
>>
>> qemu-system-arm -no-reboot -monitor null -serial stdio -nographic -M
>> realview-pbx-a9 -kernel...
>>
>> ---------------
>> SAMPLES
>> ---------------
>>
>> Passed: 11
>> Failed: 0
>> User Input: 1
>> Expected Fail: 0
>> Indeterminate: 0
>> Benchmark: 0
>> Timeout: 0
>> Invalid: 1
>> Total: 13
>> Average test time: 0:00:01.656712
>> Testing time : 0:00:21.537256
>>
>> --------------
>> FULL
>> --------------
>>
>> Passed: 547
>> Failed: 3
>> User Input: 4
>> Expected Fail: 0
>> Indeterminate: 0
>> Benchmark: 3
>> Timeout: 8
>> Invalid: 1
>> Total: 566
>> Average test time: 0:00:01.175699
>> Testing time : 0:11:05.445813
>>
>> ------------------------
>>
>> COUVERTURE-QEMU
>>
>> ------------------------
>>
>> qemu-system-arm -no-reboot -monitor null -serial stdio -nographic -M
>> realview-pbx-a9 -kernel...
>>
>> ---------------
>> SAMPLES
>> ---------------
>>
>>
>> Passed: 11
>> Failed: 0
>> User Input: 1
>> Expected Fail: 0
>> Indeterminate: 0
>> Benchmark: 0
>> Timeout: 0
>> Invalid: 1
>> Total: 13
>> Average test time: 0:00:00.617387
>> Testing time : 0:00:08.026043
>>
>> --------------
>> FULL
>> --------------
>>
>> Passed: 553
>> Failed: 4
>> User Input: 4
>> Expected Fail: 0
>> Indeterminate: 0
>> Benchmark: 3
>> Timeout: 1
>> Invalid: 1
>> Total: 566
>> Average test time: 0:00:00.671176
>> Testing time : 0:06:19.886106
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rtems.org/pipermail/devel/attachments/20170611/97c06d1f/attachment-0002.html>
More information about the devel
mailing list