Support for test coverage analysis in RTEMS 6

[VALDEZ Jose]

Hello Jerzy,

Recently we concluded a project, the RTEMS SMP Qualification which also needed to gather coverage for RTEMS (which had the participation of Andrew Butterfield from TCD/LERO). I suggest that you take a look at  https://rtems-qual.io.esa.int/, download one of the QDPs (maybe you want the GR712RC, SMP) and look for the docs/djf/svr document. There you may find the coverage folder, with the html coverage reports and in the svr.pdf the coverage summary table.

If this is what you want for your test, I suggest to take a look at qual-tool/svrmanager.py and qual-tool/ccoverage_process.py.

Basically for this what you need to do:

* Compile RTEMS with coverage flags. This will generate the gcno files
* Run your test, the reports should generate the gcda bytes (Sebastian Huber added support for coverage in RTEMS itself)
* Use/adapt ccoverage_process.py to generate the coverage reports, which makes use of the gcovr tool.

Hope this helps

José

-----Original Message-----
From: devel <devel-bounces at rtems.org> On Behalf Of Jerzy Jaskuc
Sent: 16 de março de 2022 16:46
To: devel at rtems.org
Subject: Support for test coverage analysis in RTEMS 6

Hi,

I've been looking into generating code coverage reports from RTEMS tests in RTEMS 6.
I'm using SPARC with gr712rc BSP.

I'm following the steps outlined in the coverage analysis documentation https://docs.rtems.org/branches/master/user/testing/coverage.html

However, when I run it on any tests, even sample/hello.exe I get the logs in lines of the following:

`RTEMS Testing - Tester, 6.0.not_released  Command Line: ./rtems-test --rtems-tools=/home/yoman/RTEMS/rtems/6
--log=log_leon3_sis --rtems-bsp=leon3-sis-cov --coverage /home/yoman/RTEMS/src/rtems/build/sparc/gr712rc/testsuites/validation/ts-validation-0.exe
 Host: Linux ubuntu 5.13.0-35-generic #40~20.04.1-Ubuntu SMP Mon Mar 7
09:18:32 UTC 2022 x86_64
 Python: 3.8.10 (default, Nov 26 2021, 20:14:08) [GCC 9.3.0]
Host: Linux-5.13.0-35-generic-x86_64-with-glibc2.29 (Linux ubuntu 5.13.0-35-generic #40~20.04.1-Ubuntu SMP Mon Mar 7 09:18:32 UTC 2022 x86_64
x86_64)
[1/1] p:0 f:0 u:0 e:0 I:0 B:0 t:0 L:0 i:0 W:0 | sparc/leon3-sis:
ts-validation-0.exe

Passed:        1
Failed:        0
User Input:    0
Expected Fail: 0
Indeterminate: 0
Benchmark:     0
Timeout:       0
Test too long: 0
Invalid:       0
Wrong Version: 0
Wrong Build:   0
Wrong Tools:   0
Wrong Header:  0
----------------
Total:         1

Average test time: 0:00:02.008333
Testing time     : 0:00:02.008333

Running coverage analysis (/home/yoman/RTEMS/rtems/6/bin/leon3-sis-coverage)
Function is both external and inlined: _Message_queue_Create Function is both external and inlined: _Message_queue_Create Function is both external and inlined: _Message_queue_Create Function is both external and inlined: _Thread_Initialize_information Function is both external and inlined: _Thread_Initialize_information Function is both external and inlined: _Thread_queue_Initialize Function is both external and inlined: _Thread_queue_Initialize Function is both external and inlined: _Thread_queue_Resume Function is both external and inlined: _Thread_queue_FIFO_first Function is both external and inlined: _Thread_queue_FIFO_first Function is both external and inlined: _Thread_CPU_budget_consume_and_yield
Function is both external and inlined: _Thread_CPU_budget_consume_and_yield
Function is both external and inlined: _Thread_Join Function is both external and inlined: _Thread_Join Function is both external and inlined: _Thread_Set_state_locked Function is both external and inlined: _Thread_Set_state_locked Function is both external and inlined: _Thread_Set_state_locked Function is both external and inlined: _Thread_Set_state_locked Function is both external and inlined: _Timecounter_Binuptime Function is both external and inlined: _Timecounter_Bintime Function is both external and inlined: _Timecounter_Getboottimebin Function is both external and inlined: _Thread_Priority_perform_actions Function is both external and inlined: _Thread_Priority_perform_actions Function is both external and inlined: _Thread_Clear_state_locked Function is both external and inlined: _Thread_Clear_state_locked Function is both external and inlined: _Thread_Clear_state_locked Function is both external and inlined: _Thread_Clear_state_locked Function is both external and inlined: _Malloc_System_state Function is both external and inlined: _Malloc_System_state Function is both external and inlined: _Malloc_System_state Function is both external and inlined: rtems_counter_initialize_converter
Function is both external and inlined: rtems_counter_initialize_converter
Function is both external and inlined: rtems_version_minor Function is both external and inlined: _Heap_Get_first_and_last_block Function is both external and inlined: _Heap_Get_first_and_last_block Error while running sparc-rtems6-objdump on /home/yoman/RTEMS/src/rtems/build/sparc/gr712rc/testsuites/validation/ts-validation-0.exe
Objdump error in generating objdump
Coverage time: 0:00:01.085511
Coverage generating reports
Coverage analysis finished: /home/yoman/RTEMS/rtems/6/bin`

I've tried to do a fresh VM fresh RTEMS build, results are exactly the same.
It seems like it's something to do with `objdump`? That would explain the generated report containing only unreferenced symbols.

Any tips on what's wrong or where to start looking to try and fix it?
I wanted to generate coverage reports for the tests I developed, but not sure if there's any other way than through making this part work.

Thanks and all the best!
Jerzy
_______________________________________________
devel mailing list
devel at rtems.org
http://lists.rtems.org/mailman/listinfo/devel