Support for test coverage analysis in RTEMS 6

Wed Mar 16 19:16:06 UTC 2022

Hi José,

I'm aware of the QDP, although it is pretty big so I got confused and only
followed  `11.3 Guidance for RTEMS Qualification in User’s Environment`,
which didn't work as I assume it needs actual board for tests to not
be`Invalid`
The reports generated are exactly what I'm looking for and steps you
mentioned seems much more manageable!

I can see gcno files and I can see gcda bytes within the logs of the test
run.
I will take a look at adapting the python scripts then.

Thanks a lot for your help!

All the best,
Jerzy

On Wed, 16 Mar 2022 at 17:05, VALDEZ Jose <jose.valdez at edisoft.pt> wrote:

> Hello Jerzy,
>
> Recently we concluded a project, the RTEMS SMP Qualification which also
> needed to gather coverage for RTEMS (which had the participation of Andrew
> Butterfield from TCD/LERO). I suggest that you take a look at
> https://rtems-qual.io.esa.int/, download one of the QDPs (maybe you want
> the GR712RC, SMP) and look for the docs/djf/svr document. There you may
> find the coverage folder, with the html coverage reports and in the svr.pdf
> the coverage summary table.
>
> If this is what you want for your test, I suggest to take a look at
> qual-tool/svrmanager.py and qual-tool/ccoverage_process.py.
>
> Basically for this what you need to do:
>
> * Compile RTEMS with coverage flags. This will generate the gcno files
> * Run your test, the reports should generate the gcda bytes (Sebastian
> Huber added support for coverage in RTEMS itself)
> * Use/adapt ccoverage_process.py to generate the coverage reports, which
> makes use of the gcovr tool.
>
> Hope this helps
>
> José
>
> -----Original Message-----
> From: devel <devel-bounces at rtems.org> On Behalf Of Jerzy Jaskuc
> Sent: 16 de março de 2022 16:46
> To: devel at rtems.org
> Subject: Support for test coverage analysis in RTEMS 6
>
> Hi,
>
> I've been looking into generating code coverage reports from RTEMS tests
> in RTEMS 6.
> I'm using SPARC with gr712rc BSP.
>
> I'm following the steps outlined in the coverage analysis documentation
> https://docs.rtems.org/branches/master/user/testing/coverage.html
>
> However, when I run it on any tests, even sample/hello.exe I get the logs
> in lines of the following:
>
> `RTEMS Testing - Tester, 6.0.not_released  Command Line: ./rtems-test
> --rtems-tools=/home/yoman/RTEMS/rtems/6
> --log=log_leon3_sis --rtems-bsp=leon3-sis-cov --coverage
> /home/yoman/RTEMS/src/rtems/build/sparc/gr712rc/testsuites/validation/ts-validation-0.exe
>  Host: Linux ubuntu 5.13.0-35-generic #40~20.04.1-Ubuntu SMP Mon Mar 7
> 09:18:32 UTC 2022 x86_64
>  Python: 3.8.10 (default, Nov 26 2021, 20:14:08) [GCC 9.3.0]
> Host: Linux-5.13.0-35-generic-x86_64-with-glibc2.29 (Linux ubuntu
> 5.13.0-35-generic #40~20.04.1-Ubuntu SMP Mon Mar 7 09:18:32 UTC 2022 x86_64
> x86_64)
> [1/1] p:0 f:0 u:0 e:0 I:0 B:0 t:0 L:0 i:0 W:0 | sparc/leon3-sis:
> ts-validation-0.exe
>
> Passed:        1
> Failed:        0
> User Input:    0
> Expected Fail: 0
> Indeterminate: 0
> Benchmark:     0
> Timeout:       0
> Test too long: 0
> Invalid:       0
> Wrong Version: 0
> Wrong Build:   0
> Wrong Tools:   0
> Wrong Header:  0
> ----------------
> Total:         1
>
> Average test time: 0:00:02.008333
> Testing time     : 0:00:02.008333
>
> Running coverage analysis
> (/home/yoman/RTEMS/rtems/6/bin/leon3-sis-coverage)
> Function is both external and inlined: _Message_queue_Create Function is
> both external and inlined: _Message_queue_Create Function is both external
> and inlined: _Message_queue_Create Function is both external and inlined:
> _Thread_Initialize_information Function is both external and inlined:
> _Thread_Initialize_information Function is both external and inlined:
> _Thread_queue_Initialize Function is both external and inlined:
> _Thread_queue_Initialize Function is both external and inlined:
> _Thread_queue_Resume Function is both external and inlined:
> _Thread_queue_FIFO_first Function is both external and inlined:
> _Thread_queue_FIFO_first Function is both external and inlined:
> _Thread_CPU_budget_consume_and_yield
> Function is both external and inlined: _Thread_CPU_budget_consume_and_yield
> Function is both external and inlined: _Thread_Join Function is both
> external and inlined: _Thread_Join Function is both external and inlined:
> _Thread_Set_state_locked Function is both external and inlined:
> _Thread_Set_state_locked Function is both external and inlined:
> _Thread_Set_state_locked Function is both external and inlined:
> _Thread_Set_state_locked Function is both external and inlined:
> _Timecounter_Binuptime Function is both external and inlined:
> _Timecounter_Bintime Function is both external and inlined:
> _Timecounter_Getboottimebin Function is both external and inlined:
> _Thread_Priority_perform_actions Function is both external and inlined:
> _Thread_Priority_perform_actions Function is both external and inlined:
> _Thread_Clear_state_locked Function is both external and inlined:
> _Thread_Clear_state_locked Function is both external and inlined:
> _Thread_Clear_state_locked Function is both external and inlined:
> _Thread_Clear_state_locked Function is both external and inlined:
> _Malloc_System_state Function is both external and inlined:
> _Malloc_System_state Function is both external and inlined:
> _Malloc_System_state Function is both external and inlined:
> rtems_counter_initialize_converter
> Function is both external and inlined: rtems_counter_initialize_converter
> Function is both external and inlined: rtems_version_minor Function is
> both external and inlined: _Heap_Get_first_and_last_block Function is both
> external and inlined: _Heap_Get_first_and_last_block Error while running
> sparc-rtems6-objdump on
> /home/yoman/RTEMS/src/rtems/build/sparc/gr712rc/testsuites/validation/ts-validation-0.exe
> Objdump error in generating objdump
> Coverage time: 0:00:01.085511
> Coverage generating reports
> Coverage analysis finished: /home/yoman/RTEMS/rtems/6/bin`
>
> I've tried to do a fresh VM fresh RTEMS build, results are exactly the
> same.
> It seems like it's something to do with `objdump`? That would explain the
> generated report containing only unreferenced symbols.
>
> Any tips on what's wrong or where to start looking to try and fix it?
> I wanted to generate coverage reports for the tests I developed, but not
> sure if there's any other way than through making this part work.
>
> Thanks and all the best!
> Jerzy
> _______________________________________________
> devel mailing list
> devel at rtems.org
> http://lists.rtems.org/mailman/listinfo/devel
>