Support for test coverage analysis in RTEMS 6

Wed Mar 16 23:03:10 UTC 2022

Hello Jerzy,

Actually, the same configuration can be adapted to run in SIS (which is what we use most of the times). Inside the QDP you have the file qual-tool/config-variants/sparc-gr712rc-smp-user-qual.yml. Change the file to the following (you can copy and paste the content, replacing the current one):

build-directory: build-sparc-gr712rc-smp-user-qual
post-process-items:
- uid: /dirs/djf-svr-deploy/dir
  path: /directory
  action: set
  value: ${/variant:/deployment-directory}/user_doc/djf/svr
- uid: /dirs/ddf-sdd-deploy/dir
  path: /directory
  action: set
  value: ${/variant:/deployment-directory}/user_doc/ddf/sdd
- uid: /steps/build-djf-svr
  path: /config-file
  action: set
  value: rtems/djf/svr/config_user.yml
- uid: /package-build
  path: /links
  action: set
  value:
  - role: build-step
    uid: steps/build-bsp-qual-only
  - role: build-step
    uid: steps/build-bsp-qual-only-coverage
  - role: build-step
    uid: steps/run-local-target-qual-only
  - role: build-step
    uid: steps/run-local-target-qual-only-coverage
  - role: build-step
    uid: steps/build-ddf-sdd
  - role: build-step
    uid: steps/build-djf-svr
spec-paths:
- spec-spec
- spec-glossary
- config

Then follow the steps as described in section “11.3 Guidance for RTEMS Qualification in User’s Environment” to generate your own SVR. After you have this working example, it would be easier to see how the things work. The scripts/configuration files in the qual-tool folder could be adapted to what you want (i.e: just generate coverage for your tests). Feel free to ask me if you have any difficulty!

Best regards

José

From: Jerzy Jaskuc <jaskucj at tcd.ie>
Sent: 16 de março de 2022 19:16
To: VALDEZ Jose <jose.valdez at edisoft.pt>
Cc: devel at rtems.org
Subject: Re: Support for test coverage analysis in RTEMS 6

Hi José,

I'm aware of the QDP, although it is pretty big so I got confused and only followed  `11.3 Guidance for RTEMS Qualification in User’s Environment`, which didn't work as I assume it needs actual board for tests to not be`Invalid`
The reports generated are exactly what I'm looking for and steps you mentioned seems much more manageable!

I can see gcno files and I can see gcda bytes within the logs of the test run.
I will take a look at adapting the python scripts then.

Thanks a lot for your help!

All the best,
Jerzy

On Wed, 16 Mar 2022 at 17:05, VALDEZ Jose <jose.valdez at edisoft.pt<mailto:jose.valdez at edisoft.pt>> wrote:
Hello Jerzy,

Recently we concluded a project, the RTEMS SMP Qualification which also needed to gather coverage for RTEMS (which had the participation of Andrew Butterfield from TCD/LERO). I suggest that you take a look at  https://rtems-qual.io.esa.int/, download one of the QDPs (maybe you want the GR712RC, SMP) and look for the docs/djf/svr document. There you may find the coverage folder, with the html coverage reports and in the svr.pdf the coverage summary table.

If this is what you want for your test, I suggest to take a look at qual-tool/svrmanager.py and qual-tool/ccoverage_process.py.

Basically for this what you need to do:

* Compile RTEMS with coverage flags. This will generate the gcno files
* Run your test, the reports should generate the gcda bytes (Sebastian Huber added support for coverage in RTEMS itself)
* Use/adapt ccoverage_process.py to generate the coverage reports, which makes use of the gcovr tool.

Hope this helps

José

-----Original Message-----
From: devel <devel-bounces at rtems.org<mailto:devel-bounces at rtems.org>> On Behalf Of Jerzy Jaskuc
Sent: 16 de março de 2022 16:46
To: devel at rtems.org<mailto:devel at rtems.org>
Subject: Support for test coverage analysis in RTEMS 6

Hi,

I've been looking into generating code coverage reports from RTEMS tests in RTEMS 6.
I'm using SPARC with gr712rc BSP.

I'm following the steps outlined in the coverage analysis documentation https://docs.rtems.org/branches/master/user/testing/coverage.html

However, when I run it on any tests, even sample/hello.exe I get the logs in lines of the following:

`RTEMS Testing - Tester, 6.0.not_released  Command Line: ./rtems-test --rtems-tools=/home/yoman/RTEMS/rtems/6
--log=log_leon3_sis --rtems-bsp=leon3-sis-cov --coverage /home/yoman/RTEMS/src/rtems/build/sparc/gr712rc/testsuites/validation/ts-validation-0.exe
 Host: Linux ubuntu 5.13.0-35-generic #40~20.04.1-Ubuntu SMP Mon Mar 7
09:18:32 UTC 2022 x86_64
 Python: 3.8.10 (default, Nov 26 2021, 20:14:08) [GCC 9.3.0]
Host: Linux-5.13.0-35-generic-x86_64-with-glibc2.29 (Linux ubuntu 5.13.0-35-generic #40~20.04.1-Ubuntu SMP Mon Mar 7 09:18:32 UTC 2022 x86_64
x86_64)
[1/1] p:0 f:0 u:0 e:0 I:0 B:0 t:0 L:0 i:0 W:0 | sparc/leon3-sis:
ts-validation-0.exe

Passed:        1
Failed:        0
User Input:    0
Expected Fail: 0
Indeterminate: 0
Benchmark:     0
Timeout:       0
Test too long: 0
Invalid:       0
Wrong Version: 0
Wrong Build:   0
Wrong Tools:   0
Wrong Header:  0
----------------
Total:         1

Average test time: 0:00:02.008333
Testing time     : 0:00:02.008333

Running coverage analysis (/home/yoman/RTEMS/rtems/6/bin/leon3-sis-coverage)
Function is both external and inlined: _Message_queue_Create Function is both external and inlined: _Message_queue_Create Function is both external and inlined: _Message_queue_Create Function is both external and inlined: _Thread_Initialize_information Function is both external and inlined: _Thread_Initialize_information Function is both external and inlined: _Thread_queue_Initialize Function is both external and inlined: _Thread_queue_Initialize Function is both external and inlined: _Thread_queue_Resume Function is both external and inlined: _Thread_queue_FIFO_first Function is both external and inlined: _Thread_queue_FIFO_first Function is both external and inlined: _Thread_CPU_budget_consume_and_yield
Function is both external and inlined: _Thread_CPU_budget_consume_and_yield
Function is both external and inlined: _Thread_Join Function is both external and inlined: _Thread_Join Function is both external and inlined: _Thread_Set_state_locked Function is both external and inlined: _Thread_Set_state_locked Function is both external and inlined: _Thread_Set_state_locked Function is both external and inlined: _Thread_Set_state_locked Function is both external and inlined: _Timecounter_Binuptime Function is both external and inlined: _Timecounter_Bintime Function is both external and inlined: _Timecounter_Getboottimebin Function is both external and inlined: _Thread_Priority_perform_actions Function is both external and inlined: _Thread_Priority_perform_actions Function is both external and inlined: _Thread_Clear_state_locked Function is both external and inlined: _Thread_Clear_state_locked Function is both external and inlined: _Thread_Clear_state_locked Function is both external and inlined: _Thread_Clear_state_locked Function is both external and inlined: _Malloc_System_state Function is both external and inlined: _Malloc_System_state Function is both external and inlined: _Malloc_System_state Function is both external and inlined: rtems_counter_initialize_converter
Function is both external and inlined: rtems_counter_initialize_converter
Function is both external and inlined: rtems_version_minor Function is both external and inlined: _Heap_Get_first_and_last_block Function is both external and inlined: _Heap_Get_first_and_last_block Error while running sparc-rtems6-objdump on /home/yoman/RTEMS/src/rtems/build/sparc/gr712rc/testsuites/validation/ts-validation-0.exe
Objdump error in generating objdump
Coverage time: 0:00:01.085511
Coverage generating reports
Coverage analysis finished: /home/yoman/RTEMS/rtems/6/bin`

I've tried to do a fresh VM fresh RTEMS build, results are exactly the same.
It seems like it's something to do with `objdump`? That would explain the generated report containing only unreferenced symbols.

Any tips on what's wrong or where to start looking to try and fix it?
I wanted to generate coverage reports for the tests I developed, but not sure if there's any other way than through making this part work.

Thanks and all the best!
Jerzy
_______________________________________________
devel mailing list
devel at rtems.org<mailto:devel at rtems.org>
http://lists.rtems.org/mailman/listinfo/devel