Support for test coverage analysis in RTEMS 6
VALDEZ Jose
jose.valdez at edisoft.pt
Thu Mar 17 11:24:23 UTC 2022
Hello Jerzy,
Please see my answers below.
José
From: Jerzy Jaskuc <jaskucj at tcd.ie>
Sent: 17 de março de 2022 10:44
To: VALDEZ Jose <jose.valdez at edisoft.pt>
Cc: devel at rtems.org
Subject: Re: Support for test coverage analysis in RTEMS 6
Hi José,
This is exactly what I needed, thanks so much!
I'm able to generate the reports successfully.
[JV]: Good!
Just a few small questions:
I see that in `sparc-gr712rc-smp-user-qual.yml` there's a step to build `qual-only` and a step for `qual-only-coverage`.
Do I need the steps for `qual-only` if I'm only interested in the coverage? What exactly does it do?
[JV]: These two steps compile RTEMS: ‘qual-only’, normal compilation, ‘qual-only-coverage’, coverage compilation. In this configuration you run the same tests in normal and coverage mode. It looks like you only want coverage, but you should take into attention:
* If you want to remove the normal mode, then you need to adapt the qual-tool accordingly
* In principle you shall run the tests in both modes. It may be the case that in coverage mode the tests pass and in standard mode they fail (the coverage introduces instrumented code, which might impact your test/application). I recommend that you make sure that your tests pass in both modes.
I do not know if you are aware, but the ‘qual-only’ RTEMS is a subset of the full RTEMS with the space-qualified parts. You want to use this one or the full RTEMS? If it is the later one, then the scripts need to be adapted to use the full RTEMS version.
Another thing is that for me it will be useful if i can generate coverage for existing tests + my tests, so I can sort of compare them.
I can see that there is `src/rtems-qual-only` but not `src/rtems-qual-only-coverage`, which brings a question of how are executables in `sparc-rtems6` directory for `-coverage` generated?
Will it work if I move my tests into `src/rtems-qual-only` and link them in specs?
[JV]: the src/rtems-qual-only are the source files of RTEMS (qualified subset, as described above). When compiling you generate the following:
* src/rtems-qual-only/build/sparc/gr712rc-qual-only: build products of the qual-only build
* src/rtems-qual-only/build/sparc/gr712rc-qual-only-coverage: build products of the qual-only-coverage build, includes the gcno files
* sparc-rtems6/gr712rc-qual-only: rtems libraries, normal mode (including the testsuite executables)
* sparc-rtems6/gr712rc-qual-only-coverage: rtems libraries, coverage mode (including the testsuite executables)
I think what you want is to generate coverage for your tests only. You can include your source and then update the RTEMS scripts to compile your tests. Alternatively, you can hack and compile your test apart and include the binary alongside with the RTEMS testsuite binaries. To run only your test, you could delete all the other test executables from the folder. Also you might adapt run-local-target-qual-only.yml and run-local-target-qual-only-coverage.yml, the field ignore-tests you can add the list of all RTEMS tests (or the tests you want to ignore).
Thanks a lot for your help,
All the best!
Jerzy
On Wed, 16 Mar 2022 at 23:03, VALDEZ Jose <jose.valdez at edisoft.pt<mailto:jose.valdez at edisoft.pt>> wrote:
Hello Jerzy,
Actually, the same configuration can be adapted to run in SIS (which is what we use most of the times). Inside the QDP you have the file qual-tool/config-variants/sparc-gr712rc-smp-user-qual.yml. Change the file to the following (you can copy and paste the content, replacing the current one):
build-directory: build-sparc-gr712rc-smp-user-qual
post-process-items:
- uid: /dirs/djf-svr-deploy/dir
path: /directory
action: set
value: ${/variant:/deployment-directory}/user_doc/djf/svr
- uid: /dirs/ddf-sdd-deploy/dir
path: /directory
action: set
value: ${/variant:/deployment-directory}/user_doc/ddf/sdd
- uid: /steps/build-djf-svr
path: /config-file
action: set
value: rtems/djf/svr/config_user.yml
- uid: /package-build
path: /links
action: set
value:
- role: build-step
uid: steps/build-bsp-qual-only
- role: build-step
uid: steps/build-bsp-qual-only-coverage
- role: build-step
uid: steps/run-local-target-qual-only
- role: build-step
uid: steps/run-local-target-qual-only-coverage
- role: build-step
uid: steps/build-ddf-sdd
- role: build-step
uid: steps/build-djf-svr
spec-paths:
- spec-spec
- spec-glossary
- config
Then follow the steps as described in section “11.3 Guidance for RTEMS Qualification in User’s Environment” to generate your own SVR. After you have this working example, it would be easier to see how the things work. The scripts/configuration files in the qual-tool folder could be adapted to what you want (i.e: just generate coverage for your tests). Feel free to ask me if you have any difficulty!
Best regards
José
From: Jerzy Jaskuc <jaskucj at tcd.ie<mailto:jaskucj at tcd.ie>>
Sent: 16 de março de 2022 19:16
To: VALDEZ Jose <jose.valdez at edisoft.pt<mailto:jose.valdez at edisoft.pt>>
Cc: devel at rtems.org<mailto:devel at rtems.org>
Subject: Re: Support for test coverage analysis in RTEMS 6
Hi José,
I'm aware of the QDP, although it is pretty big so I got confused and only followed `11.3 Guidance for RTEMS Qualification in User’s Environment`, which didn't work as I assume it needs actual board for tests to not be`Invalid`
The reports generated are exactly what I'm looking for and steps you mentioned seems much more manageable!
I can see gcno files and I can see gcda bytes within the logs of the test run.
I will take a look at adapting the python scripts then.
Thanks a lot for your help!
All the best,
Jerzy
On Wed, 16 Mar 2022 at 17:05, VALDEZ Jose <jose.valdez at edisoft.pt<mailto:jose.valdez at edisoft.pt>> wrote:
Hello Jerzy,
Recently we concluded a project, the RTEMS SMP Qualification which also needed to gather coverage for RTEMS (which had the participation of Andrew Butterfield from TCD/LERO). I suggest that you take a look at https://rtems-qual.io.esa.int/, download one of the QDPs (maybe you want the GR712RC, SMP) and look for the docs/djf/svr document. There you may find the coverage folder, with the html coverage reports and in the svr.pdf the coverage summary table.
If this is what you want for your test, I suggest to take a look at qual-tool/svrmanager.py and qual-tool/ccoverage_process.py.
Basically for this what you need to do:
* Compile RTEMS with coverage flags. This will generate the gcno files
* Run your test, the reports should generate the gcda bytes (Sebastian Huber added support for coverage in RTEMS itself)
* Use/adapt ccoverage_process.py to generate the coverage reports, which makes use of the gcovr tool.
Hope this helps
José
-----Original Message-----
From: devel <devel-bounces at rtems.org<mailto:devel-bounces at rtems.org>> On Behalf Of Jerzy Jaskuc
Sent: 16 de março de 2022 16:46
To: devel at rtems.org<mailto:devel at rtems.org>
Subject: Support for test coverage analysis in RTEMS 6
Hi,
I've been looking into generating code coverage reports from RTEMS tests in RTEMS 6.
I'm using SPARC with gr712rc BSP.
I'm following the steps outlined in the coverage analysis documentation https://docs.rtems.org/branches/master/user/testing/coverage.html
However, when I run it on any tests, even sample/hello.exe I get the logs in lines of the following:
`RTEMS Testing - Tester, 6.0.not_released Command Line: ./rtems-test --rtems-tools=/home/yoman/RTEMS/rtems/6
--log=log_leon3_sis --rtems-bsp=leon3-sis-cov --coverage /home/yoman/RTEMS/src/rtems/build/sparc/gr712rc/testsuites/validation/ts-validation-0.exe
Host: Linux ubuntu 5.13.0-35-generic #40~20.04.1-Ubuntu SMP Mon Mar 7
09:18:32 UTC 2022 x86_64
Python: 3.8.10 (default, Nov 26 2021, 20:14:08) [GCC 9.3.0]
Host: Linux-5.13.0-35-generic-x86_64-with-glibc2.29 (Linux ubuntu 5.13.0-35-generic #40~20.04.1-Ubuntu SMP Mon Mar 7 09:18:32 UTC 2022 x86_64
x86_64)
[1/1] p:0 f:0 u:0 e:0 I:0 B:0 t:0 L:0 i:0 W:0 | sparc/leon3-sis:
ts-validation-0.exe
Passed: 1
Failed: 0
User Input: 0
Expected Fail: 0
Indeterminate: 0
Benchmark: 0
Timeout: 0
Test too long: 0
Invalid: 0
Wrong Version: 0
Wrong Build: 0
Wrong Tools: 0
Wrong Header: 0
----------------
Total: 1
Average test time: 0:00:02.008333
Testing time : 0:00:02.008333
Running coverage analysis (/home/yoman/RTEMS/rtems/6/bin/leon3-sis-coverage)
Function is both external and inlined: _Message_queue_Create Function is both external and inlined: _Message_queue_Create Function is both external and inlined: _Message_queue_Create Function is both external and inlined: _Thread_Initialize_information Function is both external and inlined: _Thread_Initialize_information Function is both external and inlined: _Thread_queue_Initialize Function is both external and inlined: _Thread_queue_Initialize Function is both external and inlined: _Thread_queue_Resume Function is both external and inlined: _Thread_queue_FIFO_first Function is both external and inlined: _Thread_queue_FIFO_first Function is both external and inlined: _Thread_CPU_budget_consume_and_yield
Function is both external and inlined: _Thread_CPU_budget_consume_and_yield
Function is both external and inlined: _Thread_Join Function is both external and inlined: _Thread_Join Function is both external and inlined: _Thread_Set_state_locked Function is both external and inlined: _Thread_Set_state_locked Function is both external and inlined: _Thread_Set_state_locked Function is both external and inlined: _Thread_Set_state_locked Function is both external and inlined: _Timecounter_Binuptime Function is both external and inlined: _Timecounter_Bintime Function is both external and inlined: _Timecounter_Getboottimebin Function is both external and inlined: _Thread_Priority_perform_actions Function is both external and inlined: _Thread_Priority_perform_actions Function is both external and inlined: _Thread_Clear_state_locked Function is both external and inlined: _Thread_Clear_state_locked Function is both external and inlined: _Thread_Clear_state_locked Function is both external and inlined: _Thread_Clear_state_locked Function is both external and inlined: _Malloc_System_state Function is both external and inlined: _Malloc_System_state Function is both external and inlined: _Malloc_System_state Function is both external and inlined: rtems_counter_initialize_converter
Function is both external and inlined: rtems_counter_initialize_converter
Function is both external and inlined: rtems_version_minor Function is both external and inlined: _Heap_Get_first_and_last_block Function is both external and inlined: _Heap_Get_first_and_last_block Error while running sparc-rtems6-objdump on /home/yoman/RTEMS/src/rtems/build/sparc/gr712rc/testsuites/validation/ts-validation-0.exe
Objdump error in generating objdump
Coverage time: 0:00:01.085511
Coverage generating reports
Coverage analysis finished: /home/yoman/RTEMS/rtems/6/bin`
I've tried to do a fresh VM fresh RTEMS build, results are exactly the same.
It seems like it's something to do with `objdump`? That would explain the generated report containing only unreferenced symbols.
Any tips on what's wrong or where to start looking to try and fix it?
I wanted to generate coverage reports for the tests I developed, but not sure if there's any other way than through making this part work.
Thanks and all the best!
Jerzy
_______________________________________________
devel mailing list
devel at rtems.org<mailto:devel at rtems.org>
http://lists.rtems.org/mailman/listinfo/devel
More information about the devel
mailing list