Support for test coverage analysis in RTEMS 6
Jerzy Jaskuc
jaskucj at tcd.ie
Thu Mar 17 10:43:31 UTC 2022
Hi José,
This is exactly what I needed, thanks so much!
I'm able to generate the reports successfully.
Just a few small questions:
I see that in `sparc-gr712rc-smp-user-qual.yml` there's a step to build
`qual-only` and a step for `qual-only-coverage`.
Do I need the steps for `qual-only` if I'm only interested in the coverage?
What exactly does it do?
Another thing is that for me it will be useful if i can generate coverage
for existing tests + my tests, so I can sort of compare them.
I can see that there is `src/rtems-qual-only` but not
`src/rtems-qual-only-coverage`, which brings a question of how are
executables in `sparc-rtems6` directory for `-coverage` generated?
Will it work if I move my tests into `src/rtems-qual-only` and link them in
specs?
Thanks a lot for your help,
All the best!
Jerzy
On Wed, 16 Mar 2022 at 23:03, VALDEZ Jose <jose.valdez at edisoft.pt> wrote:
> Hello Jerzy,
>
>
>
> Actually, the same configuration can be adapted to run in SIS (which is
> what we use most of the times). Inside the QDP you have the file
> qual-tool/config-variants/sparc-gr712rc-smp-user-qual.yml. Change the file
> to the following (you can copy and paste the content, replacing the current
> one):
>
>
>
> build-directory: build-sparc-gr712rc-smp-user-qual
>
> post-process-items:
>
> - uid: /dirs/djf-svr-deploy/dir
>
> path: /directory
>
> action: set
>
> value: ${/variant:/deployment-directory}/user_doc/djf/svr
>
> - uid: /dirs/ddf-sdd-deploy/dir
>
> path: /directory
>
> action: set
>
> value: ${/variant:/deployment-directory}/user_doc/ddf/sdd
>
> - uid: /steps/build-djf-svr
>
> path: /config-file
>
> action: set
>
> value: rtems/djf/svr/config_user.yml
>
> - uid: /package-build
>
> path: /links
>
> action: set
>
> value:
>
> - role: build-step
>
> uid: steps/build-bsp-qual-only
>
> - role: build-step
>
> uid: steps/build-bsp-qual-only-coverage
>
> - role: build-step
>
> uid: steps/run-local-target-qual-only
>
> - role: build-step
>
> uid: steps/run-local-target-qual-only-coverage
>
> - role: build-step
>
> uid: steps/build-ddf-sdd
>
> - role: build-step
>
> uid: steps/build-djf-svr
>
> spec-paths:
>
> - spec-spec
>
> - spec-glossary
>
> - config
>
>
>
> Then follow the steps as described in section “11.3 Guidance for RTEMS
> Qualification in User’s Environment” to generate your own SVR. After you
> have this working example, it would be easier to see how the things work.
> The scripts/configuration files in the qual-tool folder could be adapted to
> what you want (i.e: just generate coverage for your tests). Feel free to
> ask me if you have any difficulty!
>
>
>
> Best regards
>
>
>
> José
>
>
>
> *From:* Jerzy Jaskuc <jaskucj at tcd.ie>
> *Sent:* 16 de março de 2022 19:16
> *To:* VALDEZ Jose <jose.valdez at edisoft.pt>
> *Cc:* devel at rtems.org
> *Subject:* Re: Support for test coverage analysis in RTEMS 6
>
>
>
> Hi José,
>
>
>
> I'm aware of the QDP, although it is pretty big so I got confused and only
> followed `11.3 Guidance for RTEMS Qualification in User’s Environment`,
> which didn't work as I assume it needs actual board for tests to not
> be`Invalid`
>
> The reports generated are exactly what I'm looking for and steps you
> mentioned seems much more manageable!
>
>
>
> I can see gcno files and I can see gcda bytes within the logs of the test
> run.
>
> I will take a look at adapting the python scripts then.
>
>
>
> Thanks a lot for your help!
>
>
>
> All the best,
>
> Jerzy
>
>
>
>
>
>
>
> On Wed, 16 Mar 2022 at 17:05, VALDEZ Jose <jose.valdez at edisoft.pt> wrote:
>
> Hello Jerzy,
>
> Recently we concluded a project, the RTEMS SMP Qualification which also
> needed to gather coverage for RTEMS (which had the participation of Andrew
> Butterfield from TCD/LERO). I suggest that you take a look at
> https://rtems-qual.io.esa.int/, download one of the QDPs (maybe you want
> the GR712RC, SMP) and look for the docs/djf/svr document. There you may
> find the coverage folder, with the html coverage reports and in the svr.pdf
> the coverage summary table.
>
> If this is what you want for your test, I suggest to take a look at
> qual-tool/svrmanager.py and qual-tool/ccoverage_process.py.
>
> Basically for this what you need to do:
>
> * Compile RTEMS with coverage flags. This will generate the gcno files
> * Run your test, the reports should generate the gcda bytes (Sebastian
> Huber added support for coverage in RTEMS itself)
> * Use/adapt ccoverage_process.py to generate the coverage reports, which
> makes use of the gcovr tool.
>
> Hope this helps
>
> José
>
> -----Original Message-----
> From: devel <devel-bounces at rtems.org> On Behalf Of Jerzy Jaskuc
> Sent: 16 de março de 2022 16:46
> To: devel at rtems.org
> Subject: Support for test coverage analysis in RTEMS 6
>
> Hi,
>
> I've been looking into generating code coverage reports from RTEMS tests
> in RTEMS 6.
> I'm using SPARC with gr712rc BSP.
>
> I'm following the steps outlined in the coverage analysis documentation
> https://docs.rtems.org/branches/master/user/testing/coverage.html
>
> However, when I run it on any tests, even sample/hello.exe I get the logs
> in lines of the following:
>
> `RTEMS Testing - Tester, 6.0.not_released Command Line: ./rtems-test
> --rtems-tools=/home/yoman/RTEMS/rtems/6
> --log=log_leon3_sis --rtems-bsp=leon3-sis-cov --coverage
> /home/yoman/RTEMS/src/rtems/build/sparc/gr712rc/testsuites/validation/ts-validation-0.exe
> Host: Linux ubuntu 5.13.0-35-generic #40~20.04.1-Ubuntu SMP Mon Mar 7
> 09:18:32 UTC 2022 x86_64
> Python: 3.8.10 (default, Nov 26 2021, 20:14:08) [GCC 9.3.0]
> Host: Linux-5.13.0-35-generic-x86_64-with-glibc2.29 (Linux ubuntu
> 5.13.0-35-generic #40~20.04.1-Ubuntu SMP Mon Mar 7 09:18:32 UTC 2022 x86_64
> x86_64)
> [1/1] p:0 f:0 u:0 e:0 I:0 B:0 t:0 L:0 i:0 W:0 | sparc/leon3-sis:
> ts-validation-0.exe
>
> Passed: 1
> Failed: 0
> User Input: 0
> Expected Fail: 0
> Indeterminate: 0
> Benchmark: 0
> Timeout: 0
> Test too long: 0
> Invalid: 0
> Wrong Version: 0
> Wrong Build: 0
> Wrong Tools: 0
> Wrong Header: 0
> ----------------
> Total: 1
>
> Average test time: 0:00:02.008333
> Testing time : 0:00:02.008333
>
> Running coverage analysis
> (/home/yoman/RTEMS/rtems/6/bin/leon3-sis-coverage)
> Function is both external and inlined: _Message_queue_Create Function is
> both external and inlined: _Message_queue_Create Function is both external
> and inlined: _Message_queue_Create Function is both external and inlined:
> _Thread_Initialize_information Function is both external and inlined:
> _Thread_Initialize_information Function is both external and inlined:
> _Thread_queue_Initialize Function is both external and inlined:
> _Thread_queue_Initialize Function is both external and inlined:
> _Thread_queue_Resume Function is both external and inlined:
> _Thread_queue_FIFO_first Function is both external and inlined:
> _Thread_queue_FIFO_first Function is both external and inlined:
> _Thread_CPU_budget_consume_and_yield
> Function is both external and inlined: _Thread_CPU_budget_consume_and_yield
> Function is both external and inlined: _Thread_Join Function is both
> external and inlined: _Thread_Join Function is both external and inlined:
> _Thread_Set_state_locked Function is both external and inlined:
> _Thread_Set_state_locked Function is both external and inlined:
> _Thread_Set_state_locked Function is both external and inlined:
> _Thread_Set_state_locked Function is both external and inlined:
> _Timecounter_Binuptime Function is both external and inlined:
> _Timecounter_Bintime Function is both external and inlined:
> _Timecounter_Getboottimebin Function is both external and inlined:
> _Thread_Priority_perform_actions Function is both external and inlined:
> _Thread_Priority_perform_actions Function is both external and inlined:
> _Thread_Clear_state_locked Function is both external and inlined:
> _Thread_Clear_state_locked Function is both external and inlined:
> _Thread_Clear_state_locked Function is both external and inlined:
> _Thread_Clear_state_locked Function is both external and inlined:
> _Malloc_System_state Function is both external and inlined:
> _Malloc_System_state Function is both external and inlined:
> _Malloc_System_state Function is both external and inlined:
> rtems_counter_initialize_converter
> Function is both external and inlined: rtems_counter_initialize_converter
> Function is both external and inlined: rtems_version_minor Function is
> both external and inlined: _Heap_Get_first_and_last_block Function is both
> external and inlined: _Heap_Get_first_and_last_block Error while running
> sparc-rtems6-objdump on
> /home/yoman/RTEMS/src/rtems/build/sparc/gr712rc/testsuites/validation/ts-validation-0.exe
> Objdump error in generating objdump
> Coverage time: 0:00:01.085511
> Coverage generating reports
> Coverage analysis finished: /home/yoman/RTEMS/rtems/6/bin`
>
> I've tried to do a fresh VM fresh RTEMS build, results are exactly the
> same.
> It seems like it's something to do with `objdump`? That would explain the
> generated report containing only unreferenced symbols.
>
> Any tips on what's wrong or where to start looking to try and fix it?
> I wanted to generate coverage reports for the tests I developed, but not
> sure if there's any other way than through making this part work.
>
> Thanks and all the best!
> Jerzy
> _______________________________________________
> devel mailing list
> devel at rtems.org
> http://lists.rtems.org/mailman/listinfo/devel
>
>
More information about the devel
mailing list