How can I use the covoar tool?

[Sebastian Huber]

On 20/03/2019 13:16, Sebastian Huber wrote:
> Hello,
>
> the RTEMS project has a covoar tool:
>
> https://git.rtems.org/rtems-tools/tree/tester/covoar
>
> I have seen some reports generated by it, but I never used it myself. 
> I have trouble to find out how it works. There is nothing in the 
> documentation:
>
> cd rtems-docs
> grep -ri covoar .
>
> There is some documentation available here:
>
> https://devel.rtems.org/wiki/GCI/Documentation/CoverageAnalysis/Coverage
>
> If I invoke the rtems-test like this in the erc32 BSP build directory, 
> I get an error:
>
> rtems-test --rtems-bsp=erc32-sis --rtems-tools=/build/rtems/5 
> --coverage --target=sparc-rtems5 .
> RTEMS Testing - Tester, 5.0.not_released
>  Command Line: /build/rtems/5/bin/rtems-test --rtems-bsp=erc32-sis 
> --rtems-tools=/build/rtems/5 --coverage --target=sparc-rtems5 .
>  Python: 3.6.5 (default, Mar 31 2018, 19:45:04) [GCC]
> Host: Linux-4.12.14-lp150.12.48-default-x86_64-with-glibc2.3.4 (Linux 
> huber-nb-linux.suse 4.12.14-lp150.12.48-default #1 SMP Tue Feb 12 
> 14:01:48 UTC 2019 (268f014) x86_64 x86_64)
> Traceback (most recent call last):
>   File "/build/rtems/5/share/rtems/tester/rt/cmd-test.py", line 42, in 
> <module>
>     test.run(sys.argv[1:], command_path = base)
>   File "/build/rtems/5/share/rtems/tester/rt/test.py", line 300, in run
>     trace = cov_trace)
>   File "/build/rtems/5/share/rtems/tester/rt/coverage.py", line 363, 
> in __init__
>     self.target = self.macros['target']
>   File "/build/rtems/5/share/rtems/rtemstoolkit/macros.py", line 181, 
> in __getitem__
>     raise IndexError('key: %s' % (key))
> IndexError: key: target
>

I made some progress to try it out myself. The next obstacle was this:

rtems-test --rtems-tools=/build/rtems/5 --rtems-bsp=leon3-sis-cov 
--coverage --no-clean --log=leon3-test.log 
sparc-rtems5/c/leon3/testsuites/samples/hello.exe
RTEMS Testing - Tester, 5.0.not_released
[1/1] p:0 f:0 u:0 e:0 I:0 B:0 t:0 i:0 W:0 | sparc/leon3-sis: hello.exe
Passed:        1
Failed:        0
User Input:    0
Expected Fail: 0
Indeterminate: 0
Benchmark:     0
Timeout:       0
Invalid:       0
Wrong Version: 0
Wrong Build:   0
Wrong Tools:   0
----------------
Total:         1
Average test time: 0:00:01.505144
Testing time     : 0:00:01.505144
error: symbol parser write failed for score

To figure out what went wrong, I removed the try/except block from 
tester/rt/coverage.py and got:

rtems-test --rtems-tools=/build/rtems/5 --rtems-bsp=leon3-sis-cov 
--coverage --no-clean --log=leon3-test.log 
sparc-rtems5/c/leon3/testsuites/samples/hello.exe
RTEMS Testing - Tester, 5.0.not_released
[1/1] p:0 f:0 u:0 e:0 I:0 B:0 t:0 i:0 W:0 | sparc/leon3-sis: hello.exe
Passed:        1
Failed:        0
User Input:    0
Expected Fail: 0
Indeterminate: 0
Benchmark:     0
Timeout:       0
Invalid:       0
Wrong Version: 0
Wrong Build:   0
Wrong Tools:   0
----------------
Total:         1
Average test time: 0:00:01.504073
Testing time     : 0:00:01.504073
Traceback (most recent call last):
   File "/build/rtems/5/share/rtems/tester/rt/cmd-test.py", line 42, in 
<module>
     test.run(sys.argv[1:], command_path = base)
   File "/build/rtems/5/share/rtems/tester/rt/test.py", line 387, in run
     coverage_runner.run()
   File "/build/rtems/5/share/rtems/tester/rt/coverage.py", line 401, in run
     parser.write_ini(sset)
   File "/build/rtems/5/share/rtems/tester/rt/coverage.py", line 293, in 
write_ini
     object_files = [o for o in os.listdir(self.symbol_sets[sset]) if 
o[-1] == 'o']
FileNotFoundError: [Errno 2] No such file or directory: 
'/sparc-rtems5/c/leon3/cpukit/score/src'

This is more useful error output from my point of view. It seems you 
have to use an absolute path for the test directory.

rtems-test --rtems-tools=/build/rtems/5 --rtems-bsp=leon3-sis-cov 
--coverage --no-clean --log=leon3-test.log 
/build/git-build/b-leon3/sparc-rtems5/c/leon3/testsuites/samples/hello.exe
[....]
Coverage generating reports
error: copying tree: /build/rtems/5/share/rtems/tester/covoar/covoar.css 
-> /build/git-build/b-leon3/leon3-sis-coverage/score/covoar.css: [Errno 
2] No such file or directory: 
'/build/rtems/5/share/rtems/tester/covoar/covoar.css'

I copied some files by hand. A full test run ended up in:

rtems-test --rtems-tools=/build/rtems/5 --rtems-bsp=leon3-sis-cov 
--coverage --no-clean --log=leon3-test.log 
/build/git-build/b-leon3/sparc-rtems5/c/leon3/testsuites
RTEMS Testing - Tester, 5.0.not_released
[  3/710] p:0   f:0   u:0   e:0   I:0   B:0   t:0   i:0   W:0   | 
sparc/leon3-sis: ada_nsecs.exe
[...]
[710/710] p:684 f:5   u:6   e:0   I:0   B:3   t:0   i:2   W:0   | 
sparc/leon3-sis: tmttest01.exe
Passed:        692
Failed:          6
User Input:      6
Expected Fail:   0
Indeterminate:   0
Benchmark:       3
Timeout:         1
Invalid:         2
Wrong Version:   0
Wrong Build:     0
Wrong Tools:     0
------------------
Total:         710
Average test time: 0:00:00.427096
Testing time     : 0:05:03.237825

Running coverage analysis: score 
(/build/git-build/b-leon3/leon3-sis-coverage/score)
Coverage time: 0:17:53.695293

Running coverage analysis: rtems 
(/build/git-build/b-leon3/leon3-sis-coverage/rtems)
Coverage time: 0:12:55.898577

Running coverage analysis: sapi 
(/build/git-build/b-leon3/leon3-sis-coverage/sapi)
error: coverage: covoar failure:: 10

-- 
Sebastian Huber, embedded brains GmbH

Address : Dornierstr. 4, D-82178 Puchheim, Germany
Phone   : +49 89 189 47 41-16
Fax     : +49 89 189 47 41-09
E-Mail  : sebastian.huber at embedded-brains.de
PGP     : Public key available on request.

Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG.