<div dir="ltr"><div dir="ltr"><br><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, May 17, 2019 at 1:05 PM Sebastian Huber <<a href="mailto:sebastian.huber@embedded-brains.de" target="_blank">sebastian.huber@embedded-brains.de</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On 20/03/2019 13:16, Sebastian Huber wrote:<br>
> Hello,<br>
><br>
> the RTEMS project has a covoar tool:<br>
><br>
> <a href="https://git.rtems.org/rtems-tools/tree/tester/covoar" rel="noreferrer" target="_blank">https://git.rtems.org/rtems-tools/tree/tester/covoar</a><br>
><br>
> I have seen some reports generated by it, but I never used it myself. <br>
> I have trouble to find out how it works. There is nothing in the <br>
> documentation:<br>
><br>
> cd rtems-docs<br>
> grep -ri covoar .<br>
><br>
> There is some documentation available here:<br>
><br>
> <a href="https://devel.rtems.org/wiki/GCI/Documentation/CoverageAnalysis/Coverage" rel="noreferrer" target="_blank">https://devel.rtems.org/wiki/GCI/Documentation/CoverageAnalysis/Coverage</a><br>
><br>
> If I invoke the rtems-test like this in the erc32 BSP build directory, <br>
> I get an error:<br>
><br>
> rtems-test --rtems-bsp=erc32-sis --rtems-tools=/build/rtems/5 <br>
> --coverage --target=sparc-rtems5 .<br>
> RTEMS Testing - Tester, 5.0.not_released<br>
> Command Line: /build/rtems/5/bin/rtems-test --rtems-bsp=erc32-sis <br>
> --rtems-tools=/build/rtems/5 --coverage --target=sparc-rtems5 .<br>
> Python: 3.6.5 (default, Mar 31 2018, 19:45:04) [GCC]<br>
> Host: Linux-4.12.14-lp150.12.48-default-x86_64-with-glibc2.3.4 (Linux <br>
> huber-nb-linux.suse 4.12.14-lp150.12.48-default #1 SMP Tue Feb 12 <br>
> 14:01:48 UTC 2019 (268f014) x86_64 x86_64)<br>
> Traceback (most recent call last):<br>
> File "/build/rtems/5/share/rtems/tester/rt/cmd-test.py", line 42, in <br>
> <module><br>
> test.run(sys.argv[1:], command_path = base)<br>
> File "/build/rtems/5/share/rtems/tester/rt/test.py", line 300, in run<br>
> trace = cov_trace)<br>
> File "/build/rtems/5/share/rtems/tester/rt/coverage.py", line 363, <br>
> in __init__<br>
> self.target = self.macros['target']<br>
> File "/build/rtems/5/share/rtems/rtemstoolkit/macros.py", line 181, <br>
> in __getitem__<br>
> raise IndexError('key: %s' % (key))<br>
> IndexError: key: target<br>
><br>
<br>
I made some progress to try it out myself. The next obstacle was this:<br>
<br>
rtems-test --rtems-tools=/build/rtems/5 --rtems-bsp=leon3-sis-cov <br>
--coverage --no-clean --log=leon3-test.log <br>
sparc-rtems5/c/leon3/testsuites/samples/hello.exe<br>
RTEMS Testing - Tester, 5.0.not_released<br>
[1/1] p:0 f:0 u:0 e:0 I:0 B:0 t:0 i:0 W:0 | sparc/leon3-sis: hello.exe<br>
Passed: 1<br>
Failed: 0<br>
User Input: 0<br>
Expected Fail: 0<br>
Indeterminate: 0<br>
Benchmark: 0<br>
Timeout: 0<br>
Invalid: 0<br>
Wrong Version: 0<br>
Wrong Build: 0<br>
Wrong Tools: 0<br>
----------------<br>
Total: 1<br>
Average test time: 0:00:01.505144<br>
Testing time : 0:00:01.505144<br>
error: symbol parser write failed for score<br>
<br>
To figure out what went wrong, I removed the try/except block from <br>
tester/rt/coverage.py and got:<br>
<br>
rtems-test --rtems-tools=/build/rtems/5 --rtems-bsp=leon3-sis-cov <br>
--coverage --no-clean --log=leon3-test.log <br>
sparc-rtems5/c/leon3/testsuites/samples/hello.exe<br>
RTEMS Testing - Tester, 5.0.not_released<br>
[1/1] p:0 f:0 u:0 e:0 I:0 B:0 t:0 i:0 W:0 | sparc/leon3-sis: hello.exe<br>
Passed: 1<br>
Failed: 0<br>
User Input: 0<br>
Expected Fail: 0<br>
Indeterminate: 0<br>
Benchmark: 0<br>
Timeout: 0<br>
Invalid: 0<br>
Wrong Version: 0<br>
Wrong Build: 0<br>
Wrong Tools: 0<br>
----------------<br>
Total: 1<br>
Average test time: 0:00:01.504073<br>
Testing time : 0:00:01.504073<br>
Traceback (most recent call last):<br>
File "/build/rtems/5/share/rtems/tester/rt/cmd-test.py", line 42, in <br>
<module><br>
test.run(sys.argv[1:], command_path = base)<br>
File "/build/rtems/5/share/rtems/tester/rt/test.py", line 387, in run<br>
coverage_runner.run()<br>
File "/build/rtems/5/share/rtems/tester/rt/coverage.py", line 401, in run<br>
parser.write_ini(sset)<br>
File "/build/rtems/5/share/rtems/tester/rt/coverage.py", line 293, in <br>
write_ini<br>
object_files = [o for o in os.listdir(self.symbol_sets[sset]) if <br>
o[-1] == 'o']<br>
FileNotFoundError: [Errno 2] No such file or directory: <br>
'/sparc-rtems5/c/leon3/cpukit/score/src'<br>
<br>
This is more useful error output from my point of view. It seems you <br>
have to use an absolute path for the test directory.<br>
<br>
rtems-test --rtems-tools=/build/rtems/5 --rtems-bsp=leon3-sis-cov <br>
--coverage --no-clean --log=leon3-test.log <br>
/build/git-build/b-leon3/sparc-rtems5/c/leon3/testsuites/samples/hello.exe<br>
[....]<br>
Coverage generating reports<br>
error: copying tree: /build/rtems/5/share/rtems/tester/covoar/covoar.css <br>
-> /build/git-build/b-leon3/leon3-sis-coverage/score/covoar.css: [Errno <br>
2] No such file or directory: <br>
'/build/rtems/5/share/rtems/tester/covoar/covoar.css'<br>
<br></blockquote><div>-rtems-tools also needs an absolute path. </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
I copied some files by hand. A full test run ended up in:<br>
<br>
rtems-test --rtems-tools=/build/rtems/5 --rtems-bsp=leon3-sis-cov <br>
--coverage --no-clean --log=leon3-test.log <br>
/build/git-build/b-leon3/sparc-rtems5/c/leon3/testsuites<br>
RTEMS Testing - Tester, 5.0.not_released<br>
[ 3/710] p:0 f:0 u:0 e:0 I:0 B:0 t:0 i:0 W:0 | <br>
sparc/leon3-sis: ada_nsecs.exe<br>
[...]<br>
[710/710] p:684 f:5 u:6 e:0 I:0 B:3 t:0 i:2 W:0 | <br>
sparc/leon3-sis: tmttest01.exe<br>
Passed: 692<br>
Failed: 6<br>
User Input: 6<br>
Expected Fail: 0<br>
Indeterminate: 0<br>
Benchmark: 3<br>
Timeout: 1<br>
Invalid: 2<br>
Wrong Version: 0<br>
Wrong Build: 0<br>
Wrong Tools: 0<br>
------------------<br>
Total: 710<br>
Average test time: 0:00:00.427096<br>
Testing time : 0:05:03.237825<br>
<br>
Running coverage analysis: score <br>
(/build/git-build/b-leon3/leon3-sis-coverage/score)<br>
Coverage time: 0:17:53.695293<br>
<br>
Running coverage analysis: rtems <br>
(/build/git-build/b-leon3/leon3-sis-coverage/rtems)<br>
Coverage time: 0:12:55.898577<br>
<br>
Running coverage analysis: sapi <br>
(/build/git-build/b-leon3/leon3-sis-coverage/sapi)<br>
error: coverage: covoar failure:: 10<br>
<br></blockquote><div>This is actually a bug in covoar or a bug somewhere in spversion01.exe</div><div>We have a ticket for this <a href="https://devel.rtems.org/ticket/3515" target="_blank">https://devel.rtems.org/ticket/3515</a>.</div><div><br></div><div>To track it down, I manually ran covoar with the trace files using the</div><div>following command ( note: you need --no-clean option in rtems-test for this)</div><div><br></div><div>covoar -S /home/lunatic/development/rtems/rtems-tools/tester/rtems/testing/coverage/leon3-sis-symbols.ini -O coverage -E/home/lunatic/development/rtems/rtems-tools/tester/rtems/testing/coverage/Explanations.txt -p RTEMS-5 sparc-rtems5/c/leon3/testsuites/sptests/spversion01.exe -v -fTSIM<br></div><div><br></div><div><div>====</div><div>.</div><div>.</div><div>.</div><div>Created unified coverage map for rtems_initialize_executive (0x0 - 0x4b)<br></div><div><div>Created unified coverage map for rtems_shutdown_executive (0x0 - 0x13)</div><div>Created unified coverage map for rtems_fatal_error_occurred (0x0 - 0x13)</div><div>Created unified coverage map for rtems_get_version_string (0x0 - 0xb)</div><div>Created unified coverage map for _IO_Initialize_all_drivers (0x0 - 0x4b)</div><div>Created unified coverage map for rtems_io_initialize (0x0 - 0x5b)</div><div>Created unified coverage map for rtems_version (0x0 - 0xb)</div><div>Created unified coverage map for rtems_version_major (0x0 - 0x7)</div><div>Created unified coverage map for rtems_version_minor (0x0 - 0x7)</div><div>error: ExecutableInfo::findCoverageMap: rtems_version_revision</div></div></div><div>====</div><div><br></div><div>So there's something in rtems_version_revision() in spversion01/</div><div>that's making it unhappy. </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
-- <br>
Sebastian Huber, embedded brains GmbH<br>
<br>
Address : Dornierstr. 4, D-82178 Puchheim, Germany<br>
Phone : +49 89 189 47 41-16<br>
Fax : +49 89 189 47 41-09<br>
E-Mail : <a href="mailto:sebastian.huber@embedded-brains.de" target="_blank">sebastian.huber@embedded-brains.de</a><br>
PGP : Public key available on request.<br>
<br>
Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG.<br>
<br>
_______________________________________________<br>
devel mailing list<br>
<a href="mailto:devel@rtems.org" target="_blank">devel@rtems.org</a><br>
<a href="http://lists.rtems.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.rtems.org/mailman/listinfo/devel</a></blockquote></div></div>