<div dir="ltr">Jiri recently added tsim's coverage format to sis. Perhaps it isn't right on all variations.<div><br></div><div>When you get reports, it is straight forward to run it on a single test and see what </div><div>methods and lines it tests.</div><div><br></div><div>Internally it could but does not keep currently track of which tests executed each line </div><div>of source code. That might be helpful in getting a global view of what tests exercise </div><div>what. But going a test at a time and focusing on what you think it does is probably</div><div>effective. </div><div><br></div><div>FWIW the coverage was near 100% when compiled at -Os before the rework to</div><div>report per subdirectory and then to use DWARF info. I can't imaging the testsuite</div><div>has gotten worse since then. </div><div><br></div><div>Coverage analysis at -Os is much easier because -O2 results in loop unrolling,</div><div>instruction interleaving, etc.</div><div><br></div><div><br></div><div>--joel</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Mar 20, 2019 at 11:11 AM Vijay Kumar Banerjee <<a href="mailto:vijaykumar9597@gmail.com">vijaykumar9597@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, 20 Mar 2019 at 21:20, Vijay Kumar Banerjee <<a href="mailto:vijaykumar9597@gmail.com" target="_blank">vijaykumar9597@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><br clear="all"><div><div dir="ltr" class="gmail-m_-2534866601514226672gmail-m_1811619683923643466gmail_signature"><div dir="ltr"><div dir="ltr"><br></div></div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, 20 Mar 2019 at 21:11, Sebastian Huber <<a href="mailto:sebastian.huber@embedded-brains.de" target="_blank">sebastian.huber@embedded-brains.de</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hello,<br>
<br>thanks for your hints.<br>
<br>On 20/03/2019 16:31, Vijay Kumar Banerjee wrote:<br>><br>><br>><br>><br>> On Wed, 20 Mar 2019 at 17:46, Sebastian Huber <br>> <<a href="mailto:sebastian.huber@embedded-brains.de" target="_blank">sebastian.huber@embedded-brains.de</a> <br>> <mailto:<a href="mailto:sebastian.huber@embedded-brains.de" target="_blank">sebastian.huber@embedded-brains.de</a>>> wrote:<br>><br>> Hello,<br>><br>> the RTEMS project has a covoar tool:<br>><br>> <a href="https://git.rtems.org/rtems-tools/tree/tester/covoar" rel="noreferrer" target="_blank">https://git.rtems.org/rtems-tools/tree/tester/covoar</a><br>><br>> I have seen some reports generated by it, but I never used it<br>> myself. I<br>> have trouble to find out how it works. There is nothing in the<br>> documentation:<br>><br>> cd rtems-docs<br>> grep -ri covoar .<br>><br>> There is some documentation available here:<br>><br>> <a href="https://devel.rtems.org/wiki/GCI/Documentation/CoverageAnalysis/Coverage" rel="noreferrer" target="_blank">https://devel.rtems.org/wiki/GCI/Documentation/CoverageAnalysis/Coverage</a><br>><br>> If I invoke the rtems-test like this in the erc32 BSP build<br>> directory, I<br>> get an error:<br>><br>> rtems-test --rtems-bsp=erc32-sis --rtems-tools=/build/rtems/5<br>> --coverage<br>> --target=sparc-rtems5 .<br>> RTEMS Testing - Tester, 5.0.not_released<br>> Command Line: /build/rtems/5/bin/rtems-test --rtems-bsp=erc32-sis<br>> --rtems-tools=/build/rtems/5 --coverage --target=sparc-rtems5 .<br>> Python: 3.6.5 (default, Mar 31 2018, 19:45:04) [GCC]<br>> Host: Linux-4.12.14-lp150.12.48-default-x86_64-with-glibc2.3.4 (Linux<br>> huber-nb-linux.suse 4.12.14-lp150.12.48-default #1 SMP Tue Feb 12<br>> 14:01:48 UTC 2019 (268f014) x86_64 x86_64)<br>> Traceback (most recent call last):<br>> File "/build/rtems/5/share/rtems/tester/rt/cmd-test.py", line<br>> 42, in<br>> <module><br>> test.run(sys.argv[1:], command_path = base)<br>> File "/build/rtems/5/share/rtems/tester/rt/test.py", line 300,<br>> in run<br>> trace = cov_trace)<br>> File "/build/rtems/5/share/rtems/tester/rt/coverage.py", line<br>> 363, in<br>> __init__<br>> self.target = self.macros['target']<br>> File "/build/rtems/5/share/rtems/rtemstoolkit/macros.py", line<br>> 181,<br>> in __getitem__<br>> raise IndexError('key: %s' % (key))<br>> IndexError: key: target<br>><br>> Hi,<br>><br>> The covoar needs the .cov files to run which is supposed to be <br>> generated by a command like<br>> this<br>> ====<br>> $HOME/development/rtems/5/bin/sparc-rtems5-sis -nouartrx -r -tlim 600 <br>> s -cov <br>> $HOME/development/rtems/kernel/erc32/sparc-rtems5/c/erc32/testsuites/samples/hello.exe<br>> ====<br>> But this runs into a segmentation fault. If my command is right <br>> (Please correct me if it isn't)<br>> then I think it's because the sis coverage is not supported for erc32. :/<br>
<br>Yes, I get also a segmentation fault. For the leon3 it seems to work:<br>
<br>sparc-rtems5-sis -leon3 -nouartrx -r -tlim 200 s -cov <br>./sparc-rtems5/c/leon3/testsuites/samples/hello.exe<br>
<br> SIS - SPARC/RISCV instruction simulator 2.13, copyright Jiri Gaisler 2019<br> Bug-reports to <a href="mailto:jiri@gaisler.se" target="_blank">jiri@gaisler.se</a><br>
<br> LEON3 emulation enabled, 1 cpus online, delta 50 clocks<br>
<br>
<br>
<br>*** BEGIN OF TEST HELLO WORLD ***<br>*** TEST VERSION: 5.0.0.45e418f922af81c9cb8d485c789ba73c95bda813<br>*** TEST STATE: EXPECTED-PASS<br>*** TEST BUILD: RTEMS_DEBUG RTEMS_NETWORKING RTEMS_POSIX_API RTEMS_SMP<br>*** TEST TOOLS: 7.4.0 20181206 (RTEMS 5, RSB <br>c41b9d0df7e5b4a5056ca50c2534380a44e92769, Newlib 3e24fbf6f)<br>Hello World<br>
<br>*** END OF TEST HELLO WORLD ***<br>
<br>cpu 0 in error mode (tt = 0x80)<br> 165222 400103e0 91d02000 ta 0<br>
<br>saved code coverage to <br>./sparc-rtems5/c/leon3/testsuites/samples/hello.exe.cov<br>
<br>><br>> about rtems-test,<br>> I have attached a patch that adds erc32-sis-cov.ini, this will fix the <br>> KeyError you're seeing,<br>> but it will return with error 10 as it's trying to run the above <br>> command. If you run rtems-test<br>> with `--no-clean` and `--coverage` options, you'll have a <br>> erc32-sis-symbols.ini file (and the .cov<br>> files, when they're supported, like in leon3), which can be used to <br>> run covoar directly like<br>> below ( but this won't run now as we don't have .cov file )<br>
<br>I tried to use leon3-sis-cov and I get:<br>
<br>rtems-test --rtems-bsp=leon3-sis-cov --rtems-tools=/build/rtems/5 <br>--coverage .<br>RTEMS Testing - Tester, 5.0.not_released<br> Command Line: /build/rtems/5/bin/rtems-test --rtems-bsp=leon3-sis-cov <br>--rtems-tools=/build/rtems/5 --coverage .<br> Python: 3.6.5 (default, Mar 31 2018, 19:45:04) [GCC]<br>Host: Linux-4.12.14-lp150.12.48-default-x86_64-with-glibc2.3.4 (Linux <br>huber-nb-linux.suse 4.12.14-lp150.12.48-default #1 SMP Tue Feb 12 <br>14:01:48 UTC 2019 (268f014) x86_64 x86_64)<br>Traceback (most recent call last):<br> File "/build/rtems/5/share/rtems/tester/rt/cmd-test.py", line 42, in <br><module><br> test.run(sys.argv[1:], command_path = base)<br> File "/build/rtems/5/share/rtems/tester/rt/test.py", line 300, in run<br> trace = cov_trace)<br> File "/build/rtems/5/share/rtems/tester/rt/coverage.py", line 363, in <br>__init__<br> self.target = self.macros['target']<br> File "/build/rtems/5/share/rtems/rtemstoolkit/macros.py", line 181, <br>in __getitem__<br> raise IndexError('key: %s' % (key))<br>IndexError: key: target<br>
<br></blockquote><div>this works fine for me ...</div><div>$HOME/development/rtems/rtems-tools/tester/rtems-test --rtems-tools=$HOME/development/rtems/5 --rtems-bsp=leon3-qemu-cov --coverage --log=leon3-test.log /home/lunatic/development/rtems/kernel/leon3/sparc-rtems5/c/leon3/testsuites/samples/hello.exe </div><div><br></div></div></div></div></div></blockquote><div>Sorry about the wrong bsp name, this works fine as well :)</div><div>$HOME/development/rtems/rtems-tools/tester/rtems-test --rtems-tools=$HOME/development/rtems/5 --rtems-bsp=leon3-sis-cov --coverage --no-clean --log=leon3-test.log /home/lunatic/development/rtems/kernel/leon3/sparc-rtems5/c/leon3/testsuites/samples/hello.exe </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div class="gmail_quote"><div></div><div>The leon3-sis-cov was added recently, so you might need to rebase to current master. If </div><div>it still gives error with python 3. Then you can try `export RTEMS_PYTHON_OVERRIDE='python2'` </div><div>to force it to use python2.</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">><br>> covoar -S <br>> $HOME/development/rtems/test/rtems-tools/tester/rtems/testing/coverage/erc32-sis-symbols.ini <br>> \<br>> -O coverage <br>> -E$HOME/development/rtems/test/rtems-tools/tester/rtems/testing/coverage/Explanations.txt <br>> \<br>> -p RTEMS-5 sparc-rtems5/c/erc32/testsuites/samples/hello.exe<br>
<br>Do I have to call rtems-test first and then covoar or is it called by <br>rtems-test itself?<br>
<br></blockquote><div>The rtems-test automatically calls covoar, there's no need to call covoar separately unless</div><div>for debugging purpose. </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">-- <br>Sebastian Huber, embedded brains GmbH<br>
<br>Address : Dornierstr. 4, D-82178 Puchheim, Germany<br>Phone : +49 89 189 47 41-16<br>Fax : +49 89 189 47 41-09<br>E-Mail : <a href="mailto:sebastian.huber@embedded-brains.de" target="_blank">sebastian.huber@embedded-brains.de</a><br>PGP : Public key available on request.<br>
<br>Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG.<br>
<br>
</blockquote></div></div></div></div>
</blockquote></div></div></div>
</blockquote></div>