How can I use the covoar tool?
Joel Sherrill
joel at rtems.org
Wed Mar 20 23:59:01 UTC 2019
Jiri recently added tsim's coverage format to sis. Perhaps it isn't right
on all variations.
When you get reports, it is straight forward to run it on a single test and
see what
methods and lines it tests.
Internally it could but does not keep currently track of which tests
executed each line
of source code. That might be helpful in getting a global view of what
tests exercise
what. But going a test at a time and focusing on what you think it does is
probably
effective.
FWIW the coverage was near 100% when compiled at -Os before the rework to
report per subdirectory and then to use DWARF info. I can't imaging the
testsuite
has gotten worse since then.
Coverage analysis at -Os is much easier because -O2 results in loop
unrolling,
instruction interleaving, etc.
--joel
On Wed, Mar 20, 2019 at 11:11 AM Vijay Kumar Banerjee <
vijaykumar9597 at gmail.com> wrote:
>
>
> On Wed, 20 Mar 2019 at 21:20, Vijay Kumar Banerjee <
> vijaykumar9597 at gmail.com> wrote:
>
>>
>>
>>
>>
>> On Wed, 20 Mar 2019 at 21:11, Sebastian Huber <
>> sebastian.huber at embedded-brains.de> wrote:
>>
>>> Hello,
>>>
>>> thanks for your hints.
>>>
>>> On 20/03/2019 16:31, Vijay Kumar Banerjee wrote:
>>> >
>>> >
>>> >
>>> >
>>> > On Wed, 20 Mar 2019 at 17:46, Sebastian Huber
>>> > <sebastian.huber at embedded-brains.de
>>> > <mailto:sebastian.huber at embedded-brains.de>> wrote:
>>> >
>>> > Hello,
>>> >
>>> > the RTEMS project has a covoar tool:
>>> >
>>> > https://git.rtems.org/rtems-tools/tree/tester/covoar
>>> >
>>> > I have seen some reports generated by it, but I never used it
>>> > myself. I
>>> > have trouble to find out how it works. There is nothing in the
>>> > documentation:
>>> >
>>> > cd rtems-docs
>>> > grep -ri covoar .
>>> >
>>> > There is some documentation available here:
>>> >
>>> >
>>> https://devel.rtems.org/wiki/GCI/Documentation/CoverageAnalysis/Coverage
>>> >
>>> > If I invoke the rtems-test like this in the erc32 BSP build
>>> > directory, I
>>> > get an error:
>>> >
>>> > rtems-test --rtems-bsp=erc32-sis --rtems-tools=/build/rtems/5
>>> > --coverage
>>> > --target=sparc-rtems5 .
>>> > RTEMS Testing - Tester, 5.0.not_released
>>> > Command Line: /build/rtems/5/bin/rtems-test --rtems-bsp=erc32-sis
>>> > --rtems-tools=/build/rtems/5 --coverage --target=sparc-rtems5 .
>>> > Python: 3.6.5 (default, Mar 31 2018, 19:45:04) [GCC]
>>> > Host: Linux-4.12.14-lp150.12.48-default-x86_64-with-glibc2.3.4
>>> (Linux
>>> > huber-nb-linux.suse 4.12.14-lp150.12.48-default #1 SMP Tue Feb 12
>>> > 14:01:48 UTC 2019 (268f014) x86_64 x86_64)
>>> > Traceback (most recent call last):
>>> > File "/build/rtems/5/share/rtems/tester/rt/cmd-test.py", line
>>> > 42, in
>>> > <module>
>>> > test.run(sys.argv[1:], command_path = base)
>>> > File "/build/rtems/5/share/rtems/tester/rt/test.py", line 300,
>>> > in run
>>> > trace = cov_trace)
>>> > File "/build/rtems/5/share/rtems/tester/rt/coverage.py", line
>>> > 363, in
>>> > __init__
>>> > self.target = self.macros['target']
>>> > File "/build/rtems/5/share/rtems/rtemstoolkit/macros.py", line
>>> > 181,
>>> > in __getitem__
>>> > raise IndexError('key: %s' % (key))
>>> > IndexError: key: target
>>> >
>>> > Hi,
>>> >
>>> > The covoar needs the .cov files to run which is supposed to be
>>> > generated by a command like
>>> > this
>>> > ====
>>> > $HOME/development/rtems/5/bin/sparc-rtems5-sis -nouartrx -r -tlim 600
>>> > s -cov
>>> >
>>> $HOME/development/rtems/kernel/erc32/sparc-rtems5/c/erc32/testsuites/samples/hello.exe
>>> > ====
>>> > But this runs into a segmentation fault. If my command is right
>>> > (Please correct me if it isn't)
>>> > then I think it's because the sis coverage is not supported for erc32.
>>> :/
>>>
>>> Yes, I get also a segmentation fault. For the leon3 it seems to work:
>>>
>>> sparc-rtems5-sis -leon3 -nouartrx -r -tlim 200 s -cov
>>> ./sparc-rtems5/c/leon3/testsuites/samples/hello.exe
>>>
>>> SIS - SPARC/RISCV instruction simulator 2.13, copyright Jiri Gaisler
>>> 2019
>>> Bug-reports to jiri at gaisler.se
>>>
>>> LEON3 emulation enabled, 1 cpus online, delta 50 clocks
>>>
>>>
>>>
>>> *** BEGIN OF TEST HELLO WORLD ***
>>> *** TEST VERSION: 5.0.0.45e418f922af81c9cb8d485c789ba73c95bda813
>>> *** TEST STATE: EXPECTED-PASS
>>> *** TEST BUILD: RTEMS_DEBUG RTEMS_NETWORKING RTEMS_POSIX_API RTEMS_SMP
>>> *** TEST TOOLS: 7.4.0 20181206 (RTEMS 5, RSB
>>> c41b9d0df7e5b4a5056ca50c2534380a44e92769, Newlib 3e24fbf6f)
>>> Hello World
>>>
>>> *** END OF TEST HELLO WORLD ***
>>>
>>> cpu 0 in error mode (tt = 0x80)
>>> 165222 400103e0 91d02000 ta 0
>>>
>>> saved code coverage to
>>> ./sparc-rtems5/c/leon3/testsuites/samples/hello.exe.cov
>>>
>>> >
>>> > about rtems-test,
>>> > I have attached a patch that adds erc32-sis-cov.ini, this will fix the
>>> > KeyError you're seeing,
>>> > but it will return with error 10 as it's trying to run the above
>>> > command. If you run rtems-test
>>> > with `--no-clean` and `--coverage` options, you'll have a
>>> > erc32-sis-symbols.ini file (and the .cov
>>> > files, when they're supported, like in leon3), which can be used to
>>> > run covoar directly like
>>> > below ( but this won't run now as we don't have .cov file )
>>>
>>> I tried to use leon3-sis-cov and I get:
>>>
>>> rtems-test --rtems-bsp=leon3-sis-cov --rtems-tools=/build/rtems/5
>>> --coverage .
>>> RTEMS Testing - Tester, 5.0.not_released
>>> Command Line: /build/rtems/5/bin/rtems-test --rtems-bsp=leon3-sis-cov
>>> --rtems-tools=/build/rtems/5 --coverage .
>>> Python: 3.6.5 (default, Mar 31 2018, 19:45:04) [GCC]
>>> Host: Linux-4.12.14-lp150.12.48-default-x86_64-with-glibc2.3.4 (Linux
>>> huber-nb-linux.suse 4.12.14-lp150.12.48-default #1 SMP Tue Feb 12
>>> 14:01:48 UTC 2019 (268f014) x86_64 x86_64)
>>> Traceback (most recent call last):
>>> File "/build/rtems/5/share/rtems/tester/rt/cmd-test.py", line 42, in
>>> <module>
>>> test.run(sys.argv[1:], command_path = base)
>>> File "/build/rtems/5/share/rtems/tester/rt/test.py", line 300, in run
>>> trace = cov_trace)
>>> File "/build/rtems/5/share/rtems/tester/rt/coverage.py", line 363, in
>>> __init__
>>> self.target = self.macros['target']
>>> File "/build/rtems/5/share/rtems/rtemstoolkit/macros.py", line 181,
>>> in __getitem__
>>> raise IndexError('key: %s' % (key))
>>> IndexError: key: target
>>>
>>> this works fine for me ...
>> $HOME/development/rtems/rtems-tools/tester/rtems-test
>> --rtems-tools=$HOME/development/rtems/5 --rtems-bsp=leon3-qemu-cov
>> --coverage --log=leon3-test.log
>> /home/lunatic/development/rtems/kernel/leon3/sparc-rtems5/c/leon3/testsuites/samples/hello.exe
>>
>> Sorry about the wrong bsp name, this works fine as well :)
> $HOME/development/rtems/rtems-tools/tester/rtems-test
> --rtems-tools=$HOME/development/rtems/5 --rtems-bsp=leon3-sis-cov
> --coverage --no-clean --log=leon3-test.log
> /home/lunatic/development/rtems/kernel/leon3/sparc-rtems5/c/leon3/testsuites/samples/hello.exe
>
>> The leon3-sis-cov was added recently, so you might need to rebase to
>> current master. If
>> it still gives error with python 3. Then you can try `export
>> RTEMS_PYTHON_OVERRIDE='python2'`
>> to force it to use python2.
>>
>>> >
>>> > covoar -S
>>> >
>>> $HOME/development/rtems/test/rtems-tools/tester/rtems/testing/coverage/erc32-sis-symbols.ini
>>>
>>> > \
>>> > -O coverage
>>> >
>>> -E$HOME/development/rtems/test/rtems-tools/tester/rtems/testing/coverage/Explanations.txt
>>>
>>> > \
>>> > -p RTEMS-5 sparc-rtems5/c/erc32/testsuites/samples/hello.exe
>>>
>>> Do I have to call rtems-test first and then covoar or is it called by
>>> rtems-test itself?
>>>
>>> The rtems-test automatically calls covoar, there's no need to call
>> covoar separately unless
>> for debugging purpose.
>>
>>> --
>>> Sebastian Huber, embedded brains GmbH
>>>
>>> Address : Dornierstr. 4, D-82178 Puchheim, Germany
>>> Phone : +49 89 189 47 41-16
>>> Fax : +49 89 189 47 41-09
>>> E-Mail : sebastian.huber at embedded-brains.de
>>> PGP : Public key available on request.
>>>
>>> Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG.
>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rtems.org/pipermail/devel/attachments/20190320/ae637d6f/attachment-0002.html>
More information about the devel
mailing list