[PATCH] rtems-tools/covoar: Wrong coverage computed

Fri Jan 25 20:16:33 UTC 2019

On Fri, Jan 25, 2019 at 12:50 PM Vijay Kumar Banerjee <
vijaykumar9597 at gmail.com> wrote:

>
>
> On Fri, 25 Jan 2019 at 23:39, Joel Sherrill <joel at rtems.org> wrote:
>
>> Sorry for the duplicate email. I didn't use the right email address and
>> the response did not go to devel@
>>
>> On Fri, Jan 25, 2019 at 8:43 AM Jiri Gaisler <jiri at gaisler.se> wrote:
>>
>>> Sorry for the noise, but I have a better patch that actually solves the
>>> root problem. I ran the full testsuite on sis and covoar against libscore.a
>>> like this:
>>>
>>> $ sparc-rtems5-ar -rcs sparc-rtems5/c/leon3/cpukit/score/libscore.a
>>> sparc-rtems5/c/leon3/cpukit/score/src/*.o
>>>
>>> $ rtems-test --rtems-bsp=leon3-sis-cov sparc-rtems5/c/leon3/testsuites
>>>
>>> $ /opt/rtems/5/share/rtems/tester/bin/covoar -v -f TSIM -S
>>> /opt/rtems/5/share/rtems/tester/rtems/testing/coverage/score-symbols.ini -O
>>> coverage -E
>>> /opt/rtems/5/share/rtems/tester/rtems/testing/coverage/Explanations.txt -p
>>> RTEMS-5  sparc-rtems5/c/leon3/testsuites/*/*.exe
>>>
>>> summary.txt:
>>>
>>> Bytes Analyzed           : 39376 Bytes Not Executed       : 2864
>>> Percentage Executed      : 92.73 Percentage Not Executed  : 7.273 Uncovered
>>> ranges found   : 108 Total branches found     : 860 Uncovered branches
>>> found : 200    61 branches always taken    139 branches never taken
>>>
>>> All symbol sizes and ranges are now correctly reported/calculated in the
>>> reports. There are two remaining issues to be solved:
>>>
>>> 1. File names are printed as 'unknown' in reports (minor problem but
>>> useful)
>>>
>> Originally we got the info via the binutils command line tools. I think
>> the
>> filename came from addr2line. When we designed it originally, Ian Lance
>> Taylor said parsing the tool output was more easier, reliable, and more
>> stable than tracking any elf reading library.
>>
>> But it was slower than accessing the data directly via ELF and DWARF.
>> Since Chris added this support for the dynamic loader, it was natural to
>> begin to replace that code. Unfortunately, there are some warts left from
>> the transition. The name may be one of them.
>>
>> We also think the beginning and end of a method are not marked properly.
>>
>> And when something is inlined, it isn't considered part of the main
>> method.
>> The original method considered assembly from inlined code part of the
>> caller. It is now reported as part of the callee. We have identified a
>> needed
>> change to (1) account for the inlined assembly as part of the caller and
>> (2) add additional tracking of the coverage of inlined methods.
>>
>> The immediate need is to get the reporting so it accurately annoates
>> the generated assembly from a method and anything inlined into it.
>> This should let us do DO-178 Level A style coverage -- we are
>> sure we use ever instruction and take/not take every branch.
>>
>> Hello,
> This is awesome to have the sis support in covoar, the very next thing
> that
> comes to my mind is adding the '-f' option in the rtems-test coverage
> script,
> this will enable us to get the HTML reports.
>
> The issues noted here, can make a very good project for GSoC and I was
> hoping to work on them, but there is a good amount of complexity involved,
> and I am confused where to get started. :)
>

This should be another thread but here's a short answer

+ Verify Jiri didn't break reporting from qemu testing

Then I recall a few odd problems:

+ method entry and exit not being marked as executed even though
we know it was.
+ inlined code not being marked because it wasn't reported as
"in the method"

If all reports look trustworthy, then look into additionally tracking
coverage of inlined methods but track it back to their source.

>
> If there is unreachable code or branches where both paths are not
>> exercised, we need tests to exercise them OR the code is useless
>> and can be removed. Either way, we win.
>>
>>> 2. We should add RISC-V support to covoar ...
>>>
>>
>> I am sure you have seen the Target* files. They are not large. Some is
>> trivial. Other parts are to recognize nops and branch instruction
>> variants.
>>
>> Is the workflow something like this ...
> 1. write Target_ARCH.cc
> 2. Add it to the TargetFactory
> 3. Add ini file in the tools .
>
> I have a question. Is there some reason that the files in covoar/ don't
> have a
> copyright notice?
>

Sloppiness. :(

The original code was written entirely by OAR. We could probably use git
blame to augment that.

>
>  On a RISC architecture, I wouldn't expect a lot of variation in nops.
>> The x86 has an awful set that range in size. :(
>>
>> Let me know if you agree to the patch ..
>>>
>>
>> The patch looks good to me.
>>
>>> _______________________________________________
>>> devel mailing list
>>> devel at rtems.org
>>> http://lists.rtems.org/mailman/listinfo/devel
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rtems.org/pipermail/devel/attachments/20190125/05019241/attachment-0002.html>