[PATCH] rtems-tools/covoar: Wrong coverage computed

[Vijay Kumar Banerjee]

On Sat, 26 Jan 2019 at 01:46, Joel Sherrill <joel at rtems.org> wrote:

>
>
> On Fri, Jan 25, 2019 at 12:50 PM Vijay Kumar Banerjee <
> vijaykumar9597 at gmail.com> wrote:
>
>>
>>
>> On Fri, 25 Jan 2019 at 23:39, Joel Sherrill <joel at rtems.org> wrote:
>>
>>> Sorry for the duplicate email. I didn't use the right email address and
>>> the response did not go to devel@
>>>
>>> On Fri, Jan 25, 2019 at 8:43 AM Jiri Gaisler <jiri at gaisler.se> wrote:
>>>
>>>> Sorry for the noise, but I have a better patch that actually solves the
>>>> root problem. I ran the full testsuite on sis and covoar against libscore.a
>>>> like this:
>>>>
>>>> $ sparc-rtems5-ar -rcs sparc-rtems5/c/leon3/cpukit/score/libscore.a
>>>> sparc-rtems5/c/leon3/cpukit/score/src/*.o
>>>>
>>>> $ rtems-test --rtems-bsp=leon3-sis-cov sparc-rtems5/c/leon3/testsuites
>>>>
>>>> $ /opt/rtems/5/share/rtems/tester/bin/covoar -v -f TSIM -S
>>>> /opt/rtems/5/share/rtems/tester/rtems/testing/coverage/score-symbols.ini -O
>>>> coverage -E
>>>> /opt/rtems/5/share/rtems/tester/rtems/testing/coverage/Explanations.txt -p
>>>> RTEMS-5  sparc-rtems5/c/leon3/testsuites/*/*.exe
>>>>
>>>> summary.txt:
>>>>
>>>> Bytes Analyzed           : 39376 Bytes Not Executed       : 2864
>>>> Percentage Executed      : 92.73 Percentage Not Executed  : 7.273 Uncovered
>>>> ranges found   : 108 Total branches found     : 860 Uncovered branches
>>>> found : 200    61 branches always taken    139 branches never taken
>>>>
>>>> All symbol sizes and ranges are now correctly reported/calculated in
>>>> the reports. There are two remaining issues to be solved:
>>>>
>>>> 1. File names are printed as 'unknown' in reports (minor problem but
>>>> useful)
>>>>
>>> Originally we got the info via the binutils command line tools. I think
>>> the
>>> filename came from addr2line. When we designed it originally, Ian Lance
>>> Taylor said parsing the tool output was more easier, reliable, and more
>>> stable than tracking any elf reading library.
>>>
>>> But it was slower than accessing the data directly via ELF and DWARF.
>>> Since Chris added this support for the dynamic loader, it was natural to
>>> begin to replace that code. Unfortunately, there are some warts left from
>>> the transition. The name may be one of them.
>>>
>>> We also think the beginning and end of a method are not marked properly.
>>>
>>> And when something is inlined, it isn't considered part of the main
>>> method.
>>> The original method considered assembly from inlined code part of the
>>> caller. It is now reported as part of the callee. We have identified a
>>> needed
>>> change to (1) account for the inlined assembly as part of the caller and
>>> (2) add additional tracking of the coverage of inlined methods.
>>>
>>> The immediate need is to get the reporting so it accurately annoates
>>> the generated assembly from a method and anything inlined into it.
>>> This should let us do DO-178 Level A style coverage -- we are
>>> sure we use ever instruction and take/not take every branch.
>>>
>>> Hello,
>> This is awesome to have the sis support in covoar, the very next thing
>> that
>> comes to my mind is adding the '-f' option in the rtems-test coverage
>> script,
>> this will enable us to get the HTML reports.
>>
>> The issues noted here, can make a very good project for GSoC and I was
>> hoping to work on them, but there is a good amount of complexity
>> involved,
>> and I am confused where to get started. :)
>>
>
> This should be another thread but here's a short answer
>
> + Verify Jiri didn't break reporting from qemu testing
>
>  Reporting from qemu works properly with the patch. Checked with rtems-test

> Then I recall a few odd problems:
>
> + method entry and exit not being marked as executed even though
> we know it was.
> + inlined code not being marked because it wasn't reported as
> "in the method"
>
> If all reports look trustworthy, then look into additionally tracking
> coverage of inlined methods but track it back to their source.
>
>
>>
>> If there is unreachable code or branches where both paths are not
>>> exercised, we need tests to exercise them OR the code is useless
>>> and can be removed. Either way, we win.
>>>
>>>> 2. We should add RISC-V support to covoar ...
>>>>
>>>
>>> I am sure you have seen the Target* files. They are not large. Some is
>>> trivial. Other parts are to recognize nops and branch instruction
>>> variants.
>>>
>>> Is the workflow something like this ...
>> 1. write Target_ARCH.cc
>> 2. Add it to the TargetFactory
>> 3. Add ini file in the tools .
>>
>> I have a question. Is there some reason that the files in covoar/ don't
>> have a
>> copyright notice?
>>
>
> Sloppiness. :(
>
> The original code was written entirely by OAR. We could probably use git
> blame to augment that.
>
>
>>
>>  On a RISC architecture, I wouldn't expect a lot of variation in nops.
>>> The x86 has an awful set that range in size. :(
>>>
>>> Let me know if you agree to the patch ..
>>>>
>>>
>>> The patch looks good to me.
>>>
>>>> _______________________________________________
>>>> devel mailing list
>>>> devel at rtems.org
>>>> http://lists.rtems.org/mailman/listinfo/devel
>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rtems.org/pipermail/devel/attachments/20190126/11008ca1/attachment-0002.html>