<offllist> Coverage support for tester
Joel Sherrill
joel at rtems.org
Wed May 30 17:35:25 UTC 2018
On Wed, May 30, 2018 at 12:25 PM, Cillian O'Donnell <cpodonnell8 at gmail.com>
wrote:
>
>
> On Wed, 30 May 2018, 18:15 Vijay Kumar Banerjee, <vijaykumar9597 at gmail.com>
> wrote:
>
>>
>>
>> On Wed, 30 May 2018, 22:42 Cillian O'Donnell, <cpodonnell8 at gmail.com>
>> wrote:
>>
>>>
>>>
>>> On Wed, 30 May 2018, 16:58 Gedare Bloom, <gedare at rtems.org> wrote:
>>>
>>>> On Wed, May 30, 2018 at 11:32 AM, Vijay Kumar Banerjee
>>>> <vijaykumar9597 at gmail.com> wrote:
>>>> > On 30 May 2018 at 20:18, Gedare Bloom <gedare at rtems.org> wrote:
>>>> >>
>>>> >> Hello Vijay,
>>>> >>
>>>> >> Do you expect/need an answer to something in here?
>>>> >>
>>>> >> gedare
>>>> >>
>>>> > Hello,
>>>> >
>>>> > I wanted to know if there were any plans on how covoar
>>>> > should store the reports when running for multisets.
>>>> > Earlier it used to be done by the coverage script,
>>>> > after the recent changes covoar can process multi sets.
>>>> >
>>>> > I think, covoar should store the reports into separate directories
>>>> > for each set . eg. score/ , rtems/ . As the coverage can already read
>>>> > from separate directories.
>>>>
>>>
>>> Sorry I thought all questions had been answered here. I think you have
>>> the right idea. Each set should be a sub-directory of coverage directory.
>>>
>>
Yes. With some top level "index.html" or similar to guide you through the
subdirectories.
The entire set of subdirectories is a single "run". One use is to provide
coverage reports
for a few BSPs as part of making a release.
>
>>> By the way I tested your changes and everything seems fine. Still have
>>> to do a review of coverage.py to see how close we are to merging.
>>>
>> I have squashed everything and sent a patch to devel at . This will make it
>> easy to go through all the changes. Please have a look.
>>
>
> Ah just seen it, will take a look. It'll be good to get Chris' thoughts
> too. We'll have something more definitive about how close it is to merging.
>
I agree with Gedare on it may be over-squashed but that may be the only way
to get it all
merged at this point. You are the third student to work on this. It is
likely changes on top
of changes on top of K's original work.
If it can't be broken apart by author, then be careful to make sure the git
log
messages indicates the authors. Unfortunately, git doesn't appear to
handle this
well/at all and there is suggested tag here.
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=451880
> >
>>>> > Any advice on how should it be approached ?
>>>>
>>>> It would help me if you could draw/write a diagram of what the
>>>> filesystem tree might look like with separate directories, and what
>>>> will go in each subdirectory.
>>>>
>>>> I don't have enough context to give any useful advice on this question.
>>>>
>>>
Please start a new thread on devel@ with this. I don't think it needs much
context for Gedare but the discussion should be part of the public record.
>
>>>> -Gedare
>>>>
>>>> >>
>>>> >> On Wed, May 30, 2018 at 10:29 AM, Vijay Kumar Banerjee
>>>> >> <vijaykumar9597 at gmail.com> wrote:
>>>> >> > On 30 May 2018 at 00:54, Joel Sherrill <joel at rtems.org> wrote:
>>>> >> >>
>>>> >> >>
>>>> >> >>
>>>> >> >> On Tue, May 29, 2018 at 11:27 AM, Vijay Kumar Banerjee
>>>> >> >> <vijaykumar9597 at gmail.com> wrote:
>>>> >> >>>
>>>> >> >>>
>>>> >> >>>
>>>> >> >>> On Tue, 29 May 2018, 20:10 Joel Sherrill, <joel at rtems.org>
>>>> wrote:
>>>> >> >>>>
>>>> >> >>>>
>>>> >> >>>>
>>>> >> >>>> On Mon, May 28, 2018 at 11:08 PM, Chris Johns <chrisj at rtems.org
>>>> >
>>>> >> >>>> wrote:
>>>> >> >>>>>
>>>> >> >>>>> On 29/5/18 4:26 am, Joel Sherrill wrote:
>>>> >> >>>>> > On Mon, May 28, 2018 at 5:43 AM, Vijay Kumar Banerjee
>>>> >> >>>>> > <vijaykumar9597 at gmail.com
>>>> >> >>>>> > <mailto:vijaykumar9597 at gmail.com>> wrote:
>>>> >> >>>>> >
>>>> >> >>>>> > Hello,
>>>> >> >>>>> >
>>>> >> >>>>> > The coverage reports are now showing results.
>>>> >> >>>>> > The current branch that holds all the changes is
>>>> >> >>>>> > the cov-tester-support branch in my forked repo
>>>> >> >>>>> >
>>>> >> >>>>> >
>>>> >> >>>>> > https://github.com/thelunatic/rtems-tools/tree/cov-tester-
>>>> support
>>>> >> >>>>> >
>>>> >> >>>>> >
>>>> >> >>>>> > <https://github.com/thelunatic/rtems-tools/tree/
>>>> cov-tester-support>
>>>> >> >>>>> >
>>>> >> >>>>> > (Please have a look into the code and test it.)
>>>> >> >>>>> >
>>>> >> >>>>> > It is close to merging (hopefully). These are the issues
>>>> >> >>>>> > that would need a fix before it can be merged :
>>>> >> >>>>> >
>>>> >> >>>>> > 1. I have added some #FIXME in the code (have a look)
>>>> >> >>>>> > in coverage script. I have set the value of the
>>>> targe to
>>>> >> >>>>> > be
>>>> >> >>>>> > sparc-rtems5, which makes it limited to sparc-rtems5
>>>> only.
>>>> >> >>>>> > We
>>>> >> >>>>> > can
>>>> >> >>>>> > include the target in the bsp ini file, That would
>>>> >> >>>>> > be a quick fix for this.
>>>> >> >>>>> >
>>>> >> >>>>> >
>>>> >> >>>>> > Yes. This needs to be fixed.
>>>> >> >>>>> >
>>>> >> >>>>> > My hack to add 4 in ObjdumpProcessor.cc needs to be addressed
>>>> >> >>>>> > also.
>>>> >> >>>>> > I am thinking of adding a method to Target_*.cc to ask for
>>>> the
>>>> >> >>>>> > size
>>>> >> >>>>> > of an
>>>> >> >>>>> > instruction.
>>>> >> >>>>> > Then pass it the last instruction. That way we can throw on
>>>> other
>>>> >> >>>>> > architectures for
>>>> >> >>>>> > now. If Chris solves this with his changes before we try
>>>> another
>>>> >> >>>>> > architecture,
>>>> >> >>>>> > great.
>>>> >> >>>>> > If not, it will be easy to fix.
>>>> >> >>>>>
>>>> >> >>>>> What is the overall requirement?
>>>> >> >>>>
>>>> >> >>>>
>>>> >> >>>> To know the ending address of the function.
>>>> >> >>>>
>>>> >> >>>> Technically there are three pieces of information:
>>>> >> >>>>
>>>> >> >>>> + start address
>>>> >> >>>> + end address
>>>> >> >>>> + size
>>>> >> >>>>
>>>> >> >>>> If you know two of those, you can compute the third.
>>>> >> >>>>
>>>> >> >>>> I don't care if this comes from DWARF, ELF, or parsing the
>>>> >> >>>> disassembly.
>>>> >> >>>> It just needs to be reliable.
>>>> >> >>>>
>>>> >> >>>> And.. I am not proposing my solution as permanent. Just to keep
>>>> us
>>>> >> >>>> moving. You want to change to an internal disassembler (which
>>>> >> >>>> would also need to have the source interspersed) and DWARF. So
>>>> >> >>>> this code would go away then.
>>>> >> >>>>
>>>> >> >>>>>
>>>> >> >>>>>
>>>> >> >>>>> What defines the function and so size are attempting to get
>>>> coverage
>>>> >> >>>>> of? What if
>>>> >> >>>>> that function calls an inline function and that function is
>>>> inlined?
>>>> >> >>>>> What if
>>>> >> >>>>> that inlined function calls another inlined function?
>>>> >> >>>>
>>>> >> >>>>
>>>> >> >>>> Then it is all inlined. It is done consistently now. I have
>>>> never
>>>> >> >>>> seen a
>>>> >> >>>> case
>>>> >> >>>> where two instances of a method differed as the assembly level.
>>>> [1]
>>>> >> >>>>
>>>> >> >>>> The actual body of the inlined method is evaluated at each
>>>> expansion
>>>> >> >>>> point.
>>>> >> >>>> That is why a few years ago, I pushed hard to reduce the
>>>> complexity
>>>> >> >>>> of
>>>> >> >>>> inline methods because we got test patch explosion when an
>>>> inlined
>>>> >> >>>> method
>>>> >> >>>> included complex conditionals.
>>>> >> >>>>
>>>> >> >>>> Similarly, I think it would be helpful to coverage and
>>>> verification
>>>> >> >>>> efforts to
>>>> >> >>>> follow the **shudder** MISRA rule which want you to write simple
>>>> >> >>>> conditional
>>>> >> >>>> expressions rather than compound ones. I have taken to writing
>>>> code
>>>> >> >>>> this
>>>> >> >>>> way as much as possible. But haven't pushed it as a coding rule.
>>>> >> >>>>
>>>> >> >>>> if (a) {
>>>> >> >>>> if (b) {
>>>> >> >>>> do x;
>>>> >> >>>> }
>>>> >> >>>> }
>>>> >> >>>>
>>>> >> >>>> Versus
>>>> >> >>>> if (a && b) {
>>>> >> >>>> do x;
>>>> >> >>>> }
>>>> >> >>>>
>>>> >> >>>> The reason is that the first is easier to analyse coverage on.
>>>> >> >>>>
>>>> >> >>>> [1] We both expect LTO could change this.
>>>> >> >>>>
>>>> >> >>>> [2] ESA did specifically mention this one though. Also in
>>>> general
>>>> >> >>>> terms,
>>>> >> >>>> an RTEMS Project response to MISRA rules. Which ones we follow,
>>>> >> >>>> reject, etc.. But I refuse to adopt hard rules which can't be
>>>> >> >>>> enforced
>>>> >> >>>> by free open source tools.
>>>> >> >>>>
>>>> >> >>>>>
>>>> >> >>>>>
>>>> >> >>>>> The DWARF data provides details about the PC low and PC high
>>>> of what
>>>> >> >>>>> is
>>>> >> >>>>> called
>>>> >> >>>>> concrete functions and then it provides the details about
>>>> inlines.
>>>> >> >>>>
>>>> >> >>>>
>>>> >> >>>> We don't (currently) report on the inlines as independent
>>>> methods.
>>>> >> >>>>
>>>> >> >>>>>
>>>> >> >>>>>
>>>> >> >>>>> >
>>>> >> >>>>> > 2. coverage used to feed ini file for each symbol to
>>>> covoar
>>>> >> >>>>> > in a loop and store the result in a separate
>>>> directory
>>>> >> >>>>> > for each symbol . Which is needed no more needed as
>>>> >> >>>>> > covoar can now process multi sets from a
>>>> >> >>>>> > single ini file. I think the results from covoar
>>>> should
>>>> >> >>>>> > be
>>>> >> >>>>> > store in a separate directory for each symbol
>>>> >> >>>>> > example :- score/
>>>> >> >>>>> >
>>>> >> >>>>> >
>>>> >> >>>>> > A bit of history will help here. Originally covoar was run
>>>> against
>>>> >> >>>>> > a
>>>> >> >>>>> > single set of
>>>> >> >>>>> > code by the scripting framework. We would do coverage on
>>>> either
>>>> >> >>>>> > "core
>>>> >> >>>>> > parts"
>>>> >> >>>>> > or "developmental" (e.g. nearly all non-networked code). The
>>>> >> >>>>> > optimization was
>>>> >> >>>>> > either at -O2 or -Os. So there were four coverage variants.
>>>> >> >>>>> >
>>>> >> >>>>> > Turned out that when we added something to "all", the
>>>> percentage
>>>> >> >>>>> > would drop
>>>> >> >>>>> > and reflect badly on the rest of the code. I remember adding
>>>> the
>>>> >> >>>>> > dosfs and
>>>> >> >>>>> > the coverage dropped almost 20 percent.
>>>> >> >>>>> >
>>>> >> >>>>> > This led to the idea that we should report on a per
>>>> >> >>>>> > directory/subsystem basis.
>>>> >> >>>>> > The score, rtems, posix, sapi, and libcsupport should have
>>>> high
>>>> >> >>>>> > coverage now
>>>> >> >>>>> > and the reports should reflect that independent of whether
>>>> the
>>>> >> >>>>> > dosfs
>>>> >> >>>>> > needs a
>>>> >> >>>>> > lot more tests.
>>>> >> >>>>> >
>>>> >> >>>>> > Before each directory/subsystem required a completely
>>>> separate run
>>>> >> >>>>> > of
>>>> >> >>>>> > covoar.
>>>> >> >>>>> > If we are headed to a solution where one analysis run of
>>>> covoar
>>>> >> >>>>> > generates different
>>>> >> >>>>> > reports, that should speed up the processing time a lot!
>>>> >> >>>>> >
>>>> >> >>>>> > The other issue is what should the top directory look
>>>> like/contain
>>>> >> >>>>> > when a single
>>>> >> >>>>> > run produces multiple subsystem reports. Seems like we would
>>>> need
>>>> >> >>>>> > at
>>>> >> >>>>> > least a
>>>> >> >>>>> > top level html and text file.
>>>> >> >>>>> >
>>>> >> >>>>> >
>>>> >> >>>>> >
>>>> >> >>>>> > 3. currently we are using the leon3-qemu-cov as the bsp.
>>>> >> >>>>> > Are we going to have two ini file for each bsp ? (
>>>> one
>>>> >> >>>>> > without coverage
>>>> >> >>>>> > and one with coverage support)
>>>> >> >>>>> >
>>>> >> >>>>> > Earlier the approach was to include a section 'coverage'
>>>> >> >>>>> > to the bsp ini to put the values we needed for coverage.
>>>> >> >>>>> > I think that approach would be more "convenient" for the
>>>> user.
>>>> >> >>>>> >
>>>> >> >>>>> >
>>>> >> >>>>> > This was something Chris suggested. I think it was to avoid
>>>> adding
>>>> >> >>>>> > to
>>>> >> >>>>> > the bsp ini file until the code was closer to merging.
>>>> >> >>>>> >
>>>> >> >>>>> > Chris.. what do you think? Long term, a section would be
>>>> nice.
>>>> >> >>>>> >
>>>> >> >>>>>
>>>> >> >>>>> Sorry I cannot remember without looking it up and I am
>>>> currently
>>>> >> >>>>> buried
>>>> >> >>>>> in
>>>> >> >>>>> family issues.
>>>> >> >>>>
>>>> >> >>>>
>>>> >> >>>> OK. Having the Python scripting loop over the sets or covoar
>>>> looping
>>>> >> >>>> over them
>>>> >> >>>> is an open issue. Historically, looping over desired symbol
>>>> sets was
>>>> >> >>>> outside
>>>> >> >>>> covoar. So looping inside covoar may take some work.
>>>> >> >>>
>>>> >> >>>
>>>> >> >>> covoar can already loop over the
>>>> >> >>> sets it seems, which is implemented
>>>> >> >>> in DesiredSymbols. But it stores all the
>>>> >> >>> reports generated from into the same directory.
>>>> >> >>
>>>> >> >>
>>>> >> >> If there is an index that makes its possible to navigate through
>>>> the
>>>> >> >> different "subsystems", then that's the key thing. You don't want
>>>> >> >> to think score is poorly covered due to dosfs.
>>>> >> >>
>>>> >> >>>
>>>> >> >>>
>>>> >> >>> P.S:Sorry for the previous mail with no message
>>>> >> >>>>
>>>> >> >>>>
>>>> >> >>>> --joel
>>>> >> >>>>>
>>>> >> >>>>>
>>>> >> >>>>> Chris
>>>> >> >>>>
>>>> >> >>>>
>>>> >> >>
>>>> >> >
>>>> >> >
>>>> >> > _______________________________________________
>>>> >> > devel mailing list
>>>> >> > devel at rtems.org
>>>> >> > http://lists.rtems.org/mailman/listinfo/devel
>>>> >
>>>> >
>>>> _______________________________________________
>>>> devel mailing list
>>>> devel at rtems.org
>>>> http://lists.rtems.org/mailman/listinfo/devel
>>>>
>>>
> _______________________________________________
> devel mailing list
> devel at rtems.org
> http://lists.rtems.org/mailman/listinfo/devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rtems.org/pipermail/devel/attachments/20180530/0aa96480/attachment-0002.html>
More information about the devel
mailing list