<offllist> Coverage support for tester
Vijay Kumar Banerjee
vijaykumar9597 at gmail.com
Wed May 30 15:32:24 UTC 2018
On 30 May 2018 at 20:18, Gedare Bloom <gedare at rtems.org> wrote:
> Hello Vijay,
>
> Do you expect/need an answer to something in here?
>
> gedare
>
> Hello,
I wanted to know if there were any plans on how covoar
should store the reports when running for multisets.
Earlier it used to be done by the coverage script,
after the recent changes covoar can process multi sets.
I think, covoar should store the reports into separate directories
for each set . eg. score/ , rtems/ . As the coverage can already read
from separate directories.
Any advice on how should it be approached ?
> On Wed, May 30, 2018 at 10:29 AM, Vijay Kumar Banerjee
> <vijaykumar9597 at gmail.com> wrote:
> > On 30 May 2018 at 00:54, Joel Sherrill <joel at rtems.org> wrote:
> >>
> >>
> >>
> >> On Tue, May 29, 2018 at 11:27 AM, Vijay Kumar Banerjee
> >> <vijaykumar9597 at gmail.com> wrote:
> >>>
> >>>
> >>>
> >>> On Tue, 29 May 2018, 20:10 Joel Sherrill, <joel at rtems.org> wrote:
> >>>>
> >>>>
> >>>>
> >>>> On Mon, May 28, 2018 at 11:08 PM, Chris Johns <chrisj at rtems.org>
> wrote:
> >>>>>
> >>>>> On 29/5/18 4:26 am, Joel Sherrill wrote:
> >>>>> > On Mon, May 28, 2018 at 5:43 AM, Vijay Kumar Banerjee
> >>>>> > <vijaykumar9597 at gmail.com
> >>>>> > <mailto:vijaykumar9597 at gmail.com>> wrote:
> >>>>> >
> >>>>> > Hello,
> >>>>> >
> >>>>> > The coverage reports are now showing results.
> >>>>> > The current branch that holds all the changes is
> >>>>> > the cov-tester-support branch in my forked repo
> >>>>> >
> >>>>> > https://github.com/thelunatic/rtems-tools/tree/cov-tester-
> support
> >>>>> >
> >>>>> > <https://github.com/thelunatic/rtems-tools/tree/cov-tester-support
> >
> >>>>> >
> >>>>> > (Please have a look into the code and test it.)
> >>>>> >
> >>>>> > It is close to merging (hopefully). These are the issues
> >>>>> > that would need a fix before it can be merged :
> >>>>> >
> >>>>> > 1. I have added some #FIXME in the code (have a look)
> >>>>> > in coverage script. I have set the value of the targe to be
> >>>>> > sparc-rtems5, which makes it limited to sparc-rtems5 only.
> We
> >>>>> > can
> >>>>> > include the target in the bsp ini file, That would
> >>>>> > be a quick fix for this.
> >>>>> >
> >>>>> >
> >>>>> > Yes. This needs to be fixed.
> >>>>> >
> >>>>> > My hack to add 4 in ObjdumpProcessor.cc needs to be addressed also.
> >>>>> > I am thinking of adding a method to Target_*.cc to ask for the size
> >>>>> > of an
> >>>>> > instruction.
> >>>>> > Then pass it the last instruction. That way we can throw on other
> >>>>> > architectures for
> >>>>> > now. If Chris solves this with his changes before we try another
> >>>>> > architecture,
> >>>>> > great.
> >>>>> > If not, it will be easy to fix.
> >>>>>
> >>>>> What is the overall requirement?
> >>>>
> >>>>
> >>>> To know the ending address of the function.
> >>>>
> >>>> Technically there are three pieces of information:
> >>>>
> >>>> + start address
> >>>> + end address
> >>>> + size
> >>>>
> >>>> If you know two of those, you can compute the third.
> >>>>
> >>>> I don't care if this comes from DWARF, ELF, or parsing the
> disassembly.
> >>>> It just needs to be reliable.
> >>>>
> >>>> And.. I am not proposing my solution as permanent. Just to keep us
> >>>> moving. You want to change to an internal disassembler (which
> >>>> would also need to have the source interspersed) and DWARF. So
> >>>> this code would go away then.
> >>>>
> >>>>>
> >>>>>
> >>>>> What defines the function and so size are attempting to get coverage
> >>>>> of? What if
> >>>>> that function calls an inline function and that function is inlined?
> >>>>> What if
> >>>>> that inlined function calls another inlined function?
> >>>>
> >>>>
> >>>> Then it is all inlined. It is done consistently now. I have never
> seen a
> >>>> case
> >>>> where two instances of a method differed as the assembly level. [1]
> >>>>
> >>>> The actual body of the inlined method is evaluated at each expansion
> >>>> point.
> >>>> That is why a few years ago, I pushed hard to reduce the complexity of
> >>>> inline methods because we got test patch explosion when an inlined
> >>>> method
> >>>> included complex conditionals.
> >>>>
> >>>> Similarly, I think it would be helpful to coverage and verification
> >>>> efforts to
> >>>> follow the **shudder** MISRA rule which want you to write simple
> >>>> conditional
> >>>> expressions rather than compound ones. I have taken to writing code
> this
> >>>> way as much as possible. But haven't pushed it as a coding rule.
> >>>>
> >>>> if (a) {
> >>>> if (b) {
> >>>> do x;
> >>>> }
> >>>> }
> >>>>
> >>>> Versus
> >>>> if (a && b) {
> >>>> do x;
> >>>> }
> >>>>
> >>>> The reason is that the first is easier to analyse coverage on.
> >>>>
> >>>> [1] We both expect LTO could change this.
> >>>>
> >>>> [2] ESA did specifically mention this one though. Also in general
> terms,
> >>>> an RTEMS Project response to MISRA rules. Which ones we follow,
> >>>> reject, etc.. But I refuse to adopt hard rules which can't be enforced
> >>>> by free open source tools.
> >>>>
> >>>>>
> >>>>>
> >>>>> The DWARF data provides details about the PC low and PC high of what
> is
> >>>>> called
> >>>>> concrete functions and then it provides the details about inlines.
> >>>>
> >>>>
> >>>> We don't (currently) report on the inlines as independent methods.
> >>>>
> >>>>>
> >>>>>
> >>>>> >
> >>>>> > 2. coverage used to feed ini file for each symbol to covoar
> >>>>> > in a loop and store the result in a separate directory
> >>>>> > for each symbol . Which is needed no more needed as
> >>>>> > covoar can now process multi sets from a
> >>>>> > single ini file. I think the results from covoar should be
> >>>>> > store in a separate directory for each symbol
> >>>>> > example :- score/
> >>>>> >
> >>>>> >
> >>>>> > A bit of history will help here. Originally covoar was run against
> a
> >>>>> > single set of
> >>>>> > code by the scripting framework. We would do coverage on either
> "core
> >>>>> > parts"
> >>>>> > or "developmental" (e.g. nearly all non-networked code). The
> >>>>> > optimization was
> >>>>> > either at -O2 or -Os. So there were four coverage variants.
> >>>>> >
> >>>>> > Turned out that when we added something to "all", the percentage
> >>>>> > would drop
> >>>>> > and reflect badly on the rest of the code. I remember adding the
> >>>>> > dosfs and
> >>>>> > the coverage dropped almost 20 percent.
> >>>>> >
> >>>>> > This led to the idea that we should report on a per
> >>>>> > directory/subsystem basis.
> >>>>> > The score, rtems, posix, sapi, and libcsupport should have high
> >>>>> > coverage now
> >>>>> > and the reports should reflect that independent of whether the
> dosfs
> >>>>> > needs a
> >>>>> > lot more tests.
> >>>>> >
> >>>>> > Before each directory/subsystem required a completely separate run
> of
> >>>>> > covoar.
> >>>>> > If we are headed to a solution where one analysis run of covoar
> >>>>> > generates different
> >>>>> > reports, that should speed up the processing time a lot!
> >>>>> >
> >>>>> > The other issue is what should the top directory look like/contain
> >>>>> > when a single
> >>>>> > run produces multiple subsystem reports. Seems like we would need
> at
> >>>>> > least a
> >>>>> > top level html and text file.
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> > 3. currently we are using the leon3-qemu-cov as the bsp.
> >>>>> > Are we going to have two ini file for each bsp ? ( one
> >>>>> > without coverage
> >>>>> > and one with coverage support)
> >>>>> >
> >>>>> > Earlier the approach was to include a section 'coverage'
> >>>>> > to the bsp ini to put the values we needed for coverage.
> >>>>> > I think that approach would be more "convenient" for the user.
> >>>>> >
> >>>>> >
> >>>>> > This was something Chris suggested. I think it was to avoid adding
> to
> >>>>> > the bsp ini file until the code was closer to merging.
> >>>>> >
> >>>>> > Chris.. what do you think? Long term, a section would be nice.
> >>>>> >
> >>>>>
> >>>>> Sorry I cannot remember without looking it up and I am currently
> buried
> >>>>> in
> >>>>> family issues.
> >>>>
> >>>>
> >>>> OK. Having the Python scripting loop over the sets or covoar looping
> >>>> over them
> >>>> is an open issue. Historically, looping over desired symbol sets was
> >>>> outside
> >>>> covoar. So looping inside covoar may take some work.
> >>>
> >>>
> >>> covoar can already loop over the
> >>> sets it seems, which is implemented
> >>> in DesiredSymbols. But it stores all the
> >>> reports generated from into the same directory.
> >>
> >>
> >> If there is an index that makes its possible to navigate through the
> >> different "subsystems", then that's the key thing. You don't want
> >> to think score is poorly covered due to dosfs.
> >>
> >>>
> >>>
> >>> P.S:Sorry for the previous mail with no message
> >>>>
> >>>>
> >>>> --joel
> >>>>>
> >>>>>
> >>>>> Chris
> >>>>
> >>>>
> >>
> >
> >
> > _______________________________________________
> > devel mailing list
> > devel at rtems.org
> > http://lists.rtems.org/mailman/listinfo/devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rtems.org/pipermail/devel/attachments/20180530/9b6dd0f7/attachment-0002.html>
More information about the devel
mailing list