<offllist> Coverage support for tester
Vijay Kumar Banerjee
vijaykumar9597 at gmail.com
Wed May 30 17:30:37 UTC 2018
On 30 May 2018 at 22:55, Cillian O'Donnell <cpodonnell8 at gmail.com> wrote:
>
>
> On Wed, 30 May 2018, 18:15 Vijay Kumar Banerjee, <vijaykumar9597 at gmail.com>
> wrote:
>
>>
>>
>> On Wed, 30 May 2018, 22:42 Cillian O'Donnell, <cpodonnell8 at gmail.com>
>> wrote:
>>
>>>
>>>
>>> On Wed, 30 May 2018, 16:58 Gedare Bloom, <gedare at rtems.org> wrote:
>>>
>>>> On Wed, May 30, 2018 at 11:32 AM, Vijay Kumar Banerjee
>>>> <vijaykumar9597 at gmail.com> wrote:
>>>> > On 30 May 2018 at 20:18, Gedare Bloom <gedare at rtems.org> wrote:
>>>> >>
>>>> >> Hello Vijay,
>>>> >>
>>>> >> Do you expect/need an answer to something in here?
>>>> >>
>>>> >> gedare
>>>> >>
>>>> > Hello,
>>>> >
>>>> > I wanted to know if there were any plans on how covoar
>>>> > should store the reports when running for multisets.
>>>> > Earlier it used to be done by the coverage script,
>>>> > after the recent changes covoar can process multi sets.
>>>> >
>>>> > I think, covoar should store the reports into separate directories
>>>> > for each set . eg. score/ , rtems/ . As the coverage can already read
>>>> > from separate directories.
>>>>
>>>
>>> Sorry I thought all questions had been answered here. I think you have
>>> the right idea. Each set should be a sub-directory of coverage directory.
>>>
>>> By the way I tested your changes and everything seems fine. Still have
>>> to do a review of coverage.py to see how close we are to merging.
>>>
>> I have squashed everything and sent a patch to devel at . This will make it
>> easy to go through all the changes. Please have a look.
>>
>
> Ah just seen it, will take a look. It'll be good to get Chris' thoughts
> too. We'll have something more definitive about how close it is to merging.
>
Sure.
> >
>>>> > Any advice on how should it be approached ?
>>>>
>>>> It would help me if you could draw/write a diagram of what the
>>>> filesystem tree might look like with separate directories, and what
>>>> will go in each subdirectory.
>>>>
>>>> I don't have enough context to give any useful advice on this question.
>>>>
>>>> -Gedare
>>>>
>>>> >>
>>>> >> On Wed, May 30, 2018 at 10:29 AM, Vijay Kumar Banerjee
>>>> >> <vijaykumar9597 at gmail.com> wrote:
>>>> >> > On 30 May 2018 at 00:54, Joel Sherrill <joel at rtems.org> wrote:
>>>> >> >>
>>>> >> >>
>>>> >> >>
>>>> >> >> On Tue, May 29, 2018 at 11:27 AM, Vijay Kumar Banerjee
>>>> >> >> <vijaykumar9597 at gmail.com> wrote:
>>>> >> >>>
>>>> >> >>>
>>>> >> >>>
>>>> >> >>> On Tue, 29 May 2018, 20:10 Joel Sherrill, <joel at rtems.org>
>>>> wrote:
>>>> >> >>>>
>>>> >> >>>>
>>>> >> >>>>
>>>> >> >>>> On Mon, May 28, 2018 at 11:08 PM, Chris Johns <chrisj at rtems.org
>>>> >
>>>> >> >>>> wrote:
>>>> >> >>>>>
>>>> >> >>>>> On 29/5/18 4:26 am, Joel Sherrill wrote:
>>>> >> >>>>> > On Mon, May 28, 2018 at 5:43 AM, Vijay Kumar Banerjee
>>>> >> >>>>> > <vijaykumar9597 at gmail.com
>>>> >> >>>>> > <mailto:vijaykumar9597 at gmail.com>> wrote:
>>>> >> >>>>> >
>>>> >> >>>>> > Hello,
>>>> >> >>>>> >
>>>> >> >>>>> > The coverage reports are now showing results.
>>>> >> >>>>> > The current branch that holds all the changes is
>>>> >> >>>>> > the cov-tester-support branch in my forked repo
>>>> >> >>>>> >
>>>> >> >>>>> >
>>>> >> >>>>> > https://github.com/thelunatic/rtems-tools/tree/cov-tester-
>>>> support
>>>> >> >>>>> >
>>>> >> >>>>> >
>>>> >> >>>>> > <https://github.com/thelunatic/rtems-tools/tree/
>>>> cov-tester-support>
>>>> >> >>>>> >
>>>> >> >>>>> > (Please have a look into the code and test it.)
>>>> >> >>>>> >
>>>> >> >>>>> > It is close to merging (hopefully). These are the issues
>>>> >> >>>>> > that would need a fix before it can be merged :
>>>> >> >>>>> >
>>>> >> >>>>> > 1. I have added some #FIXME in the code (have a look)
>>>> >> >>>>> > in coverage script. I have set the value of the
>>>> targe to
>>>> >> >>>>> > be
>>>> >> >>>>> > sparc-rtems5, which makes it limited to sparc-rtems5
>>>> only.
>>>> >> >>>>> > We
>>>> >> >>>>> > can
>>>> >> >>>>> > include the target in the bsp ini file, That would
>>>> >> >>>>> > be a quick fix for this.
>>>> >> >>>>> >
>>>> >> >>>>> >
>>>> >> >>>>> > Yes. This needs to be fixed.
>>>> >> >>>>> >
>>>> >> >>>>> > My hack to add 4 in ObjdumpProcessor.cc needs to be addressed
>>>> >> >>>>> > also.
>>>> >> >>>>> > I am thinking of adding a method to Target_*.cc to ask for
>>>> the
>>>> >> >>>>> > size
>>>> >> >>>>> > of an
>>>> >> >>>>> > instruction.
>>>> >> >>>>> > Then pass it the last instruction. That way we can throw on
>>>> other
>>>> >> >>>>> > architectures for
>>>> >> >>>>> > now. If Chris solves this with his changes before we try
>>>> another
>>>> >> >>>>> > architecture,
>>>> >> >>>>> > great.
>>>> >> >>>>> > If not, it will be easy to fix.
>>>> >> >>>>>
>>>> >> >>>>> What is the overall requirement?
>>>> >> >>>>
>>>> >> >>>>
>>>> >> >>>> To know the ending address of the function.
>>>> >> >>>>
>>>> >> >>>> Technically there are three pieces of information:
>>>> >> >>>>
>>>> >> >>>> + start address
>>>> >> >>>> + end address
>>>> >> >>>> + size
>>>> >> >>>>
>>>> >> >>>> If you know two of those, you can compute the third.
>>>> >> >>>>
>>>> >> >>>> I don't care if this comes from DWARF, ELF, or parsing the
>>>> >> >>>> disassembly.
>>>> >> >>>> It just needs to be reliable.
>>>> >> >>>>
>>>> >> >>>> And.. I am not proposing my solution as permanent. Just to keep
>>>> us
>>>> >> >>>> moving. You want to change to an internal disassembler (which
>>>> >> >>>> would also need to have the source interspersed) and DWARF. So
>>>> >> >>>> this code would go away then.
>>>> >> >>>>
>>>> >> >>>>>
>>>> >> >>>>>
>>>> >> >>>>> What defines the function and so size are attempting to get
>>>> coverage
>>>> >> >>>>> of? What if
>>>> >> >>>>> that function calls an inline function and that function is
>>>> inlined?
>>>> >> >>>>> What if
>>>> >> >>>>> that inlined function calls another inlined function?
>>>> >> >>>>
>>>> >> >>>>
>>>> >> >>>> Then it is all inlined. It is done consistently now. I have
>>>> never
>>>> >> >>>> seen a
>>>> >> >>>> case
>>>> >> >>>> where two instances of a method differed as the assembly level.
>>>> [1]
>>>> >> >>>>
>>>> >> >>>> The actual body of the inlined method is evaluated at each
>>>> expansion
>>>> >> >>>> point.
>>>> >> >>>> That is why a few years ago, I pushed hard to reduce the
>>>> complexity
>>>> >> >>>> of
>>>> >> >>>> inline methods because we got test patch explosion when an
>>>> inlined
>>>> >> >>>> method
>>>> >> >>>> included complex conditionals.
>>>> >> >>>>
>>>> >> >>>> Similarly, I think it would be helpful to coverage and
>>>> verification
>>>> >> >>>> efforts to
>>>> >> >>>> follow the **shudder** MISRA rule which want you to write simple
>>>> >> >>>> conditional
>>>> >> >>>> expressions rather than compound ones. I have taken to writing
>>>> code
>>>> >> >>>> this
>>>> >> >>>> way as much as possible. But haven't pushed it as a coding rule.
>>>> >> >>>>
>>>> >> >>>> if (a) {
>>>> >> >>>> if (b) {
>>>> >> >>>> do x;
>>>> >> >>>> }
>>>> >> >>>> }
>>>> >> >>>>
>>>> >> >>>> Versus
>>>> >> >>>> if (a && b) {
>>>> >> >>>> do x;
>>>> >> >>>> }
>>>> >> >>>>
>>>> >> >>>> The reason is that the first is easier to analyse coverage on.
>>>> >> >>>>
>>>> >> >>>> [1] We both expect LTO could change this.
>>>> >> >>>>
>>>> >> >>>> [2] ESA did specifically mention this one though. Also in
>>>> general
>>>> >> >>>> terms,
>>>> >> >>>> an RTEMS Project response to MISRA rules. Which ones we follow,
>>>> >> >>>> reject, etc.. But I refuse to adopt hard rules which can't be
>>>> >> >>>> enforced
>>>> >> >>>> by free open source tools.
>>>> >> >>>>
>>>> >> >>>>>
>>>> >> >>>>>
>>>> >> >>>>> The DWARF data provides details about the PC low and PC high
>>>> of what
>>>> >> >>>>> is
>>>> >> >>>>> called
>>>> >> >>>>> concrete functions and then it provides the details about
>>>> inlines.
>>>> >> >>>>
>>>> >> >>>>
>>>> >> >>>> We don't (currently) report on the inlines as independent
>>>> methods.
>>>> >> >>>>
>>>> >> >>>>>
>>>> >> >>>>>
>>>> >> >>>>> >
>>>> >> >>>>> > 2. coverage used to feed ini file for each symbol to
>>>> covoar
>>>> >> >>>>> > in a loop and store the result in a separate
>>>> directory
>>>> >> >>>>> > for each symbol . Which is needed no more needed as
>>>> >> >>>>> > covoar can now process multi sets from a
>>>> >> >>>>> > single ini file. I think the results from covoar
>>>> should
>>>> >> >>>>> > be
>>>> >> >>>>> > store in a separate directory for each symbol
>>>> >> >>>>> > example :- score/
>>>> >> >>>>> >
>>>> >> >>>>> >
>>>> >> >>>>> > A bit of history will help here. Originally covoar was run
>>>> against
>>>> >> >>>>> > a
>>>> >> >>>>> > single set of
>>>> >> >>>>> > code by the scripting framework. We would do coverage on
>>>> either
>>>> >> >>>>> > "core
>>>> >> >>>>> > parts"
>>>> >> >>>>> > or "developmental" (e.g. nearly all non-networked code). The
>>>> >> >>>>> > optimization was
>>>> >> >>>>> > either at -O2 or -Os. So there were four coverage variants.
>>>> >> >>>>> >
>>>> >> >>>>> > Turned out that when we added something to "all", the
>>>> percentage
>>>> >> >>>>> > would drop
>>>> >> >>>>> > and reflect badly on the rest of the code. I remember adding
>>>> the
>>>> >> >>>>> > dosfs and
>>>> >> >>>>> > the coverage dropped almost 20 percent.
>>>> >> >>>>> >
>>>> >> >>>>> > This led to the idea that we should report on a per
>>>> >> >>>>> > directory/subsystem basis.
>>>> >> >>>>> > The score, rtems, posix, sapi, and libcsupport should have
>>>> high
>>>> >> >>>>> > coverage now
>>>> >> >>>>> > and the reports should reflect that independent of whether
>>>> the
>>>> >> >>>>> > dosfs
>>>> >> >>>>> > needs a
>>>> >> >>>>> > lot more tests.
>>>> >> >>>>> >
>>>> >> >>>>> > Before each directory/subsystem required a completely
>>>> separate run
>>>> >> >>>>> > of
>>>> >> >>>>> > covoar.
>>>> >> >>>>> > If we are headed to a solution where one analysis run of
>>>> covoar
>>>> >> >>>>> > generates different
>>>> >> >>>>> > reports, that should speed up the processing time a lot!
>>>> >> >>>>> >
>>>> >> >>>>> > The other issue is what should the top directory look
>>>> like/contain
>>>> >> >>>>> > when a single
>>>> >> >>>>> > run produces multiple subsystem reports. Seems like we would
>>>> need
>>>> >> >>>>> > at
>>>> >> >>>>> > least a
>>>> >> >>>>> > top level html and text file.
>>>> >> >>>>> >
>>>> >> >>>>> >
>>>> >> >>>>> >
>>>> >> >>>>> > 3. currently we are using the leon3-qemu-cov as the bsp.
>>>> >> >>>>> > Are we going to have two ini file for each bsp ? (
>>>> one
>>>> >> >>>>> > without coverage
>>>> >> >>>>> > and one with coverage support)
>>>> >> >>>>> >
>>>> >> >>>>> > Earlier the approach was to include a section 'coverage'
>>>> >> >>>>> > to the bsp ini to put the values we needed for coverage.
>>>> >> >>>>> > I think that approach would be more "convenient" for the
>>>> user.
>>>> >> >>>>> >
>>>> >> >>>>> >
>>>> >> >>>>> > This was something Chris suggested. I think it was to avoid
>>>> adding
>>>> >> >>>>> > to
>>>> >> >>>>> > the bsp ini file until the code was closer to merging.
>>>> >> >>>>> >
>>>> >> >>>>> > Chris.. what do you think? Long term, a section would be
>>>> nice.
>>>> >> >>>>> >
>>>> >> >>>>>
>>>> >> >>>>> Sorry I cannot remember without looking it up and I am
>>>> currently
>>>> >> >>>>> buried
>>>> >> >>>>> in
>>>> >> >>>>> family issues.
>>>> >> >>>>
>>>> >> >>>>
>>>> >> >>>> OK. Having the Python scripting loop over the sets or covoar
>>>> looping
>>>> >> >>>> over them
>>>> >> >>>> is an open issue. Historically, looping over desired symbol
>>>> sets was
>>>> >> >>>> outside
>>>> >> >>>> covoar. So looping inside covoar may take some work.
>>>> >> >>>
>>>> >> >>>
>>>> >> >>> covoar can already loop over the
>>>> >> >>> sets it seems, which is implemented
>>>> >> >>> in DesiredSymbols. But it stores all the
>>>> >> >>> reports generated from into the same directory.
>>>> >> >>
>>>> >> >>
>>>> >> >> If there is an index that makes its possible to navigate through
>>>> the
>>>> >> >> different "subsystems", then that's the key thing. You don't want
>>>> >> >> to think score is poorly covered due to dosfs.
>>>> >> >>
>>>> >> >>>
>>>> >> >>>
>>>> >> >>> P.S:Sorry for the previous mail with no message
>>>> >> >>>>
>>>> >> >>>>
>>>> >> >>>> --joel
>>>> >> >>>>>
>>>> >> >>>>>
>>>> >> >>>>> Chris
>>>> >> >>>>
>>>> >> >>>>
>>>> >> >>
>>>> >> >
>>>> >> >
>>>> >> > _______________________________________________
>>>> >> > devel mailing list
>>>> >> > devel at rtems.org
>>>> >> > http://lists.rtems.org/mailman/listinfo/devel
>>>> >
>>>> >
>>>> _______________________________________________
>>>> devel mailing list
>>>> devel at rtems.org
>>>> http://lists.rtems.org/mailman/listinfo/devel
>>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rtems.org/pipermail/devel/attachments/20180530/f221da8a/attachment-0001.html>
More information about the devel
mailing list