<offllist> Coverage support for tester
Gedare Bloom
gedare at rtems.org
Wed May 30 15:58:52 UTC 2018
On Wed, May 30, 2018 at 11:32 AM, Vijay Kumar Banerjee
<vijaykumar9597 at gmail.com> wrote:
> On 30 May 2018 at 20:18, Gedare Bloom <gedare at rtems.org> wrote:
>>
>> Hello Vijay,
>>
>> Do you expect/need an answer to something in here?
>>
>> gedare
>>
> Hello,
>
> I wanted to know if there were any plans on how covoar
> should store the reports when running for multisets.
> Earlier it used to be done by the coverage script,
> after the recent changes covoar can process multi sets.
>
> I think, covoar should store the reports into separate directories
> for each set . eg. score/ , rtems/ . As the coverage can already read
> from separate directories.
>
> Any advice on how should it be approached ?
It would help me if you could draw/write a diagram of what the
filesystem tree might look like with separate directories, and what
will go in each subdirectory.
I don't have enough context to give any useful advice on this question.
-Gedare
>>
>> On Wed, May 30, 2018 at 10:29 AM, Vijay Kumar Banerjee
>> <vijaykumar9597 at gmail.com> wrote:
>> > On 30 May 2018 at 00:54, Joel Sherrill <joel at rtems.org> wrote:
>> >>
>> >>
>> >>
>> >> On Tue, May 29, 2018 at 11:27 AM, Vijay Kumar Banerjee
>> >> <vijaykumar9597 at gmail.com> wrote:
>> >>>
>> >>>
>> >>>
>> >>> On Tue, 29 May 2018, 20:10 Joel Sherrill, <joel at rtems.org> wrote:
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Mon, May 28, 2018 at 11:08 PM, Chris Johns <chrisj at rtems.org>
>> >>>> wrote:
>> >>>>>
>> >>>>> On 29/5/18 4:26 am, Joel Sherrill wrote:
>> >>>>> > On Mon, May 28, 2018 at 5:43 AM, Vijay Kumar Banerjee
>> >>>>> > <vijaykumar9597 at gmail.com
>> >>>>> > <mailto:vijaykumar9597 at gmail.com>> wrote:
>> >>>>> >
>> >>>>> > Hello,
>> >>>>> >
>> >>>>> > The coverage reports are now showing results.
>> >>>>> > The current branch that holds all the changes is
>> >>>>> > the cov-tester-support branch in my forked repo
>> >>>>> >
>> >>>>> >
>> >>>>> > https://github.com/thelunatic/rtems-tools/tree/cov-tester-support
>> >>>>> >
>> >>>>> >
>> >>>>> > <https://github.com/thelunatic/rtems-tools/tree/cov-tester-support>
>> >>>>> >
>> >>>>> > (Please have a look into the code and test it.)
>> >>>>> >
>> >>>>> > It is close to merging (hopefully). These are the issues
>> >>>>> > that would need a fix before it can be merged :
>> >>>>> >
>> >>>>> > 1. I have added some #FIXME in the code (have a look)
>> >>>>> > in coverage script. I have set the value of the targe to
>> >>>>> > be
>> >>>>> > sparc-rtems5, which makes it limited to sparc-rtems5 only.
>> >>>>> > We
>> >>>>> > can
>> >>>>> > include the target in the bsp ini file, That would
>> >>>>> > be a quick fix for this.
>> >>>>> >
>> >>>>> >
>> >>>>> > Yes. This needs to be fixed.
>> >>>>> >
>> >>>>> > My hack to add 4 in ObjdumpProcessor.cc needs to be addressed
>> >>>>> > also.
>> >>>>> > I am thinking of adding a method to Target_*.cc to ask for the
>> >>>>> > size
>> >>>>> > of an
>> >>>>> > instruction.
>> >>>>> > Then pass it the last instruction. That way we can throw on other
>> >>>>> > architectures for
>> >>>>> > now. If Chris solves this with his changes before we try another
>> >>>>> > architecture,
>> >>>>> > great.
>> >>>>> > If not, it will be easy to fix.
>> >>>>>
>> >>>>> What is the overall requirement?
>> >>>>
>> >>>>
>> >>>> To know the ending address of the function.
>> >>>>
>> >>>> Technically there are three pieces of information:
>> >>>>
>> >>>> + start address
>> >>>> + end address
>> >>>> + size
>> >>>>
>> >>>> If you know two of those, you can compute the third.
>> >>>>
>> >>>> I don't care if this comes from DWARF, ELF, or parsing the
>> >>>> disassembly.
>> >>>> It just needs to be reliable.
>> >>>>
>> >>>> And.. I am not proposing my solution as permanent. Just to keep us
>> >>>> moving. You want to change to an internal disassembler (which
>> >>>> would also need to have the source interspersed) and DWARF. So
>> >>>> this code would go away then.
>> >>>>
>> >>>>>
>> >>>>>
>> >>>>> What defines the function and so size are attempting to get coverage
>> >>>>> of? What if
>> >>>>> that function calls an inline function and that function is inlined?
>> >>>>> What if
>> >>>>> that inlined function calls another inlined function?
>> >>>>
>> >>>>
>> >>>> Then it is all inlined. It is done consistently now. I have never
>> >>>> seen a
>> >>>> case
>> >>>> where two instances of a method differed as the assembly level. [1]
>> >>>>
>> >>>> The actual body of the inlined method is evaluated at each expansion
>> >>>> point.
>> >>>> That is why a few years ago, I pushed hard to reduce the complexity
>> >>>> of
>> >>>> inline methods because we got test patch explosion when an inlined
>> >>>> method
>> >>>> included complex conditionals.
>> >>>>
>> >>>> Similarly, I think it would be helpful to coverage and verification
>> >>>> efforts to
>> >>>> follow the **shudder** MISRA rule which want you to write simple
>> >>>> conditional
>> >>>> expressions rather than compound ones. I have taken to writing code
>> >>>> this
>> >>>> way as much as possible. But haven't pushed it as a coding rule.
>> >>>>
>> >>>> if (a) {
>> >>>> if (b) {
>> >>>> do x;
>> >>>> }
>> >>>> }
>> >>>>
>> >>>> Versus
>> >>>> if (a && b) {
>> >>>> do x;
>> >>>> }
>> >>>>
>> >>>> The reason is that the first is easier to analyse coverage on.
>> >>>>
>> >>>> [1] We both expect LTO could change this.
>> >>>>
>> >>>> [2] ESA did specifically mention this one though. Also in general
>> >>>> terms,
>> >>>> an RTEMS Project response to MISRA rules. Which ones we follow,
>> >>>> reject, etc.. But I refuse to adopt hard rules which can't be
>> >>>> enforced
>> >>>> by free open source tools.
>> >>>>
>> >>>>>
>> >>>>>
>> >>>>> The DWARF data provides details about the PC low and PC high of what
>> >>>>> is
>> >>>>> called
>> >>>>> concrete functions and then it provides the details about inlines.
>> >>>>
>> >>>>
>> >>>> We don't (currently) report on the inlines as independent methods.
>> >>>>
>> >>>>>
>> >>>>>
>> >>>>> >
>> >>>>> > 2. coverage used to feed ini file for each symbol to covoar
>> >>>>> > in a loop and store the result in a separate directory
>> >>>>> > for each symbol . Which is needed no more needed as
>> >>>>> > covoar can now process multi sets from a
>> >>>>> > single ini file. I think the results from covoar should
>> >>>>> > be
>> >>>>> > store in a separate directory for each symbol
>> >>>>> > example :- score/
>> >>>>> >
>> >>>>> >
>> >>>>> > A bit of history will help here. Originally covoar was run against
>> >>>>> > a
>> >>>>> > single set of
>> >>>>> > code by the scripting framework. We would do coverage on either
>> >>>>> > "core
>> >>>>> > parts"
>> >>>>> > or "developmental" (e.g. nearly all non-networked code). The
>> >>>>> > optimization was
>> >>>>> > either at -O2 or -Os. So there were four coverage variants.
>> >>>>> >
>> >>>>> > Turned out that when we added something to "all", the percentage
>> >>>>> > would drop
>> >>>>> > and reflect badly on the rest of the code. I remember adding the
>> >>>>> > dosfs and
>> >>>>> > the coverage dropped almost 20 percent.
>> >>>>> >
>> >>>>> > This led to the idea that we should report on a per
>> >>>>> > directory/subsystem basis.
>> >>>>> > The score, rtems, posix, sapi, and libcsupport should have high
>> >>>>> > coverage now
>> >>>>> > and the reports should reflect that independent of whether the
>> >>>>> > dosfs
>> >>>>> > needs a
>> >>>>> > lot more tests.
>> >>>>> >
>> >>>>> > Before each directory/subsystem required a completely separate run
>> >>>>> > of
>> >>>>> > covoar.
>> >>>>> > If we are headed to a solution where one analysis run of covoar
>> >>>>> > generates different
>> >>>>> > reports, that should speed up the processing time a lot!
>> >>>>> >
>> >>>>> > The other issue is what should the top directory look like/contain
>> >>>>> > when a single
>> >>>>> > run produces multiple subsystem reports. Seems like we would need
>> >>>>> > at
>> >>>>> > least a
>> >>>>> > top level html and text file.
>> >>>>> >
>> >>>>> >
>> >>>>> >
>> >>>>> > 3. currently we are using the leon3-qemu-cov as the bsp.
>> >>>>> > Are we going to have two ini file for each bsp ? ( one
>> >>>>> > without coverage
>> >>>>> > and one with coverage support)
>> >>>>> >
>> >>>>> > Earlier the approach was to include a section 'coverage'
>> >>>>> > to the bsp ini to put the values we needed for coverage.
>> >>>>> > I think that approach would be more "convenient" for the user.
>> >>>>> >
>> >>>>> >
>> >>>>> > This was something Chris suggested. I think it was to avoid adding
>> >>>>> > to
>> >>>>> > the bsp ini file until the code was closer to merging.
>> >>>>> >
>> >>>>> > Chris.. what do you think? Long term, a section would be nice.
>> >>>>> >
>> >>>>>
>> >>>>> Sorry I cannot remember without looking it up and I am currently
>> >>>>> buried
>> >>>>> in
>> >>>>> family issues.
>> >>>>
>> >>>>
>> >>>> OK. Having the Python scripting loop over the sets or covoar looping
>> >>>> over them
>> >>>> is an open issue. Historically, looping over desired symbol sets was
>> >>>> outside
>> >>>> covoar. So looping inside covoar may take some work.
>> >>>
>> >>>
>> >>> covoar can already loop over the
>> >>> sets it seems, which is implemented
>> >>> in DesiredSymbols. But it stores all the
>> >>> reports generated from into the same directory.
>> >>
>> >>
>> >> If there is an index that makes its possible to navigate through the
>> >> different "subsystems", then that's the key thing. You don't want
>> >> to think score is poorly covered due to dosfs.
>> >>
>> >>>
>> >>>
>> >>> P.S:Sorry for the previous mail with no message
>> >>>>
>> >>>>
>> >>>> --joel
>> >>>>>
>> >>>>>
>> >>>>> Chris
>> >>>>
>> >>>>
>> >>
>> >
>> >
>> > _______________________________________________
>> > devel mailing list
>> > devel at rtems.org
>> > http://lists.rtems.org/mailman/listinfo/devel
>
>
More information about the devel
mailing list