[PATCH v2] tester: Add script to generate html coverage report from covoar output

[Vijay Kumar Banerjee]

On 2 June 2018 at 01:00, Vijay Kumar Banerjee <vijaykumar9597 at gmail.com>
wrote:

> On 2 June 2018 at 00:48, Joel Sherrill <joel at rtems.org> wrote:
>
>>
>>
>> On Fri, Jun 1, 2018, 11:21 AM Vijay Kumar Banerjee <
>> vijaykumar9597 at gmail.com> wrote:
>>
>>> On 1 June 2018 at 20:30, Gedare Bloom <gedare at rtems.org> wrote:
>>>
>>>> On Fri, Jun 1, 2018 at 10:28 AM, Vijay Kumar Banerjee
>>>> <vijaykumar9597 at gmail.com> wrote:
>>>> > On 1 June 2018 at 19:24, Joel Sherrill <joel at rtems.org> wrote:
>>>> >>
>>>> >>
>>>> >>
>>>> >> On Fri, Jun 1, 2018 at 2:46 AM, Vijay Kumar Banerjee
>>>> >> <vijaykumar9597 at gmail.com> wrote:
>>>> >>>
>>>> >>> Here's the list of Ideas for improvements:
>>>> >>>
>>>> >>> 1. Include the section coverage in the bsp config file.
>>>> >>>     If the section is not found then the script will show
>>>> >>>     proper error showing coverage is not supported for the
>>>> >>>     provided bsp config file.
>>>> >>>
>>>> >>> 2. Update covoar to add support for separate coverage report
>>>> >>>     for each symbol set.
>>>> >>>
>>>> >>> 3. Add a method somewhere in covoar to get the size of an
>>>> instruction
>>>> >>>     and fix the hard coded size 4 in ObjdumpProcessor.cc
>>>> >>
>>>> >>
>>>> >> What about a single XXX_run command? What about that suggestion?
>>>> >>
>>>> > The suggestion was to turn test_run and coverage_run into a single
>>>> command.
>>>> > I have kept them separate so that there's a possibility to just run
>>>> the
>>>> > test.
>>>> >
>>>> > If we want to run coverage everytime we run the test. we can do it.
>>>> > Then I think the --coverage option can be changed to --coverage-sets
>>>> > to mention the sets.
>>>> > If that's what we're looking for then I don't think a separate ticket
>>>> is
>>>> > needed,
>>>> > I can try to do it within tomorrow and submit an updated patch.
>>>> >
>>>> >>
>>>> >> Will there be an update to this patch?
>>>> >>
>>>> > This patch is working an showing results. I don't have any work
>>>> > going related to this patch currently.
>>>> > If there are any suggestions, I'll try to include all the suggested
>>>> updates
>>>> > as soon as possible and submit. So that we can get it merged.
>>>> >
>>>>
>>>> I get confused by the similarity between test_run() and coverage_run()
>>>> names, and now I'm also seeing some confusion because there is a
>>>> function coverage_run() and a class coverage_run. I suggest you remove
>>>> this function coverage_run(), and make either coverage_run.__init__()
>>>> or coverage_run.run() take the executables as a parameter directly.
>>>>
>>>> Thank you for the suggestion. :)
>>> I have removed the function and taken executables as a parameter in
>>> coverage_run.__init__()
>>>
>>> I have a question.
>>> The ini file that is fed to covoar is written by the script according to
>>> the
>>> symbols mentioned by the user. I haven't include the ini file in the
>>> patch.
>>> I'm planning to delete the file after the run, unless --no-clean option
>>> is given,
>>> which currently deletes the .cov trace files after the run.
>>>
>>
>> That makes sense.
>>
>>
>>> Can I proceed with this ?
>>>
>>
>> Yes.
>>
> Added. Thanks. :)
>
>> also, shall I include that in the .gitignore ?
>>>
>>
>> Is the name of the file constant? The same across multiple BSPs? If so,
>> then this will be a problem doing automated testing of multiple BSPs in
>> parallel.
>>
>> The ini file I'm talking about is the symbol sets config file not the bsp
> config file. yes the name is constant. Should it be unique to the bsp ?
> something like, leon3-symbols.ini ?
> How does the automated testing work?
>
>> I think the name needs to be unique enough.to support running testing
>> with coverage on multiple BSPs in parallel. That means you can't add it to
>> gitignore. And can add another issue and FIXME to the code.
>>
>>> If it is needed then I have a fix in mind. I can take the bsp name and
> add
> '-symbols.ini' to it. and add *-symbols.ini to .gitignore .
>
Shall I add this or put a fixme in the code and post a patch ?
Are there any other suggestions for the patch ?

I was looking into covoar for generating separate reports for each
symbolset.
Seems like  all the coverage reports are generated together and are written
in the _outputDirectory_ . I couldn't figure out how to cleanly address
this.
If covoar is intended to generate reports from multiple
subsystems/symbolsets,
then I think this would be a needed update. Otherwise we can do it from the
script, by feeding covoar with a single set ini and putting the result in a
separate
directory .

> Can we do this ?
>
>>
>>>
>>>> -Gedare
>>>>
>>>
>>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rtems.org/pipermail/devel/attachments/20180604/3904145d/attachment-0002.html>