GSoC 2017 -- initial proof

Sun Apr 2 17:53:56 UTC 2017

On Sun, Apr 2, 2017 at 12:33 PM, Tanu Hari Dixit <tokencolour at gmail.com>
wrote:

> Hi Andy,
>
> Joel and Gedare pointed me to the following:
>
> https://devel.rtems.org/wiki/Developer/Projects/Open/
> PythonCoverageReporting
> https://devel.rtems.org/ticket/2261
> http://kmiesowicz.blogspot.in/p/esa-socis-2014.html
>
> Last link is to the work done by a previous student.
> Hope this helps.
> My proposal does not take covoar into consideration. Though, I have
> planned to produce xml reports from rtems-tester that can be
> integrated into coverage testing. Also, I have planned to use this xml
> data to get plots (for visualizing results) as a stretch goal (you can
> look in the "Future Improvements" Section in my proposal).
>

covoar directly produces the html and text reports we have now. My
recollection is that it doesn't have to be touched and that the trick
is invoking it on a more focused basis than Core vs Developmental.
So invoke covoar from rtems-tester mulitple times on a single run
of the tests rather than twice at most.

Personally, I think Core is still a good description for the core of
the tasking (e.g. score, rtems, sapi, and posix) and that each
directory other than that should be individual. But that's my marketing
view. I could easily be argued that for consistency, it should all
be on a per directory basis.

At some point, we might discuss covoar writing a neutral format
and using Python to do more but that is pure conjecture. I have
no idea what the intermediate format would be. Writing Sphinx
output for reports would be desirable from covoar. :)

--joel

>
> Regards,
> Tanu Hari Dixit.
>
> On Sun, Apr 2, 2017 at 6:50 PM, Gedare Bloom <gedare at rtems.org> wrote:
> > Joel's e-mail client messed up my reply thread. ;) Andy, have a read
> > through https://gedare.github.io/pdf/FelShe15A.pdf if you haven't
> > found it already.
> >
> > Getting qemu-couverture to build via RSB and then integrating the
> > coverage scripts into rtems-test, would be a good start to a project.
> > There is a student (Tanu)oar dir who has proposed rtems-test
> improvements, so
> > you two would need to coordinate efforts for this piece to avoid too
> > much duplicated work/waste. Improvements to the reports etc are also
> > welcome, and then of course using the reports to guide code
> > refactoring to improve coverage is somewhat open-ended.
> >
> > -Gedare
> >
> > On Sat, Apr 1, 2017 at 9:34 PM, Joel Sherrill <joel at rtems.org> wrote:
> >>
> >>
> >> On Apr 1, 2017 7:59 PM, "Andy MacGregor" <amacgregor.2018 at comcast.net>
> >> wrote:
> >>
> >> On 03/31/2017 09:52 PM, Gedare Bloom wrote:
> >>
> >> Don't forget the deadline is Monday to submit your Final
> >> PDF and proof of enrollment. The "Final" PDF can be submitted multiple
> >> times, so go ahead and submit it when you finish a first draft. Add
> >> your draft proposal to the tracking table at
> >> https://devel.rtems.org/wiki/GSoC/2017 also, to get some possible
> >> feedback.
> >>
> >>
> >> Thanks for the welcome! (I realize I am late to the GSOC application
> table).
> >>
> >> I'm set on improving code coverage of the testsuite, but I found some
> >> questions as I'm starting to look into how its done.
> >>
> >> The link on the wiki to the current code status is currently broken, so
> I
> >> tried to generate a sample coverage report for 4.12 using ./do_coverage,
> >> ./run_coverage and ./coverage_cron as suggested on the wiki here but
> without
> >> any luck. Is there an up to date coverage report available?
> >> I modified the VERSIONS-COVERAGE file like the wiki said, and once that
> >> didn't work dug around in the script but fix it myself quickly.
> >>
> >> Could updating these coverage analysis scripts be a viable component of
> a
> >> GSOC proposal?
> >>
> >>
> >> Yes. In fact, the idea was to rework the coverage scripts in shell to be
> >> part of the rtems-tester package and in Python. There is some work on
> this
> >> by a previous student which you can find or one of us can dig out.
> Starting
> >> with it, making it work across multiple BSPs, independent of any hard
> coded
> >> paths, improving the output (we did not get far enough that I recall
> >> worrying about the quality of the output), etc could be a fair amount of
> >> work.
> >>
> >> Also we have privately discussed switching the RSB qemu upstream source
> to
> >> GNATCoverage at GitHub. This is a coverage and embedded focused
> downstream
> >> which includes the coverage trace output I used for the qemu support in
> >> covoar. I'm
> >>
> >> In addition, I'd like to actually make some improvements the code
> coverage
> >> itself.
> >> The most recent information I could find was these bar graphs from 2009.
> >> If these graphs reflect the current coverage state, I think that
> improving
> >> coverage for the i386/pc386,
> >> would be interesting because there is room for improvement in both the
> >> Baseline and Developmental groups (if these coverage statistics are up
> to
> >> date).
> >>
> >>
> >> It doesn't matter which BSP you choose for coverage since the code and
> tests
> >> are the same.
> >>
> >> This also reminds me of the other improvements we had wanted. We wanted
> to
> >> switch from Baseline and developmental to a more directory oriented
> approach
> >> for the reports. For example, the reports should be at a granularity of
> say
> >> the Dos file system or shell as reports. The way reports are generated
> now
> >> it results in a skewed view when a large body of code is added to the
> >> developmental set. Adding the Dos filesystem resulted in a 15-20% drop
> in
> >> the coverage reported in the Developmental set. That is a bad way to
> report
> >> and misleading. It detracts from the directories which have near 100%
> >> coverage.
> >>
> >>
> >> Does anyone know of a good reason to choose another BSP over i386/pc386?
> >>
> >>
> >> Users fly the SPARC leon3 and not the pc386. :)
> >>
> >>
> >> I've just posted a draft of my proposal to the GSOC page.
> >> If there's any feedback I'll be sure to update my proposal in time.
> >>
> >>
> >> Dinner time here. Hopefully my feedback makes sense and you can
> incorporate
> >> it.
> >>
> >>
> >>
> >>
> >> -Andy MacGregor
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> _______________________________________________
> >> devel mailing list
> >> devel at rtems.org
> >> http://lists.rtems.org/mailman/listinfo/devel
> >>
> >>
> >>
> >> _______________________________________________
> >> devel mailing list
> >> devel at rtems.org
> >> http://lists.rtems.org/mailman/listinfo/devel
> > _______________________________________________
> > devel mailing list
> > devel at rtems.org
> > http://lists.rtems.org/mailman/listinfo/devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rtems.org/pipermail/devel/attachments/20170402/7f52c3b2/attachment-0002.html>