GSoC 2017 -- initial proof

Sun Apr 2 18:16:46 UTC 2017

Thanks Joel for the correction.

Regards,
Tanu Hari Dixit.

On Sun, Apr 2, 2017 at 11:23 PM, Joel Sherrill <joel at rtems.org> wrote:
>
>
> On Sun, Apr 2, 2017 at 12:33 PM, Tanu Hari Dixit <tokencolour at gmail.com>
> wrote:
>>
>> Hi Andy,
>>
>> Joel and Gedare pointed me to the following:
>>
>>
>> https://devel.rtems.org/wiki/Developer/Projects/Open/PythonCoverageReporting
>> https://devel.rtems.org/ticket/2261
>> http://kmiesowicz.blogspot.in/p/esa-socis-2014.html
>>
>> Last link is to the work done by a previous student.
>> Hope this helps.
>> My proposal does not take covoar into consideration. Though, I have
>> planned to produce xml reports from rtems-tester that can be
>> integrated into coverage testing. Also, I have planned to use this xml
>> data to get plots (for visualizing results) as a stretch goal (you can
>> look in the "Future Improvements" Section in my proposal).
>
>
> covoar directly produces the html and text reports we have now. My
> recollection is that it doesn't have to be touched and that the trick
> is invoking it on a more focused basis than Core vs Developmental.
> So invoke covoar from rtems-tester mulitple times on a single run
> of the tests rather than twice at most.
>
> Personally, I think Core is still a good description for the core of
> the tasking (e.g. score, rtems, sapi, and posix) and that each
> directory other than that should be individual. But that's my marketing
> view. I could easily be argued that for consistency, it should all
> be on a per directory basis.
>
> At some point, we might discuss covoar writing a neutral format
> and using Python to do more but that is pure conjecture. I have
> no idea what the intermediate format would be. Writing Sphinx
> output for reports would be desirable from covoar. :)
>
> --joel
>
>>
>>
>> Regards,
>> Tanu Hari Dixit.
>>
>> On Sun, Apr 2, 2017 at 6:50 PM, Gedare Bloom <gedare at rtems.org> wrote:
>> > Joel's e-mail client messed up my reply thread. ;) Andy, have a read
>> > through https://gedare.github.io/pdf/FelShe15A.pdf if you haven't
>> > found it already.
>> >
>> > Getting qemu-couverture to build via RSB and then integrating the
>> > coverage scripts into rtems-test, would be a good start to a project.
>> > There is a student (Tanu)oar dir who has proposed rtems-test
>> > improvements, so
>>
>> > you two would need to coordinate efforts for this piece to avoid too
>> > much duplicated work/waste. Improvements to the reports etc are also
>> > welcome, and then of course using the reports to guide code
>> > refactoring to improve coverage is somewhat open-ended.
>> >
>> > -Gedare
>> >
>> > On Sat, Apr 1, 2017 at 9:34 PM, Joel Sherrill <joel at rtems.org> wrote:
>> >>
>> >>
>> >> On Apr 1, 2017 7:59 PM, "Andy MacGregor" <amacgregor.2018 at comcast.net>
>> >> wrote:
>> >>
>> >> On 03/31/2017 09:52 PM, Gedare Bloom wrote:
>> >>
>> >> Don't forget the deadline is Monday to submit your Final
>> >> PDF and proof of enrollment. The "Final" PDF can be submitted multiple
>> >> times, so go ahead and submit it when you finish a first draft. Add
>> >> your draft proposal to the tracking table at
>> >> https://devel.rtems.org/wiki/GSoC/2017 also, to get some possible
>> >> feedback.
>> >>
>> >>
>> >> Thanks for the welcome! (I realize I am late to the GSOC application
>> >> table).
>> >>
>> >> I'm set on improving code coverage of the testsuite, but I found some
>> >> questions as I'm starting to look into how its done.
>> >>
>> >> The link on the wiki to the current code status is currently broken, so
>> >> I
>> >> tried to generate a sample coverage report for 4.12 using
>> >> ./do_coverage,
>> >> ./run_coverage and ./coverage_cron as suggested on the wiki here but
>> >> without
>> >> any luck. Is there an up to date coverage report available?
>> >> I modified the VERSIONS-COVERAGE file like the wiki said, and once that
>> >> didn't work dug around in the script but fix it myself quickly.
>> >>
>> >> Could updating these coverage analysis scripts be a viable component of
>> >> a
>> >> GSOC proposal?
>> >>
>> >>
>> >> Yes. In fact, the idea was to rework the coverage scripts in shell to
>> >> be
>> >> part of the rtems-tester package and in Python. There is some work on
>> >> this
>> >> by a previous student which you can find or one of us can dig out.
>> >> Starting
>> >> with it, making it work across multiple BSPs, independent of any hard
>> >> coded
>> >> paths, improving the output (we did not get far enough that I recall
>> >> worrying about the quality of the output), etc could be a fair amount
>> >> of
>> >> work.
>> >>
>> >> Also we have privately discussed switching the RSB qemu upstream source
>> >> to
>> >> GNATCoverage at GitHub. This is a coverage and embedded focused
>> >> downstream
>> >> which includes the coverage trace output I used for the qemu support in
>> >> covoar. I'm
>> >>
>> >> In addition, I'd like to actually make some improvements the code
>> >> coverage
>> >> itself.
>> >> The most recent information I could find was these bar graphs from
>> >> 2009.
>> >> If these graphs reflect the current coverage state, I think that
>> >> improving
>> >> coverage for the i386/pc386,
>> >> would be interesting because there is room for improvement in both the
>> >> Baseline and Developmental groups (if these coverage statistics are up
>> >> to
>> >> date).
>> >>
>> >>
>> >> It doesn't matter which BSP you choose for coverage since the code and
>> >> tests
>> >> are the same.
>> >>
>> >> This also reminds me of the other improvements we had wanted. We wanted
>> >> to
>> >> switch from Baseline and developmental to a more directory oriented
>> >> approach
>> >> for the reports. For example, the reports should be at a granularity of
>> >> say
>> >> the Dos file system or shell as reports. The way reports are generated
>> >> now
>> >> it results in a skewed view when a large body of code is added to the
>> >> developmental set. Adding the Dos filesystem resulted in a 15-20% drop
>> >> in
>> >> the coverage reported in the Developmental set. That is a bad way to
>> >> report
>> >> and misleading. It detracts from the directories which have near 100%
>> >> coverage.
>> >>
>> >>
>> >> Does anyone know of a good reason to choose another BSP over
>> >> i386/pc386?
>> >>
>> >>
>> >> Users fly the SPARC leon3 and not the pc386. :)
>> >>
>> >>
>> >> I've just posted a draft of my proposal to the GSOC page.
>> >> If there's any feedback I'll be sure to update my proposal in time.
>> >>
>> >>
>> >> Dinner time here. Hopefully my feedback makes sense and you can
>> >> incorporate
>> >> it.
>> >>
>> >>
>> >>
>> >>
>> >> -Andy MacGregor
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> _______________________________________________
>> >> devel mailing list
>> >> devel at rtems.org
>> >> http://lists.rtems.org/mailman/listinfo/devel
>> >>
>> >>
>> >>
>> >> _______________________________________________
>> >> devel mailing list
>> >> devel at rtems.org
>> >> http://lists.rtems.org/mailman/listinfo/devel
>> > _______________________________________________
>> > devel mailing list
>> > devel at rtems.org
>> > http://lists.rtems.org/mailman/listinfo/devel
>
>