GSoC 2017 -- initial proof

[Chris Johns]

On 03/04/2017 03:53, Joel Sherrill wrote:
> On Sun, Apr 2, 2017 at 12:33 PM, Tanu Hari Dixit <tokencolour at gmail.com
> <mailto:tokencolour at gmail.com>> wrote:
>
>     Hi Andy,
>
>     Joel and Gedare pointed me to the following:
>
>     https://devel.rtems.org/wiki/Developer/Projects/Open/PythonCoverageReporting
>     <https://devel.rtems.org/wiki/Developer/Projects/Open/PythonCoverageReporting>
>     https://devel.rtems.org/ticket/2261
>     <https://devel.rtems.org/ticket/2261>
>     http://kmiesowicz.blogspot.in/p/esa-socis-2014.html
>     <http://kmiesowicz.blogspot.in/p/esa-socis-2014.html>
>
>     Last link is to the work done by a previous student.
>     Hope this helps.
>     My proposal does not take covoar into consideration. Though, I have
>     planned to produce xml reports from rtems-tester that can be
>     integrated into coverage testing. Also, I have planned to use this xml
>     data to get plots (for visualizing results) as a stretch goal (you can
>     look in the "Future Improvements" Section in my proposal).
>
>
> covoar directly produces the html and text reports we have now. My
> recollection is that it doesn't have to be touched and that the trick
> is invoking it on a more focused basis than Core vs Developmental.

I think covoar does need some refactoring. There are things we can do 
now the tool resides in the rtems-tools repo that makes the tool 
portable, more stable, faster and better.

First is the use of external tools being invoked via the `system` libc 
call to collect information that can be done directly via the 
rtemstoolkit. The makes the tool portable because the toolkit runs on 
the Windows and Unix and it is based on standards based file formats, ie 
ELF. As an example to highlight changes that need to happen take this 
function:

https://git.rtems.org/rtems-tools/tree/tester/covoar/ObjdumpProcessor.cc#n233

a) The symbols data can be loaded directly from the ELF file and it is 
quick and fully portable. See 
https://git.rtems.org/rtems-tools/tree/linkers/rtems-syms.cpp#n463 as an 
example.

b) The `system` call is not portable, eg Windows, because the command 
expects a Unix shell, pipes and `sed`. We have a requirement to support 
Windows. See 
https://git.rtems.org/rtems-tools/tree/rtemstoolkit/rld-process.h#n235 
for executing on all hosts.

c) Are the generated files cleaned up on exceptions, crashes etc. See 
https://git.rtems.org/rtems-tools/tree/rtemstoolkit/rld-process.h#n54 
and the process based support to handle temporary files and output.

I also see the running of the simulator as the function of `rtems-test` 
as it handles the jobs and manage the state of the simulator and any 
files created. It would hand the trace data to be processed to covoar 
and get back the report, ie XML data, then clean up.

> So invoke covoar from rtems-tester mulitple times on a single run
> of the tests rather than twice at most.

Sorry, I do not understand this. Is rtems-tester the repo or is this 
rtems-test the command?

> Personally, I think Core is still a good description for the core of
> the tasking (e.g. score, rtems, sapi, and posix) and that each
> directory other than that should be individual. But that's my marketing
> view. I could easily be argued that for consistency, it should all
> be on a per directory basis.

I wonder if ELF notes would help by letting us set an attribute or a few 
attributes for each function and this guides how we handle them. It 
would be nice if these are implemented close to the actual function's 
source. As we have direct access to the ELF file we can extract these 
notes and so the attributes and we can then directly map what is 
happening. For example we can have attributes (ie EFL notes) such as 
`rtems/score/cpu`, `rtems/score/threads`, and `rtems/score/mutex` etc 
and you could say cover `rtems` or `rtems/score` etc. And if we also set 
the standard the code conforms to such as `pthread_create` with the 
standard could we end up with a coverage map based on the standard? 
Using ELF notes means we do not have to manage the process by the 
position of the source in the source tree.

> At some point, we might discuss covoar writing a neutral format
> and using Python to do more but that is pure conjecture.

I think this is real and important and not conjecture. The tool 
generating HTML directly conflicts with the proposed parent wrapper, 
`rtems-test`. The `rtems-test` tool's requirement is to export data in a 
standard format like XML that can be imported in to a reporting tool to 
generate HTML etc. A user can then decide if the HTML, ReST, SQL or 
whatever meets their needs or they can take the XML and incorporate it 
into their own workflow that meets their requirements. I would like to 
see covar reports be XML and writing XML does not need a special 
library, it is just print statements. Have a look at the RSB's XML 
writer Sebastian added, 
https://git.rtems.org/rtems-source-builder/tree/source-builder/sb/reports.py#n417. 
We can use python to import the XML and generate the reports as it comes 
with suitable XML support.

> I have
> no idea what the intermediate format would be. Writing Sphinx
> output for reports would be desirable from covoar. :)

XML.

Chris