[RTEMS Project] #2920: Improve Coverage Analysis Toolset (was: Improve Coverage Analysis)

[RTEMS trac]

#2920: Improve Coverage Analysis Toolset
--------------------------+----------------------------
 Reporter:  Gedare        |       Owner:  Joel Sherrill
     Type:  project       |      Status:  assigned
 Priority:  normal        |   Milestone:  Indefinite
Component:  General       |     Version:  4.11
 Severity:  normal        |  Resolution:
 Keywords:  SoC, testing  |
--------------------------+----------------------------
Description changed by Joel Sherrill:

Old description:

> = Improve Coverage Analysis =
>
> [[PageOutline(1-5, Contents, inline)]]
>
> == Mentors ==
> Joel Sherrill
>
> == Students ==
> T.B.D.
>
> = Introduction =
>
> Improve [wiki:TBR/UserManual/RTEMS_Coverage_Analysis  Test Coverage
> Analysis] by adding more test cases, reworking code to eliminate the need
> for additional test cases, eliminating dead code, and reaching for 100%
> coverage helps reduce the likelihood of new and recurrent bugs.
>
> = Objective =
>
> Increase the percentage of code coverage achieved by the testsuite.
>
> = Requirements =
>
> Ability to [wiki:TBR/UserManual/RTEMS_Coverage_Analysis#HowitisDoneNow
> generate coverage reports] and to read/write C code.

New description:

 = Improve Coverage Analysis =

 [[PageOutline(1-5, Contents, inline)]]

 == Mentors ==
 Joel Sherrill

 == Students ==
 T.B.D.

 = Introduction =

 IThe in-house rtems coverage analysis tool 'covoar' works with RTEMS
 Tester framework to generate coverage reports on a user defined set of
 symbol libraries. This all resides in rtems-tools. It is close to
 being merged but there are still a few blockers and there is also a
 few extra improvements that would be nice to implement afterwards.

 Using this toolset to analyze the code coverage of RTEMS and improving
 that measurement is an ongoing effort with a series of incremental efforts
 to either add test cases or rework source code to improve coverage.
 Individual tickets may be filed for specific areas that need coverage
 improvement.

 = Objective =

 This project is to improve the tools used to perform the analysis and
 generate reports. These tools are part of the rtems-tester repository.

 * Remove the use of external tools that are not portable across
 platforms (Linux, BSD, Windows):

 * nm is used to generate a list of symbols in a file that is
 referenced as the symbols of interest for that set. nm is not portable
 and so must be removed. The fix for this is use rtemstoolkit and
 generate the list of symbols from the ELF files.

 * There is also a use of addr2line piped to dos2unix to find the
 source lines which match the objdump instruction lines. Again we want
 to limit the use of external tools that are not portable across
 platforms.

 Improve clean-up procedures:

 * The trace files that are generated by QEMU as each test is run in
 RTEMS Tester need to stay for later use by covoar which runs after the
 testsuite is finished. They are currently cleared up by a try finally
 statement in python, which works for Ctrl+C exit but probably not
 SIGTERM. This cannot be handled the same way as the other tempfiles as
 they are not generated by covoar. These files have a .cov ending and
 are currently generated beside the executable they match in the build
 tree.

 * covoar needs to detect target architecture internally (e.g
 sparc-rtems4.12), this can be done with get_exec_prefix() from
 rld-cc.h in rtemstoolkit.

 Further Improvements:

 * Convert the symbol_sets.cfg file to the INI format. The
 symbol_sets.cfg file is currently in a non standard format, created
 just for it. This file contains the sets of libraries of symbols and
 groups them together to be treated as a set to see what percentage of
 them the testsuite covers.

 * Generate XML reports. Currently the tools generate html and text
 reports.

 * Generate Gcov reports. There is Gcov support in covoar that is close
 to working but the gcov output generated is not trusted. The fix for
 this would involve vaildating the Gcov output to make sure it is in a
 format that the Gcov program accepts.

 * Combine how-to documentation below with existing covoar docs to make
 a ReST format doc thats integrated with the main RTEMS Tester docs.

 * Ensure the gcov output is correct and use that as the basis for
 generating reports using tools like gcov and lcov.

 * Find a way to do MCDC analysis. This is a longer term objective which
 may require improvements to gcc.

 = Requirements =

 * Covoar and rtemstoolkit (C++)
 * RTEMS Tester framework (Python)
 * Understand INI format for config files.
 * Reports (HTML and text currently, add XML)

--

--
Ticket URL: <http://devel.rtems.org/ticket/2920#comment:2>
RTEMS Project <http://www.rtems.org/>
RTEMS Project