[RTEMS Project] #2920: Improve Coverage Analysis Toolset

[RTEMS trac]

#2920: Improve Coverage Analysis Toolset
--------------------------+----------------------------
 Reporter:  Gedare Bloom  |       Owner:  Joel Sherrill
     Type:  project       |      Status:  assigned
 Priority:  normal        |   Milestone:  Indefinite
Component:  tool          |     Version:
 Severity:  normal        |  Resolution:
 Keywords:  SoC, testing  |  Blocked By:
 Blocking:                |
--------------------------+----------------------------
Description changed by Gedare Bloom:

Old description:

> = Improve Coverage Analysis =
>
> [[PageOutline(1-5, Contents, inline)]]
>
> == Mentors ==
> Joel Sherrill
>
> == Students ==
> T.B.D.
>
> = Introduction =
>
> The in-house rtems coverage analysis tool 'covoar' works with RTEMS
> Tester framework to generate coverage reports on a user defined set of
> symbol libraries. This all resides in rtems-tools. It is close to being
> merged but there are still a few blockers and improvements that would be
> nice to implement afterwards.
>
> Using this toolset to analyze the code coverage of RTEMS and improving
> that measurement is an ongoing effort with a series of incremental
> efforts to either add test cases or rework source code to improve
> coverage. Individual tickets may be filed for specific areas that need
> coverage improvement.
>
> = Objective =
>
> This project is to improve the tools used to perform coverage analysis
> and generate reports. These tools are part of the rtems-tools repository.
>
> = Prerequisites =
>
> * Ability to read/write C++ code with OOP.
> * Ability to read/write Python code with OOP.
> * Ability to debug C++ code.
> * Ability to compile and run covoar.
> * Ability to run RTEMS Tester (rtems-tools.git/tester).
> * Knowledge of INI format for configuration files.
> * Knowledge of HTML and XML.
>
> = Tasks =
>
> Some tasks have been captured as tickets:
> * #3515
> * #3462
> * #3441
> * #3851
> * #3852
>
> Other tasks include:
>
> * Remove the use of external tools that are not portable across
> platforms (Linux, BSD, Windows):
>
> * ~~nm is used to generate a list of symbols in a file that is
> referenced as the symbols of interest for that set. nm is not portable
> and so must be removed. The fix for this is use rtemstoolkit and generate
> the list of symbols from the ELF files.~~
>
> * addr2line is piped to dos2unix to find the source lines which match the
> objdump instruction lines. Again we want to limit the use of external
> tools that are not portable across platforms.
>
> Improve clean-up procedures:
>
> * The trace files that are generated by QEMU as each test is run in
> RTEMS Tester need to stay for later use by covoar which runs after the
> testsuite is finished. They are currently cleared up by a try finally
> statement in python, which works for Ctrl+C exit but probably not
> SIGTERM. This cannot be handled the same way as the other tempfiles as
> they are not generated by covoar. These files have a .cov ending and are
> currently generated beside the executable they match in the build tree.
>
> * covoar needs to detect target architecture internally (e.g.,
> sparc-rtems4.11), this can be done with get_exec_prefix() from
> rld-cc.h in rtemstoolkit.
>
> Further Improvements:
>
> * Generate XML reports. Currently the tools generate html and text
> reports.
>
> * Generate Gcov reports. There is Gcov support in covoar that is close to
> working but the gcov output generated is not trusted. The fix for this
> would involve vaildating the Gcov output to make sure it is in a format
> that the Gcov program accepts.
>
> * Combine how-to documentation below with existing covoar docs to make a
> ReST format doc thats integrated with the main RTEMS Tester docs.
>
> * Ensure the gcov output is correct and use that as the basis for
> generating reports using tools like gcov and lcov.
>
> * Find a way to do MCDC analysis. This is a longer term objective which
> may require improvements to gcc.

New description:

 = Improve Coverage Analysis =

 [[PageOutline(1-5, Contents, inline)]]

 == Mentors ==
 Joel Sherrill

 == Students ==
 T.B.D.

 = Introduction =

 The in-house rtems coverage analysis tool 'covoar' works with RTEMS Tester
 framework to generate coverage reports on a user defined set of symbol
 libraries. This all resides in rtems-tools. It is close to being merged
 but there are still a few blockers and improvements that would be nice to
 implement afterwards.

 Using this toolset to analyze the code coverage of RTEMS and improving
 that measurement is an ongoing effort with a series of incremental efforts
 to either add test cases or rework source code to improve coverage.
 Individual tickets may be filed for specific areas that need coverage
 improvement.

 = Objective =

 This project is to improve the tools used to perform coverage analysis and
 generate reports. These tools are part of the rtems-tools repository.

 = Prerequisites =

 * Ability to read/write C++ code with OOP.
 * Ability to read/write Python code with OOP.
 * Ability to debug C++ code.
 * Ability to compile and run covoar.
 * Ability to run RTEMS Tester (rtems-tools.git/tester).
 * Knowledge of INI format for configuration files.
 * Knowledge of HTML and XML.

 = Tasks =

 Some tasks have been captured as tickets:
 * #3515
 * #3462
 * #3441
 * #3851
 * #3852

 Other tasks include:

 * Remove the use of external tools that are not portable across
 platforms (Linux, BSD, Windows):

 * ~~nm is used to generate a list of symbols in a file that is referenced
 as the symbols of interest for that set. nm is not portable and so must be
 removed. The fix for this is use rtemstoolkit and generate the list of
 symbols from the ELF files.~~

 * addr2line is piped to dos2unix to find the source lines which match the
 objdump instruction lines. Again we want to limit the use of external
 tools that are not portable across platforms.

 Improve clean-up procedures:

 * The trace files that are generated by QEMU as each test is run in
 RTEMS Tester need to stay for later use by covoar which runs after the
 testsuite is finished. They are currently cleared up by a try finally
 statement in python, which works for Ctrl+C exit but probably not SIGTERM.
 This cannot be handled the same way as the other tempfiles as they are not
 generated by covoar. These files have a .cov ending and are currently
 generated beside the executable they match in the build tree.

 * covoar needs to detect target architecture internally (e.g.,
 sparc-rtems4.11), this can be done with get_exec_prefix() from
 rld-cc.h in rtemstoolkit.

 Further Improvements:

 * Generate XML reports. Currently the tools generate html and text
 reports.

 * Generate Gcov reports. There is Gcov support in covoar that is close to
 working but the gcov output generated is not trusted. The fix for this
 would involve vaildating the Gcov output to make sure it is in a format
 that the Gcov program accepts.

 * Combine how-to documentation below with existing covoar docs to make a
 ReST format doc thats integrated with the main RTEMS Tester docs.

 * Ensure the gcov output is correct and use that as the basis for
 generating reports using tools like gcov and lcov.

 * Find a way to do MCDC analysis. This is a longer term objective which
 may require improvements to gcc.

--

--
Ticket URL: <http://devel.rtems.org/ticket/2920#comment:6>
RTEMS Project <http://www.rtems.org/>
RTEMS Project