tracing tools

[Chris Johns]

On 4/11/11 3:34 AM, Xavier Raynaud wrote:
>
> I just started to investigate around trace tools available for RTEMS.
>

Great.

> Of course, I had a look at
> http://www.rtems.org/wiki/index.php/RTEMS_Trace_Tool (and
> http://code.google.com/p/rtems-tracing), but the status of this tool is
> not clear (and the wiki page said to contact Chris Johns for further
> details - here I am).
>
> My questions are:
> - What's the status of this tool ?

We have a number of parts that show the approach is sound and will work. 
The remainder of the work is to get the code together, fix bugs, 
performance issues or design holes, and to bring it up to the latest 
RTEMS code and integrate it into the HEAD.

The trace tool can serve a couple of needed functions. The first is a 
debug aid to see why something happens. Your target deadlocks. You have 
a few task lists, the various objects but what is of interesting is the 
sequence of event that lead to this. The second function is to verify 
your code. You run various tests of pieces of your code and you verify 
the input and out put data matches. This last need drives a basic 
requirement of being able to take an executable and trace it.

> - How it works ? As far as I understand, it's an instrumentation at
> source level, with function wrapping. Am I right ?

There are 3 parts (I think; it has been a while):

1. Configuration, definition and post link processing.
2. Target runtime and capture.
3. Post processing and analysis.

The configuration, definition and post link processing is first a 
definition of the functions you wish to trace and taking that definition 
to create a suitable set of wrapper functions to link against a known 
working executable RTEMS image. The approach is to use the GNU linker's 
wrap option. The ld man page has the details. The idea is to rename a 
function of interest to another name and then provide a function with 
the same name and finger print that captures data on the way in and/or 
way out. The ld wrap option does the rename and we do the rest. We read 
the debug data from the ELF file to determine the function signature to 
allow us to generate a suitable wrapper function. The requirement is to 
trace a existing executable with out changing any of the code or 
libraries used. We can instrument the code with wrapper functions and 
runtime trace support just not change the code being traced. This means 
we may not have access to the headers and compile time defines to know 
the exact signature of a function therefore the debug data is used. The 
wrapper function we generate needs to integrate into the run-time side 
of the code present on the target. The types of things required is 
runtime control for tracing a specific function's entry and exit in an 
efficient way.

The second part of the puzzle is the target runtime support and data 
capture. We need code on the target to manage the trace process. You may 
wish to instrument a number of functions how-ever your debugging or 
tracing may end up with only a few functions of interest being traced. 
To support this we have runtime control of the entry and exit tracing of 
every instrumented function. We also need a means of getting the data 
off the target or saved which is fast enough to handle the amount of 
data that may be generated. The capture engine in the current code base 
is an working example of this. It was features such as priority windows, 
triggers and trace flags. The tracing needs to be interrupt safe and now 
SMP safe.

Finally we need a suitable way of reporting the results that is useful 
and meaningful. This includes a flexible way to extract the data from 
the target.

As I said we have a number of these parts in various states of 
development. What we need is someone to pick these up and complete the 
work or to get the work completed via a support contract or contracts.

Chris