RTEMS User Level Tracing?

[Isaac Gutekunst]

Sorry for the even longer delay between replies. I've been on vacation.

I think clarifying a few things will be helpful.

barectf takes a yaml file describing a bunch of data types that can be 
logged. The python tool will generate a c file that has function like
     log_my_data_type(MyDataType* data, Stream * stream);

and a CTF descriptor

Calling that function will serialize data, and send it down stream. 
Stream is conceptual trace output buffer. Barectf interacts with the 
stream through a a set of functions, like write_bytes, and 
is_stream_full? ( I made up these names for clarity).

The bytes written to stream will be in CTF format. Therefore, putting 
them in a file with the generated descriptor will allow you to read the 
stream with babeltrace or another tool that supports CTF files.

The README on the GitHub page is actually quite good at getting you up 
to speed quickly for areas where my explanation is lacking.

So I'm thinking barectuf could be stuck into RTEMS and used as the 
serialization mechanism for all trace facilities. For the trace linker, 
we could define a set of data types for function entry, function exit, 
etc. Rather than using printk to send out the trace, we would use barectf.


I've also spent some time looking at the lttng-live protocol. If RTEMS 
did adopt barectf, or another CTF serialization solution, it would open 
up the possibility of connecting to an running RTEMS application 
directly via TCP/IP using babeltrace or tracecompass and see traces in 
real time.

The lttng-live protocol is not overly complicated, and probably could be 
implemented in a few weeks. I've almost done with a rough proof of 
concept server that has a single hard coded stream and send pre-captured 
trace data.




On 01/03/16 18:49, Chris Johns wrote:
 > On 12/22/15 02:38, Isaac Gutekunst wrote:
 >>
 >>
 >> On 12/20/2015 08:02 PM, Chris Johns wrote:
 >>> On 19/12/2015 6:36 AM, isaac.gutekunst wrote:
 >>>>
 >>>> We're interested in trace some application data in our RTEMS project.
 >>>> Here's a bunch of background and thoughts:
 >>>>
 >>>> I've read the following pages:
 >>>>      * https://devel.rtems.org/wiki/Developer/Tracing
 >>>>      * https://devel.rtems.org/wiki/Developer/Tracing/Trace_Linker
 >>>>
 >>>> I think I have an understanding of the current system as documented.
 >>>> Tracing user defined data types seems to be missing. I'm looking for
 >>>> something more like barectf. Is there any such support available, 
or any
 >>>> plans to support it? I don't want to reinvent the wheel.
 >>>
 >>> User defined data is not directly support but can be added if you add a
 >>> custom generator. There is not a great deal of documentation about how
 >>> to do this. An example generator is:
 >>>
 >>> https://git.rtems.org/rtems-tools/tree/linkers/rtld-trace-buffer.ini
 >>>
 >> Cool. I'll take a closer look at this.
 >>> You could add anything specific you want. It is not a great 
solution but
 >>> tracing into data being passed by pointers is difficult.
 >>>
 >> Definitely. You quickly come to the great serialization problem, and are
 >> bound to reinvent the wheel a few times.
 >
 > Yes and SMP makes this even more complicated. We need to ensure we 
have a low overhead solution. Work in this area has been done by Joel 
and Jennifer.
 >
I haven't figured out what the Capture Engine does exactly. Does it 
simply buffer the printk output?

Apologies for not looking into the code myself. It looks like the 
Capture Engine outputs data in some format that babletrace can 
understand and convert into CTF.

I'm proposing that we integrate barectf into the Capture Engine (or 
elsewhere), so the Capture Engine directly outputs CTF. However, I may 
be confused about what's going on.

I'm presuming that the instrumented application has calls to functions 
inserted for entry and exit to the functions a user wants to trace. 
These functions generate some data, and then the Capture Engine stores 
this and makes it available later.

I'm unclear about what this data is, and how it gets to the capture 
engine. Where is the code for this located?
 >>> You would need to use the various options to get a copy of the trace
 >>> wrapper file and then get the generator to see the headers and various
 >>> data you want.
 >>>
 >>>> If there isn't support already, maybe some parts of the trace
 >>>> infrastructure can be reused, notably the backed for storing the data,
 >>>> and/or sending it out over the network. Are there plans to make the
 >>>> Trace Linker natively output CTF data?
 >>>
 >>> We would like to support CTF output. Getting this done would need
 >>> developer funding.
 >>
 >> We would like tracing for at least user defined data, and will be
 >> developing that functionality. If we could do so in a way that helps the
 >> existing RTEMS trace, we'd love to.
 >>
 >>>
 >>>> Perhaps barectf could be used for this purpose.
 >>>
 >>> I do not know what this. Do you have a reference?
 >>>
 >> https://github.com/efficios/barectf
 >>
 >
 > Thanks
 >
 >> It takes in a YAML file, and generates a C file with functions that
 >> allow tracing user defined structs. It will output those to a custom
 >> "backend" in serialized form. The documentation is pretty good. I've
 >> used in briefly in a bare metal PIC32 project with success.
 >>
 >> A potential plan of action:
 >> 1) Make a barectf backend for RTEMS. (This could re-use existing
 >> features). Probably package/make it available via RSB.
 >
 > Is this just for data or does it handle instrumenting functions? I 
have not looked at barectf.
 >
It's really just for serializing arbitrary structs into CTF data.
 >> 2) Update the trace linker to use it (I don't really know what this
 >> means just yet).
 >
 > Are 1) and 2) has the same thing or part of the same piece of work?
 >
 > The capture engine Joel and Jennifer updated supports SMP logging. 
Can this be used to do this?
 >

Again, maybe once I have a better understanding of the Capture Engine, 
I'll know what I'm actually proposing

 > My goal is to see all support integrated into the same trace 
framework so our users have a single place for trace support. The trace 
framework can have multiple parts, they just need to coexist in a single 
framework.
 >
 >> 3) Make a mechanism/ document a standard way to integrate trace with a
 >> user application. I'd imagine this could involve a RSB package, although
 >> it's rather simple (yaml file becomes a C and H file that needs to be
 >> linked against a 'backend').
 >
 > How does this relate to the RTEMS Trace linker and the function 
wrapping that it currently implements? Can the RTEMS Trace Linker be the 
"user interface" to bring all this together? This would then provide a 
single user interface for creating a traceable RTEMS executable.
 >
I believe so. This would mostly be pushing CTF serialization into RTEMS 
and updating the Trace Linker to insert calls to barectf generated 
functions to log data, instead of the existing printk calls.

User level tracing would just be generation of additional barectf trace 
functions, outputting data to streams managed by the Capture Engine (I 
think).
 >>
 >> 4) For the ARM architecture, using the ITM to output data could be very
 >> useful. This could fit somewhere into this picture.
 >>
 >
 > Transports should be a separate part. The capture engine should be 
creating buffers and a separate module manages the user interfacing and 
data extraction. Currently thee is just a cli. I am not across the ITM 
hardware and how you manager the user interactions.
 >
Using the ITM is potentially a little painful. It can essentially be 
thought of as a UART.

If the Capture Engine just creates buffers, it shouldn't be too hard to 
consume them in delayed realtime and send them out the ITM, or anything 
really.
 >> I would be doing most or all of the work here, unless of course anyone
 >> wants to help.
 >>
 >> I'm looking for ideas and suggestions. If there is some functionality of
 >> direction you'd like us to pursue, let me know.
 >>
 >
 > Sorry about the delayed response, I have been away for a few weeks.
 >
 > Chris



On 01/03/2016 06:49 PM, Chris Johns wrote:
> On 12/22/15 02:38, Isaac Gutekunst wrote:
>>
>>
>> On 12/20/2015 08:02 PM, Chris Johns wrote:
>>> On 19/12/2015 6:36 AM, isaac.gutekunst wrote:
>>>>
>>>> We're interested in trace some application data in our RTEMS project.
>>>> Here's a bunch of background and thoughts:
>>>>
>>>> I've read the following pages:
>>>>      * https://devel.rtems.org/wiki/Developer/Tracing
>>>>      * https://devel.rtems.org/wiki/Developer/Tracing/Trace_Linker
>>>>
>>>> I think I have an understanding of the current system as documented.
>>>> Tracing user defined data types seems to be missing. I'm looking for
>>>> something more like barectf. Is there any such support available, or
>>>> any
>>>> plans to support it? I don't want to reinvent the wheel.
>>>
>>> User defined data is not directly support but can be added if you add a
>>> custom generator. There is not a great deal of documentation about how
>>> to do this. An example generator is:
>>>
>>>   https://git.rtems.org/rtems-tools/tree/linkers/rtld-trace-buffer.ini
>>>
>> Cool. I'll take a closer look at this.
>>> You could add anything specific you want. It is not a great solution but
>>> tracing into data being passed by pointers is difficult.
>>>
>> Definitely. You quickly come to the great serialization problem, and are
>> bound to reinvent the wheel a few times.
>
> Yes and SMP makes this even more complicated. We need to ensure we have
> a low overhead solution. Work in this area has been done by Joel and
> Jennifer.
>
>>> You would need to use the various options to get a copy of the trace
>>> wrapper file and then get the generator to see the headers and various
>>> data you want.
>>>
>>>> If there isn't support already, maybe some parts of the trace
>>>> infrastructure can be reused, notably the backed for storing the data,
>>>> and/or sending it out over the network. Are there plans to make the
>>>> Trace Linker natively output CTF data?
>>>
>>> We would like to support CTF output. Getting this done would need
>>> developer funding.
>>
>> We would like tracing for at least user defined data, and will be
>> developing that functionality. If we could do so in a way that helps the
>> existing RTEMS trace, we'd love to.
>>
>>>
>>>> Perhaps barectf could be used for this purpose.
>>>
>>> I do not know what this. Do you have a reference?
>>>
>> https://github.com/efficios/barectf
>>
>
> Thanks
>
>> It takes in a YAML file, and generates a C file with functions that
>> allow tracing user defined structs. It will output those to a custom
>> "backend" in serialized form. The documentation is pretty good. I've
>> used in briefly in a bare metal PIC32 project with success.
>>
>> A potential plan of action:
>> 1) Make a barectf backend for RTEMS. (This could re-use existing
>> features). Probably package/make it available via RSB.
>
> Is this just for data or does it handle instrumenting functions? I have
> not looked at barectf.
>
>> 2) Update the trace linker to use it (I don't really know what this
>> means just yet).
>
> Are 1) and 2) has the same thing or part of the same piece of work?
>
> The capture engine Joel and Jennifer updated supports SMP logging. Can
> this be used to do this?
>
> My goal is to see all support integrated into the same trace framework
> so our users have a single place for trace support. The trace framework
> can have multiple parts, they just need to coexist in a single framework.
>
>> 3) Make a mechanism/ document a standard way to integrate trace with a
>> user application. I'd imagine this could involve a RSB package, although
>> it's rather simple (yaml file becomes a C and H file that needs to be
>> linked against a 'backend').
>
> How does this relate to the RTEMS Trace linker and the function wrapping
> that it currently implements? Can the RTEMS Trace Linker be the "user
> interface" to bring all this together? This would then provide a single
> user interface for creating a traceable RTEMS executable.
>
>>
>> 4) For the ARM architecture, using the ITM to output data could be very
>> useful. This could fit somewhere into this picture.
>>
>
> Transports should be a separate part. The capture engine should be
> creating buffers and a separate module manages the user interfacing and
> data extraction. Currently thee is just a cli. I am not across the ITM
> hardware and how you manager the user interactions.
>
>> I would be doing most or all of the work here, unless of course anyone
>> wants to help.
>>
>> I'm looking for ideas and suggestions. If there is some functionality of
>> direction you'd like us to pursue, let me know.
>>
>
> Sorry about the delayed response, I have been away for a few weeks.
>
> Chris