RTEMS User Level Tracing?

[Chris Johns]

On 22/01/2016 03:10, Isaac Gutekunst wrote:
> Sorry for the even longer delay between replies. I've been on vacation.

Thanks for getting back to us. I think I answered all the parts below. :)

>
> I think clarifying a few things will be helpful.
>
> barectf takes a yaml file describing a bunch of data types that can be
> logged. The python tool will generate a c file that has function like
>      log_my_data_type(MyDataType* data, Stream * stream);

Great.

>
> and a CTF descriptor
>
> Calling that function will serialize data, and send it down stream.
> Stream is conceptual trace output buffer. Barectf interacts with the
> stream through a a set of functions, like write_bytes, and
> is_stream_full? ( I made up these names for clarity).
>
> The bytes written to stream will be in CTF format. Therefore, putting
> them in a file with the generated descriptor will allow you to read the
> stream with babeltrace or another tool that supports CTF files.

Nice.

>
> The README on the GitHub page is actually quite good at getting you up
> to speed quickly for areas where my explanation is lacking.
>
> So I'm thinking barectuf could be stuck into RTEMS and used as the
> serialization mechanism for all trace facilities.

If this can be done it would be nice.

> For the trace linker,
> we could define a set of data types for function entry, function exit,
> etc. Rather than using printk to send out the trace, we would use barectf.

Yes. You can teach the RTEMS linker to wrap barectf and abstract the 
whole generation and integration process from the user. The toolkit has 
classes for running child processes on our supported hosts.

> I've also spent some time looking at the lttng-live protocol. If RTEMS
> did adopt barectf, or another CTF serialization solution, it would open
> up the possibility of connecting to an running RTEMS application
> directly via TCP/IP using babeltrace or tracecompass and see traces in
> real time.
>
> The lttng-live protocol is not overly complicated, and probably could be
> implemented in a few weeks. I've almost done with a rough proof of
> concept server that has a single hard coded stream and send pre-captured
> trace data.

Nice.

>
> On 01/03/16 18:49, Chris Johns wrote:
>  > On 12/22/15 02:38, Isaac Gutekunst wrote:
>  >>
>  >>
>  >> On 12/20/2015 08:02 PM, Chris Johns wrote:
>  >>> On 19/12/2015 6:36 AM, isaac.gutekunst wrote:
>  >>>>
>  >>>> We're interested in trace some application data in our RTEMS project.
>  >>>> Here's a bunch of background and thoughts:
>  >>>>
>  >>>> I've read the following pages:
>  >>>>      * https://devel.rtems.org/wiki/Developer/Tracing
>  >>>>      * https://devel.rtems.org/wiki/Developer/Tracing/Trace_Linker
>  >>>>
>  >>>> I think I have an understanding of the current system as documented.
>  >>>> Tracing user defined data types seems to be missing. I'm looking for
>  >>>> something more like barectf. Is there any such support available,
> or any
>  >>>> plans to support it? I don't want to reinvent the wheel.
>  >>>
>  >>> User defined data is not directly support but can be added if you
> add a
>  >>> custom generator. There is not a great deal of documentation about how
>  >>> to do this. An example generator is:
>  >>>
>  >>> https://git.rtems.org/rtems-tools/tree/linkers/rtld-trace-buffer.ini
>  >>>
>  >> Cool. I'll take a closer look at this.
>  >>> You could add anything specific you want. It is not a great
> solution but
>  >>> tracing into data being passed by pointers is difficult.
>  >>>
>  >> Definitely. You quickly come to the great serialization problem, and
> are
>  >> bound to reinvent the wheel a few times.
>  >
>  > Yes and SMP makes this even more complicated. We need to ensure we
> have a low overhead solution. Work in this area has been done by Joel
> and Jennifer.
>  >
> I haven't figured out what the Capture Engine does exactly. Does it
> simply buffer the printk output?
>

It is the low level capture and buffering part. It takes data and 
buffers it in a low overhead SMP safe manner. The original intention was 
to separate the capture from the transport of the target so we did not 
reinvent the way that happens for each new trace system that people 
invent. This did not go past a CLI and prints.

It also manages filtering of events and triggers and it contain hooks to 
the kernel extensions to capture threading events. I needed some form of 
data to process and this was available and simple to implement. This 
tends to make the capture engine look confusing given the way the trace 
linker works and things evolved.

The original intention was to look like a digital oscilloscope. You 
enable the signals you want to look at, define the trigger and then 
prime the device and wait. When the trigger is hit you will get a 
capture of the signals around the trigger.

One of the complexity we need to manage is the amount of buffering and 
how we get it off board for further processing. I am looking at this 
from the lowest common denominator which means small memory and no 
external interfaces for runtime exporting off board. Therefore being 
able to set a trigger on the point of interest means we localise the 
data we want to capture. A simple export method is reading a block of 
memory using JTAG or GDB.

> Apologies for not looking into the code myself. It looks like the
> Capture Engine outputs data in some format that babletrace can
> understand and convert into CTF.
>
> I'm proposing that we integrate barectf into the Capture Engine (or
> elsewhere), so the Capture Engine directly outputs CTF. However, I may
> be confused about what's going on.
>
> I'm presuming that the instrumented application has calls to functions
> inserted for entry and exit to the functions a user wants to trace.
> These functions generate some data, and then the Capture Engine stores
> this and makes it available later.

Yes. The RTEMS linker and CTF bring new things and I am not fixed on how 
the Capture Engine evolves and what happens to that code. There are 
worth while parts such as SMP buffering, plus other parts could be moved 
to the RTEMS linker and managed there in a simpler way. It is confusing 
having a few different methods of capture happening.

>
> I'm unclear about what this data is, and how it gets to the capture
> engine. Where is the code for this located?
>  >>> You would need to use the various options to get a copy of the trace
>  >>> wrapper file and then get the generator to see the headers and various
>  >>> data you want.
>  >>>
>  >>>> If there isn't support already, maybe some parts of the trace
>  >>>> infrastructure can be reused, notably the backed for storing the
> data,
>  >>>> and/or sending it out over the network. Are there plans to make the
>  >>>> Trace Linker natively output CTF data?
>  >>>
>  >>> We would like to support CTF output. Getting this done would need
>  >>> developer funding.
>  >>
>  >> We would like tracing for at least user defined data, and will be
>  >> developing that functionality. If we could do so in a way that helps
> the
>  >> existing RTEMS trace, we'd love to.
>  >>
>  >>>
>  >>>> Perhaps barectf could be used for this purpose.
>  >>>
>  >>> I do not know what this. Do you have a reference?
>  >>>
>  >> https://github.com/efficios/barectf
>  >>
>  >
>  > Thanks
>  >
>  >> It takes in a YAML file, and generates a C file with functions that
>  >> allow tracing user defined structs. It will output those to a custom
>  >> "backend" in serialized form. The documentation is pretty good. I've
>  >> used in briefly in a bare metal PIC32 project with success.
>  >>
>  >> A potential plan of action:
>  >> 1) Make a barectf backend for RTEMS. (This could re-use existing
>  >> features). Probably package/make it available via RSB.
>  >
>  > Is this just for data or does it handle instrumenting functions? I
> have not looked at barectf.
>  >
> It's really just for serializing arbitrary structs into CTF data.
>  >> 2) Update the trace linker to use it (I don't really know what this
>  >> means just yet).
>  >
>  > Are 1) and 2) has the same thing or part of the same piece of work?
>  >
>  > The capture engine Joel and Jennifer updated supports SMP logging.
> Can this be used to do this?
>  >
>
> Again, maybe once I have a better understanding of the Capture Engine,
> I'll know what I'm actually proposing
>

Please have a look and see what you think is useful.

>  > My goal is to see all support integrated into the same trace
> framework so our users have a single place for trace support. The trace
> framework can have multiple parts, they just need to coexist in a single
> framework.
>  >
>  >> 3) Make a mechanism/ document a standard way to integrate trace with a
>  >> user application. I'd imagine this could involve a RSB package,
> although
>  >> it's rather simple (yaml file becomes a C and H file that needs to be
>  >> linked against a 'backend').
>  >
>  > How does this relate to the RTEMS Trace linker and the function
> wrapping that it currently implements? Can the RTEMS Trace Linker be the
> "user interface" to bring all this together? This would then provide a
> single user interface for creating a traceable RTEMS executable.
>  >
> I believe so. This would mostly be pushing CTF serialization into RTEMS
> and updating the Trace Linker to insert calls to barectf generated
> functions to log data, instead of the existing printk calls.

Can the CTF capture the calls and so the whole trace? This means we have 
a CTF generator and nothing to do with printk. The printk method of 
tracing was a simple way for me to test the RTEMS trace linker. There is 
a more complex buffering system with CLI.

> User level tracing would just be generation of additional barectf trace
> functions, outputting data to streams managed by the Capture Engine (I
> think).

Or what ever you turn the Capture Engine code into. I am fine with that 
code becoming a backend target module to support CTF tracing. If this 
code can also provide a common set of ways to get data of a target that 
would be excellent.

>  >>
>  >> 4) For the ARM architecture, using the ITM to output data could be very
>  >> useful. This could fit somewhere into this picture.
>  >>
>  >
>  > Transports should be a separate part. The capture engine should be
> creating buffers and a separate module manages the user interfacing and
> data extraction. Currently thee is just a cli. I am not across the ITM
> hardware and how you manager the user interactions.
>  >
> Using the ITM is potentially a little painful. It can essentially be
> thought of as a UART.

Ok.

>
> If the Capture Engine just creates buffers, it shouldn't be too hard to
> consume them in delayed realtime and send them out the ITM, or anything
> really.

Yes this is a good design. Users hook in how it comes of board. Some may 
fill the buffer and then drop any extra data then use JTAG to suck the 
data out.

I see 2 use cases, the streaming users who are monitoring a system and 
looking at data in real-time and users looking to fix a real-time bug 
and want a trace of what happens in a specific area of code. The RTEMS 
Trace Linker's buffering generator with the shell command is sort of 
doing the second use case. You run the code, a trigger fires and you get 
a buffer of data. You use a shell command to write it off board or to disk.

>  >> I would be doing most or all of the work here, unless of course anyone
>  >> wants to help.
>  >>
>  >> I'm looking for ideas and suggestions. If there is some
> functionality of
>  >> direction you'd like us to pursue, let me know.
>  >>
>  >
>  > Sorry about the delayed response, I have been away for a few weeks.
>  >
>  > Chris
>
>
>
> On 01/03/2016 06:49 PM, Chris Johns wrote:
>> On 12/22/15 02:38, Isaac Gutekunst wrote:
>>>
>>>
>>> On 12/20/2015 08:02 PM, Chris Johns wrote:
>>>> On 19/12/2015 6:36 AM, isaac.gutekunst wrote:
>>>>>
>>>>> We're interested in trace some application data in our RTEMS project.
>>>>> Here's a bunch of background and thoughts:
>>>>>
>>>>> I've read the following pages:
>>>>>      * https://devel.rtems.org/wiki/Developer/Tracing
>>>>>      * https://devel.rtems.org/wiki/Developer/Tracing/Trace_Linker
>>>>>
>>>>> I think I have an understanding of the current system as documented.
>>>>> Tracing user defined data types seems to be missing. I'm looking for
>>>>> something more like barectf. Is there any such support available, or
>>>>> any
>>>>> plans to support it? I don't want to reinvent the wheel.
>>>>
>>>> User defined data is not directly support but can be added if you add a
>>>> custom generator. There is not a great deal of documentation about how
>>>> to do this. An example generator is:
>>>>
>>>>   https://git.rtems.org/rtems-tools/tree/linkers/rtld-trace-buffer.ini
>>>>
>>> Cool. I'll take a closer look at this.
>>>> You could add anything specific you want. It is not a great solution
>>>> but
>>>> tracing into data being passed by pointers is difficult.
>>>>
>>> Definitely. You quickly come to the great serialization problem, and are
>>> bound to reinvent the wheel a few times.
>>
>> Yes and SMP makes this even more complicated. We need to ensure we have
>> a low overhead solution. Work in this area has been done by Joel and
>> Jennifer.
>>
>>>> You would need to use the various options to get a copy of the trace
>>>> wrapper file and then get the generator to see the headers and various
>>>> data you want.
>>>>
>>>>> If there isn't support already, maybe some parts of the trace
>>>>> infrastructure can be reused, notably the backed for storing the data,
>>>>> and/or sending it out over the network. Are there plans to make the
>>>>> Trace Linker natively output CTF data?
>>>>
>>>> We would like to support CTF output. Getting this done would need
>>>> developer funding.
>>>
>>> We would like tracing for at least user defined data, and will be
>>> developing that functionality. If we could do so in a way that helps the
>>> existing RTEMS trace, we'd love to.
>>>
>>>>
>>>>> Perhaps barectf could be used for this purpose.
>>>>
>>>> I do not know what this. Do you have a reference?
>>>>
>>> https://github.com/efficios/barectf
>>>
>>
>> Thanks
>>
>>> It takes in a YAML file, and generates a C file with functions that
>>> allow tracing user defined structs. It will output those to a custom
>>> "backend" in serialized form. The documentation is pretty good. I've
>>> used in briefly in a bare metal PIC32 project with success.
>>>
>>> A potential plan of action:
>>> 1) Make a barectf backend for RTEMS. (This could re-use existing
>>> features). Probably package/make it available via RSB.
>>
>> Is this just for data or does it handle instrumenting functions? I have
>> not looked at barectf.
>>
>>> 2) Update the trace linker to use it (I don't really know what this
>>> means just yet).
>>
>> Are 1) and 2) has the same thing or part of the same piece of work?
>>
>> The capture engine Joel and Jennifer updated supports SMP logging. Can
>> this be used to do this?
>>
>> My goal is to see all support integrated into the same trace framework
>> so our users have a single place for trace support. The trace framework
>> can have multiple parts, they just need to coexist in a single framework.
>>
>>> 3) Make a mechanism/ document a standard way to integrate trace with a
>>> user application. I'd imagine this could involve a RSB package, although
>>> it's rather simple (yaml file becomes a C and H file that needs to be
>>> linked against a 'backend').
>>
>> How does this relate to the RTEMS Trace linker and the function wrapping
>> that it currently implements? Can the RTEMS Trace Linker be the "user
>> interface" to bring all this together? This would then provide a single
>> user interface for creating a traceable RTEMS executable.
>>
>>>
>>> 4) For the ARM architecture, using the ITM to output data could be very
>>> useful. This could fit somewhere into this picture.
>>>
>>
>> Transports should be a separate part. The capture engine should be
>> creating buffers and a separate module manages the user interfacing and
>> data extraction. Currently thee is just a cli. I am not across the ITM
>> hardware and how you manager the user interactions.
>>
>>> I would be doing most or all of the work here, unless of course anyone
>>> wants to help.
>>>
>>> I'm looking for ideas and suggestions. If there is some functionality of
>>> direction you'd like us to pursue, let me know.
>>>
>>
>> Sorry about the delayed response, I have been away for a few weeks.
>>
>> Chris