[PATCH] Add low level event recording support

[Chris Johns]

On 23/1/19 6:13 pm, Sebastian Huber wrote:
> On 23/01/2019 07:49, Chris Johns wrote:
>> On 23/1/19 12:34 am, Sebastian Huber wrote:
>>> Hello Chris,
>>>
>>> On 20/12/2018 07:46, Sebastian Huber wrote:
>>>>> Sorry but I have no time to review this and consider it until next year.
>>>> No problem, take your time. I work on this since April this year from time to
>>>> time, so it can wait a couple of more weeks.
>>> had you time to look at this?
>>>
>> Sorry I have not.
>>
>> I understand there is a need for high performance tracing and welcome this work.
>> I have tended to use custom code to handle this, an example is
>> https://git.rtems.org/rtems/tree/cpukit/dev/i2c/xilinx-axi-i2c.c#n120.
> 
> You could use the record support of this patch set for this purpose.

Yes, I can see this.

>>> The focus of this new stuff is the recording of high frequency events on
>>> multiple processors. It doesn't deal with filtering or event generation.
>> Like the idea but I am confused on how what is being offered fits into what we
>> have. It seems to provide better performance but it also overlaps some of what
>> we have and is missing some other things. I cannot tell from my brief look if it
>> sits as a component in the existing RTEMS trace framework or it is completely
>> separate.
> 
> It closes a gap in the existing stuff, recording of high frequency events on a
> multiprocessor platform. For example in the output of the previous e-mail you see:
> 
> 6769.471385216:5:TCP_OUTPUT:7dd21000
> 6769.471385232:4:TCP_INPUT:7dd77100
> 
> These two events on different processors are separated by 16ns. It is impossible
> to get this independence with a recording facility which uses locks (atomic
> read-modify-write) or a global time source. You need per-processor data
> structures with per-processor synchronization and a per-processor time source
> for this. It is not complicated, it is just a ring buffer per processor.

Thanks for the detail.
>> I am OK with something new and better but we need to make sure what we offer is
>> consistent and makes sense to users. I am concerned users will become confused
>> if we have multiple approaches with separate code, set up, post processing and
>> documentation. I am fine if what we have is replaced and removed if your new
>> approach is to become a complete framework.
> 
> There is no one size fits all. This stuff just solves one problem and tries to
> do it well. You need other components that work together to get a good tracing
> solution for RTEMS. This is a major piece of work. Providing a documentation set
> along is a lot of work. Maybe we can use GSoC projects in this area.

I do not think documentation is allowed as GSoC projects. It can be part of a
development effort but not as a stand alone project.

>> Filtering and triggering as implemented is important in some cases to make sure
>> the overhead and transport processes does not become unstable but it costs at
>> run-time. If done property the overhead could be small. Manually inserting trace
>> points is problematic for some users and wrapping has limitations. I am not sure
>> we can have a single approach for all use cases.
>>
>>> Attached is a very simple example program to get the records via a TCP stream
>>> from the target. It outputs text like this:
>> How have you inserted the trace points on the target in this example? Again I am
>> missing the high level view. Is this in the detail of the other patch?
> 
> I tried to use the GNU ld --wrap, but this didn't work well since it wraps only
> undefined references. I had also a look at extending GNU ld to support also
> defined references, but this turned out to be quite difficult to me.

I am sure. I suspect we may have to look for other solutions such as DWARF.

> Some events are generated by user extensions. In the network stack I used code
> patches. The focus of this recording stuff is not the event generation, the
> filtering, the transport from A to B, visualization, etc. The focus is event
> recording of high frequency events on multiprocessor platforms. It is just one
> tiny building block that can be used in a tracing infrastructure.

This is great to hear. Should all the existing trace code be brought together in
libtrace?

>> A common transport approach would be nice for all pieces of the trace puzzle.
> 
> I think that transformation to standard formats should be done on a host
> computer and not the target.

I agree.

Chris