[PATCH] Add low level event recording support

[Sebastian Huber]

On 20/12/2018 01:46, Chris Johns wrote:
> I have read the ticket and not the patch.
>
> On 20/12/2018 08:00, Joel Sherrill wrote:
>> How does this relate.to <http://relate.to> the existing infrastructure for the
>> capture engine? It seems duplicative.
> Yes it is confusing. With this change we would have 3 possible trace systems,
> the capture engine, rtems-tld and this. I would like just one way to trace a
> live system.

This patch just adds per-processor ring buffers to record events. The 
capture engine does the same and in addition has support for triggers 
and filters. The rtems-tld generates trace events, it could use the 
stuff from this patch to record them. This was actually the original use 
case for this work which I started some months ago for the tracing GSoC 
project.

>
> I am OK with the capture engine being replaced with something new and better
> that provides all the functionality of the capture engine, ie triggers, floors
> etc. The ticket does not state if this is what will happen. It would be good if
> this can be clarified. Filtering at runtime is an important performance
> consideration.

Event recording is a separate problem from filtering and triggering 
events. This patch set only addresses the recording. It tries to do this 
as efficient as possible to be able to record high frequency events. It 
intentionally does not deal with filtering and triggering. There is a 
user extension to generate thread events, but this is just a special 
user of this event recoding.

>
> It is also not clear to me why the capture engine cannot be improved to have the
> capture method presented here integrated. Sure, currently it does not have the
> low level performance because of the locks used but it is not clear to me why
> this code cannot be changed?

The locks could be removed, the real problem are the record items.

/**
  * @brief The 32-bit format record item.
  */
typedef struct {
   uint32_t event;
   uint32_t data;
} rtems_record_item_32;

/**
  * @brief A capture timestamp.
  *
  * This is a nanosecond capture timestamp
  */
typedef uint64_t rtems_capture_time;

/*
  * @brief Capture record.
  *
  * This is a record that is written into
  * the buffer. The events includes the priority of the task
  * at the time of the context switch.
  */
typedef struct rtems_capture_record
{
   size_t             size;
   uint32_t           events;
   rtems_id           task_id;
   rtems_capture_time time;
} RTEMS_PACKED rtems_capture_record;

So, this is 8 bytes vs. 20 bytes (this is 250% more and a big deal for 
high frequency events). This capture engine ring buffer is more 
complicated due to the support for variable length records. I uses 
memcpy() to copy the data into the record item. Please compare this with 
the rtems_record_produce() assembly code in the ticket.

Getting the timestamps in the capture engine is expensive. In this patch 
the CPU counter is used in combination with periodic uptime timestamps 
for synchronization. A 22-bit field is used for the timestamp. This is 
enough to a support 4GHz CPU counter with a 1000Hz clock tick, the T4240 
has a CPU counter frequency of 1.5GHz.

>
> Can we please establish the requirements and how this sits with what we have
> sorted out before reviewing patches and details?

My use case for this is a nasty bug. We have about 50 nodes running an 
RTEMS application connected via Ethernet. The nodes operate 24/7. 
Approximately every 60 days one of the nodes crashes and this leads to a 
20 seconds service interruption. Nobody was harmed so far, but it is 
still an issue which needs to be fixed. I need a fast (faster than the 
capture engine) event recording mechanism for post-mortem analysis.

The basic requirement is recording of high frequency events on high end 
embedded SMP machines, e.g. 1.5GHz T4240 with 24 processors and 10Gbit/s 
Ethernet.

>
> An important part of any trace work is host integration and the tools to manage
> this. Getting data off via TCP, UDP, file etc is only part of the problem so it
> would good to know what the integration path on the host is.

The patch set includes a TCP client and server, see test case record01. 
Attached is a simple command line tool which I used on Linux to get the 
event records (32-bit big endian RTEMS machine, 64-bit little endian 
development machine). Some files of this patch are intended to be 
compiled on the development machine.

>
> Sorry but I have no time to review this and consider it until next year.

No problem, take your time. I work on this since April this year from 
time to time, so it can wait a couple of more weeks.

-- 
Sebastian Huber, embedded brains GmbH

Address : Dornierstr. 4, D-82178 Puchheim, Germany
Phone   : +49 89 189 47 41-16
Fax     : +49 89 189 47 41-09
E-Mail  : sebastian.huber at embedded-brains.de
PGP     : Public key available on request.

Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: main.c
Type: text/x-csrc
Size: 4424 bytes
Desc: not available
URL: <http://lists.rtems.org/pipermail/devel/attachments/20181220/5034efe2/attachment-0002.bin>