Timeline Tool

[Manuel Coutinho]

Hi

> -----Original Message-----
> From: Chris Johns [mailto:chrisj at rtems.org]
> Sent: Tuesday, October 16, 2007 3:12 AM
> To: Manuel Coutinho
> Cc: rtems-users at rtems.org
> Subject: Re: Timeline Tool
> 
>   [ Sorry about the delay ]
> 
> Manuel Coutinho wrote:
> > Sorry, sent the mail with no subject (may not have passed through some
> > mail filters :S)
> >
> >
> >
> > ------------------------------------------------------------------------
> >
> > *From:* rtems-users-bounces+manuel.coutinho=edisoft.pt at rtems.org
> > [mailto:rtems-users-bounces+manuel.coutinho=edisoft.pt at rtems.org] *On
> > Behalf Of *Manuel Coutinho
> > *Sent:* Thursday, October 04, 2007 3:51 PM
> > *To:* rtems-users at rtems.org
> > *Subject:*
> >
> >
> >
> > Hi
> >
> >
> >
> >
> >
> > Edisoft is developing (among other things related with RTEMS) a timeline
> > tool. This tool is designed to log information from the RTEMS
> > application and send it over Ethernet or serial cable to a host computer
> > that displays it to the user.
> >
> 
> It is great to see this functionality being added to RTEMS. It is needed.
> 
> >
> >
> > A similar tool is provided by RTEMS in the capture engine. The tool we
> > are developing gets some ideas from the capture engine but can trace
> > more events (e.g. schedulability events, interrupt generation, etc).
> >
> 
> The capture engine is designed to handle this type of functionality. The
> part
> missing is the changes to the core and related areas to capture the data.
> 
> I would be concerned about another different system being placed into
> RTEMS to
> perform the same function. As it stands I am concerned about the current
> overlap of functionality the capture engine has with other parts of the
> code.
> For example there is a stack checker and the capture engine also has a
> stack
> checker. Adding another tracing system would just add confusion to a user.
> 
> I am happy for a host tool to be called Timeline Tool, but I would prefer
> we
> use the term 'capture' in RTEMS. I have also kept away from the term
> logging
> or log. It is over used and means different things to different people. If
> you
> have any issues with the capture engine being able to do this job please
> let
> me know.

[Manuel] I think the inclusion of the "timeline tool" inside the capture
engine is a good idea. But we have some requirements that might require some
modifications to the current capture engine. I would prefer building a
separate tool because we don't want to change the current status of the
capture engine, i.e., its current functionality. If you (and the RTEMS
community) don't have any issues with us modifying the current capture
engine we also don't have any problems.

> 
> >
> >
> > This email describes our approach to implement the timeline tool inside
> > the RTEMS source code. We hope to receive comments from the RTEMS user
> > and developer community in order to proceed with the necessary RTEMS
> > changes and implement the timeline tool.
> >
> >
> >
> > We are thinking of allowing the user to select the TT (timeline tool) by
> > means of an option during the configuration phase of RTEMS. Example:
> >
> >
> >
> > build-rtems>../rtems471/configure \
> >
> > --target=sparc-rtems              \
> >
> > --enablebsp=erc32                 \
> >
> > --enable-rctt                     \
> >
> > --enable-networking               \
> >
> > --disable-posix
> >
> >
> >
> >
> >
> > This will enable some macros in the code that will trigger the logging
> > of information.
> 
> Agreed. It is important this type of software can be removed from the
> final
> executable.
> 
> >
> > Example:
> >
> >
> >
> > #ifdef TIMELINE_TOOL
> >
> >  log(event);
> >
> > #endif
> >
> 
> No #ifdef in the code if they can be avoided. Please use a macro or empty
> inline function:
> 
>   rtems_capture_event( event );
> 
> This code is empty when disabled.

[Manuel] From what I can tell this is a great idea. If I understood you
correctly, you are saying to make the "timeline tool" as something like an
RTEMS Manager that is included (or not) in the application Makefile. This
way the timeline tool would be always compiled and generate a dummy .rel
file and a "complete" .rel file.
Perhaps you could just give an initial pointer to get me started on this :).

> 
> >
> >
> >
> >
> > As illustrated in the Figure in attachment, if the user chooses not to
> > use the TT, the additional code is not compiled and only the original
> > RTEMS source code is present in the final executable. If the user
> > requires the TT, the additional code inside the RTEMS source tree is
> > compiled and the events are monitored by the TT.
> >
> >
> >
> >
> >
> > The timeline tool requires some RTEMS objects (one task, an extension,
> > etc) and a device driver (Ethernet or serial cable). RTEMS needs to know
> > these requirements to allocate the necessary memory space and initialize
> > the drivers. The application needs to be modified when using the
> > timeline tool to include these resources using, in particular, the
> > confdefs.h file.
> >
> 
> Yes the way RTEMS is configured needs to handled. I would go even further
> and
> add the system must also be able to handle the data flow without impacting
> the
> performance or function of the application.

[Manuel] The timeline tool will always impact the performance of the
application. The simple act of saving the event that occurred takes time and
the transmission to the host takes even more time. We plan to make the
transmission inside a task that has user configured priority and period (a
default value is provided).


> 
> In the case of the capture engine it consists of an engine that has an API
> that manages the capture process, eg triggering, filtering. A CLI or
> network
> interface uses that API to manage the capture engine. An RTEMS user with
> out a
> network interface should be able to use a tool like this. A user with an
> application specific communication channel should also be able to use this
> code. For example dumping the data to a flash disk. The separation also
> simplifies the configuration issues of --enable-capture (or --enable-rctt
> what-ever) and --disable-network.

[Manuel] We can provide an interface that is device independent (i.e. writes
to serial, Ethernet, flash disk, etc). If I understood you in your
commentary above, the timeline tool is integrated as an RTEMS Manager so the
user won't have to do --enable-rctt (or whatever) when building RTEMS. The
user will only have to do something like

managers = io semaphore message timeline

right?
> 
> >
> >
> >
> >
> > This option requires several RTEMS changes to the kernel. In particular,
> > this implies modifications in almost all RTEMS Managers. These changes
> > can be sub-divided into the following categories:
> >
> >
> >
> > 1.    Initialization Manager - the initialization manager has to be
> > modified in order to initialize the timeline tool. Small changes are
> > required here.
> >
> 
> I think I disagree :-). It is the BSP or application that should handle
> the
> capture engine type code and any user interface. If you are referring to
> the
> trace event point specific code added to the current RTEMS code and its
> related APIs then ok.

[Manuel] One of our requirements states that the timeline tool is
"independent" from the application, that is, the tool can collect
information even if the application does not call any timeline tool
function. To do this the tool must be up and running prior to the
application is started.

> 
> >
> >
> > 2.    Extension Manager - although some scheduling events can be caught
> > by the RTEMS Extension Manager, there are some events that remain
> > unmonitored. Such events are the thread insertion/removal to/from the
> > ready queue and priority changes (due, for example, to internal
> > semaphore inherence/ceiling protocols). These events are fundamental for
> > a thorough schedulability analysis. The RTEMS Extension Manager can be
> > extended to monitor these additional events. In order not to create
> > compatibility issues with existent applications, the RTEMS Extension
> > Manager will not be changed (at least, in a near future). Instead, a new
> > functionality will be added, mostly through the addition of a new
> > function: rtems_extension_createX.
> >
> 
> I agree the extension manager should stay as it is. I do not understand
> what
> an rtems_extension_createX call is. I suspect we are close in thinking and
> vary a little on terms.

[Manuel] We were planning to create a NEW structure that contains pointers
to functions that are called at critical instants (like the RTEMS Extension
Manager). These instants correspond to task insertion in a ready queue, task
priority changes and task suspending/resuming. An rtems_extension_createX
function would just set the pointers correctly. We are now thinking of
building a timeline tool Manager that incorporates this functionality.

> 
> >
> >
> > 3.    Other Managers - to monitor the API calls we envisage the
> > modification of mostly all API primitives so as to log the time the call
> > was made. This will require modifications throughout the native RTEMS
> > API source code. To change the monitored API calls (for example, to
> > monitor the rate_monotonic_period but not monitor the
> > rate_monotonic_get_status), it is required a recompilation of the TT
> > code inside RTEMS.
> >
> 
> I do not understand what you mean here. Does this mean each event trace
> point
> call can be turned on an off by some sort of compile type configuration ?
> 
> We may want to capture parameter type data. This could be tricky.

[Manuel] We are thinking of creation a file inside the RTEMS source code
where the user can modify which calls are by default traced (not logged :)).
The application can call a timeline tool API to, at runtime, enable/disable
specific traces (e.g. want to monitor semaphore calls but not region nor
partition). This file is a .c (or .h) file and by modifying these parameters
the user would have to recompile RTEMS (at least this .c file).

> 
> >
> >
> > 4.    Apart from the RTEMS Managers modifications, the integration of
> > the TT will also require changes in the build procedure to include (or
> > not) the TT code and to configure it.
> >
> 
> Agreed. Plus to also manage the mixing of other configuration type
> options.
> 
> >
> >
> > 5.    Small changes to the RTEMS Super Core will also be needed in order
> > for the TT to catch the RTEMS interval events (such as thread
> > insertion/removal from ready queue and priority changes due to semaphore
> > inherence/ceiling protocols). The required modifications inside the
> > Super Core to catch the additional internal events are very sensitive
> > since this code is a critical path and must be optimized. If the
> > Extension Manager is changed and extended to monitor these new events,
> > then a temporal overhead is always introduced because an "if" function
> > is always performed at these critical steps. However, if the Extension
> > Manager is not changed but only incorporates the new functionality
> > (e.g., through a rtems_extension_createX function), then the code
> > remains optimized (an #ifdef directive is performed).
> >
> 
> I agree we cannot leave the 'if' code in the critical paths like the
> current
> task trace code does. If your 'rtems_extension_createX' means we add or
> extend
> the extensions API then I agree.
> 
> I would like to see the changes made to the current RTEMS code kept
> separate
> from the code that handles triggering, filtering and storage of the data
> being
> captured (ie cpukit/libmisc/capture). It is important the trace point code
> is
> flexible, fast and small and building it into a kernel does not require
> the
> capture engine type code. I would also like to be able to add new trace
> points
> into applications and have those events captured into the data stream and
> handled at the host end.
> 
> The actual tracing needs to be broken down in domains, groups or what-
> ever.
> For example tasking, semaphores etc. It would concern me to have tracing
> of
> everything happening all the time. If your application requires some form
> of
> validation and needs everything and can handle the data flow then enable
> everything, but I suspect most users do not need everything or do not have
> the
> resources to handle the data rate. I have been thinking about a runtime
> switch
> inlined into the trace point that checks a table pointer. If set the
> pointer
> is called. The idea is the performance of the system is effected as little
> as
> possible unless the specific type of trace point has been asked to be
> monitored. On top of this the capture code has filtering and triggering
> but
> that is more complex and takes more processing time.
> 
> Regards
> Chris
> 

[Manuel] Since (I think) we are moving in the direction of building a
"Timeline Tool Manager", I think we don't need to create an extension to the
RTEMS Extension Manager: the new functionality is implemented by the
timeline tool Manager. This way, if the timeline tool manager is not
selected, an empty function is called and no performance is lost (no "if" is
performed). 
I also agree that the body of the "timeline tool" should be placed in a
different areas other that the RTEMS source code (e.g. libmisc/capture) but
some functions must be called from inside the remaining RTEMS source code. 
We are not thinking of allowing the application to trace specific events.
The timeline tool main purpose is to analyze what the application is doing
in terms of schedulability.
The application can define certain parameters so as to not trace every
single event. I did not mention it before because it takes a long time to
explain every parameter.
I think your idea of a table that contains flags with the information about
whether to save (or not) information that a given event has occurred is a
good one.

Regards
Manuel Coutinho

> 
>