Timeline Tool

[Chris Johns]

  [ Sorry about the delay ]

Manuel Coutinho wrote:
> Sorry, sent the mail with no subject (may not have passed through some 
> mail filters :S)
> 
>  
> 
> ------------------------------------------------------------------------
> 
> *From:* rtems-users-bounces+manuel.coutinho=edisoft.pt at rtems.org 
> [mailto:rtems-users-bounces+manuel.coutinho=edisoft.pt at rtems.org] *On 
> Behalf Of *Manuel Coutinho
> *Sent:* Thursday, October 04, 2007 3:51 PM
> *To:* rtems-users at rtems.org
> *Subject:*
> 
>  
> 
> Hi
> 
>  
> 
>  
> 
> Edisoft is developing (among other things related with RTEMS) a timeline 
> tool. This tool is designed to log information from the RTEMS 
> application and send it over Ethernet or serial cable to a host computer 
> that displays it to the user.
> 

It is great to see this functionality being added to RTEMS. It is needed.

>  
> 
> A similar tool is provided by RTEMS in the capture engine. The tool we 
> are developing gets some ideas from the capture engine but can trace 
> more events (e.g. schedulability events, interrupt generation, etc).
> 

The capture engine is designed to handle this type of functionality. The part 
missing is the changes to the core and related areas to capture the data.

I would be concerned about another different system being placed into RTEMS to
perform the same function. As it stands I am concerned about the current 
overlap of functionality the capture engine has with other parts of the code.
For example there is a stack checker and the capture engine also has a stack 
checker. Adding another tracing system would just add confusion to a user.

I am happy for a host tool to be called Timeline Tool, but I would prefer we 
use the term 'capture' in RTEMS. I have also kept away from the term logging 
or log. It is over used and means different things to different people. If you 
have any issues with the capture engine being able to do this job please let 
me know.

>  
> 
> This email describes our approach to implement the timeline tool inside 
> the RTEMS source code. We hope to receive comments from the RTEMS user 
> and developer community in order to proceed with the necessary RTEMS 
> changes and implement the timeline tool.
> 
>  
> 
> We are thinking of allowing the user to select the TT (timeline tool) by 
> means of an option during the configuration phase of RTEMS. Example:
> 
>  
> 
> build-rtems>../rtems471/configure \
> 
> --target=sparc-rtems              \
> 
> --enablebsp=erc32                 \
> 
> --enable-rctt                     \ 
> 
> --enable-networking               \
> 
> --disable-posix
> 
>  
> 
>  
> 
> This will enable some macros in the code that will trigger the logging 
> of information.

Agreed. It is important this type of software can be removed from the final
executable.

> 
> Example:
> 
>  
> 
> #ifdef TIMELINE_TOOL
> 
>  log(event);
> 
> #endif
> 

No #ifdef in the code if they can be avoided. Please use a macro or empty
inline function:

  rtems_capture_event( event );

This code is empty when disabled.

>  
> 
>  
> 
> As illustrated in the Figure in attachment, if the user chooses not to 
> use the TT, the additional code is not compiled and only the original 
> RTEMS source code is present in the final executable. If the user 
> requires the TT, the additional code inside the RTEMS source tree is 
> compiled and the events are monitored by the TT.
> 
>  
> 
>  
> 
> The timeline tool requires some RTEMS objects (one task, an extension, 
> etc) and a device driver (Ethernet or serial cable). RTEMS needs to know 
> these requirements to allocate the necessary memory space and initialize 
> the drivers. The application needs to be modified when using the 
> timeline tool to include these resources using, in particular, the 
> confdefs.h file.
> 

Yes the way RTEMS is configured needs to handled. I would go even further and 
add the system must also be able to handle the data flow without impacting the 
performance or function of the application.

In the case of the capture engine it consists of an engine that has an API 
that manages the capture process, eg triggering, filtering. A CLI or network 
interface uses that API to manage the capture engine. An RTEMS user with out a 
network interface should be able to use a tool like this. A user with an 
application specific communication channel should also be able to use this 
code. For example dumping the data to a flash disk. The separation also 
simplifies the configuration issues of --enable-capture (or --enable-rctt 
what-ever) and --disable-network.

>  
> 
>  
> 
> This option requires several RTEMS changes to the kernel. In particular, 
> this implies modifications in almost all RTEMS Managers. These changes 
> can be sub-divided into the following categories:
> 
>  
> 
> 1.    Initialization Manager - the initialization manager has to be 
> modified in order to initialize the timeline tool. Small changes are 
> required here.
> 

I think I disagree :-). It is the BSP or application that should handle the 
capture engine type code and any user interface. If you are referring to the 
trace event point specific code added to the current RTEMS code and its 
related APIs then ok.

>  
> 
> 2.    Extension Manager - although some scheduling events can be caught 
> by the RTEMS Extension Manager, there are some events that remain 
> unmonitored. Such events are the thread insertion/removal to/from the 
> ready queue and priority changes (due, for example, to internal 
> semaphore inherence/ceiling protocols). These events are fundamental for 
> a thorough schedulability analysis. The RTEMS Extension Manager can be 
> extended to monitor these additional events. In order not to create 
> compatibility issues with existent applications, the RTEMS Extension 
> Manager will not be changed (at least, in a near future). Instead, a new 
> functionality will be added, mostly through the addition of a new 
> function: rtems_extension_createX.
> 

I agree the extension manager should stay as it is. I do not understand what
an rtems_extension_createX call is. I suspect we are close in thinking and 
vary a little on terms.

>  
> 
> 3.    Other Managers - to monitor the API calls we envisage the 
> modification of mostly all API primitives so as to log the time the call 
> was made. This will require modifications throughout the native RTEMS 
> API source code. To change the monitored API calls (for example, to 
> monitor the rate_monotonic_period but not monitor the 
> rate_monotonic_get_status), it is required a recompilation of the TT 
> code inside RTEMS.
> 

I do not understand what you mean here. Does this mean each event trace point 
call can be turned on an off by some sort of compile type configuration ?

We may want to capture parameter type data. This could be tricky.

>  
> 
> 4.    Apart from the RTEMS Managers modifications, the integration of 
> the TT will also require changes in the build procedure to include (or 
> not) the TT code and to configure it.
> 

Agreed. Plus to also manage the mixing of other configuration type options.

>  
> 
> 5.    Small changes to the RTEMS Super Core will also be needed in order 
> for the TT to catch the RTEMS interval events (such as thread 
> insertion/removal from ready queue and priority changes due to semaphore 
> inherence/ceiling protocols). The required modifications inside the 
> Super Core to catch the additional internal events are very sensitive 
> since this code is a critical path and must be optimized. If the 
> Extension Manager is changed and extended to monitor these new events, 
> then a temporal overhead is always introduced because an "if" function 
> is always performed at these critical steps. However, if the Extension 
> Manager is not changed but only incorporates the new functionality 
> (e.g., through a rtems_extension_createX function), then the code 
> remains optimized (an #ifdef directive is performed).
> 

I agree we cannot leave the 'if' code in the critical paths like the current 
task trace code does. If your 'rtems_extension_createX' means we add or extend 
the extensions API then I agree.

I would like to see the changes made to the current RTEMS code kept separate 
from the code that handles triggering, filtering and storage of the data being 
captured (ie cpukit/libmisc/capture). It is important the trace point code is 
flexible, fast and small and building it into a kernel does not require the 
capture engine type code. I would also like to be able to add new trace points 
into applications and have those events captured into the data stream and 
handled at the host end.

The actual tracing needs to be broken down in domains, groups or what-ever. 
For example tasking, semaphores etc. It would concern me to have tracing of 
everything happening all the time. If your application requires some form of 
validation and needs everything and can handle the data flow then enable 
everything, but I suspect most users do not need everything or do not have the
resources to handle the data rate. I have been thinking about a runtime switch 
inlined into the trace point that checks a table pointer. If set the pointer 
is called. The idea is the performance of the system is effected as little as 
possible unless the specific type of trace point has been asked to be 
monitored. On top of this the capture code has filtering and triggering but 
that is more complex and takes more processing time.

Regards
Chris