RTEMS Tests and Profiling

[Sebastian Huber]

Hello,

in the long run we need a framework to write tests.  At the moment I have 
absolutely no time to work on this.  I need a short term solution to get 
profiling data for the tests.

Why is this profiling important?  Currently we don't know how good RTEMS is in 
terms of thread dispatch latency for example (an important characteristic for 
real-time systems).  We have some timing tests, but they yield only some 
artificial values (except tmcontext01).  The timing tests neglect cache effects 
and load from other processors on an SMP system.  The tmcontext01 is a template 
test that addresses this issue.  With the profiling data obtained for all tests 
of the RTEMS test suite we get a larger sample size.  The profiling data is in 
XML format which makes it very easy to post-process, e.g.

http://git.rtems.org/rtems/tree/testsuites/tmtests/tmcontext01/plot.py

Supporting SMP will lead to massive changes in the critical sections if we want 
to support fine grained locking.  Thus we have to show somehow that things 
don't get too bad for the single-processor systems.  This is why I work on the 
profiling first.

With the first profiling data available (smpload01) it is obvious that we need 
fine grained locking:

     <SMPLockProfilingReport name="Giant">
       <MaxAcquireTime unit="ns">245180</MaxAcquireTime>
       <MaxSectionTime unit="ns">232130</MaxSectionTime>
       <UsageCount>10102</UsageCount>
       <TotalAcquireTime unit="ns">416556635</TotalAcquireTime>
       <TotalSectionTime unit="ns">529718895</TotalSectionTime>
       <ContentionCount initialQueueLength="0">1950</ContentionCount>
       <ContentionCount initialQueueLength="1">4735</ContentionCount>
       <ContentionCount initialQueueLength="2">2441</ContentionCount>
       <ContentionCount initialQueueLength="3">976</ContentionCount>
     </SMPLockProfilingReport>

Now back to the tests.  The only practically available way to determine if a 
test was successful is to look at the "*** END OF TEST" pattern in the test 
output.  It makes sense to use a common function for test begin and end 
messages.  Now where should this function be located?

We have a testsuites/support directory with a bunch of files.  I would like to 
move away from the existing test support.  The problem is that you have to add 
at least this in every test Makefile.am:

test_SOURCES += ../../support/src/test_support.c
AM_CPPFLAGS += -I$(top_srcdir)/../support/include

I consider ../ paths in Makefiles as hacks.

You have to compile test_support.c again for each tests which increases the 
build times.

You have also hacks like this in a header file (buffer_test_io.h):

   #if defined(TEST_INIT) || defined(CONFIGURE_INIT)

     char _test_output_buffer[_TEST_OUTPUT_BUFFER_SIZE];
     int _test_output_buffer_index = 0;

     void _test_output_append(char *_buffer)
     {
       char *p;

       for ( p=_buffer ; *p ; p++ ) {
         _test_output_buffer[_test_output_buffer_index++] = *p;
         _test_output_buffer[_test_output_buffer_index]   = '\0';
         #if 0
       if ( *p == '\n' ) {
         fprintf( stderr, "BUFFER -- %s", _test_output_buffer );
         _test_output_buffer_index = 0;
        _test_output_buffer[0]   = '\0';
       }
         #endif
     if ( _test_output_buffer_index >= (_TEST_OUTPUT_BUFFER_SIZE - 80) )
       _test_output_flush();
       }
     }

     #include <termios.h>
     #include <unistd.h>

     void _test_output_flush(void)
     {
       fprintf( stderr, "%s", _test_output_buffer );
       _test_output_buffer_index = 0;
       tcdrain( 2 );
     }

     #endif

Profiling data can be output in a test fatal extension handler.

I propose to move test support code to the cpukit.  Attached is an example 
header file.

-- 
Sebastian Huber, embedded brains GmbH

Address : Dornierstr. 4, D-82178 Puchheim, Germany
Phone   : +49 89 189 47 41-16
Fax     : +49 89 189 47 41-09
E-Mail  : sebastian.huber at embedded-brains.de
PGP     : Public key available on request.

Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test.h
Type: text/x-chdr
Size: 2559 bytes
Desc: not available
URL: <http://lists.rtems.org/pipermail/devel/attachments/20140311/8a7d7db7/attachment.bin>