SMP Initialization

[Sebastian Huber]

Hello,

there is a potential problem in the SMP initialization procedure.

One processor in the system has a special role, the so called boot processor. 
Currently this is the processor with index zero.  The way to select the boot 
processor may change in the future, but what will not change is that we have a 
boot processor.

The boot processors initializes the data and BSS sections.  It performs also 
the sequential part of the RTEMS initialization.

During the sequential initialization the function

/**
  * @brief Performs CPU specific SMP initialization in the context of the boot
  * processor.
  *
  * This function is invoked on the boot processor by RTEMS during
  * initialization.  All interrupt stacks are allocated at this point in case
  * the CPU port allocates the interrupt stacks.
  *
  * The CPU port should start secondary processors now.
  *
  * @param[in] configured_cpu_count The count of processors requested by the
  * application configuration.
  *
  * @return The count of processors available for the application in the system.
  * This value is less than or equal to the configured count of processors.
  */
uint32_t _CPU_SMP_Initialize( uint32_t configured_cpu_count );

called.  This function is currently implemented by the BSPs.  An example which 
starts the processor on its own:

http://git.rtems.org/rtems/tree/c/src/lib/libbsp/sparc/leon3/smp/smp_leon3.c#n38

An example which uses U-Boot to start the second processor:

http://git.rtems.org/rtems/tree/c/src/lib/libbsp/powerpc/qoriq/startup/smp.c#n144

The return value of _CPU_SMP_Initialize() will tell the RTEMS system how many 
processors are present.

void _SMP_Handler_initialize( void )
{
   uint32_t max_cpus = rtems_configuration_get_maximum_processors();
   uint32_t cpu;

[...]

   /*
    * Discover and initialize the secondary cores in an SMP system.
    */
   max_cpus = _CPU_SMP_Initialize( max_cpus );

   _SMP_Processor_count = max_cpus;
}

If the BSP says "you have three processors", and one of them is actually not 
available, then we have a problem later.

Before the system starts multitasking there is a synchronization barrier.  This 
synchronization barrier is necessary to have a defined starting point for the 
scheduler.

void _SMP_Request_start_multitasking( void )
{
   Per_CPU_Control *self_cpu = _Per_CPU_Get();
   uint32_t ncpus = _SMP_Get_processor_count();
   uint32_t cpu;

   _Per_CPU_State_change( self_cpu, PER_CPU_STATE_READY_TO_START_MULTITASKING );

   for ( cpu = 0 ; cpu < ncpus ; ++cpu ) {
     Per_CPU_Control *per_cpu = _Per_CPU_Get_by_index( cpu );

     _Per_CPU_State_change( per_cpu, PER_CPU_STATE_REQUEST_START_MULTITASKING );
   }
}

So before this function returns ALL (!) processors must have changed into the 
PER_CPU_STATE_REQUEST_START_MULTITASKING (or into PER_CPU_STATE_SHUTDOWN which 
will terminate the system right now).

In case one of the processors doesn't start, then we will wait here FOREVER 
(unless a watchdog kill us).

There are now several ways to deal with this.

1. You can consider this a BSP bug.  The BSP told the system via 
_CPU_SMP_Initialize() that so many processors are available.  If this is not 
the case then the BSP lied and you should fix the BSP.

2. You can consider this a feature of the BSP that it tells you wrong numbers. 
  So now what to do?

2.1. You can install a watchdog driver that kills you no matter what corrupt 
systems state you have.  If you analyze the per-CPU states in this case you 
will notice that some of the processors didn't start.

2.2. You can limit the time spent waiting.  If a timeout occurs then we can 
issue a fatal error that indicates exactly the problem area.

2.2.1 Now we need a facility to measure time (e.g. the CPU counter introduced 
recently).

2.2.2 Now we need a timeout.

2.2.2.1 The RTEMS kernel cannot know a proper timeout value.

2.2.2.2 The CPU/BSP may know the timeout value.  How can the CPU/BSP tell the 
RTEMS kernel timeout value?

2.2.2.3 We can add an application configuration item that specifies the timeout 
value and move the responsibility to the application developer.

I am in favor of 1. in combination with 2.1 and 2.2.2.2.  For BSPs with 
unreliably start of secondary processors we should add a support function, e.g.

/**
  * @brief Waits for all other processors to enter the ready to start
  * multitasking state with a timeout in microseconds.
  *
  * In case one processor enters the shutdown state, this function does not
  * return.
  *
  * This function should be called only in _CPU_SMP_Initialize() if required by
  * the CPU port or BSP.
  *
  * @param[in] processor_count The processor count which will later returned by
  * _CPU_SMP_Initialize().
  * @param[in] timeout_in_us The timeout in microseconds.
  *
  * @retval true All other processors entered the ready to start multitasking
  * state.
  * @retval false Not all the other processors entered the ready to start
  * multitasking state and the timeout expired.
  */
bool _Per_CPU_State_wait_for_ready_to_start_multitasking(
   uint32_t processor_count,
   uint32_t timeout_in_us
);

This avoids the burden for the application developer to know about the timeout 
configuration option and to select a proper value.  It moves the responsibility 
to deal with issue to the BSP which knows best what to do.  In case false is 
returned it can either issue a fatal error or reduce the processor count.

-- 
Sebastian Huber, embedded brains GmbH

Address : Dornierstr. 4, D-82178 Puchheim, Germany
Phone   : +49 89 189 47 41-16
Fax     : +49 89 189 47 41-09
E-Mail  : sebastian.huber at embedded-brains.de
PGP     : Public key available on request.

Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG.