SMP boot sequence

[Sebastian Huber]

On 2013-10-24 15:42, Daniel Hellstrom wrote:
> I can see from the running on the LEON SMP hardware and analysing with GRMON
> that the init task is started and begins its execution on CPU1 before CPU0 (the
> booting CPU) has finished the RTEMS initialization in boot_card().

This is only a problem if the CPU1 performs the switch to the first scheduled 
thread too early.  There is a rendezvous in the initialization sequence for this:

typedef enum {
   /**
    * @brief The per CPU controls are initialized to zero.
    *
    * In this state the only valid field of the per CPU controls for secondary
    * processors is the per CPU state.  The secondary processors should perform
    * their basic initialization now and change into the
    * PER_CPU_STATE_READY_TO_BEGIN_MULTITASKING state once this is complete.
    *
    * The owner of the per CPU state field is the secondary processor in this
    * state.
    */
   PER_CPU_STATE_BEFORE_INITIALIZATION,

   /**
    * @brief Secondary processor is ready to begin multitasking.
    *
    * The secondary processor performed its basic initialization and is ready to
    * receive inter-processor interrupts.  Interrupt delivery must be disabled
    * in this state, but requested inter-processor interrupts must be recorded
    * and must be delivered once the secondary processor enables interrupts for
    * the first time.  The main processor will wait for all secondary processors
    * to change into this state.  In case a secondary processor does not reach
    * this state the system will not start.  The secondary processors wait now
    * for a change into the PER_CPU_STATE_BEGIN_MULTITASKING state set by the
    * main processor once all secondary processors reached the
    * PER_CPU_STATE_READY_TO_BEGIN_MULTITASKING state.
    *
    * The owner of the per CPU state field is the main processor in this state.
    */
   PER_CPU_STATE_READY_TO_BEGIN_MULTITASKING,

   /**
    * @brief Multitasking begin of secondary processor is requested.
    *
    * The main processor completed system initialization and is about to perform
    * a context switch to its heir thread.  Secondary processors should now
    * issue a context switch to the heir thread.  This normally enables
    * interrupts on the processor for the first time.
    *
    * The owner of the per CPU state field is the secondary processor in this
    * state.
    */
   PER_CPU_STATE_BEGIN_MULTITASKING,

   /**
    * @brief Normal multitasking state.
    *
    * The owner of the per CPU state field is the secondary processor in this
    * state.
    */
   PER_CPU_STATE_UP,

   /**
    * @brief This is the terminal state.
    *
    * The owner of the per CPU state field is the secondary processor in this
    * state.
    */
   PER_CPU_STATE_SHUTDOWN
} Per_CPU_State;

The secondary processors wait for the PER_CPU_STATE_BEGIN_MULTITASKING state:

void rtems_smp_secondary_cpu_initialize( void )
{
   Per_CPU_Control *self_cpu = _Per_CPU_Get();

   #if defined(RTEMS_DEBUG)
     printk( "Made it to %d -- ", _Per_CPU_Get_index( self_cpu ) );
   #endif

   _Per_CPU_Change_state( self_cpu, PER_CPU_STATE_READY_TO_BEGIN_MULTITASKING );

   _Per_CPU_Wait_for_state( self_cpu, PER_CPU_STATE_BEGIN_MULTITASKING );

   _Thread_Start_multitasking( NULL );
}

The main processor waits for all secondary processors to go into the 
PER_CPU_STATE_READY_TO_BEGIN_MULTITASKING state:

   void _SMP_Handler_initialize(void)
   {
[...]
     /*
      * Discover and initialize the secondary cores in an SMP system.
      */
     max_cpus = bsp_smp_initialize( max_cpus );

     _SMP_Processor_count = max_cpus;

     for ( cpu = 1 ; cpu < max_cpus; ++cpu ) {
       const Per_CPU_Control *per_cpu = _Per_CPU_Get_by_index( cpu );

       _Per_CPU_Wait_for_state(
         per_cpu,
         PER_CPU_STATE_READY_TO_BEGIN_MULTITASKING
       );
     }
   }

Then it indicats that it completed the low-level initialization:

void _SMP_Request_other_cores_to_perform_first_context_switch( void )
{
   uint32_t self = _SMP_Get_current_processor();
   uint32_t ncpus = _SMP_Get_processor_count();
   uint32_t cpu;

   for ( cpu = 0 ; cpu < ncpus ; ++cpu ) {
     Per_CPU_Control *per_cpu = _Per_CPU_Get_by_index( cpu );

     if ( cpu != self ) {
       _Per_CPU_Change_state( per_cpu, PER_CPU_STATE_BEGIN_MULTITASKING );
     }
   }
}

> As I
> understand the boot procedure of RTEMS global CPU interrupt is not enabled
> before the first task is actually scheduled which works fine on the
> single-core.

Interrupts are enabled in _Thread_Handler() the first time.

> However on a SMP machine the secondary CPUs have enabled global
> CPU interrupt in order to receive IPIs during the boot sequence.

This is a bug.  Interrupts must be disabled during the low-level boot process. 
  Only _Thread_Handler() should enable the interrupts.  However the secondary 
processors must be able to notice an IPI and serve it right after the 
interrupts are enabled the first time.

> At some point
> in the initialization sequence an IPI is sent to a secondary CPU, after the
> secondary CPU IPI has been handled the ISR exists into thread dispatch which
> switches in the Init task, and this is before CPU0 has completed boot_card().
> I'm not sure how this should be fixed, perhaps secondary CPUs should be waked
> later in the initialization or the init task should not be enabled until the
> complete OS has been initialized.
>
> What is the point in waking secondary CPUs before the can be given work to do?
> Of course initialization sequence would be easier to always have them on like
> in the current case... it seems that in other SMP OSes the initialization order
> have been split up in two parts, the first being single-core only, and the
> other to be multi-core "safe", I guess in the long run that is perhaps a good
> strategy for RTEMS as well where multiple CPUs could speed up the final
> initialization.
>
> Driver initialization triggered from boot_card() may create READY tasks that
> might also be executed is my guess if more than two CPUs are present.

Yes, the scheduler decides which threads run in the first place.

>
> It seems to me that this is a platform independent problem, or how have PPC,
> Intel and ARM solved this?

Interrupts must be disabled during the low-level startup.

-- 
Sebastian Huber, embedded brains GmbH

Address : Dornierstr. 4, D-82178 Puchheim, Germany
Phone   : +49 89 189 47 41-16
Fax     : +49 89 189 47 41-09
E-Mail  : sebastian.huber at embedded-brains.de
PGP     : Public key available on request.

Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG.