SMP boot sequence

[Daniel Hellstrom]

On 10/24/2013 04:08 PM, Sebastian Huber wrote:
> On 2013-10-24 15:42, Daniel Hellstrom wrote:
>> I can see from the running on the LEON SMP hardware and analysing with GRMON
>> that the init task is started and begins its execution on CPU1 before CPU0 (the
>> booting CPU) has finished the RTEMS initialization in boot_card().
>
> This is only a problem if the CPU1 performs the switch to the first scheduled thread too early.  There is a rendezvous in the initialization sequence for this:
>
> typedef enum {
>   /**
>    * @brief The per CPU controls are initialized to zero.
>    *
>    * In this state the only valid field of the per CPU controls for secondary
>    * processors is the per CPU state.  The secondary processors should perform
>    * their basic initialization now and change into the
>    * PER_CPU_STATE_READY_TO_BEGIN_MULTITASKING state once this is complete.
>    *
>    * The owner of the per CPU state field is the secondary processor in this
>    * state.
>    */
>   PER_CPU_STATE_BEFORE_INITIALIZATION,
>
>   /**
>    * @brief Secondary processor is ready to begin multitasking.
>    *
>    * The secondary processor performed its basic initialization and is ready to
>    * receive inter-processor interrupts.  Interrupt delivery must be disabled
>    * in this state, but requested inter-processor interrupts must be recorded
>    * and must be delivered once the secondary processor enables interrupts for
>    * the first time.  The main processor will wait for all secondary processors
>    * to change into this state.  In case a secondary processor does not reach
>    * this state the system will not start.  The secondary processors wait now
>    * for a change into the PER_CPU_STATE_BEGIN_MULTITASKING state set by the
>    * main processor once all secondary processors reached the
>    * PER_CPU_STATE_READY_TO_BEGIN_MULTITASKING state.
>    *
>    * The owner of the per CPU state field is the main processor in this state.
>    */
>   PER_CPU_STATE_READY_TO_BEGIN_MULTITASKING,
>
>   /**
>    * @brief Multitasking begin of secondary processor is requested.
>    *
>    * The main processor completed system initialization and is about to perform
>    * a context switch to its heir thread.  Secondary processors should now
>    * issue a context switch to the heir thread.  This normally enables
>    * interrupts on the processor for the first time.
>    *
>    * The owner of the per CPU state field is the secondary processor in this
>    * state.
>    */
>   PER_CPU_STATE_BEGIN_MULTITASKING,
>
>   /**
>    * @brief Normal multitasking state.
>    *
>    * The owner of the per CPU state field is the secondary processor in this
>    * state.
>    */
>   PER_CPU_STATE_UP,
>
>   /**
>    * @brief This is the terminal state.
>    *
>    * The owner of the per CPU state field is the secondary processor in this
>    * state.
>    */
>   PER_CPU_STATE_SHUTDOWN
> } Per_CPU_State;
>
> The secondary processors wait for the PER_CPU_STATE_BEGIN_MULTITASKING state:
>
> void rtems_smp_secondary_cpu_initialize( void )
> {
>   Per_CPU_Control *self_cpu = _Per_CPU_Get();
>
>   #if defined(RTEMS_DEBUG)
>     printk( "Made it to %d -- ", _Per_CPU_Get_index( self_cpu ) );
>   #endif
>
>   _Per_CPU_Change_state( self_cpu, PER_CPU_STATE_READY_TO_BEGIN_MULTITASKING );
>
>   _Per_CPU_Wait_for_state( self_cpu, PER_CPU_STATE_BEGIN_MULTITASKING );
>
>   _Thread_Start_multitasking( NULL );
> }
>
> The main processor waits for all secondary processors to go into the PER_CPU_STATE_READY_TO_BEGIN_MULTITASKING state:
>
>   void _SMP_Handler_initialize(void)
>   {
> [...]
>     /*
>      * Discover and initialize the secondary cores in an SMP system.
>      */
>     max_cpus = bsp_smp_initialize( max_cpus );
>
>     _SMP_Processor_count = max_cpus;
>
>     for ( cpu = 1 ; cpu < max_cpus; ++cpu ) {
>       const Per_CPU_Control *per_cpu = _Per_CPU_Get_by_index( cpu );
>
>       _Per_CPU_Wait_for_state(
>         per_cpu,
>         PER_CPU_STATE_READY_TO_BEGIN_MULTITASKING
>       );
>     }
>   }
>
> Then it indicats that it completed the low-level initialization:
>
> void _SMP_Request_other_cores_to_perform_first_context_switch( void )
> {
>   uint32_t self = _SMP_Get_current_processor();
>   uint32_t ncpus = _SMP_Get_processor_count();
>   uint32_t cpu;
>
>   for ( cpu = 0 ; cpu < ncpus ; ++cpu ) {
>     Per_CPU_Control *per_cpu = _Per_CPU_Get_by_index( cpu );
>
>     if ( cpu != self ) {
>       _Per_CPU_Change_state( per_cpu, PER_CPU_STATE_BEGIN_MULTITASKING );
>     }
>   }
> }
>
>> As I
>> understand the boot procedure of RTEMS global CPU interrupt is not enabled
>> before the first task is actually scheduled which works fine on the
>> single-core.
>
> Interrupts are enabled in _Thread_Handler() the first time.
>
>> However on a SMP machine the secondary CPUs have enabled global
>> CPU interrupt in order to receive IPIs during the boot sequence.
>
> This is a bug.  Interrupts must be disabled during the low-level boot process.  Only _Thread_Handler() should enable the interrupts.  However the secondary processors must be able to notice an IPI 
> and serve it right after the interrupts are enabled the first time.
Interrupts are off on CPU0 during boot.

I'm not sure I follow this 100%. During low-level boot interrupt should be off on all CPUs, _Thread_Handler() enables the current CPU's interrupt. So how should a secondary core receive an IPI, I mean 
an IPI is required to get to _Thread_Handler() in order to turn on interrupts. But we need interrupts enabled in order for IPIs to work?

Or are you saying that there is an endless loop waiting for CPU0 to signal multitasking via shared memory (not through IPI)? In that case I could just turn of secondary CPU interrupt.

>
>> At some point
>> in the initialization sequence an IPI is sent to a secondary CPU, after the
>> secondary CPU IPI has been handled the ISR exists into thread dispatch which
>> switches in the Init task, and this is before CPU0 has completed boot_card().
>> I'm not sure how this should be fixed, perhaps secondary CPUs should be waked
>> later in the initialization or the init task should not be enabled until the
>> complete OS has been initialized.
>>
>> What is the point in waking secondary CPUs before the can be given work to do?
>> Of course initialization sequence would be easier to always have them on like
>> in the current case... it seems that in other SMP OSes the initialization order
>> have been split up in two parts, the first being single-core only, and the
>> other to be multi-core "safe", I guess in the long run that is perhaps a good
>> strategy for RTEMS as well where multiple CPUs could speed up the final
>> initialization.
>>
>> Driver initialization triggered from boot_card() may create READY tasks that
>> might also be executed is my guess if more than two CPUs are present.
>
> Yes, the scheduler decides which threads run in the first place.
>
>>
>> It seems to me that this is a platform independent problem, or how have PPC,
>> Intel and ARM solved this?
>
> Interrupts must be disabled during the low-level startup.

Ok, I will investigate this.

Thanks,
Daniel