SMP boot sequence

[Daniel Hellstrom]

On 10/25/2013 02:40 PM, Sebastian Huber wrote:
> On 2013-10-25 14:23, Daniel Hellstrom wrote:
>>>
>>> Interrupts are enabled in _Thread_Handler() the first time.
>>>
>>>> However on a SMP machine the secondary CPUs have enabled global
>>>> CPU interrupt in order to receive IPIs during the boot sequence.
>>>
>>> This is a bug.  Interrupts must be disabled during the low-level boot
>>> process.  Only _Thread_Handler() should enable the interrupts. However the
>>> secondary processors must be able to notice an IPI and serve it right after
>>> the interrupts are enabled the first time.
>> Interrupts are off on CPU0 during boot.
>>
>> I'm not sure I follow this 100%. During low-level boot interrupt should be off
>> on all CPUs, _Thread_Handler() enables the current CPU's interrupt.
>
> Yes, interrupts must be disabled on all CPUs during low-level start.
>
>> So how
>> should a secondary core receive an IPI, I mean an IPI is required to get to
>> _Thread_Handler() in order to turn on interrupts.
>
> No, this is not the case.  Secondary cores busy wait for a per CPU state change:
>
> void rtems_smp_secondary_cpu_initialize( void )
> {
>   Per_CPU_Control *self_cpu = _Per_CPU_Get();
>
>   #if defined(RTEMS_DEBUG)
>     printk( "Made it to %d -- ", _Per_CPU_Get_index( self_cpu ) );
>   #endif
>
>   _Per_CPU_Change_state( self_cpu, PER_CPU_STATE_READY_TO_BEGIN_MULTITASKING );
>
>   _Per_CPU_Wait_for_state( self_cpu, PER_CPU_STATE_BEGIN_MULTITASKING );
>
>   _Thread_Start_multitasking( NULL );
> }
>
> The previous RTEMS SMP support required an IPI to start multitasking on secondary processors, but this was a broken concept.

Ok, I understand now. Thanks!

I've fixed so that Interrupt is now disabled on all CPUs. I will send this on the list next week, I was hoping to get something booting before I do..

But then I run into next problem dead lock -  where CPU0 lock up in _SMP_Handler_initialize() waiting for CPU1 to go into PER_CPU_STATE_READY_TO_BEGIN_MULTITASKING and CPU1 locks up in 
rtems_smp_secondary_cpu_initialize() waiting for PER_CPU_STATE_BEGIN_MULTITASKING. I will continue debugging this next week.

grmon2> bt

        %pc          %sp
   #0   0x4000e284   0x4ffffe60 <_SMP_Handler_initialize+0x88>
   #1   0x4000c8e0   0x4ffffec0 <rtems_initialize_data_structures+0xb8>
   #2   0x4000188c   0x4fffff20   <boot_card+0x20>
   #3   0x400010fc   0x4fffff80
grmon2> cpu act 1
grmon2> bt

        %pc          %sp
   #0   0x4000e1ec   0x400306c0 <_Per_CPU_Wait_for_state+0x10>
   #1   0x40002330   0x40030720 <leon3_secondary_cpu_initialize+0x64>
   #2   0x4000106c   0x40030780
grmon2> addr2line 0x40002330
/opt/rtems-4.11/src/rtems-4.11/c/src/lib/libbsp/sparc/leon3/smp/smp_leon3.c:51
grmon2> addr2line 0x4000e284
/opt/rtems-4.11/src/rtems-4.11/c/src/../../cpukit/score/src/percpu.c:91
grmon2> addr2line 0x4000e1ec
/opt/rtems-4.11/src/rtems-4.11/c/src/../../cpukit/score/src/percpu.c:91



>
>> But we need interrupts
>> enabled in order for IPIs to work?
>>
>> Or are you saying that there is an endless loop waiting for CPU0 to signal
>> multitasking via shared memory (not through IPI)? In that case I could just
>> turn of secondary CPU interrupt.
>
> Yes, exactly.
>
> However, if you send an IPI from CPU0 to CPU1 and interrupts are disabled on CPU1, then once CPU1 enables interrupts this IPI must be serviced.

I guess there is some kind of buffering going on then if multiple IPIs are being sent/received during receiver CPU has turned of interrupt. I will look into this my self.

Daniel