SMP boot sequence
daniel at gaisler.com
Fri Oct 25 13:35:09 UTC 2013
On 10/25/2013 02:40 PM, Sebastian Huber wrote:
> On 2013-10-25 14:23, Daniel Hellstrom wrote:
>>> Interrupts are enabled in _Thread_Handler() the first time.
>>>> However on a SMP machine the secondary CPUs have enabled global
>>>> CPU interrupt in order to receive IPIs during the boot sequence.
>>> This is a bug. Interrupts must be disabled during the low-level boot
>>> process. Only _Thread_Handler() should enable the interrupts. However the
>>> secondary processors must be able to notice an IPI and serve it right after
>>> the interrupts are enabled the first time.
>> Interrupts are off on CPU0 during boot.
>> I'm not sure I follow this 100%. During low-level boot interrupt should be off
>> on all CPUs, _Thread_Handler() enables the current CPU's interrupt.
> Yes, interrupts must be disabled on all CPUs during low-level start.
>> So how
>> should a secondary core receive an IPI, I mean an IPI is required to get to
>> _Thread_Handler() in order to turn on interrupts.
> No, this is not the case. Secondary cores busy wait for a per CPU state change:
> void rtems_smp_secondary_cpu_initialize( void )
> Per_CPU_Control *self_cpu = _Per_CPU_Get();
> #if defined(RTEMS_DEBUG)
> printk( "Made it to %d -- ", _Per_CPU_Get_index( self_cpu ) );
> _Per_CPU_Change_state( self_cpu, PER_CPU_STATE_READY_TO_BEGIN_MULTITASKING );
> _Per_CPU_Wait_for_state( self_cpu, PER_CPU_STATE_BEGIN_MULTITASKING );
> _Thread_Start_multitasking( NULL );
> The previous RTEMS SMP support required an IPI to start multitasking on secondary processors, but this was a broken concept.
Ok, I understand now. Thanks!
I've fixed so that Interrupt is now disabled on all CPUs. I will send this on the list next week, I was hoping to get something booting before I do..
But then I run into next problem dead lock - where CPU0 lock up in _SMP_Handler_initialize() waiting for CPU1 to go into PER_CPU_STATE_READY_TO_BEGIN_MULTITASKING and CPU1 locks up in
rtems_smp_secondary_cpu_initialize() waiting for PER_CPU_STATE_BEGIN_MULTITASKING. I will continue debugging this next week.
#0 0x4000e284 0x4ffffe60 <_SMP_Handler_initialize+0x88>
#1 0x4000c8e0 0x4ffffec0 <rtems_initialize_data_structures+0xb8>
#2 0x4000188c 0x4fffff20 <boot_card+0x20>
#3 0x400010fc 0x4fffff80
grmon2> cpu act 1
#0 0x4000e1ec 0x400306c0 <_Per_CPU_Wait_for_state+0x10>
#1 0x40002330 0x40030720 <leon3_secondary_cpu_initialize+0x64>
#2 0x4000106c 0x40030780
grmon2> addr2line 0x40002330
grmon2> addr2line 0x4000e284
grmon2> addr2line 0x4000e1ec
>> But we need interrupts
>> enabled in order for IPIs to work?
>> Or are you saying that there is an endless loop waiting for CPU0 to signal
>> multitasking via shared memory (not through IPI)? In that case I could just
>> turn of secondary CPU interrupt.
> Yes, exactly.
> However, if you send an IPI from CPU0 to CPU1 and interrupts are disabled on CPU1, then once CPU1 enables interrupts this IPI must be serviced.
I guess there is some kind of buffering going on then if multiple IPIs are being sent/received during receiver CPU has turned of interrupt. I will look into this my self.
More information about the devel