Executing Thread Migrating Due to Affinity Change

[Sebastian Huber]

On 05/29/2014 09:28 PM, Joel Sherrill wrote:
> Hi
>
> The priority affinity algorithm appears to be behaving as
> we expect from a decision making standpoint. However,
> Jennifer and I think that when a scheduled thread must
> be migrated to another core, we have a case for a new
> state in the Thread Life Cycle.

I hope we can avoid this.  Which problem do you want to address with 
this new state?

>
> I am thinking that the thread needs to have a blocking state
> set, have its context saved and be taken out of the scheduled
> set. Then a life cycle state change handler van run as an
> extension to unblock it so it can be potentially scheduled to
> execute on another processor.

The scheduler is not responsible for the thread context.  This is 
_Thread_Dispatch().  A post-switch handler can only do actions for the 
executing thread.  It would be extremely difficult to perform actions on 
behalf of another thread.  We have to keep also fine grained locking 
into account.  The _Thread_Dispatch() function is on the critical path 
for average and worst-case performance, so we should keep it as simple 
as possible.

As I wrote already in another thread you can use something like this to 
allocate an exact processor for a thread:

static inline void _Scheduler_SMP_Allocate_processor_exact(
   Scheduler_SMP_Context *self,
   Thread_Control *scheduled,
   Thread_Control *victim
)
{
   Scheduler_SMP_Node *scheduled_node = _Scheduler_SMP_Node_get( 
scheduled );
   Per_CPU_Control *cpu_of_scheduled = _Thread_Get_CPU( scheduled );
   Per_CPU_Control *cpu_of_victim = _Thread_Get_CPU( victim );
   Per_CPU_Control *cpu_self = _Per_CPU_Get();

   _Scheduler_SMP_Node_change_state(
     scheduled_node,
     SCHEDULER_SMP_NODE_SCHEDULED
   );

   _Thread_Set_CPU( scheduled, cpu_of_victim );
   _Scheduler_SMP_Update_heir( cpu_self, cpu_of_victim, scheduled );
}

You can even use this function to do things like this:

_Scheduler_SMP_Allocate_processor_exact(self, executing, other);
_Scheduler_SMP_Allocate_processor_exact(self, other, executing);

This works because the is executing indicator moved to the thread 
context and is maintained at the lowest context switch level.  For 
proper migration the scheduler must ensure that,

1. an heir thread other than the migrating thread exists on the source 
processor, and

2. the migrating thread is the heir thread on the destination processor.

-- 
Sebastian Huber, embedded brains GmbH

Address : Dornierstr. 4, D-82178 Puchheim, Germany
Phone   : +49 89 189 47 41-16
Fax     : +49 89 189 47 41-09
E-Mail  : sebastian.huber at embedded-brains.de
PGP     : Public key available on request.

Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG.