Scheduler Set Affinity Design Thoughts

[Joel Sherrill]

On 5/19/2014 1:34 AM, Sebastian Huber wrote:
> On 2014-05-16 19:48, Joel Sherrill wrote:
>> Hi
>>
>> Questions first:
>>
>> + The affinity mask must be non-empty. This means a thread must
>> be able to be scheduled on some processor.  With cluster scheduling,
>> a scheduler implementation would only be associated with a subset
>> of processors. The affinity can't be set such that the thread can't be
>> scheduled by this scheduler instance.
> Ok.
>
>> How can a scheduler instance
>> know which processors it is associated with?
> You can store this information in the scheduler context if it needs this 
> information quickly or you have iterate though the scheduler assignments:
I started to implement some validation but then looked back at
_Scheduler_Set_affinity() and it only invokes the scheduler specific
set affinity if it is in the cpu set. So it is checked out before you
get to the operation.

But, I think blindly invoking the first scheduler instance that
the thread has affinity for is dangerous and incorrect.

I do not think that _Scheduler_Set_affinity() properly
addresses the case where the old affinity set is on one scheduler
instance and the new set moves it to another. The source scheduler
doesn't appear to get a chance to remove it from its bookkeeping.

It is pretty clear that _Scheduler_Set_affinity() doesn't address this
because it blindly invokes the first scheduler instance the new
affinity matches without concern over whether that was the previous
schedule instance.

The scheduler specific affinity routine can't address this because it
won't see it leaving.

I repeat that I really don't think changing affinity should move a
thread from one scheduler instance to another.  It is non-obvious
behavior and has some subtle side-effects that need to be managed.
_Scheduler_Set_affinity() could enforce this.

Moving schedulers and changing affinity should be rare and very
explicit operations.
> RTEMS_INLINE_ROUTINE bool _Scheduler_default_Set_affinity_body(
>    const Scheduler_Control *scheduler,
>    Thread_Control          *the_thread,
>    size_t                   cpusetsize,
>    const cpu_set_t         *cpuset
> )
> {
>    size_t   cpu_max   = _CPU_set_Maximum_CPU_count( cpusetsize );
>    uint32_t cpu_count = _SMP_Get_processor_count();
>    uint32_t cpu_index;
>    bool     ok = true;
>
>    for ( cpu_index = 0 ; cpu_index < cpu_count ; ++cpu_index ) {
> #if defined(RTEMS_SMP)
>      const Scheduler_Control *scheduler_of_cpu =
>        _Scheduler_Get_by_CPU_index( cpu_index );
>
>      ok = ok
>        && ( ( CPU_ISSET_S( (int) cpu_index, cpusetsize, cpuset )
>            && scheduler == scheduler_of_cpu )
>          || ( !CPU_ISSET_S( (int) cpu_index, cpusetsize, cpuset )
>            && scheduler != scheduler_of_cpu ) );
> #else
>      (void) scheduler;
>
>      ok = ok && CPU_ISSET_S( (int) cpu_index, cpusetsize, cpuset );
> #endif
>    }
>
>    for ( ; cpu_index < cpu_max ; ++cpu_index ) {
>      ok = ok && !CPU_ISSET_S( (int) cpu_index, cpusetsize, cpuset );
>    }
>
>    _Scheduler_Set( scheduler, the_thread );
>
>    return ok;
> }
>
>
>> + Similarly, changing scheduler for a thread dynamically should not
>> allow the above to be violated. The thread must have affinity for one
>> or more of the cores. How can this be done?
> See _Scheduler_default_Set_affinity_body().
>
>> Design thoughts:
>>
>> + Change priority is very careful to avoid rescheduling an executing
>> thread. Is this optimization worth the trouble for affinity? I really have
>> trouble seeing it be worth the trouble. Changing priority is fairly
>> common thanks to priority inheritance and ceiling. Changing affinity
>> should be pretty uncommon since I think it is an application design
>> parameter.
> I think we have two use cases for the set affinity operation.
>
> 1. To select the scheduler instance.
>
> 2. To define the affinity map inside a scheduler instance.
>
> The second operation may be used more frequently.  If this can be done 
> efficiently depends on the overall implementation of the arbitrary thread 
> processor affinity support.
>
>> + If this optimization is not worth the trouble, then
>> _Scheduler_Set_affinity()
>> should be able to do something like this like when you change the
>> scheduler in _Scheduler_Set():
>>
>>      validate new affinity     // includes can run on subset for this
>> scheduler
>>      if ! changed return
>>        _Thread_Set_state( the_thread, STATES_MIGRATING );
>>          update affinity
>>      _Thread_Clear_state( the_thread, STATES_MIGRATING );
>>
>> Comments?
>>
> Yes, this is how I would do it.  Should interrupt service routines be able to 
> change the scheduler or the affinity set?
>

-- 
Joel Sherrill, Ph.D.             Director of Research & Development
joel.sherrill at OARcorp.com        On-Line Applications Research
Ask me about RTEMS: a free RTOS  Huntsville AL 35805
Support Available                (256) 722-9985