Status of SMP Work

[Joel Sherrill]

> Hello,
> 
> I would like to summarize the current status of the SMP work that is 
> already done in RTEMS.  Please correct me if something is wrong or 
> missing. 

Let me start by summarizing the goals and state of the SMP work.

   GOAL: Simplest, functional, SMP RTEMS

This meant that architectural changes to RTEMS were kept to a minimum.
We didn't push on any design assumptions -- especially in the area of
critical sections. SMP changes should not propagate into every corner
of the source. This explains the preservation of symbol names like
_Thread_Executing and _Thread_Heir when those values moved into 
the per-CPU structure.

Second, this meant that we didn't develop a fancy SMP Scheduler.
We implemented a Simple SMP Scheduler which follows the scheduling
rules used by the Deterministic Priority Scheduler and extends those
rules to scheduling tasks across multiple cores. On a uniprocessor
system, this scheduler should (and did) behave exactly like the 
Deterministic Priority Scheduler but slower. The Scheduler Framework
Gedare developed for GSOC allows a future projecct to write other
schedulers which are better for O(n) or scale to more cores efficiently.

This goal also meant that any feature which was not critical
was NOT implemented. This put items like processor affinity in
the future project bucket. The Simple SMP Scheduler can be easily
modified to support this but there is API work to make this available
to the application developer.

I personally felt it was important to provide a tool to help develop
and debug schedulers. The Scheduler Simulator uses a core set of
RTEMS plus some of the shell to provide a script driven way to
run a variety of scenarios on a prototype scheduler on any number
of cores while debugging on the host.

When OAR quit working on the SMP, the LEON3 and pc386 were executing
all smptests using grsim and qemu, respectively. We had 100% instruction
coverage. I do not recall the branch coverage but recall it was close
to 100% if not also at that level.

Overall, this was a "foundation" or "first milestone" for RTEMS SMP 
which was in good shape.

> An interesting document about real-time systems on SMP is the 
> dissertation from Björn B. Brandenburg 
> (http://www.cs.unc.edu/~bbb/diss/).  He mentions also RTEMS as a 
> classical uni-processor real-time system.

This is a true statement about RTEMS up to the addition of the
current SMP support being a classical RTOS.

 
> 1. We have now per-CPU information so that some state can be maintained 
> on a per-CPU basis:
> 
> http://git.rtems.org/rtems/tree/cpukit/score/include/rtems/score/percpu.h
> 
> Access to several global variables has been encapsulated with a function.

This is all the information needed on a per-CPU basis in the current
implementation. The wrapper methods were added to avoid propagating 
changes into other places in the source code. The following shows
how we kept the name _Thread_Executing while moving the actual value
into the per-CPU structure.
 
#define _Thread_Executing \
  _Per_CPU_Information[bsp_smp_processor_id()].executing

When built for uniprocessor configurations, bsp_smp_processor_id() is
defined as 0, so this becomes a direct reference like now but more
complicated to the compiler. There should be no change in the generated
assembly code.

 
> 2. We have now a configurable scheduler (examples are FP, EDF, CBS) 
> which makes it possible to develop SMP capable scheduler more easily.

There is an SMP Scheduler in the current tree.
 
> 3. We have a first support for spin locks. The current implementation is 
> not suitable for real-time applications:
> 
> http://git.rtems.org/rtems/tree/cpukit/score/src/smplock.c
> 
> Here interrupts are disabled before the loop and thus the disable 
> interrupt time on this core is unbounded.  We need also a lock with FIFO 
> properties like a MCS lock.

Yes. We did not change the critical section model in RTEMS. We just adapted
it to SMP. Working was better than optimal.

This was left as future work. Getting this right would be tricky and
we didn't even have SMP working.
 
> 4. Support for atomic operations is a current GSoC project and now also 
> part of the C and C++ standard.

Yes.

> 5. There are a couple of PRs with an unknown status, like
> 
> https://www.rtems.org/bugzilla/show_bug.cgi?id=1946
> https://www.rtems.org/bugzilla/buglist.cgi?quicksearch=smp

PR1946 is a "special" one. The submitter is from a company with their
own CPU. We do not have access to the CPU, tools, or simulator. They
did their own asymmetric MP port with no community interaction until
OAR had committed a large part of the SMP work. They were based on
an older version, had trouble updating their patch, and submitted
this mega patch.

Looking at the second link, it shows that Marta did break it into
pieces and submitted a handful of PRs with nicely sized patches
which need to be reviewed and merged.  I am embarrassed to say
that I honestly didn't know Marta had broken them into pieces
and submitted the other PRs. Those need review and should be
a priority.

If you can help review and merge these, it would be appreciated.
I need to apologize to Marta. Looking at the date of these, I think
I was travelling when he submitted them and I just missed them. :(

They had/have RTEMS running on a 16-core system in AMP mode. I 
believe their work is valuable and their patch needs review and
merger.

> 6. It seems that in the current SMP work the sections covered by 
> _Thread_Disable_dispatch() and _Thread_Enable_dispatch() produce a 
> critical section that is protected by a giant lock:
> 
> http://git.rtems.org/rtems/tree/cpukit/score/src/threaddispatchdisablelevel.c
> 
> These critical sections are pretty huge and I don't think this scales 
> well with an increasing processor count. Why was it not possible to move 
> the thread dispatch disable level into the per-CPU information and use 
> _ISR_Disable()/_ISR_Enable() as the giant lock?  This has the benefit 
> that we can probably use existing interrupt and exception code.  It 
> makes it also possible to gradually replace the 
> _ISR_Disable()/_ISR_Enable() sections with more specific locks, e.g. one 
> for a basic semaphore operation and one for the general blocking path.  
> I think only thread restart and delete operations are problematic here.  
> Did I miss something important?  What lead to the decision of a global 
> thread dispatch disable level?

Perfect is the enemy of working. :)

I agree. We wanted functional and simple. This can be improved once
it is stable in the current implementation. We simply did not have
the funding to tackle all these issues.

I had a simple Work Break Down for the SMP with what I saw as future
work included. I can update that and get it to you if you like.
 
> -- 
> Sebastian Huber, embedded brains GmbH
> 
> Address : Obere Lagerstr. 30, D-82178 Puchheim, Germany
> Phone   : +49 89 18 90 80 79-6
> Fax     : +49 89 18 90 80 79-9
> E-Mail  : sebastian.huber at embedded-brains.de
> PGP     : Public key available on request.
> 
> Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG.
> 
> _______________________________________________
> rtems-devel mailing list
> rtems-devel at rtems.org
> http://www.rtems.org/mailman/listinfo/rtems-devel
>