POSIX Mutex Performance

[Sergei Organov]

Joel Sherrill <joel at OARcorp.com> writes:
>
> Hi,
>
> I have been thinking about this one. The biggest thing is the first one.
> Other ideas and comments follow.
>
> + We mentioned this problem earlier but now I think I
>    can add some meat.  The RTEMS and Linux mutexes are
>    different and have different feature sets.  RTEMS supports
>    the priority inheritance and ceiling protocols.  I do not
>    see any hint of that in the Linux pthread mutex code.

Well, this indeed could be the cause of the difference, but then how do you
explain almost the same difference between RTEMS and Linux in semaphore times?

>    The default case for RTEMS and Linux  mutexes are different.
>    Linux picks a "fast" case which performs no error checking
>    and will allow you to deadlock.  RTEMS always error checks
>    and has attibutes on the base mutex whether nesting is legal
>    or an error.  Linux will let you DEADLOCK in the default
>    case!!

And what's a problem about it? I always thought that existence of a bug in a
program should be made obvious as soon as possible. Personally, I prefer to
get immediate deadlock instead of some subtle misbehavior later on, though the
best thing would be to get an immediate exception.

Most status codes returned by RTEMS calls are useful only for debugging and
thus could be replaced by some exception mechanism as in most cases
application has no clue what to do about the returned error codes and just
calls fatal handler anyway. But to call the handler it pays for checking for
error condition, and it pays for conversion between core status codes and
interface error codes even if it doesn't check for errors.

In fact I believe the real cause of the difference (along with the opaque ID
instead of pointers discussed later) is that RTEMS checks at run time for
things that are usually known at compile time. Applied to mutexes, it's
usually known at compile time if nesting and/or priority inheritance/ceiling
is required for given mutex. And in many cases some support is known to be
globally unused by the application, e.g., even if I know that I never
recursively acquire mutexes, I still pay for the ability to do that, so I
think fine-grained configurability is one of directions where RTEMS could
advance. 

Specifically for mutexes, IMHO, the pretender number 1 for unconditional
elimination from the core mutex code is support for 'simple binary semaphore'.
The simple binary semaphore could be easily implemented as core counting
semaphore with the limit for the counter set to 1. Only few lines in the
'rtems_semaphore_initialize' required as the limit for counter is already
there. I believe it will allow not only to remove "non-owner may unlock" check
but also to remove interrupts disabling on mutex operations as real mutexes
should never be locked/unlocked from ISRs.

>
>    RTEMS is a real-time operating system and wants to
>    make the application's execution predictable.  This
>    means providing tools to detect deadlock, avoid priority
>    inversion, have application limits on resource usage, etc.
>    Linux wasn't designed to meet those goals and the feature
>    set in this area shows that.

Providing tools is a good thing but it doesn't necessarily mean all of them
should always sit in the code eating processor cycles. It would be optimal if
features that take noticeable resources could be selected or turned on/off at
compile time.

Besides, detecting deadlocks is not the same as preventing them. Linux is
equally well equipped for detecting deadlocks, and, on the other hand, I don't
believe responding to an error code from RTEMS call meaning "would deadlock"
in an application is the right way to prevent deadlocks.

>
> + The Classic API Mutexes (rtems_semaphore*) are a bit more
>    optimized.  For sure, the POSIX API goes through a wrapper
>    function which could technically be avoided to save a few
>    instructions.

That's questionable as unlike POSIX API classic API needs to dispatch between
core mutexes and core semaphores at run-time, and POSIX API uses only one
level of wrapping (unlock even use 0 levels of wrapping).

>
> + One person on the list noticed that your Linux times varied
>    fairly significantly between two reports and I did not
>    seem the explanation.  You might want to doublecheck the
>    timing mechanism using something similar to the procedure
>    in the tmck check.

The OP already replied that results for those two reports have been obtained
on different hardware. Timing tests are known to be subtle so the OP results
could well be wrong anyway.

>
> + RTEMS and glibc pthreads have different design approaches
>    which impacts the create/destroy times.  RTEMS uses an
>    ID which is opaque and thus even if dereferenced by the
>    user won't harm the associated OS memory.  The Linux pthread
>    code returns direct pointers to the user.  The pointer
>    approach is a bit faster but not as safe/robust.

As RTEMS memory has no protection from user threads anyway, the real
advantage of the approach taken in the RTEMS is very questionable.

At least I always miss ability to statically allocate RTEMS resources instead
of guessing required size of work area and paying bytes of memory for dynamic
allocation.

>    The other thing to note with this approach is that RTEMS explicit
>    initializes every field and reuses a user configured and
>    finite set of objects.  These have to be reinitialized every
>    use.  So create and destroy is going to be more expensive.

I don't think this is a real problem for real-time applications as they tend
to don't allocate/destroy resources frequently or inside time-critical paths.

[...]

> A mutex is a very well understood OS object and assuming
> that everyone implemented it well, then the differences are
> going to be in default behavior selected, implementation
> structure/overhead, safety checks, and the like.

Well, but RTEMS core for whatever reason already mixes the well understood
mutex object with another well understood OS object called semaphore. RTEMS
also mixes mutexes and semaphores in the Classic API, but that could be
excused by the RTEMS history that longs from the days when mutex has not been
identified as a separate well understood OS object. Or do we in fact
understand mutex object differently while equally "well" :)?

--
Sergei.