libbsd network stack optimization tips & tricks

Jonathan Brandmeyer jbrandmeyer at
Tue May 7 16:39:15 UTC 2019

On Mon, May 6, 2019 at 12:16 AM Sebastian Huber
<sebastian.huber at> wrote:
> Some known problems are:
> 1. RTEMS is supposed to be a real-time operating system and this has
> some implications for the mutex synchronization primitives. In RTEMS an
> atomic read-modify-write operation is used in the fast path mutex obtain
> and release operations. Linux and FreeBSD provide only random fairness
> (Futex) or no fairness at all, but can use only one atomic
> read-modify-write operation in the fast path.
> 2. The NETISR(9) service is currently disabled in libbsd:
> 3. If you use TCP, then a lock contention in the TCP socket locks may
> appear. This leads to costly priority inheritance operations.

Is there a fast path available here?  Ie, can we reduce the cost of
priority inheritance by controlling the relative priorities of the
IRQS and the receiving task?  For our application, we have a
well-defined separation between soft realtime code and hard realtime
code.  Network I/O is on the soft side of the line, so there's some
flexibility in setting task priorities.

> 4. There is no zero copy receive available (it is just a matter of time
> to implement it).

Are RTEMS zero-copy sockets also based on the stack in FreeBSD?
zero_copy_sockets(9) requires buffer sizes to be at least a whole page
large.  This NIC does not support jumbo frames.  Is RTEMS also
developing zero-copy receive for standard frame sizes?  RTEMS doesn't
use the MMU for much more than access control today, so will it simply
remove the page-based restriction entirely?

> I would try to use UDP and make sure your packet consumer runs on the
> same processor as the IRQS server task. Maybe implement a zero copy
> receive as well.

We're using TCP's flow control to match the data flow rate to the
hardware's capabilities.  I'd rather avoid implementing manual flow
control of a UDP stream if I can avoid it.

There are two IRQS tasks active.  Its not obvious which core ends up
with which GEM's interrupts.

Jonathan Brandmeyer

More information about the users mailing list