libbsd network stack optimization tips & tricks

[Jonathan Brandmeyer]

On Mon, May 6, 2019 at 12:16 AM Sebastian Huber
<sebastian.huber at embedded-brains.de> wrote:
>
> Some known problems are:
>
> 1. RTEMS is supposed to be a real-time operating system and this has
> some implications for the mutex synchronization primitives. In RTEMS an
> atomic read-modify-write operation is used in the fast path mutex obtain
> and release operations. Linux and FreeBSD provide only random fairness
> (Futex) or no fairness at all, but can use only one atomic
> read-modify-write operation in the fast path.
>
> 2. The NETISR(9) service is currently disabled in libbsd:
>
> https://www.freebsd.org/cgi/man.cgi?format=html&query=netisr(9)
>
> 3. If you use TCP, then a lock contention in the TCP socket locks may
> appear. This leads to costly priority inheritance operations.

Is there a fast path available here?  Ie, can we reduce the cost of
priority inheritance by controlling the relative priorities of the
IRQS and the receiving task?  For our application, we have a
well-defined separation between soft realtime code and hard realtime
code.  Network I/O is on the soft side of the line, so there's some
flexibility in setting task priorities.

>
> 4. There is no zero copy receive available (it is just a matter of time
> to implement it).

Are RTEMS zero-copy sockets also based on the stack in FreeBSD?
zero_copy_sockets(9) requires buffer sizes to be at least a whole page
large.  This NIC does not support jumbo frames.  Is RTEMS also
developing zero-copy receive for standard frame sizes?  RTEMS doesn't
use the MMU for much more than access control today, so will it simply
remove the page-based restriction entirely?

http://nixdoc.net/man-pages/FreeBSD/zero_copy_sockets.9.html

> I would try to use UDP and make sure your packet consumer runs on the
> same processor as the IRQS server task. Maybe implement a zero copy
> receive as well.

We're using TCP's flow control to match the data flow rate to the
hardware's capabilities.  I'd rather avoid implementing manual flow
control of a UDP stream if I can avoid it.

There are two IRQS tasks active.  Its not obvious which core ends up
with which GEM's interrupts.

-- 
Jonathan Brandmeyer