libbsd network stack optimization tips & tricks
jbrandmeyer at planetiq.com
Tue May 7 16:39:15 UTC 2019
On Mon, May 6, 2019 at 12:16 AM Sebastian Huber
<sebastian.huber at embedded-brains.de> wrote:
> Some known problems are:
> 1. RTEMS is supposed to be a real-time operating system and this has
> some implications for the mutex synchronization primitives. In RTEMS an
> atomic read-modify-write operation is used in the fast path mutex obtain
> and release operations. Linux and FreeBSD provide only random fairness
> (Futex) or no fairness at all, but can use only one atomic
> read-modify-write operation in the fast path.
> 2. The NETISR(9) service is currently disabled in libbsd:
> 3. If you use TCP, then a lock contention in the TCP socket locks may
> appear. This leads to costly priority inheritance operations.
Is there a fast path available here? Ie, can we reduce the cost of
priority inheritance by controlling the relative priorities of the
IRQS and the receiving task? For our application, we have a
well-defined separation between soft realtime code and hard realtime
code. Network I/O is on the soft side of the line, so there's some
flexibility in setting task priorities.
> 4. There is no zero copy receive available (it is just a matter of time
> to implement it).
Are RTEMS zero-copy sockets also based on the stack in FreeBSD?
zero_copy_sockets(9) requires buffer sizes to be at least a whole page
large. This NIC does not support jumbo frames. Is RTEMS also
developing zero-copy receive for standard frame sizes? RTEMS doesn't
use the MMU for much more than access control today, so will it simply
remove the page-based restriction entirely?
> I would try to use UDP and make sure your packet consumer runs on the
> same processor as the IRQS server task. Maybe implement a zero copy
> receive as well.
We're using TCP's flow control to match the data flow rate to the
hardware's capabilities. I'd rather avoid implementing manual flow
control of a UDP stream if I can avoid it.
There are two IRQS tasks active. Its not obvious which core ends up
with which GEM's interrupts.
More information about the users