libbsd network stack optimization tips & tricks

[Sebastian Huber]

On 07/05/2019 18:39, Jonathan Brandmeyer wrote:
> On Mon, May 6, 2019 at 12:16 AM Sebastian Huber
> <sebastian.huber at embedded-brains.de> wrote:
>> Some known problems are:
>>
>> 1. RTEMS is supposed to be a real-time operating system and this has
>> some implications for the mutex synchronization primitives. In RTEMS an
>> atomic read-modify-write operation is used in the fast path mutex obtain
>> and release operations. Linux and FreeBSD provide only random fairness
>> (Futex) or no fairness at all, but can use only one atomic
>> read-modify-write operation in the fast path.
>>
>> 2. The NETISR(9) service is currently disabled in libbsd:
>>
>> https://www.freebsd.org/cgi/man.cgi?format=html&query=netisr(9)
>>
>> 3. If you use TCP, then a lock contention in the TCP socket locks may
>> appear. This leads to costly priority inheritance operations.
> Is there a fast path available here?  Ie, can we reduce the cost of
> priority inheritance by controlling the relative priorities of the
> IRQS and the receiving task?  For our application, we have a
> well-defined separation between soft realtime code and hard realtime
> code.  Network I/O is on the soft side of the line, so there's some
> flexibility in setting task priorities.

You can try to set the priority of the receiving task to the one of the 
IRQS and set its processor affinity to processor 0.

>
>> 4. There is no zero copy receive available (it is just a matter of time
>> to implement it).
> Are RTEMS zero-copy sockets also based on the stack in FreeBSD?
> zero_copy_sockets(9) requires buffer sizes to be at least a whole page
> large.  This NIC does not support jumbo frames.  Is RTEMS also
> developing zero-copy receive for standard frame sizes?  RTEMS doesn't
> use the MMU for much more than access control today, so will it simply
> remove the page-based restriction entirely?
>
> http://nixdoc.net/man-pages/FreeBSD/zero_copy_sockets.9.html

Since there is no user space separation in RTEMS you can directly use 
the mbufs in your application. See test zerocopy01.

>
>> I would try to use UDP and make sure your packet consumer runs on the
>> same processor as the IRQS server task. Maybe implement a zero copy
>> receive as well.
> We're using TCP's flow control to match the data flow rate to the
> hardware's capabilities.  I'd rather avoid implementing manual flow
> control of a UDP stream if I can avoid it.
>
> There are two IRQS tasks active.  Its not obvious which core ends up
> with which GEM's interrupts.

In the latest RTEMS and libbsd it should be the IRQS on processor 0.

-- 
Sebastian Huber, embedded brains GmbH

Address : Dornierstr. 4, D-82178 Puchheim, Germany
Phone   : +49 89 189 47 41-16
Fax     : +49 89 189 47 41-09
E-Mail  : sebastian.huber at embedded-brains.de
PGP     : Public key available on request.

Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG.