Questions about MBUF allocation in bsdnet

[Chris Johns]

On 23/03/2015 4:13 am, Albert Huang wrote:
> Hi, all,
>
> During developing/tracing an Ethernet device driver for ARM, I also
> setup an i386 RTEMS on qemu with NE2000 as a comparison. I found that
> MBUF allocation on both hosts does not behave the same, and might be one
> of the source of issues. One of the strange behavior is that MSIZE=128
> on both hosts, 98-byte packets are not segmented on i386 while segmented
> on ARM.

Each architecture has different alignment constraints and on top of this 
a specific driver can require further constraints so the in-memory 
layout of a specific packet can vary between difference types of hardware.

> Another issue is sometimes M_LEADINGSPACE for 84-byte/98-byte
> packet get 0 or negative.

This is the space remaining in a cluster and is used when deciding on 
pre-pending protocol headers to the user's data.

>
> In order to clarify these issues, I would like to know more about
> MBUF. So my questions about MBUF are:
>
> 1. Is MSIZE adjustable? Currently, it is set to 128, and I couldn't find
> where to modify it. Do I need to modify it to support larger Ethernet
> segment size?

It is fixed and in mbuf.h. I strongly suggest you do not touch this 
value as it is tuned for the stack we currently have.

>
> 2. What does M_EXT mean exactly? They're different on both hosts as
> well.

The TCP/IP Illustrated Vol 2 says it is set when an external buffer or 
cluster is used.

>
> 3. If a packet size is larger than MSIZE, would MCLGET() gives me two
> MBUF, or a "cluster," which means a larger chunk of MBUF's, which are
> grouped and treated as one MBUF.

It depends on pre-pending or appending data, if you already have 
available space and how much data you are looking to add.

The idea is not to copy data into new buffers each time you change the 
size. When user data is placed in an mbuf and/or cluster at the socket 
layer the amount of data to pre-pended is not known. As the data heads 
down to the driver various pieces are added and rather than copy the 
data to a new larger buffer the stack links a buffer to the start of the 
list of buffers. This "scatters" the data around the memory. The driver 
"gathers" this data as it sends it down the wire.

> Am I correct on this part? Then is it
> normal that I saw 98-byte packet segmented into two Etherent segments?
>
> I would appreciate any idea/suggestion. Thanks in advance.
>

I suggest you look around for some documentation on the topic of mbufs 
in BSD networking software. One of the advantages of RTEMS using the 
FreeBSD stack is the available documentation. Steven's book "TCP/IP 
Illustrated Vol 2" is showing its age but it is still worth a read.

Chris