Network buffers, MPC860 and data caching

[Till Straumann]

Eric Norum wrote:

> Till Straumann wrote:
> >
> > Charles-Antoine Gauthier wrote:
> >
> > >
> > > > Have a look at Charles Gauthier's & Darlene Steward's MBX BSP in the
> > > > current snapshot. It goes a different route: All RAM is cacheable,
> > > > however, once CPM buffers (not only for network) are touched, explicit
> > > > cache commands for the memory covered by the buffer are used:
> > > >
> > > >    - on output buffers, caches are flushed after writing and before
> > > >      setting the ready bit
> > > >    - on read, the cache lines for the buffer are invalidated before
> > > >      reading the buffer
> > > >
> >
> > Ok, that's what I want to do on the MPC860 powerpc FADS board.
> > I thought about how to implement this because I saw that the network and
> > console drivers use `malloc()ed' buffers. This complicates things because,
> > obviously, receive buffers must be cache line aligned. AFAIK network
> > receiving buffers are allocated by the driver, passed upstream and are
> > also deallocated upstream.
> >
> > Eventually, I decided to download the latest snapshot and have a look at
> > your implementation. I could see that your  `rtems_cache_aligned_malloc()'
> > looks pretty much like what was my first idea.
> > However, (and that's the problem stumbled over), how does `free()' like
> > the modified address returned by `rtems_cache_aligned_malloc'()? I'm
> > afraid not very much - it seems to me that free()ing
> > cache_aligned_malloc()ed memory results in heap corruption :-)
> >
>
> Network receive buffers are allocated by MCLGET, not by malloc.  An mbuf
> cluster is aligned on a cluster-sized (2 kbyte) boundary.  Network mbufs
> are aligned on an mbuf-sized (128 byte) boundary.  See bsd_init in
> src/libnetworking/rtems/rtems_glue.c.  Unless a cache line is larger
> than an mbuf there should be no problems.

Sorry, I don't quite agree.

The point is that the `databuf' starting address, i.e. where the network
interface
chip writes data must be cache line aligned. If I count the fields in m_hdr and
pkthdr,
I get 7*4 bytes, i.e. the data area is seems to be 4byte aligned. Assuming that
nobody
reads any field beyond mh_data while the network interface `owns' the buffer,
the
first cache line could be flushed before yielding the buffer to the network
driver. But this will only
be enough on a machine with a cache line size <= 5*4 = 20 bytes, e.g. the MPC
860.

Am I missing something?

Another idea: is it possible to use the M_EXT facility to provide an aligned
area
(and a corresponding deallocation routine)?
Is it legal to change the flags and the extension record on an mbuf obtained
using
MGETHEADER()?

>
>
> Console receive buffers are 16 bytes long and are statically allocated,
> but there's no attempt to cache-line-align them.  This may be a
> problem.  Charles?

Looking at mpc860/console-generic/console-generic.c it seems that the data area

is malloc()ed and free()ed locally, in the driver. While not very elegant, it
would be
possible to use a cache_aligned memory management  engine in this file.
However,
this would only solve the issue for the MPC8xx.

>
>
> I agree with your concern about passing a pointer obtained from
> rtems_cache_aligned_malloc to free.
> The snapshot I'm working with (rtems-ss-20000929) has *no* calls to
> rtems_cache_aligned_malloc, though, so there's no possibility of bad
> pointers getting passed to free().
>
> BTW -- I see you're from SLAC.  Are you considering using EPICS with
> RTEMS?

Indeed, that's what I'm envisioning. Are you going to OakRidge in November?
I hope to learn more at the collaboration meeting about the efforts to untackle

EPICS' OS-dependencies, RTEMS etc. It'd be nice talking to you about these
issues.

Regards, Till.