Network buffers, MPC860 and data caching

[Charles-Antoine Gauthier]

Eric Norum wrote:
> 
> Till Straumann wrote:
> >
> > Eric Norum wrote:
> > > Network receive buffers are allocated by MCLGET, not by malloc.  An mbuf
> > > cluster is aligned on a cluster-sized (2 kbyte) boundary.  Network mbufs
> > > are aligned on an mbuf-sized (128 byte) boundary.  See bsd_init in
> > > src/libnetworking/rtems/rtems_glue.c.  Unless a cache line is larger
> > > than an mbuf there should be no problems.
> >
> > Sorry, I don't quite agree.
> >
> > The point is that the `databuf' starting address, i.e. where the network
> > interface
> > chip writes data must be cache line aligned. If I count the fields in m_hdr and
> > pkthdr,
> > I get 7*4 bytes, i.e. the data area is seems to be 4byte aligned. Assuming that
> > nobody
> > reads any field beyond mh_data while the network interface `owns' the buffer,
> > the
> > first cache line could be flushed before yielding the buffer to the network
> > driver. But this will only
> > be enough on a machine with a cache line size <= 5*4 = 20 bytes, e.g. the MPC
> > 860.
> 
> ``What we have here is a failure to communicate.''
> 
> We must be looking at different source.  The
> c/src/lib/libcpu/powerpc/mpc8xx/console-generic/console-generic.c in the
> snapshot I've got here (rtems-ss-20000929) has neither calls to malloc
> nor free.  The CVS ID line is:
> 
>   $Id: console-generic.c,v 1.5 2000/08/25 17:25:27 joel Exp $
> 
> The receive and transmit buffer areas are:
> /*
>  *  I/O buffers and pointers to buffer descriptors.
>  *  Currently, single buffered input is done. This will work only
>  *  if the Rx interrupts are serviced quickly.
>  *
>  *  TODO: Add a least double buffering for safety.
>  */
> static volatile char rxBuf[NUM_PORTS][RXBUFSIZE];
> static volatile char txBuf[NUM_PORTS];
> 
> I agree that there is a problem here since the rxBuf is not aligned on a
> cache boundary.  A cache line which overlapped the front or back of the
> rxBuf could overwrite the values placed in the rxBuf by the SDMA
> channel.  The txBuf isn't a problem, since extra flushes would just
> write the same values into txBuf as were placed there by the
> rtems_cache_flush_multiple_data_lines call in m8xx_uart_write.  A fix
> would be to statically allocate RXBUFSIZE+CACHE_LINESIZE-1 bytes for
> each receive buffer, and then to use only the cache-aligned portion of
> the buffer.
> 
> The network driver uses mbuf clusters for incoming packets.  These are
> aligned on 2k boundaries so there's no chance of a cache-line write
> scrambling a packet buffer.  Once a packet has been received, all the
> cache lines which refer to the data are marked invalid.
> 
> Hmm...But there *is* a problem here, too.  It's not related to cache
> line boundaries, though.
> 1) CPU allocates an mbuf cluster and does some writes and reads to the
> cluster.  The cache is in writeback mode, so main memory is not updated.
> 2) CPU frees the mbuf cluster.
> 3) Driver read task allocates that mbuf cluster.
> 4) DMA engine starts reading into the mbuf cluster.
> 5) CPU decides it wants to reuse the cache line in question and flushes
> it to main memory -- kaboom -- this overwrites the value stored by the
> DMA engine!
> 
> I think that this problem could be avoided by marking the cache lines
> associated with an mbuf cluster invalid *before* pasing the mbuf cluster
> to the DMA engine, instead of after the DMA engine has finished filling
> the cluster as is now the case..  The fix would be to call
> rtems_cache_invalidate_multiple_data_lines just after the MCLGET.  You
> wouldn't have to invalidate all 2kbytes, just the 1536 bytes needed to
> hold an ethernet frame.
> 
> For transmission, once the stack has passed an mbuf to the driver output
> routine, that mbuf is `owned' by the driver so the CPU won't be writing
> to the mbuf header after the driver does the
>       rtems_cache_flush_multiple_data_lines(txBd->buffer, txBd->length);
> All mbufs are aligned on a 128 byte boundary so activity on other mbufs
> can not cause a cache-line write to affect an mbuf which has been
> flushed to main memory.  Actually, it wouldn't make any difference even
> if activity on another mbuf followed by a cache line write *did* cause
> the mbuf in question to be written since the value written would be
> exacly the same as the value in main memory anyway. The DMA engine would
> still see the same value in main memory during the transmission.
> 
> It makes no difference if the txBd->buffer is not on a cache line
> boundary.  Sure a few bytes of mbuf header get written to main memory
> along with the mbuf data, but who cares?  The important thing is that
> the bytes from buffer to buffer+length are forced out to main memory
> where the DMA engine can see them.
> 
> Charles. can you fix the two problems mentioned above?  While you're at
> it, could you add support for the FEC so we can get 100baseT Ethernet?
> Joel, I wonder if other systems with caches have similar problems?

Not RSN. I don't expect to have time to look at anything related to
RTEMS until the end of November or beginning of December. Sorry. I will
add it to the list of things to do and can deal with it at that time if
no one has fixed it by then.

> 
> >
> > Another idea: is it possible to use the M_EXT facility to provide an aligned
> > area
> > (and a corresponding deallocation routine)?
> > Is it legal to change the flags and the extension record on an mbuf obtained
> > using
> > MGETHEADER()?
> 
> Yes, you could use M_EXT but I don't see the need -- at least once the
> changes mentioned above have been made.
> 
> > > BTW -- I see you're from SLAC.  Are you considering using EPICS with
> > > RTEMS?
> >
> > Indeed, that's what I'm envisioning. Are you going to OakRidge in November?
> > I hope to learn more at the collaboration meeting about the efforts to untackle
> >
> > EPICS' OS-dependencies, RTEMS etc. It'd be nice talking to you about these
> > issues.
> >
> 
> Unfortunately I can't make it to Oak Ridge.  Teaching commitments and
> starting a new job make the trip impossible.  An expense-paid trip to
> Palo Alto sometime to give a seminar or training course on EPICS/RTEMS
> would be ideal :-) but I'll certainly give you what support I can by
> e-mail.  I'd really like to see more places using RTEMS with EPICS.
> We've got a dozen or so IOC's (68360's) running equipment here and
> things seem to be working just great.  It's really convenient having a
> <$500 (Canadian!) IOC.  You can use them everywhere.
> 
> --
> Eric Norum                                 eric at cls.usask.ca
> Canadian Light Source                      Phone: (306) 966-6308
> University of Saskatchewan                 FAX:   (306) 966-6058
> Saskatoon, Canada.

-- 
Charles-Antoine Gauthier
Institute for Information Technology   Institut de technologie de
l'information
National Research Council of Canada    Conseil national de recherches du
Canada