Network buffers, MPC860 and data caching

[Eric Norum]

Till Straumann wrote:
> 
> Eric Norum wrote:
> 
> > The fix would be to call
> > rtems_cache_invalidate_multiple_data_lines just after the MCLGET.  You
> > wouldn't have to invalidate all 2kbytes, just the 1536 bytes needed to
> > hold an ethernet frame.
> 
> That's the point where I do not agree (see my earlier message). If you call
> rtems_cache_invalidate_multiple_data_lines on a non-aligned buffer, data
> before/after the buffer hold by the cache can be lost: Assume a variable of
> type
> 
> struct {
>      int blah;
>      char buffer[1518];
> }
> 

Since we're talking about mbuf's here, I presume that you mean 124 and
not 1518.  Mbuf clusters (2k) have buffers that are completely separate
from the mbuf header.  mbufs (128 bytes) contain both header and buffer
information.

> to be 128-byte aligned (hence it's only safe to assume that `buffer' is word-
> aligned). A value is then written to `blah' which ends up in
> the data cache. If you subsequently invalidate all cache lines overlapping
> with the `buffer' field, you will also invalidate `blah' and later, an old and
> invalid value of blah will be refetched.
> 
> However, if you are sure that blah is never written while the network interface
> `owns' the data structure, you could
>   - first flush (write to memory and invalidate) the first cache line (the one that
>       overlaps with `blah')
>   - invalidate all other cache lines contained completely within `buffer'.
>   - mark the buffer ready for the networking/DMA hardware.

Yes, I agree that the first step would be necessary if the driver were
reading into an mbuf.  However, it is actually reading into an mbuf
cluster, so there is no chance of any overlap between the header and
buffer.  I can see no place in the network driver where the mbuf header
(or buffer, either local (mbuf) or not (cluster)) is written to after
the ownership has been taken over by the DMA engine.

I believe the following steps in the driver read task are necessary and
sufficient to deal with the problem.  
1) Allocate an mbuf cluster.
2) Invalidate any cache lines associated with the first 1518 bytes of
the mbuf cluster buffer.
3) Hand the buffer over to the DMA engine to fill.
4) Make sure the driver does not read/write the buffer until ownership
has been returned.

> 
> If however, a cache line is larger (in this example) than sizeof(blah), or
> if you can not guarantee that `blah' will not be written while the DMA
> engine writes to `buffer', the following scenario could happen:
> 
>   - CPU writes blah again (after the flush described above). The rest
>       of the overlapping first cache line is filled with bogus data from `buffer'
> 
>   - DMA engine fills buffer memory.
> 
>   - CPU reads cached bogus data (from the line holding `blah') which will
>       eventually be written back to memory.
> 
> Therefore, IMHO, the only workarounds for mbufs are
>  - using the M_EXT and a dedicated allocation/deallocation manager
>      that provides cache block aligned buffer space.
>  - on the MPC860 (with a cache line size of 16 bytes), the workaround
>      described in my last message could be used.

As far as I can see, as long as a cache line is less than or equal to
the size of an mbuf (128 bytes), and given that the network code writes
to neither headers nor buffers of mbufs or mbuf clusters owned by the
DMA hardware, there should be no problem.  I can't think of any
architectures with cache lines anywhere near this size.

> 
> Sorry for being pigheaded about this issue but occasional data/cache corruption
> will be very hard to track down and considering this topic with great care is quite
> important, IMHO.

I'm happy that someone out there is interested enough to spend the time
pointing out the flaws in the system.  The issues are indeed complicated
and important to get right.

-- 
Eric Norum                                 eric at cls.usask.ca
Canadian Light Source                      Phone: (306) 966-6308
University of Saskatchewan                 FAX:   (306) 966-6058
Saskatoon, Canada.