NFS Performance

[Joel Sherrill]

Steven Johnson wrote:
> Joel, Thanks for the pointer. setvbuf worked like a treat.  So, we don't
> have to hack stdio.h, thanks for the suggestion.
>
>   
I'm glad setvbuf worked.  It is always nice to not hack at a standard 
include file.
> Following are some benchmarks we got with different block sizes.  MPC862
> Processor (PowerPC) at 100Mhz, with 50Mhz SDRAM main memory and using
> the FEC (Fast Ethernet Controller @ 100Mbit/s, full duplex), its a
> custom design and doesn't use a standard BSP, but isn't dissimilar to
> the standard MPC862 BSP:
>
> Times to transfer the 7.4M test file timed using ticks_since_boot, (1
> tick is 10ms on our target):
> Buffer Size @ Ticks to transfer = ~Mbit/s
>
> 256 @ 4265  = ~1.4Mbit/s
> 512 @ 2221  = ~2.7Mbit/s
> 1024 @ 1239 = ~4.8Mbit/s
> 2048 @ 712 = ~8.4Mbit/s
> 4096 @ 439 = ~13.7Mbit/s
> 8192 @ 298 = ~20.2Mbit/s
> 16384 @ 299 =~20.2Mbit/s
> 32768 @ 298 =~20.2Mbit/s
>   

You don't mention what the NFS server is and how it does from
a similar host. 
> So, as Till said:
>
>   
>> The UDP packet size limit imposes a
>> limit of 8k per RPC call
>>     
>
> And this is demonstrated by the above figures.
>
> I understand we may be able to get it faster with some sort of
> multi-threaded reader, but for our purposes ~20Mbit/s is more than fast
> enough.  Obviously the mileage will vary on different platforms, but
> it's worth noting and trying if you are using NFS and need a bit more
> speed. 
>
> Not that I've timed it, but I would expect the latency of bigger buffers
> would be greater than the small one (it physically takes longer to
> receive 8K than 1K) so the trade off between buffer size and average
> latency would be different from application to application. 
>
> I also don't think this needs to be incorporated into the NFS code
> anywhere, but it's worth noting in the README, and maybe a Wiki page on
> it wouldn't hurt.  I'll put Till's comments on getting further
> performance, details on setvbuf and my results in the Wiki for posterity
> in the next few days.
>   
Thanks for putting this in the Wiki.  It certainly needs to become part 
of the collective
knowledge base.  Please include a code fragment for your open/setvbuf combo.

--joel
> Steven J
>
>
>
> Joel Sherrill wrote:
>
>   
>> Steven Johnson wrote:
>>
>>     
>>> Hi,
>>>
>>> We are using the NFS Client stack as found in the RTEMS CVS.  We
>>> experienced the slow performance that the readme talks about.  We were
>>> getting around 3.5Mbit/s on a 100MBit link.  Our Processor is a 100Mhz
>>> MPC862, and we are using the FEC.
>>>
>>> This isn't fast enough for us, as we need around 6-7 Mbit/s so we can
>>> read media files and play them in real time from the NFS Share.
>>>
>>> So we investigated the causes.
>>>
>>> The single biggest thing we found is BUFSIZ in stdio.h.
>>>
>>> It was defined as 1024, which gave us ~3.5Mbit/s
>>> We Changed it to 8192, and NFS Performance on our board jumped to
>>> ~19.53MBit/s
>>> We also tried 32768, which gave us ~19.99MBit/s
>>>
>>> (This is all Read performance, we don't know how fast write is, because
>>> we don't write with our application).
>>>
>>> Also, the readme for the NFS Client indicates performance sucks due to a
>>> lack of read ahead buffering.  Changing BUFSIZ to 8192 seems to provide
>>> read ahead buffering, because if we read 512 bytes, 8192 bytes are
>>> transferred from the file, and another request for 8192 bytes only
>>> occurs when we have exhausted that data.
>>>
>>> Anyway, I'm just reporting this so that anyone else who needs NFS to
>>> perform better can know what we did to achieve dramatic speed increases.
>>>
>>>   
>>>       
>> Interesting.  Since hacking stdio.h is generally not a good thing. 
>> Can you try using
>> one of the setbuf methods on the file descriptor and seeing if that
>> accomplishes the
>> same thing.
>>
>> If it does, then perhaps an open on a NFS client should implicitly
>> increase the buffer
>> size.
>>
>> This would limit the impact of the change.
>>
>> 20 Mbit/s is pretty nice performance from any computer but a 100 Mhz
>> PowerPC
>> makes it impressive.
>>
>> --joel
>>
>>     
>>> Steven J
>>>   
>>>       
>>
>>     
>
>