NFS Performance

[Till Straumann]

Joel Sherrill wrote:

> Steven Johnson wrote:
>
>> Joel, Thanks for the pointer. setvbuf worked like a treat.  So, we don't
>> have to hack stdio.h, thanks for the suggestion.
>>
>>   
>
> I'm glad setvbuf worked.  It is always nice to not hack at a standard 
> include file.
>
>> Following are some benchmarks we got with different block sizes.  MPC862
>> Processor (PowerPC) at 100Mhz, with 50Mhz SDRAM main memory and using
>> the FEC (Fast Ethernet Controller @ 100Mbit/s, full duplex), its a
>> custom design and doesn't use a standard BSP, but isn't dissimilar to
>> the standard MPC862 BSP:
>>
>> Times to transfer the 7.4M test file timed using ticks_since_boot, (1
>> tick is 10ms on our target):
>> Buffer Size @ Ticks to transfer = ~Mbit/s
>>
>> 256 @ 4265  = ~1.4Mbit/s
>> 512 @ 2221  = ~2.7Mbit/s
>> 1024 @ 1239 = ~4.8Mbit/s
>> 2048 @ 712 = ~8.4Mbit/s
>> 4096 @ 439 = ~13.7Mbit/s
>> 8192 @ 298 = ~20.2Mbit/s
>> 16384 @ 299 =~20.2Mbit/s
>> 32768 @ 298 =~20.2Mbit/s
>>   
>
>
> You don't mention what the NFS server is and how it does from
> a similar host.
>
>> So, as Till said:
>>
>>  
>>
>>> The UDP packet size limit imposes a
>>> limit of 8k per RPC call
>>>     
>>
>>
>> And this is demonstrated by the above figures.
>>
>> I understand we may be able to get it faster with some sort of
>> multi-threaded reader, but for our purposes ~20Mbit/s is more than fast
>> enough.  Obviously the mileage will vary on different platforms, but
>> it's worth noting and trying if you are using NFS and need a bit more
>> speed.
>> Not that I've timed it, but I would expect the latency of bigger buffers
>> would be greater than the small one (it physically takes longer to
>> receive 8K than 1K) so the trade off between buffer size and average
>> latency would be different from application to application.
>> I also don't think this needs to be incorporated into the NFS code
>> anywhere, but it's worth noting in the README, 
>
done - I incorporated the comments I posted yesterday into the README.
Also, I added a test code.

-- T.

>> and maybe a Wiki page on
>> it wouldn't hurt.  I'll put Till's comments on getting further
>> performance, details on setvbuf and my results in the Wiki for posterity
>> in the next few days.
>>   
>
> Thanks for putting this in the Wiki.  It certainly needs to become 
> part of the collective
> knowledge base.  Please include a code fragment for your open/setvbuf 
> combo.
>
> --joel
>
>> Steven J
>>
>>
>>
>> Joel Sherrill wrote:
>>
>>  
>>
>>> Steven Johnson wrote:
>>>
>>>    
>>>
>>>> Hi,
>>>>
>>>> We are using the NFS Client stack as found in the RTEMS CVS.  We
>>>> experienced the slow performance that the readme talks about.  We were
>>>> getting around 3.5Mbit/s on a 100MBit link.  Our Processor is a 100Mhz
>>>> MPC862, and we are using the FEC.
>>>>
>>>> This isn't fast enough for us, as we need around 6-7 Mbit/s so we can
>>>> read media files and play them in real time from the NFS Share.
>>>>
>>>> So we investigated the causes.
>>>>
>>>> The single biggest thing we found is BUFSIZ in stdio.h.
>>>>
>>>> It was defined as 1024, which gave us ~3.5Mbit/s
>>>> We Changed it to 8192, and NFS Performance on our board jumped to
>>>> ~19.53MBit/s
>>>> We also tried 32768, which gave us ~19.99MBit/s
>>>>
>>>> (This is all Read performance, we don't know how fast write is, 
>>>> because
>>>> we don't write with our application).
>>>>
>>>> Also, the readme for the NFS Client indicates performance sucks due 
>>>> to a
>>>> lack of read ahead buffering.  Changing BUFSIZ to 8192 seems to 
>>>> provide
>>>> read ahead buffering, because if we read 512 bytes, 8192 bytes are
>>>> transferred from the file, and another request for 8192 bytes only
>>>> occurs when we have exhausted that data.
>>>>
>>>> Anyway, I'm just reporting this so that anyone else who needs NFS to
>>>> perform better can know what we did to achieve dramatic speed 
>>>> increases.
>>>>
>>>>         
>>>
>>> Interesting.  Since hacking stdio.h is generally not a good thing. 
>>> Can you try using
>>> one of the setbuf methods on the file descriptor and seeing if that
>>> accomplishes the
>>> same thing.
>>>
>>> If it does, then perhaps an open on a NFS client should implicitly
>>> increase the buffer
>>> size.
>>>
>>> This would limit the impact of the change.
>>>
>>> 20 Mbit/s is pretty nice performance from any computer but a 100 Mhz
>>> PowerPC
>>> makes it impressive.
>>>
>>> --joel
>>>
>>>    
>>>
>>>> Steven J
>>>>         
>>>
>>>
>>>     
>>
>>
>>   
>
>