NFS Performance

[Till Straumann]

Steven Johnson wrote:
> Well we tried this and defining HAVE_BLKSIZE did not break our program.
>
> BUT the st_blksize is set to 4096 bytes, and we know that max
> performance seems to be when it is 8K.  The problem is we can't find
> where the 4K for st_bufsize is being set.  Its being copied by nfs from
> an fattr structure (line 2838 in nfs.c, right under a suspicious TODO
> comment), but where the fattr structure gets the 4k from is currently a
> mystery. 
It's the block size of the file on the server [I had looked at linux - 
there is
no way to tune this for the linux NFS server]
>  We could just hack NFS to not copy it from the fattr structure
> and force it to 8K, but this doesn't seem the right thing to do.
>   
Why not? That's just what I would do - supposedly,

  "The value st_blksize gives the "preferred" blocksize for efficient 
file system I/O."

I'd make it a run-time tunable feature by means of a global flag.

Comments?

-- T.
> Steven Johnson
>
> Till Straumann wrote:
>
>   
>> Here's something (regarding default block size used by newlib/stdio) I
>> found out:
>>
>> If newlib was compiled with HAVE_BLKSIZE defined then it would use
>> the st_blksize information [newlib makebuf.c does a 'stat' anyways] when
>> configuring the default buffer for stdio.
>>
>> Some filesystems (msdos, nfs) do set st_blksize, others (imfs) do not
>> but since libcsupport/src/stat.c does zero-out the stat buf argument
>> [and newlib uses BUFSIZ for st_bufsz <= 0] I believe defining
>> HAVE_BLKSIZE should be OK.
>>
>> We could use this trick to let rtems NFS transparently increase the
>> buffer size for
>> stdio to 8k (nfs v2 limit).
>>
>> -- Till
>>
>> Steven Johnson wrote:
>>
>>     
>>> Till Straumann wrote:
>>>
>>>  
>>>
>>>       
>>>> Joel Sherrill wrote:
>>>>
>>>>    
>>>>
>>>>         
>>>>> Steven Johnson wrote:
>>>>>
>>>>>      
>>>>>
>>>>>           
>>>>>> Joel, Thanks for the pointer. setvbuf worked like a treat.  So, we
>>>>>> don't
>>>>>> have to hack stdio.h, thanks for the suggestion.
>>>>>>
>>>>>>           
>>>>>>             
>>>>> I'm glad setvbuf worked.  It is always nice to not hack at a standard
>>>>> include file.
>>>>>
>>>>>      
>>>>>
>>>>>           
>>>>>> Following are some benchmarks we got with different block sizes.
>>>>>> MPC862
>>>>>> Processor (PowerPC) at 100Mhz, with 50Mhz SDRAM main memory and using
>>>>>> the FEC (Fast Ethernet Controller @ 100Mbit/s, full duplex), its a
>>>>>> custom design and doesn't use a standard BSP, but isn't dissimilar to
>>>>>> the standard MPC862 BSP:
>>>>>>
>>>>>> Times to transfer the 7.4M test file timed using ticks_since_boot, (1
>>>>>> tick is 10ms on our target):
>>>>>> Buffer Size @ Ticks to transfer = ~Mbit/s
>>>>>>
>>>>>> 256 @ 4265  = ~1.4Mbit/s
>>>>>> 512 @ 2221  = ~2.7Mbit/s
>>>>>> 1024 @ 1239 = ~4.8Mbit/s
>>>>>> 2048 @ 712 = ~8.4Mbit/s
>>>>>> 4096 @ 439 = ~13.7Mbit/s
>>>>>> 8192 @ 298 = ~20.2Mbit/s
>>>>>> 16384 @ 299 =~20.2Mbit/s
>>>>>> 32768 @ 298 =~20.2Mbit/s
>>>>>>           
>>>>>>             
>>>>> You don't mention what the NFS server is and how it does from
>>>>> a similar host.
>>>>>       
>>>>>           
>>> The NFS Server is a Linux Box (Linux version 2.4.21-0.13mdk, Mandrake
>>> Linux 9.1)
>>>
>>> It is a 2.7Ghz P4 with 1GB of ram.
>>>
>>> Ethernet controller: Broadcom Corporation BCM4401 100Base-T.
>>>
>>> From a similarly spec'd Linux box, between the machines, we can sustain
>>> about 88Mbit/s NFS transfer rate, with a single file transfer.  Not
>>> unsurprisingly, given the sophisticated caching, etc in linux setvbuf
>>> has no effect on the speed of NFS on the linux box.  I don't know if the
>>> 20Mbit/s from our rtems box is a limit on our MPC862's processing
>>> performance or the performance of the FEC, or deficiencies in the NFS
>>> stack.  Probably all of the above.  Yet given the relative level's of
>>> complexity, 20Mbit/s is very respectable in my opinion.  It is certainly
>>> fast enough for our application's needs.
>>>
>>>  
>>>
>>>       
>>>> done - I incorporated the comments I posted yesterday into the README.
>>>> Also, I added a test code.
>>>>
>>>> -- T.     
>>>>         
>>> That's great.  I'll throw up a Wiki topic on it soon.
>>>
>>> Steven J
>>>   
>>>       
>>
>>     
>
>