Sendfile for Rtems

Joel Sherrill joel.sherrill at oarcorp.com
Tue Jun 23 20:49:57 UTC 2015



On 6/23/2015 2:37 PM, Sebastian Huber wrote:
> I just had a quick look at kern_sendfile() in uipc_syscalls.c. It seems that it uses virtual memory for the file access, so this won't work on RTEMS. However it would be nice to have an optimized sendfile() which avoids one copy and uses mbufs directly.

That would be a big improvement. I think that matches the BSD semantics
that the destination has to be a socket.

Any suggestions for Sujay on mbuf size allocation? I assume you would
want to allocate them largish to make fewer filesystem read's.

Also we need to consider the case where sizeof(file) > sizeof(all mbufs).

--joel

> ----- Sujay Raj <sujayraaj at gmail.com> schrieb:
>  >
> Okay, So I will dig in to the details of the implementation of sendfile on freebsd to find if it does indeed use mmap, or if there is an implementation that doesn't.
>
> I will inform as soon as I figure it out if this is the correct way out.
>
> Thanks and Regards,
> Sujay Raj
>
> On Wed, Jun 24, 2015 at 12:49 AM, Joel Sherrill <joel.sherrill at oarcorp.com <mailto:joel.sherrill at oarcorp.com>> wrote:
>
>
>
>     >
>
>     > On 6/23/2015 2:15 PM, Gedare Bloom wrote:
>
>     >
>
>
>          > On Tue, Jun 23, 2015 at 3:13 PM, Sujay Raj <sujayraaj at gmail.com <mailto:sujayraaj at gmail.com>> wrote:
>
>          >
>
>
>              > I agree with Sebastian.
>
>              >
>
>              > Using read write in a loop can only serve as a makeshift measure.
>
>              >
>
>              > If we have consensus here, then I can work on getting sendfile backed up in
>
>              > libbsd. (though, I have some serious background reading to do before I am
>
>              > able to do it, but I think I can. )
>
>              >
>
>              >
>
>
>          > Yes this makes the most sense.
>
>          >
>
>
>     >
>
>      > If it doesn't rely on mmap(). If it does, then it won't work.
>
>
>      >
>
>      >
>
>
>              > Thanks and Regards,
>
>              > Sujay Raj
>
>              >
>
>              > On Wed, Jun 24, 2015 at 12:21 AM, Sebastian Huber
>
>              > <sebastian.huber at embedded-brains.de <mailto:sebastian.huber at embedded-brains.de>> wrote:
>
>              >
>
>
>                  >
>
>                  > I would use the sendfile system call implementation from FreeBSD and add
>
>                  > it to the libbsd. Its not a big win if you use read() and write() to
>
>                  > implement sendfile().  This makes only sense if you can use kernel space
>
>                  > zero-copy buffers.
>
>                  >
>
>                  > ----- Eduardo Silva <eduardo at monkey.io <mailto:eduardo at monkey.io>> schrieb:
>
>                  >
>
>
>                      >
>
>                      >
>
>
>                  > Sujay,
>
>                  >
>
>                  > Monkey use conditionals for each implementation: Linux, FreeBSD and OSX.
>
>                  > My suggestion is that RTEMS may implement a sendfile based on FreeBSD format
>
>                  > which can easily be emulated as a read() and further write() for socket
>
>                  > operations. That will help not only Monkey, but also other software that
>
>                  > requires such interface.
>
>                  >
>
>                  > I am not the best one to talk about RTEMS, but the Kernel always knows
>
>                  > better it buffers capacity that the program on the other side.
>
>                  >
>
>                  > do you think is possible to do that on RTEMS ?
>
>                  >
>
>                  > regards,
>
>                  >
>
>                  > On Tue, Jun 23, 2015 at 12:05 PM, Joel Sherrill
>
>                  > <joel.sherrill at oarcorp.com <mailto:joel.sherrill at oarcorp.com>> wrote:
>
>                  >
>
>
>                      >
>
>                      >
>
>                      >
>
>                      >
>
>
>                          >
>
>                          >
>
>
>                      >
>
>                      >
>
>
>                          > On 6/23/2015 12:49 PM, Sujay Raj wrote:
>
>                          >
>
>
>                      >
>
>                      >
>
>
>                          >
>
>                          >
>
>                          >
>
>                          >
>
>
>                              > I am working on porting Monkey HTTP server on rtems.
>
>                              >
>
>
>                          >
>
>                          >
>
>
>                              >
>
>                              >
>
>
>                          >
>
>                          >
>
>
>                              > Sendfile is not specified in POSIX.1-2001 or other standards and its
>
>                              > implementations vary depending on the unix/linux system being used.
>
>                              >
>
>
>                          >
>
>                          >
>
>
>                              >
>
>                              >
>
>
>                          >
>
>                          >
>
>
>                              > Freebsd has sendfile in its libc.
>
>                              >
>
>
>                          >
>
>                          >
>
>
>                              >
>
>                              >
>
>
>                      >
>
>                      >
>
>                      >
>
>
>                          >
>
>                          >
>
>
>                      >
>
>                      >
>
>
>                          > FreeBSD has a completely different API for sendfile(). Plus the source
>
>                          >
>
>
>                      >
>
>                      >
>
>
>                          > must be a regular file and the destination a socket.
>
>                          >
>
>
>                      >
>
>                      >
>
>
>                          >
>
>                          >
>
>
>                      >
>
>                      >
>
>
>                          >https://www.freebsd.org/cgi/man.cgi?query=sendfile&sektion=2
>
>                          >
>
>
>                      >
>
>                      >
>
>
>                          >
>
>                          >
>
>
>                      >
>
>                      >
>
>
>                          > This is a random discussion of sendfile() on various OSes. It mentions
>
>                          >
>
>
>                      >
>
>                      >
>
>
>                          > the assumption that the source must normally be able to be mmap'ed.
>
>                          >
>
>
>                      >
>
>                      >
>
>
>                          > Since you can't do that on RTEMS, you are stuck with a loop
>
>                          >
>
>
>                      >
>
>                      >
>
>
>                          > implementation. [1]
>
>                          >
>
>
>                      >
>
>                      >
>
>
>                          >
>
>                          >
>
>
>                      >
>
>                      >
>
>
>                          >https://groups.google.com/forum/#!topic/comp.unix.programmer/65oDSzpEzx8
>
>                          >
>
>
>                      >
>
>                      >
>
>
>                          >
>
>                          >
>
>
>                      >
>
>                      >
>
>
>                          >
>
>                          >
>
>                          >
>
>                          >
>
>
>                              > So I opened this thread to discuss how should we proceed with this
>
>                              > matter, if we should work on getting sendfile in newlib, or if there is any
>
>                              > portable solution that we can use in its place.
>
>                              >
>
>
>                          >
>
>                          >
>
>
>                              >
>
>                              >
>
>
>                      >
>
>                      >
>
>                      >
>
>
>                          >
>
>                          >
>
>
>                      >
>
>                      >
>
>
>                          > Given that (a) it is not POSIX and (b) there is no agreement between
>
>                          >
>
>
>                      >
>
>                      >
>
>
>                          > Linux and FreeBSD, it is highly unlikely newlib would accept an
>
>                          >
>
>
>                      >
>
>                      >
>
>
>                          > implementation.
>
>                          >
>
>
>                      >
>
>                      >
>
>
>                          >
>
>                          >
>
>
>                      >
>
>                      >
>
>
>                          > Given (b), I don't personally see merging it into RTEMS since we
>
>                          >
>
>
>                      >
>
>                      >
>
>
>                          > have to pick a winner and loser on the API.
>
>                          >
>
>
>                      >
>
>                      >
>
>
>                          >
>
>                          >
>
>
>                      >
>
>                      >
>
>
>                          > What's the feeling on having a default implementation in Monkey and
>
>                          >
>
>
>                      >
>
>                      >
>
>
>                          > using that as a backup? With [1] as a consideration again.
>
>                          >
>
>
>                      >
>
>                      >
>
>
>                          >
>
>                          >
>
>
>                      >
>
>                      >
>
>
>                          >
>
>                          >
>
>
>                      >
>
>                      >
>
>
>                          > [1] The size of the buffer user for read/write has to be accounted
>
>                          >
>
>
>                      >
>
>                      >
>
>
>                          > for. If this is on a thread stack, then it can't be too large or that
>
>                          >
>
>
>                      >
>
>                      >
>
>
>                          > imposes a burden on calling sendfile(). I can see an implementation
>
>                          >
>
>
>                      >
>
>                      >
>
>
>                          > doing a malloc() of the buffer and then free()'ing it.
>
>                          >
>
>
>                      >
>
>                      >
>
>                      >
>
>
>                          >
>
>                          >
>
>
>                      >
>
>                      >
>
>
>                          >
>
>                          >
>
>
>                      >
>
>                      >
>
>
>                          >
>
>                          >
>
>
>                      >
>
>                      >
>
>
>                          >
>
>                          >
>
>                          >
>
>                          >
>
>
>                              > cc to Joel and Eduardo.
>
>                              >
>
>
>                          >
>
>                          >
>
>
>                              >
>
>                              >
>
>
>                          >
>
>                          >
>
>
>                              > Thanks and Regards,
>
>                              >
>
>
>                          >
>
>                          >
>
>
>                              > Sujay Raj
>
>                              >
>
>
>                          >
>
>                          >
>
>
>                              >
>
>                              >
>
>
>                      >
>
>                      >
>
>                      >
>
>
>                          >
>
>                          >
>
>
>                      >
>
>                      >
>
>
>                          > --
>
>                          >
>
>
>                      >
>
>                      >
>
>
>                          > Joel Sherrill, Ph.D.             Director of Research & Development
>
>                          >
>
>
>                      >
>
>                      >
>
>
>                          > joel.sherrill at OARcorp.com        On-Line Applications Research
>
>                          >
>
>
>                      >
>
>                      >
>
>
>                          > Ask me about RTEMS: a free RTOS  Huntsville AL 35805
>
>                          >
>
>
>                      >
>
>                      >
>
>
>                          > Support Available (256) 722-9985 <tel:%28256%29%20722-9985>
>
>                          >
>
>
>                      >
>
>                      >
>
>
>                          >
>
>                          >
>
>
>                  >
>
>                  >
>
>                  >
>
>                  >
>
>                  > --
>
>                  > Eduardo Silva
>
>                  > Monkey Software
>
>                  >
>
>                  >
>
>
>                      >
>
>                      >
>
>
>                  >
>
>                  >
>
>                  > --
>
>                  >
>
>                  > Sebastian Huber, embedded brains GmbH
>
>                  >
>
>                  >
>
>                  > Address : Dornierstr. 4, D-82178 Puchheim, Germany
>
>                  > Phone   : +49 89 189 47 41-16 <tel:%2B49%2089%20189%2047%2041-16>
>
>                  > Fax     : +49 89 189 47 41-09 <tel:%2B49%2089%20189%2047%2041-09>
>
>                  > E-Mail  : sebastian.huber at embedded-brains.de <http://embedded-brains.de>
>
>                  > PGP     : Public key available on request.
>
>                  >
>
>                  >
>
>                  > Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG.
>
>                  >
>
>                  >
>
>                  > _______________________________________________
>
>                  > devel mailing list
>
>                  >devel at rtems.org <mailto:devel at rtems.org>
>
>                  >http://lists.rtems.org/mailman/listinfo/devel
>
>                  >
>
>
>              >
>
>              >
>
>              >
>
>              > _______________________________________________
>
>              > devel mailing list
>
>              >devel at rtems.org <mailto:devel at rtems.org>
>
>              >http://lists.rtems.org/mailman/listinfo/devel
>
>              >
>
>
>      >
>
>      > --
>
>      > Joel Sherrill, Ph.D.             Director of Research & Development
>
>      > joel.sherrill at OARcorp.com        On-Line Applications Research
>
>      > Ask me about RTEMS: a free RTOS  Huntsville AL 35805
>
>      > Support Available (256) 722-9985 <tel:%28256%29%20722-9985>
>
>      >
>
>
>
>
> --
>
> Sebastian Huber, embedded brains GmbH
>
>
> Address : Dornierstr. 4, D-82178 Puchheim, Germany
> Phone   : +49 89 189 47 41-16
> Fax     : +49 89 189 47 41-09
> E-Mail  :sebastian.huber at embedded-brains.de  <http://lists.rtems.org/mailman/listinfo/devel>
> PGP     : Public key available on request.
>
>
> Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG.
>

-- 
Joel Sherrill, Ph.D.             Director of Research & Development
joel.sherrill at OARcorp.com        On-Line Applications Research
Ask me about RTEMS: a free RTOS  Huntsville AL 35805
Support Available                (256) 722-9985



More information about the devel mailing list