#4328: New APIs added to POSIX Standard (2021)

Eshan Dhawan eshandhawan51 at gmail.com
Thu Apr 1 18:31:30 UTC 2021 

Hi Matt
Here is the list of potential sources you can look into :
List of sources
-FreeBSD Source
-NetBSD Source
-JuliaMath Libm (https://github.com/JuliaMath/openlibm.git)
-ARM's optimized routines (
https://github.com/ARM-software/optimized-routines)
-Musl libc

Also Matt your google docs link is private
You need to change its permission to commenter

Thanks
- Eshan

On Thu, Apr 1, 2021 at 5:51 PM Matthew Joyce <mfjoyce2004 at gmail.com> wrote:

> Hi Dr. Joel and Dr. Gedare,
>
> I posted my draft proposal on the GSOC 2021 page
> (https://devel.rtems.org/wiki/GSoC/2021). At your convenience, I would
> be very grateful for any comments or additional guidance you might
> have.  Please note, I found implementations of some of the "clock"
> methods on glibc...does the GNU "Lesser General Public License" meet
> the intent for what RTEMS can use?
>
> Also, regarding the spawn.h group of methods, do I understand
> correctly that they've been deliberately left out?  If so, I'm curious
> if there is anything that would still need to be done there. I noticed
> in the docs that some methods relating to new processes are supported
> in an adapted fashion (such as getpid()). Just wondering if there has
> been discussion on this for spawn so I can cover the bases.
>
> Thank you very much for your time!
>
> Sincerely,
>
> Matt
>
> On Thu, Mar 25, 2021 at 2:18 PM Joel Sherrill <joel at rtems.org> wrote:
> >
> >
> >
> > On Thu, Mar 25, 2021 at 7:14 AM Matthew Joyce <mfjoyce2004 at gmail.com>
> wrote:
> >>
> >> Hi Dr. Joel,
> >>
> >> Thanks very much, that's a big help!  Correct, I've been updating the
> >> spreadsheet as I go along. Ok, now I see that strlcat/strlcpy are used
> >> in rtems/cpukit and implemented in Newlib.
> >>
> >> One additional question, please: I haven't yet looked into the source
> >> of NetBSD or FreeBSD, but I do see that Newlib already implements
> >> ppoll (poll.cc), dladdr (dlfcn.cc), pselect (select.cc), and
> >> sockatmark (net.cc). None of them are defined in the rtems environment
> >> yet. Is there any reason why the NetBSD/FreeBSD version would be
> >> preferable to Newlib for these? Or is it just a matter of testing
> >> what's out there to find what works well in the rtems environment?
> >
> >
> > Without looking at the newlib git repo, I can tell you that the files
> > you cite are the implementation of those methods for Cygwin. Just
> > because they are in C++. :)
> >
> > The parts of the newlib repo RTEMS uses are under the newlib/
> > subdirectory not the cygwin one. Within that, there is a libc/sys and
> > only libc/sys/rtems is used for RTEMS. The others are for different
> > operating systems. There are a few places with "machine" directory
> > structures. Only the ones for the architecture you are building for
> > is used.
> >
> > As to why NetBSD for libdl, that is because portions of the code
> > originated there.
> >
> > And rtems-libbsd is based on FreeBSD. It is as close to the FreeBSD
> > source as we can keep it.
> >
> >>
> >>
> >> In my proposal I'll take your advice and work on some of the easier
> >> ones first in order to get the experience and process down.
> >
> >
> > There are tickets for a lot of methods. The rtems-docs repo has the
> > csv file (e.g. spreadsheet) which tracks RTEMS support against
> > various standards. The RTEMS POSIX Compliance Guide is generated
> > from that csv file. Between those, you can find other methods to ask
> > about. In general, if it is required by the Software Communications
> > Architecture (SCA) or FACE Technical Standard, then it is a method
> > someone expected to possibly be used in an embedded system.
> > SCA is a set of POSIX profiles focused on software defined radios and
> > the FACE Technical Standard was developed with avionics in mind.
> >
> > But any are fair game if they are actually implementable. I don;t think
> > the Compliance Guide says it yet, but we decided last year that
> > wordexp() is likely not supportable on RTEMS. The newlib
> > implementation assumes the presence of a shell with wildcard expansion
> > and ability to fork a process.
> >
> > --joel
> >
> >>
> >>
> >> Thank you again for your time!
> >>
> >> Matt
> >>
> >> On Wed, Mar 24, 2021 at 5:03 PM Joel Sherrill <joel at rtems.org> wrote:
> >> >
> >> > Wow! Good work. There is a lot to digest here. Comments interspersed.
> >> >
> >> > I assume the spreadsheet is updated.
> >> >
> >> > On Wed, Mar 24, 2021 at 7:38 AM Matthew Joyce <mfjoyce2004 at gmail.com>
> wrote:
> >> >>
> >> >> Hi Dr. Joel,
> >> >>
> >> >> I've gone over the list a few times now and see a few categories
> shaping up:
> >> >>
> >> >> 1) Already done (In Newlib source, defined in libc.a):
> >> >> a) reallocarray
> >> >> b) qsort_r
> >> >> c) memmem
> >> >> d) strlcat / strlcpy
> >> >> d) wcslcat / wcslcpy
> >> >> *Out of this group, strlcat and strlcpy also show up in
> >> >> src/rtems/cpukit. Why is that?
> >> >
> >> >
> >> > The good news is that we support these. :)
> >> >
> >> > It looks to me that strlcat and strlcpy are used in cpukit but not
> implemented
> >> > there. Where do you think they are implemented.
> >> >
> >> > This is a good example where a source code browser is helpful. grep
> can
> >> > often answer the question but a source code browser can be easier.
> Personally,
> >> > I use cscope but that is exceedingly old school. Any modern IDE
> should be
> >> > helpful.
> >> >
> >> >>
> >> >> 2) Not done yet (Do not show up in Newlib source or RTEMS):
> >> >> a) getlocalename_l
> >> >> b) posix_getdents
> >> >> c) sem_clockwait
> >> >> d) sig2str / str2sig
> >> >>
> >> >> 3) Not in Newlib; Referenced in RTEMS but hidden behind #ifdef:
> >> >> a) pthread_cond_clockwait
> >> >> (rtems/6/lib/gcc/sparc-rtems6/10.2.1/include/c++/condition_variable)
> >> >> b) pthread_mutex_clocklock
> >> >> (rtems/6/lib/gcc/sparc-rtems6/10.2.1/include/c++/mutex)
> >> >> c) pthread_rwlock_clockrdlock
> >> >> (rtems/6/lib/gcc/sparc-rtems6/10.2.1/include/c++/shared_mutex)
> >> >> c) pthread_rwlock_clockwrlock
> >> >> (rtems/6/lib/gcc/sparc-rtems6/10.2.1/include/c++/shared_mutex)
> >> >> *It looks like some groundwork was done, but the methods are not yet
> supported.
> >> >
> >> >
> >> > The paths you point to are C++ files that would implement C++ features
> >> > using the available POSIX services. So they are users, not providers.
> >> >
> >> > All of the pthread services related to these are implemented in
> >> > cpukit/posix/src. I think you can configure a clock for all these now
> >> > to be used by detailed on wait and timedwait calls. My understanding
> >> > is that these would let you use a specific clock on a per blocking
> call
> >> > basis.
> >> >
> >> > First question is which clocks are intended to be supported.
> >> >
> >> > Second is the pattern of picking which timeout queue to go on when
> >> > now it is coded to let you pick one which is used for the life of the
> object.
> >> >
> >> >>
> >> >> 4) Misc (In Newlib source, not defined in libc.a, appear in RTEMS in
> >> >> various ways)
> >> >> a) getentropy (an alternate version is defined in RTEMS
> librtemsbsd.a,
> >> >> in src/rtems/bsps/shared/dev/getentropy/getentropy-cpucounter.c. The
> >> >> comments note that it is not cryptographically secure, so it may not
> >> >> fit the bill for the getentropy() mentioned in the Open Group
> >> >> document)
> >> >
> >> >
> >> > I am far from a cryptography expert but this looks like a case where
> >> > this method would be considered supported with the disclaimer that
> >> > the quality of the entropy value depends on the BSP. If the user has
> >> > specific requirements, they will need to verify the implementation
> >> > used by the BSP by default is appropriate.
> >> >
> >> >>
> >> >> b) ppoll (appears in rtems/6/share/gdb/syscalls)
> >> >
> >> >
> >> > You need to be more careful with the grep. These again are in the
> >> > installed tools and in this case, they appear in an XML file.
> Referenced
> >> > but not implemented.
> >> >
> >> > ppoll() will need to come from rtems-libbsd. The required system call
> >> > is included but disabled currently. AFAIK this means it is possible to
> >> > provide this but that would require a more detailed discussion in case
> >> > some underlying capability is missing. Chris Johns and Sebastian
> >> > Huber would be the ones to guide here.
> >> >
> >> > Ruling: Likely possible.
> >> >
> >> >>
> >> >> c) dladdr (appears in rtems/cpukit but not defined)
> >> >
> >> >
> >> > I think this can be implemented in libdl but I am not sure if the
> >> > code from NetBSD from this would directly work or just be a guide.
> >> >
> >> >>
> >> >>
> >> >> 5) Others?
> >> >> It looks like there was work done on methods like sockatmark and
> >> >> pselect, but I don't see them supported as yet. Should those be added
> >> >> to the list or are they still being worked on?
> >> >
> >> >
> >> > These would come from rtems-libbsd.
> >> >
> >> > I think sockatmark.c is implemented in
> freebsd/lib/libc/net/sockatmark.c
> >> > but I don't know if the ioctl() is implemented. I expect it is but
> this would
> >> > at least require a test. It may just work.
> >> >
> >> > pselect() looks to be missing and would have to be ported from
> FreeBSD.
> >> >
> >> >>
> >> >> As you suggested, I'll look into NetBSD for dladdr and do some
> digging
> >> >> on the implementation of the other outstanding methods. You mentioned
> >> >> that the "clock" ones have to be strictly added to rtems/cpukit, but
> >> >> the references I found above are all in lib/gcc/sparc-rtems6/10.2.1.
> >> >> Why is that the case and what is 10.2.1? Also, I'm not sure what to
> >> >> make of getentropy and ppoll based on what I found above...at your
> >> >> convenience could you please advise?
> >> >
> >> >
> >> > Hopefully the above helped.
> >> >
> >> > You don't have to restrict your possible set to these new additions.
> >> > There are others. I think Eshan has done the research for where to
> >> > get implementations of the missing long double methods for newlib.
> >> > And there are tickets for other missing methods or specific
> capabilities
> >> > in methods that are supported. Those are quite possible to have
> >> > some alternatives that are easier to approach.
> >> >
> >> > --joel
> >> >
> >> >
> >> >>
> >> >>
> >> >> Thank you very much!
> >> >>
> >> >> Matt
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> On Sun, Mar 21, 2021 at 6:38 PM Joel Sherrill <joel at rtems.org>
> wrote:
> >> >> >
> >> >> >
> >> >> >
> >> >> > On Sun, Mar 21, 2021 at 2:28 AM Matthew Joyce <
> mfjoyce2004 at gmail.com> wrote:
> >> >> >>
> >> >> >> Gentlemen,
> >> >> >>
> >> >> >> Awesome, thanks!  I see how that works now...I'll give it a
> thorough
> >> >> >> look tomorrow and will update the spreadsheet accordingly. I'll
> pipe
> >> >> >> back up when I have a more accurate look of what's currently
> there.
> >> >> >
> >> >> >
> >> >> > Knowing what doesn't have to be done is the first step. (rtems,
> newlib, and libbsd)
> >> >> >
> >> >> > I'd be prone to look for things that are easy to add first.
> >> >> >
> >> >> > Some may not be implementable on RTEMS due to only supporting a
> >> >> > single process and no virtual memory. If you have doubts on
> whether it
> >> >> > is possible to support a specific method, speak up and let's try
> to decide.
> >> >> >
> >> >> > Then find upstream places for an implementation where possible. I
> suspect
> >> >> > all the new "clock" methods will require discussion on an
> implementation
> >> >> > pattern but those must strictly be added to rtems/cpukit with
> tests and
> >> >> > documentation. At least I can throw you that much. :)
> >> >> >
> >> >> >>
> >> >> >> Thanks again and have a great Sunday!
> >> >> >>
> >> >> >> Matt
> >> >> >>
> >> >> >> On Fri, Mar 19, 2021 at 8:27 PM Joel Sherrill <joel at rtems.org>
> wrote:
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> > On Fri, Mar 19, 2021 at 1:08 PM Gedare Bloom <gedare at rtems.org>
> wrote:
> >> >> >> >>
> >> >> >> >> On Fri, Mar 19, 2021 at 11:16 AM Matthew Joyce <
> mfjoyce2004 at gmail.com> wrote:
> >> >> >> >> >
> >> >> >> >> > Dr. Joel,
> >> >> >> >> >
> >> >> >> >> > Thanks very much...I'll keep working to get a sense of what
> goes
> >> >> >> >> > where! In the meantime, where can I look to get the ground
> truth of
> >> >> >> >> > which methods are "in RTEMS" as opposed to those in newlib?
> >> >> >> >> >
> >> >> >> >> There is only one ground truth:
> >> >> >> >> git://git.rtems.org/rtems.git
> >> >> >> >>
> >> >> >> >> And for newlib
> >> >> >> >>
> >> >> >> >> git://sourceware.org/git/newlib-cygwin.git
> >> >> >> >>
> >> >> >> >> That said, searching for the function name symbols in compiled
> >> >> >> >> libraries is a good first step to rule out newlib. Then, you
> can
> >> >> >> >> 'grep' the RTEMS source code for the function names to see if
> they
> >> >> >> >> exist there.
> >> >> >> >
> >> >> >> >
> >> >> >> > rtems/cpukit to be specitic. It won't be implemented anywhere
> else.
> >> >> >> >
> >> >> >> > And clearly we both have forgotten that networking APIs are in
> the
> >> >> >> > rtems-libbsd repository.
> >> >> >> >
> >> >> >> > https://git.rtems.org/rtems-libbsd/
> >> >> >> >
> >> >> >> > I suspect ppoll() might already be in there. Or at least
> supported by
> >> >> >> > FreeBSD.
> >> >> >> >
> >> >> >> > You should clone everything and grep the sources. newlib
> already has
> >> >> >> > qsort_r. This is the nm I used:
> >> >> >> >
> >> >> >> > $ ~/rtems-work/tools/6/bin/sparc-rtems6-nm
> ~/rtems-work/tools/6/sparc-rtems6/lib/libc.a | grep qsort_r
> >> >> >> > lib_a-bsd_qsort_r.o:
> >> >> >> > 00000000 T __bsd_qsort_r
> >> >> >> > lib_a-qsort_r.o:
> >> >> >> > 00000000 T qsort_r
> >> >> >> >
> >> >> >> > Notice the last line has "T qsort_r" which says it is defined.
> >> >> >> >
> >> >> >> > grep -r in the newlib source shows it is in
> ./libc/search/qsort_r.c
> >> >> >> >
> >> >> >> > dladdr() looks to be prototyped in RTEMS but hidden behind an
> ifdef like it
> >> >> >> > wasn't ported from NetBSD so that looks possible. It is in
> rtems.
> >> >> >> >
> >> >> >> > Those two examples should help you figure out why you missed
> >> >> >> > finding some things that were implemented.
> >> >> >> >
> >> >> >> > I need to figure out what this next POSIX version is to be
> called
> >> >> >> > so I can update the tracking spreadsheet that generates the
> RTEMS
> >> >> >> > POSIX Compliance Guide, :)
> >> >> >> >
> >> >> >> > --joel
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> > Thanks again!
> >> >> >> >> >
> >> >> >> >> > Matt
> >> >> >> >> >
> >> >> >> >> > On Fri, Mar 19, 2021 at 1:58 PM Joel Sherrill <
> joel at rtems.org> wrote:
> >> >> >> >> > >
> >> >> >> >> > > Keep devel@ on the list. :)
> >> >> >> >> > >
> >> >> >> >> > > On Fri, Mar 19, 2021 at 7:51 AM Matthew Joyce <
> mfjoyce2004 at gmail.com> wrote:
> >> >> >> >> > >>
> >> >> >> >> > >> Sir,
> >> >> >> >> > >>
> >> >> >> >> > >> Thank you for the link! I see that you're right, those
> last four are
> >> >> >> >> > >> in newlib, plus memmem(). I updated those in the Google
> Sheet.
> >> >> >> >> > >>
> >> >> >> >> > >> Now I see the newlib part, but where are you referring to
> specifically
> >> >> >> >> > >> when you say RTEMS, as in "POSIX support comes from a mix
> of RTEMS and
> >> >> >> >> > >> newlib"?
> >> >> >> >> > >
> >> >> >> >> > >
> >> >> >> >> > > POSIX is a HUGE HUGE standard and references other
> standards. One
> >> >> >> >> > > it references and pulls in is the C99 Standard C Library
> which is libc and
> >> >> >> >> > > libm. RTEMS mostly does not implement this functionality
> and relies on
> >> >> >> >> > > another open source project for those APIs. Newlib is an
> open source
> >> >> >> >> > > C Library used by RTEMS, Cygwin, and most embedded systems
> GNU tools
> >> >> >> >> > > chains.
> >> >> >> >> > >
> >> >> >> >> > > Most of the POSIX header files with RTEMS are actually in
> Newlib even
> >> >> >> >> > > if they originated with RTEMS. Many are shared with Cygwin.
> >> >> >> >> > >
> >> >> >> >> > > So methods like the string, memory, and *printf come from
> Newlib since they
> >> >> >> >> > > are in C99. We provide POSIX like threading, signals, core
> file access, and
> >> >> >> >> > > much more.
> >> >> >> >> > >
> >> >> >> >> > > It's a complementary relationship but it takes a bit to
> figure out when
> >> >> >> >> > > something should be in one or the other. The line gets
> blurred at times.
> >> >> >> >> > >
> >> >> >> >> > > Say you added a new CPU architecture implementation of a
> math
> >> >> >> >> > > method (like Eshan did last year), then it goes in newlib.
> But he also
> >> >> >> >> > > added some POSIX methods which go in RTEMS. In either case,
> >> >> >> >> > > we like tests for them in RTEMS to show they work in our
> environment.
> >> >> >> >> > >
> >> >> >> >> > > --joel
> >> >> >> >> > >
> >> >> >> >> > >
> >> >> >> >> > >
> >> >> >> >> > >>
> >> >> >> >> > >> Thanks again!
> >> >> >> >> > >>
> >> >> >> >> > >> Matt
> >> >> >> >> > >>
> >> >> >> >> > >> On Fri, Mar 19, 2021 at 1:13 PM Joel Sherrill <
> joel at rtems.org> wrote:
> >> >> >> >> > >> >
> >> >> >> >> > >> >
> >> >> >> >> > >> >
> >> >> >> >> > >> > On Fri, Mar 19, 2021, 6:40 AM Joel Sherrill <
> joel at rtems.org> wrote:
> >> >> >> >> > >> >>
> >> >> >> >> > >> >>
> >> >> >> >> > >> >>
> >> >> >> >> > >> >> On Fri, Mar 19, 2021, 5:48 AM Matthew Joyce <
> mfjoyce2004 at gmail.com> wrote:
> >> >> >> >> > >> >>>
> >> >> >> >> > >> >>>
> https://docs.google.com/spreadsheets/d/1reCNOIZC5JTwQENgl-hvG8THfQqNtlUDVy_07PYodic/edit?usp=sharing
> >> >> >> >> > >> >>>
> >> >> >> >> > >> >>> Hello,
> >> >> >> >> > >> >>>
> >> >> >> >> > >> >>> As suggested by Dr. Sherril, I've taken an initial
> look through this
> >> >> >> >> > >> >>> document
> https://www.opengroup.org/austin/docs/austin_1110.pdf and
> >> >> >> >> > >> >>> added the new methods  to a Googe Sheet, linked above.
> >> >> >> >> > >> >>>
> >> >> >> >> > >> >>> None of them appear to be in the RTEMS POSIX API
> Users Guide, but
> >> >> >> >> > >> >>> maybe that's not the right place to look. I'll stand
> by for your
> >> >> >> >> > >> >>> feedback regarding what's possible / desirable to add
> to RTEMS.
> >> >> >> >> > >> >>
> >> >> >> >> > >> >>
> >> >> >> >> > >> >> It is possible they are in our C Library or Math
> Library.  Or just not in the manual. The POSIX manual tends to be sparse
> since you can always use man pages or the POSIX standard.
> >> >> >> >> > >> >>
> >> >> >> >> > >> >> Since you have RTEMS and tools built. Find one of the
> libc.a and libm.a files in the tools install and librtemscpu.a in the RTEMS
> build or install. Then try a command something like this:
> >> >> >> >> > >> >>
> >> >> >> >> > >> >> CPU-rtems6-nm LIBRARY | grep SYMBOL
> >> >> >> >> > >> >>
> >> >> >> >> > >> >> If you see it list with T then it is in the text
> section and there.
> >> >> >> >> > >> >
> >> >> >> >> > >> >
> >> >> >> >> > >> > Following up, I initially answered from my phone and
> didn't look at source.  I am still on my phone but looked through the list
> and think the last four methods are probably the only ones currently
> supported.
> >> >> >> >> > >> >
> >> >> >> >> > >> >
> https://sourceware.org/git/?p=newlib-cygwin.git;a=tree;f=newlib/libc/string;h=ceeec602cdd0e6b5c6b002b741bda9b41da4e441;hb=HEAD
> >> >> >> >> > >> >
> >> >> >> >> > >> > POSIX support comes from a mix of RTEMS and newlib.
> That's key to this type of project.
> >> >> >> >> > >> >
> >> >> >> >> > >> > --joel
> >> >> >> >> > >> >>
> >> >> >> >> > >> >>
> >> >> >> >> > >> >>
> >> >> >> >> > >> >>>
> >> >> >> >> > >> >>> Thanks very much for your time!
> >> >> >> >> > >> >>>
> >> >> >> >> > >> >>> Sincerely,
> >> >> >> >> > >> >>>
> >> >> >> >> > >> >>> Matt
> >> >> >> >> > _______________________________________________
> >> >> >> >> > devel mailing list
> >> >> >> >> > devel at rtems.org
> >> >> >> >> > http://lists.rtems.org/mailman/listinfo/devel
> _______________________________________________
> devel mailing list
> devel at rtems.org
> http://lists.rtems.org/mailman/listinfo/devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rtems.org/pipermail/devel/attachments/20210402/6fa82d69/attachment-0001.html>

More information about the devel mailing list