#4328: New APIs added to POSIX Standard (2021)

Matthew Joyce mfjoyce2004 at gmail.com
Sun Mar 28 06:46:59 UTC 2021 

Hi Eshan,

Ok, great! Thank you for letting me know. By the way, where will it be
implemented when it's patched in? Thanks again and have a great rest
of your Sunday.

Sincerely,

Matt

On Sun, Mar 28, 2021 at 6:54 AM Eshan Dhawan <eshandhawan51 at gmail.com> wrote:
>
>
> > On 27-Mar-2021, at 1:49 AM, Matthew Joyce <mfjoyce2004 at gmail.com> wrote:
> >
> > Hi Dr. Joel,
> >
> > I finally built rtems-libbsd and see that pselect and sockatmark are
> > both defined there. I went ahead and added a "In rtems-libbsd" column
> > in the spreadsheet to reflect that.
> >
> > With those two defined, it looks like the only methods from the FACE
> > 3.0 General Purpose Profile that aren't currently supported are
> > confstr() and the <spawn.h> set. Could I ask, what is the thinking on
> > those? The man page suggests that spawn was created with embedded
> > systems in mind, but I'd guess a conscious decision was made to leave
> > them out? How about confstr?
> >
> > Thank you!
> >
> > Matt
> >
> Hi Matt
> Confstr code is ready just under styling issues.
> So maybe you could count it as Present.
>
> Thanks
> - Eshan
> >
> >> On Thu, Mar 25, 2021 at 2:18 PM Joel Sherrill <joel at rtems.org> wrote:
> >>
> >>
> >>
> >>> On Thu, Mar 25, 2021 at 7:14 AM Matthew Joyce <mfjoyce2004 at gmail.com> wrote:
> >>>
> >>> Hi Dr. Joel,
> >>>
> >>> Thanks very much, that's a big help!  Correct, I've been updating the
> >>> spreadsheet as I go along. Ok, now I see that strlcat/strlcpy are used
> >>> in rtems/cpukit and implemented in Newlib.
> >>>
> >>> One additional question, please: I haven't yet looked into the source
> >>> of NetBSD or FreeBSD, but I do see that Newlib already implements
> >>> ppoll (poll.cc), dladdr (dlfcn.cc), pselect (select.cc), and
> >>> sockatmark (net.cc). None of them are defined in the rtems environment
> >>> yet. Is there any reason why the NetBSD/FreeBSD version would be
> >>> preferable to Newlib for these? Or is it just a matter of testing
> >>> what's out there to find what works well in the rtems environment?
> >>
> >>
> >> Without looking at the newlib git repo, I can tell you that the files
> >> you cite are the implementation of those methods for Cygwin. Just
> >> because they are in C++. :)
> >>
> >> The parts of the newlib repo RTEMS uses are under the newlib/
> >> subdirectory not the cygwin one. Within that, there is a libc/sys and
> >> only libc/sys/rtems is used for RTEMS. The others are for different
> >> operating systems. There are a few places with "machine" directory
> >> structures. Only the ones for the architecture you are building for
> >> is used.
> >>
> >> As to why NetBSD for libdl, that is because portions of the code
> >> originated there.
> >>
> >> And rtems-libbsd is based on FreeBSD. It is as close to the FreeBSD
> >> source as we can keep it.
> >>
> >>>
> >>>
> >>> In my proposal I'll take your advice and work on some of the easier
> >>> ones first in order to get the experience and process down.
> >>
> >>
> >> There are tickets for a lot of methods. The rtems-docs repo has the
> >> csv file (e.g. spreadsheet) which tracks RTEMS support against
> >> various standards. The RTEMS POSIX Compliance Guide is generated
> >> from that csv file. Between those, you can find other methods to ask
> >> about. In general, if it is required by the Software Communications
> >> Architecture (SCA) or FACE Technical Standard, then it is a method
> >> someone expected to possibly be used in an embedded system.
> >> SCA is a set of POSIX profiles focused on software defined radios and
> >> the FACE Technical Standard was developed with avionics in mind.
> >>
> >> But any are fair game if they are actually implementable. I don;t think
> >> the Compliance Guide says it yet, but we decided last year that
> >> wordexp() is likely not supportable on RTEMS. The newlib
> >> implementation assumes the presence of a shell with wildcard expansion
> >> and ability to fork a process.
> >>
> >> --joel
> >>
> >>>
> >>>
> >>> Thank you again for your time!
> >>>
> >>> Matt
> >>>
> >>> On Wed, Mar 24, 2021 at 5:03 PM Joel Sherrill <joel at rtems.org> wrote:
> >>>>
> >>>> Wow! Good work. There is a lot to digest here. Comments interspersed.
> >>>>
> >>>> I assume the spreadsheet is updated.
> >>>>
> >>>> On Wed, Mar 24, 2021 at 7:38 AM Matthew Joyce <mfjoyce2004 at gmail.com> wrote:
> >>>>>
> >>>>> Hi Dr. Joel,
> >>>>>
> >>>>> I've gone over the list a few times now and see a few categories shaping up:
> >>>>>
> >>>>> 1) Already done (In Newlib source, defined in libc.a):
> >>>>> a) reallocarray
> >>>>> b) qsort_r
> >>>>> c) memmem
> >>>>> d) strlcat / strlcpy
> >>>>> d) wcslcat / wcslcpy
> >>>>> *Out of this group, strlcat and strlcpy also show up in
> >>>>> src/rtems/cpukit. Why is that?
> >>>>
> >>>>
> >>>> The good news is that we support these. :)
> >>>>
> >>>> It looks to me that strlcat and strlcpy are used in cpukit but not implemented
> >>>> there. Where do you think they are implemented.
> >>>>
> >>>> This is a good example where a source code browser is helpful. grep can
> >>>> often answer the question but a source code browser can be easier. Personally,
> >>>> I use cscope but that is exceedingly old school. Any modern IDE should be
> >>>> helpful.
> >>>>
> >>>>>
> >>>>> 2) Not done yet (Do not show up in Newlib source or RTEMS):
> >>>>> a) getlocalename_l
> >>>>> b) posix_getdents
> >>>>> c) sem_clockwait
> >>>>> d) sig2str / str2sig
> >>>>>
> >>>>> 3) Not in Newlib; Referenced in RTEMS but hidden behind #ifdef:
> >>>>> a) pthread_cond_clockwait
> >>>>> (rtems/6/lib/gcc/sparc-rtems6/10.2.1/include/c++/condition_variable)
> >>>>> b) pthread_mutex_clocklock
> >>>>> (rtems/6/lib/gcc/sparc-rtems6/10.2.1/include/c++/mutex)
> >>>>> c) pthread_rwlock_clockrdlock
> >>>>> (rtems/6/lib/gcc/sparc-rtems6/10.2.1/include/c++/shared_mutex)
> >>>>> c) pthread_rwlock_clockwrlock
> >>>>> (rtems/6/lib/gcc/sparc-rtems6/10.2.1/include/c++/shared_mutex)
> >>>>> *It looks like some groundwork was done, but the methods are not yet supported.
> >>>>
> >>>>
> >>>> The paths you point to are C++ files that would implement C++ features
> >>>> using the available POSIX services. So they are users, not providers.
> >>>>
> >>>> All of the pthread services related to these are implemented in
> >>>> cpukit/posix/src. I think you can configure a clock for all these now
> >>>> to be used by detailed on wait and timedwait calls. My understanding
> >>>> is that these would let you use a specific clock on a per blocking call
> >>>> basis.
> >>>>
> >>>> First question is which clocks are intended to be supported.
> >>>>
> >>>> Second is the pattern of picking which timeout queue to go on when
> >>>> now it is coded to let you pick one which is used for the life of the object.
> >>>>
> >>>>>
> >>>>> 4) Misc (In Newlib source, not defined in libc.a, appear in RTEMS in
> >>>>> various ways)
> >>>>> a) getentropy (an alternate version is defined in RTEMS librtemsbsd.a,
> >>>>> in src/rtems/bsps/shared/dev/getentropy/getentropy-cpucounter.c. The
> >>>>> comments note that it is not cryptographically secure, so it may not
> >>>>> fit the bill for the getentropy() mentioned in the Open Group
> >>>>> document)
> >>>>
> >>>>
> >>>> I am far from a cryptography expert but this looks like a case where
> >>>> this method would be considered supported with the disclaimer that
> >>>> the quality of the entropy value depends on the BSP. If the user has
> >>>> specific requirements, they will need to verify the implementation
> >>>> used by the BSP by default is appropriate.
> >>>>
> >>>>>
> >>>>> b) ppoll (appears in rtems/6/share/gdb/syscalls)
> >>>>
> >>>>
> >>>> You need to be more careful with the grep. These again are in the
> >>>> installed tools and in this case, they appear in an XML file. Referenced
> >>>> but not implemented.
> >>>>
> >>>> ppoll() will need to come from rtems-libbsd. The required system call
> >>>> is included but disabled currently. AFAIK this means it is possible to
> >>>> provide this but that would require a more detailed discussion in case
> >>>> some underlying capability is missing. Chris Johns and Sebastian
> >>>> Huber would be the ones to guide here.
> >>>>
> >>>> Ruling: Likely possible.
> >>>>
> >>>>>
> >>>>> c) dladdr (appears in rtems/cpukit but not defined)
> >>>>
> >>>>
> >>>> I think this can be implemented in libdl but I am not sure if the
> >>>> code from NetBSD from this would directly work or just be a guide.
> >>>>
> >>>>>
> >>>>>
> >>>>> 5) Others?
> >>>>> It looks like there was work done on methods like sockatmark and
> >>>>> pselect, but I don't see them supported as yet. Should those be added
> >>>>> to the list or are they still being worked on?
> >>>>
> >>>>
> >>>> These would come from rtems-libbsd.
> >>>>
> >>>> I think sockatmark.c is implemented in freebsd/lib/libc/net/sockatmark.c
> >>>> but I don't know if the ioctl() is implemented. I expect it is but this would
> >>>> at least require a test. It may just work.
> >>>>
> >>>> pselect() looks to be missing and would have to be ported from FreeBSD.
> >>>>
> >>>>>
> >>>>> As you suggested, I'll look into NetBSD for dladdr and do some digging
> >>>>> on the implementation of the other outstanding methods. You mentioned
> >>>>> that the "clock" ones have to be strictly added to rtems/cpukit, but
> >>>>> the references I found above are all in lib/gcc/sparc-rtems6/10.2.1.
> >>>>> Why is that the case and what is 10.2.1? Also, I'm not sure what to
> >>>>> make of getentropy and ppoll based on what I found above...at your
> >>>>> convenience could you please advise?
> >>>>
> >>>>
> >>>> Hopefully the above helped.
> >>>>
> >>>> You don't have to restrict your possible set to these new additions.
> >>>> There are others. I think Eshan has done the research for where to
> >>>> get implementations of the missing long double methods for newlib.
> >>>> And there are tickets for other missing methods or specific capabilities
> >>>> in methods that are supported. Those are quite possible to have
> >>>> some alternatives that are easier to approach.
> >>>>
> >>>> --joel
> >>>>
> >>>>
> >>>>>
> >>>>>
> >>>>> Thank you very much!
> >>>>>
> >>>>> Matt
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Sun, Mar 21, 2021 at 6:38 PM Joel Sherrill <joel at rtems.org> wrote:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Sun, Mar 21, 2021 at 2:28 AM Matthew Joyce <mfjoyce2004 at gmail.com> wrote:
> >>>>>>>
> >>>>>>> Gentlemen,
> >>>>>>>
> >>>>>>> Awesome, thanks!  I see how that works now...I'll give it a thorough
> >>>>>>> look tomorrow and will update the spreadsheet accordingly. I'll pipe
> >>>>>>> back up when I have a more accurate look of what's currently there.
> >>>>>>
> >>>>>>
> >>>>>> Knowing what doesn't have to be done is the first step. (rtems, newlib, and libbsd)
> >>>>>>
> >>>>>> I'd be prone to look for things that are easy to add first.
> >>>>>>
> >>>>>> Some may not be implementable on RTEMS due to only supporting a
> >>>>>> single process and no virtual memory. If you have doubts on whether it
> >>>>>> is possible to support a specific method, speak up and let's try to decide.
> >>>>>>
> >>>>>> Then find upstream places for an implementation where possible. I suspect
> >>>>>> all the new "clock" methods will require discussion on an implementation
> >>>>>> pattern but those must strictly be added to rtems/cpukit with tests and
> >>>>>> documentation. At least I can throw you that much. :)
> >>>>>>
> >>>>>>>
> >>>>>>> Thanks again and have a great Sunday!
> >>>>>>>
> >>>>>>> Matt
> >>>>>>>
> >>>>>>> On Fri, Mar 19, 2021 at 8:27 PM Joel Sherrill <joel at rtems.org> wrote:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Fri, Mar 19, 2021 at 1:08 PM Gedare Bloom <gedare at rtems.org> wrote:
> >>>>>>>>>
> >>>>>>>>> On Fri, Mar 19, 2021 at 11:16 AM Matthew Joyce <mfjoyce2004 at gmail.com> wrote:
> >>>>>>>>>>
> >>>>>>>>>> Dr. Joel,
> >>>>>>>>>>
> >>>>>>>>>> Thanks very much...I'll keep working to get a sense of what goes
> >>>>>>>>>> where! In the meantime, where can I look to get the ground truth of
> >>>>>>>>>> which methods are "in RTEMS" as opposed to those in newlib?
> >>>>>>>>>>
> >>>>>>>>> There is only one ground truth:
> >>>>>>>>> git://git.rtems.org/rtems.git
> >>>>>>>>>
> >>>>>>>>> And for newlib
> >>>>>>>>>
> >>>>>>>>> git://sourceware.org/git/newlib-cygwin.git
> >>>>>>>>>
> >>>>>>>>> That said, searching for the function name symbols in compiled
> >>>>>>>>> libraries is a good first step to rule out newlib. Then, you can
> >>>>>>>>> 'grep' the RTEMS source code for the function names to see if they
> >>>>>>>>> exist there.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> rtems/cpukit to be specitic. It won't be implemented anywhere else.
> >>>>>>>>
> >>>>>>>> And clearly we both have forgotten that networking APIs are in the
> >>>>>>>> rtems-libbsd repository.
> >>>>>>>>
> >>>>>>>> https://git.rtems.org/rtems-libbsd/
> >>>>>>>>
> >>>>>>>> I suspect ppoll() might already be in there. Or at least supported by
> >>>>>>>> FreeBSD.
> >>>>>>>>
> >>>>>>>> You should clone everything and grep the sources. newlib already has
> >>>>>>>> qsort_r. This is the nm I used:
> >>>>>>>>
> >>>>>>>> $ ~/rtems-work/tools/6/bin/sparc-rtems6-nm ~/rtems-work/tools/6/sparc-rtems6/lib/libc.a | grep qsort_r
> >>>>>>>> lib_a-bsd_qsort_r.o:
> >>>>>>>> 00000000 T __bsd_qsort_r
> >>>>>>>> lib_a-qsort_r.o:
> >>>>>>>> 00000000 T qsort_r
> >>>>>>>>
> >>>>>>>> Notice the last line has "T qsort_r" which says it is defined.
> >>>>>>>>
> >>>>>>>> grep -r in the newlib source shows it is in ./libc/search/qsort_r.c
> >>>>>>>>
> >>>>>>>> dladdr() looks to be prototyped in RTEMS but hidden behind an ifdef like it
> >>>>>>>> wasn't ported from NetBSD so that looks possible. It is in rtems.
> >>>>>>>>
> >>>>>>>> Those two examples should help you figure out why you missed
> >>>>>>>> finding some things that were implemented.
> >>>>>>>>
> >>>>>>>> I need to figure out what this next POSIX version is to be called
> >>>>>>>> so I can update the tracking spreadsheet that generates the RTEMS
> >>>>>>>> POSIX Compliance Guide, :)
> >>>>>>>>
> >>>>>>>> --joel
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> Thanks again!
> >>>>>>>>>>
> >>>>>>>>>> Matt
> >>>>>>>>>>
> >>>>>>>>>> On Fri, Mar 19, 2021 at 1:58 PM Joel Sherrill <joel at rtems.org> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Keep devel@ on the list. :)
> >>>>>>>>>>>
> >>>>>>>>>>> On Fri, Mar 19, 2021 at 7:51 AM Matthew Joyce <mfjoyce2004 at gmail.com> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>> Sir,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thank you for the link! I see that you're right, those last four are
> >>>>>>>>>>>> in newlib, plus memmem(). I updated those in the Google Sheet.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Now I see the newlib part, but where are you referring to specifically
> >>>>>>>>>>>> when you say RTEMS, as in "POSIX support comes from a mix of RTEMS and
> >>>>>>>>>>>> newlib"?
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> POSIX is a HUGE HUGE standard and references other standards. One
> >>>>>>>>>>> it references and pulls in is the C99 Standard C Library which is libc and
> >>>>>>>>>>> libm. RTEMS mostly does not implement this functionality and relies on
> >>>>>>>>>>> another open source project for those APIs. Newlib is an open source
> >>>>>>>>>>> C Library used by RTEMS, Cygwin, and most embedded systems GNU tools
> >>>>>>>>>>> chains.
> >>>>>>>>>>>
> >>>>>>>>>>> Most of the POSIX header files with RTEMS are actually in Newlib even
> >>>>>>>>>>> if they originated with RTEMS. Many are shared with Cygwin.
> >>>>>>>>>>>
> >>>>>>>>>>> So methods like the string, memory, and *printf come from Newlib since they
> >>>>>>>>>>> are in C99. We provide POSIX like threading, signals, core file access, and
> >>>>>>>>>>> much more.
> >>>>>>>>>>>
> >>>>>>>>>>> It's a complementary relationship but it takes a bit to figure out when
> >>>>>>>>>>> something should be in one or the other. The line gets blurred at times.
> >>>>>>>>>>>
> >>>>>>>>>>> Say you added a new CPU architecture implementation of a math
> >>>>>>>>>>> method (like Eshan did last year), then it goes in newlib. But he also
> >>>>>>>>>>> added some POSIX methods which go in RTEMS. In either case,
> >>>>>>>>>>> we like tests for them in RTEMS to show they work in our environment.
> >>>>>>>>>>>
> >>>>>>>>>>> --joel
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks again!
> >>>>>>>>>>>>
> >>>>>>>>>>>> Matt
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Fri, Mar 19, 2021 at 1:13 PM Joel Sherrill <joel at rtems.org> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Fri, Mar 19, 2021, 6:40 AM Joel Sherrill <joel at rtems.org> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Fri, Mar 19, 2021, 5:48 AM Matthew Joyce <mfjoyce2004 at gmail.com> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> https://docs.google.com/spreadsheets/d/1reCNOIZC5JTwQENgl-hvG8THfQqNtlUDVy_07PYodic/edit?usp=sharing
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hello,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> As suggested by Dr. Sherril, I've taken an initial look through this
> >>>>>>>>>>>>>>> document https://www.opengroup.org/austin/docs/austin_1110.pdf and
> >>>>>>>>>>>>>>> added the new methods  to a Googe Sheet, linked above.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> None of them appear to be in the RTEMS POSIX API Users Guide, but
> >>>>>>>>>>>>>>> maybe that's not the right place to look. I'll stand by for your
> >>>>>>>>>>>>>>> feedback regarding what's possible / desirable to add to RTEMS.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> It is possible they are in our C Library or Math Library.  Or just not in the manual. The POSIX manual tends to be sparse since you can always use man pages or the POSIX standard.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Since you have RTEMS and tools built. Find one of the libc.a and libm.a files in the tools install and librtemscpu.a in the RTEMS build or install. Then try a command something like this:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> CPU-rtems6-nm LIBRARY | grep SYMBOL
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> If you see it list with T then it is in the text section and there.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Following up, I initially answered from my phone and didn't look at source.  I am still on my phone but looked through the list and think the last four methods are probably the only ones currently supported.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> https://sourceware.org/git/?p=newlib-cygwin.git;a=tree;f=newlib/libc/string;h=ceeec602cdd0e6b5c6b002b741bda9b41da4e441;hb=HEAD
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> POSIX support comes from a mix of RTEMS and newlib. That's key to this type of project.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> --joel
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks very much for your time!
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Sincerely,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Matt
> >>>>>>>>>> _______________________________________________
> >>>>>>>>>> devel mailing list
> >>>>>>>>>> devel at rtems.org
> >>>>>>>>>> http://lists.rtems.org/mailman/listinfo/devel
> > _______________________________________________
> > devel mailing list
> > devel at rtems.org
> > http://lists.rtems.org/mailman/listinfo/devel

More information about the devel mailing list