Test documentation as a Google Code In task using ESA standards

Sat Dec 15 12:38:15 UTC 2012

On Thu, Dec 13, 2012 at 11:45 PM, Cynthia Rempel
<cynt6007 at vandals.uidaho.edu> wrote:
> Hi,
>
> Is there a link to the ESA standards we (the RTEMS development community) could look at?
>
You might get more traction asking this to the rtems-users list, where
there might be more ESA eyes. A few are on devel list, but probably
more on users.

I would prefer if we can stick to a base standardized approach, though.

> 1. From the internet research I have done so far ESA standards look like they would include requirements traceability.
> 2. Using ESA standards for satellites sounds like it would work for safety critical applications.
> 3. Since ESA would be a user interested in getting RTEMS to comply with their standards, perhaps making RTEMS compliant would be worth the additional patch load, once we can determine a format (Doxygen header, or .docs) which is agreeable to the development community.

If folks from ESA community want to "pitch in" for such an effort, I
think it is reasonable we would maintain it.

> 4. If using ESA standards is agreeable, I propose we take advantage of the cross-referencing capabilities Doxygen where possible, and use custom Doxygen labels that closely align with the selected standards.
> Perhaps (if we went with Doxygen-style formatting) the header could include:
> /**
>  * @ingroup Name of group that contains the directives tested
>  * @test paragraph of test description goes here
>  * @requirements perhaps verbatim titles from ORKID, RTEID, POSIX or other requirements being tested
>  * @additionalLabel1 additional information needed to comply with ESA standard
>  * @additionalLabel2 additional information needed to comply with ESA standard
>  */

Regardless of the standards we adopt, I like the idea of using Doxygen
to self-document our testsuites. I haven't looked closely enough at
Doxygen to see if standard labels can be used in a sensible way.

> 5. Depending on the level-of-effort, a given Google Code In task may consist of filling in part of the header (perhaps the @ingroup and copying something for the @test).
>

I think starting to add test headers makes sense to me, although it
would be best if someone can find time to create a sample for GCI
students to follow, and for the rest of us to approve of. One issue I
can think of is to resolve where in each test to put the header. The
most likely candidate to me is wherever the test includes confdefs.h,
which is usually going to be init.c or system.h

-Gedare

> Does this sound agreeable?  Looking forward to comments!
> ________________________________________
> From: Joel Sherrill [joel.sherrill at oarcorp.com]
> Sent: Wednesday, December 12, 2012 11:23 PM
> To: Cynthia Rempel; rtems-devel at rtems.org
> Subject: Re: Test documentation as a Google Code In task
>
> I am still in Munich but was lucky enough to have an ESA Gallileo person in the class. He is providing me with their standards and examples for test documentation.
>
> And I learned ALL Gallileo satellites (24?) Will be all RTEMS!!!
>
> Cynthia Rempel <cynt6007 at vandals.uidaho.edu> wrote:
>
>>Hi,
>>
>>Dr. Sherrill, Gedare, and I were discussing the possibility of having the Google Code In students improve the documentation of the tests.  Is improving the documentation of the tests of interest to the RTEMS development community?
>>
>>The benefits would be:
>>1. Better testing documentation would make it would be easier to cross-reference which tests were testing which RTEMS functions
>>2. We could make progress towards getting the testing documentation IEEE 829-2008 or another high-standard compliant, which would increase the possible uses for RTEMS, namely safety critical applications
>>
>>The challenges would be:
>>1. An increased number of patches to RTEMS
>>2. Determining a documentation format for the tests that is:
>>    a. agreeable to the RTEMS development community
>>    b. gets RTEMS testing documentation closer to the IEEE-829 (or other high-standard) compliant
>>    c. is straight forward, and concise enough, for a Google Code In task
>>
>>We are currently considering using Doxygen headers as the documentation format for the test documentation improvement, as that would make it easy to generate test documentation in many different formats.  Looking forward to comments and recommendations!
>>
>>Cynthia Rempel
>>
>>________________________________________
>>From: Gedare Bloom [gedarebloom at gmail.com]
>>Sent: Wednesday, December 12, 2012 3:27 AM
>>To: Cynthia Rempel
>>Cc: Joel Sherrill; gci2012-mentors at rtems.org
>>Subject: Re: [GCI-Mentors] Test documentation thoughts
>>
>>I like the idea. Please present it to rtems-devel mailing list for further feedback. I also am not a Doxygen expert.
>>
>>
>>On Tue, Dec 11, 2012 at 9:55 PM, Cynthia Rempel <cynt6007 at vandals.uidaho.edu<mailto:cynt6007 at vandals.uidaho.edu>> wrote:
>>Putting the test specifics in Doxygen comments may make more sense.
>>
>>If cross-referencing wouldn't get too complicated for high-schoolers, doing something akin to
>>
>>http://www.longacre-scm.com/blog/index.php/2010/08/using-doxygens-test-command-with-c
>>
>>using the RTEMS Doxygen style instead of C++ style could be very useful for the cross-referencing needed for requirements traceability. That would also give the students more coding experience.
>>
>>I'm not a Doxygen expert, but if it's possible to make custom labels, we might be able to simplify the task even more by using custom @directives and @concepts labels...
>>
>>________________________________________
>>From: Gedare Bloom [gedarebloom at gmail.com<mailto:gedarebloom at gmail.com>]
>>Sent: Tuesday, December 11, 2012 3:56 AM
>>To: Cynthia Rempel
>>Cc: Joel Sherrill; gci2012-mentors at rtems.org<mailto:gci2012-mentors at rtems.org>
>>Subject: Re: [GCI-Mentors] Test documentation thoughts
>>
>>Would it be a good idea to write the specifications in Doxygen formatting in the test source code itself and process testsuite to produce the whole set?
>>
>>Once upon a time I went through and copied every single .doc file into a spreadsheet [1]. What I wanted was one place to look at all of the test specifications and to be able to search for what test exercises what RTEMS feature quickly. Doxygen output might be helpful at producing automatically such usable information.
>>
>>[1] https://docs.google.com/document/d/1ByVTIEvpwjhOK4YVfNPZtd7TkAkEwJvN4Y8k91hOz_8/edit
>>
>>-Gedare
>>
>>
>>On Mon, Dec 10, 2012 at 10:48 PM, Cynthia Rempel <cynt6007 at vandals.uidaho.edu<mailto:cynt6007 at vandals.uidaho.edu><mailto:cynt6007 at vandals.uidaho.edu<mailto:cynt6007 at vandals.uidaho.edu>>> wrote:
>>Hi,
>>
>>Yes, getting the .docs done would be a start towards getting the code properly documented, and I suspect if there were better documented tests it would benefit some users (for reducing the cost of putting RTEMS in safety critical applications).
>>
>>The challenge to getting the students to write-up the .doc would be to ensure the tests we had students work on were either well commented, or self-explanatory.  The directions would have to be improved as well (my trial version of the test documentation task was unclaimed).
>>
>>I'm willing to identify tests I think are well enough commented, or have a detailed enough .scn for the students to write the .doc if that is desirable (I can't get to writing the list until this weekend).  I'd enjoy improving the output displayed on the screens of some of those tests... if that's desirable...
>>
>>________________________________________
>>From: Gedare Bloom [gedarebloom at gmail.com<mailto:gedarebloom at gmail.com><mailto:gedarebloom at gmail.com<mailto:gedarebloom at gmail.com>>]
>>Sent: Monday, December 10, 2012 4:33 AM
>>To: Cynthia Rempel
>>Cc: Joel Sherrill; gci2012-mentors at rtems.org<mailto:gci2012-mentors at rtems.org><mailto:gci2012-mentors at rtems.org<mailto:gci2012-mentors at rtems.org>>
>>Subject: Re: [GCI-Mentors] Test documentation thoughts
>>
>>On Mon, Dec 10, 2012 at 7:31 AM, Gedare Bloom <gedarebloom at gmail.com<mailto:gedarebloom at gmail.com><mailto:gedarebloom at gmail.com<mailto:gedarebloom at gmail.com>><mailto:gedarebloom at gmail.com<mailto:gedarebloom at gmail.com><mailto:gedarebloom at gmail.com<mailto:gedarebloom at gmail.com>>>> wrote:
>>This kind of effort only makes sense if it benefits some RTEMS user or the developer community. Improving our testing infrastructure certainly will be good for the community, but I suspect there are more foundational problems to address. Amar had been working on some of the test issues. His GSOC students project should have resulted in some infrastructure, but I don't think they got far enough for us to develop any tasks for GCI students to get in to.
>>
>>That said, I think that ensuring all of the tests contain an accurate specification (right now, the .doc file) would be a good task. When there is not a specification, I'm not sure how easy it will be for a GCI student to draft one.
>>
>>I see this idea already has an entry in the GCI projects wiki page. I think it is a good idea to work on fixing these up. That would make it easier to propose either converting to a standard approach, or maintaining our current one.
>>
>>
>>On Sun, Dec 9, 2012 at 11:03 PM, Cynthia Rempel <cynt6007 at vandals.uidaho.edu<mailto:cynt6007 at vandals.uidaho.edu><mailto:cynt6007 at vandals.uidaho.edu<mailto:cynt6007 at vandals.uidaho.edu>><mailto:cynt6007 at vandals.uidaho.edu<mailto:cynt6007 at vandals.uidaho.edu><mailto:cynt6007 at vandals.uidaho.edu<mailto:cynt6007 at vandals.uidaho.edu>>>> wrote:
>>Hi
>>
>>Attached is a file written to IEEE 829-2008 specifications...
>>
>>After going through writing one of these to IEEE 829-2008, some general thoughts are:
>>1. Writing an IEEE 829-2008 would need to be broken down into subtasks (passes?)
>>   The first pass would consist of adding the "boiler plate" or empty .tdf to the directories
>>   Before the second pass, we would need to make sure the directions were clear as to where to find the references
>>   The second pass would be to add the references for the features to be tested
>>   The third pass would be to comb through the source code for comments on pass/fail criteria
>>
>>2. I agree with Dr. Sherrill, the level of effort for this undertaking is high enough we should concentrate on test documentation that would help RTEMS to pass high certification criteria
>>
>>3. If we go ahead with test documentation, we need to sell the benefits of being IEEE 829-2008 compliant on the RTEMS developer community...
>>
>>Thanks,
>>Cynthia Rempel
>>________________________________________
>>From: gci2012-mentors-bounces at rtems.org<mailto:gci2012-mentors-bounces at rtems.org><mailto:gci2012-mentors-bounces at rtems.org<mailto:gci2012-mentors-bounces at rtems.org>><mailto:gci2012-mentors-bounces at rtems.org<mailto:gci2012-mentors-bounces at rtems.org><mailto:gci2012-mentors-bounces at rtems.org<mailto:gci2012-mentors-bounces at rtems.org>>> [gci2012-mentors-bounces at rtems.org<mailto:gci2012-mentors-bounces at rtems.org><mailto:gci2012-mentors-bounces at rtems.org<mailto:gci2012-mentors-bounces at rtems.org>><mailto:gci2012-mentors-bounces at rtems.org<mailto:gci2012-mentors-bounces at rtems.org><mailto:gci2012-mentors-bounces at rtems.org<mailto:gci2012-mentors-bounces at rtems.org>>>] on behalf of Joel Sherrill [joel.sherrill at gmail.com<mailto:joel.sherrill at gmail.com><mailto:joel.sherrill at gmail.com<mailto:joel.sherrill at gmail.com>><mailto:joel.sherrill at gmail.com<mailto:joel.sherrill at gmail.com><mailto:joel.sherrill at gmail.com<mailto:joel.sherrill at gmail.com>>>]
>>Sent: Monday, December 03, 2012 7:26 AM
>>To: gci2012-mentors at rtems.org<mailto:gci2012-mentors at rtems.org><mailto:gci2012-mentors at rtems.org<mailto:gci2012-mentors at rtems.org>><mailto:gci2012-mentors at rtems.org<mailto:gci2012-mentors at rtems.org><mailto:gci2012-mentors at rtems.org<mailto:gci2012-mentors at rtems.org>>>
>>Subject: [GCI-Mentors] test documentation thoughts
>>
>>Hi
>>
>>Cynthia and I bounced some email about this.
>>
>>http://en.wikipedia.org/wiki/IEEE_829 is on software test documentation.
>>
>>I know of SPAWAR SWEBOK as a source for example documents.
>>
>>Does anyone have any idea about examples for test documentation that
>>would adhere to the IEEE (or any other standards)?
>>
>>I see no point in doing any test documentation work if we don't have
>>a high goal based on an accepted standard -- industry, IEEE, NASA,
>>ESA, FAA, FDA, etc.
>>
>>--joel
>>
>>_______________________________________________
>>gci2012-mentors mailing list
>>gci2012-mentors at rtems.org<mailto:gci2012-mentors at rtems.org><mailto:gci2012-mentors at rtems.org<mailto:gci2012-mentors at rtems.org>><mailto:gci2012-mentors at rtems.org<mailto:gci2012-mentors at rtems.org><mailto:gci2012-mentors at rtems.org<mailto:gci2012-mentors at rtems.org>>>
>>http://www.rtems.org/mailman/listinfo/gci2012-mentors
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>_______________________________________________
>>rtems-devel mailing list
>>rtems-devel at rtems.org
>>http://www.rtems.org/mailman/listinfo/rtems-devel
>
>
> _______________________________________________
> rtems-devel mailing list
> rtems-devel at rtems.org
> http://www.rtems.org/mailman/listinfo/rtems-devel