Test documentation as a Google Code In task

Thu Dec 13 02:59:45 UTC 2012

Hi,

Dr. Sherrill, Gedare, and I were discussing the possibility of having the Google Code In students improve the documentation of the tests.  Is improving the documentation of the tests of interest to the RTEMS development community?

The benefits would be:
1. Better testing documentation would make it would be easier to cross-reference which tests were testing which RTEMS functions
2. We could make progress towards getting the testing documentation IEEE 829-2008 or another high-standard compliant, which would increase the possible uses for RTEMS, namely safety critical applications 

The challenges would be:
1. An increased number of patches to RTEMS
2. Determining a documentation format for the tests that is:
    a. agreeable to the RTEMS development community
    b. gets RTEMS testing documentation closer to the IEEE-829 (or other high-standard) compliant
    c. is straight forward, and concise enough, for a Google Code In task

We are currently considering using Doxygen headers as the documentation format for the test documentation improvement, as that would make it easy to generate test documentation in many different formats.  Looking forward to comments and recommendations!

Cynthia Rempel

________________________________________
From: Gedare Bloom [gedarebloom at gmail.com]
Sent: Wednesday, December 12, 2012 3:27 AM
To: Cynthia Rempel
Cc: Joel Sherrill; gci2012-mentors at rtems.org
Subject: Re: [GCI-Mentors] Test documentation thoughts

I like the idea. Please present it to rtems-devel mailing list for further feedback. I also am not a Doxygen expert.

On Tue, Dec 11, 2012 at 9:55 PM, Cynthia Rempel <cynt6007 at vandals.uidaho.edu<mailto:cynt6007 at vandals.uidaho.edu>> wrote:
Putting the test specifics in Doxygen comments may make more sense.

If cross-referencing wouldn't get too complicated for high-schoolers, doing something akin to

http://www.longacre-scm.com/blog/index.php/2010/08/using-doxygens-test-command-with-c

using the RTEMS Doxygen style instead of C++ style could be very useful for the cross-referencing needed for requirements traceability. That would also give the students more coding experience.

I'm not a Doxygen expert, but if it's possible to make custom labels, we might be able to simplify the task even more by using custom @directives and @concepts labels...

________________________________________
From: Gedare Bloom [gedarebloom at gmail.com<mailto:gedarebloom at gmail.com>]
Sent: Tuesday, December 11, 2012 3:56 AM
To: Cynthia Rempel
Cc: Joel Sherrill; gci2012-mentors at rtems.org<mailto:gci2012-mentors at rtems.org>
Subject: Re: [GCI-Mentors] Test documentation thoughts

Would it be a good idea to write the specifications in Doxygen formatting in the test source code itself and process testsuite to produce the whole set?

Once upon a time I went through and copied every single .doc file into a spreadsheet [1]. What I wanted was one place to look at all of the test specifications and to be able to search for what test exercises what RTEMS feature quickly. Doxygen output might be helpful at producing automatically such usable information.

[1] https://docs.google.com/document/d/1ByVTIEvpwjhOK4YVfNPZtd7TkAkEwJvN4Y8k91hOz_8/edit

-Gedare

On Mon, Dec 10, 2012 at 10:48 PM, Cynthia Rempel <cynt6007 at vandals.uidaho.edu<mailto:cynt6007 at vandals.uidaho.edu><mailto:cynt6007 at vandals.uidaho.edu<mailto:cynt6007 at vandals.uidaho.edu>>> wrote:
Hi,

Yes, getting the .docs done would be a start towards getting the code properly documented, and I suspect if there were better documented tests it would benefit some users (for reducing the cost of putting RTEMS in safety critical applications).

The challenge to getting the students to write-up the .doc would be to ensure the tests we had students work on were either well commented, or self-explanatory.  The directions would have to be improved as well (my trial version of the test documentation task was unclaimed).

I'm willing to identify tests I think are well enough commented, or have a detailed enough .scn for the students to write the .doc if that is desirable (I can't get to writing the list until this weekend).  I'd enjoy improving the output displayed on the screens of some of those tests... if that's desirable...

________________________________________
From: Gedare Bloom [gedarebloom at gmail.com<mailto:gedarebloom at gmail.com><mailto:gedarebloom at gmail.com<mailto:gedarebloom at gmail.com>>]
Sent: Monday, December 10, 2012 4:33 AM
To: Cynthia Rempel
Cc: Joel Sherrill; gci2012-mentors at rtems.org<mailto:gci2012-mentors at rtems.org><mailto:gci2012-mentors at rtems.org<mailto:gci2012-mentors at rtems.org>>
Subject: Re: [GCI-Mentors] Test documentation thoughts

On Mon, Dec 10, 2012 at 7:31 AM, Gedare Bloom <gedarebloom at gmail.com<mailto:gedarebloom at gmail.com><mailto:gedarebloom at gmail.com<mailto:gedarebloom at gmail.com>><mailto:gedarebloom at gmail.com<mailto:gedarebloom at gmail.com><mailto:gedarebloom at gmail.com<mailto:gedarebloom at gmail.com>>>> wrote:
This kind of effort only makes sense if it benefits some RTEMS user or the developer community. Improving our testing infrastructure certainly will be good for the community, but I suspect there are more foundational problems to address. Amar had been working on some of the test issues. His GSOC students project should have resulted in some infrastructure, but I don't think they got far enough for us to develop any tasks for GCI students to get in to.

That said, I think that ensuring all of the tests contain an accurate specification (right now, the .doc file) would be a good task. When there is not a specification, I'm not sure how easy it will be for a GCI student to draft one.

I see this idea already has an entry in the GCI projects wiki page. I think it is a good idea to work on fixing these up. That would make it easier to propose either converting to a standard approach, or maintaining our current one.

On Sun, Dec 9, 2012 at 11:03 PM, Cynthia Rempel <cynt6007 at vandals.uidaho.edu<mailto:cynt6007 at vandals.uidaho.edu><mailto:cynt6007 at vandals.uidaho.edu<mailto:cynt6007 at vandals.uidaho.edu>><mailto:cynt6007 at vandals.uidaho.edu<mailto:cynt6007 at vandals.uidaho.edu><mailto:cynt6007 at vandals.uidaho.edu<mailto:cynt6007 at vandals.uidaho.edu>>>> wrote:
Hi

Attached is a file written to IEEE 829-2008 specifications...

After going through writing one of these to IEEE 829-2008, some general thoughts are:
1. Writing an IEEE 829-2008 would need to be broken down into subtasks (passes?)
   The first pass would consist of adding the "boiler plate" or empty .tdf to the directories
   Before the second pass, we would need to make sure the directions were clear as to where to find the references
   The second pass would be to add the references for the features to be tested
   The third pass would be to comb through the source code for comments on pass/fail criteria

2. I agree with Dr. Sherrill, the level of effort for this undertaking is high enough we should concentrate on test documentation that would help RTEMS to pass high certification criteria

3. If we go ahead with test documentation, we need to sell the benefits of being IEEE 829-2008 compliant on the RTEMS developer community...

Thanks,
Cynthia Rempel
________________________________________
From: gci2012-mentors-bounces at rtems.org<mailto:gci2012-mentors-bounces at rtems.org><mailto:gci2012-mentors-bounces at rtems.org<mailto:gci2012-mentors-bounces at rtems.org>><mailto:gci2012-mentors-bounces at rtems.org<mailto:gci2012-mentors-bounces at rtems.org><mailto:gci2012-mentors-bounces at rtems.org<mailto:gci2012-mentors-bounces at rtems.org>>> [gci2012-mentors-bounces at rtems.org<mailto:gci2012-mentors-bounces at rtems.org><mailto:gci2012-mentors-bounces at rtems.org<mailto:gci2012-mentors-bounces at rtems.org>><mailto:gci2012-mentors-bounces at rtems.org<mailto:gci2012-mentors-bounces at rtems.org><mailto:gci2012-mentors-bounces at rtems.org<mailto:gci2012-mentors-bounces at rtems.org>>>] on behalf of Joel Sherrill [joel.sherrill at gmail.com<mailto:joel.sherrill at gmail.com><mailto:joel.sherrill at gmail.com<mailto:joel.sherrill at gmail.com>><mailto:joel.sherrill at gmail.com<mailto:joel.sherrill at gmail.com><mailto:joel.sherrill at gmail.com<mailto:joel.sherrill at gmail.com>>>]
Sent: Monday, December 03, 2012 7:26 AM
To: gci2012-mentors at rtems.org<mailto:gci2012-mentors at rtems.org><mailto:gci2012-mentors at rtems.org<mailto:gci2012-mentors at rtems.org>><mailto:gci2012-mentors at rtems.org<mailto:gci2012-mentors at rtems.org><mailto:gci2012-mentors at rtems.org<mailto:gci2012-mentors at rtems.org>>>
Subject: [GCI-Mentors] test documentation thoughts

Hi

Cynthia and I bounced some email about this.

http://en.wikipedia.org/wiki/IEEE_829 is on software test documentation.

I know of SPAWAR SWEBOK as a source for example documents.

Does anyone have any idea about examples for test documentation that
would adhere to the IEEE (or any other standards)?

I see no point in doing any test documentation work if we don't have
a high goal based on an accepted standard -- industry, IEEE, NASA,
ESA, FAA, FDA, etc.

--joel

_______________________________________________
gci2012-mentors mailing list
gci2012-mentors at rtems.org<mailto:gci2012-mentors at rtems.org><mailto:gci2012-mentors at rtems.org<mailto:gci2012-mentors at rtems.org>><mailto:gci2012-mentors at rtems.org<mailto:gci2012-mentors at rtems.org><mailto:gci2012-mentors at rtems.org<mailto:gci2012-mentors at rtems.org>>>
http://www.rtems.org/mailman/listinfo/gci2012-mentors