[SOCIS] Rtems-testing related project - more details

[Joel Sherrill]

Hopefully Chris Johns will pipe up and add more details.
In particular, where his rtems-test is.  I think it can be used
on the beagle board and some gdb simulators now.

On 5/7/2014 4:18 PM, Krzysztof Mięsowicz wrote:
> Hi,
>
> My name is Krzysztof Mięsowicz. I am interested in participating again
> in SOCIS this year and I am writing to you in hope that you could
> provide me some more details about possible projects related to new
> rtems-test infrastructure. 
>
> As dr Joel wrote on Rtems-Users facebook group, there are some
> possible improvements: (cited)
>
>     This year we want to encourage the SOCIS students to focus on
>     improvements to our new rtems-test Python-based framework. This
>     will include:
>     + adding more BSPs (simulators and real boards) to rtems-test
>

Chris will have to point us to his rtems-test repository. I can't seem to
locate it. It is in Python and drives real hardware debuggers
and simulators using the GDB/MI.

rtems-testing/sim-scripts supports more simulators but does not support
hardware debuggers. rtems-test supports a small subset of the simulators
supported by rtems-testing/sim-scripts. rtems-test needs to be worked
on so it can replace all uses of sim-scripts and obsolete them.

>     + building more simulators with the RTEMS Source
>     - if successful, sim-scripts should be obsolete
>
That should be RTEMS Source Builder.
http://git.rtems.org/rtems-source-builder/
RSB builds the basic tools and a couple of simulator configurations.
RSB and rtems-tools work together to provide a framework to manage
build instructions and patch sets for RTEMS tools. We want to expand
it so it can build every tool one needs cross. Simulators are the next
step.

Eventually, we would like to be able to even build add-on libraries
to target a specific BSP. But that is longer down the road. If someone
gets to it sooner, great!

RSB needs to be augmented to be able to build all the simulators
external to gdb that we currently use. This may involve some odd
cases like say an old qemu version plus a patch to do coldfire.
That would give a baseline to upgrade against.

>     + integrating coverage runs into rtems-test
>

rtems-testing/covoar (C++) and rtems-testing/rtems-coverage (shell)
together do RTEMS test runs for coverage and generate the coverage
reports on http://www.rtems.org/ftp/pub/rtems/people/joel/coverage/

The rtems-test tool needs to be able to do coverage runs. It needs to know
if a target supports coverage, how to turn on coverage for a target, how
to run the report generator, etc.


>     + improvements to the coverage test reporting
>     - reporting should be on "module" level. Too coarse now.
>
The report generator itself needs a refresh. Right now, it reports on two
"lumps" of code -- core and developmental. Read "critical core" and
"the kitchen sink". :)

Ideally, the coverage reports should be generated based on modules
or source directories in cpukit/. Then we can get the coverage on
score/ separate from shell/, fatfs/, or imfs. What we have seen is that
each time we add a module to the "kitchen sink" set, the covered
percentage drops because the newly added module needs improvement
in test coverage.

covoar can generate .gcov output but no one has ever validated it
against "truth" and seen if gcov, lcov, etc. can produce trustworth
reports with it.
>
>     - some scripts could be migrated to Python
>
>
The older code is in bash. Migrating it to Python as it is reworked is
desirable.
> I wonder which of these are of highest priorities and what will it
> mean exactly. Could you provide more details about these ideas and
> maybe point some starting points to get more familiar and prepare
> better proposal?
>
Everything is high priority. :)

I guess IMO I would like to see one BSP/simulator pushed through the
entire software stack, reporting improved, and shell transitioned to
Python as it was convenient/expedient. Adding other simulators
to the RSB and rtems-test/ is a bonus.  So if you want to assume we
get one student from SOCIS, that would be my suggestion.

My thinking is that if SOCIS gives us two or more students, one could
easily start at the rtems-test/ level integrating coverage runs as they
are now. Another student could focus on the coverage report generation.

Transitioning more simulators from sim-scripts/ to rtems-test/
and adding building more simulators to the RSB could be added
easily. Each simulator instance is more or less independent so
any number of students could work off a check list.

If we get two or more students, the problem can be tackled from
multiple ends.
> Regards,
> Krzysztof  

-- 
Joel Sherrill, Ph.D.             Director of Research & Development
joel.sherrill at OARcorp.com        On-Line Applications Research
Ask me about RTEMS: a free RTOS  Huntsville AL 35805
Support Available                 (256) 722-9985

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rtems.org/pipermail/devel/attachments/20140507/da03ea05/attachment-0001.html>