[SOCIS] Rtems-testing related project - more details

[Chris Johns]

On 8/05/2014 8:15 am, Joel Sherrill wrote:
> Hopefully Chris Johns will pipe up and add more details.
> In particular, where his rtems-test is.  I think it can be used
> on the beagle board and some gdb simulators now.

The RTEMS Tester is an RTEMS Tools Project and can be found in the 
rtems-tools.git repo. The web interface to the repo is 
http://git.rtems.org/rtems-tools/. The tester is under the tester 
directory. There is also a doc directory with some very basic doco.

It supports the gdb run command, gdb via the MI interface with gdb 
simulators and actual hardware via BDM or JTAG, and qemu is support. It 
provides serial port access with full termios control.

The supported BSPs can be seen here ...

http://git.rtems.org/rtems-tools/tree/tester/rtems/testing/bsps

The testing back end support can be seen here ..

http://git.rtems.org/rtems-tools/tree/tester/rtems/testing

>
> On 5/7/2014 4:18 PM, Krzysztof Mięsowicz wrote:
>> Hi,
>>
>> My name is Krzysztof Mięsowicz. I am interested in participating again
>> in SOCIS this year and I am writing to you in hope that you could
>> provide me some more details about possible projects related to new
>> rtems-test infrastructure.
>>
>> As dr Joel wrote on Rtems-Users facebook group, there are some
>> possible improvements: (cited)
>>
>>     This year we want to encourage the SOCIS students to focus on
>>     improvements to our new rtems-test Python-based framework. This
>>     will include:
>>     + adding more BSPs (simulators and real boards) to rtems-test
>>
>
> Chris will have to point us to his rtems-test repository. I can't seem to
> locate it.

See above.

> It is in Python and drives real hardware debuggers
> and simulators using the GDB/MI.
>
> rtems-testing/sim-scripts supports more simulators but does not support
> hardware debuggers. rtems-test supports a small subset of the simulators
> supported by rtems-testing/sim-scripts. rtems-test needs to be worked
> on so it can replace all uses of sim-scripts and obsolete them.
>
>>     + building more simulators with the RTEMS Source
>>     - if successful, sim-scripts should be obsolete
>>
> That should be RTEMS Source Builder.
> http://git.rtems.org/rtems-source-builder/
> RSB builds the basic tools and a couple of simulator configurations.
> RSB and rtems-tools work together to provide a framework to manage
> build instructions and patch sets for RTEMS tools. We want to expand
> it so it can build every tool one needs cross. Simulators are the next
> step.

The RSB builds qemu on most hosts. Building qemu for Windows as a 
Canadian cross is complex and I have not managed to sort this out so 
Windows is not working. Supporting Windows is outside the scope of this 
work.

There is work in tested the qemu's build and to get BSP support added to 
the rtems-tester. I am sure as you point out later in this email 
specific patches are needed to make things work.

>
> Eventually, we would like to be able to even build add-on libraries
> to target a specific BSP. But that is longer down the road. If someone
> gets to it sooner, great!
>

I have added some basic support for this via pkgconfig. The RSB contains 
a private implementation and that can be used to help manage this task. 
I still have some things to sort out and I am not sure this is ready to 
have libraries added.

> RSB needs to be augmented to be able to build all the simulators
> external to gdb that we currently use. This may involve some odd
> cases like say an old qemu version plus a patch to do coldfire.
> That would give a baseline to upgrade against.

Do you have the patches ? The RSB qemu support currently references some 
zynq patches on Qemu's patchworks so if you can find them there we can 
directly reference them.

>
>>     + integrating coverage runs into rtems-test
>>
>
> rtems-testing/covoar (C++) and rtems-testing/rtems-coverage (shell)
> together do RTEMS test runs for coverage and generate the coverage
> reports on http://www.rtems.org/ftp/pub/rtems/people/joel/coverage/
>
> The rtems-test tool needs to be able to do coverage runs. It needs to know
> if a target supports coverage, how to turn on coverage for a target, how
> to run the report generator, etc.
>

This code needs to move into the rtems-tools repo under tester/covoar 
and a waf build script added. This will make the rtems-tools.git repo 
self contained.

>
>>     + improvements to the coverage test reporting
>>     - reporting should be on "module" level. Too coarse now.
>>
> The report generator itself needs a refresh. Right now, it reports on two
> "lumps" of code -- core and developmental. Read "critical core" and
> "the kitchen sink". :)

When you say report generator do you mean coverage report generator ?

What host tools does this report generator use ? My aim is to have 
rtems-tools as self contains as possible.

>
> Ideally, the coverage reports should be generated based on modules
> or source directories in cpukit/. Then we can get the coverage on
> score/ separate from shell/, fatfs/, or imfs. What we have seen is that
> each time we add a module to the "kitchen sink" set, the covered
> percentage drops because the newly added module needs improvement
> in test coverage.

Do have more details about this task and how it would proceed ?

>
> covoar can generate .gcov output but no one has ever validated it
> against "truth" and seen if gcov, lcov, etc. can produce trustworth
> reports with it.
>>
>>     - some scripts could be migrated to Python
>>
>>
> The older code is in bash. Migrating it to Python as it is reworked is
> desirable.
>> I wonder which of these are of highest priorities and what will it
>> mean exactly. Could you provide more details about these ideas and
>> maybe point some starting points to get more familiar and prepare
>> better proposal?
>>
> Everything is high priority. :)
>
> I guess IMO I would like to see one BSP/simulator pushed through the
> entire software stack, reporting improved, and shell transitioned to
> Python as it was convenient/expedient. Adding other simulators
> to the RSB and rtems-test/ is a bonus.  So if you want to assume we
> get one student from SOCIS, that would be my suggestion.
>
> My thinking is that if SOCIS gives us two or more students, one could
> easily start at the rtems-test/ level integrating coverage runs as they
> are now. Another student could focus on the coverage report generation.
>
> Transitioning more simulators from sim-scripts/ to rtems-test/
> and adding building more simulators to the RSB could be added
> easily. Each simulator instance is more or less independent so
> any number of students could work off a check list.
>
> If we get two or more students, the problem can be tackled from
> multiple ends.

Another and more challenging task is to add support to test networking. 
This would be via the VDE support in qemu. See 
http://wiki.v2.cs.unibo.it/wiki/index.php/VDE. I do not have anything 
more to add on this topic other than this looks like a nice to do this.

>> Regards,
>> Krzysztof
>
> --
> Joel Sherrill, Ph.D.             Director of Research & Development
> joel.sherrill at OARcorp.com         On-Line Applications Research
> Ask me about RTEMS: a free RTOS  Huntsville AL 35805
> Support Available                 (256) 722-9985
>
>
>
> _______________________________________________
> rtems-devel mailing list
> rtems-devel at rtems.org
> http://www.rtems.org/mailman/listinfo/rtems-devel
>