[SOCIS] Rtems-testing related project - more details
Joel Sherrill
joel.sherrill at OARcorp.com
Thu May 8 15:48:44 UTC 2014
On 5/7/2014 6:03 PM, Chris Johns wrote:
> On 8/05/2014 8:15 am, Joel Sherrill wrote:
>> Hopefully Chris Johns will pipe up and add more details.
>> In particular, where his rtems-test is. I think it can be used
>> on the beagle board and some gdb simulators now.
> The RTEMS Tester is an RTEMS Tools Project and can be found in the
> rtems-tools.git repo. The web interface to the repo is
> http://git.rtems.org/rtems-tools/. The tester is under the tester
> directory. There is also a doc directory with some very basic doco.
>
> It supports the gdb run command, gdb via the MI interface with gdb
> simulators and actual hardware via BDM or JTAG, and qemu is support. It
> provides serial port access with full termios control.
>
> The supported BSPs can be seen here ...
>
> http://git.rtems.org/rtems-tools/tree/tester/rtems/testing/bsps
>
> The testing back end support can be seen here ..
>
> http://git.rtems.org/rtems-tools/tree/tester/rtems/testing
>
>> On 5/7/2014 4:18 PM, Krzysztof Mięsowicz wrote:
>>> Hi,
>>>
>>> My name is Krzysztof Mięsowicz. I am interested in participating again
>>> in SOCIS this year and I am writing to you in hope that you could
>>> provide me some more details about possible projects related to new
>>> rtems-test infrastructure.
>>>
>>> As dr Joel wrote on Rtems-Users facebook group, there are some
>>> possible improvements: (cited)
>>>
>>> This year we want to encourage the SOCIS students to focus on
>>> improvements to our new rtems-test Python-based framework. This
>>> will include:
>>> + adding more BSPs (simulators and real boards) to rtems-test
>>>
>> Chris will have to point us to his rtems-test repository. I can't seem to
>> locate it.
> See above.
>
>> It is in Python and drives real hardware debuggers
>> and simulators using the GDB/MI.
>>
>> rtems-testing/sim-scripts supports more simulators but does not support
>> hardware debuggers. rtems-test supports a small subset of the simulators
>> supported by rtems-testing/sim-scripts. rtems-test needs to be worked
>> on so it can replace all uses of sim-scripts and obsolete them.
>>
>>> + building more simulators with the RTEMS Source
>>> - if successful, sim-scripts should be obsolete
>>>
>> That should be RTEMS Source Builder.
>> http://git.rtems.org/rtems-source-builder/
>> RSB builds the basic tools and a couple of simulator configurations.
>> RSB and rtems-tools work together to provide a framework to manage
>> build instructions and patch sets for RTEMS tools. We want to expand
>> it so it can build every tool one needs cross. Simulators are the next
>> step.
> The RSB builds qemu on most hosts. Building qemu for Windows as a
> Canadian cross is complex and I have not managed to sort this out so
> Windows is not working. Supporting Windows is outside the scope of this
> work.
>
> There is work in tested the qemu's build and to get BSP support added to
> the rtems-tester. I am sure as you point out later in this email
> specific patches are needed to make things work.
>
>> Eventually, we would like to be able to even build add-on libraries
>> to target a specific BSP. But that is longer down the road. If someone
>> gets to it sooner, great!
>>
> I have added some basic support for this via pkgconfig. The RSB contains
> a private implementation and that can be used to help manage this task.
> I still have some things to sort out and I am not sure this is ready to
> have libraries added.
Agreed. As I say below, we need to focus on automating and improving
what we have
now, not growing that.
With that said, it would be nice to see if rtems-addon-packages could be
killed with
a few scripts in the RSB. :)
>> RSB needs to be augmented to be able to build all the simulators
>> external to gdb that we currently use. This may involve some odd
>> cases like say an old qemu version plus a patch to do coldfire.
>> That would give a baseline to upgrade against.
> Do you have the patches ? The RSB qemu support currently references some
> zynq patches on Qemu's patchworks so if you can find them there we can
> directly reference them.
I would have to dig through them but unfortunately, some are against
very specific
versions of qemu and have not been updated. I don't know how we want to
address
this. Say coldfire qemu starts with version X + patch as a baseline and
them push
that to the newest?
You know how accepting qemu is of patches. :(
FYI the coldfire patch is Till's and I attempted once in July 2011 to
submit it.
I don't recall if I ever followed up on Alexander's comments. I think
the last
time I tried, the code had been reworked to the point I couldn't tell
what to
do quickly.
http://lists.nongnu.org/archive/html/qemu-devel/2011-07/msg02067.html
Certainly taking another stab at that patch would be a good thing.
Especially
since we can do coverage analysis with runs on qemu. I did uC5282 using
that patched qemu.
The arm zynq would also be a good candidate since we could get SMP
coverage also from a single free simulator.
>>> + integrating coverage runs into rtems-test
>>>
>> rtems-testing/covoar (C++) and rtems-testing/rtems-coverage (shell)
>> together do RTEMS test runs for coverage and generate the coverage
>> reports on http://www.rtems.org/ftp/pub/rtems/people/joel/coverage/
>>
>> The rtems-test tool needs to be able to do coverage runs. It needs to know
>> if a target supports coverage, how to turn on coverage for a target, how
>> to run the report generator, etc.
>>
> This code needs to move into the rtems-tools repo under tester/covoar
> and a waf build script added. This will make the rtems-tools.git repo
> self contained.
covoar is not RTEMS specific. The associated shell scripts which drive
it in an
RTEMS specific way also need to be included in this.
Chris asked privately about report generation.
The per-coverage configuration/BSP run report generation is completely
contained
in covoar (C++). A single run is done by covoar. The wrapper for
generating multiple
bsps and the index pages is in the shell script.
>>> + improvements to the coverage test reporting
>>> - reporting should be on "module" level. Too coarse now.
>>>
>> The report generator itself needs a refresh. Right now, it reports on two
>> "lumps" of code -- core and developmental. Read "critical core" and
>> "the kitchen sink". :)
> When you say report generator do you mean coverage report generator ?
>
> What host tools does this report generator use ? My aim is to have
> rtems-tools as self contains as possible.
No external tools. Direct output from covoar for the per BSP reports and
shell script wrapper for the summary tables of runs per BSP/configuration
and the top level list of BSPs.
>> Ideally, the coverage reports should be generated based on modules
>> or source directories in cpukit/. Then we can get the coverage on
>> score/ separate from shell/, fatfs/, or imfs. What we have seen is that
>> each time we add a module to the "kitchen sink" set, the covered
>> percentage drops because the newly added module needs improvement
>> in test coverage.
> Do have more details about this task and how it would proceed ?
The first thing to understand is that covoar takes a set of .exe's,
coverage files
from the simulators, and a list of symbols of interest. You can run covoar
multiple times against the same set of .exes/.cov files with different
symbol
sets.
The procedure is as follows:
(1) build rtems in a specific configuration.
Right now, options are -O2 or -Os and POSIX enabled/disabled.
Eventually, we need to add smp enable/disable for some BSPs.
(2) run all tests on simulator with trace/coverage enabled
(3) wrapper script does:
for each "subset of source"
generate a list of symbols from compiled object
run covoar
For example, core and developmental (all) reports are two invocations
of covoar with two different symbol lists.
(4) Wrapper shell script, then adds the runs to the per BSP/config page
and generates the top level BSP index.html page in case it is a
new BSP.
Right now, each run of covoar is an entry in the "per BSP/config" page.
I think it will change to where a single coverage test/report run invokes
covoar multiple times and generates multiple report sets that are
part of the single "run for a configuration". With each report on a
symbol set being a subdirectory under the master "this run" directory.
The directory structure will end up being an extension of the current with
the directory name encoding being tweaked:
+ erc32-OsP (run for erc32 at -Os and POSIX enabled) [0]
index.html - nice links to sub-areas, summation, etc.
- all/ - equivalent to a "developmental" report now
- critical/ - equivalent to core now (name can change)
- score/ - contents look like a current coverage report
like [1] but focused on a single "module" named score.
- rtems/ ...
- posix/ ...
- libcsupport/ ...
- fatfs/...
...
- there may be 20 or more of these. Each requires generating a
symbol set and running covoar.
[0] Note that the last letter is not D or d. This is a type of report
based on a particular symbol subset being analyzed for this build.
We may want [Ss] on the end of the name to indicate S for SMP
enabled or s for disabled. Add s now always until SMP runs
supported.
[1]
http://www.rtems.org/ftp/pub/rtems/people/joel/coverage/erc32/erc32-Ospd-20121220-1418/
>> covoar can generate .gcov output but no one has ever validated it
>> against "truth" and seen if gcov, lcov, etc. can produce trustworth
>> reports with it.
>>> - some scripts could be migrated to Python
>>>
>>>
>> The older code is in bash. Migrating it to Python as it is reworked is
>> desirable.
>>> I wonder which of these are of highest priorities and what will it
>>> mean exactly. Could you provide more details about these ideas and
>>> maybe point some starting points to get more familiar and prepare
>>> better proposal?
>>>
>> Everything is high priority. :)
>>
>> I guess IMO I would like to see one BSP/simulator pushed through the
>> entire software stack, reporting improved, and shell transitioned to
>> Python as it was convenient/expedient. Adding other simulators
>> to the RSB and rtems-test/ is a bonus. So if you want to assume we
>> get one student from SOCIS, that would be my suggestion.
>>
>> My thinking is that if SOCIS gives us two or more students, one could
>> easily start at the rtems-test/ level integrating coverage runs as they
>> are now. Another student could focus on the coverage report generation.
>>
>> Transitioning more simulators from sim-scripts/ to rtems-test/
>> and adding building more simulators to the RSB could be added
>> easily. Each simulator instance is more or less independent so
>> any number of students could work off a check list.
>>
>> If we get two or more students, the problem can be tackled from
>> multiple ends.
> Another and more challenging task is to add support to test networking.
> This would be via the VDE support in qemu. See
> http://wiki.v2.cs.unibo.it/wiki/index.php/VDE. I do not have anything
> more to add on this topic other than this looks like a nice to do this.
I would view this as a longer term goal. We do need to test networking but
we also need improvements in the automation of running, posting,
interpreting, and generating alerts for the functional, coverage, and
performance
tests we have now.
I would rather have more focus on what we have and not add networking as
a requirement.
Better to test less well than more in a poor manual fashion.
>>> Regards,
>>> Krzysztof
>> --
>> Joel Sherrill, Ph.D. Director of Research & Development
>> joel.sherrill at OARcorp.com On-Line Applications Research
>> Ask me about RTEMS: a free RTOS Huntsville AL 35805
>> Support Available (256) 722-9985
>>
>>
>>
>> _______________________________________________
>> rtems-devel mailing list
>> rtems-devel at rtems.org
>> http://www.rtems.org/mailman/listinfo/rtems-devel
>>
--
Joel Sherrill, Ph.D. Director of Research & Development
joel.sherrill at OARcorp.com On-Line Applications Research
Ask me about RTEMS: a free RTOS Huntsville AL 35805
Support Available (256) 722-9985
More information about the devel
mailing list