<p dir="ltr">Thank you for your replies!</p>
<p dir="ltr">Currently, I am preparing environment and I am going to prepare proposal on Google Docs till tomorrow. And then I will share it with you for review and maybe additional improvements to it.</p>
<div class="gmail_quote">9 maj 2014 04:12 "Chris Johns" <<a href="mailto:chrisj@rtems.org">chrisj@rtems.org</a>> napisał(a):<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On 9/05/2014 1:48 am, Joel Sherrill wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
On 5/7/2014 6:03 PM, Chris Johns wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On 8/05/2014 8:15 am, Joel Sherrill wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hopefully Chris Johns will pipe up and add more details.<br>
In particular, where his rtems-test is. I think it can be used<br>
on the beagle board and some gdb simulators now.<br>
</blockquote>
The RTEMS Tester is an RTEMS Tools Project and can be found in the<br>
rtems-tools.git repo. The web interface to the repo is<br>
<a href="http://git.rtems.org/rtems-tools/" target="_blank">http://git.rtems.org/rtems-<u></u>tools/</a>. The tester is under the tester<br>
directory. There is also a doc directory with some very basic doco.<br>
<br>
It supports the gdb run command, gdb via the MI interface with gdb<br>
simulators and actual hardware via BDM or JTAG, and qemu is support. It<br>
provides serial port access with full termios control.<br>
<br>
The supported BSPs can be seen here ...<br>
<br>
<a href="http://git.rtems.org/rtems-tools/tree/tester/rtems/testing/bsps" target="_blank">http://git.rtems.org/rtems-<u></u>tools/tree/tester/rtems/<u></u>testing/bsps</a><br>
<br>
The testing back end support can be seen here ..<br>
<br>
<a href="http://git.rtems.org/rtems-tools/tree/tester/rtems/testing" target="_blank">http://git.rtems.org/rtems-<u></u>tools/tree/tester/rtems/<u></u>testing</a><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On 5/7/2014 4:18 PM, Krzysztof Mięsowicz wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi,<br>
<br>
My name is Krzysztof Mięsowicz. I am interested in participating again<br>
in SOCIS this year and I am writing to you in hope that you could<br>
provide me some more details about possible projects related to new<br>
rtems-test infrastructure.<br>
<br>
As dr Joel wrote on Rtems-Users facebook group, there are some<br>
possible improvements: (cited)<br>
<br>
This year we want to encourage the SOCIS students to focus on<br>
improvements to our new rtems-test Python-based framework. This<br>
will include:<br>
+ adding more BSPs (simulators and real boards) to rtems-test<br>
<br>
</blockquote>
Chris will have to point us to his rtems-test repository. I can't seem to<br>
locate it.<br>
</blockquote>
See above.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
It is in Python and drives real hardware debuggers<br>
and simulators using the GDB/MI.<br>
<br>
rtems-testing/sim-scripts supports more simulators but does not support<br>
hardware debuggers. rtems-test supports a small subset of the simulators<br>
supported by rtems-testing/sim-scripts. rtems-test needs to be worked<br>
on so it can replace all uses of sim-scripts and obsolete them.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
+ building more simulators with the RTEMS Source<br>
- if successful, sim-scripts should be obsolete<br>
<br>
</blockquote>
That should be RTEMS Source Builder.<br>
<a href="http://git.rtems.org/rtems-source-builder/" target="_blank">http://git.rtems.org/rtems-<u></u>source-builder/</a><br>
RSB builds the basic tools and a couple of simulator configurations.<br>
RSB and rtems-tools work together to provide a framework to manage<br>
build instructions and patch sets for RTEMS tools. We want to expand<br>
it so it can build every tool one needs cross. Simulators are the next<br>
step.<br>
</blockquote>
The RSB builds qemu on most hosts. Building qemu for Windows as a<br>
Canadian cross is complex and I have not managed to sort this out so<br>
Windows is not working. Supporting Windows is outside the scope of this<br>
work.<br>
<br>
There is work in tested the qemu's build and to get BSP support added to<br>
the rtems-tester. I am sure as you point out later in this email<br>
specific patches are needed to make things work.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Eventually, we would like to be able to even build add-on libraries<br>
to target a specific BSP. But that is longer down the road. If someone<br>
gets to it sooner, great!<br>
<br>
</blockquote>
I have added some basic support for this via pkgconfig. The RSB contains<br>
a private implementation and that can be used to help manage this task.<br>
I still have some things to sort out and I am not sure this is ready to<br>
have libraries added.<br>
</blockquote>
Agreed. As I say below, we need to focus on automating and improving<br>
what we have<br>
now, not growing that.<br>
<br>
With that said, it would be nice to see if rtems-addon-packages could be<br>
killed with<br>
a few scripts in the RSB. :)<br>
</blockquote>
<br>
Yes in the end the process needs to be as simple as possible, if however the RSB needs functionality added I need to get that done first. Here is the road block. Currently this activity is unfunded so it waits until I have the time. It is a shame it is taking time because the benefit to users in having easy to access 3rd party libraries is big.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
RSB needs to be augmented to be able to build all the simulators<br>
external to gdb that we currently use. This may involve some odd<br>
cases like say an old qemu version plus a patch to do coldfire.<br>
That would give a baseline to upgrade against.<br>
</blockquote>
Do you have the patches ? The RSB qemu support currently references some<br>
zynq patches on Qemu's patchworks so if you can find them there we can<br>
directly reference them.<br>
</blockquote>
I would have to dig through them but unfortunately, some are against<br>
very specific<br>
versions of qemu and have not been updated. I don't know how we want to<br>
address<br>
this. Say coldfire qemu starts with version X + patch as a baseline and<br>
them push<br>
that to the newest?<br>
<br>
</blockquote>
<br>
Can you build qemu with the RSB and try and see what works and what fails ? They might be fixed. The RSB is building a git version of qemu.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
You know how accepting qemu is of patches. :(<br>
</blockquote>
<br>
I wish I did know what happens here. It is a complete mystery to me. ;)<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
FYI the coldfire patch is Till's and I attempted once in July 2011 to<br>
submit it.<br>
I don't recall if I ever followed up on Alexander's comments. I think<br>
the last<br>
time I tried, the code had been reworked to the point I couldn't tell<br>
what to<br>
do quickly.<br>
<br>
<a href="http://lists.nongnu.org/archive/html/qemu-devel/2011-07/msg02067.html" target="_blank">http://lists.nongnu.org/<u></u>archive/html/qemu-devel/2011-<u></u>07/msg02067.html</a><br>
<br>
Certainly taking another stab at that patch would be a good thing.<br>
Especially<br>
since we can do coverage analysis with runs on qemu. I did uC5282 using<br>
that patched qemu.<br>
</blockquote>
<br>
The qemu patches are listed here ..<br>
<br>
<a href="http://patchwork.ozlabs.org/project/qemu-devel/list/" target="_blank">http://patchwork.ozlabs.org/<u></u>project/qemu-devel/list/</a><br>
<br>
and a cli client for patchworks is here ...<br>
<br>
<a href="http://patchwork.ozlabs.org/help/pwclient/" target="_blank">http://patchwork.ozlabs.org/<u></u>help/pwclient/</a><br>
<br>
With the client you can search for patches. I found it easier than the web interface. The pwclient config for qemu is a link on this page ..<br>
<br>
<a href="http://patchwork.ozlabs.org/project/qemu-devel/" target="_blank">http://patchwork.ozlabs.org/<u></u>project/qemu-devel/</a><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
The arm zynq would also be a good candidate since we could get SMP<br>
coverage also from a single free simulator.<br>
</blockquote>
<br>
The zynq is built by the RSB and works with the rtems-tester.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
+ integrating coverage runs into rtems-test<br>
<br>
</blockquote>
rtems-testing/covoar (C++) and rtems-testing/rtems-coverage (shell)<br>
together do RTEMS test runs for coverage and generate the coverage<br>
reports on <a href="http://www.rtems.org/ftp/pub/rtems/people/joel/coverage/" target="_blank">http://www.rtems.org/ftp/pub/<u></u>rtems/people/joel/coverage/</a><br>
<br>
The rtems-test tool needs to be able to do coverage runs. It needs to know<br>
if a target supports coverage, how to turn on coverage for a target, how<br>
to run the report generator, etc.<br>
<br>
</blockquote>
This code needs to move into the rtems-tools repo under tester/covoar<br>
and a waf build script added. This will make the rtems-tools.git repo<br>
self contained.<br>
</blockquote>
covoar is not RTEMS specific. The associated shell scripts which drive<br>
it in an<br>
RTEMS specific way also need to be included in this.<br>
<br>
Chris asked privately about report generation.<br>
<br>
The per-coverage configuration/BSP run report generation is completely<br>
contained<br>
in covoar (C++). A single run is done by covoar. The wrapper for<br>
generating multiple<br>
bsps and the index pages is in the shell script.<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
+ improvements to the coverage test reporting<br>
- reporting should be on "module" level. Too coarse now.<br>
<br>
</blockquote>
The report generator itself needs a refresh. Right now, it reports on two<br>
"lumps" of code -- core and developmental. Read "critical core" and<br>
"the kitchen sink". :)<br>
</blockquote>
When you say report generator do you mean coverage report generator ?<br>
<br>
What host tools does this report generator use ? My aim is to have<br>
rtems-tools as self contains as possible.<br>
</blockquote>
No external tools. Direct output from covoar for the per BSP reports and<br>
shell script wrapper for the summary tables of runs per BSP/configuration<br>
and the top level list of BSPs.<br>
</blockquote>
<br>
Ok. The shell scripts may be converted to Python or the tester config language.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Ideally, the coverage reports should be generated based on modules<br>
or source directories in cpukit/. Then we can get the coverage on<br>
score/ separate from shell/, fatfs/, or imfs. What we have seen is that<br>
each time we add a module to the "kitchen sink" set, the covered<br>
percentage drops because the newly added module needs improvement<br>
in test coverage.<br>
</blockquote>
Do have more details about this task and how it would proceed ?<br>
</blockquote>
The first thing to understand is that covoar takes a set of .exe's,<br>
coverage files<br>
from the simulators, and a list of symbols of interest. You can run covoar<br>
multiple times against the same set of .exes/.cov files with different<br>
symbol<br>
sets.<br>
</blockquote>
<br>
Ok. I wonder if multiple tables and used and a single pass over the trace data. It might be faster. This is something we can look at once the basics are integrated.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
The procedure is as follows:<br>
<br>
(1) build rtems in a specific configuration.<br>
Right now, options are -O2 or -Os and POSIX enabled/disabled.<br>
Eventually, we need to add smp enable/disable for some BSPs.<br>
<br>
(2) run all tests on simulator with trace/coverage enabled<br>
<br>
(3) wrapper script does:<br>
for each "subset of source"<br>
generate a list of symbols from compiled object<br>
run covoar<br>
For example, core and developmental (all) reports are two invocations<br>
of covoar with two different symbol lists.<br>
<br>
(4) Wrapper shell script, then adds the runs to the per BSP/config page<br>
and generates the top level BSP index.html page in case it is a<br>
new BSP.<br>
<br>
Right now, each run of covoar is an entry in the "per BSP/config" page.<br>
I think it will change to where a single coverage test/report run invokes<br>
covoar multiple times and generates multiple report sets that are<br>
part of the single "run for a configuration". With each report on a<br>
symbol set being a subdirectory under the master "this run" directory.<br>
<br>
The directory structure will end up being an extension of the current with<br>
the directory name encoding being tweaked:<br>
<br>
+ erc32-OsP (run for erc32 at -Os and POSIX enabled) [0]<br>
index.html - nice links to sub-areas, summation, etc.<br>
- all/ - equivalent to a "developmental" report now<br>
- critical/ - equivalent to core now (name can change)<br>
- score/ - contents look like a current coverage report<br>
like [1] but focused on a single "module" named score.<br>
- rtems/ ...<br>
- posix/ ...<br>
- libcsupport/ ...<br>
- fatfs/...<br>
...<br>
- there may be 20 or more of these. Each requires generating a<br>
symbol set and running covoar.<br>
<br>
[0] Note that the last letter is not D or d. This is a type of report<br>
based on a particular symbol subset being analyzed for this build.<br>
We may want [Ss] on the end of the name to indicate S for SMP<br>
enabled or s for disabled. Add s now always until SMP runs<br>
supported.<br>
[1]<br>
<a href="http://www.rtems.org/ftp/pub/rtems/people/joel/coverage/erc32/erc32-Ospd-20121220-1418/" target="_blank">http://www.rtems.org/ftp/pub/<u></u>rtems/people/joel/coverage/<u></u>erc32/erc32-Ospd-20121220-<u></u>1418/</a><br>
<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
covoar can generate .gcov output but no one has ever validated it<br>
against "truth" and seen if gcov, lcov, etc. can produce trustworth<br>
reports with it.<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
- some scripts could be migrated to Python<br>
<br>
<br>
</blockquote>
The older code is in bash. Migrating it to Python as it is reworked is<br>
desirable.<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I wonder which of these are of highest priorities and what will it<br>
mean exactly. Could you provide more details about these ideas and<br>
maybe point some starting points to get more familiar and prepare<br>
better proposal?<br>
<br>
</blockquote>
Everything is high priority. :)<br>
<br>
I guess IMO I would like to see one BSP/simulator pushed through the<br>
entire software stack, reporting improved, and shell transitioned to<br>
Python as it was convenient/expedient. Adding other simulators<br>
to the RSB and rtems-test/ is a bonus. So if you want to assume we<br>
get one student from SOCIS, that would be my suggestion.<br>
<br>
My thinking is that if SOCIS gives us two or more students, one could<br>
easily start at the rtems-test/ level integrating coverage runs as they<br>
are now. Another student could focus on the coverage report generation.<br>
<br>
Transitioning more simulators from sim-scripts/ to rtems-test/<br>
and adding building more simulators to the RSB could be added<br>
easily. Each simulator instance is more or less independent so<br>
any number of students could work off a check list.<br>
<br>
If we get two or more students, the problem can be tackled from<br>
multiple ends.<br>
</blockquote>
Another and more challenging task is to add support to test networking.<br>
This would be via the VDE support in qemu. See<br>
<a href="http://wiki.v2.cs.unibo.it/wiki/index.php/VDE" target="_blank">http://wiki.v2.cs.unibo.it/<u></u>wiki/index.php/VDE</a>. I do not have anything<br>
more to add on this topic other than this looks like a nice to do this.<br>
</blockquote>
<br>
I would view this as a longer term goal. We do need to test networking but<br>
we also need improvements in the automation of running, posting,<br>
interpreting, and generating alerts for the functional, coverage, and<br>
performance<br>
tests we have now.<br>
<br>
I would rather have more focus on what we have and not add networking as<br>
a requirement.<br>
Better to test less well than more in a poor manual fashion.<br>
</blockquote>
<br>
I agree. I also know the standard from the last GSoC and CGI was so high I needed more challenging tasks. :)<br>
<br>
Chris<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Regards,<br>
Krzysztof<br>
</blockquote>
--<br>
Joel Sherrill, Ph.D. Director of Research & Development<br>
joel.sherrill@OARcorp.com On-Line Applications Research<br>
Ask me about RTEMS: a free RTOS Huntsville AL 35805<br>
Support Available <a href="tel:%28256%29%20722-9985" value="+12567229985" target="_blank">(256) 722-9985</a><br>
<br>
<br>
<br>
______________________________<u></u>_________________<br>
rtems-devel mailing list<br>
<a href="mailto:rtems-devel@rtems.org" target="_blank">rtems-devel@rtems.org</a><br>
<a href="http://www.rtems.org/mailman/listinfo/rtems-devel" target="_blank">http://www.rtems.org/mailman/<u></u>listinfo/rtems-devel</a><br>
<br>
</blockquote></blockquote>
<br>
</blockquote>
</blockquote></div>