[SOCIS] Rtems-testing related project - more details
Krzysztof Mięsowicz
krzysztof.miesowicz at gmail.com
Sat May 10 08:54:23 UTC 2014
Thank you for your replies!
Currently, I am preparing environment and I am going to prepare proposal on
Google Docs till tomorrow. And then I will share it with you for review and
maybe additional improvements to it.
9 maj 2014 04:12 "Chris Johns" <chrisj at rtems.org> napisał(a):
> On 9/05/2014 1:48 am, Joel Sherrill wrote:
>
>>
>> On 5/7/2014 6:03 PM, Chris Johns wrote:
>>
>>> On 8/05/2014 8:15 am, Joel Sherrill wrote:
>>>
>>>> Hopefully Chris Johns will pipe up and add more details.
>>>> In particular, where his rtems-test is. I think it can be used
>>>> on the beagle board and some gdb simulators now.
>>>>
>>> The RTEMS Tester is an RTEMS Tools Project and can be found in the
>>> rtems-tools.git repo. The web interface to the repo is
>>> http://git.rtems.org/rtems-tools/. The tester is under the tester
>>> directory. There is also a doc directory with some very basic doco.
>>>
>>> It supports the gdb run command, gdb via the MI interface with gdb
>>> simulators and actual hardware via BDM or JTAG, and qemu is support. It
>>> provides serial port access with full termios control.
>>>
>>> The supported BSPs can be seen here ...
>>>
>>> http://git.rtems.org/rtems-tools/tree/tester/rtems/testing/bsps
>>>
>>> The testing back end support can be seen here ..
>>>
>>> http://git.rtems.org/rtems-tools/tree/tester/rtems/testing
>>>
>>> On 5/7/2014 4:18 PM, Krzysztof Mięsowicz wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> My name is Krzysztof Mięsowicz. I am interested in participating again
>>>>> in SOCIS this year and I am writing to you in hope that you could
>>>>> provide me some more details about possible projects related to new
>>>>> rtems-test infrastructure.
>>>>>
>>>>> As dr Joel wrote on Rtems-Users facebook group, there are some
>>>>> possible improvements: (cited)
>>>>>
>>>>> This year we want to encourage the SOCIS students to focus on
>>>>> improvements to our new rtems-test Python-based framework. This
>>>>> will include:
>>>>> + adding more BSPs (simulators and real boards) to rtems-test
>>>>>
>>>>> Chris will have to point us to his rtems-test repository. I can't
>>>> seem to
>>>> locate it.
>>>>
>>> See above.
>>>
>>> It is in Python and drives real hardware debuggers
>>>> and simulators using the GDB/MI.
>>>>
>>>> rtems-testing/sim-scripts supports more simulators but does not support
>>>> hardware debuggers. rtems-test supports a small subset of the simulators
>>>> supported by rtems-testing/sim-scripts. rtems-test needs to be worked
>>>> on so it can replace all uses of sim-scripts and obsolete them.
>>>>
>>>> + building more simulators with the RTEMS Source
>>>>> - if successful, sim-scripts should be obsolete
>>>>>
>>>>> That should be RTEMS Source Builder.
>>>> http://git.rtems.org/rtems-source-builder/
>>>> RSB builds the basic tools and a couple of simulator configurations.
>>>> RSB and rtems-tools work together to provide a framework to manage
>>>> build instructions and patch sets for RTEMS tools. We want to expand
>>>> it so it can build every tool one needs cross. Simulators are the next
>>>> step.
>>>>
>>> The RSB builds qemu on most hosts. Building qemu for Windows as a
>>> Canadian cross is complex and I have not managed to sort this out so
>>> Windows is not working. Supporting Windows is outside the scope of this
>>> work.
>>>
>>> There is work in tested the qemu's build and to get BSP support added to
>>> the rtems-tester. I am sure as you point out later in this email
>>> specific patches are needed to make things work.
>>>
>>> Eventually, we would like to be able to even build add-on libraries
>>>> to target a specific BSP. But that is longer down the road. If someone
>>>> gets to it sooner, great!
>>>>
>>>> I have added some basic support for this via pkgconfig. The RSB
>>> contains
>>> a private implementation and that can be used to help manage this task.
>>> I still have some things to sort out and I am not sure this is ready to
>>> have libraries added.
>>>
>> Agreed. As I say below, we need to focus on automating and improving
>> what we have
>> now, not growing that.
>>
>> With that said, it would be nice to see if rtems-addon-packages could be
>> killed with
>> a few scripts in the RSB. :)
>>
>
> Yes in the end the process needs to be as simple as possible, if however
> the RSB needs functionality added I need to get that done first. Here is
> the road block. Currently this activity is unfunded so it waits until I
> have the time. It is a shame it is taking time because the benefit to users
> in having easy to access 3rd party libraries is big.
>
> RSB needs to be augmented to be able to build all the simulators
>>>> external to gdb that we currently use. This may involve some odd
>>>> cases like say an old qemu version plus a patch to do coldfire.
>>>> That would give a baseline to upgrade against.
>>>>
>>> Do you have the patches ? The RSB qemu support currently references some
>>> zynq patches on Qemu's patchworks so if you can find them there we can
>>> directly reference them.
>>>
>> I would have to dig through them but unfortunately, some are against
>> very specific
>> versions of qemu and have not been updated. I don't know how we want to
>> address
>> this. Say coldfire qemu starts with version X + patch as a baseline and
>> them push
>> that to the newest?
>>
>>
> Can you build qemu with the RSB and try and see what works and what fails
> ? They might be fixed. The RSB is building a git version of qemu.
>
> You know how accepting qemu is of patches. :(
>>
>
> I wish I did know what happens here. It is a complete mystery to me. ;)
>
> FYI the coldfire patch is Till's and I attempted once in July 2011 to
>> submit it.
>> I don't recall if I ever followed up on Alexander's comments. I think
>> the last
>> time I tried, the code had been reworked to the point I couldn't tell
>> what to
>> do quickly.
>>
>> http://lists.nongnu.org/archive/html/qemu-devel/2011-07/msg02067.html
>>
>> Certainly taking another stab at that patch would be a good thing.
>> Especially
>> since we can do coverage analysis with runs on qemu. I did uC5282 using
>> that patched qemu.
>>
>
> The qemu patches are listed here ..
>
> http://patchwork.ozlabs.org/project/qemu-devel/list/
>
> and a cli client for patchworks is here ...
>
> http://patchwork.ozlabs.org/help/pwclient/
>
> With the client you can search for patches. I found it easier than the web
> interface. The pwclient config for qemu is a link on this page ..
>
> http://patchwork.ozlabs.org/project/qemu-devel/
>
>
>> The arm zynq would also be a good candidate since we could get SMP
>> coverage also from a single free simulator.
>>
>
> The zynq is built by the RSB and works with the rtems-tester.
>
> + integrating coverage runs into rtems-test
>>>>>
>>>>> rtems-testing/covoar (C++) and rtems-testing/rtems-coverage (shell)
>>>> together do RTEMS test runs for coverage and generate the coverage
>>>> reports on http://www.rtems.org/ftp/pub/rtems/people/joel/coverage/
>>>>
>>>> The rtems-test tool needs to be able to do coverage runs. It needs to
>>>> know
>>>> if a target supports coverage, how to turn on coverage for a target, how
>>>> to run the report generator, etc.
>>>>
>>>> This code needs to move into the rtems-tools repo under tester/covoar
>>> and a waf build script added. This will make the rtems-tools.git repo
>>> self contained.
>>>
>> covoar is not RTEMS specific. The associated shell scripts which drive
>> it in an
>> RTEMS specific way also need to be included in this.
>>
>> Chris asked privately about report generation.
>>
>> The per-coverage configuration/BSP run report generation is completely
>> contained
>> in covoar (C++). A single run is done by covoar. The wrapper for
>> generating multiple
>> bsps and the index pages is in the shell script.
>>
>>> + improvements to the coverage test reporting
>>>>> - reporting should be on "module" level. Too coarse now.
>>>>>
>>>>> The report generator itself needs a refresh. Right now, it reports on
>>>> two
>>>> "lumps" of code -- core and developmental. Read "critical core" and
>>>> "the kitchen sink". :)
>>>>
>>> When you say report generator do you mean coverage report generator ?
>>>
>>> What host tools does this report generator use ? My aim is to have
>>> rtems-tools as self contains as possible.
>>>
>> No external tools. Direct output from covoar for the per BSP reports and
>> shell script wrapper for the summary tables of runs per BSP/configuration
>> and the top level list of BSPs.
>>
>
> Ok. The shell scripts may be converted to Python or the tester config
> language.
>
> Ideally, the coverage reports should be generated based on modules
>>>> or source directories in cpukit/. Then we can get the coverage on
>>>> score/ separate from shell/, fatfs/, or imfs. What we have seen is that
>>>> each time we add a module to the "kitchen sink" set, the covered
>>>> percentage drops because the newly added module needs improvement
>>>> in test coverage.
>>>>
>>> Do have more details about this task and how it would proceed ?
>>>
>> The first thing to understand is that covoar takes a set of .exe's,
>> coverage files
>> from the simulators, and a list of symbols of interest. You can run covoar
>> multiple times against the same set of .exes/.cov files with different
>> symbol
>> sets.
>>
>
> Ok. I wonder if multiple tables and used and a single pass over the trace
> data. It might be faster. This is something we can look at once the basics
> are integrated.
>
>
>> The procedure is as follows:
>>
>> (1) build rtems in a specific configuration.
>> Right now, options are -O2 or -Os and POSIX enabled/disabled.
>> Eventually, we need to add smp enable/disable for some BSPs.
>>
>> (2) run all tests on simulator with trace/coverage enabled
>>
>> (3) wrapper script does:
>> for each "subset of source"
>> generate a list of symbols from compiled object
>> run covoar
>> For example, core and developmental (all) reports are two
>> invocations
>> of covoar with two different symbol lists.
>>
>> (4) Wrapper shell script, then adds the runs to the per BSP/config page
>> and generates the top level BSP index.html page in case it is a
>> new BSP.
>>
>> Right now, each run of covoar is an entry in the "per BSP/config" page.
>> I think it will change to where a single coverage test/report run invokes
>> covoar multiple times and generates multiple report sets that are
>> part of the single "run for a configuration". With each report on a
>> symbol set being a subdirectory under the master "this run" directory.
>>
>> The directory structure will end up being an extension of the current with
>> the directory name encoding being tweaked:
>>
>> + erc32-OsP (run for erc32 at -Os and POSIX enabled) [0]
>> index.html - nice links to sub-areas, summation, etc.
>> - all/ - equivalent to a "developmental" report now
>> - critical/ - equivalent to core now (name can change)
>> - score/ - contents look like a current coverage report
>> like [1] but focused on a single "module" named score.
>> - rtems/ ...
>> - posix/ ...
>> - libcsupport/ ...
>> - fatfs/...
>> ...
>> - there may be 20 or more of these. Each requires generating a
>> symbol set and running covoar.
>>
>> [0] Note that the last letter is not D or d. This is a type of report
>> based on a particular symbol subset being analyzed for this build.
>> We may want [Ss] on the end of the name to indicate S for SMP
>> enabled or s for disabled. Add s now always until SMP runs
>> supported.
>> [1]
>> http://www.rtems.org/ftp/pub/rtems/people/joel/coverage/
>> erc32/erc32-Ospd-20121220-1418/
>>
>>
>>
>> covoar can generate .gcov output but no one has ever validated it
>>>> against "truth" and seen if gcov, lcov, etc. can produce trustworth
>>>> reports with it.
>>>>
>>>>> - some scripts could be migrated to Python
>>>>>
>>>>>
>>>>> The older code is in bash. Migrating it to Python as it is reworked is
>>>> desirable.
>>>>
>>>>> I wonder which of these are of highest priorities and what will it
>>>>> mean exactly. Could you provide more details about these ideas and
>>>>> maybe point some starting points to get more familiar and prepare
>>>>> better proposal?
>>>>>
>>>>> Everything is high priority. :)
>>>>
>>>> I guess IMO I would like to see one BSP/simulator pushed through the
>>>> entire software stack, reporting improved, and shell transitioned to
>>>> Python as it was convenient/expedient. Adding other simulators
>>>> to the RSB and rtems-test/ is a bonus. So if you want to assume we
>>>> get one student from SOCIS, that would be my suggestion.
>>>>
>>>> My thinking is that if SOCIS gives us two or more students, one could
>>>> easily start at the rtems-test/ level integrating coverage runs as they
>>>> are now. Another student could focus on the coverage report generation.
>>>>
>>>> Transitioning more simulators from sim-scripts/ to rtems-test/
>>>> and adding building more simulators to the RSB could be added
>>>> easily. Each simulator instance is more or less independent so
>>>> any number of students could work off a check list.
>>>>
>>>> If we get two or more students, the problem can be tackled from
>>>> multiple ends.
>>>>
>>> Another and more challenging task is to add support to test networking.
>>> This would be via the VDE support in qemu. See
>>> http://wiki.v2.cs.unibo.it/wiki/index.php/VDE. I do not have anything
>>> more to add on this topic other than this looks like a nice to do this.
>>>
>>
>> I would view this as a longer term goal. We do need to test networking but
>> we also need improvements in the automation of running, posting,
>> interpreting, and generating alerts for the functional, coverage, and
>> performance
>> tests we have now.
>>
>> I would rather have more focus on what we have and not add networking as
>> a requirement.
>> Better to test less well than more in a poor manual fashion.
>>
>
> I agree. I also know the standard from the last GSoC and CGI was so high I
> needed more challenging tasks. :)
>
> Chris
>
> Regards,
>>>>> Krzysztof
>>>>>
>>>> --
>>>> Joel Sherrill, Ph.D. Director of Research & Development
>>>> joel.sherrill at OARcorp.com On-Line Applications Research
>>>> Ask me about RTEMS: a free RTOS Huntsville AL 35805
>>>> Support Available (256) 722-9985
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> rtems-devel mailing list
>>>> rtems-devel at rtems.org
>>>> http://www.rtems.org/mailman/listinfo/rtems-devel
>>>>
>>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rtems.org/pipermail/devel/attachments/20140510/de29f6ba/attachment.html>
More information about the devel
mailing list