[SOCIS] Rtems-testing related project - more details

[Chris Johns]

On 9/05/2014 1:48 am, Joel Sherrill wrote:
>
> On 5/7/2014 6:03 PM, Chris Johns wrote:
>> On 8/05/2014 8:15 am, Joel Sherrill wrote:
>>> Hopefully Chris Johns will pipe up and add more details.
>>> In particular, where his rtems-test is.  I think it can be used
>>> on the beagle board and some gdb simulators now.
>> The RTEMS Tester is an RTEMS Tools Project and can be found in the
>> rtems-tools.git repo. The web interface to the repo is
>> http://git.rtems.org/rtems-tools/. The tester is under the tester
>> directory. There is also a doc directory with some very basic doco.
>>
>> It supports the gdb run command, gdb via the MI interface with gdb
>> simulators and actual hardware via BDM or JTAG, and qemu is support. It
>> provides serial port access with full termios control.
>>
>> The supported BSPs can be seen here ...
>>
>> http://git.rtems.org/rtems-tools/tree/tester/rtems/testing/bsps
>>
>> The testing back end support can be seen here ..
>>
>> http://git.rtems.org/rtems-tools/tree/tester/rtems/testing
>>
>>> On 5/7/2014 4:18 PM, Krzysztof Mięsowicz wrote:
>>>> Hi,
>>>>
>>>> My name is Krzysztof Mięsowicz. I am interested in participating again
>>>> in SOCIS this year and I am writing to you in hope that you could
>>>> provide me some more details about possible projects related to new
>>>> rtems-test infrastructure.
>>>>
>>>> As dr Joel wrote on Rtems-Users facebook group, there are some
>>>> possible improvements: (cited)
>>>>
>>>>      This year we want to encourage the SOCIS students to focus on
>>>>      improvements to our new rtems-test Python-based framework. This
>>>>      will include:
>>>>      + adding more BSPs (simulators and real boards) to rtems-test
>>>>
>>> Chris will have to point us to his rtems-test repository. I can't seem to
>>> locate it.
>> See above.
>>
>>> It is in Python and drives real hardware debuggers
>>> and simulators using the GDB/MI.
>>>
>>> rtems-testing/sim-scripts supports more simulators but does not support
>>> hardware debuggers. rtems-test supports a small subset of the simulators
>>> supported by rtems-testing/sim-scripts. rtems-test needs to be worked
>>> on so it can replace all uses of sim-scripts and obsolete them.
>>>
>>>>      + building more simulators with the RTEMS Source
>>>>      - if successful, sim-scripts should be obsolete
>>>>
>>> That should be RTEMS Source Builder.
>>> http://git.rtems.org/rtems-source-builder/
>>> RSB builds the basic tools and a couple of simulator configurations.
>>> RSB and rtems-tools work together to provide a framework to manage
>>> build instructions and patch sets for RTEMS tools. We want to expand
>>> it so it can build every tool one needs cross. Simulators are the next
>>> step.
>> The RSB builds qemu on most hosts. Building qemu for Windows as a
>> Canadian cross is complex and I have not managed to sort this out so
>> Windows is not working. Supporting Windows is outside the scope of this
>> work.
>>
>> There is work in tested the qemu's build and to get BSP support added to
>> the rtems-tester. I am sure as you point out later in this email
>> specific patches are needed to make things work.
>>
>>> Eventually, we would like to be able to even build add-on libraries
>>> to target a specific BSP. But that is longer down the road. If someone
>>> gets to it sooner, great!
>>>
>> I have added some basic support for this via pkgconfig. The RSB contains
>> a private implementation and that can be used to help manage this task.
>> I still have some things to sort out and I am not sure this is ready to
>> have libraries added.
> Agreed. As I say below, we need to focus on automating and improving
> what we have
> now, not growing that.
>
> With that said, it would be nice to see if rtems-addon-packages could be
> killed with
> a few scripts in the RSB. :)

Yes in the end the process needs to be as simple as possible, if however 
the RSB needs functionality added I need to get that done first. Here is 
the road block. Currently this activity is unfunded so it waits until I 
have the time. It is a shame it is taking time because the benefit to 
users in having easy to access 3rd party libraries is big.

>>> RSB needs to be augmented to be able to build all the simulators
>>> external to gdb that we currently use. This may involve some odd
>>> cases like say an old qemu version plus a patch to do coldfire.
>>> That would give a baseline to upgrade against.
>> Do you have the patches ? The RSB qemu support currently references some
>> zynq patches on Qemu's patchworks so if you can find them there we can
>> directly reference them.
> I would have to dig through them but unfortunately, some are against
> very specific
> versions of qemu and have not been updated. I don't know how we want to
> address
> this. Say coldfire qemu starts with version X + patch as a baseline and
> them push
> that to the newest?
>

Can you build qemu with the RSB and try and see what works and what 
fails ? They might be fixed. The RSB is building a git version of qemu.

> You know how accepting qemu is of patches. :(

I wish I did know what happens here. It is a complete mystery to me. ;)

> FYI the coldfire patch is Till's and I attempted once in July 2011 to
> submit it.
> I don't recall if I ever followed up on Alexander's comments. I think
> the last
> time I tried, the code had been reworked to the point I couldn't tell
> what to
> do quickly.
>
> http://lists.nongnu.org/archive/html/qemu-devel/2011-07/msg02067.html
>
> Certainly taking another stab at that patch would be a good thing.
> Especially
> since we can do coverage analysis with runs on qemu. I did uC5282 using
> that patched qemu.

The qemu patches are listed here ..

http://patchwork.ozlabs.org/project/qemu-devel/list/

and a cli client for patchworks is here ...

http://patchwork.ozlabs.org/help/pwclient/

With the client you can search for patches. I found it easier than the 
web interface. The pwclient config for qemu is a link on this page ..

http://patchwork.ozlabs.org/project/qemu-devel/

>
> The arm zynq would also be a good candidate since we could get SMP
> coverage also from a single free simulator.

The zynq is built by the RSB and works with the rtems-tester.

>>>>      + integrating coverage runs into rtems-test
>>>>
>>> rtems-testing/covoar (C++) and rtems-testing/rtems-coverage (shell)
>>> together do RTEMS test runs for coverage and generate the coverage
>>> reports on http://www.rtems.org/ftp/pub/rtems/people/joel/coverage/
>>>
>>> The rtems-test tool needs to be able to do coverage runs. It needs to know
>>> if a target supports coverage, how to turn on coverage for a target, how
>>> to run the report generator, etc.
>>>
>> This code needs to move into the rtems-tools repo under tester/covoar
>> and a waf build script added. This will make the rtems-tools.git repo
>> self contained.
> covoar is not RTEMS specific. The associated shell scripts which drive
> it in an
> RTEMS specific way also need to be included in this.
>
> Chris asked privately about report generation.
>
> The per-coverage configuration/BSP run report generation is completely
> contained
> in covoar (C++). A single run is done by covoar. The wrapper for
> generating multiple
> bsps and the index pages is in the shell script.
>>>>      + improvements to the coverage test reporting
>>>>      - reporting should be on "module" level. Too coarse now.
>>>>
>>> The report generator itself needs a refresh. Right now, it reports on two
>>> "lumps" of code -- core and developmental. Read "critical core" and
>>> "the kitchen sink". :)
>> When you say report generator do you mean coverage report generator ?
>>
>> What host tools does this report generator use ? My aim is to have
>> rtems-tools as self contains as possible.
> No external tools. Direct output from covoar for the per BSP reports and
> shell script wrapper for the summary tables of runs per BSP/configuration
> and the top level list of BSPs.

Ok. The shell scripts may be converted to Python or the tester config 
language.

>>> Ideally, the coverage reports should be generated based on modules
>>> or source directories in cpukit/. Then we can get the coverage on
>>> score/ separate from shell/, fatfs/, or imfs. What we have seen is that
>>> each time we add a module to the "kitchen sink" set, the covered
>>> percentage drops because the newly added module needs improvement
>>> in test coverage.
>> Do have more details about this task and how it would proceed ?
> The first thing to understand is that covoar takes a set of .exe's,
> coverage files
> from the simulators, and a list of symbols of interest. You can run covoar
> multiple times against the same set of .exes/.cov files with different
> symbol
> sets.

Ok. I wonder if multiple tables and used and a single pass over the 
trace data. It might be faster. This is something we can look at once 
the basics are integrated.

>
> The procedure is as follows:
>
> (1) build rtems in a specific configuration.
>        Right now, options are -O2 or -Os and POSIX enabled/disabled.
>        Eventually, we need to add smp enable/disable for some BSPs.
>
> (2) run all tests on simulator with trace/coverage enabled
>
> (3) wrapper script does:
>         for each "subset of source"
>             generate a list of symbols from compiled object
>            run covoar
>       For example, core and developmental (all) reports are two invocations
>       of covoar with two different symbol lists.
>
> (4) Wrapper shell script, then adds the runs to the per BSP/config page
>        and generates the top level BSP index.html page in case it is a
> new BSP.
>
> Right now, each run of covoar is an entry in the "per BSP/config" page.
> I think it will change to where a single coverage test/report run invokes
> covoar multiple times and generates multiple report sets that are
> part of the single "run for a configuration". With each report on a
> symbol set being a subdirectory under the master "this run" directory.
>
> The directory structure will end up being an extension of the current with
> the directory name encoding being tweaked:
>
> + erc32-OsP  (run for erc32 at -Os and POSIX enabled) [0]
>    index.html - nice links to sub-areas, summation, etc.
>     - all/ - equivalent to a "developmental" report now
>     - critical/ - equivalent to core now (name can change)
>     - score/ - contents look like a current coverage report
>        like [1] but focused on a single "module" named score.
>    - rtems/ ...
>    - posix/ ...
>    - libcsupport/ ...
>    - fatfs/...
>       ...
>    - there may be 20 or more of these. Each requires generating a
>      symbol set and running covoar.
>
> [0] Note that the last letter is not D or d. This is a type of report
>        based on a particular symbol subset being analyzed for this build.
>        We may want [Ss] on the end of the name to indicate S for SMP
>        enabled or s for disabled. Add s now always until SMP runs
>       supported.
> [1]
> http://www.rtems.org/ftp/pub/rtems/people/joel/coverage/erc32/erc32-Ospd-20121220-1418/
>
>
>
>>> covoar can generate .gcov output but no one has ever validated it
>>> against "truth" and seen if gcov, lcov, etc. can produce trustworth
>>> reports with it.
>>>>      - some scripts could be migrated to Python
>>>>
>>>>
>>> The older code is in bash. Migrating it to Python as it is reworked is
>>> desirable.
>>>> I wonder which of these are of highest priorities and what will it
>>>> mean exactly. Could you provide more details about these ideas and
>>>> maybe point some starting points to get more familiar and prepare
>>>> better proposal?
>>>>
>>> Everything is high priority. :)
>>>
>>> I guess IMO I would like to see one BSP/simulator pushed through the
>>> entire software stack, reporting improved, and shell transitioned to
>>> Python as it was convenient/expedient. Adding other simulators
>>> to the RSB and rtems-test/ is a bonus.  So if you want to assume we
>>> get one student from SOCIS, that would be my suggestion.
>>>
>>> My thinking is that if SOCIS gives us two or more students, one could
>>> easily start at the rtems-test/ level integrating coverage runs as they
>>> are now. Another student could focus on the coverage report generation.
>>>
>>> Transitioning more simulators from sim-scripts/ to rtems-test/
>>> and adding building more simulators to the RSB could be added
>>> easily. Each simulator instance is more or less independent so
>>> any number of students could work off a check list.
>>>
>>> If we get two or more students, the problem can be tackled from
>>> multiple ends.
>> Another and more challenging task is to add support to test networking.
>> This would be via the VDE support in qemu. See
>> http://wiki.v2.cs.unibo.it/wiki/index.php/VDE. I do not have anything
>> more to add on this topic other than this looks like a nice to do this.
>
> I would view this as a longer term goal. We do need to test networking but
> we also need improvements in the automation of running, posting,
> interpreting, and generating alerts for the functional, coverage, and
> performance
> tests we have now.
>
> I would rather have more focus on what we have and not add networking as
> a requirement.
>   Better to test less well than more in a poor manual fashion.

I agree. I also know the standard from the last GSoC and CGI was so high 
I needed more challenging tasks. :)

Chris

>>>> Regards,
>>>> Krzysztof
>>> --
>>> Joel Sherrill, Ph.D.             Director of Research & Development
>>> joel.sherrill at OARcorp.com         On-Line Applications Research
>>> Ask me about RTEMS: a free RTOS  Huntsville AL 35805
>>> Support Available                 (256) 722-9985
>>>
>>>
>>>
>>> _______________________________________________
>>> rtems-devel mailing list
>>> rtems-devel at rtems.org
>>> http://www.rtems.org/mailman/listinfo/rtems-devel
>>>
>