ESA SOCIS - Fault injection tools topic
Joel Sherrill
joel.sherrill at OARcorp.com
Thu Aug 1 13:42:13 UTC 2013
On 7/29/2013 3:09 PM, Krzysztof Mięsowicz wrote:
> Thank you for all replies very much.
>
> I read about libfiu and it seems really interesting and I believe it
> can be quite easy applied to RTEMS testing. I hope so :)
>
The process of using preloaded libraries to intercept calls and
break them may not work on RTEMS at all. Plus the idea of
running a process under another process is foreign since we
only have one process. I don't have a good feeling on libfiu
for RTEMS on an admittedly cursory examination.
API parameter fuzzing was done in the report I posted.
I did a quick google and found http://peachfuzzer.com/
which looks like a FLOSS software package to what was
done in that report. The Classic APIs were described in
terms of their data types. For each data type, "interesting"
values were defined. Say NULL for argument pointers,
or 0xfffffffc and 0xffffffff for the start of a region or
partition.
As I recall, the challenge is that the way RTEMS APIs
work, you often need a valid object ID or N-1 parameters
to be valid to get to the Nth parameter. So there was
some custom test setup to write to go along with the
automatically generated fuzzing code. Depending on the
fuzzing package, this may be handled better.
From my perspective, API parameter fuzzing was valuable
at the time. It turned up some odd bugs that wouldn't
likely have been caught otherwise.
With all that said, I don't know how much you can
accomplish in the time period. The goal would have
to be to provide the foundation for a permanent
addition to the test suite which can be added to.
You can't possibly finish all APIs so we would have to
define an interesting subset.
> According to Joel's mail:
>
> /Another area which we need help in is the coverage and testing
> scripts. The coverage reporting really should be by directory or
> capability. It is now reported in "lumps" that are too large.
> Also rewriting as much of this as possible in Python including the
> sim-scripts is highly desirable./
>
> Do you mean that coverage reporting framework should be rewrited and
> reports should be generated for source directories instead of core and
> developmental parts (as it is done now) ?
Yes. Having core is OK but developmental really sucks. When I
added filesystems, the overall percentage dropped significantly.
What we want to see from an analysis and marketing perspective
is that directory X is 99.9% but this new directory Y is at 50%
and needs testing attention.
As we move more into continuous integration testing, this
fine grained reporting will be even more valuable.
The scripts that drive this process are in shell. We are moving to using
Python as our host tools scripting language. So these should be
rewritten.
This project is more defined and has a short term completion.
> And what particularly do you mean by "rewriting as much of THIS as
> possible" - do you mean all coverage reporting scripts which is now
> written as bash scripts? Do I understand you correctly?
>
Yes. There is "covoar" which is C++ but the contents of rtems-coverage
are pretty
much all shell scripts with some template files.
The rtems-testing git module has sim-scripts to run simulators and we want
to move them to Python as well. In the process we want to both them to
a single command rather than one per BSP. Invoked omething like
rtems-sim [args] BSP
And rtems-testing has other scripts.
> Greetings,
> Krzysiek
>
>
> 2013/7/29 Rempel, Cynthia <cynt6007 at vandals.uidaho.edu
> <mailto:cynt6007 at vandals.uidaho.edu>>
>
> Hi,
>
> Did a quick write-up on the fault-injection open-project...
> http://blitiri.com.ar/p/libfiu/ looks interesting... (the python
> bindings have been Debian packaged)
> it would probably need to be evaluated more thoroughly as part of
> the project though...
>
> Thanks,
> Cindy
> ________________________________________
> From: rtems-devel-bounces at rtems.org
> <mailto:rtems-devel-bounces at rtems.org>
> [rtems-devel-bounces at rtems.org
> <mailto:rtems-devel-bounces at rtems.org>] on behalf of Rempel,
> Cynthia [cynt6007 at vandals.uidaho.edu
> <mailto:cynt6007 at vandals.uidaho.edu>]
> Sent: Monday, July 29, 2013 9:06 AM
> To: Joel Sherrill; Gedare Bloom
> Cc: Krzysztof Mięsowicz; rtems-devel at rtems.org
> <mailto:rtems-devel at rtems.org>
> Subject: RE: ESA SOCIS - Fault injection tools topic
>
> Hi,
>
> Thanks for the link! This will be helpful with fleshing out the
> project...
> Yes, the initial idea was to get a tool the RTEMS development
> community could use to help with certification efforts...
> and having the certification report would be very helpful...
> Given this information, we can start to write an open-project page
> for this...
>
> The criteria we typically use for evaluating other projects are:
> 1. Free and open source (usually modified BSD or special exception
> GPLv2, but as this is testing GPL would be fine)
> 2. Works under Linux and Windows
> 3. Still under maintainence
>
> Thanks!
> Cindy
> ________________________________________
> From: rtems-devel-bounces at rtems.org
> <mailto:rtems-devel-bounces at rtems.org>
> [rtems-devel-bounces at rtems.org
> <mailto:rtems-devel-bounces at rtems.org>] on behalf of Joel Sherrill
> [joel.sherrill at OARcorp.com]
> Sent: Monday, July 29, 2013 8:23 AM
> To: Gedare Bloom
> Cc: Krzysztof Mięsowicz; rtems-devel at rtems.org
> <mailto:rtems-devel at rtems.org>
> Subject: Re: ESA SOCIS - Fault injection tools topic
>
> If I forgot the link:
>
> ftp://www.rtems.org/pub/rtems/esa_validation_report_450/
>
> On 7/29/2013 9:45 AM, Gedare Bloom wrote:
> > This seems like a good project. Ballista looks like it is "dead"
> > upstream. You'll want to scope your project to decide what / how
> much
> > of RTEMS you can reasonably instrument with fault injection. I also
> > wonder if it would be sensible to first start with a tool that can
> > support test input "fuzzing", which is related to but somewhat
> > different than fault injection. I think "fuzzing" is more
> > appropriately related to test coverage /testing, whereas fault
> > injection can include also non-valid inputs and faults, such as
> > flipping bits in code or modifying values on-the-fly.
> >
> > On Mon, Jul 29, 2013 at 4:25 AM, Krzysztof Mięsowicz
> > <krzysztof.miesowicz at gmail.com
> <mailto:krzysztof.miesowicz at gmail.com>> wrote:
> >> Hi!
> >>
> >> My name is Krzysiek. I participated in ESA SOCIS 2012 for RTEMS
> and worked
> >> on improving test coverage. I'm interested in participating in
> this edition
> >> of SOCIS again.
> >>
> >> I browse Open Projects page of RTEMS and I have question about
> topic related
> >> to fault injection tools. Have you any tips or suggestions to
> this topic
> >> (especially which tools are most desired or promising and how
> this project
> >> should look in details).
> >>
> >> I googled a bit about fault injection and free tools and find some:
> >> Ballista, LFI, Fuzz, or JACA which potentially could be applied
> to RTEMS
> >> testing. But I'm looking forward to any tips or thoughts from
> you and
> >> additionaly is it possible to be selected again this year.
> >>
> >> Thanks in advance for your replies.
> >> Krzysiek Mięsowicz
> >>
> >> _______________________________________________
> >> rtems-devel mailing list
> >> rtems-devel at rtems.org <mailto:rtems-devel at rtems.org>
> >> http://www.rtems.org/mailman/listinfo/rtems-devel
> >>
> > _______________________________________________
> > rtems-devel mailing list
> > rtems-devel at rtems.org <mailto:rtems-devel at rtems.org>
> > http://www.rtems.org/mailman/listinfo/rtems-devel
>
>
> --
> Joel Sherrill, Ph.D. Director of Research & Development
> joel.sherrill at OARcorp.com On-Line Applications Research
> Ask me about RTEMS: a free RTOS Huntsville AL 35805
> Support Available (256) 722-9985 <tel:%28256%29%20722-9985>
>
> _______________________________________________
> rtems-devel mailing list
> rtems-devel at rtems.org <mailto:rtems-devel at rtems.org>
> http://www.rtems.org/mailman/listinfo/rtems-devel
>
>
> _______________________________________________
> rtems-devel mailing list
> rtems-devel at rtems.org <mailto:rtems-devel at rtems.org>
> http://www.rtems.org/mailman/listinfo/rtems-devel
>
>
>
--
Joel Sherrill, Ph.D. Director of Research & Development
joel.sherrill at OARcorp.com On-Line Applications Research
Ask me about RTEMS: a free RTOS Huntsville AL 35805
Support Available (256) 722-9985
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rtems.org/pipermail/devel/attachments/20130801/7e66820d/attachment-0001.html>
More information about the devel
mailing list