<html>
<head>
<meta content="text/html; charset=ISO-8859-2"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">On 7/29/2013 3:09 PM, Krzysztof
Mięsowicz wrote:<br>
</div>
<blockquote
cite="mid:CANNpagrYNqESnWV48fo_U=o+GZbqHBwCXp8QHv1UZkZ=4gh1rg@mail.gmail.com"
type="cite">
<div dir="ltr">Thank you for all replies very much.
<div><br>
</div>
<div>I read about libfiu and it seems really interesting and I
believe it can be quite easy applied to RTEMS testing. I hope
so :) </div>
<div><br>
</div>
</div>
</blockquote>
The process of using preloaded libraries to intercept calls and<br>
break them may not work on RTEMS at all. Plus the idea of<br>
running a process under another process is foreign since we<br>
only have one process. I don't have a good feeling on libfiu<br>
for RTEMS on an admittedly cursory examination. <br>
<br>
API parameter fuzzing was done in the report I posted. <br>
I did a quick google and found <a href="http://peachfuzzer.com/">http://peachfuzzer.com/</a><br>
which looks like a FLOSS software package to what was<br>
done in that report. The Classic APIs were described in <br>
terms of their data types. For each data type, "interesting" <br>
values were defined. Say NULL for argument pointers, <br>
or 0xfffffffc and 0xffffffff for the start of a region or<br>
partition. <br>
<br>
As I recall, the challenge is that the way RTEMS APIs<br>
work, you often need a valid object ID or N-1 parameters<br>
to be valid to get to the Nth parameter. So there was<br>
some custom test setup to write to go along with the <br>
automatically generated fuzzing code. Depending on the<br>
fuzzing package, this may be handled better.<br>
<br>
From my perspective, API parameter fuzzing was valuable<br>
at the time. It turned up some odd bugs that wouldn't <br>
likely have been caught otherwise. <br>
<br>
With all that said, I don't know how much you can<br>
accomplish in the time period. The goal would have<br>
to be to provide the foundation for a permanent <br>
addition to the test suite which can be added to.<br>
You can't possibly finish all APIs so we would have to<br>
define an interesting subset.<br>
<blockquote
cite="mid:CANNpagrYNqESnWV48fo_U=o+GZbqHBwCXp8QHv1UZkZ=4gh1rg@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>
According to Joel's mail:</div>
<div><span style="font-family:arial,sans-serif;font-size:13px"><br>
</span></div>
<div><i><span
style="font-family:arial,sans-serif;font-size:13px">Another
area which we need help in is the coverage and testing</span><br
style="font-family:arial,sans-serif;font-size:13px">
<span style="font-family:arial,sans-serif;font-size:13px">scripts.
The coverage reporting really should be by directory or</span><br
style="font-family:arial,sans-serif;font-size:13px">
<span style="font-family:arial,sans-serif;font-size:13px">capability.
It is now reported in "lumps" that are too large.</span><br
style="font-family:arial,sans-serif;font-size:13px">
<span style="font-family:arial,sans-serif;font-size:13px">Also
rewriting as much of this as possible in Python including
the</span><br
style="font-family:arial,sans-serif;font-size:13px">
<span style="font-family:arial,sans-serif;font-size:13px">sim-scripts
is highly desirable.</span></i><br>
</div>
<div><br>
</div>
<div>Do you mean that coverage reporting framework should be
rewrited and reports should be generated for source
directories instead of core and developmental parts (as it is
done now) ?</div>
</div>
</blockquote>
Yes. Having core is OK but developmental really sucks. When I<br>
added filesystems, the overall percentage dropped significantly.<br>
What we want to see from an analysis and marketing perspective<br>
is that directory X is 99.9% but this new directory Y is at 50%<br>
and needs testing attention.<br>
<br>
As we move more into continuous integration testing, this<br>
fine grained reporting will be even more valuable. <br>
<br>
The scripts that drive this process are in shell. We are moving to
using<br>
Python as our host tools scripting language. So these should be <br>
rewritten. <br>
<br>
This project is more defined and has a short term completion.<br>
<blockquote
cite="mid:CANNpagrYNqESnWV48fo_U=o+GZbqHBwCXp8QHv1UZkZ=4gh1rg@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>And what particularly do you mean by "rewriting as much of
THIS as possible" - do you mean all coverage reporting scripts
which is now written as bash scripts? Do I understand you
correctly? </div>
<div><br>
</div>
</div>
</blockquote>
Yes. There is "covoar" which is C++ but the contents of
rtems-coverage are pretty<br>
much all shell scripts with some template files.<br>
<br>
The rtems-testing git module has sim-scripts to run simulators and
we want<br>
to move them to Python as well. In the process we want to both them
to<br>
a single command rather than one per BSP. Invoked omething like <br>
<br>
rtems-sim [args] BSP <br>
<br>
And rtems-testing has other scripts.<br>
<blockquote
cite="mid:CANNpagrYNqESnWV48fo_U=o+GZbqHBwCXp8QHv1UZkZ=4gh1rg@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>Greetings,</div>
<div>Krzysiek</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">2013/7/29 Rempel, Cynthia <span
dir="ltr"><<a moz-do-not-send="true"
href="mailto:cynt6007@vandals.uidaho.edu" target="_blank">cynt6007@vandals.uidaho.edu</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
Did a quick write-up on the fault-injection open-project...<br>
<a moz-do-not-send="true"
href="http://blitiri.com.ar/p/libfiu/" target="_blank">http://blitiri.com.ar/p/libfiu/</a>
looks interesting... (the python bindings have been Debian
packaged)<br>
it would probably need to be evaluated more thoroughly as
part of the project though...<br>
<br>
Thanks,<br>
Cindy<br>
________________________________________<br>
From: <a moz-do-not-send="true"
href="mailto:rtems-devel-bounces@rtems.org">rtems-devel-bounces@rtems.org</a>
[<a moz-do-not-send="true"
href="mailto:rtems-devel-bounces@rtems.org">rtems-devel-bounces@rtems.org</a>]
on behalf of Rempel, Cynthia [<a moz-do-not-send="true"
href="mailto:cynt6007@vandals.uidaho.edu">cynt6007@vandals.uidaho.edu</a>]<br>
Sent: Monday, July 29, 2013 9:06 AM<br>
To: Joel Sherrill; Gedare Bloom<br>
Cc: Krzysztof Mięsowicz; <a moz-do-not-send="true"
href="mailto:rtems-devel@rtems.org">rtems-devel@rtems.org</a><br>
Subject: RE: ESA SOCIS - Fault injection tools topic<br>
<div class="HOEnZb">
<div class="h5"><br>
Hi,<br>
<br>
Thanks for the link! This will be helpful with fleshing
out the project...<br>
Yes, the initial idea was to get a tool the RTEMS
development community could use to help with
certification efforts...<br>
and having the certification report would be very
helpful...<br>
Given this information, we can start to write an
open-project page for this...<br>
<br>
The criteria we typically use for evaluating other
projects are:<br>
1. Free and open source (usually modified BSD or special
exception GPLv2, but as this is testing GPL would be
fine)<br>
2. Works under Linux and Windows<br>
3. Still under maintainence<br>
<br>
Thanks!<br>
Cindy<br>
________________________________________<br>
From: <a moz-do-not-send="true"
href="mailto:rtems-devel-bounces@rtems.org">rtems-devel-bounces@rtems.org</a>
[<a moz-do-not-send="true"
href="mailto:rtems-devel-bounces@rtems.org">rtems-devel-bounces@rtems.org</a>]
on behalf of Joel Sherrill [<a class="moz-txt-link-abbreviated" href="mailto:joel.sherrill@OARcorp.com">joel.sherrill@OARcorp.com</a>]<br>
Sent: Monday, July 29, 2013 8:23 AM<br>
To: Gedare Bloom<br>
Cc: Krzysztof Mięsowicz; <a moz-do-not-send="true"
href="mailto:rtems-devel@rtems.org">rtems-devel@rtems.org</a><br>
Subject: Re: ESA SOCIS - Fault injection tools topic<br>
<br>
If I forgot the link:<br>
<br>
<a moz-do-not-send="true"
href="ftp://www.rtems.org/pub/rtems/esa_validation_report_450/"
target="_blank">ftp://www.rtems.org/pub/rtems/esa_validation_report_450/</a><br>
<br>
On 7/29/2013 9:45 AM, Gedare Bloom wrote:<br>
> This seems like a good project. Ballista looks like
it is "dead"<br>
> upstream. You'll want to scope your project to
decide what / how much<br>
> of RTEMS you can reasonably instrument with fault
injection. I also<br>
> wonder if it would be sensible to first start with
a tool that can<br>
> support test input "fuzzing", which is related to
but somewhat<br>
> different than fault injection. I think "fuzzing"
is more<br>
> appropriately related to test coverage /testing,
whereas fault<br>
> injection can include also non-valid inputs and
faults, such as<br>
> flipping bits in code or modifying values
on-the-fly.<br>
><br>
> On Mon, Jul 29, 2013 at 4:25 AM, Krzysztof
Mięsowicz<br>
> <<a moz-do-not-send="true"
href="mailto:krzysztof.miesowicz@gmail.com">krzysztof.miesowicz@gmail.com</a>>
wrote:<br>
>> Hi!<br>
>><br>
>> My name is Krzysiek. I participated in ESA
SOCIS 2012 for RTEMS and worked<br>
>> on improving test coverage. I'm interested in
participating in this edition<br>
>> of SOCIS again.<br>
>><br>
>> I browse Open Projects page of RTEMS and I have
question about topic related<br>
>> to fault injection tools. Have you any tips or
suggestions to this topic<br>
>> (especially which tools are most desired or
promising and how this project<br>
>> should look in details).<br>
>><br>
>> I googled a bit about fault injection and free
tools and find some:<br>
>> Ballista, LFI, Fuzz, or JACA which potentially
could be applied to RTEMS<br>
>> testing. But I'm looking forward to any tips or
thoughts from you and<br>
>> additionaly is it possible to be selected again
this year.<br>
>><br>
>> Thanks in advance for your replies.<br>
>> Krzysiek Mięsowicz<br>
>><br>
>> _______________________________________________<br>
>> rtems-devel mailing list<br>
>> <a moz-do-not-send="true"
href="mailto:rtems-devel@rtems.org">rtems-devel@rtems.org</a><br>
>> <a moz-do-not-send="true"
href="http://www.rtems.org/mailman/listinfo/rtems-devel"
target="_blank">http://www.rtems.org/mailman/listinfo/rtems-devel</a><br>
>><br>
> _______________________________________________<br>
> rtems-devel mailing list<br>
> <a moz-do-not-send="true"
href="mailto:rtems-devel@rtems.org">rtems-devel@rtems.org</a><br>
> <a moz-do-not-send="true"
href="http://www.rtems.org/mailman/listinfo/rtems-devel"
target="_blank">http://www.rtems.org/mailman/listinfo/rtems-devel</a><br>
<br>
<br>
--<br>
Joel Sherrill, Ph.D. Director of Research
& Development<br>
<a class="moz-txt-link-abbreviated" href="mailto:joel.sherrill@OARcorp.com">joel.sherrill@OARcorp.com</a> On-Line Applications
Research<br>
Ask me about RTEMS: a free RTOS Huntsville AL 35805<br>
Support Available <a
moz-do-not-send="true" href="tel:%28256%29%20722-9985"
value="+12567229985">(256) 722-9985</a><br>
<br>
_______________________________________________<br>
rtems-devel mailing list<br>
<a moz-do-not-send="true"
href="mailto:rtems-devel@rtems.org">rtems-devel@rtems.org</a><br>
<a moz-do-not-send="true"
href="http://www.rtems.org/mailman/listinfo/rtems-devel"
target="_blank">http://www.rtems.org/mailman/listinfo/rtems-devel</a><br>
<br>
<br>
_______________________________________________<br>
rtems-devel mailing list<br>
<a moz-do-not-send="true"
href="mailto:rtems-devel@rtems.org">rtems-devel@rtems.org</a><br>
<a moz-do-not-send="true"
href="http://www.rtems.org/mailman/listinfo/rtems-devel"
target="_blank">http://www.rtems.org/mailman/listinfo/rtems-devel</a><br>
<br>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
Joel Sherrill, Ph.D. Director of Research & Development
<a class="moz-txt-link-abbreviated" href="mailto:joel.sherrill@OARcorp.com">joel.sherrill@OARcorp.com</a> On-Line Applications Research
Ask me about RTEMS: a free RTOS Huntsville AL 35805
Support Available (256) 722-9985 </pre>
</body>
</html>