[PATCH 4/6] testsuite: Add expected-fail to psim

[Chris Johns]

On 6/5/20 7:35 pm, Sebastian Huber wrote:
> On 06/05/2020 10:41, chrisj at rtems.org wrote:
> 
>> From: Chris Johns<chrisj at rtems.org>
>>
>> Updates #2962
>> ---
>>   bsps/powerpc/psim/config/psim-testsuite.tcfg | 22 ++++++++++++++++++++
>>   1 file changed, 22 insertions(+)
>>   create mode 100644 bsps/powerpc/psim/config/psim-testsuite.tcfg
>>
>> diff --git a/bsps/powerpc/psim/config/psim-testsuite.tcfg 
>> b/bsps/powerpc/psim/config/psim-testsuite.tcfg
>> new file mode 100644
>> index 0000000000..b0d2a05086
>> --- /dev/null
>> +++ b/bsps/powerpc/psim/config/psim-testsuite.tcfg
>> @@ -0,0 +1,22 @@
>> +#
>> +# PSIM RTEMS Test Database.
>> +#
>> +# Format is one line per test that is_NOT_  built.
>> +#
>> +
>> +expected-fail: fsimfsgeneric01
>> +expected-fail: block11
>> +expected-fail: rbheap01
>> +expected-fail: termios01
>> +expected-fail: ttest01
>> +expected-fail: psx12
>> +expected-fail: psxchroot01
>> +expected-fail: psxfenv01
>> +expected-fail: psximfs02
>> +expected-fail: psxpipe01
>> +expected-fail: spextensions01
>> +expected-fail: spfatal31
>> +expected-fail: spfifo02
>> +expected-fail: spmountmgr01
>> +expected-fail: spprivenv01
>> +expected-fail: spstdthreads01
> 
> I don't think these tests are expected to fail. If they fail, then there 
> is a bug somewhere.

Yes we hope no tests fail but they can and do. Excluding tests because 
they fail would be incorrect. In the 5.1 release these bugs are present 
so we expect, or maybe it should say, we know the test will fail. With 
this change any thing that appears in the failure column is "unexpected" 
and that means the user build of the release does not match the state we 
"expect" and it is worth investigation by the user.

Without these tests being tagged this way the user would have no idea 
where the stand after a build and test run and that would mean we would 
have to make sure a release has no failures. I consider that as not 
practical or realistic.

Chris