[PATCH 4/6] testsuite: Add expected-fail to psim

[Sebastian Huber]

On 06/05/2020 12:00, Chris Johns wrote:

> On 6/5/20 7:35 pm, Sebastian Huber wrote:
>> On 06/05/2020 10:41, chrisj at rtems.org wrote:
>>
>>> From: Chris Johns<chrisj at rtems.org>
>>>
>>> Updates #2962
>>> ---
>>>   bsps/powerpc/psim/config/psim-testsuite.tcfg | 22 
>>> ++++++++++++++++++++
>>>   1 file changed, 22 insertions(+)
>>>   create mode 100644 bsps/powerpc/psim/config/psim-testsuite.tcfg
>>>
>>> diff --git a/bsps/powerpc/psim/config/psim-testsuite.tcfg 
>>> b/bsps/powerpc/psim/config/psim-testsuite.tcfg
>>> new file mode 100644
>>> index 0000000000..b0d2a05086
>>> --- /dev/null
>>> +++ b/bsps/powerpc/psim/config/psim-testsuite.tcfg
>>> @@ -0,0 +1,22 @@
>>> +#
>>> +# PSIM RTEMS Test Database.
>>> +#
>>> +# Format is one line per test that is_NOT_  built.
>>> +#
>>> +
>>> +expected-fail: fsimfsgeneric01
>>> +expected-fail: block11
>>> +expected-fail: rbheap01
>>> +expected-fail: termios01
>>> +expected-fail: ttest01
>>> +expected-fail: psx12
>>> +expected-fail: psxchroot01
>>> +expected-fail: psxfenv01
>>> +expected-fail: psximfs02
>>> +expected-fail: psxpipe01
>>> +expected-fail: spextensions01
>>> +expected-fail: spfatal31
>>> +expected-fail: spfifo02
>>> +expected-fail: spmountmgr01
>>> +expected-fail: spprivenv01
>>> +expected-fail: spstdthreads01
>>
>> I don't think these tests are expected to fail. If they fail, then 
>> there is a bug somewhere.
>
> Yes we hope no tests fail but they can and do. Excluding tests because 
> they fail would be incorrect. In the 5.1 release these bugs are 
> present so we expect, or maybe it should say, we know the test will 
> fail. With this change any thing that appears in the failure column is 
> "unexpected" and that means the user build of the release does not 
> match the state we "expect" and it is worth investigation by the user.
>
> Without these tests being tagged this way the user would have no idea 
> where the stand after a build and test run and that would mean we 
> would have to make sure a release has no failures. I consider that as 
> not practical or realistic. 
Maybe we need another state, e.g. something-is-broken-please-fix-it.