[PATCH 4/6] testsuite: Add expected-fail to psim

Tue May 12 13:22:43 UTC 2020

On Tue, May 12, 2020 at 4:11 AM Chris Johns <chrisj at rtems.org> wrote:

> On 12/5/20 5:15 pm, Sebastian Huber wrote:
> > Hello,
> >
> > On 09/05/2020 03:30, Gedare Bloom wrote:
> >>>>> Without these tests being tagged this way the user would have no
> >>>>> idea where the stand after a build and test run and that would mean
> >>>>> we would have to make sure a release has no failures. I consider
> >>>>> that as not practical or realistic.
> >>>> Maybe we need another state, e.g. something-is-broken-please-fix-it.
> >>> I do not think so, it is implicit in the failure or the test is
> >>> broken. The only change is to add unexpected-pass, that will be on
> >>> master after the 5 branch.
> >>>
> >> I disagree with this in principle, and it should be reverted after we
> >> branch 5. It's fine for now to get the release state sync'd, but we
> >> should find a long-term solution that distinguishes the cases:
> >> 1. we don't expect this test to pass on this bsp
> >> 2. we expect this test to pass, but know it doesn't currently
> >>
> >> They are two very different things, and I don't like conflating them
> >> into one "expected-fail" case
> > originally, I had the same point of view. What I didn't take into
> > account was the perspective of the tester. Now, I think it is perfectly
> > fine to flag these tests as expected failure test states. Because right
> > now, due to some known bugs such as https://devel.rtems.org/ticket/3982
> > and probably also some more issues, these tests fail. On this BSP and
> > this RTEMS version, they will always fail. This is not some sort of
> > random failure. When we change test states to expected failure I think
> > we should make sure that a ticket exists, which captures that there are
> > some test results which indicate issues (expected failure test state).
> > The ticket system is the better place to manage this. We should not use
> > the test states for this. The test states should be used to figure out
> > changes between different test runs. They should enable also to quickly
> > check if the outcome of a test run yields the expected results for a
> > certain RTEMS version and BSP.
>
> Thanks. It is clear to me we lack documentation on this topic and this
> is an oversight on my part which I will attempt to correct.
>
> I have reviewed Dejagnu and considered other things like the withdrawn
> IEEE 1003.3 standard and there are states we have that need to change
> but I think the original intent is the right path.
>
> The Dejagnu states are documented here:
>
>
> https://www.gnu.org/software/dejagnu/manual/A-POSIX-Conforming-Test-Framework.html#A-POSIX-Conforming-Test-Framework
>
> And the exit codes are:
>
> https://www.gnu.org/software/dejagnu/manual/Runtest.html#Runtest
>
> For me they define the goal and intent.
>
> The test states are metadata for the tester so it can determine the
> result of any given set of tests in relation to the expected state of
> the test when it was built. You need to detach yourself from being a
> developer and put yourself in the position of a tester who's task is to
> give an overall pass or fail for a specific build of RTEMS without
> needing to consider the specifics of any test, bug or feature.
>
> The primary requirement is to allow machine check of the results to
> determine regressions. A regression is a failure, pass or unresolved
> result that was not expected.
>
> My current thinking for test states are:
>
> PASS:
> The test has succeeded and passed without a failure.
>
> UNEXCEPTED-PASS:
> The test has succeeded when it was expected to fail.
>
> FAIL:
> The test has not succeeded and has failed when it was expected to pass.
> The failure can be a failed assert, unhandled exception, resource
> constraint, or a faulty test.
>
> EXCEPTED-FAIL:
> The test has not succeeded and has failed and this is expected.
>
> UNRESOLVED:
> The test has not completed and the result cannot be determined. The
> result can be unresolved because the test did not start or end, test
> harness failure, insufficient computing resources for the test harness
> to function correctly.
>
> EXCEPTED-UNRESOLVED:
> The test has not completed and the result cannot be determined and this
> is expected.
>
> INDETERMINATE:
> The test has succeeded, has failed or in unresolved. The test is an edge
> case where the test can pass, can fail, can be unresolved and this is
> expected.
>
> USER-INPUT:
> The test has not completed and the result is unresolved because it
> requires user intervention that cannot be provided.
>
> BENCHMARK:
> The test performs a performance type test. These are currently not
> supported.
>
> UNTESTED:
> The test has not run and is a place holder for a real test that is not
> yet provided.
>
> UNSUPPORTED:
> The test is not supported for this build of RTEMS, BSP or architecture.
>
> Note:
>
> 1. Any expected failures, unresolved, or indeterminate test results are
> considered faults and require fixing.
>
> 2. The nature of a failure cannot be inferred from the test's metadata
> state.
>
> 3. The timeout and invalid states will be merged into UNRESOLVED.
>
> 4. The excluded state will be changed to UNSUPPORTED.
>
> 5. The metadata is placed in each test because is it an effective way to
> capture the state. Tests can be run as a group, stand alone or at
> different location and the test results can determine a regression. The
> version of the test harness does not need to match the RTEMS build.
>

Not to be dense but what state do tests which fail but have not been
investigated
yet go? GCC just leaves those as FAIL and releases happen with those on
secondary targets.  I know FAILs are undesirable for primary targets with
GCC.

I don't want to see a test that fails but we don't know why binned
somewhere
it will never get investigated.

>
>
>
> This list of test states account for some missing states. It also adds
> some states I do not see being available until we move to a new build
> system. For UNTESTED and UNSUPPORTED I see a template test being built
> and run and does nothing. This is important because it means we get a
> complete set of test results that are complete and consistent for all BSPs.
>
> I can attend to this change before releasing 5.1 or it can be done on
> master and we can determine if it is back ported to 5.2[34..].
>

I have previously stated that this is a good goal but it is moving the goal
line
for the 5.x releases. I would propose we be happy with the fact we have
reported test results at all for the first time a release happens.  Let's
not
let the perfect be the enemy of the good. In this case, the good is quite
a bit better than previous releases. We need to be more conscious of this

I'm also concerned this task is bigger than you think based solely on the
number
of BSPs we have and the number we can execute tests on simulators. My
build sweep has at least 21 BSPs (hand count) it is testing on simulator
and
I didn't count the handful of qemu based ones.

To get an accurate assessment, I think you would have to temporarily
let all tests build for a BSP so you would know which are disabled because
they don't fit. Then the rest of the tests in the .tcfg which are not ld
overflow
issues would have to be examined and categorized.  I don't think the
current list of "don't build" tests fits nicely into one of the new
categories.

>
> The change will come with documentation to explain thing a little better.
>

+1

>
> I hope this addresses the issues we have and I am sorry for creating a
> disturbance so close to a release.
>

It's a good goal but I think the timing is wrong.

--joel

>
> Chris
> _______________________________________________
> devel mailing list
> devel at rtems.org
> http://lists.rtems.org/mailman/listinfo/devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rtems.org/pipermail/devel/attachments/20200512/f4ba045a/attachment.html>