[PATCH rtems-tools] tester/report: Apply test excludes to fatal-error

[Chris Johns]

On 18/8/2023 11:51 pm, Kinsey Moore wrote:
> On Thu, Aug 17, 2023 at 7:25 PM Chris Johns <chrisj at rtems.org
> <mailto:chrisj at rtems.org>> wrote:
> 
>     On 17/8/2023 10:43 pm, Kinsey Moore wrote:
>     > On Wed, Aug 16, 2023 at 9:47 PM Chris Johns <chrisj at rtems.org
>     <mailto:chrisj at rtems.org>
>     > <mailto:chrisj at rtems.org <mailto:chrisj at rtems.org>>> wrote:
>     >
>     >     On 17/8/2023 7:36 am, Kinsey Moore wrote:
>     >     > Before the fatal-error test result type was introduced, minimum.exe was
>     >     > classified as an invalid test since it lacked a proper test header and
>     >     > trailer. This applies the test exclusions to all test states to avoid
>     >     > this happening again in the future.
>     >
>     >     I do not think this right. What state are you seeing?
>     >
>     >
>     > The minimum test fails with a fatal error:
>     >     "*** FATAL ***",
>     >     "fatal source: 0 (INTERNAL_ERROR_CORE)",
>     >     "fatal code: 5 (INTERNAL_ERROR_THREAD_EXITTED)",
>     >     "RTEMS version: 6.0.0.c76c98c344d803fa80361884c4cc79f0b3607ec8",
>     >     "RTEMS tools: 12.3.1 20230626 (RTEMS 6, RSB
>     > 8e568b2ca3489d6bfa48e1d29618ea9b48a5b408, Newlib 4c7d0df)",
>     >     "executing thread ID: 0x09010001",
>     >
>     > As far as I'm aware, this is a normal exit for minimum since it does nothing
>     > which includes not properly exiting the test by allowing Init() to return.
>     >
>     >
>     >     While we cannot determine the pass state because there is no end
>     message we can
>     >     determine if the test has timed out or is too long and they should be
>     a fail.
>     >
>     >
>     > This is not timing out or taking too long as per the above. I could move the
>     > check to cover just the invalid/fatal-error checks instead of all of them
>     if you
>     > prefer.
> 
>     This looks to me like an issue in minimum and it looks to me like minimal has
>     some other issue. What is printing out the message? If there is code in minimal
>     to print this error then why not print a normal test start and end banner?
> 
>     My understanding is minimal has not means to output anything.
> 
> 
> It seems to be intentional:
> static void *Init( uintptr_t ignored )
> {
>   /* initialize application */
> 
>   /* Real application would call idle loop functionality */
> 
>   /* but in this case, just return and fall into a fatal error */
> 
>   return NULL;
> }
> 
> Even if no console is requested (not even simple console), BSP_output_char is
> still available.

Is this happening for all BSPs and all archs?

When the original filter for minimal was made minimal never reported anything so
what has changed to make this happen?

I think what we have is right because it does not suppress real issues minimal
could have if there is a regression. I think there is more to this we need to
understand before we decide to change the tester filtering.

Chris