rtems tester: Why does rtems_test_assert() just exit?

[dufault at hda.com]

> On Sep 22, 2019, at 18:45 , Chris Johns <chrisj at rtems.org> wrote:
> 
> On 23/9/19 12:33 am, dufault at hda.com wrote:
>>> On Sep 22, 2019, at 07:12 , <dufault at hda.com> <dufault at hda.com> wrote:
>>>> On Sep 21, 2019, at 17:29 , dufault at hda.com wrote:
>>>>> On Sep 21, 2019, at 17:04 , Chris Johns <chrisj at rtems.org> wrote:
>>> (…)
>>>>> The tester is designed to avoid cycling power. A clean working target should
>>>>> only need a single power on at the start and a single power off at the end.
>>>>> Repeated power cycling can stress power supplies and targets, some switch modes
>>>>> do not like it. Some targets have hardware reset issues and may need a power
>>>>> cycle and I have seen uboot get wedged. The power cycle logic is for those
>>>>> cases. I would not expect this being needed on each boot. Also it is a reset
>>>>> which can be a power cycle or a wired reset signal however some SOCs these days
>>>>> have separate power on reset (POR) logic and a reset signal logic.
>>>>> 
>>>>> Chris
>>>> 
>>>> You are bringing up most of my concerns in your response, I’ll go over your response properly tomorrow AM when I’ll have free time.  Thanks.
>>>> 
>>> I think if I change "rtems_test_assert()" to invoke "exit(RTEMS_FATAL_SOURCE_ASSERT)" instead of "exit(0)" and then use the following for the reset expression it will do what I would like.  I'm sure there is a better way to do that regular expression but I'll use this for a test.
>>> 
>>> target_reset_regex = .* RTEMS_FATAL_SOURCE_BDBUF, .*|.* RTEMS_FATAL_SOURCE_APPLICATION, .*|.* RTEMS_FATAL_SOURCE_BSP, .*|.* RTEMS_FATAL_SOURCE_ASSERT, .*|.* RTEMS_FATAL_SOURCE_STACK_CHECKER, .*|.* RTEMS_FATAL_SOURCE_EXCEPTION, .*|.* RTEMS_FATAL_SOURCE_SMP, .*|.* RTEMS_FATAL_SOURCE_PANIC, .*|.* RTEMS_FATAL_SOURCE_INVALID_HEAP_FREE, .*
> 
> The tester knows the test has finished with '*** END OF TEST ' so it will end
> the test when this message is seen. If your host is fast or the console is
> buffered the test will finish and be looking for the test begin message and if a
> reset trigger is seen in the output from the end of the previous test the board
> will reset as part of the start of the next test.
> 
> In theory a good quality target should not need any reset triggers however it
> will need a start trigger to catch a test failing without any output and restarting.
> 
>> This doesn't work as I'd hoped.  There is pending input to the tester from the tested device's serial port console.  If the tester matches the reset condition *after* the first reset but *before* the board restarts it will issue the reset again.
>> 
>> In the case of "unlimited.exe" the following is what happened.  I watched the tester power cycle the board off and on about 15 times before it advanced to the next test.
>> 
>> After issuing the reset command should the tester advance through the console output without taking action until it sees the start expression?
>> 
>> (...)
>> ]  TEST3 : signal task 0a01034e to delete, task 844 ending.
>> =>  target reset condition detected
>> =>  target reset: ./power-ctl 1 off-on
>> ]  TEST3 : signal task 0a01034f to delete, fatal source: RTEMS_FATAL_SOURCE_EXCEPTION, error code: 9018872
>> ] exception vector 7 (0x7)
>> ]   next PC or address of fault = 0x00890038
>> (...)
>> ]   executing thread ID = 0x0a01034f, name = ai45
>> ] Stack Trace: 
>> ]   IP: 0x00890038, LR: 0x0089003a
>> =>  target reset condition detected
>> =>  target reset: ./power-ctl 1 off-on
>> ] fatal source: RTEMS_FATAL_SOURCE_EXCEPTION, error code: 9018088
>> ] exception vector 3 (0x3)
> 
> Is this from the power being turned off? If it is this would cause a loop if in
> the reset trigger.
> 

No, whatever error happened in "unlmited.exe" generated a cascade of errors and so they are all in the console output for the tester to read.  The tester reads through them sequentially and keeps cycling power off and on.

I did put in the change for the tester to cycle to a start expression after it issues a reset without issuing multiple resets.  That almost did what I wanted in that it cycled to the start without any more resets.  The tester did not end the test, though, so it issued a second reset when the board requested a TFTP transfer request.

I'll think about this a little more, I don't think these are really RESET regular expressions but an "outside of the current tester's framework" failure regular expression.

These RTEMS_FATAL_FUBAR errors mean:
- The test has failed, and the test is going to reset the board momentarily.  There's no need for the tester to do a reset, the tester can fail this test and advance to the next test and download the next test at the next TFTP transfer request.
- The test has failed catastrophically, and we're now going to get a possibly infinite cascade of these errors.  Rather than wait for a three minute timeout of cascading errors it would be nice to count to a certain number, reset, fail the test and continue to the next one.

My goal is also to approach the ideal of an initial power-up / final power-down power sequence.  I have a few thoughts about the user input tests as well but I'll think more about it first.

Peter
-----------------
Peter Dufault
HD Associates, Inc.      Software and System Engineering

This email is delivered through the public internet using protocols subject to interception and tampering.