or1k test was .. Re: [PATCH] or1k: New cache manager.

Hesham Moustafa heshamelmatary at gmail.com
Tue Sep 16 20:16:17 UTC 2014


Breakpoint 1, _RBTree_Next (node=0x40890, dir=RBT_RIGHT)
    at ../../../../../../rtems/c/src/../../cpukit/score/src/rbtreenext.c:35
35  RBTree_Direction opp_dir = _RBTree_Opposite_direction( dir );
(gdb) n
36  RBTree_Node     *current = node->child[ dir ];
(gdb) n
37  RBTree_Node     *next = NULL;
(gdb) p/x current
$1 = 0xa010002
(gdb) p/x node->child
$2 = {0x3d254, 0xa010002}
(gdb) p dir
$3 = RBT_RIGHT
(gdb) p/x &node->child
$4 = 0x40894
(gdb) i r
r0             0x0 0
r1             0x439a8 0x439a8
r2             0x439c8 0x439c8

r1 = sp
r2 = fp


(gdb) b _CPU_Context_Initialize
Breakpoint 1 at 0x2f648: file
../../../../../../../../rtems/c/src/../../cpukit/score/cpu/or1k/or1k-context-initialize.c,
line 33.
(gdb) c
Continuing.

Breakpoint 1, _CPU_Context_Initialize (context=0x3fe28,
stack_area_begin=0x41c10, stack_area_size=4096, new_level=0,
entry_point=0x34078 <_Thread_Handler>,
    is_fp=false, tls_area=0x0) at
../../../../../../../../rtems/c/src/../../cpukit/score/cpu/or1k/or1k-context-initialize.c:33
33  uint32_t stack = ((uint32_t) stack_area_begin) - 200;
(gdb) c
Continuing.

Breakpoint 1, _CPU_Context_Initialize (context=0x403e0,
stack_area_begin=0x42c18, stack_area_size=4096, new_level=0,
entry_point=0x34078 <_Thread_Handler>,
    is_fp=false, tls_area=0x0) at
../../../../../../../../rtems/c/src/../../cpukit/score/cpu/or1k/or1k-context-initialize.c:33
33  uint32_t stack = ((uint32_t) stack_area_begin) - 200;
(gdb) c
Continuing.

Breakpoint 1, _CPU_Context_Initialize (context=0x40970,
stack_area_begin=0x43c20, stack_area_size=8192, new_level=0,
entry_point=0x34078 <_Thread_Handler>,
    is_fp=false, tls_area=0x0) at
../../../../../../../../rtems/c/src/../../cpukit/score/cpu/or1k/or1k-context-initialize.c:33
33  uint32_t stack = ((uint32_t) stack_area_begin) - 200;


(gdb) p/x &bsp_section_work_end
$5 = 0x2000000
(gdb) p/x &bsp_section_work_size
$6 = 0x1fc13c0
(gdb) p/x &bsp_section_work_begin
$7 = 0x3ec40

The corresponding linkcmds.base part is:

.work : ALIGN_WITH_INPUT {
    /*
     * The work section will occupy the remaining REGION_WORK region and
     * contains the RTEMS work space and heap.
     */
    bsp_section_work_begin = .;
    . += ORIGIN (REGION_WORK) + LENGTH (REGION_WORK) - ABSOLUTE (.);
    bsp_section_work_end = .;
  } > REGION_WORK AT > REGION_WORK
  bsp_section_work_size = bsp_section_work_end - bsp_section_work_begin;

  .stack : ALIGN_WITH_INPUT {
    bsp_section_stack_end = .;
  } > REGION_STACK AT > REGION_STACK
  bsp_section_stack_size = bsp_section_stack_begin - bsp_section_stack_end;

  RamBase = ORIGIN (REGION_WORK);
  RamSize = LENGTH (REGION_WORK);
  WorkAreaBase = bsp_section_work_begin;
  HeapSize = 0;
}

On Tue, Sep 16, 2014 at 9:45 PM, Hesham Moustafa
<heshamelmatary at gmail.com> wrote:
> Breakpoint 2, 0x00000600 in _unalign ()
> (gdb) bt
> #0  0x00000600 in _unalign ()
> #1  0x0002ec4c in _RBTree_Next (
>     node=0x40890, dir=RBT_RIGHT)
>     at ../../../../../../rtems/c/src/../../cpukit/score/src/rbtreenext.c:35
> #2  0x0002e2f4 in _RBTree_Successor (
>     node=0x40890)
>     at ../../cpukit/../../../or1ksim/lib/include/rtems/score/rbtree.h:512
> #3  0x0002e8c0 in _RBTree_Extract (
>     the_rbtree=0x4198c,
>     the_node=0x40890)
>     at ../../../../../../rtems/c/src/../../cpukit/score/src/rbtreeextract.c:106
> #4  0x00021524 in _RBTree_Get (
>     the_rbtree=0x4198c, dir=RBT_LEFT)
>     at ../../cpukit/../../../or1ksim/lib/include/rtems/score/rbtree.h:540
> #5  0x000215c8 in _Thread_queue_Dequeue
>     (the_thread_queue=0x4198c)
> ---Type <return> to continue, or q <return> to quit---
>     at ../../../../../../rtems/c/src/../../cpukit/score/src/threadqdequeue.c:51
> #6  0x00017c14 in _CORE_semaphore_Surrender (the_semaphore=0x4198c,
>     id=436273153,
>     api_semaphore_mp_support=0x0)
>     at ../../../../../../rtems/c/src/../../cpukit/score/src/coresemsurrender.c:37
> #7  0x00014868 in rtems_semaphore_release (id=436273153)
>     at ../../../../../../rtems/c/src/../../cpukit/rtems/src/semrelease.c:102
> #8  0x00026cfc in rtems_libio_unlock ()
>     at ../../cpukit/../../../or1ksim/lib/include/rtems/libio_.h:253
> #9  0x00026d5c in rtems_filesystem_default_unlock (mt_entry=0x49ce0)
>     at ../../../../../../rtems/c/src/../../cpukit/libfs/src/defaults/default_loc---Type
> <return> to continue, or q <return> to quit---
> k_and_unlock.c:39
> #10 0x0002920c in rtems_filesystem_instance_unlock (loc=0x49c5c)
>     at ../../cpukit/../../../or1ksim/lib/include/rtems/libio_.h:292
> #11 0x00029268 in rtems_filesystem_location_free (loc=0x49c5c)
>     at ../../../../../../rtems/c/src/../../cpukit/libcsupport/src/freenode.c:29
> #12 0x00029734 in rtems_libio_free (
>     iop=0x49c50)
>     at ../../../../../../rtems/c/src/../../cpukit/libcsupport/src/libio.c:136
> #13 0x0002912c in close (fd=0)
>     at ../../../../../../rtems/c/src/../../cpukit/libcsupport/src/close.c:38
> #14 0x000064b0 in rtems_libio_exit ()
>     at ../../../../../../rtems/c/src/../../cpukit/libcsupport/src/libio_exit.c:31
> ---Type <return> to continue, or q <return> to quit---
> #15 0x0003b058 in _exit (status=0)
>     at ../../../../../../rtems/c/src/../../cpukit/libcsupport/src/newlibc_exit.c:46
> #16 0x00034798 in exit (code=0)
>     at ../../../../../gcc-4.8.3/newlib/libc/stdlib/exit.c:70
> #17 0x00002e3c in Test_task (unused=1)
>     at ../../../../../../../rtems/c/src/../../testsuites/samples/ticker/tasks.c:41
> #18 0x000340f0 in _Thread_Handler ()
>     at ../../../../../../rtems/c/src/../../cpukit/score/src/threadhandler.c:192
> #19 0x00034078 in _User_extensions_Thread_exitted (executing=0x40890)
>     at ../../cpukit/../../../or1ksim/lib/include/rtems/score/userextimpl.h:243
> Backtrace stopped: frame did not save the PC
> (gdb)
>
>
> It breaks at _RBTree_Next specifically at the following line:
>  while ( ( current = current->child[ opp_dir ] ) != NULL )
>
> (gdb) p current->child[ opp_dir ]
> Cannot access memory at address 0xa010006
> (gdb) p current
> $1 = (RBTree_Node *) 0xa010002
>
> This address is invalid, the current memory length should be only 32
> MB (0x2000000)
>
> http://git.rtems.org/rtems/tree/c/src/lib/libbsp/or1k/or1ksim/startup/linkcmds#n20
>
> So I guest current->child is overwritten somehow?
>
> On Tue, Sep 16, 2014 at 9:21 PM, Joel Sherrill
> <joel.sherrill at oarcorp.com> wrote:
>> On 9/16/2014 2:17 PM, Hesham Moustafa wrote:
>>>
>>> On Tue, Sep 16, 2014 at 8:42 PM, Joel Sherrill
>>> <joel.sherrill at oarcorp.com> wrote:
>>>>
>>>>
>>>> On 9/16/2014 1:34 PM, Hesham Moustafa wrote:
>>>>>
>>>>> On Tue, Sep 16, 2014 at 8:15 PM, Joel Sherrill
>>>>> <joel.sherrill at oarcorp.com> wrote:
>>>>>>
>>>>>> On 9/16/2014 12:54 PM, Hesham Moustafa wrote:
>>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>> On Tue, Sep 16, 2014 at 7:47 PM, Joel Sherrill
>>>>>>> <joel.sherrill at oarcorp.com> wrote:
>>>>>>>>
>>>>>>>> I don't understand this but I got it applied.
>>>>>>>>
>>>>>>>> I manually edited the saved email to delete the preinstall.am
>>>>>>>> changes.  I committed the rest. Then I ran bootstrap -p myself
>>>>>>>> and folded that into the rest of your patch.
>>>>>>>>
>>>>>>>> It should all be committed now.
>>>>>>>>
>>>>>>> Thanks for doing this, me too do not know what's wrong. BTW, commits
>>>>>>> are not mirrored on github since 4 days ago.
>>>>>>>
>>>>>>>> How about some new test results. :)
>>>>>>>>
>>>>>>> I did run one last night, no big progress since previous results :( Is
>>>>>>> there any tool, script, utility program or whatever that I can use to
>>>>>>> detect wrong memory access (i.e, stack overwrite, heap corruption,
>>>>>>> access to another task context)? I tried to add -fstack-protector-all
>>>>>>> to gcc, but QEMU did not get anything or core-dump, ticker just hangs.
>>>>>>
>>>>>> I haven't checked into how gcc does its stack overwrite protection.
>>>>>>
>>>>>> The tests by themselves don't have these problems. The first
>>>>>> possible source is incorrect layout of sections to memory by
>>>>>> the linker script. There is some debug code in boot
>>>>>>
>>>>>> There used to be debug printk's in bspgetworkarea.c so you
>>>>>> could check if areas overlapped. That usually causes bad issues
>>>>>> though. Let's go through some basics:
>>>>>>
>>>>>> + Does hello world run and exit cleanly?
>>>>>>
>>>>> The output of Hello World is:
>>>>>
>>>>> *** BEGIN OF TEST HELLO WORLD ***
>>>>> Hello World
>>>>> *** END OF TEST HELLO WORLD ***
>>>>> Fatal Error 5.0 Halted
>>>>>
>>>>>   From GDB:
>>>>>
>>>>> Breakpoint 1, _Terminate (
>>>>>       the_source=RTEMS_FATAL_SOURCE_EXIT, is_internal=false,
>>>>> the_error=0)
>>>>>       at
>>>>> ../../../../../../rtems/c/src/../../cpukit/score/src/interr.c:39
>>>>> 39  _ISR_Disable_without_giant( level );
>>>>> (gdb) bt
>>>>> #0  _Terminate (
>>>>>       the_source=RTEMS_FATAL_SOURCE_EXIT, is_internal=false,
>>>>> the_error=0)
>>>>>       at
>>>>> ../../../../../../rtems/c/src/../../cpukit/score/src/interr.c:39
>>>>> #1  0x0003b5f8 in rtems_shutdown_executive (result=0)
>>>>>       at
>>>>> ../../../../../../rtems/c/src/../../cpukit/sapi/src/exshutdown.c:21
>>>>> #2  0x0003b350 in _exit (status=0)
>>>>>       at
>>>>>
>>>>> ../../../../../../rtems/c/src/../../cpukit/libcsupport/src/newlibc_exit.c:47
>>>>> #3  0x0002cc30 in exit (code=0)
>>>>>       at ../../../../../gcc-4.8.3/newlib/libc/stdlib/exit.c:70
>>>>> #4  0x00002184 in Init (ignored=253816)
>>>>>       at
>>>>>
>>>>> ../../../../../../../rtems/c/src/../../testsuites/samples/hello/init.c:33
>>>>> ---Type <return> to continue, or q <return> to quit---
>>>>> #5  0x0002c5b8 in _Thread_Handler ()
>>>>>       at
>>>>> ../../../../../../rtems/c/src/../../cpukit/score/src/threadhandler.c:192
>>>>> #6  0x0002c540 in _User_extensions_Thread_exitted (executing=0x40080)
>>>>>       at
>>>>> ../../cpukit/../../../or1ksim/lib/include/rtems/score/userextimpl.h:243
>>>>
>>>> This is normal and OK. Look at the arguments to _Terminate.
>>>>>>
>>>>>> + How far does ticker get?
>>>>>>
>>>>> Ticker goes to the end:
>>>>>
>>>>> *** BEGIN OF TEST CLOCK TICK ***
>>>>> TA1  - rtems_clock_get_tod - 09:00:00   12/31/1988
>>>>> TA2  - rtems_clock_get_tod - 09:00:00   12/31/1988
>>>>> TA3  - rtems_clock_get_tod - 09:00:00   12/31/1988
>>>>> TA1  - rtems_clock_get_tod - 09:00:05   12/31/1988
>>>>> TA2  - rtems_clock_get_tod - 09:00:10   12/31/1988
>>>>> TA1  - rtems_clock_get_tod - 09:00:10   12/31/1988
>>>>> TA3  - rtems_clock_get_tod - 09:00:15   12/31/1988
>>>>> TA1  - rtems_clock_get_tod - 09:00:15   12/31/1988
>>>>> TA2  - rtems_clock_get_tod - 09:00:20   12/31/1988
>>>>> TA1  - rtems_clock_get_tod - 09:00:20   12/31/1988
>>>>> TA1  - rtems_clock_get_tod - 09:00:25   12/31/1988
>>>>> TA3  - rtems_clock_get_tod - 09:00:30   12/31/1988
>>>>> TA2  - rtems_clock_get_tod - 09:00:30   12/31/1988
>>>>> TA1  - rtems_clock_get_tod - 09:00:30   12/31/1988
>>>>> *** END OF TEST CLOCK TICK ***
>>>>> Fatal Error 9.276564 Halted
>>>>>
>>>>>   From GDB:
>>>>>
>>>>> (gdb) break _Terminate
>>>>> Breakpoint 1 at 0x19a68: file
>>>>> ../../../../../../rtems/c/src/../../cpukit/score/src/interr.c, line
>>>>> 39.
>>>>> (gdb) break _OR1K_Exception_default
>>>>> Breakpoint 2 at 0x2686c: file
>>>>>
>>>>>
>>>>> ../../../../../../../../rtems/c/src/../../cpukit/score/cpu/or1k/or1k-exception-default.c,
>>>>> line 22.
>>>>> (gdb) c
>>>>> The program is not being run.
>>>>> (gdb) target remote :50001
>>>>> Remote debugging using :50001
>>>>> 0x00000100 in _reset ()
>>>>> (gdb) c
>>>>> Continuing.
>>>>>
>>>>> Breakpoint 2, _OR1K_Exception_default (vector=6, frame=0x43854) at
>>>>>
>>>>>
>>>>> ../../../../../../../../rtems/c/src/../../cpukit/score/cpu/or1k/or1k-exception-default.c:22
>>>>> 22  rtems_fatal( RTEMS_FATAL_SOURCE_EXCEPTION, (rtems_fatal_code) frame
>>>>> );
>>>>> (gdb) bt
>>>>> #0  _OR1K_Exception_default (vector=6, frame=0x43854) at
>>>>>
>>>>>
>>>>> ../../../../../../../../rtems/c/src/../../cpukit/score/cpu/or1k/or1k-exception-default.c:22
>>>>> #1  0x00026980 in jump_to_c_handler ()
>>>>> Backtrace stopped: frame did not save the PC
>>>>>
>>>>> vector 6 is _unalign exception.
>>>>
>>>> Set a break point at exit() (I think) and rtems_shutdown_executive(). You
>>>> could start in the task which calls whatever kicks off the shutdown
>>>> sequence.
>>>> It looks like something in the shutdown procedure trips over something.
>>>> This might be easy to debug.
>>>>
>>> I did add just a function call to _CPU_Exception_frame_print(frame);
>>>   from _OR1K_Exception_default(uint32_t vector, CPU_Exception_frame
>>> *frame)
>>> And ticker exits normally without even entering
>>> _OR1K_Exception_defaul as it did before. This is very weird. Does this
>>> mean that some areas of the code are overlapped from the linker
>>> script?
>>
>> I doubt it. I suspect something unitialized or not aligned properly.
>>
>> Set a breakpoint at
>> http://git.rtems.org/rtems/tree/testsuites/samples/ticker/tasks.c#n40
>> next over the print and then step through rtems_test_exit() and see
>> where it faults.
>>>>
>>>> If the fault address is in the exception data, you can map that back to
>>>> the
>>>> nm file and see what file that was in, then that might help.
>>>>>>
>>>>>> + Have you tried the trick I suggested earlier to disable the
>>>>>> real clock tick driver, use the simulator idle tick code, and
>>>>>> disable all the tests that are known to fail that way. This
>>>>>> will eliminate the ISR code as an issue because you won't
>>>>>> have any (if console output if polled).  See h8sim for
>>>>>> an example. Should be a Makefile.am change, adding
>>>>>> an include to the testsuite configuration file, building
>>>>>> and running.
>>>>>>
>>
>> --
>> Joel Sherrill, Ph.D.             Director of Research & Development
>> joel.sherrill at OARcorp.com        On-Line Applications Research
>> Ask me about RTEMS: a free RTOS  Huntsville AL 35805
>> Support Available                (256) 722-9985
>>



More information about the devel mailing list