<div dir="ltr"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">breaking at _Terminate and doing a back trace will give<br> you the exact line the error is raised from. </blockquote><div>I don't know why I did not focus on this earlier! So I put a breakpoint at the <a href="https://git.rtems.org/rtems/tree/cpukit/score/src/heapfree.c#n73" target="_blank">line</a> which detects a fault and now we know which is the corrupted address:</div><div>-----------------------------------------------------------------</div><div>Continuing.<br><br>Thread 1 hit Breakpoint 9, _Heap_Protection_check_free_block (heap=0x202ba8 <_Workspace_Area>, block=0x206fec) at /home/richi/quick-start/LatestStrong/src/rtems/c/src/../../cpukit/score/src/heapfree.c:73<br>73 _Heap_Protection_block_error( heap, block, HEAP_ERROR_FREE_PATTERN );<br>(gdb) p current<br>$7 = (uintptr_t *) 0x207bac<br></div><div>-----------------------------------------------------------------<br></div><div><br></div><div>but when I try to find which function actually changes its value by putting a watchpoint, I get this:</div><div>-----------------------------------------------------------------<br></div><div>(gdb) watch *(uintptr_t *) 0x207bac<br>Hardware watchpoint 7: *(uintptr_t *) 0x207bac<br></div><div>(gdb) reset<br>Loading section .start, size 0xa5c lma 0x100000<br>...<br>Transfer rate: 3102 KB/sec, 1855 bytes/write.<br>(gdb) c<br>Continuing.<br><br>Thread 3 hit Hardware watchpoint 7: *(uintptr_t *) 0x207bac<br><br>Old value = 1134949<br>New value = 1050747<br>0x00102c06 in _SMP_Get_current_processor () at /home/richi/quick-start/LatestStrong/src/rtems/cpukit/include/rtems/score/smp.h:65<br>65 {<br>(gdb) <br>Continuing.<br><br>Thread 3 hit Hardware watchpoint 7: *(uintptr_t *) 0x207bac<br><br>Old value = 1050747<br>New value = 2128816<br>0x00102bf6 in _ARM_Wait_for_event () at /home/richi/quick-start/LatestStrong/src/rtems/cpukit/score/cpu/arm/include/rtems/score/cpu.h:505<br>505 {<br>(gdb) <br>Continuing.<br>[Switching to Thread 1.1]<br><br>Thread 1 hit Breakpoint 6, Init (argument=2107944) at /home/richi/quick-start/LatestStrong/src/rtems/c/src/../../testsuites/sptests/sp02/init.c:26<br>26 TEST_BEGIN();<br>(gdb) <br>Continuing.<br><br>Thread 1 hit Hardware watchpoint 7: *(uintptr_t *) 0x207bac<br><br>Old value = 2128816<br>New value = 3876142303<br>_Heap_Protection_delay_block_free (heap=0x202ba8 <_Workspace_Area>, block=0x206fec) at /home/richi/quick-start/LatestStrong/src/rtems/c/src/../../cpukit/score/src/heapfree.c:55<br>55 for ( current = pattern_begin; current != pattern_end; ++current ) {<br>(gdb) <br>Continuing.<br>[Switching to Thread 1.3]<br><br>Thread 3 hit Hardware watchpoint 7: *(uintptr_t *) 0x207bac<br><br>Old value = 3876142303<br>New value = 2128816<br>0x00102bf6 in _ARM_Wait_for_event () at /home/richi/quick-start/LatestStrong/src/rtems/cpukit/score/cpu/arm/include/rtems/score/cpu.h:505<br>505 {<br>(gdb) bt<br>#0 0x00102bf6 in _ARM_Wait_for_event () at /home/richi/quick-start/LatestStrong/src/rtems/cpukit/score/cpu/arm/include/rtems/score/cpu.h:505<br>#1 0x001008a8 in bsp_start_hook_0 () at /home/richi/quick-start/LatestStrong/src/rtems/c/src/lib/libbsp/arm/realview-pbx-a9/../../../../../../bsps/arm/realview-pbx-a9/start/bspstarthooks.c:76<br>#2 0x00100244 in bsp_start_vector_table_end () at /home/richi/quick-start/LatestStrong/src/rtems/c/src/lib/libbsp/arm/realview-pbx-a9/../../../../../../bsps/arm/shared/start/start.S:464<br>Backtrace stopped: previous frame identical to this frame (corrupt stack?)<br>(gdb) <br></div><div>-----------------------------------------------------------------<br></div><div><br></div><div>The value of HEAP_FREE_PATTERN is 3876142303. So I know that the function _ARM_Wait_for_event is the one that actually corrupts this value. But, the backtrace does not help much. Which function calls this function? What is the trace of this function?</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Mar 6, 2021 at 10:28 AM Richi Dubey <<a href="mailto:richidubey@gmail.com" target="_blank">richidubey@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">This is just the detection point. The allocate is doing a validity check and something is wrong from an overwrite. <br>FWIW this is pretty early in the test I think.</blockquote><div>Good point, the corruption has already happened earlier, and yes its quite early : </div><div>...</div><div>#13 0x0010860e in rtems_task_create (name=1413558560, initial_priority=1, stack_size=4096, initial_modes=0, attribute_set=0, id=0x202444 <Task_id+4>) at /home/richi/quick-start/LatestStrong/src/rtems/c/src/../../cpukit/rtems/src/taskcreate.c:84<br></div><div>#14 0x001015fa in Init (argument=2107944) at /home/richi/quick-start/LatestStrong/src/rtems/c/src/../../testsuites/sptests/sp02/init.c:101<br></div><div><br></div><div><br></div><div><br></div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Look manually at say 32 bytes at that address at various points during the program's execution. I think this is one where a binary search for the corrupting action happens.<br></blockquote><div>Yes, or maybe I would try to manually put a watchpoint at all the 32 bytes starting 0x202ba8 and see if it works.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Is this using your scheduler as the default? If so, I'd be suspicious of anything allocated for it and if you were riding outside an area allocated for you.</blockquote><div>Yes! Maybe the scheduler access a variable outside its bound (maybe an array element outside its array size), but if that is true, there should be a lot more failure with HEAP_ERROR. I would still give it a look.</div></div><div><br></div><div>Thanks again for your help.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Mar 5, 2021 at 7:01 PM Joel Sherrill <<a href="mailto:joel@rtems.org" target="_blank">joel@rtems.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto"><div><br><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Mar 4, 2021, 11:31 PM Richi Dubey <<a href="mailto:richidubey@gmail.com" target="_blank">richidubey@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi,<div><br></div><div>Thanks to both of you for helping me out with this!</div><div><br></div><div>When I backtrace on _Terminate: I get this:</div><div><br></div><div>Init -> rtems_task_create -> ... -> _Heap_Allocate -> ... ->_Heap_Protection_check_free_block -> _Heap_Protection_block_error ->_Heap_Protection_block_error_default -> _Terminate (the_source=RTEMS_FATAL_SOURCE_HEAP, the_error=2125180). So I will try to debug this trace.<br></div></div></blockquote></div></div><div dir="auto"><br></div><div dir="auto">This is just the detection point. The allocate is doing a validity check and something is wrong from an overwrite. </div><div dir="auto"><br></div><div dir="auto">FWIW this is pretty early in the test I think.</div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div></div><div><br></div><div>Also, setting a watchpoint doesn't help:</div></div></blockquote></div></div><div dir="auto"><br></div><div dir="auto">Look manually at say 32 bytes at that address at various points during the program's execution. I think this is one where a binary search for the corrupting action happens.</div><div dir="auto"><br></div><div dir="auto">Is this using your scheduler as the default? If so, I'd be suspicious of anything allocated for it and if you were riding outside an area allocated for you. </div><div dir="auto"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>Thread 1 hit Breakpoint 6, Init (argument=2107944) at /home/richi/quick-start/LatestStrong/src/rtems/c/src/../../testsuites/sptests/sp02/init.c:26<br>26 TEST_BEGIN();<br>(gdb) watch *(unsigned int)* 0x202ba8<br>Hardware watchpoint 13: *(unsigned int)* 0x202ba8<br>(gdb) c<br>Continuing.<br><br>Thread 1 hit Breakpoint 5, _Terminate (the_source=RTEMS_FATAL_SOURCE_HEAP, the_error=2125180) at /home/richi/quick-start/LatestStrong/src/rtems/c/src/../../cpukit/score/src/interr.c:38<br>38 _User_extensions_Fatal( the_source, the_error );<br></div><div><br></div><div> </div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Mar 3, 2021 at 10:20 PM Gedare Bloom <<a href="mailto:gedare@rtems.org" rel="noreferrer" target="_blank">gedare@rtems.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Wed, Mar 3, 2021 at 9:49 AM Gedare Bloom <<a href="mailto:gedare@rtems.org" rel="noreferrer" target="_blank">gedare@rtems.org</a>> wrote:<br>
><br>
> On Wed, Mar 3, 2021 at 9:28 AM Joel Sherrill <<a href="mailto:joel.sherrill@gmail.com" rel="noreferrer" target="_blank">joel.sherrill@gmail.com</a>> wrote:<br>
> ><br>
> ><br>
> ><br>
> > On Wed, Mar 3, 2021 at 9:50 AM Gedare Bloom <<a href="mailto:gedare@rtems.org" rel="noreferrer" target="_blank">gedare@rtems.org</a>> wrote:<br>
> >><br>
> >> On Wed, Mar 3, 2021 at 12:08 AM Richi Dubey <<a href="mailto:richidubey@gmail.com" rel="noreferrer" target="_blank">richidubey@gmail.com</a>> wrote:<br>
> >> ><br>
> >> > What's the element written after the free? I set a watch at the exact block location, but it doesn't work:<br>
> >> ><br>
> >> > Hardware watchpoint 7: *0x202ba8<br>
> >> > (gdb) watch *0x206fec<br>
> >> > Hardware watchpoint 8: *0x206fec<br>
> >> That's just the first byte in the block. If you can figure out which<br>
> >> bytes/words in the block get accessed that would help you.<br>
> ><br>
> ><br>
> > What about watch *(unsigned int)* 0x202ba8?<br>
> ><br>
> > Won't that look at more bytes?<br>
><br>
> And this is just the first byte of the workspace area.<br>
><br>
4 bytes :)<br>
> ><br>
> > In case you do need to look at more bytes in the fence...<br>
> > breaking at _Terminate and doing a back trace will give<br>
> > you the exact line the error is raised from. You can then set a<br>
> > breakpoint at that on the next line and look at local variables.<br>
> > The corruption may be in the fence somewhere beyond the<br>
> > first 32-bits.<br>
> ><br>
In the case of heap corruption, the corruption is detected after it<br>
already happened. Narrowing down when/where the corruption happens is<br>
necessary. The next thing to do would be to examine the pattern that<br>
triggers the violation, and see where it got modified. This might<br>
provide a byte address to set a watch on.<br>
<br>
> > Sometimes it is easy to binary search for an issue like this<br>
> > on a simulator. But with a watchpoint, you should be able to<br>
> > determine the precise word which is corrupted in the fence<br>
> > and break on that write.<br>
> ><br>
> > --joel<br>
> >><br>
> >><br>
> >> > (gdb) c<br>
> >> > Continuing.<br>
> >> ><br>
> >> > Thread 1 hit Breakpoint 6, Init (argument=2107944) at /home/richi/quick-start/LatestStrong/src/rtems/c/src/../../testsuites/sptests/sp02/init.c:26<br>
> >> > 26 TEST_BEGIN();<br>
> >> > (gdb)<br>
> >> > Continuing.<br>
> >> ><br>
> >> > Thread 1 hit Breakpoint 5, _Terminate (the_source=RTEMS_FATAL_SOURCE_HEAP, the_error=2125180) at /home/richi/quick-start/LatestStrong/src/rtems/c/src/../../cpukit/score/src/interr.c:38<br>
> >> > 38 _User_extensions_Fatal( the_source, the_error );<br>
> >> > (gdb)<br>
> >> > Continuing.<br>
> >> ><br>
> >> > Thread 1 hit Breakpoint 4, bsp_reset () at /home/richi/quick-start/LatestStrong/src/rtems/c/src/lib/libbsp/arm/realview-pbx-a9/../../../../../../bsps/arm/realview-pbx-a9/start/bspreset.c:19<br>
> >> > 19 volatile uint32_t *sys_lock = (volatile uint32_t *) 0x10000020;<br>
> >> > (gdb)<br>
> >> ><br>
> >> > On Wed, Mar 3, 2021 at 12:31 PM Sebastian Huber <<a href="mailto:sebastian.huber@embedded-brains.de" rel="noreferrer" target="_blank">sebastian.huber@embedded-brains.de</a>> wrote:<br>
> >> >><br>
> >> >> On 02/03/2021 05:44, Richi Dubey wrote:<br>
> >> >><br>
> >> >> ><br>
> >> >> > (gdb) p *(Heap_Error_context*)(0x00206d7c)<br>
> >> >> > $5 = {<br>
> >> >> > heap = 0x202ba8 <_Workspace_Area>,<br>
> >> >> > block = 0x206fec,<br>
> >> >> > reason = HEAP_ERROR_FREE_PATTERN<br>
> >> >> > }<br>
> >> >><br>
> >> >> If it is always the same address, then you can set a watch point to an<br>
> >> >> element which is written after the free to catch the function which<br>
> >> >> writes into this area.<br>
> >> >><br>
> >> >> --<br>
> >> >> embedded brains GmbH<br>
> >> >> Herr Sebastian HUBER<br>
> >> >> Dornierstr. 4<br>
> >> >> 82178 Puchheim<br>
> >> >> Germany<br>
> >> >> email: <a href="mailto:sebastian.huber@embedded-brains.de" rel="noreferrer" target="_blank">sebastian.huber@embedded-brains.de</a><br>
> >> >> phone: +49-89-18 94 741 - 16<br>
> >> >> fax: +49-89-18 94 741 - 08<br>
> >> >><br>
> >> >> Registergericht: Amtsgericht München<br>
> >> >> Registernummer: HRB 157899<br>
> >> >> Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler<br>
> >> >> Unsere Datenschutzerklärung finden Sie hier:<br>
> >> >> <a href="https://embedded-brains.de/datenschutzerklaerung/" rel="noreferrer noreferrer" target="_blank">https://embedded-brains.de/datenschutzerklaerung/</a><br>
> >> >><br>
> >> > _______________________________________________<br>
> >> > devel mailing list<br>
> >> > <a href="mailto:devel@rtems.org" rel="noreferrer" target="_blank">devel@rtems.org</a><br>
> >> > <a href="http://lists.rtems.org/mailman/listinfo/devel" rel="noreferrer noreferrer" target="_blank">http://lists.rtems.org/mailman/listinfo/devel</a><br>
> >> _______________________________________________<br>
> >> devel mailing list<br>
> >> <a href="mailto:devel@rtems.org" rel="noreferrer" target="_blank">devel@rtems.org</a><br>
> >> <a href="http://lists.rtems.org/mailman/listinfo/devel" rel="noreferrer noreferrer" target="_blank">http://lists.rtems.org/mailman/listinfo/devel</a><br>
</blockquote></div>
</blockquote></div></div></div>
</blockquote></div>
</blockquote></div>