<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">Your approach (associate addresses in
stack trace with source code) is what I usually do, too.<br>
Also disassemble the fault location and get hints from the
register values.<br>
<br>
I assume between #2 and #3 there was also a power-cycle.<br>
<br>
It is interesting that the fault affects different threads and
areas of code. This has a strong smell of<br>
memory/stack corruption. I.e., your culprit could be code which is
totally unrelated with where the<br>
fault occurs.<br>
<br>
In the first case (program exception, e.g., due to illegal
instruction) you'd inspect the fault address<br>
and verify that it holds an illegal instruction. You'd then check
if the address itself is ok, i.e., somewhere<br>
in the text segment where it would make sense. It could be that
the text was overwritten. Otherwise,<br>
the problem occurred earlier e.g., by jumping to a corrupted
pointer value.<br>
<br>
If you are able to locate the corrupted memory then sometimes its
contents can give you a<br>
hint as to what wrote to it. Otherwise an 'electric fence' tool
can be quite useful. This is a library<br>
which wraps malloc & friends to allocate memory always so that
start or end are aligned on a<br>
page-boundary. It uses the MMU to protect adjacent pages and thus
causes an exception on attempt<br>
to write outside of the allocated region (with standard MMUs it is
only possible to protect either the<br>
beginning or the end of the region).<br>
<br>
HTH<br>
- Till<br>
<br>
On 06/26/2017 08:40 PM, Matt Rippa wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAJdne7=AA2cyaMeBfWZw2wNpM9R96-zCtCGSrXSASkYfRj8G-Q@mail.gmail.com">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<div dir="ltr">
<div class="gmail_default" style="font-size:small">Hi Till, all:</div>
<div class="gmail_default" style="font-size:small"><br>
</div>
<div class="gmail_default" style="font-size:small">Last week we
recommissioned one of our telescope systems using RTEMS and
EPICS on the MVME2307 BSP. After nearly two days of uptime
(~42 hours), we get various unrecoverable exceptions (shown
below). This occurs suddenly when the system is "idle" and not
being used for many hours.</div>
<div class="gmail_default" style="font-size:small"><br>
</div>
<div class="gmail_default" style="font-size:small">Any
suggestions tracking down this stack trace, finding the
offending application code, or getting this to fail-faster? </div>
<div class="gmail_default" style="font-size:small">We've
produced an object dump available at the following link: <a
href="https://github.com/rcardenes/crcs-info"
moz-do-not-send="true">https://github.com/rcardenes/crcs-info</a></div>
<div class="gmail_default" style="font-size:small"><br>
</div>
<div class="gmail_default" style="font-size:small">Has anyone
else used the MVME2307 BSP?</div>
<div class="gmail_default" style="font-size:small"><br>
</div>
<div class="gmail_default" style="font-size:small">Thanks,</div>
<div class="gmail_default" style="font-size:small">-Matt</div>
<div class="gmail_default" style="font-size:small"><br>
</div>
<div class="gmail_default" style="font-size:small">System
details:</div>
<div class="gmail_default" style="font-size:small">RTEMS: 4.10.2</div>
<div class="gmail_default" style="font-size:small">EPICS:
3.14.12.4. </div>
<div class="gmail_default" style="font-size:small">Hardware:
MVME-2700 (using mvme2307 alias)</div>
<div class="gmail_default" style="font-size:small">VME Boards: </div>
<div class="gmail_default" style="font-size:small">
<ol>
<li>Bancomm 635 Time Board<br>
</li>
<li>PMAC 1</li>
<li>Xycom-240</li>
</ol>
</div>
<div class="gmail_default" style="font-size:small"><br>
</div>
<div class="gmail_default" style="font-size:small"><br>
</div>
<div class="gmail_default" style="font-size:small"><b>Event #1:</b></div>
<div class="gmail_default" style="font-size:small">
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex"><span
style="font-family:monospace"><span
style="color:rgb(0,0,0)">Jun 23 15:19:43 E) PORT: crcs,
MSG: </span><span
style="color:rgb(255,255,255);background-color:rgb(0,0,0)">Exception</span><span
style="color:rgb(0,0,0)"> handler called for exception 7
(0x7) <br>
</span></span><span style="font-family:monospace">Jun 23
15:19:43 E) PORT: crcs, MSG: #011 Next PC or Address of
fault = 011B010C <br>
</span><span style="font-family:monospace">Jun 23 15:19:43
E) PORT: crcs, MSG: #011 Saved MSR = 0008B032 <br>
</span><span style="font-family:monospace">Jun 23 15:19:43
E) PORT: crcs, MSG: #011 Context: Task ID 0x0A01004B <br>
</span><span style="font-family:monospace">Jun 23 15:19:43
E) PORT: crcs, MSG: #011 R0 = 01010101 R1 = 00AF5A40 R2
= 00000000 R3 = 007F8014 <br>
</span><span style="font-family:monospace">Jun 23 15:19:43
E) PORT: crcs, MSG: #011 R4 = 00000014 R5 = 007AD148 R6
= 00AF62BC R7 = 00000000 <br>
</span><span style="font-family:monospace">Jun 23 15:19:43
E) PORT: crcs, MSG: #011 R8 = 000031A8 R9 = 011B0101
R10 = 00AF62B8 R11 = 007AD148 <br>
</span><span style="font-family:monospace">Jun 23 15:19:44
E) PORT: crcs, MSG: #011 R12 = 22000022 R13 = 001E4B90
R14 = AEFE7BBF R15 = 77FF7FBB <br>
</span><span style="font-family:monospace">Jun 23 15:19:44
E) PORT: crcs, MSG: #011 R16 = AA7FBFFC R17 = FF7FF5FF
R18 = 001E0000 R19 = 00000000 <br>
</span><span style="font-family:monospace">Jun 23 15:19:44
E) PORT: crcs, MSG: #011 R20 = 000634A0 R21 = 001929E8
R22 = 001A2284 R23 = 00191098 <br>
</span><span style="font-family:monospace">Jun 23 15:19:44
E) PORT: crcs, MSG: #011 R24 = 00000000 R25 = 00000001
R26 = 00000001 R27 = 001A85C4 <br>
</span><span style="font-family:monospace">Jun 23 15:19:44
E) PORT: crcs, MSG: #011 R28 = 00AF62BC R29 = 00000028
R30 = 00419260 R31 = 007AD148 <br>
</span><span style="font-family:monospace">Jun 23 15:19:44
E) PORT: crcs, MSG: #011 CR = 22042028 <br>
</span><span style="font-family:monospace">Jun 23 15:19:44
E) PORT: crcs, MSG: #011 CTR = 011B00FF <br>
</span><span style="font-family:monospace">Jun 23 15:19:44
E) PORT: crcs, MSG: #011 XER = 20000000 <br>
</span><span style="font-family:monospace">Jun 23 15:19:44
E) PORT: crcs, MSG: #011 LR = 0006371C <br>
</span><span style="font-family:monospace">Jun 23 15:19:44
E) PORT: crcs, MSG: #011 DAR = 00000000 <br>
</span><span style="font-family:monospace">Jun 23 15:19:44
E) PORT: crcs, MSG: Stack Trace: <br>
</span><span style="font-family:monospace">Jun 23 15:19:44
E) PORT: crcs, MSG: IP: 0x011B010C, LR: 0x0006371C <br>
</span><span style="font-family:monospace">Jun 23 15:19:44
E) PORT: crcs, MSG: --^ 0x0006371C--^ 0x00082FDC--^
0x000E7094--^ 0x00136048--^ 0x00135F6C <br>
</span><span style="font-family:monospace">Jun 23 15:19:44
E) PORT: crcs, MSG: Suspending faulting task (0x0A01004B)
</span></blockquote>
</div>
<div class="gmail_default" style="font-size:small">...</div>
<div class="gmail_default" style="font-size:small"><br>
</div>
<div class="gmail_default" style="font-size:small">Using the
memory map for the Exception 7 we reach the following trace:<br>
</div>
<div class="gmail_default">
<div class="gmail_default"><br>
</div>
</div>
<blockquote style="margin:0px 0px 0px
40px;border:none;padding:0px">
<div class="gmail_default">
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">0x00135f6c
_Thread_Handler</blockquote>
</div>
<div class="gmail_default">
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">/gem_sw/targetOS/RTEMS/source/rtems/rtems-4.10.2/cpukit/score/src/threadhandler.c:80</blockquote>
</div>
<div class="gmail_default">
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">0x00136048</blockquote>
</div>
<div class="gmail_default">
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">/gem_sw/targetOS/RTEMS/source/rtems/rtems-4.10.2/cpukit/score/src/threadhandler.c:145</blockquote>
</div>
<div class="gmail_default">
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">0x000e7094</blockquote>
</div>
<div class="gmail_default">
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">/gem_sw/epics/R3.14.12.4/base/src/libCom/osi/os/RTEMS/osdThread.c:169</blockquote>
</div>
<div class="gmail_default">
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">0x00082fdc</blockquote>
</div>
<div class="gmail_default">
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">/gem_sw/epics/R3.14.12.4/base/src/db/dbEvent.c:883</blockquote>
</div>
<div class="gmail_default">
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">0x0006371c</blockquote>
</div>
<div class="gmail_default">
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">/gem_sw/targetOS/RTEMS/source/tools/gcc-4.4.7/newlib/libc/stdio/vfscanf.c:270</blockquote>
</div>
</blockquote>
<div class="gmail_default" style="font-size:small"><br>
</div>
<div class="gmail_default" style="font-size:small">Power Cycle</div>
<div class="gmail_default" style="font-size:small"><br>
</div>
<div class="gmail_default" style="font-size:small"><b>Event #2:</b></div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px
solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><span
style="font-family:monospace"><span style="color:rgb(0,0,0)">Jun
23 18:32:20 E) PORT: crcs, MSG: </span><span
style="color:rgb(255,255,255);background-color:rgb(0,0,0)">Exception</span><span
style="color:rgb(0,0,0)"> handler called for exception 3
(0x3) </span><br>
Jun 23 18:32:20 E) PORT: crcs, MSG: #011 Next PC or Address
of fault = 001541E4 <br>
Jun 23 18:32:20 E) PORT: crcs, MSG: #011 Saved MSR =
00009032 <br>
Jun 23 18:32:20 E) PORT: crcs, MSG: #011 Context: Task ID
0x09010001 <br>
Jun 23 18:32:20 E) PORT: crcs, MSG: #011 R0 = FFF5A4DA R1
= 00366720 R2 = 00000000 R3 = FFF0A442 <br>
Jun 23 18:32:20 E) PORT: crcs, MSG: #011 R4 = 00352498 R5
= 00000008 R6 = 00000000 R7 = FFF0A442 <br>
Jun 23 18:32:20 E) PORT: crcs, MSG: #011 R8 = 00352498 R9
= 00000002 R10 = 00000000 R11 = 00000001 <br>
Jun 23 18:32:20 E) PORT: crcs, MSG: #011 R12 = 003BC0E0 R13
= 001E4B90 R14 = 00000000 R15 = 00000000 <br>
Jun 23 18:32:21 E) PORT: crcs, MSG: #011 R16 = 00000000 R17
= 00000000 R18 = 00000000 R19 = 00000000 <br>
Jun 23 18:32:21 E) PORT: crcs, MSG: #011 R20 = 00000000 R21
= 00000000 R22 = 00000000 R23 = 00000000 <br>
Jun 23 18:32:21 E) PORT: crcs, MSG: #011 R24 = 00000000 R25
= 00000000 R26 = 00000000 R27 = 00000000 <br>
Jun 23 18:32:21 E) PORT: crcs, MSG: #011 R28 = FFF0A340 R29
= 00000000 R30 = FFF0A341 R31 = 0034D5B0 <br>
Jun 23 18:32:21 E) PORT: crcs, MSG: #011 CR = 40000004 <br>
Jun 23 18:32:21 E) PORT: crcs, MSG: #011 CTR = 0012E3AC <br>
Jun 23 18:32:21 E) PORT: crcs, MSG: #011 XER = 00000000 <br>
Jun 23 18:32:21 E) PORT: crcs, MSG: #011 LR = 000F2CDC <br>
Jun 23 18:32:21 E) PORT: crcs, MSG: #011 DAR = FFF0A442 <br>
Jun 23 18:32:21 E) PORT: crcs, MSG: Stack Trace: <br>
Jun 23 18:32:21 E) PORT: crcs, MSG: IP: 0x001541E4, LR:
0x000F2CDC <br>
Jun 23 18:32:21 E) PORT: crcs, MSG: --^ 0x00135F6C <br>
Jun 23 18:32:21 E) PORT: crcs, MSG: Suspending faulting
task (0x09010001)</span></blockquote>
<div><br>
</div>
<div class="gmail_default" style="font-size:small"><b>Event
#3:</b></div>
<div class="gmail_default" style="font-size:small">
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex"><span
style="font-family:monospace"><span
style="color:rgb(0,0,0)">Jun 24 07:37:50 E) PORT: crcs,
MSG: *Exception handler called for </span><span
style="color:rgb(255,255,255);background-color:rgb(0,0,0)">exception
8</span><span style="color:rgb(0,0,0)"> (0x8) <br>
</span></span><span style="font-family:monospace">Jun 24
07:37:50 E) PORT: crcs, MSG: #011 Next PC or Address of
fault = 0015BB08 <br>
</span><span style="font-family:monospace">Jun 24 07:37:50
E) PORT: crcs, MSG: #011 Saved MSR = 00009032 <br>
</span><span style="font-family:monospace">Jun 24 07:37:50
E) PORT: crcs, MSG: #011 Context: Task ID 0x09010001 <br>
</span><span style="font-family:monospace">Jun 24 07:37:50
E) PORT: crcs, MSG: #011 R0 = 00157754 R1 = 003664B8 R2
= 00000000 R3 = 00366758 <br>
</span><span style="font-family:monospace">Jun 24 07:37:50
E) PORT: crcs, MSG: #011 R4 = 0036661C R5 = 001B4C68 R6
= 00366610 R7 = 00000000 <br>
</span><span style="font-family:monospace">Jun 24 07:37:50
E) PORT: crcs, MSG: #011 R8 = 005F5370 R9 = 00642450
R10 = 0034C940 R11 = 00366688 <br>
</span><span style="font-family:monospace">Jun 24 07:37:50
E) PORT: crcs, MSG: #011 R12 = 40000048 R13 = 001E4B90
R14 = 001B4C68 R15 = 00000000 <br>
</span><span style="font-family:monospace">Jun 24 07:37:51
E) PORT: crcs, MSG: #011 R16 = 00000000 R17 = 00000000
R18 = 00000000 R19 = 00366610 <br>
</span><span style="font-family:monospace">Jun 24 07:37:51
E) PORT: crcs, MSG: #011 R20 = 00000000 R21 = 00000000
R22 = 00366758 R23 = 001B4CC0 <br>
</span><span style="font-family:monospace">Jun 24 07:37:51
E) PORT: crcs, MSG: #011 R24 = 0036661C R25 = 00377E40
R26 = 00000000 R27 = 0000002F <br>
</span><span style="font-family:monospace">Jun 24 07:37:51
E) PORT: crcs, MSG: #011 R28 = 00377E40 R29 = 001E49B0
R30 = 00377E40 R31 = 00000000 <br>
</span><span style="font-family:monospace">Jun 24 07:37:51
E) PORT: crcs, MSG: #011 CR = 40000048 <br>
</span><span style="font-family:monospace">Jun 24 07:37:51
E) PORT: crcs, MSG: #011 CTR = 00000000 <br>
</span><span style="font-family:monospace">Jun 24 07:37:51
E) PORT: crcs, MSG: #011 XER = 00000000 <br>
</span><span style="font-family:monospace">Jun 24 07:37:51
E) PORT: crcs, MSG: #011 LR = 00157754 <br>
</span><span style="font-family:monospace">Jun 24 07:37:51
E) PORT: crcs, MSG: #011 DAR = 00000000 <br>
</span><span style="font-family:monospace">Jun 24 07:37:51
E) PORT: crcs, MSG: Stack Trace: <br>
</span><span style="font-family:monospace">Jun 24 07:37:51
E) PORT: crcs, MSG: IP: 0x0015BB08, LR: 0x00157754 <br>
</span><span style="font-family:monospace">Jun 24 07:37:51
E) PORT: crcs, MSG: --^ 0x00157754--^ 0x000F2820--^
0x000F2DA0--^ 0x001E49BD--^ 0x00135F6C <br>
</span><span style="font-family:monospace">Jun 24 07:37:51
E) PORT: crcs, MSG: Suspending faulting task (0x09010001)
</span></blockquote>
<span style="font-family:monospace">
<br>
</span></div>
<div class="gmail_default" style="font-size:small"><span
style="font-family:monospace"><br>
</span></div>
<div class="gmail_default" style="font-size:small"><br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px
solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"> <span
style="font-family:monospace"><br>
</span></blockquote>
</div>
</blockquote>
<br>
</body>
</html>