<div dir="ltr"><div class="gmail_default" style="font-size:small">Hi Till,</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">Thanks for your advice. Looking at the EPICS release notes, there have been many fixes between EPICS 3.14.12.4 and 3.14.12.6. <a href="http://www.aps.anl.gov/epics/base/R3-14/12-docs/RELEASE_NOTES.html">http://www.aps.anl.gov/epics/base/R3-14/12-docs/RELEASE_NOTES.html</a> So my first step was upgrading and rebuilding all my support code.</div><div class="gmail_default" style="font-size:small">That's working now but I won't get back to the telescope for a few days.</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">I took a look at your electric fence. This looks very useful. I have to say I learned few things about the linker I never knew about.</div><div class="gmail_default" style="font-size:small">I built the library and linked it as instructed. My first attempts today caused a seq fault immediately on iocInit(). I've attached relevant files.</div><div class="gmail_default" style="font-size:small">I need more time to look at this.</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">Cheers,</div><div class="gmail_default" style="font-size:small">-Matt</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jul 3, 2017 at 9:48 AM, Till Straumann <span dir="ltr"><<a href="mailto:strauman@slac.stanford.edu" target="_blank">strauman@slac.stanford.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div class="m_685219899655768212moz-cite-prefix">Your approach (associate addresses in
stack trace with source code) is what I usually do, too.<br>
Also disassemble the fault location and get hints from the
register values.<br>
<br>
I assume between #2 and #3 there was also a power-cycle.<br>
<br>
It is interesting that the fault affects different threads and
areas of code. This has a strong smell of<br>
memory/stack corruption. I.e., your culprit could be code which is
totally unrelated with where the<br>
fault occurs.<br>
<br>
In the first case (program exception, e.g., due to illegal
instruction) you'd inspect the fault address<br>
and verify that it holds an illegal instruction. You'd then check
if the address itself is ok, i.e., somewhere<br>
in the text segment where it would make sense. It could be that
the text was overwritten. Otherwise,<br>
the problem occurred earlier e.g., by jumping to a corrupted
pointer value.<br>
<br>
If you are able to locate the corrupted memory then sometimes its
contents can give you a<br>
hint as to what wrote to it. Otherwise an 'electric fence' tool
can be quite useful. This is a library<br>
which wraps malloc & friends to allocate memory always so that
start or end are aligned on a<br>
page-boundary. It uses the MMU to protect adjacent pages and thus
causes an exception on attempt<br>
to write outside of the allocated region (with standard MMUs it is
only possible to protect either the<br>
beginning or the end of the region).<br>
<br>
HTH<span class="HOEnZb"><font color="#888888"><br>
- Till</font></span><div><div class="h5"><br>
<br>
On 06/26/2017 08:40 PM, Matt Rippa wrote:<br>
</div></div></div><div><div class="h5">
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_default" style="font-size:small">Hi Till, all:</div>
<div class="gmail_default" style="font-size:small"><br>
</div>
<div class="gmail_default" style="font-size:small">Last week we
recommissioned one of our telescope systems using RTEMS and
EPICS on the MVME2307 BSP. After nearly two days of uptime
(~42 hours), we get various unrecoverable exceptions (shown
below). This occurs suddenly when the system is "idle" and not
being used for many hours.</div>
<div class="gmail_default" style="font-size:small"><br>
</div>
<div class="gmail_default" style="font-size:small">Any
suggestions tracking down this stack trace, finding the
offending application code, or getting this to fail-faster? </div>
<div class="gmail_default" style="font-size:small">We've
produced an object dump available at the following link: <a href="https://github.com/rcardenes/crcs-info" target="_blank">https://github.com/rcardenes/<wbr>crcs-info</a></div>
<div class="gmail_default" style="font-size:small"><br>
</div>
<div class="gmail_default" style="font-size:small">Has anyone
else used the MVME2307 BSP?</div>
<div class="gmail_default" style="font-size:small"><br>
</div>
<div class="gmail_default" style="font-size:small">Thanks,</div>
<div class="gmail_default" style="font-size:small">-Matt</div>
<div class="gmail_default" style="font-size:small"><br>
</div>
<div class="gmail_default" style="font-size:small">System
details:</div>
<div class="gmail_default" style="font-size:small">RTEMS: 4.10.2</div>
<div class="gmail_default" style="font-size:small">EPICS:
3.14.12.4. </div>
<div class="gmail_default" style="font-size:small">Hardware:
MVME-2700 (using mvme2307 alias)</div>
<div class="gmail_default" style="font-size:small">VME Boards: </div>
<div class="gmail_default" style="font-size:small">
<ol>
<li>Bancomm 635 Time Board<br>
</li>
<li>PMAC 1</li>
<li>Xycom-240</li>
</ol>
</div>
<div class="gmail_default" style="font-size:small"><br>
</div>
<div class="gmail_default" style="font-size:small"><br>
</div>
<div class="gmail_default" style="font-size:small"><b>Event #1:</b></div>
<div class="gmail_default" style="font-size:small">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span style="font-family:monospace"><span style="color:rgb(0,0,0)">Jun 23 15:19:43 E) PORT: crcs,
MSG: </span><span style="color:rgb(255,255,255);background-color:rgb(0,0,0)">Exception</span><span style="color:rgb(0,0,0)"> handler called for exception 7
(0x7) <br>
</span></span><span style="font-family:monospace">Jun 23
15:19:43 E) PORT: crcs, MSG: #011 Next PC or Address of
fault = 011B010C <br>
</span><span style="font-family:monospace">Jun 23 15:19:43
E) PORT: crcs, MSG: #011 Saved MSR = 0008B032 <br>
</span><span style="font-family:monospace">Jun 23 15:19:43
E) PORT: crcs, MSG: #011 Context: Task ID 0x0A01004B <br>
</span><span style="font-family:monospace">Jun 23 15:19:43
E) PORT: crcs, MSG: #011 R0 = 01010101 R1 = 00AF5A40 R2
= 00000000 R3 = 007F8014 <br>
</span><span style="font-family:monospace">Jun 23 15:19:43
E) PORT: crcs, MSG: #011 R4 = 00000014 R5 = 007AD148 R6
= 00AF62BC R7 = 00000000 <br>
</span><span style="font-family:monospace">Jun 23 15:19:43
E) PORT: crcs, MSG: #011 R8 = 000031A8 R9 = 011B0101
R10 = 00AF62B8 R11 = 007AD148 <br>
</span><span style="font-family:monospace">Jun 23 15:19:44
E) PORT: crcs, MSG: #011 R12 = 22000022 R13 = 001E4B90
R14 = AEFE7BBF R15 = 77FF7FBB <br>
</span><span style="font-family:monospace">Jun 23 15:19:44
E) PORT: crcs, MSG: #011 R16 = AA7FBFFC R17 = FF7FF5FF
R18 = 001E0000 R19 = 00000000 <br>
</span><span style="font-family:monospace">Jun 23 15:19:44
E) PORT: crcs, MSG: #011 R20 = 000634A0 R21 = 001929E8
R22 = 001A2284 R23 = 00191098 <br>
</span><span style="font-family:monospace">Jun 23 15:19:44
E) PORT: crcs, MSG: #011 R24 = 00000000 R25 = 00000001
R26 = 00000001 R27 = 001A85C4 <br>
</span><span style="font-family:monospace">Jun 23 15:19:44
E) PORT: crcs, MSG: #011 R28 = 00AF62BC R29 = 00000028
R30 = 00419260 R31 = 007AD148 <br>
</span><span style="font-family:monospace">Jun 23 15:19:44
E) PORT: crcs, MSG: #011 CR = 22042028 <br>
</span><span style="font-family:monospace">Jun 23 15:19:44
E) PORT: crcs, MSG: #011 CTR = 011B00FF <br>
</span><span style="font-family:monospace">Jun 23 15:19:44
E) PORT: crcs, MSG: #011 XER = 20000000 <br>
</span><span style="font-family:monospace">Jun 23 15:19:44
E) PORT: crcs, MSG: #011 LR = 0006371C <br>
</span><span style="font-family:monospace">Jun 23 15:19:44
E) PORT: crcs, MSG: #011 DAR = 00000000 <br>
</span><span style="font-family:monospace">Jun 23 15:19:44
E) PORT: crcs, MSG: Stack Trace: <br>
</span><span style="font-family:monospace">Jun 23 15:19:44
E) PORT: crcs, MSG: IP: 0x011B010C, LR: 0x0006371C <br>
</span><span style="font-family:monospace">Jun 23 15:19:44
E) PORT: crcs, MSG: --^ 0x0006371C--^ 0x00082FDC--^
0x000E7094--^ 0x00136048--^ 0x00135F6C <br>
</span><span style="font-family:monospace">Jun 23 15:19:44
E) PORT: crcs, MSG: Suspending faulting task (0x0A01004B)
</span></blockquote>
</div>
<div class="gmail_default" style="font-size:small">...</div>
<div class="gmail_default" style="font-size:small"><br>
</div>
<div class="gmail_default" style="font-size:small">Using the
memory map for the Exception 7 we reach the following trace:<br>
</div>
<div class="gmail_default">
<div class="gmail_default"><br>
</div>
</div>
<blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px">
<div class="gmail_default">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">0x00135f6c
_Thread_Handler</blockquote>
</div>
<div class="gmail_default">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">/gem_sw/targetOS/RTEMS/source/<wbr>rtems/rtems-4.10.2/cpukit/<wbr>score/src/threadhandler.c:80</blockquote>
</div>
<div class="gmail_default">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">0x00136048</blockquote>
</div>
<div class="gmail_default">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">/gem_sw/targetOS/RTEMS/source/<wbr>rtems/rtems-4.10.2/cpukit/<wbr>score/src/threadhandler.c:145</blockquote>
</div>
<div class="gmail_default">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">0x000e7094</blockquote>
</div>
<div class="gmail_default">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">/gem_sw/epics/R3.14.12.4/base/<wbr>src/libCom/osi/os/RTEMS/<wbr>osdThread.c:169</blockquote>
</div>
<div class="gmail_default">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">0x00082fdc</blockquote>
</div>
<div class="gmail_default">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">/gem_sw/epics/R3.14.12.4/base/<wbr>src/db/dbEvent.c:883</blockquote>
</div>
<div class="gmail_default">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">0x0006371c</blockquote>
</div>
<div class="gmail_default">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">/gem_sw/targetOS/RTEMS/source/<wbr>tools/gcc-4.4.7/newlib/libc/<wbr>stdio/vfscanf.c:270</blockquote>
</div>
</blockquote>
<div class="gmail_default" style="font-size:small"><br>
</div>
<div class="gmail_default" style="font-size:small">Power Cycle</div>
<div class="gmail_default" style="font-size:small"><br>
</div>
<div class="gmail_default" style="font-size:small"><b>Event #2:</b></div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><span style="font-family:monospace"><span style="color:rgb(0,0,0)">Jun
23 18:32:20 E) PORT: crcs, MSG: </span><span style="color:rgb(255,255,255);background-color:rgb(0,0,0)">Exception</span><span style="color:rgb(0,0,0)"> handler called for exception 3
(0x3) </span><br>
Jun 23 18:32:20 E) PORT: crcs, MSG: #011 Next PC or Address
of fault = 001541E4 <br>
Jun 23 18:32:20 E) PORT: crcs, MSG: #011 Saved MSR =
00009032 <br>
Jun 23 18:32:20 E) PORT: crcs, MSG: #011 Context: Task ID
0x09010001 <br>
Jun 23 18:32:20 E) PORT: crcs, MSG: #011 R0 = FFF5A4DA R1
= 00366720 R2 = 00000000 R3 = FFF0A442 <br>
Jun 23 18:32:20 E) PORT: crcs, MSG: #011 R4 = 00352498 R5
= 00000008 R6 = 00000000 R7 = FFF0A442 <br>
Jun 23 18:32:20 E) PORT: crcs, MSG: #011 R8 = 00352498 R9
= 00000002 R10 = 00000000 R11 = 00000001 <br>
Jun 23 18:32:20 E) PORT: crcs, MSG: #011 R12 = 003BC0E0 R13
= 001E4B90 R14 = 00000000 R15 = 00000000 <br>
Jun 23 18:32:21 E) PORT: crcs, MSG: #011 R16 = 00000000 R17
= 00000000 R18 = 00000000 R19 = 00000000 <br>
Jun 23 18:32:21 E) PORT: crcs, MSG: #011 R20 = 00000000 R21
= 00000000 R22 = 00000000 R23 = 00000000 <br>
Jun 23 18:32:21 E) PORT: crcs, MSG: #011 R24 = 00000000 R25
= 00000000 R26 = 00000000 R27 = 00000000 <br>
Jun 23 18:32:21 E) PORT: crcs, MSG: #011 R28 = FFF0A340 R29
= 00000000 R30 = FFF0A341 R31 = 0034D5B0 <br>
Jun 23 18:32:21 E) PORT: crcs, MSG: #011 CR = 40000004 <br>
Jun 23 18:32:21 E) PORT: crcs, MSG: #011 CTR = 0012E3AC <br>
Jun 23 18:32:21 E) PORT: crcs, MSG: #011 XER = 00000000 <br>
Jun 23 18:32:21 E) PORT: crcs, MSG: #011 LR = 000F2CDC <br>
Jun 23 18:32:21 E) PORT: crcs, MSG: #011 DAR = FFF0A442 <br>
Jun 23 18:32:21 E) PORT: crcs, MSG: Stack Trace: <br>
Jun 23 18:32:21 E) PORT: crcs, MSG: IP: 0x001541E4, LR:
0x000F2CDC <br>
Jun 23 18:32:21 E) PORT: crcs, MSG: --^ 0x00135F6C <br>
Jun 23 18:32:21 E) PORT: crcs, MSG: Suspending faulting
task (0x09010001)</span></blockquote>
<div><br>
</div>
<div class="gmail_default" style="font-size:small"><b>Event
#3:</b></div>
<div class="gmail_default" style="font-size:small">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span style="font-family:monospace"><span style="color:rgb(0,0,0)">Jun 24 07:37:50 E) PORT: crcs,
MSG: *Exception handler called for </span><span style="color:rgb(255,255,255);background-color:rgb(0,0,0)">exception
8</span><span style="color:rgb(0,0,0)"> (0x8) <br>
</span></span><span style="font-family:monospace">Jun 24
07:37:50 E) PORT: crcs, MSG: #011 Next PC or Address of
fault = 0015BB08 <br>
</span><span style="font-family:monospace">Jun 24 07:37:50
E) PORT: crcs, MSG: #011 Saved MSR = 00009032 <br>
</span><span style="font-family:monospace">Jun 24 07:37:50
E) PORT: crcs, MSG: #011 Context: Task ID 0x09010001 <br>
</span><span style="font-family:monospace">Jun 24 07:37:50
E) PORT: crcs, MSG: #011 R0 = 00157754 R1 = 003664B8 R2
= 00000000 R3 = 00366758 <br>
</span><span style="font-family:monospace">Jun 24 07:37:50
E) PORT: crcs, MSG: #011 R4 = 0036661C R5 = 001B4C68 R6
= 00366610 R7 = 00000000 <br>
</span><span style="font-family:monospace">Jun 24 07:37:50
E) PORT: crcs, MSG: #011 R8 = 005F5370 R9 = 00642450
R10 = 0034C940 R11 = 00366688 <br>
</span><span style="font-family:monospace">Jun 24 07:37:50
E) PORT: crcs, MSG: #011 R12 = 40000048 R13 = 001E4B90
R14 = 001B4C68 R15 = 00000000 <br>
</span><span style="font-family:monospace">Jun 24 07:37:51
E) PORT: crcs, MSG: #011 R16 = 00000000 R17 = 00000000
R18 = 00000000 R19 = 00366610 <br>
</span><span style="font-family:monospace">Jun 24 07:37:51
E) PORT: crcs, MSG: #011 R20 = 00000000 R21 = 00000000
R22 = 00366758 R23 = 001B4CC0 <br>
</span><span style="font-family:monospace">Jun 24 07:37:51
E) PORT: crcs, MSG: #011 R24 = 0036661C R25 = 00377E40
R26 = 00000000 R27 = 0000002F <br>
</span><span style="font-family:monospace">Jun 24 07:37:51
E) PORT: crcs, MSG: #011 R28 = 00377E40 R29 = 001E49B0
R30 = 00377E40 R31 = 00000000 <br>
</span><span style="font-family:monospace">Jun 24 07:37:51
E) PORT: crcs, MSG: #011 CR = 40000048 <br>
</span><span style="font-family:monospace">Jun 24 07:37:51
E) PORT: crcs, MSG: #011 CTR = 00000000 <br>
</span><span style="font-family:monospace">Jun 24 07:37:51
E) PORT: crcs, MSG: #011 XER = 00000000 <br>
</span><span style="font-family:monospace">Jun 24 07:37:51
E) PORT: crcs, MSG: #011 LR = 00157754 <br>
</span><span style="font-family:monospace">Jun 24 07:37:51
E) PORT: crcs, MSG: #011 DAR = 00000000 <br>
</span><span style="font-family:monospace">Jun 24 07:37:51
E) PORT: crcs, MSG: Stack Trace: <br>
</span><span style="font-family:monospace">Jun 24 07:37:51
E) PORT: crcs, MSG: IP: 0x0015BB08, LR: 0x00157754 <br>
</span><span style="font-family:monospace">Jun 24 07:37:51
E) PORT: crcs, MSG: --^ 0x00157754--^ 0x000F2820--^
0x000F2DA0--^ 0x001E49BD--^ 0x00135F6C <br>
</span><span style="font-family:monospace">Jun 24 07:37:51
E) PORT: crcs, MSG: Suspending faulting task (0x09010001)
</span></blockquote>
<span style="font-family:monospace">
<br>
</span></div>
<div class="gmail_default" style="font-size:small"><span style="font-family:monospace"><br>
</span></div>
<div class="gmail_default" style="font-size:small"><br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"> <span style="font-family:monospace"><br>
</span></blockquote>
</div>
</blockquote>
<br>
</div></div></div>
</blockquote></div><br></div>