Beatnik RTEMS4.10.2 vmeTsi148 ISR: ERROR: no handler registered

[Matt Rippa]

Hi Andrew,

Thanks for your email. It helped me learn a little more. We have a
bancomm635 timing board installed, physically.
However we've stopped initializing it some time ago in our startup. So the
interrupt handler isn't configured as you pointed out. I know what to do to
prevent this from happening again. I think the consequence of this is
extreme (our system stopped working).

I haven't tracked down vector 0x45 yet. But we also have 2 GreenSprings
IPAC carriers for our canbus cards that can generate interrupts. These
isr's should all be configured. This uses the drvTip810 driver.
https://github.com/epics-modules/ipac/tree/2.14

We can run the t810Report 3 if we see this happen again.

-Matt

$ egrep -rn --include="*.c" BC635VEC bancomm
bancomm/bancommApp/src/drvBc635.c:121:#define BC635VEC    0x40
bancomm/bancommApp/src/drvBc635.c:702:        if( (status =
devConnectInterruptVME(BC635VEC, isr_bc635, NULL) ) != OK )
bancomm/bancommApp/src/drvBc635.c:707:        pbc635->vector = BC635VEC;
     /* Interrupt vector */



On Thu, Mar 11, 2021 at 5:59 PM Johnson, Andrew N. <anj at anl.gov> wrote:

> Hi Matt,
>
> You appear to have had 2 separate unconfigured interrupts, which triggered
> about 5 minutes apart. Editing the log slightly to make that clearer:
>
> > <8/3/2021 8:00:34 hst>vmeTsi148 ISR: ERROR: no handler registered (level
> 2) IACK 0x00000045 -- DISABLING level 2
> > <8/3/2021 8:05:14 hst>vmeTsi148 ISR: ERROR: no handler registered (level
> 4) IACK 0x00000040 -- DISABLING level 4
>
>
> I would guess that those IACK numbers 0x45 and 0x40 are VME interrupt
> vectors, so they probably came from one or more VME cards in the crate. My
> VxWorks BSPs have routines that let me see the complete interrupt vector
> table, but I don’t know if the Beatnik BSP has anything similar. You should
> have a record of what vectors are used for each VME card that can generate
> interrupts, so check whether those interrupts should have registered
> handlers or not.
>
> It’s also possible that these interrupts might have been generated by the
> Tsi148 chip itself. In many VxWorks VME BSPs the on-board drivers route
> their interrupts through the same vector table even though there’s no
> specific need to do it that way. There are VMEbus SYSFail and PowerFail
> interrupts for example that the Tempe (Tsi148) chip can generate, although
> none of the chip interrupts should have been enabled without registering a
> handler for them first. SYSfail can be asserted by another VME board so
> that wouldn’t give you a definitive answer if it turned out to be that.
>
> Hope this gives you some clues,
>
> - Andrew
>
>
>
> On Mar 11, 2021, at 9:35 PM, Matt Rippa via Tech-talk <
> tech-talk at aps.anl.gov> wrote:
> >
> > It looks like the IRQEntry vector triggering this interrupt was null.
> I'm not sure what condition would arise leading to this.
> > Maybe memory corruption or a bus error.
> >
> >
> https://git.rtems.org/rtems/tree/c/src/lib/libbsp/shared/vmeUniverse/vmeTsi148.c?h=4.10.2#n1562
> >
> > On Thu, Mar 11, 2021 at 2:42 PM Matt Rippa <mrippa at gemini.edu> wrote:
> > Hello,
> >
> > Our Primary Mirror control system has a beatnik bsp running on a
> mvme6100 with RTEMS4.10.2 and EPICS 3.14.12.8 (which understandably is not
> actively supported). We've experienced a single event that we've never seen
> before. vmeTsi148 ISR: ERROR: no handler
> >
> > After this it appears all interrupts were disabled and the VME bus
> communications are down. This system is normally well behaved with uptimes
> measured in months. In 2020 we had an uptime of 301 days!
> >
> > Has anyone seen an error like this on the beatnik bsp?
> >
> > Subsequent to the error which corresponds closely with a casr command,
> we see dbScan warnings in our logs. These persisted several times per
> minute until the system was reboot.
> >
> > Thanks for any insight.
> >
> > -Matt Rippa
> >
> >
> > <8/3/2021 8:00:32 hst>pcs-mk-ioc> casr
> > <8/3/2021 8:00:33 hst>Channel Access Server V4.13
> > <8/3/2021 8:00:33 hst>Connected circuits:
> > <8/3/2021 8:00:33 hst>TCP xxxxxxxxxxxxxxxxxxxx, V4.13, 6 Channels,
> Priority=0
> > <8/3/2021 8:00:33 hst>TCP xxxxxxxxxxxxxxxxxxxx, V4.13, 134 Channels,
> Priority=80
> > <8/3/2021 8:00:33 hst>TCP , V4.13, 1 Channels, Priority=0
> > <8/3/2021 8:00:33 hst>TCP , V4.13, 5 Channels, Priority=0
> > <8/3/2021 8:00:33 hst>TCP , V4.13, 8 Channels, Priority=0
> > <8/3/2021 8:00:33 hst>TCP , V4.13, 60 Channels, Priority=0
> > <8/3/2021 8:00:33 hst>TCP , V4.13, 9 Channels, Priority=0
> > <8/3/2021 8:00:34 hst>TCP , V4.13, 9 Channels, Priority=0
> > <8/3/2021 8:00:34 hst>TCP , V4.13, 71 Channels, Priority=0
> > <8/3/2021 8:00:34 hst>TCP , V4.13, 5 Channels, Priority=0
> > <8/3/2021 8:00:34 hst>TCP , V4.13, 359 Channels, Priority=20
> > <8/3/2021 8:00:34 hst>TCP , 741 Channels, Priority=20
> > <8/3/2021 8:00:34 hst>TCP , V4.13, 5 Channels, Priority=20
> > <8/3/2021 8:00:34 hst>pcs-mk-ioc> vmeTsi148 ISR: ERROR: no handler
> registered (level 2) IACK 0x00000045 -- DISABLING level 2
> > <8/3/2021 8:05:14 hst>vmeTsi148 ISR: ERROR: no handler registered (level
> 4) IACK 0x00000040 -- DISABLING level 4
> > <8/3/2021 8:34:54 hst>dbScan warning from '.2 second' scan thread:
> > <8/3/2021 8:34:54 hst>        Scan processing averages 0.300 seconds
> (0.300 .. 0.300).
> > <8/3/2021 8:34:54 hst>        Over-runs have now happened 10 times in a
> row.
> > <8/3/2021 8:34:54 hst>        To fix this, move some records to a slower
> scan rate.
> > <8/3/2021 8:34:54 hst>
> > <8/3/2021 8:35:09 hst>dbScan warning from '.2 second' scan thread:
> > <8/3/2021 8:35:09 hst>        Scan processing averages 0.300 seconds
> (0.300 .. 0.300).
> > <8/3/2021 8:35:09 hst>        Over-runs have now happened 10 times in a
> row.
> > <8/3/2021 8:35:09 hst>        To fix this, move some records to a slower
> scan rate.
> > <8/3/2021 8:35:09 hst>
> > ...
> >
>
> --
> Complexity comes for free, simplicity you have to work for.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rtems.org/pipermail/users/attachments/20210312/3de9f8de/attachment.html>