Beatnik RTEMS4.10.2 vmeTsi148 ISR: ERROR: no handler registered

[Johnson, Andrew N.]

Hi Matt,

You appear to have had 2 separate unconfigured interrupts, which triggered about 5 minutes apart. Editing the log slightly to make that clearer:

> <8/3/2021 8:00:34 hst>vmeTsi148 ISR: ERROR: no handler registered (level 2) IACK 0x00000045 -- DISABLING level 2 
> <8/3/2021 8:05:14 hst>vmeTsi148 ISR: ERROR: no handler registered (level 4) IACK 0x00000040 -- DISABLING level 4


I would guess that those IACK numbers 0x45 and 0x40 are VME interrupt vectors, so they probably came from one or more VME cards in the crate. My VxWorks BSPs have routines that let me see the complete interrupt vector table, but I don’t know if the Beatnik BSP has anything similar. You should have a record of what vectors are used for each VME card that can generate interrupts, so check whether those interrupts should have registered handlers or not.

It’s also possible that these interrupts might have been generated by the Tsi148 chip itself. In many VxWorks VME BSPs the on-board drivers route their interrupts through the same vector table even though there’s no specific need to do it that way. There are VMEbus SYSFail and PowerFail interrupts for example that the Tempe (Tsi148) chip can generate, although none of the chip interrupts should have been enabled without registering a handler for them first. SYSfail can be asserted by another VME board so that wouldn’t give you a definitive answer if it turned out to be that.

Hope this gives you some clues,

- Andrew



On Mar 11, 2021, at 9:35 PM, Matt Rippa via Tech-talk <tech-talk at aps.anl.gov> wrote:
> 
> It looks like the IRQEntry vector triggering this interrupt was null. I'm not sure what condition would arise leading to this.
> Maybe memory corruption or a bus error.
> 
> https://git.rtems.org/rtems/tree/c/src/lib/libbsp/shared/vmeUniverse/vmeTsi148.c?h=4.10.2#n1562
> 
> On Thu, Mar 11, 2021 at 2:42 PM Matt Rippa <mrippa at gemini.edu> wrote:
> Hello,
> 
> Our Primary Mirror control system has a beatnik bsp running on a mvme6100 with RTEMS4.10.2 and EPICS 3.14.12.8 (which understandably is not actively supported). We've experienced a single event that we've never seen before. vmeTsi148 ISR: ERROR: no handler 
> 
> After this it appears all interrupts were disabled and the VME bus communications are down. This system is normally well behaved with uptimes measured in months. In 2020 we had an uptime of 301 days!
> 
> Has anyone seen an error like this on the beatnik bsp?
> 
> Subsequent to the error which corresponds closely with a casr command, we see dbScan warnings in our logs. These persisted several times per minute until the system was reboot.
> 
> Thanks for any insight.
> 
> -Matt Rippa 
> 
> 
> <8/3/2021 8:00:32 hst>pcs-mk-ioc> casr 
> <8/3/2021 8:00:33 hst>Channel Access Server V4.13 
> <8/3/2021 8:00:33 hst>Connected circuits: 
> <8/3/2021 8:00:33 hst>TCP xxxxxxxxxxxxxxxxxxxx, V4.13, 6 Channels, Priority=0 
> <8/3/2021 8:00:33 hst>TCP xxxxxxxxxxxxxxxxxxxx, V4.13, 134 Channels, Priority=80
> <8/3/2021 8:00:33 hst>TCP , V4.13, 1 Channels, Priority=0 
> <8/3/2021 8:00:33 hst>TCP , V4.13, 5 Channels, Priority=0 
> <8/3/2021 8:00:33 hst>TCP , V4.13, 8 Channels, Priority=0 
> <8/3/2021 8:00:33 hst>TCP , V4.13, 60 Channels, Priority=0 
> <8/3/2021 8:00:33 hst>TCP , V4.13, 9 Channels, Priority=0 
> <8/3/2021 8:00:34 hst>TCP , V4.13, 9 Channels, Priority=0 
> <8/3/2021 8:00:34 hst>TCP , V4.13, 71 Channels, Priority=0 
> <8/3/2021 8:00:34 hst>TCP , V4.13, 5 Channels, Priority=0 
> <8/3/2021 8:00:34 hst>TCP , V4.13, 359 Channels, Priority=20 
> <8/3/2021 8:00:34 hst>TCP , 741 Channels, Priority=20 
> <8/3/2021 8:00:34 hst>TCP , V4.13, 5 Channels, Priority=20 
> <8/3/2021 8:00:34 hst>pcs-mk-ioc> vmeTsi148 ISR: ERROR: no handler registered (level 2) IACK 0x00000045 -- DISABLING level 2 
> <8/3/2021 8:05:14 hst>vmeTsi148 ISR: ERROR: no handler registered (level 4) IACK 0x00000040 -- DISABLING level 4
> <8/3/2021 8:34:54 hst>dbScan warning from '.2 second' scan thread: 
> <8/3/2021 8:34:54 hst>        Scan processing averages 0.300 seconds (0.300 .. 0.300). 
> <8/3/2021 8:34:54 hst>        Over-runs have now happened 10 times in a row. 
> <8/3/2021 8:34:54 hst>        To fix this, move some records to a slower scan rate. 
> <8/3/2021 8:34:54 hst> 
> <8/3/2021 8:35:09 hst>dbScan warning from '.2 second' scan thread: 
> <8/3/2021 8:35:09 hst>        Scan processing averages 0.300 seconds (0.300 .. 0.300). 
> <8/3/2021 8:35:09 hst>        Over-runs have now happened 10 times in a row. 
> <8/3/2021 8:35:09 hst>        To fix this, move some records to a slower scan rate. 
> <8/3/2021 8:35:09 hst>
> ...
> 

-- 
Complexity comes for free, simplicity you have to work for.