EPICS ioc with RTEMS on Raspberry Pi

Wed Apr 22 20:17:48 UTC 2026

Fascinating. Thanks for that detailed report.

You indicated USB enumerating continues running in the background.

I am educated-guessing that a counter maximum is reached, and then mistakenly breached. I would look for a test of that counter and correct it from allowing greater-than comparing.

Let us know how it turns out.

I am developing an app for Raspberry Pi Zero 2 W (stripped-down 3B+). I wasn't expecting any potential problem like this.

-- John

> On Apr 22, 2026, at 1:25 PM, Miroslaw Dach <miroslaw.dach at gmail.com> wrote:
> 
> 
> Hi All,
> 
> I'm running an EPICS ioc server (EPICS 7.0.10)  with RTEMS 6.2 on a Raspberry Pi 3B+ with rtems-libbsd  (6-freebsd-14) and
> encountering a reproducible system hang after several minutes of operation.
> Through systematic elimination testing I've narrowed the root cause to the
> DWC OTG USB controller driver. I'd appreciate any recommendations on how
> to address this. I can of course use the EPICS ioc server under linux on RPi but just tried to have the RTEMS - Hard real time system which is much more deterministic.
> The boot time for the RPi with EPICS/RTEMS is around 7 sec which is extremely fast!
> 
> Environment
> -----------
> - Board: Raspberry Pi 3B+ (BCM2837, boardrev a020d3)
> - RTEMS: rtems-6.2 (ARM/ARMv4/raspberrypi2)
> - Network stack: rtems-libbsd (RTEMS_BSD_CONFIG_BSP_CONFIG + RTEMS_BSD_CONFIG_INIT)
> - Ethernet: LAN7515 USB Ethernet (muge driver, via DWC OTG)
> - Application: EPICS IOC (but hang occurs with minimal/empty IOC as well)
> - Console: UART serial (/dev/ttyS0)
> 
> Symptom
> -------
> The system boots and runs normally, then the entire system freezes —
> including the UART serial console (which is not USB-dependent). No crash
> message, no stack dump — a hard hang requiring power cycle.
> 
> The time-to-hang depends on the system tick rate:
> - CONFIGURE_MICROSECONDS_PER_TICK=500  (2 kHz): hangs after ~2 minutes
> - CONFIGURE_MICROSECONDS_PER_TICK=10000 (100 Hz): hangs after ~5-6 minutes
> 
> During the hang, the UART becomes completely unresponsive, suggesting a
> kernel-level deadlock or interrupt handler issue rather than an
> application-level problem.
> 
> Elimination testing performed
> -----------------------------
> I systematically disabled components to isolate the cause. All tests below
> used CONFIGURE_MICROSECONDS_PER_TICK=10000 (100 Hz):
> 
> 1. Disabled all EPICS database records (no record processing) -> still hangs
> 2. Disabled periodic NTP sync (no socket operations) -> still hangs
> 3. Disabled IP configuration (no ifconfig, no route, no network traffic,
>    but USB/DWC OTG still initialised via RTEMS_BSD_CONFIG_BSP_CONFIG)
>    -> still hangs (~5-6 min), USB hub enumeration continues in background:
>       ugen1.2: <vendor 0x0424 product 0x2514> at usbus1
>       uhub1 on uhub0
>       ...
>    Console output is garbled by concurrent USB enumeration messages,
>    suggesting interrupt contention.
> 4. Commented out RTEMS_BSD_CONFIG_BSP_CONFIG and the
>    #include <bsp/nexus-devices.h> to prevent DWC OTG initialisation
>    -> STABLE, ran for 14+ minutes with no hang (test stopped manually)
> 
> The libbsd software stack (loopback, sockets, telnetd) continues to
> function in test 4 — only the hardware BSP devices (DWC OTG, muge,
> uhub, ukphy) are excluded.
> 
> Minimal reproduction
> --------------------
> Build an RTEMS 6.2 application for raspberrypi2 BSP with:
> 
>   #define RTEMS_BSD_CONFIG_BSP_CONFIG
>   #define RTEMS_BSD_CONFIG_INIT
>   #include <machine/rtems-bsd-config.h>
>   #include <bsp/nexus-devices.h>     /* in one translation unit */
> 
>   #define CONFIGURE_MICROSECONDS_PER_TICK 10000
> 
> The application does not need to configure any network interface or
> perform any USB transfers — the DWC OTG hub polling alone triggers
> the hang after ~5-6 minutes.
> 
> Commenting out RTEMS_BSD_CONFIG_BSP_CONFIG and the nexus-devices.h
> include eliminates the hang.
> 
> Boot log (abbreviated, from hanging configuration)
> --------------------------------------------------
> RTEMS RPi 3B+ 1.3 (1GB) [00a020d3]
> nexus0: <RTEMS Nexus device>
> dwcotg0: <DWC OTG 2.0 integrated USB controller> on nexus0
> usbus1 on dwcotg0
> usbus1: 480Mbps High Speed USB v2.0
> ugen1.1: <DWCOTG OTG Root HUB> at usbus1
> uhub0 on usbus1
> uhub0: <DWCOTG OTG Root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
> uhub0: 1 port with 1 removable, self powered
> ugen1.2: <vendor 0x0424 product 0x2514> at usbus1
> uhub1 on uhub0
> uhub1: <vendor 0x0424 product 0x2514, class 9/0, rev 2.00/b.b3, addr 2> on usbus1
> uhub1: 4 ports with 3 removable, self powered
> ugen1.3: <vendor 0x0424 product 0x2514> at usbus1
> uhub2 on uhub1
> uhub2: <vendor 0x0424 product 0x2514, class 9/0, rev 2.00/b.b3, addr 3> on usbus1
> uhub2: 3 ports with 2 removable, self powered
> ugen1.4: <vendor 0x0424 product 0x7800> at usbus1
> muge0: <vendor 0x0424 product 0x7800, rev 2.10/3.00, addr 4> on usbus1
> muge0: Chip ID 0x7800 rev 0002
> miibus0: <MII bus> on muge0
> ukphy0: <Generic IEEE 802.3u media interface> PHY 1 on miibus0
> info: ue0: <USB Ethernet> on muge0
> [system hangs after ~5-6 minutes, UART unresponsive]
> 
> Questions
> ---------
> 1. Is this a known issue with the DWC OTG driver on RPi 3B+?
> 2. Are there any configuration options (hub polling interval, interrupt
>    coalescing, DMA settings) that might work around the problem?
> 3. Would a newer version of rtems-libbsd contain fixes for this?
> 4. Is there an alternative Ethernet driver approach for RPi 3B+ that
>    avoids the DWC OTG USB path?
> 5. Is there any known project which uses RPi with RTEMS?
> (it looks like that the option to run RTEMS on RPi4 or RPi 5 can not be considered since the BSP in RTEMS kernel  is not yet finalised.
> The RPi4 or RPi 5 would be much better candidates vs RPi 3 B+ since they use direct connection to Ethernet instead of the USB-Ethernet) 
> 
> Thank you for any guidance.
> 
> Mirek
> 
> 
> _______________________________________________
> users mailing list
> users at rtems.org
> http://lists.rtems.org/mailman/listinfo/users