<div dir="ltr">Hi John,<br><br>Thanks for your e-mail and your suggestion.<div><br>Follow-up to my earlier report about the DWC OTG USB hang on RPi 3B+<br>with RTEMS 6. I've done extensive instrumentation and applied BCM2835-<br>specific errata workarounds from the Linux dwc2 driver. Here are the<br>complete findings:<br><br>I added diagnostic counters (#ifdef __rtems__) to the FreeBSD dwc_otg<br>driver, printed every 1 second from the 10ms timer callback:<br><br>- yld: times dwc_otg_interrupt_poll_locked() hit its 16-iteration cap<br>- afail: times dwc_otg_host_channel_alloc() failed (no free channels)<br>- hok: successful halt completions processed<br>- rxd: RX data discarded (no active endpoint listening)<br>- stuck: channels with wait_halted=1 but allocated=0 (leaked channels)<br>- filt: filter ISR entry/exit count (to detect stuck ISR)<br>- thr: thread handler entry/exit count<br>- gi: last GINTSTS register value<br><br>Results: channel exhaustion theory DISPROVED<br>Across all tests, every counter remained perfectly healthy right up to<br>the instant of the freeze:<br><br> DWC_OTG t=17500: yld=0 afail=0 hok=180038 rxd=0 ch=8/16<br> alloc=0 wh=0 stuck=0 filt=738178/738178 thr=7280/7280<br> gi=0x04000031<br><br>- yld=0: the 16-iteration poll loop cap was NEVER hit<br>- afail=0: channel allocation NEVER failed<br>- stuck=0: NO channels ever leaked (wait_halted was always cleared)<br>- filt balanced: filter ISR entered and exited the same number of times<br>- thr balanced: thread handler entered and exited the same number of times<br>- gi=0x04000031: benign (host mode + TX FIFOs empty + RX FIFO non-empty)<br><br>The ~1054 halt completions/second were steady with no degradation.<br><br>Results: total CPU freeze, not software deadlock<br><br>I also tried an independent RTEMS timer (rtems_timer_fire_after) as a<br>heartbeat, firing outside the USB bus lock. Both the USB timer and the<br>independent heartbeat stopped simultaneously, confirming the entire ARM<br>core freezes — not just the USB subsystem.<br><br>No RTEMS fatal error handler was triggered (I installed one via<br>CONFIGURE_INITIAL_EXTENSIONS). This means it is NOT a standard ARM<br>data abort — the CPU simply stops executing, likely due to an AHB bus<br>lockup caused by the DWC OTG controller hardware.<br><br>BCM2835 errata workarounds applied<br><br>Based on the Linux dwc2 driver (params.c, core.c) and the Ultibo<br>project documentation, I applied three workarounds:<br><br>1. FIFO size cap: BCM2835 hardware reports 4096 words but only 4080<br> exist. Cap sc_fifo_size to 4080*4 bytes to prevent FIFO overrun.<br> (In practice, the hardware on my board reported <= 4080, so this<br> cap was NOT triggered.)<br><br>2. GAHBCFG AHB burst configuration: Broadcom redefined bits [4:1] of<br> GAHBCFG for AXI burst control. Linux dwc2 sets ahbcfg=0x10 for<br> BCM2835. Changed from GAHBCFG_GLBLINTRMSK (0x01) to 0x11.<br><br>3. AHB idle wait after core reset: Added a loop waiting for<br> GRSTCTL_AHBIDLE after GRSTCTL_CSFTRST, plus 250ms settling delay<br> (Linux dwc2 uses 100+ ms).<br><br>Results with workarounds:<br>- Before: hangs after ~86 seconds (2 kHz tick)<br>- After: hangs after ~175 seconds (2 kHz tick)<br><br>The workarounds approximately doubled the time-to-hang but did NOT<br>eliminate it. All software counters remained healthy throughout.<br><br>The hang is caused most probably by a combination of factors:<br><br>1. The DWC OTG controller on BCM2835/BCM2837 requires sub-125µs<br> interrupt response times for USB split transaction phases (Start<br> Split → Complete Split through the USB hub). The RPi 3B+ Ethernet<br> goes through a USB hub (LAN7515), making every packet a split<br> transaction.<br><br>2. The Linux kernel addresses this with a dedicated FIQ (Fast Interrupt<br> Request) handler (dwc_otg_fiq_fsm.c) that executes complete split<br> transactions in FIQ context, bypassing the normal interrupt stack.<br> Without FIQ, "certain USB devices become completely unusable."<br><br>3 The FreeBSD dwc_otg driver used by RTEMS handles all split<br> transactions in normal interrupt context. On RTEMS, the interrupt<br> filter and thread handler run back-to-back in the interrupt server<br> task (nexus_intr_with_filter in rtems-kernel-nexus.c), with no<br> preemption point between them.<br><br>4 When the controller's split transaction timing is violated, it<br> enters an unrecoverable state that locks the AHB bus, freezing the<br> entire ARM core including UART and system timers.<br><br>The timing correlation with tick rate confirms this: higher tick rate<br>= more frequent scheduling = more interrupt latency jitter = faster<br>timing violation.</div><div><br>A proper fix requires most probably implementing FIQ-based split transaction handling<br>in the RTEMS BSP for BCM2835/BCM2837, similar to what Linux does in<br>dwc_otg_fiq_fsm.c. This is a significant undertaking but is essential<br>for reliable USB operation on RPi 3B+ (and RPi Zero 2 W, which uses<br>the same SoC).<br><br>The GAHBCFG and reset sequence workarounds should also be applied as<br>they improve stability.<br><br>Please give me your thoughts on that. Maybe it is easier to finalise the BSP for RPi 4 and 5?<br><br>Best Regards<br>Mirek<br></div></div><br><div class="gmail_quote gmail_quote_container"><div dir="ltr" class="gmail_attr">śr., 22 kwi 2026 o 13:17 John Howard <<a href="mailto:echosoft.llc@gmail.com">echosoft.llc@gmail.com</a>> napisał(a):<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Fascinating. Thanks for that detailed report.<br>
<br>
You indicated USB enumerating continues running in the background.<br>
<br>
I am educated-guessing that a counter maximum is reached, and then mistakenly breached. I would look for a test of that counter and correct it from allowing greater-than comparing.<br>
<br>
Let us know how it turns out.<br>
<br>
I am developing an app for Raspberry Pi Zero 2 W (stripped-down 3B+). I wasn't expecting any potential problem like this.<br>
<br>
-- John<br>
<br>
> On Apr 22, 2026, at 1:25 PM, Miroslaw Dach <<a href="mailto:miroslaw.dach@gmail.com" target="_blank">miroslaw.dach@gmail.com</a>> wrote:<br>
> <br>
> <br>
> Hi All,<br>
> <br>
> I'm running an EPICS ioc server (EPICS 7.0.10) with RTEMS 6.2 on a Raspberry Pi 3B+ with rtems-libbsd (6-freebsd-14) and<br>
> encountering a reproducible system hang after several minutes of operation.<br>
> Through systematic elimination testing I've narrowed the root cause to the<br>
> DWC OTG USB controller driver. I'd appreciate any recommendations on how<br>
> to address this. I can of course use the EPICS ioc server under linux on RPi but just tried to have the RTEMS - Hard real time system which is much more deterministic.<br>
> The boot time for the RPi with EPICS/RTEMS is around 7 sec which is extremely fast!<br>
> <br>
> Environment<br>
> -----------<br>
> - Board: Raspberry Pi 3B+ (BCM2837, boardrev a020d3)<br>
> - RTEMS: rtems-6.2 (ARM/ARMv4/raspberrypi2)<br>
> - Network stack: rtems-libbsd (RTEMS_BSD_CONFIG_BSP_CONFIG + RTEMS_BSD_CONFIG_INIT)<br>
> - Ethernet: LAN7515 USB Ethernet (muge driver, via DWC OTG)<br>
> - Application: EPICS IOC (but hang occurs with minimal/empty IOC as well)<br>
> - Console: UART serial (/dev/ttyS0)<br>
> <br>
> Symptom<br>
> -------<br>
> The system boots and runs normally, then the entire system freezes —<br>
> including the UART serial console (which is not USB-dependent). No crash<br>
> message, no stack dump — a hard hang requiring power cycle.<br>
> <br>
> The time-to-hang depends on the system tick rate:<br>
> - CONFIGURE_MICROSECONDS_PER_TICK=500 (2 kHz): hangs after ~2 minutes<br>
> - CONFIGURE_MICROSECONDS_PER_TICK=10000 (100 Hz): hangs after ~5-6 minutes<br>
> <br>
> During the hang, the UART becomes completely unresponsive, suggesting a<br>
> kernel-level deadlock or interrupt handler issue rather than an<br>
> application-level problem.<br>
> <br>
> Elimination testing performed<br>
> -----------------------------<br>
> I systematically disabled components to isolate the cause. All tests below<br>
> used CONFIGURE_MICROSECONDS_PER_TICK=10000 (100 Hz):<br>
> <br>
> 1. Disabled all EPICS database records (no record processing) -> still hangs<br>
> 2. Disabled periodic NTP sync (no socket operations) -> still hangs<br>
> 3. Disabled IP configuration (no ifconfig, no route, no network traffic,<br>
> but USB/DWC OTG still initialised via RTEMS_BSD_CONFIG_BSP_CONFIG)<br>
> -> still hangs (~5-6 min), USB hub enumeration continues in background:<br>
> ugen1.2: <vendor 0x0424 product 0x2514> at usbus1<br>
> uhub1 on uhub0<br>
> ...<br>
> Console output is garbled by concurrent USB enumeration messages,<br>
> suggesting interrupt contention.<br>
> 4. Commented out RTEMS_BSD_CONFIG_BSP_CONFIG and the<br>
> #include <bsp/nexus-devices.h> to prevent DWC OTG initialisation<br>
> -> STABLE, ran for 14+ minutes with no hang (test stopped manually)<br>
> <br>
> The libbsd software stack (loopback, sockets, telnetd) continues to<br>
> function in test 4 — only the hardware BSP devices (DWC OTG, muge,<br>
> uhub, ukphy) are excluded.<br>
> <br>
> Minimal reproduction<br>
> --------------------<br>
> Build an RTEMS 6.2 application for raspberrypi2 BSP with:<br>
> <br>
> #define RTEMS_BSD_CONFIG_BSP_CONFIG<br>
> #define RTEMS_BSD_CONFIG_INIT<br>
> #include <machine/rtems-bsd-config.h><br>
> #include <bsp/nexus-devices.h> /* in one translation unit */<br>
> <br>
> #define CONFIGURE_MICROSECONDS_PER_TICK 10000<br>
> <br>
> The application does not need to configure any network interface or<br>
> perform any USB transfers — the DWC OTG hub polling alone triggers<br>
> the hang after ~5-6 minutes.<br>
> <br>
> Commenting out RTEMS_BSD_CONFIG_BSP_CONFIG and the nexus-devices.h<br>
> include eliminates the hang.<br>
> <br>
> Boot log (abbreviated, from hanging configuration)<br>
> --------------------------------------------------<br>
> RTEMS RPi 3B+ 1.3 (1GB) [00a020d3]<br>
> nexus0: <RTEMS Nexus device><br>
> dwcotg0: <DWC OTG 2.0 integrated USB controller> on nexus0<br>
> usbus1 on dwcotg0<br>
> usbus1: 480Mbps High Speed USB v2.0<br>
> ugen1.1: <DWCOTG OTG Root HUB> at usbus1<br>
> uhub0 on usbus1<br>
> uhub0: <DWCOTG OTG Root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1<br>
> uhub0: 1 port with 1 removable, self powered<br>
> ugen1.2: <vendor 0x0424 product 0x2514> at usbus1<br>
> uhub1 on uhub0<br>
> uhub1: <vendor 0x0424 product 0x2514, class 9/0, rev 2.00/b.b3, addr 2> on usbus1<br>
> uhub1: 4 ports with 3 removable, self powered<br>
> ugen1.3: <vendor 0x0424 product 0x2514> at usbus1<br>
> uhub2 on uhub1<br>
> uhub2: <vendor 0x0424 product 0x2514, class 9/0, rev 2.00/b.b3, addr 3> on usbus1<br>
> uhub2: 3 ports with 2 removable, self powered<br>
> ugen1.4: <vendor 0x0424 product 0x7800> at usbus1<br>
> muge0: <vendor 0x0424 product 0x7800, rev 2.10/3.00, addr 4> on usbus1<br>
> muge0: Chip ID 0x7800 rev 0002<br>
> miibus0: <MII bus> on muge0<br>
> ukphy0: <Generic IEEE 802.3u media interface> PHY 1 on miibus0<br>
> info: ue0: <USB Ethernet> on muge0<br>
> [system hangs after ~5-6 minutes, UART unresponsive]<br>
> <br>
> Questions<br>
> ---------<br>
> 1. Is this a known issue with the DWC OTG driver on RPi 3B+?<br>
> 2. Are there any configuration options (hub polling interval, interrupt<br>
> coalescing, DMA settings) that might work around the problem?<br>
> 3. Would a newer version of rtems-libbsd contain fixes for this?<br>
> 4. Is there an alternative Ethernet driver approach for RPi 3B+ that<br>
> avoids the DWC OTG USB path?<br>
> 5. Is there any known project which uses RPi with RTEMS?<br>
> (it looks like that the option to run RTEMS on RPi4 or RPi 5 can not be considered since the BSP in RTEMS kernel is not yet finalised.<br>
> The RPi4 or RPi 5 would be much better candidates vs RPi 3 B+ since they use direct connection to Ethernet instead of the USB-Ethernet) <br>
> <br>
> Thank you for any guidance.<br>
> <br>
> Mirek<br>
> <br>
> <br>
> _______________________________________________<br>
> users mailing list<br>
> <a href="mailto:users@rtems.org" target="_blank">users@rtems.org</a><br>
> <a href="http://lists.rtems.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.rtems.org/mailman/listinfo/users</a><br>
</blockquote></div>