<div dir="ltr">Hi Andrei,<div>thank you for the elaborated answer !</div><div><br></div><div>I checked my STM32 Cube settings, I have 3 enabled interrupts and they all have the preemption priority/sub priority set to zero !</div><div>it seems I ran into the same issue you had in 2015 :)</div><div>I will take your advice - change the priorities, regenerate the code and see what happens.</div><div><br></div><div>regards,</div><div>Catalin</div><div><br><div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Apr 3, 2019 at 6:03 PM <<a href="mailto:groups@chichak.ca">groups@chichak.ca</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;">This sounds like a problem I had in 2015 on an STM32 that Sebastian helped me get around. At the end of the ordeal I wrote:<div><br></div><div>"<span style="font-family:SabonRoman">A bit of review to begin with; I am working with an STM32F4 ARM Cortex M4F processor’s ADC section. A feature of this ADC is the ability to have conversions triggered by a timer (great for evenly sampled signals), the results transferred using double buffered DMA, giving you an interrupt when the DMA buffer is half full, then again when the buffer is full.</span></div><br style="font-family:SabonRoman"><span style="font-family:SabonRoman">To let my task know when there was data ready to process, the DMA half/full complete interrupt routines would call rtems_event_send. The task would pend on the events, with a timeout in case something screwed up.</span><br style="font-family:SabonRoman"><br style="font-family:SabonRoman"><span style="font-family:SabonRoman">In my case, the timer would trigger 14 channels of ADC conversions to happen 400 times per second. This would yield 200 half full and 200 full interrupts per second, each calling rtems_event_send.</span><br style="font-family:SabonRoman"><br style="font-family:SabonRoman"><span style="font-family:SabonRoman">This action would proceed for a few thousand seconds and then the program would crash and, doing some painful debugging, I managed to repeatedly catch the system attempting to expire some “watchdogs”, which I believe is the squelching of outstanding timeouts on satisfied rtems_event_receive calls.</span><br style="font-family:SabonRoman"><br style="font-family:SabonRoman"><br style="font-family:SabonRoman"><br style="font-family:SabonRoman"><span style="font-family:SabonRoman">After trying a whole bunch of dead ends, Sebastian Huber asked me about the priorities of the interrupts being generated. </span><br style="font-family:SabonRoman"><br style="font-family:SabonRoman"><span style="font-family:SabonRoman">The ARM architecture uses a vectored interrupt structure quite similar to the MC68xxx processors, where a device generates an interrupt and the address of the service routine is automatically picked up from a known place in a table without having to poll a bunch of registers to figure out what happened and branch off to the handler. The ARM processors have assignable priorities on most of the interrupts, so if two interrupts assert at the same time, or if a higher priority interrupt happens while an interrupt is in progress, you can predict what happens.</span><br style="font-family:SabonRoman"><br style="font-family:SabonRoman"><span style="font-family:SabonRoman">What I didn’t know is that RTEMS implements something called Non-Maskable Interrupts (NMI). The software NMIs don’t seem to be like hardware NMIs (a hardware interrupt that can not be turned off), they just have the same name (much like the event watchdogs that aren’t like the hardware watchdogs).</span><br style="font-family:SabonRoman"><br style="font-family:SabonRoman"><span style="font-family:SabonRoman">What I learned was that RTEMS NMIs are interrupt routines that are not allowed to use any RTEMS facilities. So, I presume, these routines would be used for dealing with devices that don’t need to interact with task code. The upside is that the interrupts can be entered bypassing RTEMS’ overhead.</span><br style="font-family:SabonRoman"><br style="font-family:SabonRoman"><span style="font-family:SabonRoman">A drawback is that if you call for RTEMS facilities from within one of these routines, apparently, your code becomes rather crashy.</span><br style="font-family:SabonRoman"><br style="font-family:SabonRoman"><br style="font-family:SabonRoman"><br style="font-family:SabonRoman"><br style="font-family:SabonRoman"><span style="font-family:SabonRoman">To differentiate between NMI routines and a regular ISR that can call RTEMS facilities, the developers use the interrupt priorities and a mask. The NMI determination is not specific to the ARM family, each architecture has a mask that determines which bits are used to determine if an interrupt routine is an NMI or an ISR.</span><br style="font-family:SabonRoman"><br style="font-family:SabonRoman"><br style="font-family:SabonRoman"><span style="font-family:SabonRoman">ARM uses an 8 bit priority and a priority in the range of 0x00-0x7F indicates an NMI. On an ARM, the lower the number, the more urgent the interrupt, so NMIs have higher urgency than ISRs that can use RTEMS facilities.</span><br style="font-family:SabonRoman"><br style="font-family:SabonRoman"><br style="font-family:SabonRoman"><span style="font-family:SabonRoman">On the STM32F4, only 4 bits of 8 of priority are implemented, the 4 MSBs with the lower 4 being set to 0 (other Cortex M4 implementations have other combinations). In ST’s CubeMX tool, you can set the interrupt priority of the various interrupt sources in the range of 0-15 and Cube generates code to take care of the bit shifting for you. In my case I had set my priorities to 1,2,3 and 6. Shifted, these became 0x10, 0x20, 0x30, and 0x60. Since these numbers are all below 0x80, the RTEMS code was interpreting these interrupts as NMIs, bypassing a bunch of the necessary code to support RTEMS calls.</span><br style="font-family:SabonRoman"><br style="font-family:SabonRoman"><span style="font-family:SabonRoman">By changing my interrupt priorities to 9, 10, 11, and 14 (shifting gives 0x90, 0xA0, 0xB0, and 0xE0), the interrupt routines lost their NMI nature and the system immediately became dead stable with a 1kHz tick interrupt rate, 2 ADC DMA interrupts at 200Hz each, and a CAN interrupt at about 36Hz.</span><font face="SabonRoman">”</font><div><font face="SabonRoman"><br></font></div><div><br></div><div><br></div><div>When I ported RTEMS5 to the STM32F7, I ran into the same issue and used the same method to get around it.</div><div><br></div><div>I hope this helps.</div><div><br></div><div>Andrei</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><div><div><blockquote type="cite"><div>On 2019-April-03, at 07:46, Sebastian Huber <<a href="mailto:sebastian.huber@embedded-brains.de" target="_blank">sebastian.huber@embedded-brains.de</a>> wrote:</div><br class="gmail-m_718148936417713759Apple-interchange-newline"><div><div>On 03/04/2019 15:41, Catalin Demergian wrote:<br><blockquote type="cite">yes, I realized yesterday evening that gIntrErrs could be incremented in the second if.<br>so I rewrote it like this<br><br>int gIntrptErrs;<br>int gInsertErrs;<br><br>RTEMS_INLINE_ROUTINE void _Scheduler_priority_Ready_queue_enqueue(<br> Chain_Node *node,<br> Scheduler_priority_Ready_queue *ready_queue,<br> Priority_bit_map_Control *bit_map<br>)<br>{<br> Chain_Control *ready_chain = ready_queue->ready_chain;<br> //_Assert(_ISR_Get_level() != 0);<br> if(_ISR_Get_level() == 0)<br>gIntrptErrs++;<br><br> cnt_before = _Chain_Node_count_unprotected(ready_chain);<br> _Chain_Append/*_unprotected*/( ready_chain, node );<br> cnt_after = _Chain_Node_count_unprotected(ready_chain);<br><br> if(cnt_after != cnt_before + 1)<br>gInsertErrs++;<br><br> _Priority_bit_map_Add( bit_map, &ready_queue->Priority_map );<br>}<br><br>It didn't seem that we enter that code with interrupts enabled .. output was<br># cpuuse<br>-------------------------------------------------------------------------------<br> CPU USAGE BY THREAD<br>------------+----------------------------------------+---------------+---------<br> ID | NAME | SECONDS | PERCENT<br>------------+----------------------------------------+---------------+---------<br>*cdemergian build 11.15 gIntrptErrs=0 gInsertErrs=2*<br> 0x09010001 | IDLE | 244.595117 | 99.238<br> 0x0a010001 | UI1 | 1.000929 | 0.406<br> 0x0a010002 | ntwk | 0.099342 | 0.040<br> 0x0a010003 | SCtx | 0.068705 | 0.027<br> 0x0a010004 | SCrx | 0.089272 | 0.036<br> 0x0a010005 | eRPC | 0.000050 | 0.000<br> 0x0a010006 | SHLL | 0.550608 | 0.223<br> 0x0b010001 | | 0.000096 | 0.000<br> 0x0b010002 | | 0.068307 | 0.027<br>------------+----------------------------------------+---------------+---------<br> TIME SINCE LAST CPU USAGE RESET IN SECONDS: 246.528065<br>-------------------------------------------------------------------------------<br>[/] #<br>Not all time time, most of the runs both globals were zero, which is wierd ..<br><br>I also tried the patch. The issue was reproduced as well.<br>[/] # cpuuse<br>-------------------------------------------------------------------------------<br> CPU USAGE BY THREAD<br>------------+----------------------------------------+---------------+---------<br> ID | NAME | SECONDS | PERCENT<br>------------+----------------------------------------+---------------+---------<br>*cdemergian build 16.25 gIntrptErrs=233694 gInsertErrs=1*<br> 0x09010001 | IDLE | 94.488726 | 98.619<br> 0x0a010001 | UI1 | 1.000931 | 1.044<br> 0x0a010002 | ntwk | 0.030101 | 0.031<br> 0x0a010003 | SCtx | 0.021441 | 0.022<br> 0x0a010004 | SCrx | 0.027176 | 0.028<br> 0x0a010005 | eRPC | 0.000049 | 0.000<br> 0x0a010006 | SHLL | 0.215693 | 0.225<br> 0x0b010001 | | 0.000096 | 0.000<br> 0x0b010002 | | 0.027211 | 0.028<br>------------+----------------------------------------+---------------+---------<br> TIME SINCE LAST CPU USAGE RESET IN SECONDS: 95.867059<br>-------------------------------------------------------------------------------<br><br>we are getting big numbers for gIntrptErrs (is that normal ? I don't understand all the aspects of the patch just yet)<br></blockquote><br>Can you set a break point to the gIntrptErrs++ and print the stack traces?<br><br>-- <br>Sebastian Huber, embedded brains GmbH<br><br>Address : Dornierstr. 4, D-82178 Puchheim, Germany<br>Phone : +49 89 189 47 41-16<br>Fax : +49 89 189 47 41-09<br><a href="mailto:sebastian.huber@embedded-brains.de" target="_blank">E-Mail : sebastian.huber@embedded-brains.de</a><br>PGP : Public key available on request.<br><br>Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG.<br><br>_______________________________________________<br>users mailing list<br><a href="mailto:users@rtems.org" target="_blank">users@rtems.org</a><br><a href="http://lists.rtems.org/mailman/listinfo/users" target="_blank">http://lists.rtems.org/mailman/listinfo/users</a></div></div></blockquote></div><br></div></div></div>_______________________________________________<br>
users mailing list<br>
<a href="mailto:users@rtems.org" target="_blank">users@rtems.org</a><br>
<a href="http://lists.rtems.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.rtems.org/mailman/listinfo/users</a></blockquote></div>