<div dir="ltr">Hi,<div>My device just passed a 24h long duration test, I can say now that this issue is history. </div><div>Since we were looking at this for quite a while, I would like to thank you, Sebastian for your prompt & useful support !</div><div><br></div><div>regards,</div><div>Catalin</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Mar 29, 2019 at 11:56 AM Catalin Demergian <<a href="mailto:demergian@gmail.com">demergian@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi, </div><div>We had some time ago (sept/oct 2018) a long discussion where I was suspecting a</div><div>scheduler issue (subject "rtems_message_queue_receive/rtems_event_receive issues")</div><div><br></div><div>We got to the point where I realized that _Chain_Append_unprotected might fail to add an</div><div>element in the queue, with the effect of having a task in a funny state where state=READY, but</div><div>the task will not be in the ready chain, so the task will never get CPU time anymore since a task</div><div>needs to be blocked in order to be unblocked when new data arrives.</div><div><br></div><div>We were using USB then, but this issue re-became hot because we just got the same issue</div><div>over serial :)</div><div>I believe there is a possible chain of events that can make _Chain_Append_unprotected to fail, </div><div>explanations follow.</div><div><br></div><div><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US">/*</span></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><b><span lang="EN-US">* @note It does NOT disable interrupts to ensure the atomicity of the</span></b></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><b><span lang="EN-US">*       append operation.</span></b></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US">*/</span></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US"> </span></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US">RTEMS_INLINE_ROUTINE void _Chain_Append_unprotected(</span></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US">  Chain_Control *the_chain,</span></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US">  Chain_Node    *the_node</span></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US">)</span></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US">{</span></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US">  Chain_Node *tail = _Chain_Tail( the_chain );</span></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US">  Chain_Node *old_last = tail->previous;</span></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US"> </span></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US">  the_node->next = tail;</span></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><b><span lang="EN-US">  tail->previous = the_node;</span></b></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><b><span lang="EN-US">  old_last->next = the_node;</span></b></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US">  the_node->previous = old_last;</span></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US">}</span></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="color:rgb(31,73,125)">The</span></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><b><span lang="EN-US">  tail->previous = the_node;</span></b></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="color:rgb(31,73,125)"><b style="font-size:11pt;color:rgb(34,34,34)"><span lang="EN-US">  old_last->next = the_node;</span></b></span></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="color:rgb(31,73,125)">lines are the ones that actually add the element</span></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="color:rgb(31,73,125)">to the ready chain.</span></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="color:rgb(31,73,125)">If a thread executes those lines, but just before executing</span></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="color:rgb(31,73,125)">the_node->previous = old_last;</span></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="color:rgb(31,73,125)">another thread comes to add another node in this chain, it will set another node in </span></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="color:rgb(31,73,125)">tail->previous and old_last->next, and as a result, when the interrupted</span></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="color:rgb(31,73,125)">thread will continue to execute the last line, it will be for nothing, because the initial node will not be added to the ready chain.</span></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="color:rgb(31,73,125)"><br></span></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="color:rgb(31,73,125)">If this chain of events occur (<b>and after a while they will</b>), we get starvation for that task.</span></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-family:Arial,Helvetica,sans-serif;font-size:small">I'm reproducing this issue in a long duration test, the duration before this happens varies from run to run, but it always happens.</span></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><br></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><b>What I'm proposing is the following</b>: call _Chain_Append instead of _Chain_Append_unprotected in schedulerpriorityimpl.h, _Scheduler_priority_Ready_queue_enqueue function.</p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><br></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US">void _Chain_Append(</span></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US">  Chain_Control *the_chain,</span></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US">  Chain_Node    *node</span></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US">)</span></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US">{</span></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US">  ISR_Level level;</span></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US"> </span></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US">  _ISR_Disable( level );</span></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US">    _Chain_Append_unprotected( the_chain, node );</span></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-US">  </span><span lang="IT">_ISR_Enable( level );</span></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="IT">}</span></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="IT"><br></span></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="IT">This way the add-element-to-chain operation becomes atomic.</span></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="IT">I was able to run a long duration test (8 hrs) in my setup with this fix successfully.</span></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="IT"><br></span></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="IT">What do you think ?</span></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><br></p></div><div>regards,</div><div>Catalin</div><div><br></div></div>
</blockquote></div>