File system deadlock troubleshooting
mbenson at windhoverlabs.com
Tue Oct 8 14:30:44 UTC 2019
I don't have a test case that I can post or give away, yet. I may put one
Yes. All 4 tasks are on that same conditional wait. The condition is a
response from a block device. The test case is hammering the RAM disk. I
wrote some of the flash drivers but as far as I know, the RAM disk was used
as is so I haven't looked at the code. That conditional wait does not have
a time out so it just pends forever.
Thanks. I'll try the RFS trace when I have time. Unfortunately, this is
late in the development cycle so I don't have ready access to the article
under test. Loading it with non production code takes time. Trying to
reproduce it on a non production environment. For now, all I have is shell
commands and manually decoding dumped memory.
Somebody more knowledgable on the inner workings of the RTEMS kernel
expressed concern a while ago that heavy use of the RAM disk could
basically out run the kernel worker thread. I noted it but didn't ask
about the mechanics of how or why that could happen. Are requests to the
worker threads in a lossy queue? Is it possible the request is getting
On Mon, Oct 7, 2019 at 11:30 PM Chris Johns <chrisj at rtems.org> wrote:
> On 8/10/19 12:53 pm, Mathew Benson wrote:
> > I'm using RTEMS 5 on a LEON3. I'm troubleshooting a failure condition
> > occurs when stress test reading and writing to and from RAM disk. RAM
> disk to
> > RAM disk. When the condition is tripped, it appears that I have 4 tasks
> > are pending on conditions that just never happens.
> Do you have a test case?
> > The task command shows:
> > ID NAME SHED PRI STATE MODES EVENTS WAITINFO
> > 0a01000c TSKA UPD 135 MTX P:T:nA NONE RFS
> > 0a01001f TSKB UPD 135 CV P:T:nA NONE bdbuf
> > 0a010020 TSKC UPD 150 MTX P:T:nA NONE RFS
> > 0a010032 TSKD UPD 245 MTX P:T:nA NONE RFS
> It looks like TSKA, TSKC and TSKD are waiting for the RFS lock and TSKB is
> blocked in a bdbuf access. I wonder why that is blocked?
> The RFS hold's it lock over the bdbuf calls.
> > None of my tasks appear to be failed. Nobody is pending on anything
> > except the 4 above. The conditional wait is a single shared resource so
> > attempt to access the file system after this happens results in yet
> > forever pended task.
> > Digging into source code, it appears that the kernel is waiting for a
> > response from a block device, but just didn't get what its expecting.
> The next
> > thing is to determine which block device the kernel is pending on, what
> > expected response is, and what the block device actually did. Can
> anybody shed
> > some light on this or recommend some debugging steps? I'm trying to
> > all I can do before I start manually decoding machine code.
> The RFS has trace support you can access via
> `rtems/rfs/rtems-rfs-trace.h`. You
> can set the trace mask in your code or you can can call
> `rtems_rfs_trace_shell_command()` with suitable arguments or hook it to an
> existing shell. There is a buffer trace flag that show the release calls
> to bdbuf ..
> There is no trace call to get or read. Maybe add a get/read trace as well.
> The RAM disk also has trace in the code which can be enabled by editing
> the file.
CEO | Chief Engineer
Windhover Labs, LLC
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the users