RTEMS_FATAL_SOURCE_EXCEPTION in RTEMS

Thu May 2 10:05:22 UTC 2019

----- Ursprüngliche Mail -----
> Von: "Amarnath MB" <amarnath.mb at mistralsolutions.com>
> An: "Christian Mauderer" <christian.mauderer at embedded-brains.de>
> CC: "RTEMS Users" <users at rtems.org>, "Ravi G Patil" <ravigp at mistralsolutions.com>, "Shekhar Suman Singh"
> <shekhar.s at mistralsolutions.com>
> Gesendet: Donnerstag, 2. Mai 2019 11:18:48
> Betreff: Re: RTEMS_FATAL_SOURCE_EXCEPTION in RTEMS

> Hi Christian,
> 
> As per your suggestion, i tried disabling  global interrupt before my erase
> call and enabling it afterwards. With this setup the test routine is
> passing without giving exception. So i think, interrupt was the issue here.
> 

Hello Amarnath,

good. Than you have at least a hint toward the problem. I wouldn't see it as the solution. Like already discussed, depending on your application, this can lead to missed interrupts. If you have a very specific case for writing the flash that isn't during your normal operation (for example firmware updates) you are most likely fine with that solution. But if you write for example log files, you might get a quite unpredictable behavior later.

> Flash controller is implemented using the AHB to external SRAM interface in
> the Cortex-M System Design Kit and there are no pins shared between RAM and
> NOR Flash.  We can't doubt that flash is blocking bus accesses for the RAM,
> because the test executes fine in the uboot running from RAM.

So it's a custom FPGA based design? Most likely your RAM and Flash controller are both connected to the same system bus (AHB in that case). I don't know the modules you mentioned but you might want to take a look at the documentation whether they lock the AHB or not.

U-Boot most likely doesn't use interrupts. I assume you are waiting for a flash operation to be finished in some busy wait loop there (polling a flag or similar). That loop can be fully put into the processor cache (if you have one which is quite likely). So in your U-Boot test case, the external RAM most likely isn't accessed during the flash operations. If you have a problem with conflicting bus accesses you maybe only see them there if you disable all caches.

In RTEMS on the other hand you are running with enabled interrupts and task switches. So RAM access are more or less guaranteed during a longer flash operation if you don't lock the interrupts.

> 
> You had mentioned there are a lot of possible reasons for a
> RTEMS_FATAL_SOURCE_EXCEPTION, is it somewhere documented what all reasons
> can cause this exception. Any references would be very much helpful.

Basically for an ARM system, you have that source:

cpukit/score/cpu/arm/arm-exception-default.c:24:  rtems_fatal( RTEMS_FATAL_SOURCE_EXCEPTION, (rtems_fatal_code) frame );

That is default handler for all exceptions where no extra handler is installed. So basically every exception that is listed in the ARM manual can be the source.

> 
> Thank you once again for your prompt response, it really saved lot of our
> time.
> 
> Below is the exception frame we got,
> FYI, 0x0 - 0x007FFFFF is NOR Flash and 0x40000000 to 0x7FFFFFFF the RAM.

Yust a note: Putting memory at 0 might hides some NULL-pointer accesses. If you can do that in your design you might want to think about putting some boot ROM there that isn't accessible by the application (for example locked via MPU). 

Of course there can be worse types of memory Flash (which should be mostly a read only memory). I had a controller with some RAM there once. Finding NULL-pointer accesses on such a system is really nasty.

> 
> *** FATAL ***
> fatal source: 9 (RTEMS_FATAL_SOURCE_EXCEPTION)
> 
> R0   = 0x00000040 R8  = 0x00100000
> R1   = 0x00080000 R9  = 0x00000001
> R2   = 0x00000048 R10 = 0x4010a3e8
> R3   = 0x00000048 R11 = 0x4011737c
> R4   = 0x00000a00 R12 = 0x00000000
> R5   = 0x00000500 SP  = 0x40100c40
> R6   = 0x00000055 LR  = 0x4000a36c
> R7   = 0x000000aa PC  = 0x003fde80
> CPSR = 0x200000d2 VEC = 0x00000001
> RTEMS version: 5.0.0.
> RTEMS tools: 7.4.0 20181206 (RTEMS 5, RSB
> 40ae056f12e1cbe530f76a3ebd1e2ac745a888ef, Newlib
> dc6e94551f09d3a983afd571478d63a09d6f66fa)
> executing thread ID: 0x08a010002
> executing thread name: Alpi

Your program counter (PC) points to a Flash address here. Are you sure that your application runs entirely from RAM?

Best regards

Christian

> 
> *Thank you & Regards,*
> *Amarnath MB*
> 
> 
> 
> On Thu, May 2, 2019 at 12:49 AM Christian Mauderer <
> christian.mauderer at embedded-brains.de> wrote:
> 
>>
>> ----- Ursprüngliche Mail -----
>> > Von: "Amarnath MB" <amarnath.mb at mistralsolutions.com>
>> > An: "Christian Mauderer" <christian.mauderer at embedded-brains.de>
>> > CC: "RTEMS Users" <users at rtems.org>, "Ravi G Patil" <
>> ravigp at mistralsolutions.com>, "Shekhar Suman Singh"
>> > <shekhar.s at mistralsolutions.com>
>> > Gesendet: Mittwoch, 1. Mai 2019 17:07:13
>> > Betreff: Re: RTEMS_FATAL_SOURCE_EXCEPTION in RTEMS
>>
>> > Hi Christian,
>> >
>> > Thanks for your quick response.
>> > Our boot process is like, first u-boot will be loaded from the NOR flash
>> > and then the uboot copies RTEMS application from NOR flash to the RAM.
>> > After that RTEMS executes entirely from RAM.
>> >
>> > How can i issue a global interrupt disable? Is it using *
>> > rtems_interrupt_disable(0)*?
>> > One more doubt, if i issue a global interrupt disable then will there be
>> > chance that i can miss few clock ticks?
>> >
>> > *Thank you & Regards,*
>> > *Amarnath MB*
>> >
>>
>> Hello Amarnath,
>>
>> with a global interrupt lock, all kinds of events can be missed including
>> clock ticks. Most should be processed just a little late but some
>> interfaces might loose packets or data (depending on the data rate / flash
>> access times). So if not necessary it's not a good solution.
>>
>> The rtems_interrupt_disable() function should be called with a level
>> argument. See the example at
>> https://docs.rtems.org/branches/master/c-user/interrupt_manager.html#rtems-interrupt-disable.
>> Note that for SMP configurations, you might have to use other functions.
>>
>> But if your application runs entirely from RAM than the bus access
>> shouldn't be the problem. So the interrupt might isn't the right guess.
>>
>> Do you have any more information on the exception and where it happens?
>> Some output or a stack trace from a debugger?
>>
>> What kind of flash controller is used? Can it block bus accesses for the
>> RAM? Does your RAM share pins with the flash?
>>
>> With kind regards
>>
>> Christian
>>
>> >
>> >
>> > On Wed, May 1, 2019 at 8:16 PM Christian Mauderer <
>> > christian.mauderer at embedded-brains.de> wrote:
>> >
>> >> ----- Ursprüngliche Mail -----
>> >> > Von: "Amarnath MB" <amarnath.mb at mistralsolutions.com>
>> >> > An: "RTEMS Users" <users at rtems.org>
>> >> > CC: "Ravi G Patil" <ravigp at mistralsolutions.com>, "Shekhar Suman
>> Singh"
>> >> <shekhar.s at mistralsolutions.com>
>> >> > Gesendet: Mittwoch, 1. Mai 2019 16:28:23
>> >> > Betreff: RTEMS_FATAL_SOURCE_EXCEPTION in RTEMS
>> >>
>> >> > Hi All,
>> >> >
>> >> > I'm developing RTEMS 5.00 BSP and device drivers for a custom
>> ARM926EJ-S
>> >> > core. I was successful in porting and building BSP for the ARM core.
>> >> >
>> >> > We are facing a strange issue with the generic NOR flash driver we
>> have
>> >> > developed. Our device drivers are designed such that it can be used
>> with
>> >> > RTEMS as well as the bare metal programs.
>> >> > Our driver is working fine in bare metal program (erase, read, write
>> >> > everything), but when we use the driver with RTEMS application and
>> issue
>> >> an
>> >> > erase call, then the application gives RTEMS_FATAL_SOURCE_EXCEPTION.
>> >> >
>> >> > For testing driver in RTEMS, we have added each test routine as a
>> custom
>> >> > shell command using rtems_shell_add_cmd().
>> >> >
>> >> > For your info we also tested the same driver with u-boot and it works
>> >> > fines.
>> >> > Can anyone guide me on this issue?
>> >> >
>> >> > *Thank you & Regards,*
>> >> > *Amarnath MB*
>> >> >
>> >>
>> >> Hello Amarnath,
>> >>
>> >> it's a little hard with the given information to tell you a concrete
>> >> problem. There are a lot of possible reasons for a
>> >> RTEMS_FATAL_SOURCE_EXCEPTION.
>> >>
>> >> From your description, my guess would be some problem with an interrupt.
>> >> Most U-Boot code avoids interrupts. I don't know about your bare metal
>> >> application but maybe in some test application, you don't have too much
>> >> interrupts too. In RTEMS you have most likely at least a periodic tick
>> >> interrupt.
>> >>
>> >> Do you execute your code from the same flash or do you keep some data in
>> >> the flash? In that case it could be an access error during a flash
>> erase or
>> >> write (for example caused by an interrupt). In that case you might have
>> to
>> >> do a global interrupt disable before you enter certain routines.
>> >>
>> >> Best regards
>> >>
>> >> Christian
>>

-- 
--------------------------------------------
embedded brains GmbH
Christian Mauderer
Dornierstr. 4
D-82178 Puchheim
Germany
email: christian.mauderer at embedded-brains.de
Phone: +49-89-18 94 741 - 18
Fax:   +49-89-18 94 741 - 08
PGP: Public key available on request.

Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG.