[EXTERNAL] Re: Problem running RTEMS on raspberrypi3
Cudmore, Alan P. (GSFC-5820)
alan.p.cudmore at nasa.gov
Mon Dec 16 13:27:06 UTC 2019
Has the RPI3 UART/Console been addressed in RTEMS?
My understanding is that the RPI 1, Zero (without wireless) and 2 all share the same UART/Console driver. But the Raspberry Pi 3 and Zero W use that UART to talk to the Bluetooth module. The “mini” UART is then used for the console.
So, at a minimum, I think the RPI3 needs a different console driver.
https://www.raspberrypi.org/documentation/configuration/uart.md
From: devel <devel-bounces at rtems.org> on behalf of Niteesh <gsnb.gn at gmail.com>
Date: Monday, December 16, 2019 at 8:02 AM
To: Christian Mauderer <list at c-mauderer.de>
Cc: "rtems-devel at rtems.org" <devel at rtems.org>
Subject: [EXTERNAL] Re: Problem running RTEMS on raspberrypi3
I got uboot running on my raspi3. But I can't figure out to load and run a custom kernel. Can you explain the steps or point me to some
reference.
On Mon, Dec 16, 2019 at 5:13 PM Niteesh <gsnb.gn at gmail.com<mailto:gsnb.gn at gmail.com>> wrote:
On Mon, Dec 16, 2019 at 2:36 AM Christian Mauderer <list at c-mauderer.de<mailto:list at c-mauderer.de>> wrote:
On 15/12/2019 21:29, Niteesh wrote:
>
>
> On Mon, Dec 16, 2019 at 12:53 AM Christian Mauderer <list at c-mauderer.de<mailto:list at c-mauderer.de>
> <mailto:list at c-mauderer.de<mailto:list at c-mauderer.de>>> wrote:
>
> On 15/12/2019 19:46, Niteesh wrote:
> >
> >
> > On Sun, Dec 15, 2019 at 10:15 PM Christian Mauderer
> <list at c-mauderer.de<mailto:list at c-mauderer.de> <mailto:list at c-mauderer.de<mailto:list at c-mauderer.de>>
> > <mailto:list at c-mauderer.de<mailto:list at c-mauderer.de> <mailto:list at c-mauderer.de<mailto:list at c-mauderer.de>>>> wrote:
> >
> > Hello Niteesh,
> >
> > On 15/12/2019 09:05, Niteesh wrote:
> > > I am trying to get RTEMS examples running on the RPI3, the
> RPI3 is
> > > similar to RPI2 so the examples built for RPI2 should
> technically
> > run on
> > > the RPi3.But they don't :(, I am really sure of what is causing
> > the problem.
> >
> > Note that there are at least two different versions of the
> RPi3 which
> > use different chips. The original RPi3 which uses a BCM2837
> (same like
> > later versions of RPi2) and the RPi3+ which uses a BCM2837B0.
> Broadcom
> > is always quite sparse with documentation so it's not easy to
> tell the
> > differences. Which one do you have?
> >
> > I have Rpi3 model b v1.2 which uses BCM2837 SOC, in my bare-metal
> > programming I used the
> > 2835 doc as a reference because the only major difference these
> two SOC
> > is the peripheral base address
> > offset. But this is arm cpu is also capable of executing in 64bit
> mode.
>
> OK. Did you check, whether the offset is correct? In the raspberrypi.h
> in RTEMS there is the following define:
>
> #if (BSP_IS_RPI2 == 1)
> #define RPI_PERIPHERAL_BASE 0x3F000000
> #else
> #define RPI_PERIPHERAL_BASE 0x20000000
> #endif
>
> The offsets are right.
Good.
>
> >
> > > I followed the steps
> > >
> >
> from http://alanstechnotes.blogspot.com/2013/03/running-your-first-rtems-program-on.html<https://urldefense.proofpoint.com/v2/url?u=http-3A__alanstechnotes.blogspot.com_2013_03_running-2Dyour-2Dfirst-2Drtems-2Dprogram-2Don.html&d=DwMFaQ&c=ApwzowJNAKKw3xye91w7BE1XMRKi2LN9kiMk5Csz9Zk&r=mW-jVVDIxBn4bZU1jogCo9FDuDuPszsoVj1mqBvCXzw&m=mgmnjRRtQ0UiBECza6LBUdPRMlI07FCjTRHH5qFTYRk&s=RXd0TjeCxMP_Y43iLHMAMB4sv7Tfq1pjRRVKJbqfyJg&e=> (modified
> > > commands to use rtems5) to build the kernel img.
> >
> > It's a bit odd that the Bootloader doesn't use some image
> format like
> > U-Boot but if that's the case for Raspberry, that's OK.
> >
> > Do you want me to try U-Boot, I was planning to use it for my
> bare-metal
> > stuff because copying the kernel
> > to SD-card was a real pain. Will it even work with RTEMS?
>
> The manual that you linked uses the default Raspberry bootloader. I'm
> not sure whether it's an U-Boot. If you skip the bootloader entirely,
> your SDRAM might isn't initialized.
>
> The manual uses the default bootloader. I don't think we have to worry
> about the SDRAM initialization
> because all of that is taken care of by the GPU.
Sounds OK.
> When using Uboot, the
> GPU will load the uboot image and
> pass the control to the CPU. And then the uboot continue's it's execution.
>
I don't wanted to suggest to use an extra U-Boot. I was just not sure
whether the stage 3 loader is an U-Boot. Your approach sounds fine so far.
>
>
> PS: You answered that further below. You are using the stage 3 loader.
>
> >
> > > I did try running it on
> > > Qemu but it doesn't always work, sometimes it gives
> weird output.
> >
> > How did you run it on Qemu? Did you build some image for that too?
> >
> > qemu-system-arm -M raspi2 -m 1G -kernel hello.exe -serial mon:stdio
> > -nographic
> > *
> > *
> > *
> > qemu-system-aarch64: GLib: g_mapped_file_unref: assertion 'file !=
> NULL'
> > failed *I get this error
> > while trying to emulate raspi3.
>
> That sounds like a problem with Qemu. Is there some official test image
> for rpi3 on qemu? Note that this isn't really relevant for your current
> problem. So if you don't have some manual just ignore the question.
>
> >
> > I ran qemu along with GDB to find what was causing the wrong output. I
> > am really not sure if this is right,
> > I still have a lot to learn, but my assumption's using GDB are as
> follows.
> > There are 4 active thread which run the same code.
> >
> > (gdb) info thread
> > Id Target Id Frame
> > * 1 Thread 1.1 (CPU#0 [running]) _start () at
> >
> ../../../../../../../../rtems/c/src/lib/libbsp/arm/raspberrypi/../../../../../../bsps/arm/shared/start/start.S:153
> > 2 Thread 1.2 (CPU#1 [running]) _start () at
> >
> ../../../../../../../../rtems/c/src/lib/libbsp/arm/raspberrypi/../../../../../../bsps/arm/shared/start/start.S:153
> > 3 Thread 1.3 (CPU#2 [running]) _start () at
> >
> ../../../../../../../../rtems/c/src/lib/libbsp/arm/raspberrypi/../../../../../../bsps/arm/shared/start/start.S:153
> > 4 Thread 1.4 (CPU#3 [running]) _start () at
> >
> ../../../../../../../../rtems/c/src/lib/libbsp/arm/raspberrypi/../../../../../../bsps/arm/shared/start/start.S:153
>
> In this case that are not threads but it's the CPU cores. GDB shows them
> as threads. Most likely it wouldn't be able to detect the RTEMS threads.
>
> It's a bit odd that they are all pointing to start.S:153. That's the
> entry point for the program. It looks like not even one instruction has
> been executed yet.
>
> I took this output before executing the program, that the reason why not
> even a single instruction has been
> executed yet.
OK.
>
> >
> > After some time one of the thread call's the BSP reset function
> this is
> > when the program crashes, the other threads complain "*executing
> thread
> > is NULL*"
>
> I would rather assume that one core tries to do the initialization while
> the others hang in a endless loop till they are needed. The one core
> doing the initialization work hits an exception somewhere and calls the
> exception handler which calls the bsp reset function.
>
> The executing thread is NULL is a sign that it happens somewhere during
> initialization when the RTEMS threading hasn't been started yet.
>
> The PC has an odd value. The linker command file tells that there is a
> RAM_MMU at 0x00100000. It only puts a bsp_translation_table there but
> there shouldn't be any code. So I don't know what the processor is doing
> there. You could try to set a breakpoint on the address 0x00100fc4 and
> take a look at why the processor is there with a "bt" (backtrace).
>
> When I re-run it again, it now stops at a different address. As you said
> that the other cores are put
> in an endless loop, I don't think that's is happening. I single stepped
> the instruction and later at some point checked the threads
>
> (gdb) info threads
>
>
> Target Id Frame
> 1 Thread 1.1 (CPU#0 [running]) arm_ccsidr_get_line_power
> (ccsidr=<optimized out>)
> at
> /home/niteesh/development/rtems/kernel/rtems/cpukit/score/cpu/arm/include/libcpu/arm-cp15.h:850
> 2 Thread 1.2 (CPU#1 [running]) arm_cp15_cache_invalidate_level
> (inst_data_fl=0, level=1)
> at
> /home/niteesh/development/rtems/kernel/rtems/cpukit/score/cpu/arm/include/libcpu/arm-cp15.h:1162
> 3 Thread 1.3 (CPU#2 [running]) arm_ccsidr_get_line_power
> (ccsidr=<optimized out>)
> at
> /home/niteesh/development/rtems/kernel/rtems/cpukit/score/cpu/arm/include/libcpu/arm-cp15.h:850
> * 4 Thread 1.4 (CPU#3 [running])
> arm_cp15_get_cache_size_id_for_level (level_and_inst_dat=0)
> at
> /home/niteesh/development/rtems/kernel/rtems/cpukit/score/cpu/arm/include/libcpu/arm-cp15.h:936
> (gdb)
>
> They all are executing different instructions at the same time.
Some of the initialization is done on all cores. Some isn't. I took a
look at the initialization and it seems that I was wrong: There is no
wait loop. All processors are running through the initialization
process. Some just skip parts. The part where they really start to
differ is in bsp_start_hook_0.
> I> googled about just running one thread or CPU as you said at a time and
> used "*set scheduler-locking on" *on doing this I always get the right
> output.
>
> (gdb) info threads
> Id Target Id Frame
> * 1 Thread 1.1 (CPU#0 [running]) bsp_reset ()
> at
> ../../../../../../../../rtems/c/src/lib/libbsp/arm/raspberrypi/../../../../../../bsps/arm/raspberrypi/start/bspreset.c:18
> 2 Thread 1.2 (CPU#1 [running]) _start ()
> at
> ../../../../../../../../rtems/c/src/lib/libbsp/arm/raspberrypi/../../../../../../bsps/arm/shared/start/start.S:153
> 3 Thread 1.3 (CPU#2 [running]) _start ()
> at
> ../../../../../../../../rtems/c/src/lib/libbsp/arm/raspberrypi/../../../../../../bsps/arm/shared/start/start.S:153
> 4 Thread 1.4 (CPU#3 [running]) _start ()
> at
> ../../../../../../../../rtems/c/src/lib/libbsp/arm/raspberrypi/../../../../../../bsps/arm/shared/start/start.S:153
> (gdb)
>
> The above command allow's only a single thread to run.
Maybe there is a timing difference between the simulator and the real
hardware. I'm not sure how well tested the SMP code is on the Raspberry.
There can be a hidden bug.
Just a guess: If there is a bug it could be possible that you hit it
with your rpi3 too. Maybe it would be good to try a single core version
of the BSP. I assume you have configured with "--enable-smp"? Can you
try to build without it?
I built 2 versions with SMP enabled and disabled, the one we are talking about is the SMP disabled version, I ran
the example with SMP enabled, still, the error's are similar, I only difference is, in the disabled one, there are only 4 or less panic's
(maybe corresponding to 4 cpu's) but the other one has a higher number of panics.
> Won't it be a good idea to make a separate BSP for rpi3?
As soon as it is necessary: Sure. But from what you told me it seems
that the hardware is very similar so that we won't hit this point soon.
Or do you already see differences that would make it necessary.
I haven't had a look at the details but it could also be possible to
unify the BSPs and entirely remove the rpi2 variant if the information
from the flattened device tree are used.
Can you explain how FDT work in RTEMS. Can you mention some BSP's which use FDT so I can use them as a reference to learn.
I previously took a look at the beagle FDT project (#3784), you mentioned about hardcoded values and initialization functions, can
you explain more about what exactly do the initialization functions do? Do they assign a function to a particular pin, like in raspi
the pins are multiplexed for various functions, so do the initialization functions assign those pins to a particular function?
And also please explain how does the initialization of the system happen from the DT file.
>
> > *** FATAL ***
> > fatal source: 9 (RTEMS_FATAL_SOURCE_EXCEPTION)
> >
> > R0 = 0x400005e6 R8 = 0x00000000
> > R1 = 0x00000001 R9 = 0x00000000
> > R2 = 0xbffffa1a R10 = 0x00000000
> > R3 = 0x00000000 R11 = 0x00000000
> > R4 = 0x002001db R12 = 0x00000000
> > R5 = 0x00000000 SP = 0x00300bd0
> > R6 = 0x00000000 LR = 0x00100fc4
> > R7 = 0x00000000 PC = 0x00100fc4
> > CPSR = 0x000001d3 VEC = 0x00000002
> > FPEXC = 0x40000000
> > FPSCR = 0x00000000
> > D00 = 0x0000000000000000
> > D01 = 0x0000000000000000
> > D02 = 0x0000000000000000
> > D03 = 0x0000000000000000
> > D04 = 0x0000000000000000
> > D05 = 0x0000000000000000
> > D06 = 0x0000000000000000
> > D07 = 0x0000000000000000
> > D08 = 0x0000000000000000
> > D09 = 0x0000000000000000
> > D10 = 0x0000000000000000
> > D11 = 0x0000000000000000
> > D12 = 0x0000000000000000
> > D13 = 0x0000000000000000
> > D14 = 0x0000000000000000
> > D15 = 0x0000000000000000
> > D16 = 0x0000000000000000
> > D17 = 0x0000000000000010
> > D18 = 0x0000000000000000
> > D19 = 0x0000000000000000
> > D20 = 0x0000000000000000
> > D21 = 0x0000000000000000
> > D22 = 0x0000000000000000
> > D23 = 0x0000000000000000
> > D24 = 0x0000000000000000
> > D25 = 0x0000000000000000
> > D26 = 0x0000000000000000
> > D27 = 0x0000000000000000
> > D28 = 0x0000000000000000
> > D29 = 0x0000000000000000
> > D30 = 0x0000000000000000
> > D31 = 0x0000000000000000
> > RTEMS version:
> 5.0.0.c6d8589bb00a9d2a5a094c68c90290df1dc44807-modified
> > RTEMS tools: 7.5.0 20191114 (RTEMS 5, RSB
> > 83fa79314dd87c0a8c78fd642b2cea3138be8dd6, Newlib 3e24fbf6f)
> > executing thread is NULL
> >
> > > The steps that I followed are:
> > > 1. Created a bootable SD card using raspbian.
> > > 2. Replaced the kernel.img file with RTEMS kernel.img file and
> > modified
> > > the config.txt to boot from the RTEMs kernel (boots in
> aarch32 bit
> > mode).
> > > I am still not able to wrap my head around the RPI bsp build
> process.
> > > This is what I understood as of now, correct me if I am wrong.
> > > Both RPi and Rpi2 are based on the same BSP, they just
> differ in the
> > > peripheral offsets, hardcoded checks are used to select the
> right
> > offset
> > > at the time of compiling
> >
> > >From what I know of the Raspberry BSPs that is correct.
> >
> > > and the linkercmd file is responsible for
> > > building the final executable file.
> >
> > The linkercmd file is - like for all programs - responsible
> where the
> > memory regions are that can be used for code or data. So you
> could more
> > or less explain it like you did.
> >
> > > I looked at the linker script, it seem's to have the start
> section at
> > > address 0x200000, I also loaded it in GDB and the start
> address is
> > > *Start address 0x200080,*
> >
> > I agree with that. The different start in GDB is most likely
> because
> > there is a vector table in front (at least if the Broadcom chip is
> > similar to a lot of other processors that I have encountered).
> >
> > Does that mean that you have a debugger connected to the
> raspberry? Can
> > you load code with it? If yes: Is the bootloader executed
> before you
> > load your code? Otherwise the SDRAM might isn't initialized yet.
> >
> > I don't have a debugger connected to it. I from what I have SDRAM is
> > initialized by the 3 stage bootloader(start.elf).
>
> That should be OK and it answers my question above.
>
> >
> > > I did some bare metal programming on RPI3
> > > there I had the start section at address 0x8000 Is this causing
> > the problem?
> >
> > I assume that you used some internal RAM when you did bare metal
> > programming. You maybe even skipped one or two bootloader
> stages. From a
> > quick look Raspberry has a quite complex boot process with at
> least
> > three bootloaders:
> http://lions-wing.net/maker/raspberry-1/boot.html<https://urldefense.proofpoint.com/v2/url?u=http-3A__lions-2Dwing.net_maker_raspberry-2D1_boot.html&d=DwMFaQ&c=ApwzowJNAKKw3xye91w7BE1XMRKi2LN9kiMk5Csz9Zk&r=mW-jVVDIxBn4bZU1jogCo9FDuDuPszsoVj1mqBvCXzw&m=mgmnjRRtQ0UiBECza6LBUdPRMlI07FCjTRHH5qFTYRk&s=RKIi8InlPNqYxRJyxDcr_gruhLWweSnjRhwcrAxKYFM&e=>
> >
> > I don't think I have skipped any stages. The boot process is
> exactly the
> > same as how it boot's a normal raspbian or any other linux
> > distro, I just to replace the linux kernel with my own kernel.
>
> Sounds reasonable. Does the bootloader print anything where it puts the
> kernel image? Maybe the start address changed during the raspberry
> versions.
>
> the default kernel load address is 0x8000 in 32bit mode and 0x80000 in
> 64bit mode I have no idea about the raspberry 1,
> but the load address is same for rpi2 and 3.
That sounds odd. Do you have a memory map somewhere? From the linker
command file it seems quite clear that RTEMS is build for a 0x200000.
>
> >
> >
> > > I have no idea on how to debug this, any suggestion on how
> to start
> > > would be really helpfull.
> > >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rtems.org/pipermail/devel/attachments/20191216/e92edbf0/attachment-0001.html>
More information about the devel
mailing list