<div dir="ltr"><div dir="ltr">On Mon, Dec 16, 2019 at 2:36 AM Christian Mauderer <<a href="mailto:list@c-mauderer.de">list@c-mauderer.de</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
<br>
On 15/12/2019 21:29, Niteesh wrote:<br>
> <br>
> <br>
> On Mon, Dec 16, 2019 at 12:53 AM Christian Mauderer <<a href="mailto:list@c-mauderer.de" target="_blank">list@c-mauderer.de</a><br>
> <mailto:<a href="mailto:list@c-mauderer.de" target="_blank">list@c-mauderer.de</a>>> wrote:<br>
> <br>
> On 15/12/2019 19:46, Niteesh wrote:<br>
> ><br>
> ><br>
> > On Sun, Dec 15, 2019 at 10:15 PM Christian Mauderer<br>
> <<a href="mailto:list@c-mauderer.de" target="_blank">list@c-mauderer.de</a> <mailto:<a href="mailto:list@c-mauderer.de" target="_blank">list@c-mauderer.de</a>><br>
> > <mailto:<a href="mailto:list@c-mauderer.de" target="_blank">list@c-mauderer.de</a> <mailto:<a href="mailto:list@c-mauderer.de" target="_blank">list@c-mauderer.de</a>>>> wrote:<br>
> ><br>
> > Hello Niteesh,<br>
> ><br>
> > On 15/12/2019 09:05, Niteesh wrote:<br>
> > > I am trying to get RTEMS examples running on the RPI3, the<br>
> RPI3 is<br>
> > > similar to RPI2 so the examples built for RPI2 should<br>
> technically<br>
> > run on<br>
> > > the RPi3.But they don't :(, I am really sure of what is causing<br>
> > the problem.<br>
> ><br>
> > Note that there are at least two different versions of the<br>
> RPi3 which<br>
> > use different chips. The original RPi3 which uses a BCM2837<br>
> (same like<br>
> > later versions of RPi2) and the RPi3+ which uses a BCM2837B0.<br>
> Broadcom<br>
> > is always quite sparse with documentation so it's not easy to<br>
> tell the<br>
> > differences. Which one do you have?<br>
> ><br>
> > I have Rpi3 model b v1.2 which uses BCM2837 SOC, in my bare-metal<br>
> > programming I used the <br>
> > 2835 doc as a reference because the only major difference these<br>
> two SOC<br>
> > is the peripheral base address<br>
> > offset. But this is arm cpu is also capable of executing in 64bit<br>
> mode.<br>
> <br>
> OK. Did you check, whether the offset is correct? In the raspberrypi.h<br>
> in RTEMS there is the following define:<br>
> <br>
> #if (BSP_IS_RPI2 == 1)<br>
> #define RPI_PERIPHERAL_BASE 0x3F000000<br>
> #else<br>
> #define RPI_PERIPHERAL_BASE 0x20000000<br>
> #endif<br>
> <br>
> The offsets are right.<br>
<br>
Good.<br>
<br>
> <br>
> ><br>
> > > I followed the steps<br>
> > ><br>
> > <br>
> from <a href="http://alanstechnotes.blogspot.com/2013/03/running-your-first-rtems-program-on.html" rel="noreferrer" target="_blank">http://alanstechnotes.blogspot.com/2013/03/running-your-first-rtems-program-on.html</a> (modified<br>
> > > commands to use rtems5) to build the kernel img.<br>
> ><br>
> > It's a bit odd that the Bootloader doesn't use some image<br>
> format like<br>
> > U-Boot but if that's the case for Raspberry, that's OK.<br>
> ><br>
> > Do you want me to try U-Boot, I was planning to use it for my<br>
> bare-metal<br>
> > stuff because copying the kernel<br>
> > to SD-card was a real pain. Will it even work with RTEMS?<br>
> <br>
> The manual that you linked uses the default Raspberry bootloader. I'm<br>
> not sure whether it's an U-Boot. If you skip the bootloader entirely,<br>
> your SDRAM might isn't initialized.<br>
> <br>
> The manual uses the default bootloader. I don't think we have to worry<br>
> about the SDRAM initialization<br>
> because all of that is taken care of by the GPU.<br>
<br>
Sounds OK.<br>
<br>
> When using Uboot, the<br>
> GPU will load the uboot image and<br>
> pass the control to the CPU. And then the uboot continue's it's execution.<br>
> <br>
<br>
I don't wanted to suggest to use an extra U-Boot. I was just not sure<br>
whether the stage 3 loader is an U-Boot. Your approach sounds fine so far.<br>
<br>
> <br>
> <br>
> PS: You answered that further below. You are using the stage 3 loader.<br>
> <br>
> ><br>
> > > I did try running it on<br>
> > > Qemu but it doesn't always work, sometimes it gives<br>
> weird output.<br>
> ><br>
> > How did you run it on Qemu? Did you build some image for that too?<br>
> ><br>
> > qemu-system-arm -M raspi2 -m 1G -kernel hello.exe -serial mon:stdio<br>
> > -nographic<br>
> > *<br>
> > *<br>
> > *<br>
> > qemu-system-aarch64: GLib: g_mapped_file_unref: assertion 'file !=<br>
> NULL'<br>
> > failed *I get this error <br>
> > while trying to emulate raspi3.<br>
> <br>
> That sounds like a problem with Qemu. Is there some official test image<br>
> for rpi3 on qemu? Note that this isn't really relevant for your current<br>
> problem. So if you don't have some manual just ignore the question.<br>
> <br>
> ><br>
> > I ran qemu along with GDB to find what was causing the wrong output. I<br>
> > am really not sure if this is right,<br>
> > I still have a lot to learn, but my assumption's using GDB are as<br>
> follows.<br>
> > There are 4 active thread which run the same code.<br>
> ><br>
> > (gdb) info thread<br>
> > Id Target Id Frame<br>
> > * 1 Thread 1.1 (CPU#0 [running]) _start () at<br>
> > <br>
> ../../../../../../../../rtems/c/src/lib/libbsp/arm/raspberrypi/../../../../../../bsps/arm/shared/start/start.S:153<br>
> > 2 Thread 1.2 (CPU#1 [running]) _start () at<br>
> > <br>
> ../../../../../../../../rtems/c/src/lib/libbsp/arm/raspberrypi/../../../../../../bsps/arm/shared/start/start.S:153<br>
> > 3 Thread 1.3 (CPU#2 [running]) _start () at<br>
> > <br>
> ../../../../../../../../rtems/c/src/lib/libbsp/arm/raspberrypi/../../../../../../bsps/arm/shared/start/start.S:153<br>
> > 4 Thread 1.4 (CPU#3 [running]) _start () at<br>
> > <br>
> ../../../../../../../../rtems/c/src/lib/libbsp/arm/raspberrypi/../../../../../../bsps/arm/shared/start/start.S:153<br>
> <br>
> In this case that are not threads but it's the CPU cores. GDB shows them<br>
> as threads. Most likely it wouldn't be able to detect the RTEMS threads.<br>
> <br>
> It's a bit odd that they are all pointing to start.S:153. That's the<br>
> entry point for the program. It looks like not even one instruction has<br>
> been executed yet.<br>
> <br>
> I took this output before executing the program, that the reason why not<br>
> even a single instruction has been<br>
> executed yet.<br>
<br>
OK.<br>
<br>
> <br>
> ><br>
> > After some time one of the thread call's the BSP reset function<br>
> this is<br>
> > when the program crashes, the other threads complain "*executing<br>
> thread<br>
> > is NULL*"<br>
> <br>
> I would rather assume that one core tries to do the initialization while<br>
> the others hang in a endless loop till they are needed. The one core<br>
> doing the initialization work hits an exception somewhere and calls the<br>
> exception handler which calls the bsp reset function.<br>
> <br>
> The executing thread is NULL is a sign that it happens somewhere during<br>
> initialization when the RTEMS threading hasn't been started yet.<br>
> <br>
> The PC has an odd value. The linker command file tells that there is a<br>
> RAM_MMU at 0x00100000. It only puts a bsp_translation_table there but<br>
> there shouldn't be any code. So I don't know what the processor is doing<br>
> there. You could try to set a breakpoint on the address 0x00100fc4 and<br>
> take a look at why the processor is there with a "bt" (backtrace).<br>
> <br>
> When I re-run it again, it now stops at a different address. As you said<br>
> that the other cores are put<br>
> in an endless loop, I don't think that's is happening. I single stepped<br>
> the instruction and later at some point checked the threads<br>
> <br>
> (gdb) info threads <br>
> <br>
> <br>
> Target Id Frame<br>
> 1 Thread 1.1 (CPU#0 [running]) arm_ccsidr_get_line_power<br>
> (ccsidr=<optimized out>)<br>
> at<br>
> /home/niteesh/development/rtems/kernel/rtems/cpukit/score/cpu/arm/include/libcpu/arm-cp15.h:850<br>
> 2 Thread 1.2 (CPU#1 [running]) arm_cp15_cache_invalidate_level<br>
> (inst_data_fl=0, level=1)<br>
> at<br>
> /home/niteesh/development/rtems/kernel/rtems/cpukit/score/cpu/arm/include/libcpu/arm-cp15.h:1162<br>
> 3 Thread 1.3 (CPU#2 [running]) arm_ccsidr_get_line_power<br>
> (ccsidr=<optimized out>)<br>
> at<br>
> /home/niteesh/development/rtems/kernel/rtems/cpukit/score/cpu/arm/include/libcpu/arm-cp15.h:850<br>
> * 4 Thread 1.4 (CPU#3 [running])<br>
> arm_cp15_get_cache_size_id_for_level (level_and_inst_dat=0)<br>
> at<br>
> /home/niteesh/development/rtems/kernel/rtems/cpukit/score/cpu/arm/include/libcpu/arm-cp15.h:936<br>
> (gdb)<br>
> <br>
> They all are executing different instructions at the same time.<br>
<br>
Some of the initialization is done on all cores. Some isn't. I took a<br>
look at the initialization and it seems that I was wrong: There is no<br>
wait loop. All processors are running through the initialization<br>
process. Some just skip parts. The part where they really start to<br>
differ is in bsp_start_hook_0.<br>
<br>
> I> googled about just running one thread or CPU as you said at a time and<br>
> used "*set scheduler-locking on" *on doing this I always get the right<br>
> output. <br>
> <br>
> (gdb) info threads<br>
> Id Target Id Frame<br>
> * 1 Thread 1.1 (CPU#0 [running]) bsp_reset ()<br>
> at<br>
> ../../../../../../../../rtems/c/src/lib/libbsp/arm/raspberrypi/../../../../../../bsps/arm/raspberrypi/start/bspreset.c:18<br>
> 2 Thread 1.2 (CPU#1 [running]) _start ()<br>
> at<br>
> ../../../../../../../../rtems/c/src/lib/libbsp/arm/raspberrypi/../../../../../../bsps/arm/shared/start/start.S:153<br>
> 3 Thread 1.3 (CPU#2 [running]) _start ()<br>
> at<br>
> ../../../../../../../../rtems/c/src/lib/libbsp/arm/raspberrypi/../../../../../../bsps/arm/shared/start/start.S:153<br>
> 4 Thread 1.4 (CPU#3 [running]) _start ()<br>
> at<br>
> ../../../../../../../../rtems/c/src/lib/libbsp/arm/raspberrypi/../../../../../../bsps/arm/shared/start/start.S:153<br>
> (gdb)<br>
> <br>
> The above command allow's only a single thread to run.<br>
<br>
Maybe there is a timing difference between the simulator and the real<br>
hardware. I'm not sure how well tested the SMP code is on the Raspberry.<br>
There can be a hidden bug.<br>
<br>
Just a guess: If there is a bug it could be possible that you hit it<br>
with your rpi3 too. Maybe it would be good to try a single core version<br>
of the BSP. I assume you have configured with "--enable-smp"? Can you<br>
try to build without it?<br></blockquote><div>I built 2 versions with SMP enabled and disabled, the one we are talking about is the SMP disabled version, I ran</div><div>the example with SMP enabled, still, the error's are similar, I only difference is, in the disabled one, there are only 4 or less panic's</div><div>(maybe corresponding to 4 cpu's) but the other one has a higher number of panics.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
> Won't it be a good idea to make a separate BSP for rpi3?<br>
<br>
As soon as it is necessary: Sure. But from what you told me it seems<br>
that the hardware is very similar so that we won't hit this point soon.<br>
Or do you already see differences that would make it necessary.<br>
<br>
I haven't had a look at the details but it could also be possible to<br>
unify the BSPs and entirely remove the rpi2 variant if the information<br>
from the flattened device tree are used.<br></blockquote><div>Can you explain how FDT work in RTEMS. Can you mention some BSP's which use FDT so I can use them as a reference to learn.</div><div>I previously took a look at the beagle FDT project (#3784), you mentioned about hardcoded values and initialization functions, can</div><div>you explain more about what exactly do the initialization functions do? Do they assign a function to a particular pin, like in raspi</div><div>the pins are multiplexed for various functions, so do the initialization functions assign those pins to a particular function?</div><div><br></div><div>And also please explain how does the initialization of the system happen from the DT file.</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
> <br>
> > *** FATAL ***<br>
> > fatal source: 9 (RTEMS_FATAL_SOURCE_EXCEPTION)<br>
> ><br>
> > R0 = 0x400005e6 R8 = 0x00000000<br>
> > R1 = 0x00000001 R9 = 0x00000000<br>
> > R2 = 0xbffffa1a R10 = 0x00000000<br>
> > R3 = 0x00000000 R11 = 0x00000000<br>
> > R4 = 0x002001db R12 = 0x00000000<br>
> > R5 = 0x00000000 SP = 0x00300bd0<br>
> > R6 = 0x00000000 LR = 0x00100fc4<br>
> > R7 = 0x00000000 PC = 0x00100fc4<br>
> > CPSR = 0x000001d3 VEC = 0x00000002<br>
> > FPEXC = 0x40000000<br>
> > FPSCR = 0x00000000<br>
> > D00 = 0x0000000000000000<br>
> > D01 = 0x0000000000000000<br>
> > D02 = 0x0000000000000000<br>
> > D03 = 0x0000000000000000<br>
> > D04 = 0x0000000000000000<br>
> > D05 = 0x0000000000000000<br>
> > D06 = 0x0000000000000000<br>
> > D07 = 0x0000000000000000<br>
> > D08 = 0x0000000000000000<br>
> > D09 = 0x0000000000000000<br>
> > D10 = 0x0000000000000000<br>
> > D11 = 0x0000000000000000<br>
> > D12 = 0x0000000000000000<br>
> > D13 = 0x0000000000000000<br>
> > D14 = 0x0000000000000000<br>
> > D15 = 0x0000000000000000<br>
> > D16 = 0x0000000000000000<br>
> > D17 = 0x0000000000000010<br>
> > D18 = 0x0000000000000000<br>
> > D19 = 0x0000000000000000<br>
> > D20 = 0x0000000000000000<br>
> > D21 = 0x0000000000000000<br>
> > D22 = 0x0000000000000000<br>
> > D23 = 0x0000000000000000<br>
> > D24 = 0x0000000000000000<br>
> > D25 = 0x0000000000000000<br>
> > D26 = 0x0000000000000000<br>
> > D27 = 0x0000000000000000<br>
> > D28 = 0x0000000000000000<br>
> > D29 = 0x0000000000000000<br>
> > D30 = 0x0000000000000000<br>
> > D31 = 0x0000000000000000<br>
> > RTEMS version:<br>
> 5.0.0.c6d8589bb00a9d2a5a094c68c90290df1dc44807-modified<br>
> > RTEMS tools: 7.5.0 20191114 (RTEMS 5, RSB<br>
> > 83fa79314dd87c0a8c78fd642b2cea3138be8dd6, Newlib 3e24fbf6f)<br>
> > executing thread is NULL<br>
> ><br>
> > > The steps that I followed are:<br>
> > > 1. Created a bootable SD card using raspbian.<br>
> > > 2. Replaced the kernel.img file with RTEMS kernel.img file and<br>
> > modified<br>
> > > the config.txt to boot from the RTEMs kernel (boots in<br>
> aarch32 bit<br>
> > mode).<br>
> > > I am still not able to wrap my head around the RPI bsp build<br>
> process.<br>
> > > This is what I understood as of now, correct me if I am wrong. <br>
> > > Both RPi and Rpi2 are based on the same BSP, they just<br>
> differ in the<br>
> > > peripheral offsets, hardcoded checks are used to select the<br>
> right<br>
> > offset<br>
> > > at the time of compiling<br>
> ><br>
> > >From what I know of the Raspberry BSPs that is correct.<br>
> ><br>
> > > and the linkercmd file is responsible for<br>
> > > building the final executable file.<br>
> ><br>
> > The linkercmd file is - like for all programs - responsible<br>
> where the<br>
> > memory regions are that can be used for code or data. So you<br>
> could more<br>
> > or less explain it like you did.<br>
> ><br>
> > > I looked at the linker script, it seem's to have the start<br>
> section at<br>
> > > address 0x200000, I also loaded it in GDB and the start<br>
> address is<br>
> > > *Start address 0x200080,*<br>
> ><br>
> > I agree with that. The different start in GDB is most likely<br>
> because<br>
> > there is a vector table in front (at least if the Broadcom chip is<br>
> > similar to a lot of other processors that I have encountered).<br>
> ><br>
> > Does that mean that you have a debugger connected to the<br>
> raspberry? Can<br>
> > you load code with it? If yes: Is the bootloader executed<br>
> before you<br>
> > load your code? Otherwise the SDRAM might isn't initialized yet.<br>
> ><br>
> > I don't have a debugger connected to it. I from what I have SDRAM is<br>
> > initialized by the 3 stage bootloader(start.elf).<br>
> <br>
> That should be OK and it answers my question above.<br>
> <br>
> ><br>
> > > I did some bare metal programming on RPI3<br>
> > > there I had the start section at address 0x8000 Is this causing<br>
> > the problem?<br>
> ><br>
> > I assume that you used some internal RAM when you did bare metal<br>
> > programming. You maybe even skipped one or two bootloader<br>
> stages. From a<br>
> > quick look Raspberry has a quite complex boot process with at<br>
> least<br>
> > three bootloaders:<br>
> <a href="http://lions-wing.net/maker/raspberry-1/boot.html" rel="noreferrer" target="_blank">http://lions-wing.net/maker/raspberry-1/boot.html</a><br>
> ><br>
> > I don't think I have skipped any stages. The boot process is<br>
> exactly the<br>
> > same as how it boot's a normal raspbian or any other linux<br>
> > distro, I just to replace the linux kernel with my own kernel. <br>
> <br>
> Sounds reasonable. Does the bootloader print anything where it puts the<br>
> kernel image? Maybe the start address changed during the raspberry<br>
> versions.<br>
> <br>
> the default kernel load address is 0x8000 in 32bit mode and 0x80000 in<br>
> 64bit mode I have no idea about the raspberry 1,<br>
> but the load address is same for rpi2 and 3.<br>
<br>
That sounds odd. Do you have a memory map somewhere? From the linker<br>
command file it seems quite clear that RTEMS is build for a 0x200000.<br>
<br>
> <br>
> ><br>
> ><br>
> > > I have no idea on how to debug this, any suggestion on how<br>
> to start<br>
> > > would be really helpfull. <br>
> > ><br>
> ><br>
> <br>
</blockquote></div></div>