[PATCH] arm/cortex-a: Fix cache flush/invalidate after u-boot.

[Pavel Pisa]

Hello Chris,

On Monday 15 of August 2016 07:30:56 Chris Johns wrote:
> This is a copy of the patch from Pavel to fix some strange behaviour with
> data cache, instruction cache and MMU being enabled by u-boot on the
> RaspberryPi.
>
> Closes #2774.

My code can have issues with SMP. If the arm_a9mpcore_start_hook_0
is called by secondary CPU then only local caches should be fully
invalidated.

So to be on safe side I suggest following changes to run
destructive invalidate only on CPU #0. May it be 
  rtems_cache_flush_entire_data
  rtems_cache_invalidate_entire_data
  rtems_cache_invalidate_entire_instruction
should be completely replaced for (cpu_id == 0) else block
by arm-cache-l1.h :

  arm_cache_l1_clean_and_invalidate_entire_data
  arm_cache_l1_invalidate_entire_instruction

which do not reach shared L2 cache and are guaranteed to not
disturb other cores operation.

If you test that chage is OK for Zynq with SMP then I reintroduce
change to RPi code (it would worth to have RPi2 SMP working for that
testing but that is near to bottom of my own TODO list).

> > May it be extend
> >
> >    rtems/c/src/lib/libbsp/arm/xilinx-zynq/configure.ac
> >
> > with
> >
> >    RTEMS_BSPOPTS_SET([BSP_START_IN_HYP_SUPPORT],[*],[1])
> >    RTEMS_BSPOPTS_HELP([BSP_START_IN_HYP_SUPPORT], [Support start of BSP
> > in ARM HYP mode]) AM_CONDITIONAL(BSP_START_IN_HYP_SUPPORT,test
> > "$BSP_START_IN_HYP_SUPPORT" = "1")
> >
> > to support possible start in HYPervisor mode as well
>
> I saw this change but I do not fully understand the issues are that are
> being solved. I know recent RPi's enter in hypervisor mode. What boot
> loader does the RPi have?
>
> I saw this change but I do not fully understand the issues are that are
> being solved. I know recent RPi's enter in hypervisor mode. What boot
> loader does the RPi have?

It seems that recent RPi firmware enters kernel image (RTEMS application)
in HYP mode. Even if U-boot is used then it is enterred in HYP mode
and it pass control to loaded application in HYP mode.
RTEMS startup with BSP_START_IN_HYP_SUPPORT == 1 is implemented such
way that it detect CPU state and if HYP startup is used it
switches back to SVC. There should be no issues with loaders
starting in SVC as well as witch ARM architecture variants which
do not support HYP. But as for anything else, it worth to be
checked.

>  .../libbsp/arm/shared/include/arm-a9mpcore-start.h | 29
> ++++++++++++++++++++++ 1 file changed, 29 insertions(+)
>
> diff --git a/c/src/lib/libbsp/arm/shared/include/arm-a9mpcore-start.h
> b/c/src/lib/libbsp/arm/shared/include/arm-a9mpcore-start.h index
> 7d6185b..08a4d7b 100644
> --- a/c/src/lib/libbsp/arm/shared/include/arm-a9mpcore-start.h
> +++ b/c/src/lib/libbsp/arm/shared/include/arm-a9mpcore-start.h
> @@ -129,8 +129,37 @@ BSP_START_TEXT_SECTION static inline void
> arm_a9mpcore_start_hook_0(void) volatile a9mpcore_scu *scu =
>      (volatile a9mpcore_scu *) BSP_ARM_A9MPCORE_SCU_BASE;
>    uint32_t cpu_id = arm_cortex_a9_get_multiprocessor_cpu_id();
> +  uint32_t sctlr_val;
>
> +  sctlr_val = arm_cp15_get_control();
> +
> +  /*
> +   * Current U-boot loader seems to start kernel image
> +   * with I and D caches on and MMU enabled.
> +   * If RTEMS application image finds that cache is on
> +   * during startup then disable caches.
> +   */
> +  if (sctlr_val & (ARM_CP15_CTRL_I | ARM_CP15_CTRL_C | ARM_CP15_CTRL_M)) {
> +    if (sctlr_val & (ARM_CP15_CTRL_C | ARM_CP15_CTRL_M)) {
> +      /*
> +       * If the data cache is on then ensure that it is clean
> +       * before switching off to be extra carefull.
> +       */
        if (cpu_id == 0) {
> +      rtems_cache_flush_entire_data();
> +      rtems_cache_invalidate_entire_data();
        } else {
          arm_cache_l1_clean_and_invalidate_entire_data();
        }

> +    }
> +    arm_cp15_flush_prefetch_buffer();
> +    sctlr_val &= ~(ARM_CP15_CTRL_I | ARM_CP15_CTRL_C | ARM_CP15_CTRL_M |
> ARM_CP15_CTRL_A); +    arm_cp15_set_control(sctlr_val);
> +  }
     if (cpu_id == 0) {
> +    rtems_cache_invalidate_entire_data();
> +    rtems_cache_invalidate_entire_instruction();
     } else {
       arm_cache_l1_invalidate_entire_data();
       arm_cache_l1_invalidate_entire_instruction();
     }
>    arm_cp15_branch_predictor_invalidate_all();
> +  arm_cp15_tlb_invalidate();
> +  arm_cp15_flush_prefetch_buffer();
> +
> +  /* Clear Translation Table Base Control Register */
> +  arm_cp15_set_translation_table_base_control_register(0);
>
>    if (cpu_id == 0) {
>      arm_a9mpcore_start_scu_enable(scu);