i386 SMP buils - Was: libchip/network/if_fxp.c: do not use rtems_interrupt_disable.

[Pavel Pisa]

Hello Chris,

On Monday 17 of October 2016 03:00:56 Chris Johns wrote:
> I have built and tested these patches with no issues and 0 spurious
> interrupts.
>
> The changes look fine to me.

Thanks much for testing. I have pushed changes to the master.

I have spent considerable time over weekend by debugging
i386 SMP build but without success.

I have found and fixed some problems, I get to the state when
two CPUs runs under QEMU and even IPI is exchanged but then
execution finish by different kinds of fatal exceptions.

On the other hand, SMP build when only one CPU is enabled
runs stable under QEMU.

Some found issues and corrections tested with

  CONFIGURE_SMP_MAXIMUM_PROCESSORS  2

According to the more sources, INIT and SIPI duplication
is not required on modern CPUs. INIT is ignored so no problem,
but if the second SIPI is delayed too much then the first
one causes complete start of the second core reaching RTEMS
secondary_cpu_initialize() and then next SIPI resets the core
and it starts again so you count CPU twice. The specification
is strict about 200usec between two SIPI. But according to
some discussion that second SIPI is not required on modern systems.
Next modification tries bring up the second core by single SIPI
and if the core does not reach RTEMS code in really long time
it expect old system and restries the second SIPI.
Probably LAPIC version can be used there to switch this workaround
for newer cores at all. 

diff --combined c/src/lib/libbsp/i386/shared/smp/smp-imps.c
--- a/c/src/lib/libbsp/i386/shared/smp/smp-imps.c
+++ b/c/src/lib/libbsp/i386/shared/smp/smp-imps.c
@@@ -301,12 -301,12 +334,21 @@@
     */
  
    if (proc->apic_ver >= APIC_VER_NEW) {
--    int i;
--    for (i = 1; i <= 2; i++) {
++    int retry = 2;
++    while (retry) {
++      int wait_for_ap = 100;
++
++      printk("boot_cpu sending SIPI\n");
++
        send_ipi(apicid, LAPIC_ICR_DM_SIPI | ((bootaddr >> 12) & 0xFF));
--      UDELAY(1000);
++      do {
++        UDELAY(10);
++        if (acked_ap_cpus != _Atomic_Load_uint(&imps_ap_cpus_acked, ATOMIC_ORDER_SEQ_CST))
++          retry = 0;
++      } while(wait_for_ap--);
      }
    }
++  printk("boot_cpu secondary acked\n");
  
    /*
     *  Generic CPU startup sequence ends here, the rest is cleanup.

Another strange thing for SMP is, that secondary CPUs continues
to run with temporary GDT set by CPU SIPI trampoline code at 0x70000.
So I have added function to setup the secondary CPU the same GDT
as is used on the primary

diff --combined c/src/lib/libbsp/i386/shared/smp/smp-imps.c
--- a/c/src/lib/libbsp/i386/shared/smp/smp-imps.c
+++ b/c/src/lib/libbsp/i386/shared/smp/smp-imps.c
@@@ -785,7 -785,7 +829,7 @@@ static void secondary_cpu_initialize(vo
  {
    int apicid;
  
--  asm volatile( "lidt IDT_Descriptor" );
++  _load_segments_secondary();
  
    apicid = IMPS_LAPIC_READ(LAPIC_SPIV);
    IMPS_LAPIC_WRITE(LAPIC_SPIV, apicid|LAPIC_SPIV_ENABLE_APIC);
@@@ -794,6 -794,6 +838,8 @@@
    enable_sse();
  #endif
  
++  _Atomic_Fetch_add_uint(&imps_ap_cpus_acked, 1, ATOMIC_ORDER_SEQ_CST);
++
    _SMP_Start_multitasking_on_secondary_processor();
  }

+++ b/c/src/lib/libbsp/i386/pc386/startup/ldsegs.S
++
++	.p2align 4
++
++        PUBLIC ( _load_segments_secondary)
++SYM (_load_segments_secondary):
++
++	lgdt SYM(gdtdesc)
++	lidt SYM(IDT_Descriptor)
++
++	/* Load CS, flush prefetched queue */
++	ljmp $0x8, $next_step_secondary
++
++next_step_secondary:
++        /* Load segment registers */
++	movw $0x10, ax
++	movw ax, ss
++	movw ax, ds
++	movw ax, es
++	movw ax, fs
++	movw ax, gs
++	ret

As for GDT, I am not sure if one GDT for all CPUs can work.
Problem is TSS. RTEMS runs in ring 0 only, so TSS is not required
to switch between ring 3 and ring 0 but I am not sure if there
is not some situation when TSS is accessed anyway. The additional
TSS is required for sure if we want to catch double fault exception.

To be on safe side, it would worth to move user descriptors
to LDT, left global GDT to point to LDT and area for primary
CPU TSS and allocate new GDT and TSS for each CPU during bringup.

Generally, I am not expert for x86 and do not like this area much
so if some student or somebody else wants to exercise in this
remains of dark age, I would be happy and help with some hints.

On the other hand, RTEMS should provide reasonable x6 SMP
support for testing and may it be Jailhouse or other interresting
hypervisor base divisions to RT and generic OS domains.
Other option is to left i386 SMP support unsolved and add
x86_64 support which skips many of these oddities like TSS and GDT.
(OK GDT is required but only that simple short lived in the trampoline
is enough).

Best wishes,

Pavel