[GSoC - x86_64] Interrupt manager and and port-specific glue - was Re: [GSoC - x86_64 - automake] Limit CFLAGS to specific source for librtemsbsp.a

Wed Aug 1 16:50:35 UTC 2018

Thanks for the background!

Let's use the gen5200 as the ongoing example - my confusion arises
here (correct me if any of the following points is incorrect!). In
overly simplified call chains:

Register interrupt handler:
-  bsp_interrupt_facility_initialize() -> ppc_exc_set_handler(vec,
C_dispatch_irq_handler) -> ppc_exc_handler_table[vec] =
C_dispatch_irq_handler

Interrupt handler called:
- C_dispatch_irq_handler -> dispatch -> bsp_interrupt_handler_dispatch
(irq-generic.h) -> bsp_interrupt_handler_table[index].handler()

What I'm confused about is how the bsp_interrupt_handler_table is
updated at all - I just haven't found the link between how the entries
in the two tables are synchronized, where the tables are:

1) the ppc_exc_handler_table (the processor IDT) and
2) the bsp_interrupt_handler_table (the RTEMS interrupt table)

------------------------------------------------------------
Another similar chain of confusion for i386 is:
- rtems_interrupt_catch (intrcatch.c) -> _ISR_Install_vector (isr.h)
-> _CPU_ISR_install_vector (i386/idt.c) -> idt_entry_tbl[vector]
updated

But the i386 dispatch code chain is:
- _ISR_Handler (i386/irq_asm.S) -> BSP_dispatch_isr (i386/irq.c) ->
bsp_interrupt_handler_dispatch (irq-generic.h) ->
bsp_interrupt_handler_table[index].handler()

But I don't see any updates to bsp_interrupt_handler_table that would
let this work.
-------------------------------------------------------------

Would you happen to know what that "missing link" is?

On Wed, Aug 1, 2018 at 9:07 PM, Joel Sherrill <joel at rtems.org> wrote:
>
>
> On Wed, Aug 1, 2018 at 10:11 AM, Gedare Bloom <gedare at rtems.org> wrote:
>>
>> On Wed, Aug 1, 2018 at 9:15 AM, Amaan Cheval <amaan.cheval at gmail.com>
>> wrote:
>> > That's definitely very illuminating, thank you so much for all the
>> > details!
>> >
>> > A few more questions that have arisen for me. Feel free to skip over
>> > them (I'll likely figure them out given enough time, so I'm only
>> > asking in case any of them are obvious to anyone):
>> >
>> > - The i386 doesn't use CPU_Interrupt_frame at all. It seems like it
>> > stores some of the data onto the stack?
>> >
>> the interrupt frame structure was introduced during 4.11 development.
>> probably i386 never got updated to use a struct to encapsulate the
>> interrupt frame. the interrupt frame should contain the registers that
>> are preserved by the interrupt entry code I believe.
>
>
> +1
>
> Historically, there was no structure to represent the set of
> registers and information saved on interrupt entry. Over time
> this has been added.
>
> i386  also is missing the SMP synchronization check in the
> middle of the context which ensures it is safe for a thread to
> be migrated.
>
>>
>> > - There used to be defines in cpu.h regarding hardware/software based
>> > interrupt stacks, and how they'd be setup, which were made
>> > superfluous[1] - I'm not quite sure how these are meant to work - I
>> > see references to "stack high" and "stack low" and I'm not quite sure
>> > what the code is referencing when using those.
>> >
>>
>> a hardware interrupt stack is one that the hardware switches to during
>> an interrupt. i think m68k has such.
>>
>> most interrupt stacks in RTEMS are software-managed, meaning that
>> RTEMS explicitly switches the stack region off the task stack and to
>> an interrupt stack region.
>>
>> some stacks start high and grow down, and some stacks start low and
>> grow up. maybe this is what the "stack high" and "stack low" you
>> mention are in relation to?
>
>
> They are used to denote the top and bottom of the memory reserved
> for the interrupt stack. One important use is in
> cpukit/libmisc/stackchk/check.c
> to report on usage.
>
>>
>>
>> > - c/src/lib/libbsp/no_cpu/no_bsp/Makefile.am doesn't include
>> > irq-sources.am, by the way (this is part of why I used to think a lot
>> > of what your email mentioned was unnecessary, until you...ahem,
>> > pre-empted that line of thought and helped clarify it :P). Should I
>> > add a ticket to update the no_bsp code to be more in line with current
>> > use?
>> >
>> Sure. I don't know that anyone is in particular maintaining
>> no_cpu/no_bsp since we can't compile it, it is basically best effort
>> stuff that sometimes we miss updating.
>
>
> +1
>
> Also there are variations based on simple vectored and PIC vectored
> architectures.
>
> The architecture is responsible for the managing the minimal actions
> based on what the CPU does for an interrupt/exception. Logically
> a PIC is part of the BSP even if it comes integrated with the CPU as
> is common on x86 and SoCs. The PIC could fairly easily change
> and the architecture is harder to change.
>
>>
>>
>> > - My understanding of _ISR_Handler is that it'll be the handler for
>> > _all_ interrupt vectors by default - it'll then dispatch interrupts to
>> > user-handlers (or internal handlers, for the timer, for eg.). Is that
>> > right? (I don't quite understand its interaction with the RTEMS
>> > interrupt manager yet, but irq-generic's "bsp_interrupt_handler_table"
>> > seems to be the RTEMS equivalent to the processor-specific vector
>> > table, and "bsp_interrupt_handler_dispatch" seems to call the actual
>> > handler within that table as appropriate. Accurate? (I just haven't
>> > found how that table actually gets its handlers setup besides during
>> > initialization, since rtems_interrupt_catch just calls
>> > _CPU_install_vector, which updates the processor vector table, not the
>> > RTEMS interrupt manager vector table.)
>> >
>>
>> You have discovered a couple of different but related interrupt
>> processing code bases.  I can see why you get confused.
>>
>> Basically, a CPU port should support two kinds of interrupts that may
>> be installed, "RTEMS" and "Raw" interrupts. The "Raw" interrupts are
>> installed directly in the processor's vector table. For processors
>> that use simple vectored interrupts, the "RTEMS" interrupts install a
>> call to the _ISR_Handler() function in the processor's vector table,
>> and will put the user's isr function pointer in the
>> _ISR_Vector_table(), which is the RTEMS Interrupt Manager's vector
>> table.
>>
>> I'm not really familiar with the processors that use a different model
>> for interrupt handling besides simple vectored. Probably, you will
>> have to study one of them.
>>
>> This irq-generic bsp_interrupt_* code is used by the "IRQ Server" that
>> builds from the CPU port capabilities to allow for some advanced
>> features like chaining multiple isrs from the same source. I don't
>> think you want to be focusing on those right now, but I could be
>> mistaken. I haven't spent much time looking at the IRQ Server
>> codebase.
>
>
> For the most part, the bsp_interrupt_* code is hung off of one
> (or more) of the simple vectored sources. On many architectures,
> you have very few "CPU vectors" and the bsp_interrupt_* is
> the secondary layer managing the PIC. Notice that it is all
> in C and thus isn't first level processing. And it doesn't deal
> with RTEMS critical section, interrupt entry/exit, and dispatching..
>
> For example, in its original form, the PowerPC didn't have many
> "exceptions" (e.g. CPU vectors) at all. In fact, there was a single
> External Exception which was always tied to a board dependent
> PIC. The first level processing of PPC exceptions is in assembly
> and deals with what I pointed you to in various _ISR_Handler's.
> The External Exception handler now process bsp_interrupt_*.
> The gen5200 BSP irq.c file has a bsp_interrupt_* implementation
> which registers for two "exceptions" which both go to
> C_dispatch_irq_handler() which reads the PIC, determines
> the source(s), and dispatches (via bsp_interrupt_handler_dispatch()
> the registered IRQ.
>
> If you think of this as two levels of processing, it should help.
>
> The two levels of processing is for RTEMS interrupts which can
> unblock threads, etc. The first level ensures that locking and
> dispatching is handled.
>
> As Gedare stated, there are also Raw interrupts which ignore
> RTEMS and can't use RTEMS services. These should never
> be anything but the highest priority interrupt in a system. The
> RTEMS first level processing can switch to a dedicated
> interrupt stack and mixing RTEMS and Raw IRQs can
> screw up the stack handling as well as the interrupt nesting
> count.
>
> --joel
>
>>
>>
>> Gedare
>>
>> > - My understanding of the interaction between RTEMS' interrupt manager
>> > (i.e. support for nested interrupts and thread dispatch once an
>> > interrupt ends) and the BSP's processor-specific interrupt manager
>> > (code to use the APIC and IDT in my case) is that they're tied
>> > together through the use of irq-generic.c's "bsp_interrupt_initialize"
>> > - is that right? (m68k never seems to call it, though, so perhaps
>> > not?)
>> >
>> > Sorry about the rambling! To reiterate, I'll likely figure it out
>> > given enough time, so if the answers aren't at the top of your head, I
>> > can figure it out without wasting your time :)
>> >
>> > [1] https://devel.rtems.org/ticket/3459#comment:11
>> >
>> > On Wed, Aug 1, 2018 at 3:18 AM, Joel Sherrill <joel at rtems.org> wrote:
>> >>
>> >>
>> >> On Tue, Jul 31, 2018 at 3:05 PM, Amaan Cheval <amaan.cheval at gmail.com>
>> >> wrote:
>> >>>
>> >>> Hm, I'm not sure what to look for in the other ports specifically,
>> >>> really.
>> >>> The BSP porting documentation doesn't have a section on interrupts, so
>> >>> I'm
>> >>> doing this on more of an "as it comes up" basis.
>> >>>
>> >>> What I've got right now (the interrupt handlers in C) are what I need
>> >>> for
>> >>> calibrating the APIC timer (through the PIT) - so simply hooking IRQ0
>> >>> (for
>> >>> the timer) and IRQ7 (spurious vector), since those are needed for the
>> >>> timer
>> >>> work to continue.
>> >>>
>> >>> What constitutes as a requirement for basic interrupt support?
>> >>
>> >>
>> >> There used to be a generic porting guide. I can see that this
>> >> particular
>> >> section
>> >> has bit rotted some but the interrupt dispatching section. Some of this
>> >> will have evolved to support SMP and fine grained locking but the
>> >> pseudo-code
>> >> here will give you a push toward the right line of thinking:
>> >>
>> >>
>> >> https://docs.rtems.org/releases/rtemsdocs-4.10.2/share/rtems/html/porting/porting00034.html
>> >>
>> >> The idea is that you need to ensure RTEMS knows it is inside an
>> >> interrupt
>> >> and the current locking scheme (old was dispatching, new is ...) is
>> >> honored.
>> >>
>> >> The ARM and PowerPC (plus RISCV) are good ports to look at for how SMP
>> >> plays into this. But the CPU supplement is thin for their interrupt
>> >> processing.
>> >>
>> >>
>> >> This is the CPU Architecture supplement section for the m68k. This is a
>> >> relatively simple
>> >> architecture to describe. There is also a section for the i386 which
>> >> reads
>> >> similarly.
>> >>
>> >>
>> >> https://docs.rtems.org/branches/master/cpu-supplement/m68xxx_and_coldfire.html#interrupt-processing
>> >>
>> >> Personally, I find the m68k a fairly easy processor to read assembly
>> >> in.
>> >> Look at cpukit/score/cpu/m68k/cpu_asm.S and _ISR_Handler to see what
>> >> is done there w/o SMP. On the m68k _ISR_Handler is directly put into
>> >> the
>> >> vector table. But this isn't the most similar example for you.
>> >>
>> >> For the i386 (better example), it is in bsps/i386/shared/irq/irq_asm.S
>> >> with
>> >> the
>> >> same name. There _ISR_Handler is installed via the
>> >> DISTINCT_INTERRUPT_ENTRY
>> >> macros at the bottom of the file where some prologue jumps to the
>> >> common
>> >> _ISR_Handler and then the actions are similar. Usually _ISR_Handler
>> >> type of
>> >> code ends up invoking a PIC decode method in normal C without an
>> >> interrupt attribute.
>> >>
>> >> Long and multi-architecture answer but maybe that makes sense. The goal
>> >> in ticker.exe is to take a number of tick interrupts which don't
>> >> schedule
>> >> and
>> >> then take one that does -- it schedules a preemption of the idle
>> >> thread.
>> >>
>> >> Hope this helps.
>> >>
>> >> --joel
>> >>
>> >>
>> >>
>> >>>
>> >>>
>> >>> On Wed, Aug 1, 2018, 1:29 AM Joel Sherrill <joel at rtems.org> wrote:
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Tue, Jul 31, 2018 at 2:52 PM, Amaan Cheval
>> >>>> <amaan.cheval at gmail.com>
>> >>>> wrote:
>> >>>>>
>> >>>>> Hi Chris!
>> >>>>>
>> >>>>> I currently have code like this in
>> >>>>> c/src/lib/libbsp/x86_64/amd64/Makefile.am:
>> >>>>>
>> >>>>>     librtemsbsp_a_SOURCES +=
>> >>>>> ../../../../../../bsps/x86_64/amd64/interrupts/handlers.c
>> >>>>>     # XXX: Needed to use GCC "interrupt" attribute directives - can
>> >>>>> we
>> >>>>> pass these
>> >>>>>     # flags only for the handlers.c source file (compile to an
>> >>>>> object
>> >>>>> file first and
>> >>>>>     # then link with the rest for librtemsbsp.a?)
>> >>>>>     librtemsbsp_a_CFLAGS = -mgeneral-regs-only
>> >>>>>
>> >>>>> The CFLAGS arg is required to allow us to use
>> >>>>> "__attribute__((interrupt))" to setup interrupt handlers in C. (See
>> >>>>> [1] and ctrl+f "interrupt" for more.)
>> >>>>>
>> >>>>> Is there a way to not force the CFLAGS for _all_ of librtemsbsp, but
>> >>>>> to limit it only to handlers.c?
>> >>>>>
>> >>>>> If not, is the above code something that would be acceptable to have
>> >>>>> upstream?
>> >>>>>
>> >>>>> [1]
>> >>>>>
>> >>>>> https://gcc.gnu.org/onlinedocs/gcc/x86-Function-Attributes.html#x86-Function-Attributes
>> >>>>
>> >>>>
>> >>>> Are we basically talking about the outermost layer of your interrupt
>> >>>> dispatching?
>> >>>>
>> >>>>
>> >>>> Have you looked at the basic approach taken by the other ports? They
>> >>>> end
>> >>>> up switching the stack pointer to a dedicated stack on the outermost
>> >>>> interrupt
>> >>>> and, if a context switch/dispatch is needed, arrange for the
>> >>>> interrupted
>> >>>> task to call _Thread_Dispatch.But tinker with its stack so some
>> >>>> registers
>> >>>> are saved and it looks like it made the call itself.
>> >>>>
>> >>>> If you can do it in C, I am ok with an attribute. I just don't think
>> >>>> you
>> >>>> can pull off all the stack and return to dispatch magic that way.
>> >>>>
>> >>>> --joel
>> >>>>
>> >>
>
>