[GSoC - x86_64] Interrupt manager and and port-specific glue - was Re: [GSoC - x86_64 - automake] Limit CFLAGS to specific source for librtemsbsp.a

[Amaan Cheval]

Okay, I think I understand finally. Sorry about the rambling!

When rtems_interrupt_catch is called, that's installing a "raw" ISR by
modifying the processor specific table itself, so _ISR_Handler is
never called, but the user ISR is.

When rtems_interrupt_handler_install is called, that's an "RTEMS
interrupt", and we go through the _ISR_Handler -> dispatch route I
laid out earlier, leading to eventually the user's ISR.

Thank you for letting me rubber-duck with you, everyone (let me know
if anything above sounds off, though!) :P

On Wed, Aug 1, 2018 at 10:20 PM, Amaan Cheval <amaan.cheval at gmail.com> wrote:
> Thanks for the background!
>
> Let's use the gen5200 as the ongoing example - my confusion arises
> here (correct me if any of the following points is incorrect!). In
> overly simplified call chains:
>
> Register interrupt handler:
> -  bsp_interrupt_facility_initialize() -> ppc_exc_set_handler(vec,
> C_dispatch_irq_handler) -> ppc_exc_handler_table[vec] =
> C_dispatch_irq_handler
>
> Interrupt handler called:
> - C_dispatch_irq_handler -> dispatch -> bsp_interrupt_handler_dispatch
> (irq-generic.h) -> bsp_interrupt_handler_table[index].handler()
>
> What I'm confused about is how the bsp_interrupt_handler_table is
> updated at all - I just haven't found the link between how the entries
> in the two tables are synchronized, where the tables are:
>
> 1) the ppc_exc_handler_table (the processor IDT) and
> 2) the bsp_interrupt_handler_table (the RTEMS interrupt table)
>
> ------------------------------------------------------------
> Another similar chain of confusion for i386 is:
> - rtems_interrupt_catch (intrcatch.c) -> _ISR_Install_vector (isr.h)
> -> _CPU_ISR_install_vector (i386/idt.c) -> idt_entry_tbl[vector]
> updated
>
> But the i386 dispatch code chain is:
> - _ISR_Handler (i386/irq_asm.S) -> BSP_dispatch_isr (i386/irq.c) ->
> bsp_interrupt_handler_dispatch (irq-generic.h) ->
> bsp_interrupt_handler_table[index].handler()
>
> But I don't see any updates to bsp_interrupt_handler_table that would
> let this work.
> -------------------------------------------------------------
>
> Would you happen to know what that "missing link" is?
>
> On Wed, Aug 1, 2018 at 9:07 PM, Joel Sherrill <joel at rtems.org> wrote:
>>
>>
>> On Wed, Aug 1, 2018 at 10:11 AM, Gedare Bloom <gedare at rtems.org> wrote:
>>>
>>> On Wed, Aug 1, 2018 at 9:15 AM, Amaan Cheval <amaan.cheval at gmail.com>
>>> wrote:
>>> > That's definitely very illuminating, thank you so much for all the
>>> > details!
>>> >
>>> > A few more questions that have arisen for me. Feel free to skip over
>>> > them (I'll likely figure them out given enough time, so I'm only
>>> > asking in case any of them are obvious to anyone):
>>> >
>>> > - The i386 doesn't use CPU_Interrupt_frame at all. It seems like it
>>> > stores some of the data onto the stack?
>>> >
>>> the interrupt frame structure was introduced during 4.11 development.
>>> probably i386 never got updated to use a struct to encapsulate the
>>> interrupt frame. the interrupt frame should contain the registers that
>>> are preserved by the interrupt entry code I believe.
>>
>>
>> +1
>>
>> Historically, there was no structure to represent the set of
>> registers and information saved on interrupt entry. Over time
>> this has been added.
>>
>> i386  also is missing the SMP synchronization check in the
>> middle of the context which ensures it is safe for a thread to
>> be migrated.
>>
>>>
>>> > - There used to be defines in cpu.h regarding hardware/software based
>>> > interrupt stacks, and how they'd be setup, which were made
>>> > superfluous[1] - I'm not quite sure how these are meant to work - I
>>> > see references to "stack high" and "stack low" and I'm not quite sure
>>> > what the code is referencing when using those.
>>> >
>>>
>>> a hardware interrupt stack is one that the hardware switches to during
>>> an interrupt. i think m68k has such.
>>>
>>> most interrupt stacks in RTEMS are software-managed, meaning that
>>> RTEMS explicitly switches the stack region off the task stack and to
>>> an interrupt stack region.
>>>
>>> some stacks start high and grow down, and some stacks start low and
>>> grow up. maybe this is what the "stack high" and "stack low" you
>>> mention are in relation to?
>>
>>
>> They are used to denote the top and bottom of the memory reserved
>> for the interrupt stack. One important use is in
>> cpukit/libmisc/stackchk/check.c
>> to report on usage.
>>
>>>
>>>
>>> > - c/src/lib/libbsp/no_cpu/no_bsp/Makefile.am doesn't include
>>> > irq-sources.am, by the way (this is part of why I used to think a lot
>>> > of what your email mentioned was unnecessary, until you...ahem,
>>> > pre-empted that line of thought and helped clarify it :P). Should I
>>> > add a ticket to update the no_bsp code to be more in line with current
>>> > use?
>>> >
>>> Sure. I don't know that anyone is in particular maintaining
>>> no_cpu/no_bsp since we can't compile it, it is basically best effort
>>> stuff that sometimes we miss updating.
>>
>>
>> +1
>>
>> Also there are variations based on simple vectored and PIC vectored
>> architectures.
>>
>> The architecture is responsible for the managing the minimal actions
>> based on what the CPU does for an interrupt/exception. Logically
>> a PIC is part of the BSP even if it comes integrated with the CPU as
>> is common on x86 and SoCs. The PIC could fairly easily change
>> and the architecture is harder to change.
>>
>>>
>>>
>>> > - My understanding of _ISR_Handler is that it'll be the handler for
>>> > _all_ interrupt vectors by default - it'll then dispatch interrupts to
>>> > user-handlers (or internal handlers, for the timer, for eg.). Is that
>>> > right? (I don't quite understand its interaction with the RTEMS
>>> > interrupt manager yet, but irq-generic's "bsp_interrupt_handler_table"
>>> > seems to be the RTEMS equivalent to the processor-specific vector
>>> > table, and "bsp_interrupt_handler_dispatch" seems to call the actual
>>> > handler within that table as appropriate. Accurate? (I just haven't
>>> > found how that table actually gets its handlers setup besides during
>>> > initialization, since rtems_interrupt_catch just calls
>>> > _CPU_install_vector, which updates the processor vector table, not the
>>> > RTEMS interrupt manager vector table.)
>>> >
>>>
>>> You have discovered a couple of different but related interrupt
>>> processing code bases.  I can see why you get confused.
>>>
>>> Basically, a CPU port should support two kinds of interrupts that may
>>> be installed, "RTEMS" and "Raw" interrupts. The "Raw" interrupts are
>>> installed directly in the processor's vector table. For processors
>>> that use simple vectored interrupts, the "RTEMS" interrupts install a
>>> call to the _ISR_Handler() function in the processor's vector table,
>>> and will put the user's isr function pointer in the
>>> _ISR_Vector_table(), which is the RTEMS Interrupt Manager's vector
>>> table.
>>>
>>> I'm not really familiar with the processors that use a different model
>>> for interrupt handling besides simple vectored. Probably, you will
>>> have to study one of them.
>>>
>>> This irq-generic bsp_interrupt_* code is used by the "IRQ Server" that
>>> builds from the CPU port capabilities to allow for some advanced
>>> features like chaining multiple isrs from the same source. I don't
>>> think you want to be focusing on those right now, but I could be
>>> mistaken. I haven't spent much time looking at the IRQ Server
>>> codebase.
>>
>>
>> For the most part, the bsp_interrupt_* code is hung off of one
>> (or more) of the simple vectored sources. On many architectures,
>> you have very few "CPU vectors" and the bsp_interrupt_* is
>> the secondary layer managing the PIC. Notice that it is all
>> in C and thus isn't first level processing. And it doesn't deal
>> with RTEMS critical section, interrupt entry/exit, and dispatching..
>>
>> For example, in its original form, the PowerPC didn't have many
>> "exceptions" (e.g. CPU vectors) at all. In fact, there was a single
>> External Exception which was always tied to a board dependent
>> PIC. The first level processing of PPC exceptions is in assembly
>> and deals with what I pointed you to in various _ISR_Handler's.
>> The External Exception handler now process bsp_interrupt_*.
>> The gen5200 BSP irq.c file has a bsp_interrupt_* implementation
>> which registers for two "exceptions" which both go to
>> C_dispatch_irq_handler() which reads the PIC, determines
>> the source(s), and dispatches (via bsp_interrupt_handler_dispatch()
>> the registered IRQ.
>>
>> If you think of this as two levels of processing, it should help.
>>
>> The two levels of processing is for RTEMS interrupts which can
>> unblock threads, etc. The first level ensures that locking and
>> dispatching is handled.
>>
>> As Gedare stated, there are also Raw interrupts which ignore
>> RTEMS and can't use RTEMS services. These should never
>> be anything but the highest priority interrupt in a system. The
>> RTEMS first level processing can switch to a dedicated
>> interrupt stack and mixing RTEMS and Raw IRQs can
>> screw up the stack handling as well as the interrupt nesting
>> count.
>>
>> --joel
>>
>>>
>>>
>>> Gedare
>>>
>>> > - My understanding of the interaction between RTEMS' interrupt manager
>>> > (i.e. support for nested interrupts and thread dispatch once an
>>> > interrupt ends) and the BSP's processor-specific interrupt manager
>>> > (code to use the APIC and IDT in my case) is that they're tied
>>> > together through the use of irq-generic.c's "bsp_interrupt_initialize"
>>> > - is that right? (m68k never seems to call it, though, so perhaps
>>> > not?)
>>> >
>>> > Sorry about the rambling! To reiterate, I'll likely figure it out
>>> > given enough time, so if the answers aren't at the top of your head, I
>>> > can figure it out without wasting your time :)
>>> >
>>> > [1] https://devel.rtems.org/ticket/3459#comment:11
>>> >
>>> > On Wed, Aug 1, 2018 at 3:18 AM, Joel Sherrill <joel at rtems.org> wrote:
>>> >>
>>> >>
>>> >> On Tue, Jul 31, 2018 at 3:05 PM, Amaan Cheval <amaan.cheval at gmail.com>
>>> >> wrote:
>>> >>>
>>> >>> Hm, I'm not sure what to look for in the other ports specifically,
>>> >>> really.
>>> >>> The BSP porting documentation doesn't have a section on interrupts, so
>>> >>> I'm
>>> >>> doing this on more of an "as it comes up" basis.
>>> >>>
>>> >>> What I've got right now (the interrupt handlers in C) are what I need
>>> >>> for
>>> >>> calibrating the APIC timer (through the PIT) - so simply hooking IRQ0
>>> >>> (for
>>> >>> the timer) and IRQ7 (spurious vector), since those are needed for the
>>> >>> timer
>>> >>> work to continue.
>>> >>>
>>> >>> What constitutes as a requirement for basic interrupt support?
>>> >>
>>> >>
>>> >> There used to be a generic porting guide. I can see that this
>>> >> particular
>>> >> section
>>> >> has bit rotted some but the interrupt dispatching section. Some of this
>>> >> will have evolved to support SMP and fine grained locking but the
>>> >> pseudo-code
>>> >> here will give you a push toward the right line of thinking:
>>> >>
>>> >>
>>> >> https://docs.rtems.org/releases/rtemsdocs-4.10.2/share/rtems/html/porting/porting00034.html
>>> >>
>>> >> The idea is that you need to ensure RTEMS knows it is inside an
>>> >> interrupt
>>> >> and the current locking scheme (old was dispatching, new is ...) is
>>> >> honored.
>>> >>
>>> >> The ARM and PowerPC (plus RISCV) are good ports to look at for how SMP
>>> >> plays into this. But the CPU supplement is thin for their interrupt
>>> >> processing.
>>> >>
>>> >>
>>> >> This is the CPU Architecture supplement section for the m68k. This is a
>>> >> relatively simple
>>> >> architecture to describe. There is also a section for the i386 which
>>> >> reads
>>> >> similarly.
>>> >>
>>> >>
>>> >> https://docs.rtems.org/branches/master/cpu-supplement/m68xxx_and_coldfire.html#interrupt-processing
>>> >>
>>> >> Personally, I find the m68k a fairly easy processor to read assembly
>>> >> in.
>>> >> Look at cpukit/score/cpu/m68k/cpu_asm.S and _ISR_Handler to see what
>>> >> is done there w/o SMP. On the m68k _ISR_Handler is directly put into
>>> >> the
>>> >> vector table. But this isn't the most similar example for you.
>>> >>
>>> >> For the i386 (better example), it is in bsps/i386/shared/irq/irq_asm.S
>>> >> with
>>> >> the
>>> >> same name. There _ISR_Handler is installed via the
>>> >> DISTINCT_INTERRUPT_ENTRY
>>> >> macros at the bottom of the file where some prologue jumps to the
>>> >> common
>>> >> _ISR_Handler and then the actions are similar. Usually _ISR_Handler
>>> >> type of
>>> >> code ends up invoking a PIC decode method in normal C without an
>>> >> interrupt attribute.
>>> >>
>>> >> Long and multi-architecture answer but maybe that makes sense. The goal
>>> >> in ticker.exe is to take a number of tick interrupts which don't
>>> >> schedule
>>> >> and
>>> >> then take one that does -- it schedules a preemption of the idle
>>> >> thread.
>>> >>
>>> >> Hope this helps.
>>> >>
>>> >> --joel
>>> >>
>>> >>
>>> >>
>>> >>>
>>> >>>
>>> >>> On Wed, Aug 1, 2018, 1:29 AM Joel Sherrill <joel at rtems.org> wrote:
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> On Tue, Jul 31, 2018 at 2:52 PM, Amaan Cheval
>>> >>>> <amaan.cheval at gmail.com>
>>> >>>> wrote:
>>> >>>>>
>>> >>>>> Hi Chris!
>>> >>>>>
>>> >>>>> I currently have code like this in
>>> >>>>> c/src/lib/libbsp/x86_64/amd64/Makefile.am:
>>> >>>>>
>>> >>>>>     librtemsbsp_a_SOURCES +=
>>> >>>>> ../../../../../../bsps/x86_64/amd64/interrupts/handlers.c
>>> >>>>>     # XXX: Needed to use GCC "interrupt" attribute directives - can
>>> >>>>> we
>>> >>>>> pass these
>>> >>>>>     # flags only for the handlers.c source file (compile to an
>>> >>>>> object
>>> >>>>> file first and
>>> >>>>>     # then link with the rest for librtemsbsp.a?)
>>> >>>>>     librtemsbsp_a_CFLAGS = -mgeneral-regs-only
>>> >>>>>
>>> >>>>> The CFLAGS arg is required to allow us to use
>>> >>>>> "__attribute__((interrupt))" to setup interrupt handlers in C. (See
>>> >>>>> [1] and ctrl+f "interrupt" for more.)
>>> >>>>>
>>> >>>>> Is there a way to not force the CFLAGS for _all_ of librtemsbsp, but
>>> >>>>> to limit it only to handlers.c?
>>> >>>>>
>>> >>>>> If not, is the above code something that would be acceptable to have
>>> >>>>> upstream?
>>> >>>>>
>>> >>>>> [1]
>>> >>>>>
>>> >>>>> https://gcc.gnu.org/onlinedocs/gcc/x86-Function-Attributes.html#x86-Function-Attributes
>>> >>>>
>>> >>>>
>>> >>>> Are we basically talking about the outermost layer of your interrupt
>>> >>>> dispatching?
>>> >>>>
>>> >>>>
>>> >>>> Have you looked at the basic approach taken by the other ports? They
>>> >>>> end
>>> >>>> up switching the stack pointer to a dedicated stack on the outermost
>>> >>>> interrupt
>>> >>>> and, if a context switch/dispatch is needed, arrange for the
>>> >>>> interrupted
>>> >>>> task to call _Thread_Dispatch.But tinker with its stack so some
>>> >>>> registers
>>> >>>> are saved and it looks like it made the call itself.
>>> >>>>
>>> >>>> If you can do it in C, I am ok with an attribute. I just don't think
>>> >>>> you
>>> >>>> can pull off all the stack and return to dispatch magic that way.
>>> >>>>
>>> >>>> --joel
>>> >>>>
>>> >>
>>
>>