[GSoC - x86_64] Interrupt manager and and port-specific glue - was Re: [GSoC - x86_64 - automake] Limit CFLAGS to specific source for librtemsbsp.a

Wed Aug 1 17:12:01 UTC 2018

If my previous email _is_ in fact correct, could someone confirm?
Because this excerpt in the documentation here seems to contradict it
(which was what lead to the confusion in the first place):

https://docs.rtems.org/branches/master/c-user/interrupt_manager.html#establishing-an-isr

With my emphasis:

> The rtems_interrupt_catch directive establishes an ISR for the system. The address of the ISR and its associated CPU vector number are specified to this directive. This directive installs the **RTEMS interrupt wrapper in the processor’s Interrupt Vector Table and the address of the user’s ISR in the RTEMS’ Vector Table**. This directive returns the previous contents of the specified vector in the RTEMS’ Vector Table.

On Wed, Aug 1, 2018 at 10:39 PM, Amaan Cheval <amaan.cheval at gmail.com> wrote:
> Okay, I think I understand finally. Sorry about the rambling!
>
> When rtems_interrupt_catch is called, that's installing a "raw" ISR by
> modifying the processor specific table itself, so _ISR_Handler is
> never called, but the user ISR is.
>
> When rtems_interrupt_handler_install is called, that's an "RTEMS
> interrupt", and we go through the _ISR_Handler -> dispatch route I
> laid out earlier, leading to eventually the user's ISR.
>
> Thank you for letting me rubber-duck with you, everyone (let me know
> if anything above sounds off, though!) :P
>
> On Wed, Aug 1, 2018 at 10:20 PM, Amaan Cheval <amaan.cheval at gmail.com> wrote:
>> Thanks for the background!
>>
>> Let's use the gen5200 as the ongoing example - my confusion arises
>> here (correct me if any of the following points is incorrect!). In
>> overly simplified call chains:
>>
>> Register interrupt handler:
>> -  bsp_interrupt_facility_initialize() -> ppc_exc_set_handler(vec,
>> C_dispatch_irq_handler) -> ppc_exc_handler_table[vec] =
>> C_dispatch_irq_handler
>>
>> Interrupt handler called:
>> - C_dispatch_irq_handler -> dispatch -> bsp_interrupt_handler_dispatch
>> (irq-generic.h) -> bsp_interrupt_handler_table[index].handler()
>>
>> What I'm confused about is how the bsp_interrupt_handler_table is
>> updated at all - I just haven't found the link between how the entries
>> in the two tables are synchronized, where the tables are:
>>
>> 1) the ppc_exc_handler_table (the processor IDT) and
>> 2) the bsp_interrupt_handler_table (the RTEMS interrupt table)
>>
>> ------------------------------------------------------------
>> Another similar chain of confusion for i386 is:
>> - rtems_interrupt_catch (intrcatch.c) -> _ISR_Install_vector (isr.h)
>> -> _CPU_ISR_install_vector (i386/idt.c) -> idt_entry_tbl[vector]
>> updated
>>
>> But the i386 dispatch code chain is:
>> - _ISR_Handler (i386/irq_asm.S) -> BSP_dispatch_isr (i386/irq.c) ->
>> bsp_interrupt_handler_dispatch (irq-generic.h) ->
>> bsp_interrupt_handler_table[index].handler()
>>
>> But I don't see any updates to bsp_interrupt_handler_table that would
>> let this work.
>> -------------------------------------------------------------
>>
>> Would you happen to know what that "missing link" is?
>>
>> On Wed, Aug 1, 2018 at 9:07 PM, Joel Sherrill <joel at rtems.org> wrote:
>>>
>>>
>>> On Wed, Aug 1, 2018 at 10:11 AM, Gedare Bloom <gedare at rtems.org> wrote:
>>>>
>>>> On Wed, Aug 1, 2018 at 9:15 AM, Amaan Cheval <amaan.cheval at gmail.com>
>>>> wrote:
>>>> > That's definitely very illuminating, thank you so much for all the
>>>> > details!
>>>> >
>>>> > A few more questions that have arisen for me. Feel free to skip over
>>>> > them (I'll likely figure them out given enough time, so I'm only
>>>> > asking in case any of them are obvious to anyone):
>>>> >
>>>> > - The i386 doesn't use CPU_Interrupt_frame at all. It seems like it
>>>> > stores some of the data onto the stack?
>>>> >
>>>> the interrupt frame structure was introduced during 4.11 development.
>>>> probably i386 never got updated to use a struct to encapsulate the
>>>> interrupt frame. the interrupt frame should contain the registers that
>>>> are preserved by the interrupt entry code I believe.
>>>
>>>
>>> +1
>>>
>>> Historically, there was no structure to represent the set of
>>> registers and information saved on interrupt entry. Over time
>>> this has been added.
>>>
>>> i386  also is missing the SMP synchronization check in the
>>> middle of the context which ensures it is safe for a thread to
>>> be migrated.
>>>
>>>>
>>>> > - There used to be defines in cpu.h regarding hardware/software based
>>>> > interrupt stacks, and how they'd be setup, which were made
>>>> > superfluous[1] - I'm not quite sure how these are meant to work - I
>>>> > see references to "stack high" and "stack low" and I'm not quite sure
>>>> > what the code is referencing when using those.
>>>> >
>>>>
>>>> a hardware interrupt stack is one that the hardware switches to during
>>>> an interrupt. i think m68k has such.
>>>>
>>>> most interrupt stacks in RTEMS are software-managed, meaning that
>>>> RTEMS explicitly switches the stack region off the task stack and to
>>>> an interrupt stack region.
>>>>
>>>> some stacks start high and grow down, and some stacks start low and
>>>> grow up. maybe this is what the "stack high" and "stack low" you
>>>> mention are in relation to?
>>>
>>>
>>> They are used to denote the top and bottom of the memory reserved
>>> for the interrupt stack. One important use is in
>>> cpukit/libmisc/stackchk/check.c
>>> to report on usage.
>>>
>>>>
>>>>
>>>> > - c/src/lib/libbsp/no_cpu/no_bsp/Makefile.am doesn't include
>>>> > irq-sources.am, by the way (this is part of why I used to think a lot
>>>> > of what your email mentioned was unnecessary, until you...ahem,
>>>> > pre-empted that line of thought and helped clarify it :P). Should I
>>>> > add a ticket to update the no_bsp code to be more in line with current
>>>> > use?
>>>> >
>>>> Sure. I don't know that anyone is in particular maintaining
>>>> no_cpu/no_bsp since we can't compile it, it is basically best effort
>>>> stuff that sometimes we miss updating.
>>>
>>>
>>> +1
>>>
>>> Also there are variations based on simple vectored and PIC vectored
>>> architectures.
>>>
>>> The architecture is responsible for the managing the minimal actions
>>> based on what the CPU does for an interrupt/exception. Logically
>>> a PIC is part of the BSP even if it comes integrated with the CPU as
>>> is common on x86 and SoCs. The PIC could fairly easily change
>>> and the architecture is harder to change.
>>>
>>>>
>>>>
>>>> > - My understanding of _ISR_Handler is that it'll be the handler for
>>>> > _all_ interrupt vectors by default - it'll then dispatch interrupts to
>>>> > user-handlers (or internal handlers, for the timer, for eg.). Is that
>>>> > right? (I don't quite understand its interaction with the RTEMS
>>>> > interrupt manager yet, but irq-generic's "bsp_interrupt_handler_table"
>>>> > seems to be the RTEMS equivalent to the processor-specific vector
>>>> > table, and "bsp_interrupt_handler_dispatch" seems to call the actual
>>>> > handler within that table as appropriate. Accurate? (I just haven't
>>>> > found how that table actually gets its handlers setup besides during
>>>> > initialization, since rtems_interrupt_catch just calls
>>>> > _CPU_install_vector, which updates the processor vector table, not the
>>>> > RTEMS interrupt manager vector table.)
>>>> >
>>>>
>>>> You have discovered a couple of different but related interrupt
>>>> processing code bases.  I can see why you get confused.
>>>>
>>>> Basically, a CPU port should support two kinds of interrupts that may
>>>> be installed, "RTEMS" and "Raw" interrupts. The "Raw" interrupts are
>>>> installed directly in the processor's vector table. For processors
>>>> that use simple vectored interrupts, the "RTEMS" interrupts install a
>>>> call to the _ISR_Handler() function in the processor's vector table,
>>>> and will put the user's isr function pointer in the
>>>> _ISR_Vector_table(), which is the RTEMS Interrupt Manager's vector
>>>> table.
>>>>
>>>> I'm not really familiar with the processors that use a different model
>>>> for interrupt handling besides simple vectored. Probably, you will
>>>> have to study one of them.
>>>>
>>>> This irq-generic bsp_interrupt_* code is used by the "IRQ Server" that
>>>> builds from the CPU port capabilities to allow for some advanced
>>>> features like chaining multiple isrs from the same source. I don't
>>>> think you want to be focusing on those right now, but I could be
>>>> mistaken. I haven't spent much time looking at the IRQ Server
>>>> codebase.
>>>
>>>
>>> For the most part, the bsp_interrupt_* code is hung off of one
>>> (or more) of the simple vectored sources. On many architectures,
>>> you have very few "CPU vectors" and the bsp_interrupt_* is
>>> the secondary layer managing the PIC. Notice that it is all
>>> in C and thus isn't first level processing. And it doesn't deal
>>> with RTEMS critical section, interrupt entry/exit, and dispatching..
>>>
>>> For example, in its original form, the PowerPC didn't have many
>>> "exceptions" (e.g. CPU vectors) at all. In fact, there was a single
>>> External Exception which was always tied to a board dependent
>>> PIC. The first level processing of PPC exceptions is in assembly
>>> and deals with what I pointed you to in various _ISR_Handler's.
>>> The External Exception handler now process bsp_interrupt_*.
>>> The gen5200 BSP irq.c file has a bsp_interrupt_* implementation
>>> which registers for two "exceptions" which both go to
>>> C_dispatch_irq_handler() which reads the PIC, determines
>>> the source(s), and dispatches (via bsp_interrupt_handler_dispatch()
>>> the registered IRQ.
>>>
>>> If you think of this as two levels of processing, it should help.
>>>
>>> The two levels of processing is for RTEMS interrupts which can
>>> unblock threads, etc. The first level ensures that locking and
>>> dispatching is handled.
>>>
>>> As Gedare stated, there are also Raw interrupts which ignore
>>> RTEMS and can't use RTEMS services. These should never
>>> be anything but the highest priority interrupt in a system. The
>>> RTEMS first level processing can switch to a dedicated
>>> interrupt stack and mixing RTEMS and Raw IRQs can
>>> screw up the stack handling as well as the interrupt nesting
>>> count.
>>>
>>> --joel
>>>
>>>>
>>>>
>>>> Gedare
>>>>
>>>> > - My understanding of the interaction between RTEMS' interrupt manager
>>>> > (i.e. support for nested interrupts and thread dispatch once an
>>>> > interrupt ends) and the BSP's processor-specific interrupt manager
>>>> > (code to use the APIC and IDT in my case) is that they're tied
>>>> > together through the use of irq-generic.c's "bsp_interrupt_initialize"
>>>> > - is that right? (m68k never seems to call it, though, so perhaps
>>>> > not?)
>>>> >
>>>> > Sorry about the rambling! To reiterate, I'll likely figure it out
>>>> > given enough time, so if the answers aren't at the top of your head, I
>>>> > can figure it out without wasting your time :)
>>>> >
>>>> > [1] https://devel.rtems.org/ticket/3459#comment:11
>>>> >
>>>> > On Wed, Aug 1, 2018 at 3:18 AM, Joel Sherrill <joel at rtems.org> wrote:
>>>> >>
>>>> >>
>>>> >> On Tue, Jul 31, 2018 at 3:05 PM, Amaan Cheval <amaan.cheval at gmail.com>
>>>> >> wrote:
>>>> >>>
>>>> >>> Hm, I'm not sure what to look for in the other ports specifically,
>>>> >>> really.
>>>> >>> The BSP porting documentation doesn't have a section on interrupts, so
>>>> >>> I'm
>>>> >>> doing this on more of an "as it comes up" basis.
>>>> >>>
>>>> >>> What I've got right now (the interrupt handlers in C) are what I need
>>>> >>> for
>>>> >>> calibrating the APIC timer (through the PIT) - so simply hooking IRQ0
>>>> >>> (for
>>>> >>> the timer) and IRQ7 (spurious vector), since those are needed for the
>>>> >>> timer
>>>> >>> work to continue.
>>>> >>>
>>>> >>> What constitutes as a requirement for basic interrupt support?
>>>> >>
>>>> >>
>>>> >> There used to be a generic porting guide. I can see that this
>>>> >> particular
>>>> >> section
>>>> >> has bit rotted some but the interrupt dispatching section. Some of this
>>>> >> will have evolved to support SMP and fine grained locking but the
>>>> >> pseudo-code
>>>> >> here will give you a push toward the right line of thinking:
>>>> >>
>>>> >>
>>>> >> https://docs.rtems.org/releases/rtemsdocs-4.10.2/share/rtems/html/porting/porting00034.html
>>>> >>
>>>> >> The idea is that you need to ensure RTEMS knows it is inside an
>>>> >> interrupt
>>>> >> and the current locking scheme (old was dispatching, new is ...) is
>>>> >> honored.
>>>> >>
>>>> >> The ARM and PowerPC (plus RISCV) are good ports to look at for how SMP
>>>> >> plays into this. But the CPU supplement is thin for their interrupt
>>>> >> processing.
>>>> >>
>>>> >>
>>>> >> This is the CPU Architecture supplement section for the m68k. This is a
>>>> >> relatively simple
>>>> >> architecture to describe. There is also a section for the i386 which
>>>> >> reads
>>>> >> similarly.
>>>> >>
>>>> >>
>>>> >> https://docs.rtems.org/branches/master/cpu-supplement/m68xxx_and_coldfire.html#interrupt-processing
>>>> >>
>>>> >> Personally, I find the m68k a fairly easy processor to read assembly
>>>> >> in.
>>>> >> Look at cpukit/score/cpu/m68k/cpu_asm.S and _ISR_Handler to see what
>>>> >> is done there w/o SMP. On the m68k _ISR_Handler is directly put into
>>>> >> the
>>>> >> vector table. But this isn't the most similar example for you.
>>>> >>
>>>> >> For the i386 (better example), it is in bsps/i386/shared/irq/irq_asm.S
>>>> >> with
>>>> >> the
>>>> >> same name. There _ISR_Handler is installed via the
>>>> >> DISTINCT_INTERRUPT_ENTRY
>>>> >> macros at the bottom of the file where some prologue jumps to the
>>>> >> common
>>>> >> _ISR_Handler and then the actions are similar. Usually _ISR_Handler
>>>> >> type of
>>>> >> code ends up invoking a PIC decode method in normal C without an
>>>> >> interrupt attribute.
>>>> >>
>>>> >> Long and multi-architecture answer but maybe that makes sense. The goal
>>>> >> in ticker.exe is to take a number of tick interrupts which don't
>>>> >> schedule
>>>> >> and
>>>> >> then take one that does -- it schedules a preemption of the idle
>>>> >> thread.
>>>> >>
>>>> >> Hope this helps.
>>>> >>
>>>> >> --joel
>>>> >>
>>>> >>
>>>> >>
>>>> >>>
>>>> >>>
>>>> >>> On Wed, Aug 1, 2018, 1:29 AM Joel Sherrill <joel at rtems.org> wrote:
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>> On Tue, Jul 31, 2018 at 2:52 PM, Amaan Cheval
>>>> >>>> <amaan.cheval at gmail.com>
>>>> >>>> wrote:
>>>> >>>>>
>>>> >>>>> Hi Chris!
>>>> >>>>>
>>>> >>>>> I currently have code like this in
>>>> >>>>> c/src/lib/libbsp/x86_64/amd64/Makefile.am:
>>>> >>>>>
>>>> >>>>>     librtemsbsp_a_SOURCES +=
>>>> >>>>> ../../../../../../bsps/x86_64/amd64/interrupts/handlers.c
>>>> >>>>>     # XXX: Needed to use GCC "interrupt" attribute directives - can
>>>> >>>>> we
>>>> >>>>> pass these
>>>> >>>>>     # flags only for the handlers.c source file (compile to an
>>>> >>>>> object
>>>> >>>>> file first and
>>>> >>>>>     # then link with the rest for librtemsbsp.a?)
>>>> >>>>>     librtemsbsp_a_CFLAGS = -mgeneral-regs-only
>>>> >>>>>
>>>> >>>>> The CFLAGS arg is required to allow us to use
>>>> >>>>> "__attribute__((interrupt))" to setup interrupt handlers in C. (See
>>>> >>>>> [1] and ctrl+f "interrupt" for more.)
>>>> >>>>>
>>>> >>>>> Is there a way to not force the CFLAGS for _all_ of librtemsbsp, but
>>>> >>>>> to limit it only to handlers.c?
>>>> >>>>>
>>>> >>>>> If not, is the above code something that would be acceptable to have
>>>> >>>>> upstream?
>>>> >>>>>
>>>> >>>>> [1]
>>>> >>>>>
>>>> >>>>> https://gcc.gnu.org/onlinedocs/gcc/x86-Function-Attributes.html#x86-Function-Attributes
>>>> >>>>
>>>> >>>>
>>>> >>>> Are we basically talking about the outermost layer of your interrupt
>>>> >>>> dispatching?
>>>> >>>>
>>>> >>>>
>>>> >>>> Have you looked at the basic approach taken by the other ports? They
>>>> >>>> end
>>>> >>>> up switching the stack pointer to a dedicated stack on the outermost
>>>> >>>> interrupt
>>>> >>>> and, if a context switch/dispatch is needed, arrange for the
>>>> >>>> interrupted
>>>> >>>> task to call _Thread_Dispatch.But tinker with its stack so some
>>>> >>>> registers
>>>> >>>> are saved and it looks like it made the call itself.
>>>> >>>>
>>>> >>>> If you can do it in C, I am ok with an attribute. I just don't think
>>>> >>>> you
>>>> >>>> can pull off all the stack and return to dispatch magic that way.
>>>> >>>>
>>>> >>>> --joel
>>>> >>>>
>>>> >>
>>>
>>>