[GSoC - x86_64] Interrupt manager and and port-specific glue - was Re: [GSoC - x86_64 - automake] Limit CFLAGS to specific source for librtemsbsp.a

[Amaan Cheval]

Thanks for all the help! I have a simple test using the RTEMS
interrupt manager working successfully (tested by calling
rtems_interrupt_handler_install for vector 0, and then triggering a
divide-by-0 exception).

Could someone shed any light on why the i386 only hooks the first 17
vectors as "RTEMS interrupts"?

Here's what I'm aiming to get done before the GSoC deadline:

- Remap PIC (masking/disabling the PIC doesn't stop it from generating
spurious interrupts (IRQ7), which would look like exceptions to us)
- Disable PIC
- Enable APIC (done already, but confirm it plays well with the recent
changes to the IDT)
- Enable the PIT timer and use it to calibrate the APIC timer
- Clock driver using the APIC timer - (1) generate interrupts on ticks
and (2) tc_get_timecount function which calculates total time passed
through calculating (number of IRQs occured * time_per_irq +
time_passed_since_last_irq (through tick counter))

This does seem a bit ambitious given how short we are on time - I'll
finish this up even after the deadline if need be.

What should our minimum deliverable be for this period? Should we try
to upstream the interrupt support before I finish the clock driver? (I
think we can have this discussion on Wednesday or so, since by then
I'll likely know how much progress on the clock driver remains.)

Sorry about the rush near the deadline - getting the APIC functioning
had a bunch more yak-shaving style issues than I anticipated.

On Wed, Aug 1, 2018 at 10:51 PM, Joel Sherrill <joel at rtems.org> wrote:
> I have started to reply twice but you jumped in ahead. :)
>
> On Wed, Aug 1, 2018 at 12:12 PM, Amaan Cheval <amaan.cheval at gmail.com>
> wrote:
>>
>> If my previous email _is_ in fact correct, could someone confirm?
>> Because this excerpt in the documentation here seems to contradict it
>> (which was what lead to the confusion in the first place):
>>
>>
>> https://docs.rtems.org/branches/master/c-user/interrupt_manager.html#establishing-an-isr
>>
>> With my emphasis:
>>
>> > The rtems_interrupt_catch directive establishes an ISR for the system.
>> > The address of the ISR and its associated CPU vector number are specified to
>> > this directive. This directive installs the **RTEMS interrupt wrapper in the
>> > processor’s Interrupt Vector Table and the address of the user’s ISR in the
>> > RTEMS’ Vector Table**. This directive returns the previous contents of the
>> > specified vector in the RTEMS’ Vector Table.
>
>
> Almost but Gedare and I left out a detail. rtems_interrupt_catch is ONLY
> used on pure simple vectored architectures which do not use the
> bsp_interrupt_*
> or <rtems/irq.h> interfaces. Some embedded MCUs are so simple and have
> plenty of vectors so you don't need the complexity of supporting a PIC. For
> example, the m68k family had 256 direct vectors and I don't recall ever
> seeing
> a PIC.[1]
>
> You should assume that you can ignore rtems_interrupt_catch and simple
> vectored support for x86_64. See cpukit/rtems/intrcatch.c and I hope you
> see an ifdef that results in the code disappearing on your port. Simple
> vectored is FACE is your cpu.h. :)
>
> [1] Disclaimer: The support for the <rtems/irq.h> interfaces is critical to
> the
> libbsd stack. We haven't discussed it but any architecture that is
> sufficient
> to run the new stack will have to support this interface. If someone wants
> the
> new stack on a 68040 VME board or a Coldfire board, we will have to find
> the simplest, non-bloated way to support this. When doing the MIPS Malta,
> we just converted the MIPS architecture away from simple vectored.
>
> So support the bsp_interrupt_* infrastructure. :)
>
> --joel
>
>>
>>
>> On Wed, Aug 1, 2018 at 10:39 PM, Amaan Cheval <amaan.cheval at gmail.com>
>> wrote:
>> > Okay, I think I understand finally. Sorry about the rambling!
>> >
>> > When rtems_interrupt_catch is called, that's installing a "raw" ISR by
>> > modifying the processor specific table itself, so _ISR_Handler is
>> > never called, but the user ISR is.
>> >
>> > When rtems_interrupt_handler_install is called, that's an "RTEMS
>> > interrupt", and we go through the _ISR_Handler -> dispatch route I
>> > laid out earlier, leading to eventually the user's ISR.
>> >
>> > Thank you for letting me rubber-duck with you, everyone (let me know
>> > if anything above sounds off, though!) :P
>> >
>> > On Wed, Aug 1, 2018 at 10:20 PM, Amaan Cheval <amaan.cheval at gmail.com>
>> > wrote:
>> >> Thanks for the background!
>> >>
>> >> Let's use the gen5200 as the ongoing example - my confusion arises
>> >> here (correct me if any of the following points is incorrect!). In
>> >> overly simplified call chains:
>> >>
>> >> Register interrupt handler:
>> >> -  bsp_interrupt_facility_initialize() -> ppc_exc_set_handler(vec,
>> >> C_dispatch_irq_handler) -> ppc_exc_handler_table[vec] =
>> >> C_dispatch_irq_handler
>> >>
>> >> Interrupt handler called:
>> >> - C_dispatch_irq_handler -> dispatch -> bsp_interrupt_handler_dispatch
>> >> (irq-generic.h) -> bsp_interrupt_handler_table[index].handler()
>> >>
>> >> What I'm confused about is how the bsp_interrupt_handler_table is
>> >> updated at all - I just haven't found the link between how the entries
>> >> in the two tables are synchronized, where the tables are:
>> >>
>> >> 1) the ppc_exc_handler_table (the processor IDT) and
>> >> 2) the bsp_interrupt_handler_table (the RTEMS interrupt table)
>> >>
>> >> ------------------------------------------------------------
>> >> Another similar chain of confusion for i386 is:
>> >> - rtems_interrupt_catch (intrcatch.c) -> _ISR_Install_vector (isr.h)
>> >> -> _CPU_ISR_install_vector (i386/idt.c) -> idt_entry_tbl[vector]
>> >> updated
>> >>
>> >> But the i386 dispatch code chain is:
>> >> - _ISR_Handler (i386/irq_asm.S) -> BSP_dispatch_isr (i386/irq.c) ->
>> >> bsp_interrupt_handler_dispatch (irq-generic.h) ->
>> >> bsp_interrupt_handler_table[index].handler()
>> >>
>> >> But I don't see any updates to bsp_interrupt_handler_table that would
>> >> let this work.
>> >> -------------------------------------------------------------
>> >>
>> >> Would you happen to know what that "missing link" is?
>> >>
>> >> On Wed, Aug 1, 2018 at 9:07 PM, Joel Sherrill <joel at rtems.org> wrote:
>> >>>
>> >>>
>> >>> On Wed, Aug 1, 2018 at 10:11 AM, Gedare Bloom <gedare at rtems.org>
>> >>> wrote:
>> >>>>
>> >>>> On Wed, Aug 1, 2018 at 9:15 AM, Amaan Cheval <amaan.cheval at gmail.com>
>> >>>> wrote:
>> >>>> > That's definitely very illuminating, thank you so much for all the
>> >>>> > details!
>> >>>> >
>> >>>> > A few more questions that have arisen for me. Feel free to skip
>> >>>> > over
>> >>>> > them (I'll likely figure them out given enough time, so I'm only
>> >>>> > asking in case any of them are obvious to anyone):
>> >>>> >
>> >>>> > - The i386 doesn't use CPU_Interrupt_frame at all. It seems like it
>> >>>> > stores some of the data onto the stack?
>> >>>> >
>> >>>> the interrupt frame structure was introduced during 4.11 development.
>> >>>> probably i386 never got updated to use a struct to encapsulate the
>> >>>> interrupt frame. the interrupt frame should contain the registers
>> >>>> that
>> >>>> are preserved by the interrupt entry code I believe.
>> >>>
>> >>>
>> >>> +1
>> >>>
>> >>> Historically, there was no structure to represent the set of
>> >>> registers and information saved on interrupt entry. Over time
>> >>> this has been added.
>> >>>
>> >>> i386  also is missing the SMP synchronization check in the
>> >>> middle of the context which ensures it is safe for a thread to
>> >>> be migrated.
>> >>>
>> >>>>
>> >>>> > - There used to be defines in cpu.h regarding hardware/software
>> >>>> > based
>> >>>> > interrupt stacks, and how they'd be setup, which were made
>> >>>> > superfluous[1] - I'm not quite sure how these are meant to work - I
>> >>>> > see references to "stack high" and "stack low" and I'm not quite
>> >>>> > sure
>> >>>> > what the code is referencing when using those.
>> >>>> >
>> >>>>
>> >>>> a hardware interrupt stack is one that the hardware switches to
>> >>>> during
>> >>>> an interrupt. i think m68k has such.
>> >>>>
>> >>>> most interrupt stacks in RTEMS are software-managed, meaning that
>> >>>> RTEMS explicitly switches the stack region off the task stack and to
>> >>>> an interrupt stack region.
>> >>>>
>> >>>> some stacks start high and grow down, and some stacks start low and
>> >>>> grow up. maybe this is what the "stack high" and "stack low" you
>> >>>> mention are in relation to?
>> >>>
>> >>>
>> >>> They are used to denote the top and bottom of the memory reserved
>> >>> for the interrupt stack. One important use is in
>> >>> cpukit/libmisc/stackchk/check.c
>> >>> to report on usage.
>> >>>
>> >>>>
>> >>>>
>> >>>> > - c/src/lib/libbsp/no_cpu/no_bsp/Makefile.am doesn't include
>> >>>> > irq-sources.am, by the way (this is part of why I used to think a
>> >>>> > lot
>> >>>> > of what your email mentioned was unnecessary, until you...ahem,
>> >>>> > pre-empted that line of thought and helped clarify it :P). Should I
>> >>>> > add a ticket to update the no_bsp code to be more in line with
>> >>>> > current
>> >>>> > use?
>> >>>> >
>> >>>> Sure. I don't know that anyone is in particular maintaining
>> >>>> no_cpu/no_bsp since we can't compile it, it is basically best effort
>> >>>> stuff that sometimes we miss updating.
>> >>>
>> >>>
>> >>> +1
>> >>>
>> >>> Also there are variations based on simple vectored and PIC vectored
>> >>> architectures.
>> >>>
>> >>> The architecture is responsible for the managing the minimal actions
>> >>> based on what the CPU does for an interrupt/exception. Logically
>> >>> a PIC is part of the BSP even if it comes integrated with the CPU as
>> >>> is common on x86 and SoCs. The PIC could fairly easily change
>> >>> and the architecture is harder to change.
>> >>>
>> >>>>
>> >>>>
>> >>>> > - My understanding of _ISR_Handler is that it'll be the handler for
>> >>>> > _all_ interrupt vectors by default - it'll then dispatch interrupts
>> >>>> > to
>> >>>> > user-handlers (or internal handlers, for the timer, for eg.). Is
>> >>>> > that
>> >>>> > right? (I don't quite understand its interaction with the RTEMS
>> >>>> > interrupt manager yet, but irq-generic's
>> >>>> > "bsp_interrupt_handler_table"
>> >>>> > seems to be the RTEMS equivalent to the processor-specific vector
>> >>>> > table, and "bsp_interrupt_handler_dispatch" seems to call the
>> >>>> > actual
>> >>>> > handler within that table as appropriate. Accurate? (I just haven't
>> >>>> > found how that table actually gets its handlers setup besides
>> >>>> > during
>> >>>> > initialization, since rtems_interrupt_catch just calls
>> >>>> > _CPU_install_vector, which updates the processor vector table, not
>> >>>> > the
>> >>>> > RTEMS interrupt manager vector table.)
>> >>>> >
>> >>>>
>> >>>> You have discovered a couple of different but related interrupt
>> >>>> processing code bases.  I can see why you get confused.
>> >>>>
>> >>>> Basically, a CPU port should support two kinds of interrupts that may
>> >>>> be installed, "RTEMS" and "Raw" interrupts. The "Raw" interrupts are
>> >>>> installed directly in the processor's vector table. For processors
>> >>>> that use simple vectored interrupts, the "RTEMS" interrupts install a
>> >>>> call to the _ISR_Handler() function in the processor's vector table,
>> >>>> and will put the user's isr function pointer in the
>> >>>> _ISR_Vector_table(), which is the RTEMS Interrupt Manager's vector
>> >>>> table.
>> >>>>
>> >>>> I'm not really familiar with the processors that use a different
>> >>>> model
>> >>>> for interrupt handling besides simple vectored. Probably, you will
>> >>>> have to study one of them.
>> >>>>
>> >>>> This irq-generic bsp_interrupt_* code is used by the "IRQ Server"
>> >>>> that
>> >>>> builds from the CPU port capabilities to allow for some advanced
>> >>>> features like chaining multiple isrs from the same source. I don't
>> >>>> think you want to be focusing on those right now, but I could be
>> >>>> mistaken. I haven't spent much time looking at the IRQ Server
>> >>>> codebase.
>> >>>
>> >>>
>> >>> For the most part, the bsp_interrupt_* code is hung off of one
>> >>> (or more) of the simple vectored sources. On many architectures,
>> >>> you have very few "CPU vectors" and the bsp_interrupt_* is
>> >>> the secondary layer managing the PIC. Notice that it is all
>> >>> in C and thus isn't first level processing. And it doesn't deal
>> >>> with RTEMS critical section, interrupt entry/exit, and dispatching..
>> >>>
>> >>> For example, in its original form, the PowerPC didn't have many
>> >>> "exceptions" (e.g. CPU vectors) at all. In fact, there was a single
>> >>> External Exception which was always tied to a board dependent
>> >>> PIC. The first level processing of PPC exceptions is in assembly
>> >>> and deals with what I pointed you to in various _ISR_Handler's.
>> >>> The External Exception handler now process bsp_interrupt_*.
>> >>> The gen5200 BSP irq.c file has a bsp_interrupt_* implementation
>> >>> which registers for two "exceptions" which both go to
>> >>> C_dispatch_irq_handler() which reads the PIC, determines
>> >>> the source(s), and dispatches (via bsp_interrupt_handler_dispatch()
>> >>> the registered IRQ.
>> >>>
>> >>> If you think of this as two levels of processing, it should help.
>> >>>
>> >>> The two levels of processing is for RTEMS interrupts which can
>> >>> unblock threads, etc. The first level ensures that locking and
>> >>> dispatching is handled.
>> >>>
>> >>> As Gedare stated, there are also Raw interrupts which ignore
>> >>> RTEMS and can't use RTEMS services. These should never
>> >>> be anything but the highest priority interrupt in a system. The
>> >>> RTEMS first level processing can switch to a dedicated
>> >>> interrupt stack and mixing RTEMS and Raw IRQs can
>> >>> screw up the stack handling as well as the interrupt nesting
>> >>> count.
>> >>>
>> >>> --joel
>> >>>
>> >>>>
>> >>>>
>> >>>> Gedare
>> >>>>
>> >>>> > - My understanding of the interaction between RTEMS' interrupt
>> >>>> > manager
>> >>>> > (i.e. support for nested interrupts and thread dispatch once an
>> >>>> > interrupt ends) and the BSP's processor-specific interrupt manager
>> >>>> > (code to use the APIC and IDT in my case) is that they're tied
>> >>>> > together through the use of irq-generic.c's
>> >>>> > "bsp_interrupt_initialize"
>> >>>> > - is that right? (m68k never seems to call it, though, so perhaps
>> >>>> > not?)
>> >>>> >
>> >>>> > Sorry about the rambling! To reiterate, I'll likely figure it out
>> >>>> > given enough time, so if the answers aren't at the top of your
>> >>>> > head, I
>> >>>> > can figure it out without wasting your time :)
>> >>>> >
>> >>>> > [1] https://devel.rtems.org/ticket/3459#comment:11
>> >>>> >
>> >>>> > On Wed, Aug 1, 2018 at 3:18 AM, Joel Sherrill <joel at rtems.org>
>> >>>> > wrote:
>> >>>> >>
>> >>>> >>
>> >>>> >> On Tue, Jul 31, 2018 at 3:05 PM, Amaan Cheval
>> >>>> >> <amaan.cheval at gmail.com>
>> >>>> >> wrote:
>> >>>> >>>
>> >>>> >>> Hm, I'm not sure what to look for in the other ports
>> >>>> >>> specifically,
>> >>>> >>> really.
>> >>>> >>> The BSP porting documentation doesn't have a section on
>> >>>> >>> interrupts, so
>> >>>> >>> I'm
>> >>>> >>> doing this on more of an "as it comes up" basis.
>> >>>> >>>
>> >>>> >>> What I've got right now (the interrupt handlers in C) are what I
>> >>>> >>> need
>> >>>> >>> for
>> >>>> >>> calibrating the APIC timer (through the PIT) - so simply hooking
>> >>>> >>> IRQ0
>> >>>> >>> (for
>> >>>> >>> the timer) and IRQ7 (spurious vector), since those are needed for
>> >>>> >>> the
>> >>>> >>> timer
>> >>>> >>> work to continue.
>> >>>> >>>
>> >>>> >>> What constitutes as a requirement for basic interrupt support?
>> >>>> >>
>> >>>> >>
>> >>>> >> There used to be a generic porting guide. I can see that this
>> >>>> >> particular
>> >>>> >> section
>> >>>> >> has bit rotted some but the interrupt dispatching section. Some of
>> >>>> >> this
>> >>>> >> will have evolved to support SMP and fine grained locking but the
>> >>>> >> pseudo-code
>> >>>> >> here will give you a push toward the right line of thinking:
>> >>>> >>
>> >>>> >>
>> >>>> >>
>> >>>> >> https://docs.rtems.org/releases/rtemsdocs-4.10.2/share/rtems/html/porting/porting00034.html
>> >>>> >>
>> >>>> >> The idea is that you need to ensure RTEMS knows it is inside an
>> >>>> >> interrupt
>> >>>> >> and the current locking scheme (old was dispatching, new is ...)
>> >>>> >> is
>> >>>> >> honored.
>> >>>> >>
>> >>>> >> The ARM and PowerPC (plus RISCV) are good ports to look at for how
>> >>>> >> SMP
>> >>>> >> plays into this. But the CPU supplement is thin for their
>> >>>> >> interrupt
>> >>>> >> processing.
>> >>>> >>
>> >>>> >>
>> >>>> >> This is the CPU Architecture supplement section for the m68k. This
>> >>>> >> is a
>> >>>> >> relatively simple
>> >>>> >> architecture to describe. There is also a section for the i386
>> >>>> >> which
>> >>>> >> reads
>> >>>> >> similarly.
>> >>>> >>
>> >>>> >>
>> >>>> >>
>> >>>> >> https://docs.rtems.org/branches/master/cpu-supplement/m68xxx_and_coldfire.html#interrupt-processing
>> >>>> >>
>> >>>> >> Personally, I find the m68k a fairly easy processor to read
>> >>>> >> assembly
>> >>>> >> in.
>> >>>> >> Look at cpukit/score/cpu/m68k/cpu_asm.S and _ISR_Handler to see
>> >>>> >> what
>> >>>> >> is done there w/o SMP. On the m68k _ISR_Handler is directly put
>> >>>> >> into
>> >>>> >> the
>> >>>> >> vector table. But this isn't the most similar example for you.
>> >>>> >>
>> >>>> >> For the i386 (better example), it is in
>> >>>> >> bsps/i386/shared/irq/irq_asm.S
>> >>>> >> with
>> >>>> >> the
>> >>>> >> same name. There _ISR_Handler is installed via the
>> >>>> >> DISTINCT_INTERRUPT_ENTRY
>> >>>> >> macros at the bottom of the file where some prologue jumps to the
>> >>>> >> common
>> >>>> >> _ISR_Handler and then the actions are similar. Usually
>> >>>> >> _ISR_Handler
>> >>>> >> type of
>> >>>> >> code ends up invoking a PIC decode method in normal C without an
>> >>>> >> interrupt attribute.
>> >>>> >>
>> >>>> >> Long and multi-architecture answer but maybe that makes sense. The
>> >>>> >> goal
>> >>>> >> in ticker.exe is to take a number of tick interrupts which don't
>> >>>> >> schedule
>> >>>> >> and
>> >>>> >> then take one that does -- it schedules a preemption of the idle
>> >>>> >> thread.
>> >>>> >>
>> >>>> >> Hope this helps.
>> >>>> >>
>> >>>> >> --joel
>> >>>> >>
>> >>>> >>
>> >>>> >>
>> >>>> >>>
>> >>>> >>>
>> >>>> >>> On Wed, Aug 1, 2018, 1:29 AM Joel Sherrill <joel at rtems.org>
>> >>>> >>> wrote:
>> >>>> >>>>
>> >>>> >>>>
>> >>>> >>>>
>> >>>> >>>> On Tue, Jul 31, 2018 at 2:52 PM, Amaan Cheval
>> >>>> >>>> <amaan.cheval at gmail.com>
>> >>>> >>>> wrote:
>> >>>> >>>>>
>> >>>> >>>>> Hi Chris!
>> >>>> >>>>>
>> >>>> >>>>> I currently have code like this in
>> >>>> >>>>> c/src/lib/libbsp/x86_64/amd64/Makefile.am:
>> >>>> >>>>>
>> >>>> >>>>>     librtemsbsp_a_SOURCES +=
>> >>>> >>>>> ../../../../../../bsps/x86_64/amd64/interrupts/handlers.c
>> >>>> >>>>>     # XXX: Needed to use GCC "interrupt" attribute directives -
>> >>>> >>>>> can
>> >>>> >>>>> we
>> >>>> >>>>> pass these
>> >>>> >>>>>     # flags only for the handlers.c source file (compile to an
>> >>>> >>>>> object
>> >>>> >>>>> file first and
>> >>>> >>>>>     # then link with the rest for librtemsbsp.a?)
>> >>>> >>>>>     librtemsbsp_a_CFLAGS = -mgeneral-regs-only
>> >>>> >>>>>
>> >>>> >>>>> The CFLAGS arg is required to allow us to use
>> >>>> >>>>> "__attribute__((interrupt))" to setup interrupt handlers in C.
>> >>>> >>>>> (See
>> >>>> >>>>> [1] and ctrl+f "interrupt" for more.)
>> >>>> >>>>>
>> >>>> >>>>> Is there a way to not force the CFLAGS for _all_ of
>> >>>> >>>>> librtemsbsp, but
>> >>>> >>>>> to limit it only to handlers.c?
>> >>>> >>>>>
>> >>>> >>>>> If not, is the above code something that would be acceptable to
>> >>>> >>>>> have
>> >>>> >>>>> upstream?
>> >>>> >>>>>
>> >>>> >>>>> [1]
>> >>>> >>>>>
>> >>>> >>>>>
>> >>>> >>>>> https://gcc.gnu.org/onlinedocs/gcc/x86-Function-Attributes.html#x86-Function-Attributes
>> >>>> >>>>
>> >>>> >>>>
>> >>>> >>>> Are we basically talking about the outermost layer of your
>> >>>> >>>> interrupt
>> >>>> >>>> dispatching?
>> >>>> >>>>
>> >>>> >>>>
>> >>>> >>>> Have you looked at the basic approach taken by the other ports?
>> >>>> >>>> They
>> >>>> >>>> end
>> >>>> >>>> up switching the stack pointer to a dedicated stack on the
>> >>>> >>>> outermost
>> >>>> >>>> interrupt
>> >>>> >>>> and, if a context switch/dispatch is needed, arrange for the
>> >>>> >>>> interrupted
>> >>>> >>>> task to call _Thread_Dispatch.But tinker with its stack so some
>> >>>> >>>> registers
>> >>>> >>>> are saved and it looks like it made the call itself.
>> >>>> >>>>
>> >>>> >>>> If you can do it in C, I am ok with an attribute. I just don't
>> >>>> >>>> think
>> >>>> >>>> you
>> >>>> >>>> can pull off all the stack and return to dispatch magic that
>> >>>> >>>> way.
>> >>>> >>>>
>> >>>> >>>> --joel
>> >>>> >>>>
>> >>>> >>
>> >>>
>> >>>
>
>