[GSoC - x86_64] Interrupt manager and and port-specific glue - was Re: [GSoC - x86_64 - automake] Limit CFLAGS to specific source for librtemsbsp.a

Wed Aug 1 15:37:49 UTC 2018

On Wed, Aug 1, 2018 at 10:11 AM, Gedare Bloom <gedare at rtems.org> wrote:

> On Wed, Aug 1, 2018 at 9:15 AM, Amaan Cheval <amaan.cheval at gmail.com>
> wrote:
> > That's definitely very illuminating, thank you so much for all the
> details!
> >
> > A few more questions that have arisen for me. Feel free to skip over
> > them (I'll likely figure them out given enough time, so I'm only
> > asking in case any of them are obvious to anyone):
> >
> > - The i386 doesn't use CPU_Interrupt_frame at all. It seems like it
> > stores some of the data onto the stack?
> >
> the interrupt frame structure was introduced during 4.11 development.
> probably i386 never got updated to use a struct to encapsulate the
> interrupt frame. the interrupt frame should contain the registers that
> are preserved by the interrupt entry code I believe.
>

+1

Historically, there was no structure to represent the set of
registers and information saved on interrupt entry. Over time
this has been added.

i386  also is missing the SMP synchronization check in the
middle of the context which ensures it is safe for a thread to
be migrated.

> > - There used to be defines in cpu.h regarding hardware/software based
> > interrupt stacks, and how they'd be setup, which were made
> > superfluous[1] - I'm not quite sure how these are meant to work - I
> > see references to "stack high" and "stack low" and I'm not quite sure
> > what the code is referencing when using those.
> >
>
> a hardware interrupt stack is one that the hardware switches to during
> an interrupt. i think m68k has such.
>
> most interrupt stacks in RTEMS are software-managed, meaning that
> RTEMS explicitly switches the stack region off the task stack and to
> an interrupt stack region.
>
> some stacks start high and grow down, and some stacks start low and
> grow up. maybe this is what the "stack high" and "stack low" you
> mention are in relation to?
>

They are used to denote the top and bottom of the memory reserved
for the interrupt stack. One important use is in
cpukit/libmisc/stackchk/check.c
to report on usage.

>
> > - c/src/lib/libbsp/no_cpu/no_bsp/Makefile.am doesn't include
> > irq-sources.am, by the way (this is part of why I used to think a lot
> > of what your email mentioned was unnecessary, until you...ahem,
> > pre-empted that line of thought and helped clarify it :P). Should I
> > add a ticket to update the no_bsp code to be more in line with current
> > use?
> >
> Sure. I don't know that anyone is in particular maintaining
> no_cpu/no_bsp since we can't compile it, it is basically best effort
> stuff that sometimes we miss updating.
>

+1

Also there are variations based on simple vectored and PIC vectored
architectures.

The architecture is responsible for the managing the minimal actions
based on what the CPU does for an interrupt/exception. Logically
a PIC is part of the BSP even if it comes integrated with the CPU as
is common on x86 and SoCs. The PIC could fairly easily change
and the architecture is harder to change.

>
> > - My understanding of _ISR_Handler is that it'll be the handler for
> > _all_ interrupt vectors by default - it'll then dispatch interrupts to
> > user-handlers (or internal handlers, for the timer, for eg.). Is that
> > right? (I don't quite understand its interaction with the RTEMS
> > interrupt manager yet, but irq-generic's "bsp_interrupt_handler_table"
> > seems to be the RTEMS equivalent to the processor-specific vector
> > table, and "bsp_interrupt_handler_dispatch" seems to call the actual
> > handler within that table as appropriate. Accurate? (I just haven't
> > found how that table actually gets its handlers setup besides during
> > initialization, since rtems_interrupt_catch just calls
> > _CPU_install_vector, which updates the processor vector table, not the
> > RTEMS interrupt manager vector table.)
> >
>
> You have discovered a couple of different but related interrupt
> processing code bases.  I can see why you get confused.
>
> Basically, a CPU port should support two kinds of interrupts that may
> be installed, "RTEMS" and "Raw" interrupts. The "Raw" interrupts are
> installed directly in the processor's vector table. For processors
> that use simple vectored interrupts, the "RTEMS" interrupts install a
> call to the _ISR_Handler() function in the processor's vector table,
> and will put the user's isr function pointer in the
> _ISR_Vector_table(), which is the RTEMS Interrupt Manager's vector
> table.
>
> I'm not really familiar with the processors that use a different model
> for interrupt handling besides simple vectored. Probably, you will
> have to study one of them.
>
> This irq-generic bsp_interrupt_* code is used by the "IRQ Server" that
> builds from the CPU port capabilities to allow for some advanced
> features like chaining multiple isrs from the same source. I don't
> think you want to be focusing on those right now, but I could be
> mistaken. I haven't spent much time looking at the IRQ Server
> codebase.
>

For the most part, the bsp_interrupt_* code is hung off of one
(or more) of the simple vectored sources. On many architectures,
you have very few "CPU vectors" and the bsp_interrupt_* is
the secondary layer managing the PIC. Notice that it is all
in C and thus isn't first level processing. And it doesn't deal
with RTEMS critical section, interrupt entry/exit, and dispatching..

For example, in its original form, the PowerPC didn't have many
"exceptions" (e.g. CPU vectors) at all. In fact, there was a single
External Exception which was always tied to a board dependent
PIC. The first level processing of PPC exceptions is in assembly
and deals with what I pointed you to in various _ISR_Handler's.
The External Exception handler now process bsp_interrupt_*.
The gen5200 BSP irq.c file has a bsp_interrupt_* implementation
which registers for two "exceptions" which both go to
C_dispatch_irq_handler() which reads the PIC, determines
the source(s), and dispatches (via bsp_interrupt_handler_dispatch()
the registered IRQ.

If you think of this as two levels of processing, it should help.

The two levels of processing is for RTEMS interrupts which can
unblock threads, etc. The first level ensures that locking and
dispatching is handled.

As Gedare stated, there are also Raw interrupts which ignore
RTEMS and can't use RTEMS services. These should never
be anything but the highest priority interrupt in a system. The
RTEMS first level processing can switch to a dedicated
interrupt stack and mixing RTEMS and Raw IRQs can
screw up the stack handling as well as the interrupt nesting
count.

--joel

>
> Gedare
>
> > - My understanding of the interaction between RTEMS' interrupt manager
> > (i.e. support for nested interrupts and thread dispatch once an
> > interrupt ends) and the BSP's processor-specific interrupt manager
> > (code to use the APIC and IDT in my case) is that they're tied
> > together through the use of irq-generic.c's "bsp_interrupt_initialize"
> > - is that right? (m68k never seems to call it, though, so perhaps
> > not?)
> >
> > Sorry about the rambling! To reiterate, I'll likely figure it out
> > given enough time, so if the answers aren't at the top of your head, I
> > can figure it out without wasting your time :)
> >
> > [1] https://devel.rtems.org/ticket/3459#comment:11
> >
> > On Wed, Aug 1, 2018 at 3:18 AM, Joel Sherrill <joel at rtems.org> wrote:
> >>
> >>
> >> On Tue, Jul 31, 2018 at 3:05 PM, Amaan Cheval <amaan.cheval at gmail.com>
> >> wrote:
> >>>
> >>> Hm, I'm not sure what to look for in the other ports specifically,
> really.
> >>> The BSP porting documentation doesn't have a section on interrupts, so
> I'm
> >>> doing this on more of an "as it comes up" basis.
> >>>
> >>> What I've got right now (the interrupt handlers in C) are what I need
> for
> >>> calibrating the APIC timer (through the PIT) - so simply hooking IRQ0
> (for
> >>> the timer) and IRQ7 (spurious vector), since those are needed for the
> timer
> >>> work to continue.
> >>>
> >>> What constitutes as a requirement for basic interrupt support?
> >>
> >>
> >> There used to be a generic porting guide. I can see that this particular
> >> section
> >> has bit rotted some but the interrupt dispatching section. Some of this
> >> will have evolved to support SMP and fine grained locking but the
> >> pseudo-code
> >> here will give you a push toward the right line of thinking:
> >>
> >> https://docs.rtems.org/releases/rtemsdocs-4.10.2/
> share/rtems/html/porting/porting00034.html
> >>
> >> The idea is that you need to ensure RTEMS knows it is inside an
> interrupt
> >> and the current locking scheme (old was dispatching, new is ...) is
> honored.
> >>
> >> The ARM and PowerPC (plus RISCV) are good ports to look at for how SMP
> >> plays into this. But the CPU supplement is thin for their interrupt
> >> processing.
> >>
> >>
> >> This is the CPU Architecture supplement section for the m68k. This is a
> >> relatively simple
> >> architecture to describe. There is also a section for the i386 which
> reads
> >> similarly.
> >>
> >> https://docs.rtems.org/branches/master/cpu-supplement/m68xxx_and_
> coldfire.html#interrupt-processing
> >>
> >> Personally, I find the m68k a fairly easy processor to read assembly in.
> >> Look at cpukit/score/cpu/m68k/cpu_asm.S and _ISR_Handler to see what
> >> is done there w/o SMP. On the m68k _ISR_Handler is directly put into the
> >> vector table. But this isn't the most similar example for you.
> >>
> >> For the i386 (better example), it is in bsps/i386/shared/irq/irq_asm.S
> with
> >> the
> >> same name. There _ISR_Handler is installed via the
> DISTINCT_INTERRUPT_ENTRY
> >> macros at the bottom of the file where some prologue jumps to the common
> >> _ISR_Handler and then the actions are similar. Usually _ISR_Handler
> type of
> >> code ends up invoking a PIC decode method in normal C without an
> >> interrupt attribute.
> >>
> >> Long and multi-architecture answer but maybe that makes sense. The goal
> >> in ticker.exe is to take a number of tick interrupts which don't
> schedule
> >> and
> >> then take one that does -- it schedules a preemption of the idle thread.
> >>
> >> Hope this helps.
> >>
> >> --joel
> >>
> >>
> >>
> >>>
> >>>
> >>> On Wed, Aug 1, 2018, 1:29 AM Joel Sherrill <joel at rtems.org> wrote:
> >>>>
> >>>>
> >>>>
> >>>> On Tue, Jul 31, 2018 at 2:52 PM, Amaan Cheval <amaan.cheval at gmail.com
> >
> >>>> wrote:
> >>>>>
> >>>>> Hi Chris!
> >>>>>
> >>>>> I currently have code like this in
> >>>>> c/src/lib/libbsp/x86_64/amd64/Makefile.am:
> >>>>>
> >>>>>     librtemsbsp_a_SOURCES +=
> >>>>> ../../../../../../bsps/x86_64/amd64/interrupts/handlers.c
> >>>>>     # XXX: Needed to use GCC "interrupt" attribute directives - can
> we
> >>>>> pass these
> >>>>>     # flags only for the handlers.c source file (compile to an object
> >>>>> file first and
> >>>>>     # then link with the rest for librtemsbsp.a?)
> >>>>>     librtemsbsp_a_CFLAGS = -mgeneral-regs-only
> >>>>>
> >>>>> The CFLAGS arg is required to allow us to use
> >>>>> "__attribute__((interrupt))" to setup interrupt handlers in C. (See
> >>>>> [1] and ctrl+f "interrupt" for more.)
> >>>>>
> >>>>> Is there a way to not force the CFLAGS for _all_ of librtemsbsp, but
> >>>>> to limit it only to handlers.c?
> >>>>>
> >>>>> If not, is the above code something that would be acceptable to have
> >>>>> upstream?
> >>>>>
> >>>>> [1]
> >>>>> https://gcc.gnu.org/onlinedocs/gcc/x86-Function-
> Attributes.html#x86-Function-Attributes
> >>>>
> >>>>
> >>>> Are we basically talking about the outermost layer of your interrupt
> >>>> dispatching?
> >>>>
> >>>>
> >>>> Have you looked at the basic approach taken by the other ports? They
> end
> >>>> up switching the stack pointer to a dedicated stack on the outermost
> >>>> interrupt
> >>>> and, if a context switch/dispatch is needed, arrange for the
> interrupted
> >>>> task to call _Thread_Dispatch.But tinker with its stack so some
> registers
> >>>> are saved and it looks like it made the call itself.
> >>>>
> >>>> If you can do it in C, I am ok with an attribute. I just don't think
> you
> >>>> can pull off all the stack and return to dispatch magic that way.
> >>>>
> >>>> --joel
> >>>>
> >>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rtems.org/pipermail/devel/attachments/20180801/4162dd06/attachment-0002.html>