Including paravirtualization headers in RTEMS

Gedare Bloom gedare at rtems.org
Mon May 26 18:46:59 UTC 2014 

On Mon, May 26, 2014 at 1:10 PM, Cláudio Silva <claudiodcsilva at gmail.com> wrote:
> RTEMS in POK is converging towards such a design, with the exception
> of still using libpok as an abstraction layer. Following this
> discussion, if everyone involved agrees, we should start to guide GSOC
> students working on this to implement some of the functionality of
> libpok inside RTEMS (all functionality related with
> CPU/Interrupt/Timer/Console virtualization). Everything up to the trap
> instruction should be in libbsp/libbpu.
Makes sense to me.

> With regard to the "libpartitioning", I really don't know where to
> place it. Your proposal seems fine, my only question is how to manage
> the different functionalities provided by different kernels. When you
> have a common functionality you can just abstract it using a score
> implementation that is translated to the specific partitioning kernel
> interface. It is a bit more ugly if we end up with APIs that only are
> used in partitioned RTEMS and for a single partitioning kernel.
>
I would guess most of the partition services will be related to some
standard e.g. ARINC/FACE, so that might be a useful place to start
when considering what the RTEMS API to the partitioning kernel should
look like.

We can also consider a "pass-through" API that allows to execute
arbitrary functions passing arbitrary arguments to the partitioning
kernel. (Think something like syscalls...
rtems_partitioning_execute(...); Not pretty or easy to test, but it
may reduce such clutter if some partitioning kernels offer a lot of
excess services that are not common or part of the standard.

Gedare

> Cláudio
>
> On Mon, May 26, 2014 at 1:23 PM, Chris Johns <chrisj at rtems.org> wrote:
>> On 26/05/2014 8:47 pm, Cláudio Silva wrote:
>>>
>>> Hello,
>>>
>>> I agree with you. POK should define a virtual CPU and RTEMS should
>>> support this POK virtual CPU interface like it does with any other
>>> "normal" architecture. Instead of writing directly to protected
>>> registers, RTEMS setups arguments and executes a trap. As a result,
>>> the functionality from the previously discussed abstraction layer can
>>> be implemented inside the hardware dependent sections of RTEMS.
>>
>>
>> Interesting idea and it makes sense, well to me. Given this is not what we
>> have at the moment we need to find something else.
>>
>>
>>> My only question arises when we consider other interfaces that are not
>>> directly related with CPU/Timer/Interrupt virtualization, but are
>>> nonetheless necessary.  A simple example of this would be the ports,
>>> error reporting and partition state management interfaces that are
>>> provided by most partitioning kernels. This functionality must be
>>> exposed to the user application but RTEMS has no equivalent API
>>> defined (in score/sapi) because most of these APIs only make sense in
>>> a partitioned environment. We could implement them inside the BSP but
>>> that feels like an hack. Alternatively, we could implement them as a
>>> simple RTEMS lib.
>>
>>
>> This makes sense and an important point to consider. Can we add a
>> libpartioning library to the cpukit and in this we implement an API and
>> within that API provide the specific partitioning support for POK or any
>> other host determined via configure ? Applications that use these APIs can
>> only run in partitioned environments but that is ok. We would require a
>> specific set of tests to test this library and built only when building a
>> partitioned RTEMS.
>>
>> Thanks for the discussion. It is helping me understand the problem much
>> better.
>>
>> Chris
>>
>>
>>>
>>> Cláudio
>>>
>>> On Mon, May 26, 2014 at 1:36 AM, Chris Johns <chrisj at rtems.org> wrote:
>>>>
>>>> On 26/05/2014 4:39 am, Cláudio Silva wrote:
>>>>>
>>>>>
>>>>> Sorry all, miss send. Returning to where I was:
>>>>>
>>>>> Yes, Partitions are fully linked executables created "off-line". No
>>>>> linking occurs in run-time. The partition elf is simply loaded in to
>>>>> memory by a partition loader or by the bootloader.
>>>>>
>>>>> Typically, you have a thin abstraction layer defining simple functions
>>>>> that will trap into the partitioning kernel. Something like:
>>>>>
>>>>>
>>>>>
>>>>> https://github.com/t-crest/ospat/blob/master/libpok/middleware/portqueueingsend.c
>>>>>
>>>>
>>>> Thanks. This lines up with my understand.
>>>>
>>>> I think RTEMS needs to view POK or any virtual environment as a black box
>>>> and we cannot assume anything about the environment other than the ABI
>>>> presented to us. We cannot look through to the host system and do things
>>>> as
>>>> a result. Maintaining a strict black box view may not be possible however
>>>> we
>>>> should start with the black box and move to a position that is practical
>>>> and
>>>> possible.
>>>>
>>>> We should not borrow (reference in the POK source tree) header files to
>>>> pick
>>>> up common definitions directly or indirectly via static libraries. Any
>>>> interfaces defined by the partition's host need to be formally exported
>>>> from
>>>> the project and made available. This defines the ABI we are working
>>>> against.
>>>> I see POK is highly configurable so this complicates what is exported.
>>>>
>>>> If I build POK with port queueing enabled and build a guest that uses it
>>>> then I reconfigure POK and remove port queueing however I do not rebuild
>>>> the
>>>> guest will POK detect the problem and handle the mismatch ?
>>>>
>>>>
>>>>> In POK + RTEMS case, this library is being compiled by POK's compiler
>>>>> and then linked to each partitions' executable.  When this partition is
>>>>> an RTEMS one, the remaining binaries (RTEMS+Application) were compiled
>>>>> with the RTEMS' compiler. Therefore you are linking two sets of
>>>>> binaries produced by different compilers.
>>>>
>>>>
>>>>
>>>> We cannot do this. Given the type of applications expected to use POK I
>>>> would expect an audit to flag this as a fail. There are a number of
>>>> reasons
>>>> why this is not a sustainable path forward, the most simple one is having
>>>> to
>>>> have code in the RTEMS partition built with a compiler we do not control.
>>>> In
>>>> this example the dependency is backwards, for example will the POK
>>>> project
>>>> make sure its compiler can always build the interface and work with
>>>> RTEMS,
>>>> ie all tests pass. I suspect the POK maintainers will have limited or no
>>>> real interest in RTEMS's development process and I see this as normal and
>>>> expected. I do not see this path as being sustainable and I do not
>>>> support
>>>> it. As a hack to get things going it has been perfectly fine.
>>>>
>>>>
>>>>>
>>>>> I guess this is also true for the recently submitted Xtratum BSP.
>>>>>
>>>>
>>>> Current Xtratum BSP is not going to happen so lets put that one to one
>>>> side.
>>>>
>>>>  From the RTEMS point of view I see POK as no different to any
>>>> architecture
>>>> we support. We support running on hard IP in a device, for example a
>>>> Blackfin, a mix of hard IP and soft IP, for example a Zynq, soft IP, for
>>>> example a NIOS2, hard IP (or mixed soft IP) and software intervention in
>>>> a
>>>> virtual environment, for example POK, and full software emulation via
>>>> simulation, for example qemu.
>>>>
>>>> For RTEMS to support an architecture we need separation so each part can
>>>> vary in version, the version dependencies flow down to RTEMS and not the
>>>> other direction, and we need a way to test. Testing means we have a
>>>> stable
>>>> version of POK built on our test system and we build RTEMS against that.
>>>>
>>>>
>>>>> We have several options available:
>>>>> - Let the two binaries (RTEMS + abstraction lib), compiled with
>>>>> different compilers, to be linked together (how it is currently
>>>>> happening)
>>>>
>>>>
>>>>
>>>> No.
>>>>
>>>>
>>>>> - Compile the partitioning kernel abstraction layer with the RTEMS
>>>>> compiler - this will result in a complex configuration and build
>>>>> procedure.
>>>>
>>>>
>>>>
>>>> This is possible however is depends on how this is implement and where
>>>> the
>>>> code resides.
>>>>
>>>> If the code lives inside the POK source tree and references files that
>>>> are
>>>> core to POK we have an issue. This approach would place a burden on the
>>>> POK
>>>> project to have a current set of RTEMS tools for the architectures
>>>> supported
>>>> and to test the builds work. It means POK maintainers would need to
>>>> monitor
>>>> and track the RTEMS development process. The dependency here is from
>>>> RTEMS
>>>> to POK and as I mentioned before the dependence should be the other away
>>>> around.
>>>>
>>>> The code could be a separate project or sub-project of POK and both
>>>> projects
>>>> need to maintain it. The POK project would need to maintain the ABI and
>>>> RTEMS would need to maintain the dependency with RTEMS tools and RTEMS.
>>>> For
>>>> RTEMS how we interface to this because the next issue to solve.
>>>>
>>>>
>>>>> - Implement the functionality of the abstraction layer completely
>>>>> inside RTEMS as part of the libbsp/libcpu - This can be a good
>>>>> solution for the CPU/TImer/Interrupt part of the abstraction layer.
>>>>
>>>>
>>>>
>>>> If the POK interface is stable and the header dependency can be made thin
>>>> this works for me. In the ideal world if something changes in the POK ABI
>>>> they would publish an update and we would import it. There would be
>>>> versioning issues, ie backwards compatibility, and this is normal and
>>>> nothing new for us.
>>>>
>>>>
>>>>> However, for other elements like the queueing and sampling port APIs
>>>>> that have no match inside RTEMS...
>>>>
>>>>
>>>>
>>>> Could you please elaborate a little more ? Are these not supported
>>>> because
>>>> RTEMS application's would not use them or this is still to be done ?
>>>>
>>>> Chris
>>>>
>>>>
>>>>> -
>>>>>
>>>>>
>>>>> Best Regards,
>>>>> Cláudio
>>>>>
>>>>> On Sun, May 25, 2014 at 7:22 PM, Cláudio Silva
>>>>> <claudiodcsilva at gmail.com>
>>>>> wrote:
>>>>>>
>>>>>>
>>>>>> Hello Chris,
>>>>>>
>>>>>> Yes, Partitions are fully linked executables created "off-line". No
>>>>>> linking occurs in run-time. The partition elf is simply loaded in to
>>>>>> memory by a partition loader or the bootloader.
>>>>>>
>>>>>> Typically, you have a thin abstraction layer defining simple functions
>>>>>> that will trap into the partitioning kernel. Something like:
>>>>>>
>>>>>>
>>>>>> On Sat, May 24, 2014 at 12:27 AM, Chris Johns <chrisj at rtems.org> wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 23/05/2014 6:46 pm, Philipp Eppelt wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 05/21/2014 03:54 PM, Gedare Bloom wrote:> On Wed, May 21, 2014 at
>>>>>>>>> 4:04
>>>>>>>>> AM, Christian Mauderer
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> <Christian.Mauderer at embedded-brains.de> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> First of all: Thanks for your comments. You will find answers
>>>>>>>>>>> below.
>>>>>>>>>>>
>>>>>>>>>>> Am 20.05.2014 16:58, schrieb Gedare Bloom:
>>>>>>>>>>>
>>>>>>>>>>> That is correct. It's part of XtratuM. Is there some preferred way
>>>>>>>>>>> of
>>>>>>>>>>> marking such headers?
>>>>>>>>>>>
>>>>>>>>>> Not that I know of. We have discussed a similar issue with the POK
>>>>>>>>>> paravirtualization project. The problem is to allow external code
>>>>>>>>>> linking to RTEMS. The design should be considered carefully and
>>>>>>>>>> probably discussed in a separate thread.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> To pick up this discussion, I like first to picture the problem,
>>>>>>>> second
>>>>>>>> to collect possible solutions and third name some known problems with
>>>>>>>> the solutions.
>>>>>>>>
>>>>>>>
>>>>>>> Thank you for posting this email. I followed the links and looked at
>>>>>>> the
>>>>>>> code and realised I need to ask more questions.
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> In a paravirtualized environment we need to make function calls from
>>>>>>>> our
>>>>>>>> guest system to the host system. The implementation of these
>>>>>>>> functions
>>>>>>>> rely sole on the host system.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Are they function calls ? To me a function call assumes some sort of
>>>>>>> linking
>>>>>>> process between the guest and the host system either static or via
>>>>>>> some
>>>>>>> form
>>>>>>> of runtime binding, ie a link editor. I also assume a guest is an
>>>>>>> RTEMS
>>>>>>> partition executable and the host is the POK microkernel providing the
>>>>>>> partitioned environment in some sort of protective environment.
>>>>>>>
>>>>>>>
>>>>>>>> Hence, at compile time the guest system is missing the implementation
>>>>>>>> of
>>>>>>>> this host specific functions.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> My understanding is partitions are loaded into an ELF image that POK
>>>>>>> loads
>>>>>>> and executes. Is there some sort of linking happening when the
>>>>>>> partition
>>>>>>> is
>>>>>>> written into the ELF image or when it is being loaded ?
>>>>>>>
>>>>>>> For me this is the really important because I am wanting to first
>>>>>>> understand
>>>>>>> the which pieces of code are compiled into object files and by which
>>>>>>> compilers and then archived into static libraries and then linked to
>>>>>>> any
>>>>>>> RTEMS code compiled with an RTEMS compiler. I do not understand if
>>>>>>> there
>>>>>>> are
>>>>>>> function calls into the POK kernel or syscall type calls which I
>>>>>>> consider an
>>>>>>> ABI, ie a trap or VM exception.
>>>>>>>
>>>>>>> If I can understand these needs I will be in a better position to help
>>>>>>> with
>>>>>>> the remaining topics in this email.
>>>>>>>
>>>>>>> Chris
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> To provide the guest system with the function implementation there
>>>>>>>> are
>>>>>>>> a
>>>>>>>> couple of possible solutions.
>>>>>>>>
>>>>>>>> (A) Provide a host library
>>>>>>>> The host compiles a library containing the function implementations
>>>>>>>> including all dependencies. This host library is passed to the guest
>>>>>>>> system, to resolve the `undefined reference` errors during link time.
>>>>>>>> In the case of POK+RTEMS the resulting RTEMS binary is a valid POK
>>>>>>>> partition and can run without further modifications.
>>>>>>>> The recently propose XantruM BSP seems to follow a similar approach.
>>>>>>>>
>>>>>>>> (B) Use function stubs and build a jump table in the host
>>>>>>>> The guest system makes use of function stubs, written into a special
>>>>>>>> section in the binary to build. This section is analysed by the host
>>>>>>>> system to intercept function calls  and replace them with the host
>>>>>>>> implementation of the calls.
>>>>>>>> This approach is used in L4Linux and I gave it a shot in L4RTEMS.
>>>>>>>> [1] shows the macro for the function stubs, [2] the hosts resolver
>>>>>>>> part.
>>>>>>>>
>>>>>>>> (C) add your solution
>>>>>>>>
>>>>>>>>
>>>>>>>> Problems with the solutions
>>>>>>>>
>>>>>>>> P(A) Dependencies and foreign code
>>>>>>>> The library is host code, which has to be maintained, which can get
>>>>>>>> out
>>>>>>>> of sync between BSP and host. Licensing is another question.
>>>>>>>> To my knowledge this approach won't get merged upstream.
>>>>>>>>
>>>>>>>> P(B) Support by the host system
>>>>>>>> The host system needs a piece of code, loading the guest binary and
>>>>>>>> providing it with knowledge about the environment
>>>>>>>> (external_resolver).
>>>>>>>> We haven't discussed this approach in detail.
>>>>>>>>
>>>>>>>>
>>>>>>>> I hope, this gives a good overview and we can have a fruitful
>>>>>>>> discussion.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Philipp
>>>>>>>>
>>>>>>>>
>>>>>>>> [1]
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> https://github.com/phipse/L4RTEMS/blob/master/c/src/lib/libbsp/l4vcpu/pc386/startup/l4lib.h
>>>>>>>> [2]
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> https://github.com/phipse/L4RTEMS/blob/f5a32ed0121b3e5f0ac8ae4c53e636eab2257fb1/l4/pkg/RTEMS_wrapper/server/src/res.c
>>>>>>>> _______________________________________________
>>>>>>>> rtems-devel mailing list
>>>>>>>> rtems-devel at rtems.org
>>>>>>>> http://www.rtems.org/mailman/listinfo/rtems-devel
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> rtems-devel mailing list
>>>>>>> rtems-devel at rtems.org
>>>>>>> http://www.rtems.org/mailman/listinfo/rtems-devel
>
> _______________________________________________
> rtems-devel mailing list
> rtems-devel at rtems.org
> http://www.rtems.org/mailman/listinfo/rtems-devel

More information about the devel mailing list