Including paravirtualization headers in RTEMS

Chris Johns chrisj at rtems.org
Mon May 26 12:23:52 UTC 2014 

On 26/05/2014 8:47 pm, Cláudio Silva wrote:
> Hello,
>
> I agree with you. POK should define a virtual CPU and RTEMS should
> support this POK virtual CPU interface like it does with any other
> "normal" architecture. Instead of writing directly to protected
> registers, RTEMS setups arguments and executes a trap. As a result,
> the functionality from the previously discussed abstraction layer can
> be implemented inside the hardware dependent sections of RTEMS.

Interesting idea and it makes sense, well to me. Given this is not what 
we have at the moment we need to find something else.

> My only question arises when we consider other interfaces that are not
> directly related with CPU/Timer/Interrupt virtualization, but are
> nonetheless necessary.  A simple example of this would be the ports,
> error reporting and partition state management interfaces that are
> provided by most partitioning kernels. This functionality must be
> exposed to the user application but RTEMS has no equivalent API
> defined (in score/sapi) because most of these APIs only make sense in
> a partitioned environment. We could implement them inside the BSP but
> that feels like an hack. Alternatively, we could implement them as a
> simple RTEMS lib.

This makes sense and an important point to consider. Can we add a 
libpartioning library to the cpukit and in this we implement an API and 
within that API provide the specific partitioning support for POK or any 
other host determined via configure ? Applications that use these APIs 
can only run in partitioned environments but that is ok. We would 
require a specific set of tests to test this library and built only when 
building a partitioned RTEMS.

Thanks for the discussion. It is helping me understand the problem much 
better.

Chris

>
> Cláudio
>
> On Mon, May 26, 2014 at 1:36 AM, Chris Johns <chrisj at rtems.org> wrote:
>> On 26/05/2014 4:39 am, Cláudio Silva wrote:
>>>
>>> Sorry all, miss send. Returning to where I was:
>>>
>>> Yes, Partitions are fully linked executables created "off-line". No
>>> linking occurs in run-time. The partition elf is simply loaded in to
>>> memory by a partition loader or by the bootloader.
>>>
>>> Typically, you have a thin abstraction layer defining simple functions
>>> that will trap into the partitioning kernel. Something like:
>>>
>>>
>>> https://github.com/t-crest/ospat/blob/master/libpok/middleware/portqueueingsend.c
>>>
>>
>> Thanks. This lines up with my understand.
>>
>> I think RTEMS needs to view POK or any virtual environment as a black box
>> and we cannot assume anything about the environment other than the ABI
>> presented to us. We cannot look through to the host system and do things as
>> a result. Maintaining a strict black box view may not be possible however we
>> should start with the black box and move to a position that is practical and
>> possible.
>>
>> We should not borrow (reference in the POK source tree) header files to pick
>> up common definitions directly or indirectly via static libraries. Any
>> interfaces defined by the partition's host need to be formally exported from
>> the project and made available. This defines the ABI we are working against.
>> I see POK is highly configurable so this complicates what is exported.
>>
>> If I build POK with port queueing enabled and build a guest that uses it
>> then I reconfigure POK and remove port queueing however I do not rebuild the
>> guest will POK detect the problem and handle the mismatch ?
>>
>>
>>> In POK + RTEMS case, this library is being compiled by POK's compiler
>>> and then linked to each partitions' executable.  When this partition is
>>> an RTEMS one, the remaining binaries (RTEMS+Application) were compiled
>>> with the RTEMS' compiler. Therefore you are linking two sets of
>>> binaries produced by different compilers.
>>
>>
>> We cannot do this. Given the type of applications expected to use POK I
>> would expect an audit to flag this as a fail. There are a number of reasons
>> why this is not a sustainable path forward, the most simple one is having to
>> have code in the RTEMS partition built with a compiler we do not control. In
>> this example the dependency is backwards, for example will the POK project
>> make sure its compiler can always build the interface and work with RTEMS,
>> ie all tests pass. I suspect the POK maintainers will have limited or no
>> real interest in RTEMS's development process and I see this as normal and
>> expected. I do not see this path as being sustainable and I do not support
>> it. As a hack to get things going it has been perfectly fine.
>>
>>
>>>
>>> I guess this is also true for the recently submitted Xtratum BSP.
>>>
>>
>> Current Xtratum BSP is not going to happen so lets put that one to one side.
>>
>>  From the RTEMS point of view I see POK as no different to any architecture
>> we support. We support running on hard IP in a device, for example a
>> Blackfin, a mix of hard IP and soft IP, for example a Zynq, soft IP, for
>> example a NIOS2, hard IP (or mixed soft IP) and software intervention in a
>> virtual environment, for example POK, and full software emulation via
>> simulation, for example qemu.
>>
>> For RTEMS to support an architecture we need separation so each part can
>> vary in version, the version dependencies flow down to RTEMS and not the
>> other direction, and we need a way to test. Testing means we have a stable
>> version of POK built on our test system and we build RTEMS against that.
>>
>>
>>> We have several options available:
>>> - Let the two binaries (RTEMS + abstraction lib), compiled with
>>> different compilers, to be linked together (how it is currently
>>> happening)
>>
>>
>> No.
>>
>>
>>> - Compile the partitioning kernel abstraction layer with the RTEMS
>>> compiler - this will result in a complex configuration and build
>>> procedure.
>>
>>
>> This is possible however is depends on how this is implement and where the
>> code resides.
>>
>> If the code lives inside the POK source tree and references files that are
>> core to POK we have an issue. This approach would place a burden on the POK
>> project to have a current set of RTEMS tools for the architectures supported
>> and to test the builds work. It means POK maintainers would need to monitor
>> and track the RTEMS development process. The dependency here is from RTEMS
>> to POK and as I mentioned before the dependence should be the other away
>> around.
>>
>> The code could be a separate project or sub-project of POK and both projects
>> need to maintain it. The POK project would need to maintain the ABI and
>> RTEMS would need to maintain the dependency with RTEMS tools and RTEMS. For
>> RTEMS how we interface to this because the next issue to solve.
>>
>>
>>> - Implement the functionality of the abstraction layer completely
>>> inside RTEMS as part of the libbsp/libcpu - This can be a good
>>> solution for the CPU/TImer/Interrupt part of the abstraction layer.
>>
>>
>> If the POK interface is stable and the header dependency can be made thin
>> this works for me. In the ideal world if something changes in the POK ABI
>> they would publish an update and we would import it. There would be
>> versioning issues, ie backwards compatibility, and this is normal and
>> nothing new for us.
>>
>>
>>> However, for other elements like the queueing and sampling port APIs
>>> that have no match inside RTEMS...
>>
>>
>> Could you please elaborate a little more ? Are these not supported because
>> RTEMS application's would not use them or this is still to be done ?
>>
>> Chris
>>
>>
>>> -
>>>
>>>
>>> Best Regards,
>>> Cláudio
>>>
>>> On Sun, May 25, 2014 at 7:22 PM, Cláudio Silva <claudiodcsilva at gmail.com>
>>> wrote:
>>>>
>>>> Hello Chris,
>>>>
>>>> Yes, Partitions are fully linked executables created "off-line". No
>>>> linking occurs in run-time. The partition elf is simply loaded in to
>>>> memory by a partition loader or the bootloader.
>>>>
>>>> Typically, you have a thin abstraction layer defining simple functions
>>>> that will trap into the partitioning kernel. Something like:
>>>>
>>>>
>>>> On Sat, May 24, 2014 at 12:27 AM, Chris Johns <chrisj at rtems.org> wrote:
>>>>>
>>>>> On 23/05/2014 6:46 pm, Philipp Eppelt wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 05/21/2014 03:54 PM, Gedare Bloom wrote:> On Wed, May 21, 2014 at
>>>>>>> 4:04
>>>>>>> AM, Christian Mauderer
>>>>>>>>
>>>>>>>>
>>>>>>>> <Christian.Mauderer at embedded-brains.de> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> First of all: Thanks for your comments. You will find answers below.
>>>>>>>>>
>>>>>>>>> Am 20.05.2014 16:58, schrieb Gedare Bloom:
>>>>>>>>>
>>>>>>>>> That is correct. It's part of XtratuM. Is there some preferred way
>>>>>>>>> of
>>>>>>>>> marking such headers?
>>>>>>>>>
>>>>>>>> Not that I know of. We have discussed a similar issue with the POK
>>>>>>>> paravirtualization project. The problem is to allow external code
>>>>>>>> linking to RTEMS. The design should be considered carefully and
>>>>>>>> probably discussed in a separate thread.
>>>>>>
>>>>>>
>>>>>>
>>>>>> To pick up this discussion, I like first to picture the problem, second
>>>>>> to collect possible solutions and third name some known problems with
>>>>>> the solutions.
>>>>>>
>>>>>
>>>>> Thank you for posting this email. I followed the links and looked at the
>>>>> code and realised I need to ask more questions.
>>>>>
>>>>>
>>>>>>
>>>>>> In a paravirtualized environment we need to make function calls from
>>>>>> our
>>>>>> guest system to the host system. The implementation of these functions
>>>>>> rely sole on the host system.
>>>>>
>>>>>
>>>>>
>>>>> Are they function calls ? To me a function call assumes some sort of
>>>>> linking
>>>>> process between the guest and the host system either static or via some
>>>>> form
>>>>> of runtime binding, ie a link editor. I also assume a guest is an RTEMS
>>>>> partition executable and the host is the POK microkernel providing the
>>>>> partitioned environment in some sort of protective environment.
>>>>>
>>>>>
>>>>>> Hence, at compile time the guest system is missing the implementation
>>>>>> of
>>>>>> this host specific functions.
>>>>>
>>>>>
>>>>>
>>>>> My understanding is partitions are loaded into an ELF image that POK
>>>>> loads
>>>>> and executes. Is there some sort of linking happening when the partition
>>>>> is
>>>>> written into the ELF image or when it is being loaded ?
>>>>>
>>>>> For me this is the really important because I am wanting to first
>>>>> understand
>>>>> the which pieces of code are compiled into object files and by which
>>>>> compilers and then archived into static libraries and then linked to any
>>>>> RTEMS code compiled with an RTEMS compiler. I do not understand if there
>>>>> are
>>>>> function calls into the POK kernel or syscall type calls which I
>>>>> consider an
>>>>> ABI, ie a trap or VM exception.
>>>>>
>>>>> If I can understand these needs I will be in a better position to help
>>>>> with
>>>>> the remaining topics in this email.
>>>>>
>>>>> Chris
>>>>>
>>>>>
>>>>>>
>>>>>> To provide the guest system with the function implementation there are
>>>>>> a
>>>>>> couple of possible solutions.
>>>>>>
>>>>>> (A) Provide a host library
>>>>>> The host compiles a library containing the function implementations
>>>>>> including all dependencies. This host library is passed to the guest
>>>>>> system, to resolve the `undefined reference` errors during link time.
>>>>>> In the case of POK+RTEMS the resulting RTEMS binary is a valid POK
>>>>>> partition and can run without further modifications.
>>>>>> The recently propose XantruM BSP seems to follow a similar approach.
>>>>>>
>>>>>> (B) Use function stubs and build a jump table in the host
>>>>>> The guest system makes use of function stubs, written into a special
>>>>>> section in the binary to build. This section is analysed by the host
>>>>>> system to intercept function calls  and replace them with the host
>>>>>> implementation of the calls.
>>>>>> This approach is used in L4Linux and I gave it a shot in L4RTEMS.
>>>>>> [1] shows the macro for the function stubs, [2] the hosts resolver
>>>>>> part.
>>>>>>
>>>>>> (C) add your solution
>>>>>>
>>>>>>
>>>>>> Problems with the solutions
>>>>>>
>>>>>> P(A) Dependencies and foreign code
>>>>>> The library is host code, which has to be maintained, which can get out
>>>>>> of sync between BSP and host. Licensing is another question.
>>>>>> To my knowledge this approach won't get merged upstream.
>>>>>>
>>>>>> P(B) Support by the host system
>>>>>> The host system needs a piece of code, loading the guest binary and
>>>>>> providing it with knowledge about the environment (external_resolver).
>>>>>> We haven't discussed this approach in detail.
>>>>>>
>>>>>>
>>>>>> I hope, this gives a good overview and we can have a fruitful
>>>>>> discussion.
>>>>>>
>>>>>> Cheers,
>>>>>> Philipp
>>>>>>
>>>>>>
>>>>>> [1]
>>>>>>
>>>>>>
>>>>>> https://github.com/phipse/L4RTEMS/blob/master/c/src/lib/libbsp/l4vcpu/pc386/startup/l4lib.h
>>>>>> [2]
>>>>>>
>>>>>>
>>>>>> https://github.com/phipse/L4RTEMS/blob/f5a32ed0121b3e5f0ac8ae4c53e636eab2257fb1/l4/pkg/RTEMS_wrapper/server/src/res.c
>>>>>> _______________________________________________
>>>>>> rtems-devel mailing list
>>>>>> rtems-devel at rtems.org
>>>>>> http://www.rtems.org/mailman/listinfo/rtems-devel
>>>>>>
>>>>> _______________________________________________
>>>>> rtems-devel mailing list
>>>>> rtems-devel at rtems.org
>>>>> http://www.rtems.org/mailman/listinfo/rtems-devel

More information about the devel mailing list