libbsd development policy clarification needed?

Mon Feb 6 15:27:04 UTC 2023

On 2023-02-06 16:06, Gedare Bloom wrote:
> On Mon, Feb 6, 2023 at 2:26 AM Christian MAUDERER
> <christian.mauderer at embedded-brains.de> wrote:
>>
>> On 2023-02-06 10:02, Christian MAUDERER wrote:
>>> On 2023-02-05 11:38, Christian Mauderer wrote:
>>>>
>>>>
>>>> Am 04.02.23 um 01:25 schrieb Gedare Bloom:
>>>>> On Fri, Feb 3, 2023 at 3:29 PM Christian Mauderer <oss at c-mauderer.de>
>>>>> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> Am 3. Februar 2023 22:52:48 MEZ schrieb Gedare Bloom
>>>>>> <gedare at rtems.org>:
>>>>>>> On Fri, Feb 3, 2023 at 2:39 PM Christian Mauderer
>>>>>>> <oss at c-mauderer.de> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Am 3. Februar 2023 22:12:06 MEZ schrieb Gedare Bloom
>>>>>>>> <gedare at rtems.org>:
>>>>>>>>> On Fri, Feb 3, 2023 at 12:52 PM <oss at c-mauderer.de> wrote:
>>>>>>>>>>
>>>>>>>>>> Hello Gedare,
>>>>>>>>>>
>>>>>>>>>> Am 03.02.23 um 19:51 schrieb Gedare Bloom:
>>>>>>>>>>> On Thu, Feb 2, 2023 at 11:24 PM Christian MAUDERER
>>>>>>>>>>> <christian.mauderer at embedded-brains.de> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hello Karel,
>>>>>>>>>>>>
>>>>>>>>>>>> On 2023-02-02 12:43, Karel Gardas wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>       Guys,
>>>>>>>>>>>>>
>>>>>>>>>>>>> recently I needed to work with RTEMS/NFS. As this is provided
>>>>>>>>>>>>> by libbsd
>>>>>>>>>>>>> I took this and following two sentences below from master branch
>>>>>>>>>>>>> description provided in README I took as granted that master
>>>>>>>>>>>>> does have
>>>>>>>>>>>>> all the features which are currently available and provided
>>>>>>>>>>>>> by the project:
>>>>>>>>>>>>>
>>>>>>>>>>>>> "This branch must be used for libbsd development. Back ports
>>>>>>>>>>>>> to the
>>>>>>>>>>>>> 6-freebsd-12 are allowed."
>>>>>>>>>>>>>
>>>>>>>>>>>>> I was surprised to be proven wrong then by Fabrizio here:
>>>>>>>>>>>>> https://devel.rtems.org/ticket/4723
>>>>>>>>>>>>>
>>>>>>>>>>>>> and by later investigation which shows that 6-freebsd-12 branch
>>>>>>>>>>>>> accumulated NFS work by Chris done in 2021 which is not
>>>>>>>>>>>>> presented on
>>>>>>>>>>>>> master. I've investigated just NFS as this was my focus here.
>>>>>>>>>>>>>
>>>>>>>>>>>>> So if 6-freebsd-12 became development branch of some sort,
>>>>>>>>>>>>> then it would
>>>>>>>>>>>>> be great to have that clarified in the project README file to
>>>>>>>>>>>>> prevent
>>>>>>>>>>>>> users confusion? Or if the policy is still the same, then
>>>>>>>>>>>>> perhaps some
>>>>>>>>>>>>> branch sync is needed here?
>>>>>>>>>>>>
>>>>>>>>>>>> That currently is an open issue. Basically there is a pending
>>>>>>>>>>>> patch set
>>>>>>>>>>>> that should fix that since several months. But there is a
>>>>>>>>>>>> disagreement
>>>>>>>>>>>> about some of the changes in that patch set (and about the
>>>>>>>>>>>> patches
>>>>>>>>>>>> checked in to 6-freebsd-12). Therefore, it still hasn't been
>>>>>>>>>>>> merged.
>>>>>>>>>>>>
>>>>>>>>>>>> If you want to know some more about the problematic points, I
>>>>>>>>>>>> recommend
>>>>>>>>>>>> reading this (long) thread:
>>>>>>>>>>>>
>>>>>>>>>>>> https://lists.rtems.org/pipermail/devel/2023-January/074164.html
>>>>>>>>>>>>
>>>>>>>>>>>> The statement that development has to happen on the master
>>>>>>>>>>>> branch is
>>>>>>>>>>>> still true. The master is intended to track the FreeBSD upstream
>>>>>>>>>>>> development. Only changes on that branch are guaranteed to
>>>>>>>>>>>> live through
>>>>>>>>>>>> an upgrade to a newer base version of FreeBSD. It's very
>>>>>>>>>>>> unfortunate,
>>>>>>>>>>>> that there are some patches on the 6-freebsd-12 branch only.
>>>>>>>>>>>> On the long
>>>>>>>>>>>> term, that issue has to be resolved.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I have been investigating this problem in the background, and I
>>>>>>>>>>> have
>>>>>>>>>>> some findings and some questions. First, I have found that
>>>>>>>>>>> there is a
>>>>>>>>>>> most-common ancestor between master and 6-freebsd-12 at commit
>>>>>>>>>>> https://git.rtems.org/rtems-libbsd/commit/?h=6-freebsd-12&id=2ce13cf6dc73855f28bc7edbbc64dc4b482a4976
>>>>>>>>>>> This is at least promising that the discrepancy between the
>>>>>>>>>>> branches
>>>>>>>>>>> can be resolved.
>>>>>>>>>>>
>>>>>>>>>>> The proposed pending patch set to "fix" the NFS issue does not
>>>>>>>>>>> fix the
>>>>>>>>>>> underlying problem. Instead, it introduces further divergence
>>>>>>>>>>> between
>>>>>>>>>>> the branches. I would instead suggest that we should resolve to
>>>>>>>>>>> fix
>>>>>>>>>>> the underlying problem. I can see two paths forward.
>>>>>>>>>>>
>>>>>>>>>>> 1. Abandon 6-freebsd-12 after releasing 6. This is probably not
>>>>>>>>>>> ideal
>>>>>>>>>>> since what I understand is some users have projects based on
>>>>>>>>>>> 6-freebsd-12 and would like an upgrade path. (I guess there is
>>>>>>>>>>> also
>>>>>>>>>>> the option to abandon master, which also makes little sense.)
>>>>>>>>>>
>>>>>>>>>> A variant for this would be to introduce a 6-freebsd-13 that is
>>>>>>>>>> based on
>>>>>>>>>> the master branch as soon as we have one. That would allow a longer
>>>>>>>>>> maintenance because FreeBSD 12 reaches it's EoL December 2023.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2. Pull commits from 6-freebsd-12 into master to make sure
>>>>>>>>>>> master is
>>>>>>>>>>> the development head. in the future, reject patches that only go
>>>>>>>>>>> toward release branches. This has its own problems too. It can
>>>>>>>>>>> realistically only be done in three ways:
>>>>>>>>>>
>>>>>>>>>> Please note that Sebastian mentioned that the file descriptors
>>>>>>>>>> broke the
>>>>>>>>>> NTP support (at least I think it was NTP; possible that it was
>>>>>>>>>> another
>>>>>>>>>> submodule). So picking the current version of the patches into the
>>>>>>>>>> master without adding fixes makes the master unusable for some
>>>>>>>>>> cases.
>>>>>>>>>>
>>>>>>>>> Fixing the problems after making the branches the same will be
>>>>>>>>> better
>>>>>>>>> for the long-term, if we can find a path to do it.
>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2a: Rebase master and cherry-pick commits from 6-freebsd-12 and
>>>>>>>>>>> master
>>>>>>>>>>> back into master. This rewrites the history of master, and
>>>>>>>>>>> unfortunately will cause the head of 5-freebsd-12 and the tags for
>>>>>>>>>>> rtems-5 to no longer exist on the master branch. They will
>>>>>>>>>>> still exist
>>>>>>>>>>> in the '5' branch. The advantage is in the end there will be a
>>>>>>>>>>> linear
>>>>>>>>>>> history of development on master that reflects the timeline of
>>>>>>>>>>> actual
>>>>>>>>>>> development that spanned both branches. Theoretically, this should
>>>>>>>>>>> make it easier to git-bisect.
>>>>>>>>>>>
>>>>>>>>>>> 2b: Cherry-pick commits from 6-freebsd-12 to master and fix
>>>>>>>>>>> conflicts.
>>>>>>>>>>> This puts all the missing commits from 6-freebsd-12 on to the
>>>>>>>>>>> current
>>>>>>>>>>> head of master. I don't know how messy this would be. It ends up
>>>>>>>>>>> making the history of master convoluted to understand, with
>>>>>>>>>>> fairly old
>>>>>>>>>>> commits from 2018 being placed on top of newer commits from 2020s.
>>>>>>>>>>>
>>>>>>>>>>> 2c: Merge 6-freebsd-12 into master and fixup conflicts in the
>>>>>>>>>>> merge
>>>>>>>>>>> commit. This is pretty similar to 2a but ends up with a non-linear
>>>>>>>>>>> history and a merge commit. It may be a fairly complex merge
>>>>>>>>>>> commit.
>>>>>>>>>>
>>>>>>>>>> For all of the 2x solutions: The commits from 6-freebsd-12 can't
>>>>>>>>>> just be
>>>>>>>>>> cherry-picked. You have to re-import the NFS files from the FreeBSD
>>>>>>>>>> master version that is used as base for the current libbsd master.
>>>>>>>>>> Otherwise we mix different FreeBSD source versions. We had that
>>>>>>>>>> some
>>>>>>>>>> time back in libbsd and Sebastian needed a lot of time cleaning
>>>>>>>>>> that up.
>>>>>>>>>>
>>>>>>>>> Understood.
>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> To get a sense of the difference between the two branches, I
>>>>>>>>>>> have done
>>>>>>>>>>> the following command:
>>>>>>>>>>> $ git log --pretty=oneline master...6-freebsd-12 > ../log.txt
>>>>>>>>>>> This uses the ... (three-dot) Symmetric Difference Notation. The
>>>>>>>>>>> result of that is a 750 line file, so 750 commits are different
>>>>>>>>>>> between the two branches. Some of those commits are actually
>>>>>>>>>>> the same
>>>>>>>>>>> content, but they have different parents so different hashes. In a
>>>>>>>>>>> rebase or merge situation, those commits should end up the
>>>>>>>>>>> same. There
>>>>>>>>>>> may be other git-fu to find just the patches that are unique in
>>>>>>>>>>> the
>>>>>>>>>>> two branches.
>>>>>>>>>>
>>>>>>>>>> 750 commits are a bit too much. The order of patches on master and
>>>>>>>>>> 6-freebsd-12 isn't always the same. I wrote a small python
>>>>>>>>>> script to
>>>>>>>>>> find the differences somewhere in 2021 when I needed the
>>>>>>>>>> differences in
>>>>>>>>>> a discussion with (I think) Joel. It compares based on author and
>>>>>>>>>> subject which gives a quite good estimate for libbsd. You can
>>>>>>>>>> find the
>>>>>>>>>> script here:
>>>>>>>>>>
>>>>>>>>>>      https://gist.github.com/a82e21eb250cb96c3a36f107b92dab09
>>>>>>>>>>
>>>>>>>>>> That shows 138 different commits. 86 are only on 6-freebsd-12.
>>>>>>>>>> Complete
>>>>>>>>>> output is here:
>>>>>>>>>>
>>>>>>>>>>      https://gist.github.com/93a2b19a5bf4cd8a6263ae29aa359ac2
>>>>>>>>>>
>>>>>>>>>>    From these you can ignore the "Update to FreeBSD..." commits
>>>>>>>>>> and some
>>>>>>>>>> of the cleanup patches. I assume that quite some of them can be
>>>>>>>>>> cherry-picked with only small or no changes. For example patches
>>>>>>>>>> that
>>>>>>>>>> only add or update drivers in rtemsbsd should work without
>>>>>>>>>> problems. So
>>>>>>>>>> I think a more realistic number of problematic patches should be
>>>>>>>>>> 30 to 50.
>>>>>>>>>>
>>>>>>>>> Oh, thanks, this is nice. It may be a reasonable place to start by
>>>>>>>>> looking at what is "Only on master" and to consider back-porting to
>>>>>>>>> 6-freebsd-12. That may reduce the delta somewhat in a way that makes
>>>>>>>>> sense.
>>>>>>>>
>>>>>>>> We really should do the other direction. Not all features oft
>>>>>>>> master have to be on the stable branch. But all features of the
>>>>>>>> stable one should be on master.
>>>>>>>>
>>>>>>> I agree, but since there are many fewer changes only on master, if we
>>>>>>> can merge those into 6-freebsd-12, then it becomes simpler to just
>>>>>>> call 6-freebsd-12 as the new master. We can replace the current master
>>>>>>> with this branch, and then update freebsd versions. I think this can
>>>>>>> be a lower burden process to follow, except it requires acceptance of
>>>>>>> the commits that are on 6-freebsd-12, which is a different matter.
>>>>>>
>>>>>> You can't just make 6-freebsd-12 a new master. It tracks the FreeBSD
>>>>>> 12 branch.
>>>>>>
>>>>>> Every block of updates to a newer FreeBSD base revision was several
>>>>>> days of work for Sebastian. There are a few on master.
>>>>>>
>>>>>> An upgrade from a branch tracking FreeBSD 12 to one tracking FreeBSD
>>>>>> 13 is most likely more something in the several week range of work.
>>>>>>
>>>>>>>
>>>>>>>> master tracks the FreeBSD master. As soon as it reaches the point
>>>>>>>> where FreeBSD branched off version 13, we can create a
>>>>>>>> 6-freebsd-13 or 7-freebsd-13 from master (depending on which RTEMS
>>>>>>>> version we have then). All features not on master would be lost
>>>>>>>> when we do that.
>>>>>>>>
>>>>>>>> The current 6-freebsd-12 on the other hand should be considered
>>>>>>>> stable. We won't upgrade it to a 7-freensd-12 or similar. It will
>>>>>>>> be a maintenance branch as soon as we reach the release. We can
>>>>>>>> backport features that have been tested on master if there are
>>>>>>>> users for that. But we don't have to. Just like we don't backport
>>>>>>>> every new feature from RTEMS 6 to 5.
>>>>>>>>
>>>>>>> I view this as equivalent to the proposals to revert commits that are
>>>>>>> on 6-freebsd-12. Doing this means that we are discarding quite a lot
>>>>>>> of software engineering effort that was done on 6-freebsd-12. I would
>>>>>>> rather that we determine a way that can retain that effort. At this
>>>>>>> point, there is probably little way to make everyone happy.
>>>>>>
>>>>>> I think you misunderstood my intention. We definitively should port
>>>>>> the patches from 6-freebsd-12 to master so that they are on a future
>>>>>> libbsd branch. Like said above: I just don't think it's realistic to
>>>>>> base a new developed branch on the 12 one.
>>>>>>
>>>>> I see. This is good to consider.
>>>>>
>>>>>> If I remember correctly, Sebastian's branch that contains his
>>>>>> suggested fixes already ported Chris NFS work to master first and
>>>>>> then added Sebastian's patches that revert the parts where he
>>>>>> disagrees.
>>>>>>
>>>>> I would leave it to them to determine if the NFSv4 is fully supported.
>>>>>
>>>>>> That leaves some driver patches on the 12 branch that are most
>>>>>> likely a lot simpler to port to master.
>>>>>>
>>>>>> My guess would be: Porting the stuff from 6-freebsd-12 to master
>>>>>> should be a few days maximum to a compile clean state. Upgrading
>>>>>> 6-freebsd-12 to something that would follow FreeBSD master again
>>>>>> should be something in the range of weeks.
>>>>>>
>>>>> OK, then we should likely follow the path of porting from 6-freebsd-12
>>>>> to master. I would leave the issue of what to do with FDs as a
>>>>> separate concern that can be addressed once we solve this underlying
>>>>> problem.
>>>>>
>>>>> What would you think is the best way to handle the process though? It
>>>>> would seem to still require some kind of cherry-picking + patching
>>>>> FreeBSD and passing some tests? Perhaps it is best to do it that way,
>>>>> on top of the current master branch, attempting to pick up patches
>>>>> from the shared ancestor. ("Only in 6-freebsd-12").
>>>>
>>>> I would usually just cherry-pick commits that don't import anything
>>>> from FreeBSD. All commits that do import files need a bit more work:
>>>> For these, the files from FreeBSD master have to be re-importet.
>>>>
>>>> It's basically the same process that I use to back-port something from
>>>> master to 6-freebsd-12. And it's why CONTRIBUTING.md tells that you
>>>> should have one commit that only adds the unmodified imported files.
>>>> With that it's easy to re-import the files from a different version
>>>> and re-apply the patches.
>>>>
>>>> Best regards
>>>>
>>>> Christian
>>>
>>> I tried to cherry-pick some of the commits yesterday (hobby time on
>>> Sunday!). The first few patches should work quite well. There are some
>>> minor problems that still have to be cleaned up to make it compile clean.
>>>
>>> One of the early ones that makes problems is the path-mappings feature
>>> from Jan Sommer because on master something changed the same code parts.
>>> We need someone who understands both changes to port that.
>>>
>>> About starting with Chris kernel symbol namespace tool patch
>>> (59f652fe88) the changes start to get bigger and the patches are harder
>>> to apply. I think the kernel symbol namespace is one of the core patches
>>> that would be important to port, but I don't know the code surrounding
>>> it well enough that I can do it fast.
>>>
>>> How should we approach that work? One patch at a time? Maybe together
>>> with pinging the original authors to ask them whether they can port
>>> their patches to master or at least test them there?
>>>
>>> For the VFS and NFS parts, it depends a bit on the solution of the
>>> discussion. Sebastian already worked on a re-implementation of NFS on
>>> master (not yet entirely finished):
>>>
>>>     https://github.com/sebhub/rtems-libbsd/commits/master-update
>>>
>>> Of course that is based on his solution without the file descriptors and
>>> without the central system calls. So it depends on the results of the
>>> discussion between Sebastian and Chris whether that patch can be used to
>>> have the NFS functionality in master.
>>>
>>
>> When re-reading the text I noted that mentioning that branch here can
>> lead into a wrong direction. To make my intention clear: I think we
>> should concentrate on all unrelated patches at the moment. Parallel we
>> should try to push the NFS/VFS discussion and help Chris and Sebastian
>> to agree on one solution that is acceptable for both. Depending on that
>> solution, it could be simpler to just use that branch and maybe port
>> only small parts to have a functionally equivalent solution on both
>> branches with a slightly different history.
>>
> Yes, thanks. This looks like a good analysis. I would definitely
> prefer to get master and 6-freebsd-12 into harmony. Then it is a
> little simpler to discuss the other problems in libbsd.
> 
> Vijay had similar kind of success (and problems) as you did with
> cherry-picking off of 6-freebsd-12. I think he made it a little
> further. I guess one possible route forward would be to begin working
> on this effort and share results, pushing to master when some
> relatively stable milestones are reached.
> 
> Gedare

Hello Gedare,

I made sure to remove every patch that is not compile clean and pushed 
my current working version here:

https://gitlab.com/c-mauderer/rtems-libbsd/-/tree/cm/20230206-port-patches-to-master

At the moment it's cherry-picking patches from 6-freebsd-12 to master as 
long as they apply and compile. I didn't make sure yet that everything 
works or is in a usefull order.

Comparison between this and 6-freebsd-12 is hiere:

https://gist.github.com/620ae08d88dca054c45100a84fdaaee8

Only 35 patches difference left if you ignore the "Update to FreeBSD" 
patches that are branch specific.

A big one will be Chris' 'Implement portable kernel symbol namespace 
tool' and the follow up commits that clean up the kern-symbols. That is 
an important improvement but I don't know yet what is necessary to 
update these patches to master.

Do you think we should create a new mailinglist thread?
Topic: 'libbsd master and 6-freebsd-12 cleanup; help wanted'
Goal: Port all patches from 6-freebsd-12 to master that are not branch 
specific and not related to the currently open questions regarding the 
correct aproach to NFS.
Status: Short summary of what we discussed and what we reached.

Best regards

Christian

> 
>> Best regards
>>
>> Christian
>>
>>> Note that Sebastian also mentioned, that FreeBSD changed a lot in the
>>> NFS/VFS in the last few months. So it's possible that both solutions
>>> need quite some changes after we reach that version of FreeBSD.
>>>
>>> Best regards
>>>
>>> Christian
>>>
>>>>
>>>>>
>>>>>> Best regards
>>>>>>
>>>>>> Christian
>>>>>>
>>>>>>>
>>>>>>> it will be much easier to debate the merits of proposed patches to
>>>>>>> address optimization etc., if we have a shared basis of features and
>>>>>>> development. The way this has been heading is going to lead us
>>>>>>> directly into a non-viable mode of trying to support several open
>>>>>>> development branches of libbsd. I can easily see someone wanting to
>>>>>>> carry forward 6-freebsd-12 commits to rtems7. If we don't have a way
>>>>>>> to do that, we will definitely end up with something like
>>>>>>> 7-freebsd-12.
>>>>>>>
>>>>>>> We have to deal with the fact that there are legitimate developments
>>>>>>> on 6-freebsd-12 that users may expect in future RTEMS releases. I
>>>>>>> cannot agree with having a release of 6-freebsd-12 with rtems6 that is
>>>>>>> followed by a release of 7-freebsd-1x that drops some features such as
>>>>>>> support for NFSv4. It is not pragmatic. But we also don't have the
>>>>>>> resources to be maintaining multiple branches of libbsd.
>>>>>>>
>>>>>>>> For example, I don't know of a user for the BBB framebuffer on
>>>>>>>> RTEMS 6. It's nice that it is there on master. It will become a
>>>>>>>> stable feature on the next major branch (one of 6 or 7-freebsd-13
>>>>>>>> or 14). But there is no necessity to backport it to the mostly
>>>>>>>> stable 6-freebsd-12.
>>>>>>>>
>>>>>>> I have asked Vijay to look at backporting it anyway, to reduce the
>>>>>>> difference between the branches. I want to get us out of this backward
>>>>>>> way of development / change management that has evolved over time in
>>>>>>> libbsd. How best to get us to a state where new development only goes
>>>>>>> to master is for me still the main question. Once we have this fixed,
>>>>>>> it is my intention to thoroughly reject any development that is only
>>>>>>> submitted to a "stable" branch in the future.
>>>>>>>
>>>>>>> Gedare
>>>>>>>
>>>>>>>> Best regards
>>>>>>>>
>>>>>>>> Christian
>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Best regards
>>>>>>>>>>
>>>>>>>>>> Christian
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> In any case, doing this in a way that ensures the commits build
>>>>>>>>>>> and
>>>>>>>>>>> tests run is challenging due to the interactions with the
>>>>>>>>>>> rtems.git,
>>>>>>>>>>> toolchain, and the submodules. >
>>>>>>>>>>> After the 6-freebsd-12 and master are made consistent, then it
>>>>>>>>>>> becomes
>>>>>>>>>>> possible to update freebsd and also to consider what ways can be
>>>>>>>>>>> chosen to fix problems in 6-freebsd-12.
>>>>>>>>>>>
>>>>>>>>>>> -Gedare
>>>>>>>>>>>
>>>>>>>>>>>> Best regards
>>>>>>>>>>>>
>>>>>>>>>>>> Christian
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm fine with either way, as a user I just need clear not
>>>>>>>>>>>>> confusing
>>>>>>>>>>>>> project message...
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>> Karel
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> devel mailing list
>>>>>>>>>>>>> devel at rtems.org
>>>>>>>>>>>>> http://lists.rtems.org/mailman/listinfo/devel
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> --------------------------------------------
>>>>>>>>>>>> embedded brains GmbH
>>>>>>>>>>>> Herr Christian MAUDERER
>>>>>>>>>>>> Dornierstr. 4
>>>>>>>>>>>> 82178 Puchheim
>>>>>>>>>>>> Germany
>>>>>>>>>>>> email:  christian.mauderer at embedded-brains.de
>>>>>>>>>>>> phone:  +49-89-18 94 741 - 18
>>>>>>>>>>>> mobile: +49-176-152 206 08
>>>>>>>>>>>>
>>>>>>>>>>>> Registergericht: Amtsgericht München
>>>>>>>>>>>> Registernummer: HRB 157899
>>>>>>>>>>>> Vertretungsberechtigte Geschäftsführer: Peter Rasmussen,
>>>>>>>>>>>> Thomas Dörfler
>>>>>>>>>>>> Unsere Datenschutzerklärung finden Sie hier:
>>>>>>>>>>>> https://embedded-brains.de/datenschutzerklaerung/
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> devel mailing list
>>>>>>>>>>>> devel at rtems.org
>>>>>>>>>>>> http://lists.rtems.org/mailman/listinfo/devel
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> devel mailing list
>>>>>>>>>>> devel at rtems.org
>>>>>>>>>>> http://lists.rtems.org/mailman/listinfo/devel
>>>>>>>>>
>>>>>>>
>>>>>
>>>
>>
>> --
>> --------------------------------------------
>> embedded brains GmbH
>> Herr Christian MAUDERER
>> Dornierstr. 4
>> 82178 Puchheim
>> Germany
>> email:  christian.mauderer at embedded-brains.de
>> phone:  +49-89-18 94 741 - 18
>> mobile: +49-176-152 206 08
>>
>> Registergericht: Amtsgericht München
>> Registernummer: HRB 157899
>> Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler
>> Unsere Datenschutzerklärung finden Sie hier:
>> https://embedded-brains.de/datenschutzerklaerung/

-- 
--------------------------------------------
embedded brains GmbH
Herr Christian MAUDERER
Dornierstr. 4
82178 Puchheim
Germany
email:  christian.mauderer at embedded-brains.de
phone:  +49-89-18 94 741 - 18
mobile: +49-176-152 206 08

Registergericht: Amtsgericht München
Registernummer: HRB 157899
Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler
Unsere Datenschutzerklärung finden Sie hier:
https://embedded-brains.de/datenschutzerklaerung/