libbsd development policy clarification needed?

Gedare Bloom gedare at rtems.org
Sat Feb 4 00:25:23 UTC 2023 

On Fri, Feb 3, 2023 at 3:29 PM Christian Mauderer <oss at c-mauderer.de> wrote:
>
>
>
> Am 3. Februar 2023 22:52:48 MEZ schrieb Gedare Bloom <gedare at rtems.org>:
> >On Fri, Feb 3, 2023 at 2:39 PM Christian Mauderer <oss at c-mauderer.de> wrote:
> >>
> >>
> >>
> >> Am 3. Februar 2023 22:12:06 MEZ schrieb Gedare Bloom <gedare at rtems.org>:
> >> >On Fri, Feb 3, 2023 at 12:52 PM <oss at c-mauderer.de> wrote:
> >> >>
> >> >> Hello Gedare,
> >> >>
> >> >> Am 03.02.23 um 19:51 schrieb Gedare Bloom:
> >> >> > On Thu, Feb 2, 2023 at 11:24 PM Christian MAUDERER
> >> >> > <christian.mauderer at embedded-brains.de> wrote:
> >> >> >>
> >> >> >> Hello Karel,
> >> >> >>
> >> >> >> On 2023-02-02 12:43, Karel Gardas wrote:
> >> >> >>>
> >> >> >>>     Guys,
> >> >> >>>
> >> >> >>> recently I needed to work with RTEMS/NFS. As this is provided by libbsd
> >> >> >>> I took this and following two sentences below from master branch
> >> >> >>> description provided in README I took as granted that master does have
> >> >> >>> all the features which are currently available and provided by the project:
> >> >> >>>
> >> >> >>> "This branch must be used for libbsd development. Back ports to the
> >> >> >>> 6-freebsd-12 are allowed."
> >> >> >>>
> >> >> >>> I was surprised to be proven wrong then by Fabrizio here:
> >> >> >>> https://devel.rtems.org/ticket/4723
> >> >> >>>
> >> >> >>> and by later investigation which shows that 6-freebsd-12 branch
> >> >> >>> accumulated NFS work by Chris done in 2021 which is not presented on
> >> >> >>> master. I've investigated just NFS as this was my focus here.
> >> >> >>>
> >> >> >>> So if 6-freebsd-12 became development branch of some sort, then it would
> >> >> >>> be great to have that clarified in the project README file to prevent
> >> >> >>> users confusion? Or if the policy is still the same, then perhaps some
> >> >> >>> branch sync is needed here?
> >> >> >>
> >> >> >> That currently is an open issue. Basically there is a pending patch set
> >> >> >> that should fix that since several months. But there is a disagreement
> >> >> >> about some of the changes in that patch set (and about the patches
> >> >> >> checked in to 6-freebsd-12). Therefore, it still hasn't been merged.
> >> >> >>
> >> >> >> If you want to know some more about the problematic points, I recommend
> >> >> >> reading this (long) thread:
> >> >> >>
> >> >> >> https://lists.rtems.org/pipermail/devel/2023-January/074164.html
> >> >> >>
> >> >> >> The statement that development has to happen on the master branch is
> >> >> >> still true. The master is intended to track the FreeBSD upstream
> >> >> >> development. Only changes on that branch are guaranteed to live through
> >> >> >> an upgrade to a newer base version of FreeBSD. It's very unfortunate,
> >> >> >> that there are some patches on the 6-freebsd-12 branch only. On the long
> >> >> >> term, that issue has to be resolved.
> >> >> >>
> >> >> >
> >> >> > I have been investigating this problem in the background, and I have
> >> >> > some findings and some questions. First, I have found that there is a
> >> >> > most-common ancestor between master and 6-freebsd-12 at commit
> >> >> > https://git.rtems.org/rtems-libbsd/commit/?h=6-freebsd-12&id=2ce13cf6dc73855f28bc7edbbc64dc4b482a4976
> >> >> > This is at least promising that the discrepancy between the branches
> >> >> > can be resolved.
> >> >> >
> >> >> > The proposed pending patch set to "fix" the NFS issue does not fix the
> >> >> > underlying problem. Instead, it introduces further divergence between
> >> >> > the branches. I would instead suggest that we should resolve to fix
> >> >> > the underlying problem. I can see two paths forward.
> >> >> >
> >> >> > 1. Abandon 6-freebsd-12 after releasing 6. This is probably not ideal
> >> >> > since what I understand is some users have projects based on
> >> >> > 6-freebsd-12 and would like an upgrade path. (I guess there is also
> >> >> > the option to abandon master, which also makes little sense.)
> >> >>
> >> >> A variant for this would be to introduce a 6-freebsd-13 that is based on
> >> >> the master branch as soon as we have one. That would allow a longer
> >> >> maintenance because FreeBSD 12 reaches it's EoL December 2023.
> >> >>
> >> >> >
> >> >> > 2. Pull commits from 6-freebsd-12 into master to make sure master is
> >> >> > the development head. in the future, reject patches that only go
> >> >> > toward release branches. This has its own problems too. It can
> >> >> > realistically only be done in three ways:
> >> >>
> >> >> Please note that Sebastian mentioned that the file descriptors broke the
> >> >> NTP support (at least I think it was NTP; possible that it was another
> >> >> submodule). So picking the current version of the patches into the
> >> >> master without adding fixes makes the master unusable for some cases.
> >> >>
> >> >Fixing the problems after making the branches the same will be better
> >> >for the long-term, if we can find a path to do it.
> >> >
> >> >> >
> >> >> > 2a: Rebase master and cherry-pick commits from 6-freebsd-12 and master
> >> >> > back into master. This rewrites the history of master, and
> >> >> > unfortunately will cause the head of 5-freebsd-12 and the tags for
> >> >> > rtems-5 to no longer exist on the master branch. They will still exist
> >> >> > in the '5' branch. The advantage is in the end there will be a linear
> >> >> > history of development on master that reflects the timeline of actual
> >> >> > development that spanned both branches. Theoretically, this should
> >> >> > make it easier to git-bisect.
> >> >> >
> >> >> > 2b: Cherry-pick commits from 6-freebsd-12 to master and fix conflicts.
> >> >> > This puts all the missing commits from 6-freebsd-12 on to the current
> >> >> > head of master. I don't know how messy this would be. It ends up
> >> >> > making the history of master convoluted to understand, with fairly old
> >> >> > commits from 2018 being placed on top of newer commits from 2020s.
> >> >> >
> >> >> > 2c: Merge 6-freebsd-12 into master and fixup conflicts in the merge
> >> >> > commit. This is pretty similar to 2a but ends up with a non-linear
> >> >> > history and a merge commit. It may be a fairly complex merge commit.
> >> >>
> >> >> For all of the 2x solutions: The commits from 6-freebsd-12 can't just be
> >> >> cherry-picked. You have to re-import the NFS files from the FreeBSD
> >> >> master version that is used as base for the current libbsd master.
> >> >> Otherwise we mix different FreeBSD source versions. We had that some
> >> >> time back in libbsd and Sebastian needed a lot of time cleaning that up.
> >> >>
> >> >Understood.
> >> >
> >> >> >
> >> >> > To get a sense of the difference between the two branches, I have done
> >> >> > the following command:
> >> >> > $ git log --pretty=oneline master...6-freebsd-12 > ../log.txt
> >> >> > This uses the ... (three-dot) Symmetric Difference Notation. The
> >> >> > result of that is a 750 line file, so 750 commits are different
> >> >> > between the two branches. Some of those commits are actually the same
> >> >> > content, but they have different parents so different hashes. In a
> >> >> > rebase or merge situation, those commits should end up the same. There
> >> >> > may be other git-fu to find just the patches that are unique in the
> >> >> > two branches.
> >> >>
> >> >> 750 commits are a bit too much. The order of patches on master and
> >> >> 6-freebsd-12 isn't always the same. I wrote a small python script to
> >> >> find the differences somewhere in 2021 when I needed the differences in
> >> >> a discussion with (I think) Joel. It compares based on author and
> >> >> subject which gives a quite good estimate for libbsd. You can find the
> >> >> script here:
> >> >>
> >> >>    https://gist.github.com/a82e21eb250cb96c3a36f107b92dab09
> >> >>
> >> >> That shows 138 different commits. 86 are only on 6-freebsd-12. Complete
> >> >> output is here:
> >> >>
> >> >>    https://gist.github.com/93a2b19a5bf4cd8a6263ae29aa359ac2
> >> >>
> >> >>  From these you can ignore the "Update to FreeBSD..." commits and some
> >> >> of the cleanup patches. I assume that quite some of them can be
> >> >> cherry-picked with only small or no changes. For example patches that
> >> >> only add or update drivers in rtemsbsd should work without problems. So
> >> >> I think a more realistic number of problematic patches should be 30 to 50.
> >> >>
> >> >Oh, thanks, this is nice. It may be a reasonable place to start by
> >> >looking at what is "Only on master" and to consider back-porting to
> >> >6-freebsd-12. That may reduce the delta somewhat in a way that makes
> >> >sense.
> >>
> >> We really should do the other direction. Not all features oft master have to be on the stable branch. But all features of the stable one should be on master.
> >>
> >I agree, but since there are many fewer changes only on master, if we
> >can merge those into 6-freebsd-12, then it becomes simpler to just
> >call 6-freebsd-12 as the new master. We can replace the current master
> >with this branch, and then update freebsd versions. I think this can
> >be a lower burden process to follow, except it requires acceptance of
> >the commits that are on 6-freebsd-12, which is a different matter.
>
> You can't just make 6-freebsd-12 a new master. It tracks the FreeBSD 12 branch.
>
> Every block of updates to a newer FreeBSD base revision was several days of work for Sebastian. There are a few on master.
>
> An upgrade from a branch tracking FreeBSD 12 to one tracking FreeBSD 13 is most likely more something in the several week range of work.
>
> >
> >> master tracks the FreeBSD master. As soon as it reaches the point where FreeBSD branched off version 13, we can create a 6-freebsd-13 or 7-freebsd-13 from master (depending on which RTEMS version we have then). All features not on master would be lost when we do that.
> >>
> >> The current 6-freebsd-12 on the other hand should be considered stable. We won't upgrade it to a 7-freensd-12 or similar. It will be a maintenance branch as soon as we reach the release. We can backport features that have been tested on master if there are users for that. But we don't have to. Just like we don't backport every new feature from RTEMS 6 to 5.
> >>
> >I view this as equivalent to the proposals to revert commits that are
> >on 6-freebsd-12. Doing this means that we are discarding quite a lot
> >of software engineering effort that was done on 6-freebsd-12. I would
> >rather that we determine a way that can retain that effort. At this
> >point, there is probably little way to make everyone happy.
>
> I think you misunderstood my intention. We definitively should port the patches from 6-freebsd-12 to master so that they are on a future libbsd branch. Like said above: I just don't think it's realistic to base a new developed branch on the 12 one.
>
I see. This is good to consider.

> If I remember correctly, Sebastian's branch that contains his suggested fixes already ported Chris NFS work to master first and then added Sebastian's patches that revert the parts where he disagrees.
>
I would leave it to them to determine if the NFSv4 is fully supported.

> That leaves some driver patches on the 12 branch that are most likely a lot simpler to port to master.
>
> My guess would be: Porting the stuff from 6-freebsd-12 to master should be a few days maximum to a compile clean state. Upgrading 6-freebsd-12 to something that would follow FreeBSD master again should be something in the range of weeks.
>
OK, then we should likely follow the path of porting from 6-freebsd-12
to master. I would leave the issue of what to do with FDs as a
separate concern that can be addressed once we solve this underlying
problem.

What would you think is the best way to handle the process though? It
would seem to still require some kind of cherry-picking + patching
FreeBSD and passing some tests? Perhaps it is best to do it that way,
on top of the current master branch, attempting to pick up patches
from the shared ancestor. ("Only in 6-freebsd-12").

> Best regards
>
> Christian
>
> >
> >it will be much easier to debate the merits of proposed patches to
> >address optimization etc., if we have a shared basis of features and
> >development. The way this has been heading is going to lead us
> >directly into a non-viable mode of trying to support several open
> >development branches of libbsd. I can easily see someone wanting to
> >carry forward 6-freebsd-12 commits to rtems7. If we don't have a way
> >to do that, we will definitely end up with something like
> >7-freebsd-12.
> >
> >We have to deal with the fact that there are legitimate developments
> >on 6-freebsd-12 that users may expect in future RTEMS releases. I
> >cannot agree with having a release of 6-freebsd-12 with rtems6 that is
> >followed by a release of 7-freebsd-1x that drops some features such as
> >support for NFSv4. It is not pragmatic. But we also don't have the
> >resources to be maintaining multiple branches of libbsd.
> >
> >> For example, I don't know of a user for the BBB framebuffer on RTEMS 6. It's nice that it is there on master. It will become a stable feature on the next major branch (one of 6 or 7-freebsd-13 or 14). But there is no necessity to backport it to the mostly stable 6-freebsd-12.
> >>
> >I have asked Vijay to look at backporting it anyway, to reduce the
> >difference between the branches. I want to get us out of this backward
> >way of development / change management that has evolved over time in
> >libbsd. How best to get us to a state where new development only goes
> >to master is for me still the main question. Once we have this fixed,
> >it is my intention to thoroughly reject any development that is only
> >submitted to a "stable" branch in the future.
> >
> >Gedare
> >
> >> Best regards
> >>
> >> Christian
> >>
> >> >
> >> >> Best regards
> >> >>
> >> >> Christian
> >> >>
> >> >> >
> >> >> > In any case, doing this in a way that ensures the commits build and
> >> >> > tests run is challenging due to the interactions with the rtems.git,
> >> >> > toolchain, and the submodules. >
> >> >> > After the 6-freebsd-12 and master are made consistent, then it becomes
> >> >> > possible to update freebsd and also to consider what ways can be
> >> >> > chosen to fix problems in 6-freebsd-12.
> >> >> >
> >> >> > -Gedare
> >> >> >
> >> >> >> Best regards
> >> >> >>
> >> >> >> Christian
> >> >> >>
> >> >> >>>
> >> >> >>> I'm fine with either way, as a user I just need clear not confusing
> >> >> >>> project message...
> >> >> >>>
> >> >> >>> Thanks!
> >> >> >>> Karel
> >> >> >>>
> >> >> >>>
> >> >> >>> _______________________________________________
> >> >> >>> devel mailing list
> >> >> >>> devel at rtems.org
> >> >> >>> http://lists.rtems.org/mailman/listinfo/devel
> >> >> >>
> >> >> >> --
> >> >> >> --------------------------------------------
> >> >> >> embedded brains GmbH
> >> >> >> Herr Christian MAUDERER
> >> >> >> Dornierstr. 4
> >> >> >> 82178 Puchheim
> >> >> >> Germany
> >> >> >> email:  christian.mauderer at embedded-brains.de
> >> >> >> phone:  +49-89-18 94 741 - 18
> >> >> >> mobile: +49-176-152 206 08
> >> >> >>
> >> >> >> Registergericht: Amtsgericht München
> >> >> >> Registernummer: HRB 157899
> >> >> >> Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler
> >> >> >> Unsere Datenschutzerklärung finden Sie hier:
> >> >> >> https://embedded-brains.de/datenschutzerklaerung/
> >> >> >> _______________________________________________
> >> >> >> devel mailing list
> >> >> >> devel at rtems.org
> >> >> >> http://lists.rtems.org/mailman/listinfo/devel
> >> >> > _______________________________________________
> >> >> > devel mailing list
> >> >> > devel at rtems.org
> >> >> > http://lists.rtems.org/mailman/listinfo/devel
> >> >
> >

More information about the devel mailing list