User manual and new build system

[Sebastian Huber]

On 27/11/2019 01:45, Joel Sherrill wrote:
> 
> 
> On Sun, Nov 24, 2019 at 4:44 PM Chris Johns <chrisj at rtems.org 
> <mailto:chrisj at rtems.org>> wrote:
> 
>     On 23/11/19 1:31 am, Sebastian Huber wrote:
>      > On 22/11/2019 15:25, Joel Sherrill wrote:
>      >> On Fri, Nov 22, 2019, 8:19 AM Sebastian Huber
>      >> <sebastian.huber at embedded-brains.de
>     <mailto:sebastian.huber at embedded-brains.de>
>      >> <mailto:sebastian.huber at embedded-brains.de
>     <mailto:sebastian.huber at embedded-brains.de>>> wrote:
>      >>     I am about to add the documentation of the new build system
>     to the user
>      >>     manual. There is no longer a bootstrap of the sources
>     necessary. Should
>      >>     I remove all information for the bootstrapping from the user
>     manual?
>      >>     Should there be a chapter for the old build system?
>      >>
>      >> Don't remove the old process until the code is removed. I
>     thought we had
>      >> agreed to a final autoconf based release. A release with both
>     will help ensure
>      >> it is functionally equivalent and provide a baseline for comparison.
>      >
>      > I would remove the old build system as soon as possible. Shipping
>     a release with
>      > two build systems makes maintenance more difficult. Getting rid of
>      > Automake/Autoconf would remove these packages from the RSB build
>     for example.
> 
>     After a some consideration over the weekend I am not sure what we
>     should do and
>     I think we need to discuss it some more.
> 
>     We need to answer the important question of user churn. A change of
>     this type
>     will have a flow on effect to established users and any
>     infrastructure they have
>     to build and deploy RTEMS. An example is the `5/bsps` build sets in
>     the RTEMS, a
>     build system change breaks those configurations. I am sure there are
>     many other
>     scripts in use today. I also know there are sites running 4.11 and 5
>     in parallel
>     and building each will break.
> 
>     We need to decide if this churn to our users is something we want now.
> 
>     I currently do not favor a release with both build systems being
>     present.
> 
> 
> My big concern is ensuring that we know the new build system works 
> exactly like
> the old one. And by works, I mean produces the same binary code with the 
> same
> configure options expressed in the new build system's manner.

Yes, I think we need a check list which everyone interested in a 
particular BSP should use to determine if the new build system produces 
sane results.

> Here are some
> things we need to consider. These are of varying complexity and type.
> 
> (1)  Users historically test "dot zero" releases and the "dot one" 
> incorporates
> a lot of fixes that we really wish had been in the "dot zero".  This 
> raises the
> question of when do we get enough community testing to have confidence
> the new system has no breakages from the old one?

Breakages can occur due to build system changes, code changes, new 
compiler versions, etc. I think this is a general problem and not 
specific to the build system change. If no testing is done before a 
release, then we should accept this and ship a 5.2 release as the "real" 
one.

> 
> (2) rtems-bsp-builder will need to change to support the new build system.
> I think this is a must have to ensure a certain level of no breakages.
> 
> (2a)  Related: Do we know the new build system builds all the configurations
> rtems-bsp-builder performs?

It would be nice if this could be done by someone knowing the 
rtems-bsp-builder code. I had a quick look into it and it is not 
something I can do in couple of minutes.

I tested some build variations with a simple script.

> 
> (3) BSPs have a number of variables which can be overridden. How will we
> know these still are supported and work?

The BSP options were converted by a Python script:

https://git.rtems.org/sebh/rtems.git/tree/gen-bsps.py?h=build

It is quite likely that I converted all options. This should be one of 
the check list items.

> 
> (4)  an unknown number of instructions will have to change. [This may be
> time to purge/convert/fix more wiki content. This is not necessarily a bad
> thing -- just a stick to beat us to do something.]

Yes, but I cannot change this over night. My general long term approach 
is to move content to the Sphinx documentation and remove wiki content 
and README files.

> 
> (5) Developer disruption. I am hoping this doesn't have the negative impact
> switching from CVS to git did. But it could. It isn't that the switch 
> over itself
> is bad. It's just that we can forget all the side-effects and human 
> retraining.

I don't think that changing a configure command line to the new 
config.ini file will be a very hard job.

> 
> My overarching concern with all of these is "risk reduction". The primary
> risk is breaking something. And the risk reduction is (1) testing before 
> switch
> over and (2) having a reference after switch over for behavior?
> 
> So what's the criteria for switching over? I have no idea what the 
> acceptance
> criteria is. Can we define that?

For the integration my criteria would be:

1. The new build system builds all tests (maybe except Ada).

2. You can build libbsd with a couple of BSPs.

3. The rtems-examples work using the legacy Makefiles (seems to work).

4. Test runs on a couple of BSPs show the same pass/failures compared to 
the old build system.

> 
> What can that reference be? I think this is from heaviest to lightest.
> 
> + Build systems in parallel on same branch for N time?
> 
> + final release with old build system and then kill it.
> 
> + final autoconf snapshot known to work -- implies testing the snapshot 
> which
> we don't do right now..
> 
> + (lightest) tag RSB, tools, rtems, and libbsd repos with a 
> "pre-autoconf-death"
> tag. This at least lets someone easily go back to that point in time 
> across the repos.
> You can't do this easily with git log because you would have to find the 
> right place
> in multiple repositories.
> 
> I probably can be convinced that the lightest option is acceptable. Whatever
> the reference point is, it just needs to be documented what it is and how to
> recreate it.

I would merge the new build system in one patch into the RTEMS sources. 
Then I would add a configuration option to the old configure script 
(e.g. "--I-only-want-to-compare-results-with-the-new-build-system"). 
This basically disables the normal use. It intentionally breaks existing 
configuration scripts to that you know something happens. You can still 
build with the old build system. You should do this only the compare the 
results of the old and new build system. The new build system should be 
used, fixed and improved. In a three month (maybe more?) period we keep 
the old build system in the sources and documentation. Afterwards we 
remove it completely.

Before the stuff is removed, we tag the repositories.

> 
> 
>     If we decide it is OK to have the churn we can proceed as we
>     currently are. The
>     build changes can be merged and the autotools build system can be
>     removed.
> 
>     If we decide we would like to make RTEMS 5 the last autotools
>     release we need to
>     make the release and that would mean a focused effort by more than
>     just me. We
>     cannot leave all the hard work done by Sebastian for a new build
>     system on hold.
> 
> I agree on not letting the new build system hold.

Good.

> 
> 
>     Either path effects someone, a developer, for example Sebastian, our
>     users with
>     churn or me and I hope other core developers with a release.
> 
> 
> Churn will occur at some point. My concern is risk.

Yes, at some point all the configuration scripts to build a BSP need a 
change.

> 
> 
>     A release means getting the release itself debugged then we need to
>     test it. It
>     also means there may be possible changes and we need to be mindful
>     of the effect
>     that might have on the work Sebastian has already done.
> 
> 
> And knowing that we don't have good infrastructure to test those 
> snapshots right now,
> I would lean to having tags in the repos. We have good testing on the repos.
> 
> Perhaps after the build system switch, we can focus on testing the 
> snapshots.
> 
> We only have a finite amount of volunteer time to make this transition. 
> Where is
> the effort best spent?
> 
> My gut hunch is on updating the testing infrastructure and documentation 
> after
> the switch to get back to full speed as quickly as possible.
> 
> 
>     I look forward to hearing what our developers and community think.
> 
> 
> Me too.
> 
> I'm not trying to put extra burden, work or churn on anyone. I'm just 
> concerned
> that this build system transition has inherent risks. We need to do what 
> we can
> to reduce risk before the switch, provide a known (good?) reference point,
> and make sure we have a complete work list. And commitment to finish that
> work list.
> 
> And we all have holidays. etc coming up.

 From my point of view it would be nice to have at least the patches 
integrated in the next one or two weeks.

-- 
Sebastian Huber, embedded brains GmbH

Address : Dornierstr. 4, D-82178 Puchheim, Germany
Phone   : +49 89 189 47 41-16
Fax     : +49 89 189 47 41-09
E-Mail  : sebastian.huber at embedded-brains.de
PGP     : Public key available on request.

Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG.