Waf migration proposal.

[Amar Takhar]

The following is a proposal to lay out the beginnings of a plan to move from the 
current auto* build system to the new waf build located at 
http://git.rtems.org/amar/waf.git.

This is meant as an initial overview open for discussion.  The entire project 
will be fleshed out on the wiki and using milestones on the roadmap populated 
with tickets for each task that needs to be completed.  This will help give 
everyone an idea of progress and a place to collect any issues that may arise.


* 1 Initial commit
  ~~~~~~~~~~~~~~~~
  1.1 move headers
  1.2 introduce waf
  1.3 apply all include fixes (from 'fix' branch)
  1.4 remove documentation

* 2 Tests
  ~~~~~~~
  2.1 migrate tests to waf

* 3 Source Move
  ~~~~~~~~~~~~~
  3.1 new source layout

* 4 Buildbot
  ~~~~~~~~~~
  4.1 Introduce automated building using buildbot

* 5 Tools
  ~~~~~~~
  5.1 build all tools, run all test suites to ensure RTEMS builds as 
      expected.

* 6 Verification
  ~~~~~~~~~~~~~~
  6.1 Final version containing all testing required in order to verify the 
      build.
  6.2 create new rtems-docs repository with new reST + Sphinx documentation 
      containing data from docs/ (This is already complete)


It was brought up previously the above may be too much at once.  I've had a lot 
of time to think about it and I believe there is no other way to do it for the 
following reasons:

 * The current build is monolithic in nature.  It literally touches everything 
   if you only want to build 1 BSP.
 * There is no way to verify old+new other than hashing the important parts of 
   the resulting object files.  Using this method it does not matter how much we 
   change at once.
 * The incremental changes, in their smallest units are so large that each one 
   is a radical change that requires a lot of testing.  If we do each one then 
   test or do it all and test we garner no benefit doing it in between steps 
   other than 5 times the work.  This is due to our test method: object 
   checking and testsuite.
 * Trying to make changes using the current build system is worse as the auto* 
   build provides no way to verify itself whereas waf does.
 * waf offers us the opportunity to build RTEMS in an extremely concise manner.  
   Once we verify it we know it will build that way on every system or fail 
   during the build.

While moving forward with this we will still have 4.11,4.12 or however many 
iterations of the 4.x branch until the waf build can be called stable.  While 
there are concerns about RTEMS being seen as unstable this can be mitigated or 
completely removed by detailing our intentions and not cutting any releases of 
the 'master' branch.

I will be writing a tool that will map old source paths to new ones.  You could 
for example go:

# ./find.py <file>
  - get a list of files that match that file in their new location
# ./find <path/to/file>
  - same a above just a more exact search
# ./find.py <hash>
  - locate a file based on old file hash

This will be useful to allow users to make their way around the new source.  It
will also offer an easy way to port patches to the new branch from 4.x.

Also, no code freeze will be necessary.  None of the source file contents will
change only their names.  It should be possible to use the above tool to write
something that will automatically convert patches if someone want to do this.
Otherwise as long as you point it to the new file all patches will apply
cleanly.

The expected timeline from milestones 1-5 inclusive is 1 month.

To be clear: this is a proposal where all development moves to the 4.x branches 
until this project is complete.  Once done commits into the 4.x branches can be 
merged into master one by one where they will go through review and testing via 
Buildbot.  Porting should in most cases be trivial as pointing to the new file 
location.  Any build changes will be trivial to port to waf from auto*.

Any and all comments welcome feel free to ask for clarification on any points!


Amar.