Build failures w/RSB

[Chris Johns]

On 23/03/2015 6:50 am, Doug Hawkins wrote:
> Hi,
>
> I put my questions at the bottom.  Intervening is "reference" material.
>
> I recently read about RTEMS and it seems like a great addition to my
> project (on a TI LaunchPad), but I'm having trouble building it.  I'm
> running Linux Mint Debian Edition 64bit.
>
> I cloned the git repository for RTEMS and tried to configure it (but
> automake was a newer version, so there were many warnings). Then I
> noticed the comments about RTEMS Source Builder and thought that might
> be a useful way to proceed, since it integrates many of the other
> packages I'm using (newlib, gdb, etc.).
>
> My first question that I could not find in the documentation
> (https://ftp.rtems.org/pub/rtems/people/chrisj/source-builder/source-builder.html):
> How can I take advantage of the hundreds of MB's I've already
> downloaded?  Too late for me, but an answer might help someone else.

The documentation is now located at https://docs.rtems.org/rsb. I need 
to add a redirect.

>
> I ran sb-check and everything was fine.  Then gritted my teeth and ran
> sb-set-builder and re-downloaded everything. But it failed at an
> "#include <zlib.h>" while compiling gcc-4.9.2: lto_compress.c... I found
> the header in the gcc tree (should have been <zlib/zlib.h> and re-ran
> "sb-set-builder".

The package in question does not check for the header during an early 
configure stage so the build proceeds for a while before it fails. The 
RSB is only wrapping the build process.

>
> Since everything seems to be reset, my <zlib.h> code change was cast
> out, so the failure occurred again. I finally decided to try installing
> "zlib1g-dev" package on my system -- that got me through this pickle,
> though I thought it should look for the local file.
>

What host operating system ?

> I also tried the --no-clean and --no-download options, to see if that
> would speed up this re-cycling.

A bug was fixed with these options on the weekend. The 
--with-*/--without-* option parser ate an extra option and that broke 
the builds.

> But then I learned that not everything
> was already downloaded, so the next failure occurred when the gdb source
> package wasn't found.  Each "build" cycle costs me 25 minutes on my
> four-core i7 machine.

If you hit the --with-download bug then this could happen.

> Re-starting without the --no-download & --no-clean options (I've given
> up on "optimizing" this process) -- next failure is telling me that
> python2.7 doesn't exist (it is installed on my system).  I've spent the
> last couple hours searching through configure code & run logs to try to
> understand what part of python2.7 it needs.
>
> I did find the gdb configure call just before the failure, but when I
> re-ran it, it completed successfully, duplicating the log messages up
> to, but except the last 40 lines or so.  The start of that sequence
> (which seemed too generic to locate) was:
>
> checking whether NLS is requested... no
> checking whether makeinfo --split-size=5000000 supports @click... yes
> checking for default auto-load directory... $debugdir:$datadir/auto-load
> checking for default auto-load safe-path... $debugdir:$datadir/auto-load
>
> I've finally given up!

Which host ? What architecture ?

>  Here are the remaining questions running through
> my mind.
>
> 1) Is there any way I can optimize the compile process (not rebuild
> everything when a failure occurs)?  Specifically: how do I "resume"
> sb-set-builder?  It appears that I have to check all repositories &
> rebuild everything each time I run it.

Sorry there is nothing currently that can do this, and my time is 
limited. It is a repeated question and worthy of a full answer. If there 
is something you can add to the documentation in the doc directory of 
the RSB for your host that helps please provide a patch.

The complexity of adding this is the side effects it beings. The main 
focus of the RSB is to provide a user with a tool set that consistent 
and repeated. Add such a feature would require I make sure there are no 
side effects of the build process which could result in a partial or 
mixed build that may appear ok but is not. The feature would require a 
hash of the contents of each stage would need to be made and verified. 
This means you cannot cheat and alter a completed build stage and doing 
so would restart that stage. At the moment the RSB makes no assumptions 
about files or data in the build tree and a restart feature would 
require adding 'cached' data and state information and my experiences 
with 'cached' data and state is the complexity rises.

Looking at the timely builds such as a tool set the autotools is fast, 
and binutils is fast, so it is basically gcc that is taking the time. I 
do not have the figures in front of me but lets say is it 75% of the 
total time and it is also the main reason a build fails, is adding the 
feature and the complexity worth while. From where I sit having to 
implement it I can not as sure given there are other things that RTEMS 
could have before this.

The other approach is to teach the RSB to look for libraries. The RSB 
has no integration with host package management system and I think this 
is a good thing. This means we would need to look for a different 
approach to solving this problem. An idea I have been considering is to 
create a series of waf scripts that check for various libraries. Each 
package in a build set can define a list of checks to be made and the 
RSB collects these when the configuration files are read and then 
executables the tests before running. This would expose any system 
libraries or tools not present early. The only restriction is not adding 
a dependency to packages built before hand.

Finally, this RSB has been having a lot of churn recently as a range of 
changes for the 4.11 have been applied. My hope is this will slow down 
once we have gdb-7.9 sorted, and the recent rtems-tools waf issue has 
been resolved.

>
> 2) When a failure occurs, how can I locate the "real" problem and find
> information on how to resolve it?  It seems that the source-builder is
> too complex & doesn't have enough documentation for me to dig deep
> enough into the sorts of problems I'm encountering.

If you have suggestions for the documentation in the 'doc' directory 
please make them.

> On other
> "meta-builders" I've worked with, there is usually a large header pushed
> into the log file for each execution segment, with all the necessary
> options to re-run that segment manually, but I couldn't find anything
> like that in the log file.

There was no .log file in the directory you ran the command in ?

If you get an issue it is best to post the rsb-report-* file. Each 
package can have different configure and build systems and each results 
in a different way to find errors. For autoconf type configure you need 
to find the config.log file and then know how to find the error in it.

>
> I had a look through some of set-builder's python code, but I'd need
> more documentation there to understand some of the structures (I was
> hoping to redirect the sources to my already existing repositories).
>
> My log file is over 90MB now, the error log files are each about 280kB.
> I could compress them and send, if needed, but won't do that until
> requested.

So you found the .log file and the error reports files ?

>
> Any help that can be offered would be greatly appreciated.
>

I hope this helps.

Chris