How to build librtemscpu.a and librtemsbsp.a in the future?

[Chris Johns]

On 26/03/2018 17:32, Sebastian Huber wrote:
> On 25/03/18 05:20, Chris Johns wrote:
>> On 24/3/18 1:27 am, Sebastian Huber wrote:
>>> Hello,
>>>
>>> I tried to consolidate the cpukit Makefile.am. This ended up in the following
>>> problem. The content of librtemscpu.a depends on the CPU port. This is currently
>>> done via a non-standard
>>>
>>> _SUBDIRS = ../@RTEMS_CPU@
>>>
>>> construct.
>> Where is this? I cannot find it.
> 
> https://git.rtems.org/rtems/tree/cpukit/score/cpu/Makefile.am#n1
> 

I am confused, the link's version does not have '../'?

>>
>>> The wrapup/Makefile.am combines several static libraries into one
>>> librtemscpu.a. With librtemsbsp.am we have a similar problem. This archive
>>> unpack and re-create step seems to be a bit expensive.
>> Yes this is an awkward work around. I seem to remember this is borrowed from
>> automake and adapted for us. No matter where is comes from it is a hack and a
>> pain to maintain. With nested makes we do not have access to the list of objects
>> so we need to extract, capture and create to merge.
>>
>>> Why can't we use an MRI script, e.g.
>>>
>>> create ./sparc-rtems5/erc32/lib/librtemscpu.a
>>> addlib ./sparc-rtems5/c/erc32/cpukit/librtemscpu.a
>>> addlib ./sparc-rtems5/c/erc32/cpukit/score/cpu/sparc/libscorecpu.a
>>> save
>>> end
>>>
>>> ?
>> If this was supported by the standard GNU script I would be more receptive but
>> even then I feel it is papering over build system weaknesses.
> 
> I just searched for "merge static libraries gnu" and found this. It seems to work.
> 

I am sure it does work, however it is an old standard created when MRI was
active in the embedded market and users wanted to migrate existing projects to
GNU. I would have expected most projects would have migrated off that format ...
unless they still use MRI tools or they also have a broken build system :)

>> Is the MRI format
>> still alive in any MRI or Mentor products?
> 
> I don't know.
> 

Same.

>>
>>> Would conditional Makefile.am parts be more efficient, e.g. in
>>> cpukit/Makefile.am
>>>
>>> if CPU_ARM
>>> include score/cpu/arm/make.am
>>> endif
>>> if CPU_XXX
>>> ...
>> I think this would be a major piece of work. You would need to make sure any
>> globals at the make level do not step on each other.
>>
>>> How would waf deal with this problem?
>>>
>> Efficiently. See:
>>
>>   https://git.rtems.org/rtems-libbsd/tree/libbsd_waf.py#n2469
>>
>> Chris
> 
> So, waf has a global view of the project. 

Yes, waf builds a complete picture of the build and jobs it has to run before it
starts running jobs and then it can keep all cores busy. The overhead it has is
similar for all hosts where Python is running natively and the number execs
calls made is minimal which helps on hosts where the exec call is costly, ie MacOS.

Have a look at a single compile line for us at the moment and the number of exec
calls made. It is crazy.

> I think this is similar to a single top-level Makefile.am with various "if XXX" constructs.

Could be.

> I am not sure how we want to proceed with the build system stuff.

It is becoming more and more important this work starts to happen. I am happy to
make the change to waf if we can attract the funding to do it.

I have pushed the existing build system as far as I want to without major
refactoring. I would like to test a recent autoconf and automake to see if we
can move to a recent version. The recent perl issues highlights the problems we
can expect to face if we do not change.

> Should we first clean up the existing build system first? 

I have no interesting in a "clean up" or a refactor. I have played with it
enough in recent times with the parallel build and pre-install work to know it
is a deep pit with much gnashing of teeth, rolling of eyes, and terrible roars.

The major issue we have is the time configure takes. I you play with the
rtems-bsp-builder and the --jobs option you will get an idea. If you look at
Joel's latest build:

https://lists.rtems.org/pipermail/build/2018-March/000554.html

jobs is '--jobs=6/12' which means run 6 parallel RTEMS BSP builds and use 12
cores per build. We can do this because of the time configure takes using a
single core. I think Joel has 24 cores on this box. This setting builds a BSP
every 18 seconds on average which is impressive but it is a hack.

> Should it be possible to run an
> old and new build system in parallel for a certain time?

Yes it should be and this is the plan. The removal of pre-install fixed a build
issue exposed with parallel building but it also served the more important role
of making a change to waf simpler.

chris