How to build librtemscpu.a and librtemsbsp.a in the future?

Sat Mar 31 22:04:34 UTC 2018

On 31/3/18 5:07 am, Sebastian Huber wrote:
> ----- Am 29. Mrz 2018 um 0:16 schrieb Chris Johns chrisj at rtems.org:
> 
>> On 28/03/2018 16:30, Sebastian Huber wrote:
>>> On 28/03/18 02:35, Chris Johns wrote:
>>>> On 27/03/2018 17:17, Sebastian Huber wrote:
>>>>> On 27/03/18 01:15, Chris Johns wrote:
>>>>>> On 26/03/2018 17:32, Sebastian Huber wrote:
>>>>>>> On 25/03/18 05:20, Chris Johns wrote:
>>>>>>>> On 24/3/18 1:27 am, Sebastian Huber wrote:
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> I tried to consolidate the cpukit Makefile.am. This ended up in the
>>>>>>>>> following
>>>>>>>>> problem. The content of librtemscpu.a depends on the CPU port. This is
>>>>>>>>> currently
>>>>>>>>> done via a non-standard
>>>>>>>>>
>>>>>>>>> _SUBDIRS = ../@RTEMS_CPU@
>>>>>>>>>
>>>>>>>>> construct.
>>>>>>>> Where is this? I cannot find it.
>>>>>>> https://git.rtems.org/rtems/tree/cpukit/score/cpu/Makefile.am#n1
>>>>>>>
>>>>>> I am confused, the link's version does not have '../'?
>>>>> Sorry, I should have used copy and paste.
>>>>>
>>>>>> [...]
>>>>>>> I am not sure how we want to proceed with the build system stuff.
>>>>>> It is becoming more and more important this work starts to happen. I am
>>>>>> happy to
>>>>>> make the change to waf if we can attract the funding to do it.
>>>>>>
>>>>>> I have pushed the existing build system as far as I want to without major
>>>>>> refactoring. I would like to test a recent autoconf and automake to see if we
>>>>>> can move to a recent version. The recent perl issues highlights the problems we
>>>>>> can expect to face if we do not change.
>>>>>>
>>>>>>> Should we first clean up the existing build system first?
>>>>>> I have no interesting in a "clean up" or a refactor. I have played with it
>>>>>> enough in recent times with the parallel build and pre-install work to know it
>>>>>> is a deep pit with much gnashing of teeth, rolling of eyes, and terrible roars.
>>>>> I think there are some issues left which need a clean up:
>>>>>
>>>>> * Use of RTEMS_RELLDFLAGS'
>>>> I have not looked into this stuff. What is happening with these flags and these
>>>> *.rel files?
>>>>
>>>> Is this some sort of component system using incremental objects ?
>>>
>>> I think it is a hack to enable the use of LIBADD. It was used mainly by the
>>> libcpu.
>>
>> Some BSPs also have this?
> 
> There are not that many spots left to clean up:
> 
> grep -r --include='Makefile.am' RTEMS_RELLDFLAGS                                                                                                  
> c/src/lib/libbsp/powerpc/beatnik/Makefile.am:network_support_rel_LDFLAGS = $(RTEMS_RELLDFLAGS)
> c/src/lib/libbsp/powerpc/beatnik/Makefile.am:network_if_mve_tmp_rel_LDFLAGS  = $(RTEMS_RELLDFLAGS)
> c/src/lib/libbsp/powerpc/beatnik/Makefile.am:network_if_gfe_tmp_rel_LDFLAGS = $(RTEMS_RELLDFLAGS)
> c/src/lib/libbsp/powerpc/beatnik/Makefile.am:network_if_em_tmp_rel_LDFLAGS = $(RTEMS_RELLDFLAGS)
> c/src/lib/libbsp/powerpc/mvme5500/Makefile.am:network_rel_LDFLAGS = $(RTEMS_RELLDFLAGS)
> c/src/lib/libbsp/powerpc/mvme5500/Makefile.am:mvme5500start___OBJEXT__LDFLAGS = $(RTEMS_RELLDFLAGS)
> c/src/lib/libbsp/powerpc/motorola_powerpc/Makefile.am:polledIO_rel_LDFLAGS = $(RTEMS_RELLDFLAGS)
> cpukit/libfs/src/nfsclient/Makefile.am:dirutils_rel_LDFLAGS = $(RTEMS_RELLDFLAGS)
> cpukit/libfs/src/nfsclient/Makefile.am:nfs_rel_LDFLAGS = $(RTEMS_RELLDFLAGS)
> cpukit/libfs/src/nfsclient/Makefile.am:rpcio_rel_LDFLAGS = $(RTEMS_RELLDFLAGS)
> 

OK

>>
>>>
>>>>
>>>>> * Command line defines
>>>>>
>>>> Lets leave that to the ticket on this topic.
>>>>
>>>>> * Host tools should be removed or moved to rtems-tools
>>>> Agreed.
>>>>
>>>>>> The major issue we have is the time configure takes. I you play with the
>>>>>> rtems-bsp-builder and the --jobs option you will get an idea. If you look at
>>>>>> Joel's latest build:
>>>>>>
>>>>>> https://lists.rtems.org/pipermail/build/2018-March/000554.html
>>>>>>
>>>>>> jobs is '--jobs=6/12' which means run 6 parallel RTEMS BSP builds and use 12
>>>>>> cores per build. We can do this because of the time configure takes using a
>>>>>> single core. I think Joel has 24 cores on this box. This setting builds a BSP
>>>>>> every 18 seconds on average which is impressive but it is a hack.
>>>>> Yes, I think we need only two configure runs. One in the top-level and one in
>>>>> the BSP (for the BSP options).
>>>>>
>>>> That would be nice. I suggest we resolve how the tree looks in the other thread
>>>> before we proceed. It would provide clarity for this work and a solid direction.
>>>
>>> The host tools are a blocking point for me to do anything with the configure.ac
>>> files.
>>>
>>> Another issue are the complicated test program Makefile.am.
>>
>> What issues are you seeing?
> 
> It is a mixture of an Automake program and the classic RTEMS makefile support.
> 

Ah, OK.

>>
>>> We have the optional
>>> BSP-specific post-link command which transforms an .exe file (ELF) into a .ralf
>>> file.
>>
>> I see printing the size as wasted build time and we should remove that step. 
> 
> Yes, we definitely should not print this stuff.
> 

Agreed.

>> You
>> can run size on any static ELF executable and get the sizes if you are
>> interested and it is easy to script something to print all test executable
>> sizes.
>>
>> I am not sure about the post-link parts, I can see they are useful even as an
>> example and I can also see the case for RTEMS stopping at a static ELF file and
>> simplifying the BSP configs.
>>
>> The current way the post-link is done is specific to building the samples and
>> tests and any Makefile.inc support, after that the details of what needs to
>> happen are not accessible. Mkaefile.inc support is to be remove so this leave
>> the samples and tests. The rtems_waf tool cannot perform a post-link operation
>> because nothing is captured in the pkgconfig file and adding it from the current
>> Makefile fragment would be a difficult task. This is why the .pc work stalled.
> 
> Normally, an ELF file is all you need. A post link step is quite common for embedded targets, but from a host operating system point of view it is quite exotic.
> 

Yes, and I think it is important we can capture this detail someway because
collecting this type of information from the web can be confusing.

>>
>>> The RTEMS tester duplicates this feature somehow.
>>
>> Yes it does. It has to because of the last point, the details are not openly
>> available from a built or installed BSP. The RTEMS Tester's support is effective
>> but not optimal.
>>
>>> How would waf deal with this post-link step?
>>
>> I do not know, it is something I have not considered.
>>
>> The problem I see with what we currently have is not the operation itself it is
>> access to the details outside of the RTEMS build system. If that can be solved
>> at the BSP level as an exported controlled interface we can build eco-system
>> support around it.
>>
>> For waf I would look at adding a per BSP configuration in a format that can be
>> exported easily. Once that information exists I would have the RTEMS tester use
>> it as well. In waf you could see if a post-link operation is configured and add
>> a build job based on a rule, ie a more advanced version of the current RTEMS
>> tester's `target_pretest_command`. How and what requires looking at all the
>> existing post-link commands to see what can be done.
> 
> Ok, so per se waf supports a post link step.
> 

In waf compiling and linking are build steps so adding a build step after
linking is easy and requires nothing special, you just create a rule based
build object with the executable as the source and the target in the binary file.

>>
>> Just wondering aloud, if the BSP configuration was a pkgconfig file and not a
>> makefile fragment and the RTEMS build system parsed it would that help? It could
>> then be installed or an extended version with build specific details capturing
>> any details we want. Pkg-config supports variables so we could have something
>> like (hand made and cleaned up version installed by RTEMS):
>>
>> $ cat /tmp/arm-rtems5-xilinx_zynq_zedboard.pc
>> #
>> # pkg-config support file for RTEMS BSP xilinx_zynq_zedboard
>> #
>> # Warning: This stuff is experimental and may be changed at any time.
>> #
>> rtems_major=5
>> arch=arm
>> bsp=xilinx_zynq_zedboard
>>
>> arch_bsp=${arch}/${bsp}
>> arch_prefix=${arch}-rtems${rtems_major}
>>
>> prefix=/opt/work/rtems/bsps/5
>> exec_prefix=/opt/work/rtems/bsps/5/${arch_prefix}
>> libdir=${exec_prefix}/${bsp}/lib
>> includedir=${exec_prefix}/${bsp}/lib/include
>>
>> CFLAGS=-march=armv7-a -mthumb -mfpu=neon -mfloat-abi=hard -mtune=cortex-a9 -O2
>> -g -ffunction-sections -fdata-sections
>>
>> Name: arm-rtems5-xilinx_zynq_zedboard
>> Version: 5.0.0
>> Description: RTEMS BSP xilinx_zynq_zedboard
>> Libs:
>> Cflags: -qrtems -B${exec_prefix}/lib/ -B${libdir}/ --specs bsp_specs ${CFLAGS}
>>
>> start_addr=0x00104000
>> entry_addr=0x00104000
>>
>> bin=${arch_prefix}-objdump -R -S --strip-debug -O binary @EXE@ @EXE at .bin && \
>>    cat @EXE at .bin | gzip -9 > @EXE at .gz && \
>>    mkimage -A arm -O rtems -T kernel -a ${start_addr} -e ${entry_addr} \
>>    -n "RTEMS" -d @EXE at .gz @EXE at .img
>>
>> Asking for the variable returns:
>>
>> $ PKG_CONFIG_PATH=/tmp pkg-config --variable=bin arm-rtems5-xilinx_zynq_zedboard
>> arm-rtems5-objdump -R -S --strip-debug -O binary @EXE@ @EXE at .bin && cat
>> @EXE at .bin | gzip -9 > @EXE at .gz && mkimage -A arm -O rtems -T kernel -a
>> 0x00104000 -e 0x00104000 -n "RTEMS" -d @EXE at .gz @EXE at .img
>>
>> (I am not sure the gzip pipe and redirect would work but it shows the idea)
>>
>> We could define and document a set of standard variables with known functions.
>>
>> BSPs could provide more, for example DFT, BSP options, etc and the BSP
>> documentation would list those.
>>
>> Detecting a variable could be checking `--print-variables` or the empty result
>> pkg-config gives.
>>
>> It would really improve rtems_waf having this detail. I could remove the
>> 'tweaks' support I need to have for some BSPs.
>>
>> Any build system that supports pkg-config could be used.
> 
> Ok, this sounds like a good direction. I think it is acceptable if we break the existing post link support during the build system conversion if we have a suitable replacement and a proof of concept. Maintained BSPs can then easily resurrect this feature.

I agree. We should capture what exists as best we can.

> 
>>
>>> Is there a standard support for a post-link step in Automake?
>>
>> I do not know.
>>
>>>
>>> The "make dist" is broken?
>>>
>>
>> Is it used? I do not use it to make a release.
> 
> Good, then I don't have to care about this. 

Yes.

> Was it ever used? There is a lot of stuff in the Makefile.am for the make dist.

I suppose it was given the amount of related stuff in the build system but I
have never used it or maintained it.

Chris