How to build start.o using waf?

[Chris Johns]

On 4/10/19 4:21 pm, Sebastian Huber wrote:
> On 03/10/2019 04:32, Chris Johns wrote:
>> On 3/10/19 3:30 am, Gedare Bloom wrote:
>>> On Wed, Oct 2, 2019 at 5:12 AM Sebastian Huber
>>> <sebastian.huber at embedded-brains.de> wrote:
>>>>
>>>> On 30/09/2019 15:14, Sebastian Huber wrote:
>>>>> Hello,
>>>>>
>>>>> I would like to work on a new build system prototype. The idea is to use
>>>>> specification items maintained by Doorstop (YAML files), a Python
>>>>> configuration script and waf to build RTEMS and the tests. This is
>>>>> similar to the libbsd build. The difference is that in libbsd the build
>>>>> data is maintained directly in Python code (libbsd.py).
>>
>> A key design aspect is how the configuration of RTEMS is handled and maintained.
>> Amar's solution provides a specific model for managing settings and specifically
>> BSP options. We need a way to add options for a BSP that lets us collect,
>> document and validate these settings. We need a way to query and review option
>> defaults. The solution needs to be able to add and remove BSPs with minimal
>> impact and this leads to being able to build a BSP that is external to the RTEMS
>> source tree. There are BSPs that are private or export restricted. Another issue
>> to consider is how we deprecate, update or alter options.
>>
>> I suspect we will have an iterating design around the internal design and
>> implementation complexity and suitable external user work-flows.
> 
> Now is a good time to create a wish list for the new build system. I cannot
> implement everything, but I can try to make a substantial initial step forward.

I agree.

> Doorstop is a tool to manage a directed, acyclic graph of items. Each item has a
> set of attributes (key-value pairs). Some attributes are pre-defined by Doorstop
> with requirements management in mind. You can also add an arbitrary number of
> custom attributes. With this data structure you can tackle a lot of problems.
> The items are stored in YAML format files.

Note YAML is not part of the standard Python, it is add-on package. This means
using it would mean we would need to provide support on a number of hosts.

> My approach would be to store all the data that defines the build in build item
> files. The wscript file reads the build items, configures the build and performs
> the build as specified also by command line options, e.g. which BSP, with tests,
> etc. The build item files can be also used by other tools.
> 
> A BSP is represented by a BSP build item which links to BSP option items, object
> items (start.o) and file groups for librtemsbsp.a.
> 
> A BSP is just a node in the graph, you can ask for example: if I remove this BSP
> item, does it create isolated items? The isolated items can be removed if the
> BSP is removed. If you change an option, you can figure out the impact, e.g.
> which BSP uses this option.

I had not considered going to level but what you say seems consistent with my
thinking on how configuration should happen. With out the detail it is hard so I
will make a general statement that a change to the build system has to serve the
general RTEMS community first and this means easy to access and manage
configurations generated by RTEMS. I ask RTEMS to configure a BSP and a
configuration file is generated. I can then edit the file to alter specific
settings I need. The is what Amar did.

And I do not see Doorstop as a part of the general community :)

> Since the build is driven by item files which just reside in a directory, you
> can also point the build script to another directory with items defined
> externally. The question is if waf is capable to build files outside the top
> directory. Building externally defined stuff has a very low priority for me. I
> just want to point out that there are no fundamental obstacles with this approach.

Yes this is an issue and something I would need to discuss with Thomas.

> The details of BSP options can be easily managed with specialized attributes,
> e.g. type, min, max, default value, etc. We can utilize the full power of
> 
> https://docs.python.org/3.6/library/optparse.html
> 
> which is a very nice Python module. We may use
> 
> https://docs.python.org/3.6/library/optparse.html#optparse-option-callbacks
> 

Sure ...

> to provide services we need for this BSP (and others):
> 
> https://git.rtems.org/rtems/tree/c/src/lib/libbsp/arm/atsam/configure.ac
> 
> The callbacks can be stored in the items files as Python code and imported with
> exec(). The callbacks need an ability to install a post-processing handler which
> is invoked during the waf configure(), e.g. to generate the linker command file
> based on options.

Amar implemented this back in 2015 ....

https://git.rtems.org/amar/waf-old.git/tree/py/config?h=fix

... and it is worth reviewing. It is simple and specific.

The process needs to be wrapped so Python programming skill are not needed to
update and add new options or BSPs. Adding complex data structures in config
files can be hard to debug and fix if the provided support is not strong. A user
adding or working on a BSP has to be able to navigate and fix something they
break. The more complex the structuring of the data the more complex the support
needs to be.

Chris