How to build start.o using waf?

Fri Oct 4 07:55:58 UTC 2019

On 04/10/2019 09:20, Chris Johns wrote:
> On 4/10/19 4:21 pm, Sebastian Huber wrote:
>> On 03/10/2019 04:32, Chris Johns wrote:
>>> On 3/10/19 3:30 am, Gedare Bloom wrote:
>>>> On Wed, Oct 2, 2019 at 5:12 AM Sebastian Huber
>>>> <sebastian.huber at embedded-brains.de> wrote:
>>>>>
>>>>> On 30/09/2019 15:14, Sebastian Huber wrote:
>>>>>> Hello,
>>>>>>
>>>>>> I would like to work on a new build system prototype. The idea is to use
>>>>>> specification items maintained by Doorstop (YAML files), a Python
>>>>>> configuration script and waf to build RTEMS and the tests. This is
>>>>>> similar to the libbsd build. The difference is that in libbsd the build
>>>>>> data is maintained directly in Python code (libbsd.py).
>>>
>>> A key design aspect is how the configuration of RTEMS is handled and maintained.
>>> Amar's solution provides a specific model for managing settings and specifically
>>> BSP options. We need a way to add options for a BSP that lets us collect,
>>> document and validate these settings. We need a way to query and review option
>>> defaults. The solution needs to be able to add and remove BSPs with minimal
>>> impact and this leads to being able to build a BSP that is external to the RTEMS
>>> source tree. There are BSPs that are private or export restricted. Another issue
>>> to consider is how we deprecate, update or alter options.
>>>
>>> I suspect we will have an iterating design around the internal design and
>>> implementation complexity and suitable external user work-flows.
>>
>> Now is a good time to create a wish list for the new build system. I cannot
>> implement everything, but I can try to make a substantial initial step forward.
> 
> I agree.
> 
>> Doorstop is a tool to manage a directed, acyclic graph of items. Each item has a
>> set of attributes (key-value pairs). Some attributes are pre-defined by Doorstop
>> with requirements management in mind. You can also add an arbitrary number of
>> custom attributes. With this data structure you can tackle a lot of problems.
>> The items are stored in YAML format files.
> 
> Note YAML is not part of the standard Python, it is add-on package. This means
> using it would mean we would need to provide support on a number of hosts.

The libyaml package seems to be used in a lot of other packages and can 
be installed with pip. If the use of YAML for the build would be a show 
stopper for you, it would be good to know this early.

> 
>> My approach would be to store all the data that defines the build in build item
>> files. The wscript file reads the build items, configures the build and performs
>> the build as specified also by command line options, e.g. which BSP, with tests,
>> etc. The build item files can be also used by other tools.
>>
>> A BSP is represented by a BSP build item which links to BSP option items, object
>> items (start.o) and file groups for librtemsbsp.a.
>>
>> A BSP is just a node in the graph, you can ask for example: if I remove this BSP
>> item, does it create isolated items? The isolated items can be removed if the
>> BSP is removed. If you change an option, you can figure out the impact, e.g.
>> which BSP uses this option.
> 
> I had not considered going to level but what you say seems consistent with my
> thinking on how configuration should happen. With out the detail it is hard so I
> will make a general statement that a change to the build system has to serve the
> general RTEMS community first and this means easy to access and manage
> configurations generated by RTEMS. I ask RTEMS to configure a BSP and a
> configuration file is generated. I can then edit the file to alter specific
> settings I need. The is what Amar did.

My approach would be to configure the build via command line options and 
then the configuration is read-only for the build. Being able to alter 
an existing configuration seems a bit complex to me.

> 
> And I do not see Doorstop as a part of the general community :)

You would not need Doorstop to build a BSP, just to maintain a BSP.

> 
>> Since the build is driven by item files which just reside in a directory, you
>> can also point the build script to another directory with items defined
>> externally. The question is if waf is capable to build files outside the top
>> directory. Building externally defined stuff has a very low priority for me. I
>> just want to point out that there are no fundamental obstacles with this approach.
> 
> Yes this is an issue and something I would need to discuss with Thomas.
> 
>> The details of BSP options can be easily managed with specialized attributes,
>> e.g. type, min, max, default value, etc. We can utilize the full power of
>>
>> https://docs.python.org/3.6/library/optparse.html
>>
>> which is a very nice Python module. We may use
>>
>> https://docs.python.org/3.6/library/optparse.html#optparse-option-callbacks
>>
> 
> Sure ...
> 
>> to provide services we need for this BSP (and others):
>>
>> https://git.rtems.org/rtems/tree/c/src/lib/libbsp/arm/atsam/configure.ac
>>
>> The callbacks can be stored in the items files as Python code and imported with
>> exec(). The callbacks need an ability to install a post-processing handler which
>> is invoked during the waf configure(), e.g. to generate the linker command file
>> based on options.
> 
> Amar implemented this back in 2015 ....
> 
> https://git.rtems.org/amar/waf-old.git/tree/py/config?h=fix
> 
> ... and it is worth reviewing. It is simple and specific.

Yes, I have this check out. However, I was not able to configure or 
build anything due to Python/waf errors.

> 
> The process needs to be wrapped so Python programming skill are not needed to
> update and add new options or BSPs. Adding complex data structures in config
> files can be hard to debug and fix if the provided support is not strong. A user
> adding or working on a BSP has to be able to navigate and fix something they
> break. The more complex the structuring of the data the more complex the support
> needs to be.

Yes, but I think we can require some Python skills from a BSP developer.

-- 
Sebastian Huber, embedded brains GmbH

Address : Dornierstr. 4, D-82178 Puchheim, Germany
Phone   : +49 89 189 47 41-16
Fax     : +49 89 189 47 41-09
E-Mail  : sebastian.huber at embedded-brains.de
PGP     : Public key available on request.

Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG.