How to build start.o using waf?

[Chris Johns]

On 4/10/19 5:55 pm, Sebastian Huber wrote:
> On 04/10/2019 09:20, Chris Johns wrote:
>> On 4/10/19 4:21 pm, Sebastian Huber wrote:
>>> On 03/10/2019 04:32, Chris Johns wrote:
>>>> On 3/10/19 3:30 am, Gedare Bloom wrote:
>>>>> On Wed, Oct 2, 2019 at 5:12 AM Sebastian Huber
>>>>> <sebastian.huber at embedded-brains.de> wrote:
>>>>>>
>>>>>> On 30/09/2019 15:14, Sebastian Huber wrote:
>>>>>>> Hello,
>>>>>>>
>>>>>>> I would like to work on a new build system prototype. The idea is to use
>>>>>>> specification items maintained by Doorstop (YAML files), a Python
>>>>>>> configuration script and waf to build RTEMS and the tests. This is
>>>>>>> similar to the libbsd build. The difference is that in libbsd the build
>>>>>>> data is maintained directly in Python code (libbsd.py).
>>>>
>>>> A key design aspect is how the configuration of RTEMS is handled and
>>>> maintained.
>>>> Amar's solution provides a specific model for managing settings and
>>>> specifically
>>>> BSP options. We need a way to add options for a BSP that lets us collect,
>>>> document and validate these settings. We need a way to query and review option
>>>> defaults. The solution needs to be able to add and remove BSPs with minimal
>>>> impact and this leads to being able to build a BSP that is external to the
>>>> RTEMS
>>>> source tree. There are BSPs that are private or export restricted. Another
>>>> issue
>>>> to consider is how we deprecate, update or alter options.
>>>>
>>>> I suspect we will have an iterating design around the internal design and
>>>> implementation complexity and suitable external user work-flows.
>>>
>>> Now is a good time to create a wish list for the new build system. I cannot
>>> implement everything, but I can try to make a substantial initial step forward.
>>
>> I agree.
>>
>>> Doorstop is a tool to manage a directed, acyclic graph of items. Each item has a
>>> set of attributes (key-value pairs). Some attributes are pre-defined by Doorstop
>>> with requirements management in mind. You can also add an arbitrary number of
>>> custom attributes. With this data structure you can tackle a lot of problems.
>>> The items are stored in YAML format files.
>>
>> Note YAML is not part of the standard Python, it is add-on package. This means
>> using it would mean we would need to provide support on a number of hosts.
> 
> The libyaml package seems to be used in a lot of other packages and can be
> installed with pip. If the use of YAML for the build would be a show stopper for
> you, it would be good to know this early.

It is not about me saying yes or no, I am only highlighting the issue. I suggest
you check and test the different host types we support to make sure there is a
sensible realistic way to support YAML. This goes for any dependency we come
across. If you do not do this and no one comments or draws attention to any
potential issue the whole project has to live with the outcome once it is merged.

>>> My approach would be to store all the data that defines the build in build item
>>> files. The wscript file reads the build items, configures the build and performs
>>> the build as specified also by command line options, e.g. which BSP, with tests,
>>> etc. The build item files can be also used by other tools.
>>>
>>> A BSP is represented by a BSP build item which links to BSP option items, object
>>> items (start.o) and file groups for librtemsbsp.a.
>>>
>>> A BSP is just a node in the graph, you can ask for example: if I remove this BSP
>>> item, does it create isolated items? The isolated items can be removed if the
>>> BSP is removed. If you change an option, you can figure out the impact, e.g.
>>> which BSP uses this option.
>>
>> I had not considered going to level but what you say seems consistent with my
>> thinking on how configuration should happen. With out the detail it is hard so I
>> will make a general statement that a change to the build system has to serve the
>> general RTEMS community first and this means easy to access and manage
>> configurations generated by RTEMS. I ask RTEMS to configure a BSP and a
>> configuration file is generated. I can then edit the file to alter specific
>> settings I need. The is what Amar did.
> 
> My approach would be to configure the build via command line options and then
> the configuration is read-only for the build. 

It is not clear to me what the command line options are configuring. There are
build options and target/BSP options. It you include BSP options there are far
more than could sensibly be handled on the command line.

> Being able to alter an existing configuration seems a bit complex to me.

It can be simple, Amar's build used the INI format, it was easy to review and
alter and did not add any extra host dependencies. The build checked and
validated the config on each build and it did it quickly. Also the config does
not need to be generated and altered each time, it can be held in a user's
workspace used to build RTEMS.

Command line options for BSPOPT type configs is something I do not support. I do
not like what we have with configure BSPOPT_* and making them normal looking
option does not change this.

>> And I do not see Doorstop as a part of the general community :)
> 
> You would not need Doorstop to build a BSP, just to maintain a BSP.

Joel and I have made it clear certification cannot add overhead or complexity to
RTEMS that is not considered a normal workflow. Joel and I need to make sure the
project does not become side tracked by a certification process where the
overheads and complexity to develop RTEMS make it harder, we want the bar to be
lower not higher. Certification is a once in a decade activity for a specific
and important user while the development is a daily one. We need to find a
balance that works.

Changes like a workflow originating from Doorstop need to be clearly explained
at a high level. Recently you have started this build system work, you have
testing underway and you have made a reference to generated API headers. We
should not need to join the dots. I feel you run a risk of hitting a barrier if
what you are proposing is not simply explained at a high level rather than via a
series of bits and pieces posted as you work through the detail looking for
solutions. I appreciate the complexity and difficulties you are facing and I
respect your desire for a complete end to end solution however I am not
convinced it is worth doing when we adding in the community side of this open
source project. I am more than happy to be convinced it can be made to work.

>>> Since the build is driven by item files which just reside in a directory, you
>>> can also point the build script to another directory with items defined
>>> externally. The question is if waf is capable to build files outside the top
>>> directory. Building externally defined stuff has a very low priority for me. I
>>> just want to point out that there are no fundamental obstacles with this
>>> approach.
>>
>> Yes this is an issue and something I would need to discuss with Thomas.
>>
>>> The details of BSP options can be easily managed with specialized attributes,
>>> e.g. type, min, max, default value, etc. We can utilize the full power of
>>>
>>> https://docs.python.org/3.6/library/optparse.html
>>>
>>> which is a very nice Python module. We may use
>>>
>>> https://docs.python.org/3.6/library/optparse.html#optparse-option-callbacks
>>>
>>
>> Sure ...
>>
>>> to provide services we need for this BSP (and others):
>>>
>>> https://git.rtems.org/rtems/tree/c/src/lib/libbsp/arm/atsam/configure.ac
>>>
>>> The callbacks can be stored in the items files as Python code and imported with
>>> exec(). The callbacks need an ability to install a post-processing handler which
>>> is invoked during the waf configure(), e.g. to generate the linker command file
>>> based on options.
>>
>> Amar implemented this back in 2015 ....
>>
>> https://git.rtems.org/amar/waf-old.git/tree/py/config?h=fix
>>
>> ... and it is worth reviewing. It is simple and specific.
> 
> Yes, I have this check out. However, I was not able to configure or build
> anything due to Python/waf errors.

I have not tried it for a long time. Maybe ping Amar and ask.

>> The process needs to be wrapped so Python programming skill are not needed to
>> update and add new options or BSPs. Adding complex data structures in config
>> files can be hard to debug and fix if the provided support is not strong. A user
>> adding or working on a BSP has to be able to navigate and fix something they
>> break. The more complex the structuring of the data the more complex the support
>> needs to be.
> 
> Yes, but I think we can require some Python skills from a BSP developer.

Agreed.

Chris