How to build start.o using waf?

[Sebastian Huber]

On 05/10/2019 01:05, Chris Johns wrote:
> On 4/10/19 5:55 pm, Sebastian Huber wrote:
>> On 04/10/2019 09:20, Chris Johns wrote:
>>> On 4/10/19 4:21 pm, Sebastian Huber wrote:
>>>> On 03/10/2019 04:32, Chris Johns wrote:
>>>>> On 3/10/19 3:30 am, Gedare Bloom wrote:
>>>>>> On Wed, Oct 2, 2019 at 5:12 AM Sebastian Huber
>>>>>> <sebastian.huber at embedded-brains.de> wrote:
>>>>>>>
>>>>>>> On 30/09/2019 15:14, Sebastian Huber wrote:
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> I would like to work on a new build system prototype. The idea is to use
>>>>>>>> specification items maintained by Doorstop (YAML files), a Python
>>>>>>>> configuration script and waf to build RTEMS and the tests. This is
>>>>>>>> similar to the libbsd build. The difference is that in libbsd the build
>>>>>>>> data is maintained directly in Python code (libbsd.py).
>>>>>
>>>>> A key design aspect is how the configuration of RTEMS is handled and
>>>>> maintained.
>>>>> Amar's solution provides a specific model for managing settings and
>>>>> specifically
>>>>> BSP options. We need a way to add options for a BSP that lets us collect,
>>>>> document and validate these settings. We need a way to query and review option
>>>>> defaults. The solution needs to be able to add and remove BSPs with minimal
>>>>> impact and this leads to being able to build a BSP that is external to the
>>>>> RTEMS
>>>>> source tree. There are BSPs that are private or export restricted. Another
>>>>> issue
>>>>> to consider is how we deprecate, update or alter options.
>>>>>
>>>>> I suspect we will have an iterating design around the internal design and
>>>>> implementation complexity and suitable external user work-flows.
>>>>
>>>> Now is a good time to create a wish list for the new build system. I cannot
>>>> implement everything, but I can try to make a substantial initial step forward.
>>>
>>> I agree.
>>>
>>>> Doorstop is a tool to manage a directed, acyclic graph of items. Each item has a
>>>> set of attributes (key-value pairs). Some attributes are pre-defined by Doorstop
>>>> with requirements management in mind. You can also add an arbitrary number of
>>>> custom attributes. With this data structure you can tackle a lot of problems.
>>>> The items are stored in YAML format files.
>>>
>>> Note YAML is not part of the standard Python, it is add-on package. This means
>>> using it would mean we would need to provide support on a number of hosts.
>>
>> The libyaml package seems to be used in a lot of other packages and can be
>> installed with pip. If the use of YAML for the build would be a show stopper for
>> you, it would be good to know this early.
> 
> It is not about me saying yes or no, I am only highlighting the issue. I suggest
> you check and test the different host types we support to make sure there is a
> sensible realistic way to support YAML. This goes for any dependency we come
> across. If you do not do this and no one comments or draws attention to any
> potential issue the whole project has to live with the outcome once it is merged.

Ok, this makes sense, sorry for not finding this out myself. I sent a 
help request to the users list:

https://lists.rtems.org/pipermail/users/2019-October/067184.html

> 
>>>> My approach would be to store all the data that defines the build in build item
>>>> files. The wscript file reads the build items, configures the build and performs
>>>> the build as specified also by command line options, e.g. which BSP, with tests,
>>>> etc. The build item files can be also used by other tools.
>>>>
>>>> A BSP is represented by a BSP build item which links to BSP option items, object
>>>> items (start.o) and file groups for librtemsbsp.a.
>>>>
>>>> A BSP is just a node in the graph, you can ask for example: if I remove this BSP
>>>> item, does it create isolated items? The isolated items can be removed if the
>>>> BSP is removed. If you change an option, you can figure out the impact, e.g.
>>>> which BSP uses this option.
>>>
>>> I had not considered going to level but what you say seems consistent with my
>>> thinking on how configuration should happen. With out the detail it is hard so I
>>> will make a general statement that a change to the build system has to serve the
>>> general RTEMS community first and this means easy to access and manage
>>> configurations generated by RTEMS. I ask RTEMS to configure a BSP and a
>>> configuration file is generated. I can then edit the file to alter specific
>>> settings I need. The is what Amar did.
>>
>> My approach would be to configure the build via command line options and then
>> the configuration is read-only for the build.
> 
> It is not clear to me what the command line options are configuring. There are
> build options and target/BSP options. It you include BSP options there are far
> more than could sensibly be handled on the command line.
> 
>> Being able to alter an existing configuration seems a bit complex to me.
> 
> It can be simple, Amar's build used the INI format, it was easy to review and
> alter and did not add any extra host dependencies. The build checked and
> validated the config on each build and it did it quickly. Also the config does
> not need to be generated and altered each time, it can be held in a user's
> workspace used to build RTEMS.
> 
> Command line options for BSPOPT type configs is something I do not support. I do
> not like what we have with configure BSPOPT_* and making them normal looking
> option does not change this.

Ok, this is a sound approach. Configuring two or more BSPs via the 
command line would definitely not work well. I will try to implement it 
like this:

[arch/bsp]
RTEMS_DEBUG = On
RTEMS_SMP = Off
SOME_OPTION = "ABC"

We can also add BSP aliases to allow building of one BSP variant with 
different option values:

[arch/custom]
alias = bsp
RTEMS_DEBUG = Off
RTEMS_SMP = On
SOME_OPTION = "123"

The BSP options would come from YAML build specification items.

We may query the options via a command, e.g.

./waf options sparc/erc32

-> Get BSP options (with comments) for erc32 BSP of the sparc architecture

./waf options sparc

-> Get BSP options for all sparc BSPs

./waf options

-> Get BSP options for all BSPs

> 
>>> And I do not see Doorstop as a part of the general community :)
>>
>> You would not need Doorstop to build a BSP, just to maintain a BSP.
> 
> Joel and I have made it clear certification cannot add overhead or complexity to
> RTEMS that is not considered a normal workflow. Joel and I need to make sure the
> project does not become side tracked by a certification process where the
> overheads and complexity to develop RTEMS make it harder, we want the bar to be
> lower not higher. Certification is a once in a decade activity for a specific
> and important user while the development is a daily one. We need to find a
> balance that works.
> 
> Changes like a workflow originating from Doorstop need to be clearly explained
> at a high level. Recently you have started this build system work, you have
> testing underway and you have made a reference to generated API headers. We
> should not need to join the dots. I feel you run a risk of hitting a barrier if
> what you are proposing is not simply explained at a high level rather than via a
> series of bits and pieces posted as you work through the detail looking for
> solutions. I appreciate the complexity and difficulties you are facing and I
> respect your desire for a complete end to end solution however I am not
> convinced it is worth doing when we adding in the community side of this open
> source project. I am more than happy to be convinced it can be made to work.
[...]

I don't know why you are so sceptical. The pre-qualification activity is 
not only a source of overhead, it bears also chances and may end up with 
benefits for the project, e.g. an easier maintenance. My aim is to get 
rid of some redundancy in the project code base. At the moment I just 
have to ask questions and try to figure out what works and what doesn't 
work.

There is just one thing settled from my point of view and this is the 
use of Doorstop for the specification parts.

I will send an updated Software Requirements Engineering chapter next 
week. It contains more details with respect to the specification of 
RTEMS. Starting with the build system is attractive since going from 
Autoconf/Automake to something else is an easy way to improve things for 
the project. It is also a self-contained area. It helps to get familiar 
with the Doorstop concepts.

-- 
Sebastian Huber, embedded brains GmbH

Address : Dornierstr. 4, D-82178 Puchheim, Germany
Phone   : +49 89 189 47 41-16
Fax     : +49 89 189 47 41-09
E-Mail  : sebastian.huber at embedded-brains.de
PGP     : Public key available on request.

Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG.