How to build start.o using waf?

[Chris Johns]

On 7/10/19 7:00 pm, Sebastian Huber wrote:
> On 05/10/2019 01:05, Chris Johns wrote:
>> On 4/10/19 5:55 pm, Sebastian Huber wrote:
>>> On 04/10/2019 09:20, Chris Johns wrote:
>>>> On 4/10/19 4:21 pm, Sebastian Huber wrote:
>>>>> On 03/10/2019 04:32, Chris Johns wrote:
>>>>>> On 3/10/19 3:30 am, Gedare Bloom wrote:
>>>>>>> On Wed, Oct 2, 2019 at 5:12 AM Sebastian Huber
>>>>>>> <sebastian.huber at embedded-brains.de> wrote:
>>>>>>>>
>>>>>>>> On 30/09/2019 15:14, Sebastian Huber wrote:
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> I would like to work on a new build system prototype. The idea is to use
>>>>>>>>> specification items maintained by Doorstop (YAML files), a Python
>>>>>>>>> configuration script and waf to build RTEMS and the tests. This is
>>>>>>>>> similar to the libbsd build. The difference is that in libbsd the build
>>>>>>>>> data is maintained directly in Python code (libbsd.py).
>>>>>>
>>>>>> A key design aspect is how the configuration of RTEMS is handled and
>>>>>> maintained.
>>>>>> Amar's solution provides a specific model for managing settings and
>>>>>> specifically
>>>>>> BSP options. We need a way to add options for a BSP that lets us collect,
>>>>>> document and validate these settings. We need a way to query and review
>>>>>> option
>>>>>> defaults. The solution needs to be able to add and remove BSPs with minimal
>>>>>> impact and this leads to being able to build a BSP that is external to the
>>>>>> RTEMS
>>>>>> source tree. There are BSPs that are private or export restricted. Another
>>>>>> issue
>>>>>> to consider is how we deprecate, update or alter options.
>>>>>>
>>>>>> I suspect we will have an iterating design around the internal design and
>>>>>> implementation complexity and suitable external user work-flows.
>>>>>
>>>>> Now is a good time to create a wish list for the new build system. I cannot
>>>>> implement everything, but I can try to make a substantial initial step
>>>>> forward.
>>>>
>>>> I agree.
>>>>
>>>>> Doorstop is a tool to manage a directed, acyclic graph of items. Each item
>>>>> has a
>>>>> set of attributes (key-value pairs). Some attributes are pre-defined by
>>>>> Doorstop
>>>>> with requirements management in mind. You can also add an arbitrary number of
>>>>> custom attributes. With this data structure you can tackle a lot of problems.
>>>>> The items are stored in YAML format files.
>>>>
>>>> Note YAML is not part of the standard Python, it is add-on package. This means
>>>> using it would mean we would need to provide support on a number of hosts.
>>>
>>> The libyaml package seems to be used in a lot of other packages and can be
>>> installed with pip. If the use of YAML for the build would be a show stopper for
>>> you, it would be good to know this early.
>>
>> It is not about me saying yes or no, I am only highlighting the issue. I suggest
>> you check and test the different host types we support to make sure there is a
>> sensible realistic way to support YAML. This goes for any dependency we come
>> across. If you do not do this and no one comments or draws attention to any
>> potential issue the whole project has to live with the outcome once it is merged.
> 
> Ok, this makes sense, sorry for not finding this out myself. 

No problem.

> I sent a help
> request to the users list:
> 
> https://lists.rtems.org/pipermail/users/2019-October/067184.html

Great.

>>>>> My approach would be to store all the data that defines the build in build
>>>>> item
>>>>> files. The wscript file reads the build items, configures the build and
>>>>> performs
>>>>> the build as specified also by command line options, e.g. which BSP, with
>>>>> tests,
>>>>> etc. The build item files can be also used by other tools.
>>>>>
>>>>> A BSP is represented by a BSP build item which links to BSP option items,
>>>>> object
>>>>> items (start.o) and file groups for librtemsbsp.a.
>>>>>
>>>>> A BSP is just a node in the graph, you can ask for example: if I remove
>>>>> this BSP
>>>>> item, does it create isolated items? The isolated items can be removed if the
>>>>> BSP is removed. If you change an option, you can figure out the impact, e.g.
>>>>> which BSP uses this option.
>>>>
>>>> I had not considered going to level but what you say seems consistent with my
>>>> thinking on how configuration should happen. With out the detail it is hard
>>>> so I
>>>> will make a general statement that a change to the build system has to serve
>>>> the
>>>> general RTEMS community first and this means easy to access and manage
>>>> configurations generated by RTEMS. I ask RTEMS to configure a BSP and a
>>>> configuration file is generated. I can then edit the file to alter specific
>>>> settings I need. The is what Amar did.
>>>
>>> My approach would be to configure the build via command line options and then
>>> the configuration is read-only for the build.
>>
>> It is not clear to me what the command line options are configuring. There are
>> build options and target/BSP options. It you include BSP options there are far
>> more than could sensibly be handled on the command line.
>>
>>> Being able to alter an existing configuration seems a bit complex to me.
>>
>> It can be simple, Amar's build used the INI format, it was easy to review and
>> alter and did not add any extra host dependencies. The build checked and
>> validated the config on each build and it did it quickly. Also the config does
>> not need to be generated and altered each time, it can be held in a user's
>> workspace used to build RTEMS.
>>
>> Command line options for BSPOPT type configs is something I do not support. I do
>> not like what we have with configure BSPOPT_* and making them normal looking
>> option does not change this.
> 
> Ok, this is a sound approach. Configuring two or more BSPs via the command line
> would definitely not work well. I will try to implement it like this:
> 
> [arch/bsp]
> RTEMS_DEBUG = On
> RTEMS_SMP = Off
> SOME_OPTION = "ABC"
> 
> We can also add BSP aliases to allow building of one BSP variant with different
> option values:
> 
> [arch/custom]
> alias = bsp
> RTEMS_DEBUG = Off
> RTEMS_SMP = On
> SOME_OPTION = "123"

This is the sort of thing I would expect to see.

> The BSP options would come from YAML build specification items.

Sure, the format is open and depends on solving how we manage users when we know
an extra step for the environment set up is needed.

> We may query the options via a command, e.g.
> 
> ./waf options sparc/erc32
> 

Amar and I both settled on rtems-config as a stand alone command to do this. The
idea is this tool is installed along with the BSP config data so it is available
inside and outside the build tree. This becomes the formal interface for the
configuration and separates the implementation details.

The separate installed command lets users access important build details for a
BSP in their selected build system and deployment environment knowing the
interface is stable. A BSP may change which means the results returned can vary
between releases however that is a secondary issue between BSP maintainers and
the BSP users.

> -> Get BSP options (with comments) for erc32 BSP of the sparc architecture
> 
> ./waf options sparc
> 

Hmmm waf has taken the options space and I am not sure if this is confusing.

I am OK with `./waf --list-arch sparc` to list the specifics for an arch, is BSPs.

> -> Get BSP options for all sparc BSPs
> 
> ./waf options
> 
> -> Get BSP options for all BSPs
> 

When does the configure phase happen?

I can see something like:

 ./waf --list
 Architectures:
  sparc
  arm
  powerpc
  x86

 ./waf --list sparc
 BSPs: sparc
  erc32
  leon3
  leon2

 ./waf --list sparc/erc32
 BSP options: sparc/erc32
  console: /dev/console
  SOME_OPTION: 123
  ...

To configure:

 ./waf configure --rtems-bsp sparc/erc32

which creates a config build file, for example rtems-sparc-erc32-config. You
could then query it with:

 ./rtems-config sparc/erc32 arch
 sparc-rtems5
 ./rtems-config sparc/erc32 cc
 sparc-rtems5-gcc
 ./rtems-config sparc/erc32 console
 /dev/console

At this point in time I am thinking the config file would be edited. This is not
fixed as I am not considered much beyond here.

>>
>>>> And I do not see Doorstop as a part of the general community :)
>>>
>>> You would not need Doorstop to build a BSP, just to maintain a BSP.
>>
>> Joel and I have made it clear certification cannot add overhead or complexity to
>> RTEMS that is not considered a normal workflow. Joel and I need to make sure the
>> project does not become side tracked by a certification process where the
>> overheads and complexity to develop RTEMS make it harder, we want the bar to be
>> lower not higher. Certification is a once in a decade activity for a specific
>> and important user while the development is a daily one. We need to find a
>> balance that works.
>>
>> Changes like a workflow originating from Doorstop need to be clearly explained
>> at a high level. Recently you have started this build system work, you have
>> testing underway and you have made a reference to generated API headers. We
>> should not need to join the dots. I feel you run a risk of hitting a barrier if
>> what you are proposing is not simply explained at a high level rather than via a
>> series of bits and pieces posted as you work through the detail looking for
>> solutions. I appreciate the complexity and difficulties you are facing and I
>> respect your desire for a complete end to end solution however I am not
>> convinced it is worth doing when we adding in the community side of this open
>> source project. I am more than happy to be convinced it can be made to work.
> [...]
> 
> I don't know why you are so sceptical. 

I am not sure sceptical is the right word but I can see how you may think this.
I have not been shown how this will all come together and work and that is
understandable as it is a work in progress. As a result it is difficult for me
to say yes this is the way forward so I am attempting to present a high level
view of how I will evaluate what is presented. Also my views are evolving as
time goes on.

Certification is new to us all as an open source project. As a result I can see
the possibility of us being swept up in what can be done and over reaching and
leaving behind the community that has brought us to the place we are current at.
At the moment I can add features to RTEMS without needing to create a series of
requirements base around a 3rd party standard. I feel the burden of
responsibility rests on those who have a vested interest in a certified RTEMS
and not the community and so think there is a need for some clear separation at
some level. Again if I add a feature do I need to maintain the requirements? If
the origin of how RTEMS is built and maintained is Doorstop then I think it is
fair for me to question this.

> The pre-qualification activity is not
> only a source of overhead, it bears also chances and may end up with benefits
> for the project, e.g. an easier maintenance. 

I agree and why I support and encourage this activity. We need to find a
workable balance.

> My aim is to get rid of some redundancy in the project code base. 

I can see this and it is the starting point or origin that I am interested in.

> At the moment I just have to ask questions
> and try to figure out what works and what doesn't work.

I understand.

> There is just one thing settled from my point of view and this is the use of
> Doorstop for the specification parts.

Yes and given the results you have present I do as well.

> I will send an updated Software Requirements Engineering chapter next week. It
> contains more details with respect to the specification of RTEMS. Starting with
> the build system is attractive since going from Autoconf/Automake to something
> else is an easy way to improve things for the project. It is also a
> self-contained area. It helps to get familiar with the Doorstop concepts.

Maybe we can review it in person when we meet next week?

Chris