How to build start.o using waf?

Wed Oct 9 23:17:24 UTC 2019

On Mon, Oct 7, 2019 at 6:45 PM Chris Johns <chrisj at rtems.org> wrote:
>
> On 7/10/19 7:00 pm, Sebastian Huber wrote:
> > On 05/10/2019 01:05, Chris Johns wrote:
> >> On 4/10/19 5:55 pm, Sebastian Huber wrote:
> >>> On 04/10/2019 09:20, Chris Johns wrote:
> >>>> On 4/10/19 4:21 pm, Sebastian Huber wrote:
> >>>>> On 03/10/2019 04:32, Chris Johns wrote:
> >>>>>> On 3/10/19 3:30 am, Gedare Bloom wrote:
> >>>>>>> On Wed, Oct 2, 2019 at 5:12 AM Sebastian Huber
> >>>>>>> <sebastian.huber at embedded-brains.de> wrote:
> >>>>>>>>
> >>>>>>>> On 30/09/2019 15:14, Sebastian Huber wrote:
> >>>>>>>>> Hello,
> >>>>>>>>>
> >>>>>>>>> I would like to work on a new build system prototype. The idea is to use
> >>>>>>>>> specification items maintained by Doorstop (YAML files), a Python
> >>>>>>>>> configuration script and waf to build RTEMS and the tests. This is
> >>>>>>>>> similar to the libbsd build. The difference is that in libbsd the build
> >>>>>>>>> data is maintained directly in Python code (libbsd.py).
> >>>>>>
> >>>>>> A key design aspect is how the configuration of RTEMS is handled and
> >>>>>> maintained.
> >>>>>> Amar's solution provides a specific model for managing settings and
> >>>>>> specifically
> >>>>>> BSP options. We need a way to add options for a BSP that lets us collect,
> >>>>>> document and validate these settings. We need a way to query and review
> >>>>>> option
> >>>>>> defaults. The solution needs to be able to add and remove BSPs with minimal
> >>>>>> impact and this leads to being able to build a BSP that is external to the
> >>>>>> RTEMS
> >>>>>> source tree. There are BSPs that are private or export restricted. Another
> >>>>>> issue
> >>>>>> to consider is how we deprecate, update or alter options.
> >>>>>>
> >>>>>> I suspect we will have an iterating design around the internal design and
> >>>>>> implementation complexity and suitable external user work-flows.
> >>>>>
> >>>>> Now is a good time to create a wish list for the new build system. I cannot
> >>>>> implement everything, but I can try to make a substantial initial step
> >>>>> forward.
> >>>>
> >>>> I agree.
> >>>>
> >>>>> Doorstop is a tool to manage a directed, acyclic graph of items. Each item
> >>>>> has a
> >>>>> set of attributes (key-value pairs). Some attributes are pre-defined by
> >>>>> Doorstop
> >>>>> with requirements management in mind. You can also add an arbitrary number of
> >>>>> custom attributes. With this data structure you can tackle a lot of problems.
> >>>>> The items are stored in YAML format files.
> >>>>
> >>>> Note YAML is not part of the standard Python, it is add-on package. This means
> >>>> using it would mean we would need to provide support on a number of hosts.
> >>>
> >>> The libyaml package seems to be used in a lot of other packages and can be
> >>> installed with pip. If the use of YAML for the build would be a show stopper for
> >>> you, it would be good to know this early.
> >>
> >> It is not about me saying yes or no, I am only highlighting the issue. I suggest
> >> you check and test the different host types we support to make sure there is a
> >> sensible realistic way to support YAML. This goes for any dependency we come
> >> across. If you do not do this and no one comments or draws attention to any
> >> potential issue the whole project has to live with the outcome once it is merged.
> >
> > Ok, this makes sense, sorry for not finding this out myself.
>
> No problem.
>
> > I sent a help
> > request to the users list:
> >
> > https://lists.rtems.org/pipermail/users/2019-October/067184.html
>
> Great.
>
> >>>>> My approach would be to store all the data that defines the build in build
> >>>>> item
> >>>>> files. The wscript file reads the build items, configures the build and
> >>>>> performs
> >>>>> the build as specified also by command line options, e.g. which BSP, with
> >>>>> tests,
> >>>>> etc. The build item files can be also used by other tools.
> >>>>>
> >>>>> A BSP is represented by a BSP build item which links to BSP option items,
> >>>>> object
> >>>>> items (start.o) and file groups for librtemsbsp.a.
> >>>>>
> >>>>> A BSP is just a node in the graph, you can ask for example: if I remove
> >>>>> this BSP
> >>>>> item, does it create isolated items? The isolated items can be removed if the
> >>>>> BSP is removed. If you change an option, you can figure out the impact, e.g.
> >>>>> which BSP uses this option.
> >>>>
> >>>> I had not considered going to level but what you say seems consistent with my
> >>>> thinking on how configuration should happen. With out the detail it is hard
> >>>> so I
> >>>> will make a general statement that a change to the build system has to serve
> >>>> the
> >>>> general RTEMS community first and this means easy to access and manage
> >>>> configurations generated by RTEMS. I ask RTEMS to configure a BSP and a
> >>>> configuration file is generated. I can then edit the file to alter specific
> >>>> settings I need. The is what Amar did.
> >>>
> >>> My approach would be to configure the build via command line options and then
> >>> the configuration is read-only for the build.
> >>
> >> It is not clear to me what the command line options are configuring. There are
> >> build options and target/BSP options. It you include BSP options there are far
> >> more than could sensibly be handled on the command line.
> >>
> >>> Being able to alter an existing configuration seems a bit complex to me.
> >>
> >> It can be simple, Amar's build used the INI format, it was easy to review and
> >> alter and did not add any extra host dependencies. The build checked and
> >> validated the config on each build and it did it quickly. Also the config does
> >> not need to be generated and altered each time, it can be held in a user's
> >> workspace used to build RTEMS.
> >>
> >> Command line options for BSPOPT type configs is something I do not support. I do
> >> not like what we have with configure BSPOPT_* and making them normal looking
> >> option does not change this.
> >
> > Ok, this is a sound approach. Configuring two or more BSPs via the command line
> > would definitely not work well. I will try to implement it like this:
> >
> > [arch/bsp]
> > RTEMS_DEBUG = On
> > RTEMS_SMP = Off
> > SOME_OPTION = "ABC"
> >
> > We can also add BSP aliases to allow building of one BSP variant with different
> > option values:
> >
> > [arch/custom]
> > alias = bsp
> > RTEMS_DEBUG = Off
> > RTEMS_SMP = On
> > SOME_OPTION = "123"
>
> This is the sort of thing I would expect to see.
>
> > The BSP options would come from YAML build specification items.
>
> Sure, the format is open and depends on solving how we manage users when we know
> an extra step for the environment set up is needed.
>
> > We may query the options via a command, e.g.
> >
> > ./waf options sparc/erc32
> >
>
> Amar and I both settled on rtems-config as a stand alone command to do this. The
> idea is this tool is installed along with the BSP config data so it is available
> inside and outside the build tree. This becomes the formal interface for the
> configuration and separates the implementation details.
>
> The separate installed command lets users access important build details for a
> BSP in their selected build system and deployment environment knowing the
> interface is stable. A BSP may change which means the results returned can vary
> between releases however that is a secondary issue between BSP maintainers and
> the BSP users.
>
> > -> Get BSP options (with comments) for erc32 BSP of the sparc architecture
> >
> > ./waf options sparc
> >
>
> Hmmm waf has taken the options space and I am not sure if this is confusing.
>
> I am OK with `./waf --list-arch sparc` to list the specifics for an arch, is BSPs.
>
> > -> Get BSP options for all sparc BSPs
> >
> > ./waf options
> >
> > -> Get BSP options for all BSPs
> >
>
> When does the configure phase happen?
>
> I can see something like:
>
>  ./waf --list
>  Architectures:
>   sparc
>   arm
>   powerpc
>   x86
>
>  ./waf --list sparc
>  BSPs: sparc
>   erc32
>   leon3
>   leon2
>
>  ./waf --list sparc/erc32
>  BSP options: sparc/erc32
>   console: /dev/console
>   SOME_OPTION: 123
>   ...
>
> To configure:
>
>  ./waf configure --rtems-bsp sparc/erc32
>
> which creates a config build file, for example rtems-sparc-erc32-config. You
> could then query it with:
>
>  ./rtems-config sparc/erc32 arch
>  sparc-rtems5
>  ./rtems-config sparc/erc32 cc
>  sparc-rtems5-gcc
>  ./rtems-config sparc/erc32 console
>  /dev/console
>
> At this point in time I am thinking the config file would be edited. This is not
> fixed as I am not considered much beyond here.
>
> >>
> >>>> And I do not see Doorstop as a part of the general community :)
> >>>
> >>> You would not need Doorstop to build a BSP, just to maintain a BSP.
> >>
> >> Joel and I have made it clear certification cannot add overhead or complexity to
> >> RTEMS that is not considered a normal workflow. Joel and I need to make sure the
> >> project does not become side tracked by a certification process where the
> >> overheads and complexity to develop RTEMS make it harder, we want the bar to be
> >> lower not higher. Certification is a once in a decade activity for a specific
> >> and important user while the development is a daily one. We need to find a
> >> balance that works.
> >>
> >> Changes like a workflow originating from Doorstop need to be clearly explained
> >> at a high level. Recently you have started this build system work, you have
> >> testing underway and you have made a reference to generated API headers. We
> >> should not need to join the dots. I feel you run a risk of hitting a barrier if
> >> what you are proposing is not simply explained at a high level rather than via a
> >> series of bits and pieces posted as you work through the detail looking for
> >> solutions. I appreciate the complexity and difficulties you are facing and I
> >> respect your desire for a complete end to end solution however I am not
> >> convinced it is worth doing when we adding in the community side of this open
> >> source project. I am more than happy to be convinced it can be made to work.
> > [...]
> >
> > I don't know why you are so sceptical.
>
> I am not sure sceptical is the right word but I can see how you may think this.
> I have not been shown how this will all come together and work and that is
> understandable as it is a work in progress. As a result it is difficult for me
> to say yes this is the way forward so I am attempting to present a high level
> view of how I will evaluate what is presented. Also my views are evolving as
> time goes on.
>
> Certification is new to us all as an open source project. As a result I can see
> the possibility of us being swept up in what can be done and over reaching and
> leaving behind the community that has brought us to the place we are current at.
> At the moment I can add features to RTEMS without needing to create a series of
> requirements base around a 3rd party standard. I feel the burden of
> responsibility rests on those who have a vested interest in a certified RTEMS
> and not the community and so think there is a need for some clear separation at
> some level. Again if I add a feature do I need to maintain the requirements? If
> the origin of how RTEMS is built and maintained is Doorstop then I think it is
> fair for me to question this.
>
> > The pre-qualification activity is not
> > only a source of overhead, it bears also chances and may end up with benefits
> > for the project, e.g. an easier maintenance.
>
> I agree and why I support and encourage this activity. We need to find a
> workable balance.
>
> > My aim is to get rid of some redundancy in the project code base.
>
> I can see this and it is the starting point or origin that I am interested in.
>
> > At the moment I just have to ask questions
> > and try to figure out what works and what doesn't work.
>
> I understand.
>
I agree with both of you on many points made in this thread. It's a
bit unfortunate this great conversation has been buried in the
innocuous sounding subject. :) I would also like to see some
high-level view about the vision / pieces coming together, but I also
appreciate that it can be hard to see that view when you are "down in
the weeds" working it all out.

Pre-certification will greatly improve RTEMS Project software
engineering and the quality of the open-source products involved. I'm
even not sure about the argument that it 'must not make things
harder', because I'm certain some things will become easier, some
things may be harder; if we have a strong argument at the high-level
that it is close to 'net neutral negative impact' on the development
team and doesn't make it harder to onboard new developers (GSoC
students, etc), then this effort will be a clear win to me.
Unfortunately, we can't make those predictions with the knowledge we
have today. It would be greatly helpful to have some kind of master
plan or vision for the open-source community to try to grapple with
the pros/cons of the pre-certification effort's proposed changes,
instead of trying to take them piecemeal.

Gedare

> > There is just one thing settled from my point of view and this is the use of
> > Doorstop for the specification parts.
>
> Yes and given the results you have present I do as well.
>
> > I will send an updated Software Requirements Engineering chapter next week. It
> > contains more details with respect to the specification of RTEMS. Starting with
> > the build system is attractive since going from Autoconf/Automake to something
> > else is an easy way to improve things for the project. It is also a
> > self-contained area. It helps to get familiar with the Doorstop concepts.
>
> Maybe we can review it in person when we meet next week?
>
> Chris
> _______________________________________________
> devel mailing list
> devel at rtems.org
> http://lists.rtems.org/mailman/listinfo/devel