Change build specification files from YAML to JSON?

[Chris Johns]

On 26/4/2023 6:22 pm, Sebastian Huber wrote:
> On 26.04.23 01:52, Chris Johns wrote:
>> On 25/4/2023 5:35 pm, Sebastian Huber wrote:
>>> Tooling makes sense if you have a high change rate.
>>
>> That is not the use case I am discussing and it is not a factor.
>>
>>> This is not the case for the RTEMS build specification items.
>>
>> I am not saying it is.
>>
>>> For which use case do we need tooling support?
>>
>> We need tooling to lets us all understand this build system. YAML is easy to
>> learn however the data-structures, rules and the large number of file fragments
>> we have are complex and not apparent to anyone looking at it even if you have
>> invested time doing so. It reflects the fact that RTEMS is complicated to build.
> 
> I don't think building an RTEMS BSP is complicated once you have the tools
> installed.

Sorry, my "user" is someone developing something for RTEMS and needing to update
the spec files. I was not clear about this.

>> I am advocate for what we have and will continue to be one so I am attempting to
>> address some things we could do better. History has shown the RTEMS core
>> developers are not great judges of the community's view of the project's
>> usability and accessibility.
>>
>> The analogy is to consider the git command then the roles github and gitlab
>> provide with their user interfaces and tooling.
>>
>>> For a new option or BSP, you just copy and past from existing files/BSPs.
>>> Changing a specific part of an existing file is just copy and paste some lines
>>> and edit them in most cases.
>>
>> My experience tells me it is not as simple as this. There is an element of guess
>> work a change is right. The recent dependency cases highlighted this and the
>> need for checking of some form to be present in the rtems.git repo.
>>
>> We need to step back and consider the role of the build system before we discuss
>> implementation details.
>>
>> The first requirement by a large margin is ease of use for users and the
>> community to make contributions. Meeting this requirement can be done a number
>> of ways. For example a user focused format for the relationships rather than one
>> that aids machine integration. The original ground work Amar did for the move to
>> waf was to define the build in Python as documented by waf. It was simple and
>> transparent. Another example is a machine focused format however we need tooling
>> to help the users manage making changes and accessing what is happening without
>> needing to learn the details of how it is implemented.
>>
>> For tooling my order of importance is:
>>
>> 1. Visualise the structures and dependencies in a human readable manner. An
>> indication of which file is doing what would be helpful.
>>
>> 2. A check of changes users make that raises errors, dependency problems, etc.
>> This can be a command you run if you are making changes and does not need to run
>> every build.
>>
>> I see JSON and tooling as linked together. I also not expect complete tooling to
>> be present for the change to happen. All I am wanting is the need for tooling be
>> agreed on and it is located in rtems.git. The development can then happen and
>> evolve as the community sees a need.
> 
> From my experience with waf I would not say that the waf build system is simple
> and transparent. Things which are very easy with make are complicated with waf
> at least for me. 

Yes this is valid and I agree. If you are outside the scope of something simple
it can get hard. It does some things well and easily and others are hard. Waf is
not alone here.

> I asked a couple of waf questions on the RTEMS mailing list
> which no core developer could answer. The support from the waf developer was a
> great help, but without this help I would not have been able to write the waf
> based build system for RTEMS.

Developing a framework does require a different level of integration with waf
and it can be involved. Waf is good as the base of a framework and I think what
we have is a good example.

I agree with Joel, we have not explained or sold the change as well as we need
to have.

What ever the format I would like the ability to create user focused tools but
currently the code to load the YAML (pickled for not) is not accessible as a
module in rtems.git.

> I think our biggest obstacle to improve things in the area you outlined above is
> the scattered project infrastructure with several Git repositories and the lack
> of CI pipelines. I will probably start a discussion about this topic because I
> think our current project setting has severe maintenance issues.

I am not so sure, there are problems with both approaches. We recently stripped
the rtems.git repo down to be what it is now and so moving back to a single
integrated repo would be returning to the past.

If you consider a single repo to be the right approach then I suggest you
present that first and explain how we release, possible ecosystem changes and
how existing users deploy systems? I work with users who have CI with the repos
we provide using open deployment methods and they have no issues. This is for
RTEMS 5 and 6.

Chris