Discussion: How to handle HALs, SDKs and libraries

[Karel Gardas]

On 7/15/23 09:47, oss at c-mauderer.de wrote:
> Am 15.07.23 um 01:48 schrieb Karel Gardas:
>>
>>   Hello,
>>
>> I've created setup where I've put updated STM32H7 HAL consisting of 
>> two submodules:
>>
>> - stm32h7xx_hal_driver
>> - cmsis_device_h7
>>
>> and ARM's
>>
>> - CMSIS_5
>>
>> into hals/arm/stm32h7 to follow related BSP code location.
> 
> I think I would prefer "contrib/hals", "external/hals" or some other 
> more generic top directory name. It would allow to use that for other 
> parts like for example zlib in "contrib/libs/zlib" in the future (if we 
> agree on having libs like that there too).

Indeed, name is not the problem here. Both looks fine, somewhat prefer 
external a bit.

> At the moment, you have absolute URLs for your submodules:
> 
>    url = https://github.com/ARM-software/CMSIS_5.git
>    url = https://github.com/karelfv/stm32h7xx_hal_driver.git
>    url = https://github.com/karelfv/cmsis_device_h7.git
> 
> If we use submodules, I would go for relative paths and clone the 
> repositories to our git.rtems.org and to all mirrors. It has the 
> advantage that even if some upstream URL changes, we won't be affected.

This is dirty PoC not a final solution. Due to this I've settled on 
absolute urls and not relative. For final solution relative are needed, 
but both url (relative/abolute) and name/location in RTEMS tree is very 
easy to change based on decision.

> I like submodules because they are well-supported by the usual tools. 

Honestly. I like submodules idea, but hate its implementation. Some reasons:

- submodules were added as a light weight feature and during the 
development of git/submodule feature they accumulated some more.

- submodules started as way to get commit id of the external project to 
the tree. But only master branch was supported. (not suitable for RTEMS)

- later submodules added capability to use tag (still not suitable for 
RTEMS)

- later submodules added capability to use different branch from the 
project, but only a head of it. (finally what RTEMS need).

- in the field you can see gits supporting different sets of submodule 
features described above.

- submodules are easy to be broken. Personal experience from Haskell 
compiler (GHC) project which uses submodules a lot. I have git 2.25 and 
I always have an issue to pull from GHC repo since alwyas for me one 
submodule (when changed) will be broken for me. Conclusion: clone with 
submodules may kind of work, but later pull may be tricky.

- besides submodules are not supported in 'got' which is 3rd party 
independent implementation of git like SCM with more sane UI

- due to submodules trickiness development policy for external 
integration with project code modification (aka HAL patches) needs to be 
very well written. I smell horrible increase in cognitive load here...

> Nonetheless, we should take a look at some of the reasons why other 
> projects decided against them. For example Google has the "repo" tool 
> used in Android. Zephyr has the "west" tool. OpenWRT uses "feeds" for 
> something similar. I'm sure there are more.
> 
>    https://gerrit.googlesource.com/git-repo/
>    https://docs.zephyrproject.org/latest/develop/west/why.html#
>    https://openwrt.org/docs/guide-developer/feeds
> 
> All solve a similar problem. Part of them even overlap already with 
> functionality from our waf based build system. We should consider 
> whether we need any of the functionality of these tools and whether we 
> want to re-implement it or whether we just want to use one of the tools 
> and integrate with it.

Oh, do I sense starting academic debate now? The point of my dirty PoC 
was to ground that a bit and show real stuff to get a feel of possibilities.

Anyway, based on my PoC experience I would probably prefer something 
more KISS and also more flexible by tool(s) being also aware of RTEMS BSPs

- first lets start with just 'boot' and 'update' scripts hosted in the 
repo and used to clone/update externals would be needed. And put 
'external' into git ignore.

- later boot/update my be made BSP aware. I mean when doing development 
on H7 I do not need HALs for Xilinx and other ARM/v8/Rx stuff. Neither I 
need HAL for NXP/Microchip chips.

- may this boot/update functionality be made part of ./waf tooling? 
Perhaps it may....

So based on this I can see how Zephyr gravitated to west...


>> I've also completely removed HAL code from the BSP directory and also 
>> CMSIS v4 files from bsps/arm/include to verify compilation is really 
>> using submodules.
> 
> I think the CMSIS headers are used by some other BSPs too. So that's 
> fine for a prototype. But for the final solution, we need something 
> different.

The PoC needs to prove some things hence removal of CMSIS from the tree. 
Nothing more. It's a "gain experience" vehicle, not a final solution. 
And btw, you will not be able to find final solution without 
touching/testing stuff and getting your hands dirty with it.

Everything has its own price: (i) adaptation/support of west for RTEMS 
or (ii) writing homegrown script(s) or (iii) never ending fight with git 
submodule incompatibilities and user confusion and most importantly (iv) 
never ending discussion or discussion fading away without any decision :-)

Thanks for all your comments!

Karel