Discussion: How to handle HALs, SDKs and libraries

Sat Jul 15 07:47:20 UTC 2023

Hello Karel,

thanks for trying that. I wouldn't object to add it to 6 but I wouldn't 
expect it either because it means a lot of adaptions to the release 
scripts. Chris most likely can tell you more about whether it's even 
thinkable at that point in the release cycle or not.

Am 15.07.23 um 01:48 schrieb Karel Gardas:
> 
>   Hello,
> 
> I've created setup where I've put updated STM32H7 HAL consisting of two 
> submodules:
> 
> - stm32h7xx_hal_driver
> - cmsis_device_h7
> 
> and ARM's
> 
> - CMSIS_5
> 
> into hals/arm/stm32h7 to follow related BSP code location.

I think I would prefer "contrib/hals", "external/hals" or some other 
more generic top directory name. It would allow to use that for other 
parts like for example zlib in "contrib/libs/zlib" in the future (if we 
agree on having libs like that there too).

At the moment, you have absolute URLs for your submodules:

   url = https://github.com/ARM-software/CMSIS_5.git
   url = https://github.com/karelfv/stm32h7xx_hal_driver.git
   url = https://github.com/karelfv/cmsis_device_h7.git

If we use submodules, I would go for relative paths and clone the 
repositories to our git.rtems.org and to all mirrors. It has the 
advantage that even if some upstream URL changes, we won't be affected.

I like submodules because they are well-supported by the usual tools. 
Nonetheless, we should take a look at some of the reasons why other 
projects decided against them. For example Google has the "repo" tool 
used in Android. Zephyr has the "west" tool. OpenWRT uses "feeds" for 
something similar. I'm sure there are more.

   https://gerrit.googlesource.com/git-repo/
   https://docs.zephyrproject.org/latest/develop/west/why.html#
   https://openwrt.org/docs/guide-developer/feeds

All solve a similar problem. Part of them even overlap already with 
functionality from our waf based build system. We should consider 
whether we need any of the functionality of these tools and whether we 
want to re-implement it or whether we just want to use one of the tools 
and integrate with it.

> 
> I've also completely removed HAL code from the BSP directory and also 
> CMSIS v4 files from bsps/arm/include to verify compilation is really 
> using submodules.

I think the CMSIS headers are used by some other BSPs too. So that's 
fine for a prototype. But for the final solution, we need something 
different.

> 
> If you like to see how this is looking, just:
> 
> $ git clone https://github.com/karelfv/rtems.git rtems
> $ cd rtems
> $ git checkout rtems-stm32h7-hal-git-submodules
> $ git submodule update --init
> 
> And see bsps/arm/stm32h7/ and hals/arm/stm32h7/ and bsps/arm/include 
> subdirectories.
> 
> You can also compile if you have arm tools:
> 
> $ ./waf bspdefaults --rtems-bsps=arm/stm32h757i-eval > stm32h757i-eval.ini
> $ ./waf configure --rtems-bsps=arm/stm32h757i-eval 
> --rtems-config=./stm32h757i-eval.ini --rtems-tools=<tools> 
> --prefix=<prefix>
> $ ./waf
> 
> CAVEAT: only STM32H757i-eval BSP is supported and only compilation. 
> Installation is not supported yet.
> 
> Note: HAL is updated to latest STMicro code and all related changes done 
> in the past by Sebastian and Christian are merged into submodules 
> projects except Doxygen tag change done by Sebastian. This is to be 
> discussed topic if doxygen changes belongs to submodule or are not 
> needed anymore...

I agree that the doxygen shouldn't be necessary any more. But we should 
make sure that our Doxyfile doesn't pick up the submodules. So that 
would need some of the EXCLUDE* options set to the path of the submodules.

Best regards

Christian

> 
> hello.exe/ticker.exe/paranoia.exe runs fine.
> 
> Comments welcome!
> 
> Karel
> 
> 
> On 7/14/23 13:53, Karel Gardas wrote:
>>
>> Hello,
>>
>> are we really that close to RTEMS 6 release that none of this is 
>> acceptable to do now?
>>
>> Asking since I'd also like to update stm32h7 HAL. I may do that 
>> manually or I may do that submodule way which may perhaps save some of 
>> the manual work involved as some of the patches may not be needed 
>> anymore. It depends on resolution to following questions:
>>
>> - stm32h7 hal contains Sebastian's modification/additions of Doxygen 
>> tags. Do we need to preserve that in the submodule? I guess not, but 
>> I'm not sure and this question of Doxygen markups was not solved in 
>> this thread yet AFAIK.
>>
>> - all ARMs HALs probably also depends on particular version of CMSIS 
>> files, so I guess we should use separate submodules of CMSIS per HAL. 
>> At least STM recommends that: 
>> https://htmlpreview.github.io/?https://github.com/STMicroelectronics/STM32CubeH7/blob/master/Release_Notes.html -- this would certainly be improvement in comparison with current state of the art besides update of CMSIS alone we would also support manufacture tested CMSIS version for their HAL...
>>
>> - where to put submodules? What is preferred location? For stm32h7 case!
>>    (i) hals/arm/stm32h7
>>   or
>>    (ii) deps/arm/stm32h7
>>   or
>>    (iii) contrib/arm/stm32h7
>>   or
>>    (iv) anything else preferred?
>>
>>
>> I may do stm32h7 hal + its CMSIS dependency now (or next week), but 
>> only if the chance of inclusion into RTEMS 6 is reasonably high. 
>> Otherwise I would need to go more safer route of manual in-tree HAL 
>> update assuming this is still acceptable for RTEMS 6 release.
>>
>> Testing of stm32h7 changes is on my shoulders, hardware is here and 
>> customer is using the platform in field. Offering this as an 
>> opportunity to test the submodule vehicle ASAP on some part of RTEMS...
>>
>> Thanks,
>> Karel
>>
>> On 5/23/23 09:26, Christian MAUDERER wrote:
>>> Hello,
>>>
>>> I recently updated the HAL in the i.MXRT BSP. I used the same 
>>> approach that we use for a lot of similar cases: Import the sources 
>>> into RTEMS and adapt them slightly so that they work for us. So 
>>> basically a Clone-and-Own approach.
>>>
>>> During the discussion of the patches, some concerns were raised, 
>>> whether we should find a better solution to handle HALs, SDKs and 
>>> similar cases. We should start discussing a solution that can be used 
>>> after the 6 release so that maybe someone can start to work on a 
>>> prototype.
>>>
>>> Some example cases are:
>>>
>>> - the mcux_sdk in the imxrt BSP
>>> - the hal in the stm32h7 BSP
>>> - general ARM CMSIS files
>>> - zlib
>>> - libfdt
>>>
>>> One solution could be to build these libraries external and only link 
>>> RTEMS with them. There are disadvantages to this aproach:
>>>
>>> - Also in my experience, the API of the HALs / SDKs / libraries seems 
>>> to be quite stable, it's possible that there are combinations where 
>>> some unexpected change breaks a driver or makes it impossible to link 
>>> the applications.
>>>
>>> - BSPs rely on basic drivers from these libraries (like console or 
>>> clock driver). If we link against the libraries, the testsuite 
>>> wouldn't build any more without preinstalled libraries.
>>>
>>> Another solution could be to include libraties like that as 
>>> submodules and build them using the RTEMS build system. We could 
>>> clone the repos onto the RTEMS git server, and add necessary patches. 
>>> Advantage would be that it is more similar to the process that we 
>>> currently have. Another advantage is that we have a known-working 
>>> version of the files. Upstream updates could be either merged or we 
>>> could rebase our patches to a new version.
>>>
>>>  From my point of view, the second option would be the better one 
>>> especially because we have a tested, fixed version of the library 
>>> instead telling the user to just use some random version that might 
>>> or might not work.
>>>
>>> Regardless which aproach we use: We have to think about how to handle 
>>> that on releases. In the link aproach (first case), we have to 
>>> somehow archive source tar balls and some kind of build recipe. In 
>>> the submodule aproach, we could checkout all submodules and pack the 
>>> files into the RTEMS release tar ball. So I would expect that the 
>>> second aproach has less impact here too.
>>>
>>> Comments? Improvements? Better suggestions?
>>>
>>> Best regards
>>>
>>> Christian
>>
>> _______________________________________________
>> devel mailing list
>> devel at rtems.org
>> http://lists.rtems.org/mailman/listinfo/devel
> 
> _______________________________________________
> devel mailing list
> devel at rtems.org
> http://lists.rtems.org/mailman/listinfo/devel