Draft for moving network headers from RTEMS to newlib

[Sebastian Huber]

On 26/04/16 10:27, Chris Johns wrote:
> On 26/04/2016 17:31, Sebastian Huber wrote:
>> On 26/04/16 07:51, Chris Johns wrote:
>>> On 26/04/2016 01:06, Christian Mauderer wrote:
>>>> currently we try to remove the network specific POSIX headers from
>>>> RTEMS. Instead, we add current headers from FreeBSD to newlib. This 
>>>> will
>>>> simplify the build process of some libraries that depend on the 
>>>> network
>>>> (like LibreSSL).
>>>
>>> What does this work flow offer over building and installing an RTEMS
>>> kernel for a BSP and adding that path to the packages include paths?
>>
>> You don't build per BSP in this scenario. You build per multilib and
>> have only one set of header files installed.
>
> Hmm multilib, this is what I thought was happening. I see this 
> exploding the libraries if packages are built this way. Do we manage 
> this or not bother and just accept all variants have to be built?

The multilibs are there for a purpose. In case a particular architecture 
has superfluous multilibs, then we should remove them. I see an 
explosion of libraries if we build them for each BSP. We have count of 
BSPs > count of multilibs, otherwise the GCC configuration is broken.

>
>> You are able to use the
>> network header files during GCC build, which makes it easier to build
>> the run-time libraries of Ada and Go.
>
> Nice. Maybe limiting what is added to just building languages would be 
> a nice first starting point.
>
>>
>>>
>>> How do you get the flags for the compiler to build the package?
>>
>> See attached script which builds for example libpng for all multilibs.
>
> Multilibs has not had a very successful history with RTEMS (outside of 
> gcc). They are not very user friendly and are a source of questions 
> and issues.

I think multilibs are great for libraries which don't depend on BSP 
specifics. What is the benefit of per BSP libpng for example?

>
> I would like to see something more than a script posted before I get 
> excited and accepting of this path. I would like to understand the 
> work flow for the project, the developers and the users?
>
>> You don't need BSPs installed to do this. The libpng is just an add-on
>> to the tool chain.
>
> Sure, I understand this and it is nice. I hope users do not forget to 
> build the BSP. ;)

The work flow is build the tool chain, build all support libraries you 
are interested in, then build the actual BSPs.

>
>>
>>>
>>> How are any tests present in the package built and linked?
>>
>> Tests are executables, so they need a BSP.
>>
>
> Yes, so how do these packages get tested?

How are packages tested currently?

You can only test using a particular BSP.

>
>>>
>>> Do we risk limited the functionality of a package by restricting the
>>> headers exposed to only standards based headers? There are headers
>>> which some packages use that will not be present.
>>
>> In case a common header file is missing for a particular package, then
>> we can add this file to Newlib.
>>
>
> If a package changes or something happens we would need a new version 
> of tools. 

Yes, but this area is not really volatile. Its not that we have a new 
POSIX standard every two months. The old network stack uses about 20 
year old header files.

> I am concerned about the management of the detail here and finding a 
> suitable set of headers for all packages across all architecture may 
> become complicated if there are incompatibilities between 
> architectures and/or packages. With standards based headers we have 
> the standard to fall back on and we can get the package fixed. When we 
> move away from standards based headers we open up a range potential 
> issue. Adding the headers is easy, removing would be hard if not 
> impossible. Once we head down this path will be difficult to turn back.

The place of the header file makes no difference here. For the user its 
the same if you remove the file from Newlib or RTEMS.

>
> It is a massive task to get the tools to build for supported 
> architectures _now_ and I hope this path does not increase the work 
> load here.

If we add Ada and Go to the standard languages, then it could get more 
difficult.

>
> Do you have limits or boundaries for suitable headers? Who administers 
> these boundaries?
>
> What is the procedure for a user to add a header?

Like everything else in RTEMS this will be user driven. Someone is not 
satisfied with the current situation and will work out a fix.

>
> If a header is missing does this means the user cannot build a 
> package? In other words can we mix how we currently build packages 
> with this way of building packages?

You can still build against a BSP which presents the full set of header 
files.

>
>>>
>>>> Further it will be another step into the direction of
>>>> extracting the old RTEMS network stack and build it as an independent
>>>> package.
>>>
>>> Is this just a specialised version of the generic vertical integration
>>> problem being discussed in the civetweb thread?
>>>
>>> I am not against standards based headers like the ones being discussed
>>> here being move to newlib however I currently do not see what the
>>> advantage is and how value is being adding over a specific build order
>>> of packages.
>>
>> Building libraries against a particular BSP using actually BSP
>> independent header files is not really a great overall setup.
>
> Correct each BSP is independent and that gives me the ability to vary 
> and change things in RTEMS without needing to repeat tools builds or 
> create separate sandboxes at the tools level. The whole point of the 
> cpukit was to multilib build the generic code and avoid the 
> duplication in the BSPs however in the end it became clear a single 
> specific BSP is all most users need and overhead of that specific 
> build and install even with per BSP headers is faster and simpler than 
> the whole multilib path.
>
> Having a single tree of headers is fine if they are stable and not 
> changing. I do not use a single prefix and prefer separate prefixes so 
> I can mix and match and play. Having more and more header become part 
> of the tools removes an important part of this process.
>
> It comes down to the type of header and how often they change.

How often do standard header files change? Its much more likely that a 
tool chain update happens due to a GCC release or configuration change.

>
>>
>>>
>>> I still see all the normal issues of CFLAGS, LDFLAGS, 3rd party
>>> package dependence still being present once this work is completed.
>>
>> For libraries, the LDFLAGS are irrelevant.
>
> What about testing? This needs to be resolved.

Yes, this needs to be resolved, but we cannot fix all problems at once. 
We have to start somewhere. Moving the standard network header files to 
Newlib gives us the ability to

* more easily build the run-time libraries for Ada and Go,
* provide one set of header files for use with three different network 
stacks (old, libbsd, lwIP),
* in case we share these header files with Cygwin, then they are exposed 
to a large set of packages, and
* enable multilib based builds for network support libraries like SSL, 
HTTP server.

>
>> One advantage is that the
>> built-in search paths are sufficient if you build against a multilib
>> (except the machine flags), see attached script.
>
> This assumes you install everything under the one prefix. Unless you 
> are on a production build I do not recommend this. If you are building 
> and installing different libraries under different prefixes you are 
> back at the same issue of handling include paths so include paths will 
> always need to work.
>
> Please understand I am not against this, I would just like the all 
> detail exposed and understood.
>
> Chris

I don't understand why you need these libraries per BSP if they have 
absolutely no BSP dependency. I use these multilib based libraries for 
several years and it works quite well.

-- 
Sebastian Huber, embedded brains GmbH

Address : Dornierstr. 4, D-82178 Puchheim, Germany
Phone   : +49 89 189 47 41-16
Fax     : +49 89 189 47 41-09
E-Mail  : sebastian.huber at embedded-brains.de
PGP     : Public key available on request.

Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG.