Building Tools From Source

[Ralf Corsepius]

On 09/21/2012 06:49 AM, Chris Johns wrote:
> Ralf Corsepius wrote:
>> On 09/21/2012 12:15 AM, Chris Johns wrote:
>>> Ralf Corsepius wrote:
>>>>
>>>> * The symlinks are a way to build these libraries from source while
>>>> building GCC. This means static linking and therefore is not
>>>> necessarily a good idea.
>>>>
>>>
>>> I think for RTEMS having these libraries built with GCC (statically
>>> linked or not) has advantages. The reasons statically linking can be
>>> considered harmful are well known [1] how-ever having these libraries
>>> that are an important part of the compiler controlled as part of the
>>> compiler's configuration set goes a long way to simplifying the
>>> verification and validation audit. If I was currently responsible for
>>> building a critical application I would like to be able to audit and
>>> provide to the V&V team the exact configuration used for testing,
>>> development and release. For me this means aways building the tools from
>>> source.
>>
>> Well, that's the common argument proponents of static linkage come up
>> with. With my Fedora Packaging Committee hat on, I cannot avoid to call
>> this view "short-sighted" and "dangerous".
>
> I agree a distribution would have this view and if I was in your
> position with that hat on I would 100% agree.
>
> RTEMS is not a distribution and that 'hat' is not as good a fit.

The toolchains as I provide them for some Linux variants are "SW add-on 
package suites". They are not actually any different from any "SW 
package suites" OS vendors supply. The only real difference is the 
origin of the packages. The working principles behind them are the same.

> RTEMS
> is an operating system and in a lot cases orthogonal to the requirements
> the development host has.

Well, I do not agree. The RTEMS toolchains are just a "set of packages", 
which, due to being OpenSource can be supplied and maintained in 
different ways.

One way is to use an OS's package-management and the resources vendors 
supply and trust them that they are "doing it right". Another way is to 
"do-it-yourself", to ignore vendors' resources and to trust "in yourselves".

Both ways have their justifications.

> Making sure I can come back to the application
> being developed many many years later and can generate the same
> instructions given the same input source code is an overriding requirement.

To be able to achieve this, you'd have to take a snapshot of the whole 
environment (Comprising the whole OS, i.e. you'd have to take a binary 
image) and not only the tools, because other (vendor supplied) packages 
may also impact "reproducability" (Notably libc).

>> Apart from the technical problems static linkage can imply (It's often
>> not possible - Does not apply to gmp, mpfr and mpc), experience from
>> history tells, in longer terms, static linkage is guaranteed to be hit
>> by the downsides of static linkage by different extends.
>
> The key requirement is being able to rebuild the compiler and generate
> the same instructions. If the host being used is not able to perform
> that task that host is questionable.

As written above, this only works if you to keep the host OS frozen, 
with all the bugs it might contain and not to upgrade it.

In this case, in the end, whether using static or dynamic linkage 
doesn't make a real difference.

Conversely, if having tripped over bugs, selectively applying updates 
(e.g. replacing shared libs) occasionally is helpful.

>> Prominent example from history: The zlib incident ca. 15 years ago.
>> Then, zlib was suffering from security senitive bugs, which had gone
>> through unnoticed for a long time. Tnanks to the fact everybody used
>> statically linked zlibs, then, when these issues were finially noticed,
>> fixing this issue was very tedious for OS vendors and product vendors.
>
> I agree this is a good fit for a host and a distribution how-ever the
> security of a host does not effect the instructions the compiler creates
> given the RTEMS source code.

Sure, the RTEMS toolchains are not a web-server nor web-browser, so the 
security concerns are very low.

Nevertheless ... you never know. E.g. libc may invoke net-worked 
services underneath and a memory leak in a statically linked library be 
exploited ...

>> In other words, when vendor supplied libraries suffer from bugs, you're
>> better off using shared libs.
>
> Except if there is a possible chance the code generated changes then you
> have a problem. Going back and getting the old source and building the
> compiler a different way because you need those specific dependent parts
> exactly the same would also be questionable if the way you build the
> compiler changes. Loading the old OS into a VM is a possible solution if
> the VM can host that specific version of the host OS.
No disagreement.

>> I don't know if Apple or a trustworthy 3rd
>> party supplies binaries for gmp, mpfr, mpc on MacOS (I would be
>> surprised if this does not apply), you're likely better off using these
>> libraries and to apply shared linkage and not to use your "homegrown"
>> static versions.
>
> I have no idea what Apple provide on MacOS, or what is provided on
> FreeBSD (I use both) nor do I need to be concerned.

Well, that's the difference in your way of thinking and mine.

I trust more in OS vendors' dedicated package maintainers who are 
supposed to be deeply familiar with packages and supposed to take more 
care about packages, than I ever could myself.

Another lesson learnt throughout the years: Building everything yourself 
only provides a "feeling of safety and calmness", often based on 
unawareness of issues, often due to lack of intimacy with packages.

As a consequence of this, my strategy is:
- For active works, use actively maintained distros and keep "everything 
rolling".
- For dead works (aka. "long term maintenance") and if reproducability 
is of concern, use frozen images

Ralf