Building Tools From Source

[Chris Johns]

Ralf Corsepius wrote:
> On 09/21/2012 06:49 AM, Chris Johns wrote:
>> Ralf Corsepius wrote:
>>> On 09/21/2012 12:15 AM, Chris Johns wrote:
>>>> Ralf Corsepius wrote:
>>>>>
>>>>> * The symlinks are a way to build these libraries from source while
>>>>> building GCC. This means static linking and therefore is not
>>>>> necessarily a good idea.
>>>>>
>>>>
>>>> I think for RTEMS having these libraries built with GCC (statically
>>>> linked or not) has advantages. The reasons statically linking can be
>>>> considered harmful are well known [1] how-ever having these libraries
>>>> that are an important part of the compiler controlled as part of the
>>>> compiler's configuration set goes a long way to simplifying the
>>>> verification and validation audit. If I was currently responsible for
>>>> building a critical application I would like to be able to audit and
>>>> provide to the V&V team the exact configuration used for testing,
>>>> development and release. For me this means aways building the tools
>>>> from
>>>> source.
>>>
>>> Well, that's the common argument proponents of static linkage come up
>>> with. With my Fedora Packaging Committee hat on, I cannot avoid to call
>>> this view "short-sighted" and "dangerous".
>>
>> I agree a distribution would have this view and if I was in your
>> position with that hat on I would 100% agree.
>>
>> RTEMS is not a distribution and that 'hat' is not as good a fit.
>
> The toolchains as I provide them for some Linux variants are "SW add-on
> package suites". They are not actually any different from any "SW
> package suites" OS vendors supply. The only real difference is the
> origin of the packages. The working principles behind them are the same.
>

Yes I agree and when I use Linux I welcome the packages. I would not 
consider bootstrapping Linux from source. I how-ever do differ from you 
when it comes to the parts that make up an RTEMS application stack, ie 
the tools, newlib, RTEMS source and finally the application. I prefer 
source.

>> RTEMS
>> is an operating system and in a lot cases orthogonal to the requirements
>> the development host has.
>
> Well, I do not agree. The RTEMS toolchains are just a "set of packages",
> which, due to being OpenSource can be supplied and maintained in
> different ways.

And I see a set of packages (for a specific set of hosts) _provide_ the 
RTEMS tools. It is a subtle different. I see the RTEMS tools as the 
source and patches and the packages a specific means of providing them 
to those users who use those host operating systems. I also see a script 
or build-from-source tool as another means of providing the RTEMS tools.

> One way is to use an OS's package-management and the resources vendors
> supply and trust them that they are "doing it right". Another way is to
> "do-it-yourself", to ignore vendors' resources and to trust "in
> yourselves".

I do not question the Linux vendors on providing a working Linux host. 
If the same source is used with a script to build the tools verses the 
packages I would expect the same RTEMS source to generate the same 
instructions.

>
> Both ways have their justifications.
>

Agreed and it is good to discuss this and to make sure RTEMS users are 
aware they have a choice.

>> Making sure I can come back to the application
>> being developed many many years later and can generate the same
>> instructions given the same input source code is an overriding
>> requirement.
>
> To be able to achieve this, you'd have to take a snapshot of the whole
> environment (Comprising the whole OS, i.e. you'd have to take a binary
> image) and not only the tools, because other (vendor supplied) packages
> may also impact "reproducability" (Notably libc).
>

Yes libc can impact code generation. I have seen the sort routine change 
in libc and effect the way registers where allocated on the local stack 
of functions on the m68k arch. The code was equivalent how-ever 
different. This sort of change between releases of an OS or between 
different hosts can be frustrating.

I have had to make a release to a system about 4 or 5 years after the 
last release. I agreed with the customer that 2 previous releases built 
and matched byte or byte meant the tools could be used for the needed 
bug fix. We had a whole snapshot of the OS as install disks and a test 
procedure to set it up how-ever the PC buses and devices did not match 
and the OS could not be made to run. I used a newer OS with all the 
source from before plus some needed host related patches taken from the 
gcc archives and verified the code byte for byte. The result was 
successful and I suspect flying over you from time to time.

>>> Prominent example from history: The zlib incident ca. 15 years ago.
>>> Then, zlib was suffering from security senitive bugs, which had gone
>>> through unnoticed for a long time. Tnanks to the fact everybody used
>>> statically linked zlibs, then, when these issues were finially noticed,
>>> fixing this issue was very tedious for OS vendors and product vendors.
>>
>> I agree this is a good fit for a host and a distribution how-ever the
>> security of a host does not effect the instructions the compiler creates
>> given the RTEMS source code.
>
> Sure, the RTEMS toolchains are not a web-server nor web-browser, so the
> security concerns are very low.
>
> Nevertheless ... you never know. E.g. libc may invoke net-worked
> services underneath and a memory leak in a statically linked library be
> exploited ...
>

Sure. I am only talking about these extra packages that gcc has started 
to use and integrate. Security of development hosts is something RTEMS 
users need to deal with themselves and take responsibility for.

>>> I don't know if Apple or a trustworthy 3rd
>>> party supplies binaries for gmp, mpfr, mpc on MacOS (I would be
>>> surprised if this does not apply), you're likely better off using these
>>> libraries and to apply shared linkage and not to use your "homegrown"
>>> static versions.
>>
>> I have no idea what Apple provide on MacOS, or what is provided on
>> FreeBSD (I use both) nor do I need to be concerned.
>
> Well, that's the difference in your way of thinking and mine.
>
> I trust more in OS vendors' dedicated package maintainers who are
> supposed to be deeply familiar with packages and supposed to take more
> care about packages, than I ever could myself.
>

Yes agree for the host OS. I hope that the tools we release are tested 
and stable. I am only talking about gcc and the dependent libraries it 
now includes and making sure these do not vary even if the OS vendor has 
decided they need to for reasons specific to the host os.

Chris