N2X, GR740 and coverage: The impact of a linkcmds patch from 2014 to gcov_init calls

Tue Feb 27 16:30:05 UTC 2018

Hi RTEMS developers.

What follows is a detailed bug report - and a suggested fix - regarding 
the impact on calls to "__gcov_init" from a patch made to sparc's 
linkscmds file back in 2014.

The executive summary of the problem:  In the process of running some 
coverage tests on a GR740 board for the qualified math library for space, 
it became clear that even though the code was compiled with "
-ftest-coverage" and "-fprofile-arcs", there were no calls made upon 
startup towards "__gcov_init". The coverage checks depended on that, 
because these calls pass pointers to structures that hold key information 
related to coverage.

In similar experiments done with N2X boards and older version of the tools 
(gcc 4.8 and RTEMS4.11), these automated calls to "__gcov_init" were 
properly made during startup, as expected. 

In attempting to debug this failure, the current structure of the code was 
then followed.  In our - admittedly - limited understanding, it appears 
that the compiler-generated "stubs" that call "__gcov_init" are in the end 
meant to exist between the __CTOR_LIST__ and __CTOR_END__ symbols in the 
final binary - in the form of function pointers (i.e. their addresses are 
meant to exist in this "list"). 

To give a more specific example, on an N2X board that was running the 
binaries built with the older RTEMS toolchain, the CTOR_LIST ended up with 
content like this:

$ sparc-rtems4.11-objdump -x -d -S ../bin.debug.leon3.OAR/fputest
...
4001d100 <__CTOR_LIST__>:
4001d100:       ff ff ff ff 40 00 13 ac 40 00 14 54 40 00 18 54 
.... at ...@..T at ..T
4001d110:       40 00 1b 38 40 00 22 4c 40 00 25 c0 @..8 at ."L at .%.
4001d11c <__CTOR_END__>:
4001d11c:       00 00 00 00                                         ....

Notice that the list start with 0xffffffff (i.e. -1) and proceeds with 
400013ac, 40001454, etc. 
And here's the coverage-generated "stub" that lives at 400013ac - and will 
call "__gcov_init":

...
400013ac:       9d e3 bf a0     save  %sp, -96, %sp
400013b0:       03 10 00 7a     sethi  %hi(0x4001e800), %g1
400013b4:       90 10 60 50     or  %g1, 0x50, %o0      ! 4001e850 
<__gcov_.Init+0x18>
400013b8:       40 00 03 ad     call  4000226c <__gcov_init>
400013bc:       01 00 00 00     nop
400013c0:       81 e8 00 00     restore
400013c4:       81 c3 e0 08     retl
400013c8:       01 00 00 00     nop

When we built the same code with either (a) a very recent toolchain from 
the mainline (RTEMS 5, RSB 703532cb04c6990fb21e97cb7347a16e9df11108 ) or 
(b) using the RCC1.3rc3 toolchain from Gaisler - in both cases targetting 
the GR740 - the result was different:

000242c0 <__CTOR_LIST__>:
   242c0:       ff ff ff ff                                         ....

000242c4 <__CTOR_END__>:
   242c4:       00 00 00 00 00 00 14 60 00 00 15 24 00 00 19 cc 
.......`...$....
   242d4:       00 00 1d 34 00 00 24 74 00 00 28 08 ...4..$t..(.

It appears that for some reason the list of stubs ends up somehow 
accumulated on the CTOR_END instead of the CTOR_LIST - indeed, 00001460 is 
a coverage "stub" that is meant to call __gcov_init:

00001460 
<_GLOBAL__sub_I_00100_0__Linker_set__Sysinit_rtems_filesystem_initialize>:
    1460:       9d e3 bf a0     save  %sp, -96, %sp
    1464:       03 00 00 96     sethi  %hi(0x25800), %g1
    1468:       90 10 61 20     or  %g1, 0x120, %o0     ! 25920 
<__gcov_.Init+0x18>
    146c:       40 00 04 10     call  24ac <__gcov_init>
    1470:       01 00 00 00     nop
    1474:       81 e8 00 00     restore
    1478:       81 c3 e0 08     retl
    147c:       01 00 00 00     nop

But unfortunately, no-one calls this stub - because the "list processing 
code" remains the same - it starts at CTOR_END-4, and goes back until it 
meets a -1:

grmon2> bp __do_global_ctors_aux
  Software breakpoint 1 at <__do_global_ctors_aux>

grmon2> run

  CPU 0:  breakpoint 1 hit
          0x000241b4: 3b000090  sethi  %hi(0x24000), %i5
  CPU 1:  Power down mode
  CPU 2:  Power down mode
  CPU 3:  Power down mode

grmon2> disassemble 0x000241b4
   241b4:       3b 00 00 90     sethi  %hi(0x24000), %i5
   241b8:       ba 17 62 c4     or  %i5, 0x2c4, %i5     ! 242c4 
<__CTOR_END__>
   241bc:       c2 07 7f fc     ld  [ %i5 + -4 ], %g1
   241c0:       80 a0 7f ff     cmp  %g1, -1
   241c4:       02 80 00 08     be  241e4 <__do_global_ctors_aux+0x34>
   241c8:       ba 07 7f fc     add  %i5, -4, %i5
   241cc:       9f c0 40 00     call  %g1
   241d0:       ba 07 7f fc     add  %i5, -4, %i5
   241d4:       c2 07 40 00     ld  [ %i5 ], %g1
   241d8:       80 a0 7f ff     cmp  %g1, -1
   241dc:       12 bf ff fc     bne  241cc <__do_global_ctors_aux+0x1c>
   241e0:       01 00 00 00     nop
   241e4:       81 c7 e0 08     ret
   241e8:       81 e8 00 00     restore

This code will clearly fail to call our coverage stubs -  it will 
immediately "see" the -1, and return.

After quite a lot of hunting, we traced this failure to a patch in the 
"linkcmds.base" - done back in 2014 ( commit 95cb09ed746 ) :

This commit changed "c/src/lib/libbsp/sparc/shared/startup/linkcmds.base" 
from this:

    KEEP (*(SORT(.ctors.*)))
    KEEP (*(.ctors))
    KEEP (*crtbegin.o(.dtors))
    KEEP (*crtbegin?.o(.dtors))
    KEEP (*(EXCLUDE_FILE (*crtend.o *crtend?.o ) .dtors))
    KEEP (*(SORT(.dtors.*)))
    KEEP (*(.dtors))

...to this:

    KEEP (*(EXCLUDE_FILE (*crtend.o *crtend?.o ) .ctors))
    KEEP (*(SORT(.ctors*)))
    KEEP (*crtbegin.o(.dtors))
    KEEP (*crtbegin?.o(.dtors))
    KEEP (*(EXCLUDE_FILE (*crtend.o *crtend?.o ) .dtors))
    KEEP (*(SORT(.dtors*)))

Reverting this change, the gcov stubs revert back to their proper place, 
and __gcov_inits are called properly.

The reason I tried this patch, was because I noticed that the coverage 
stubs are no longer placed in ".ctor" sections - they are instead placed 
by the newer compilers at ".ctor.NUMBER" sections (with NUMBER set to 
65435 in my case - but I am guessing this will change from invocation to 
invocation). 

The comment provided by Joel back in 2014, only indicated that this patch 
was done to address C++ concerns:

commit 95cb09ed746e7daeca2158c7ecdf0249cfcbc5c8
Author: Joel Sherrill <joel.sherrill at oarcorp.com>
Date:   Wed Apr 2 11:39:20 2014 -0500

    sparc/shared/.../linkcmds.base: Correct C++ support
    Add KEEP() for .eh_frame*, .ctor*, and .dtor*.

...but it appears that it also breaks the proper operation of coverage 
stubs.

Can you please confirm whether this is indeed a bug - and/or indicate 
whether there's some other workaround that can be used besides "hacking" 
the linkcmds this way, to make coverage functionality operate properly?

Thanks in advance,
Thanassis.

Thanassis Tsiodras
Real-time Embedded Software Engineer 
System, Software and Technology Department

ESTEC
Keplerlaan 1, PO Box 299
NL-2200 AG Noordwijk, The Netherlands
Thanassis.Tsiodras at esa.int | www.esa.int
T +31 71 565 5332
This message and any attachments are intended for the use of the addressee or addressees only.
The unauthorised disclosure, use, dissemination or copying (either in whole or in part) of its
content is not permitted.
If you received this message in error, please notify the sender and delete it from your system.
Emails can be altered and their integrity cannot be guaranteed by the sender.

Please consider the environment before printing this email.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rtems.org/pipermail/devel/attachments/20180227/3d4e4e93/attachment-0001.html>