Fwd: [PATCH 2/4] covoar: Use DWARF to map addresses to source files and lines.
Joel Sherrill
joel at rtems.org
Mon May 7 19:36:12 UTC 2018
On Mon, May 7, 2018 at 2:16 PM, Cillian O'Donnell <cpodonnell8 at gmail.com>
wrote:
> Heres the gdb backtrace. rld-dwarf.cpp rld-path showing up near the
> bottom. I don't have valgrind, I'll get it now and see what I find there.
>
> (gdb) bt
> #0 0x00007ffff74aa428 in __GI_raise (sig=sig at entry=6)
> at ../sysdeps/unix/sysv/linux/raise.c:54
> #1 0x00007ffff74ac02a in __GI_abort () at abort.c:89
> #2 0x00007ffff74ec7ea in __libc_message (do_abort=do_abort at entry=2,
> fmt=fmt at entry=0x7ffff7605ed8 "*** Error in `%s': %s: 0x%s ***\n")
> at ../sysdeps/posix/libc_fatal.c:175
> #3 0x00007ffff74f537a in malloc_printerr (ar_ptr=<optimized out>,
> ptr=<optimized out>, str=0x7ffff7605f50 "free(): invalid next size
> (fast)",
> action=3) at malloc.c:5006
> #4 _int_free (av=<optimized out>, p=<optimized out>, have_lock=0) at
> malloc.c:3867
> #5 0x00007ffff74f953c in __GI___libc_free (mem=<optimized out>) at
> malloc.c:2968
> #6 0x000000000043fcee in __gnu_cxx::new_allocator<std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > >::deallocate
> (this=0x7fffffffc730,
> __p=<optimized out>) at /usr/include/c++/5/ext/new_allocator.h:110
> #7 std::allocator_traits<std::allocator<std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > > >::deallocate (__a=...,
> __n=<optimized out>, __p=<optimized out>)
> at /usr/include/c++/5/bits/alloc_traits.h:517
> #8 std::_Vector_base<std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> >,
> std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > > >::_M_deallocate (this=0x7fffffffc730,
> __n=<optimized out>, __p=<optimized out>)
> at /usr/include/c++/5/bits/stl_vector.h:178
> #9 std::_Vector_base<std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> >,
> std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > > >::~_Vector_base (this=0x7fffffffc730,
> __in_chrg=<optimized out>) at /usr/include/c++/5/bits/stl_vector.h:160
> #10 std::vector<std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > > >::~vector
> (this=0x7fffffffc730,
> __in_chrg=<optimized out>) at /usr/include/c++/5/bits/stl_vector.h:425
> #11 rld::path::path_abs (path=...) at ../rtemstoolkit/rld-path.cpp:132
> #12 0x000000000042bc4a in rld::dwarf::compilation_unit::compilation_unit (
> this=0x7fffffffca30, debug=..., die=..., offset=<optimized out>)
> at ../rtemstoolkit/rld-dwarf.cpp:486
> #13 0x000000000042d205 in rld::dwarf::file::load_debug (this=this at entry
> =0x6bc568)
> at ../rtemstoolkit/rld-dwarf.cpp:758
> #14 0x000000000040dec5 in Coverage::ExecutableInfo::ExecutableInfo
> (this=0x6bc3c0,
> theExecutableName=<optimized out>, theLibraryName=0x0)
> at ../tester/covoar/ExecutableInfo.cc:33
> #15 0x00000000004054b6 in main (argc=<optimized out>, argv=<optimized out>)
> at ../tester/covoar/covoar.cc:330
>
OK. We will have to help Chris get to the point he can run covoar. We can
see at #11-#15
that this occurs processing the DWARF.
That leaves us in the following state:
+ Vijay - keep working without this patch series :)
+ Cillian/Vijay - Chris will need to have a coverage run to test this.
+ Cillian/Vijay - I also need to get to where I am running your version.
+ My patch will let this work enough for Chris to debug this failure.
+ Very near future, we need rld file dirname() and eliminate use of
libgen.h completely
in covoar. If we have coding conventions for native code written
somewhere, advice
on this should be added. [I think our coding conventions don't address
native issues.]
Unless someone sees what went wrong, I think this one is Chris' and we wait
until
he gets back from his short holiday.
--joel
>
>
> On 7 May 2018 at 20:03, Joel Sherrill <joel at rtems.org> wrote:
>
>>
>>
>> On Mon, May 7, 2018 at 1:59 PM, Cillian O'Donnell <cpodonnell8 at gmail.com>
>> wrote:
>>
>>>
>>>
>>> On 7 May 2018 at 18:56, Joel Sherrill <joel at rtems.org> wrote:
>>>
>>>> Hi
>>>>
>>>> I have attached a workaround. It seems that libgen.h has this:
>>>>
>>>> ========================================================
>>>> /* Return final component of PATH.
>>>>
>>>> This is the weird XPG version of this function. It sometimes will
>>>> modify its argument. Therefore we normally use the GNU version (in
>>>> <string.h>) and only if this header is included make the XPG
>>>> version available under the real name. */
>>>> extern char *__xpg_basename (char *__path) __THROW;
>>>> #define basename __xpg_basename
>>>> ========================================================
>>>>
>>>> Chris has used basename as a method name and even though that should
>>>> be perfectly acceptable, the macro above gets expanded, the name
>>>> gets changed (in some places) to rld::path::__xpg_basename()
>>>>
>>>> r.lowSourceLine = rld::path::basename (location);
>>>>
>>>> IMO the fix is to add dirname to rld-files, use rld basename and dirname
>>>> exclusively, and avoid libgen.h at all costs.
>>>>
>>>> I didn't do that much work. I got lucky in a couple of files by
>>>> removing the
>>>> include of libgen.h since it wasn't needed but I had to leave it in one
>>>> place.
>>>>
>>>> $ grep dirname *.cc
>>>> GcovData.cc: dirname( path );
>>>>
>>>> Hopefully this lets you all proceed with Chris' patches and my slight
>>>> hack
>>>> in place.
>>>>
>>>
>>> Thanks Joel! The good news is that's building now with only a couple of
>>> warnings.
>>>
>>> The bad news is..
>>>
>>> cpod at cpod ~/development/rtems/leon3 $ covoar -S
>>> /home/cpod/development/rtems/test/rtems-tools/tester/rtems/t
>>> esting/coverage/score-symbols.ini -O /home/cpod/coverage_test/test/score
>>> -E/home/cpod/development/rtems/test/rtems-tools/tester/rtems
>>> /testing/coverage/Explanations.txt -p RTEMS-5 -v
>>> sparc-rtems5/c/leon3/testsuites/samples/hello/hello.exe
>>> *** Error in `covoar': free(): invalid next size (fast):
>>> 0x0000000001000360 ***
>>> ======= Backtrace: =========
>>> /lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7fcd0b3477e5]
>>> /lib/x86_64-linux-gnu/libc.so.6(+0x8037a)[0x7fcd0b35037a]
>>> /lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7fcd0b35453c]
>>> covoar[0x43fcee]
>>> covoar[0x42bc4a]
>>> covoar[0x42d205]
>>> covoar[0x40dec5]
>>> covoar[0x4054b6]
>>> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7fcd0b2f0830]
>>> covoar[0x4073e9]
>>> ======= Memory map: ========
>>> 00400000-004a8000 r-xp 00000000 08:07 3276534
>>> /home/cpod/development/rtems/test/rtems-tools/build/tester/covoar/covoar
>>> 006a7000-006a8000 r--p 000a7000 08:07 3276534
>>> /home/cpod/development/rtems/test/rtems-tools/build/tester/covoar/covoar
>>> 006a8000-006a9000 rw-p 000a8000 08:07 3276534
>>> /home/cpod/development/rtems/test/rtems-tools/build/tester/covoar/covoar
>>> 006a9000-006aa000 rw-p 00000000 00:00 0
>>> 00c6a000-01022000 rw-p 00000000 00:00 0
>>> [heap]
>>> 7fcd04000000-7fcd04021000 rw-p 00000000 00:00 0
>>> 7fcd04021000-7fcd08000000 ---p 00000000 00:00 0
>>> 7fcd0a9ae000-7fcd0ac36000 rw-p 00000000 00:00 0
>>> 7fcd0ac36000-7fcd0afc7000 r--p 00000000 08:07 2759194
>>> /home/cpod/development/rtems/leon3/sparc-rtems5/c/leon3/test
>>> suites/samples/hello/hello.exe
>>> 7fcd0afc7000-7fcd0b0cf000 r-xp 00000000 08:07 1838070
>>> /lib/x86_64-linux-gnu/libm-2.23.so
>>> 7fcd0b0cf000-7fcd0b2ce000 ---p 00108000 08:07 1838070
>>> /lib/x86_64-linux-gnu/libm-2.23.so
>>> 7fcd0b2ce000-7fcd0b2cf000 r--p 00107000 08:07 1838070
>>> /lib/x86_64-linux-gnu/libm-2.23.so
>>> 7fcd0b2cf000-7fcd0b2d0000 rw-p 00108000 08:07 1838070
>>> /lib/x86_64-linux-gnu/libm-2.23.so
>>> 7fcd0b2d0000-7fcd0b490000 r-xp 00000000 08:07 1842682
>>> /lib/x86_64-linux-gnu/libc-2.23.so
>>> 7fcd0b490000-7fcd0b690000 ---p 001c0000 08:07 1842682
>>> /lib/x86_64-linux-gnu/libc-2.23.so
>>> 7fcd0b690000-7fcd0b694000 r--p 001c0000 08:07 1842682
>>> /lib/x86_64-linux-gnu/libc-2.23.so
>>> 7fcd0b694000-7fcd0b696000 rw-p 001c4000 08:07 1842682
>>> /lib/x86_64-linux-gnu/libc-2.23.so
>>> 7fcd0b696000-7fcd0b69a000 rw-p 00000000 00:00 0
>>> 7fcd0b69a000-7fcd0b6b0000 r-xp 00000000 08:07 1836727
>>> /lib/x86_64-linux-gnu/libgcc_s.so.1
>>> 7fcd0b6b0000-7fcd0b8af000 ---p 00016000 08:07 1836727
>>> /lib/x86_64-linux-gnu/libgcc_s.so.1
>>> 7fcd0b8af000-7fcd0b8b0000 rw-p 00015000 08:07 1836727
>>> /lib/x86_64-linux-gnu/libgcc_s.so.1
>>> 7fcd0b8b0000-7fcd0ba22000 r-xp 00000000 08:07 1049309
>>> /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
>>> 7fcd0ba22000-7fcd0bc22000 ---p 00172000 08:07 1049309
>>> /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
>>> 7fcd0bc22000-7fcd0bc2c000 r--p 00172000 08:07 1049309
>>> /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
>>> 7fcd0bc2c000-7fcd0bc2e000 rw-p 0017c000 08:07 1049309
>>> /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
>>> 7fcd0bc2e000-7fcd0bc32000 rw-p 00000000 00:00 0
>>> 7fcd0bc32000-7fcd0bc58000 r-xp 00000000 08:07 1838072
>>> /lib/x86_64-linux-gnu/ld-2.23.so
>>> 7fcd0bd72000-7fcd0be30000 rw-p 00000000 00:00 0
>>> 7fcd0be56000-7fcd0be57000 rw-p 00000000 00:00 0
>>> 7fcd0be57000-7fcd0be58000 r--p 00025000 08:07 1838072
>>> /lib/x86_64-linux-gnu/ld-2.23.so
>>> 7fcd0be58000-7fcd0be59000 rw-p 00026000 08:07 1838072
>>> /lib/x86_64-linux-gnu/ld-2.23.so
>>> 7fcd0be59000-7fcd0be5a000 rw-p 00000000 00:00 0
>>> 7ffc5053b000-7ffc5055d000 rw-p 00000000 00:00 0
>>> [stack]
>>> 7ffc505d5000-7ffc505d7000 r--p 00000000 00:00 0
>>> [vvar]
>>> 7ffc505d7000-7ffc505d9000 r-xp 00000000 00:00 0
>>> [vdso]
>>> ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00
>>> 0 [vsyscall]
>>> Aborted
>>>
>>
>> OK. Sadly this is progress. :)
>>
>> Can you get a real backtrace in gdb?
>>
>> Or run it under valgrind? It is either a double free or a write past the
>> end of a buffer. Based on the
>> error mesage from glibc (" *** Error in `covoar': free(): invalid next
>> size (fast): 0x0000000001000360 ***"),
>> I suspect it is a write past the end of a buffer and valgrind would be
>> better to spot that.
>>
>>
>>>
>>>
>>>> Ultimate solution is probably simple. We just need to hear from Chris.
>>>>
>>>> --joel
>>>>
>>>>
>>>> On Mon, May 7, 2018 at 12:42 PM, Joel Sherrill <joel at rtems.org> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Mon, May 7, 2018 at 12:36 PM, Vijay Kumar Banerjee <
>>>>> vijaykumar9597 at gmail.com> wrote:
>>>>>
>>>>>>
>>>>>> On Mon, 7 May 2018, 22:57 Cillian O'Donnell, <cpodonnell8 at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Yeah I'm seeing the same as Joel, at least were further then we were
>>>>>>> :).
>>>>>>>
>>>>>>> I've been mostly working on the rtems-tester support, so just to
>>>>>>> give an update. I spent all day Saturday and today on it. It's taking
>>>>>>> longer than expected, re-orienting myself and deciding what is needed and
>>>>>>> not needed now with the changes in covoar. The problems are not difficult,
>>>>>>> it's just taking some time re-organizing everything. My time is limited
>>>>>>> during the week, so it'll probably be next weekend before I can finish it
>>>>>>> off. Vijay if you have time during the week I could push what I have and
>>>>>>> you could take a stab at some of them and then I could it finish off next
>>>>>>> weekend if you haven't already.
>>>>>>>
>>>>>> please send them, I can look into them for sure.
>>>>>>
>>>>>
>>>>> Keep plugging away.
>>>>>
>>>>> I think there is something wrong in rld related to basename. My first
>>>>> thought was that the undefined symbol was because we didn't include the
>>>>> library. Now I think it is because something got misconfigured on
>>>>> Centos/Fedora/etc. Looking into this.
>>>>>
>>>>>
>>>>>>
>>>>>>> On 7 May 2018 at 15:40, Joel Sherrill <joel at rtems.org> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, May 7, 2018 at 6:01 AM, Vijay Kumar Banerjee <
>>>>>>>> vijaykumar9597 at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> I have added the path to libdwarf here
>>>>>>>>>
>>>>>>>>
>>>>>>>> That worked for me to build but not to link.
>>>>>>>>
>>>>>>>> I'm not sure why this rld symbol turned up missing on Centos 7.
>>>>>>>>
>>>>>>>> ====================================================
>>>>>>>>
>>>>>>>> $ ./waf -v
>>>>>>>> Waf: Entering directory `/home/joel/rtems-work/rtems-tools/build'
>>>>>>>> [228/229] Linking build/tester/covoar/trace-converter
>>>>>>>> 09:38:25 runner ['/usr/bin/g++', 'tester/covoar/TraceConverter.cc.2.o',
>>>>>>>> 'tester/covoar/TraceList.cc.2.o', 'tester/covoar/TraceReaderBase.cc.2.o',
>>>>>>>> 'tester/covoar/TraceReaderLogQEMU.cc.2.o',
>>>>>>>> 'tester/covoar/TraceWriterBase.cc.2.o',
>>>>>>>> 'tester/covoar/TraceWriterQEMU.cc.2.o',
>>>>>>>> '-o/home/joel/rtems-work/rtems-tools/build/tester/covoar/trace-converter',
>>>>>>>> '-Wl,-Bstatic', '-Ltester/covoar', '-Lrtemstoolkit', '-lccovoar', '-lrld',
>>>>>>>> '-ldwarf', '-lelf', '-liberty', '-Wl,-Bdynamic']
>>>>>>>> [229/229] Linking build/tester/covoar/covoar
>>>>>>>> 09:38:25 runner ['/usr/bin/g++', 'tester/covoar/covoar.cc.3.o',
>>>>>>>> '-o/home/joel/rtems-work/rtems-tools/build/tester/covoar/covoar',
>>>>>>>> '-Wl,-Bstatic', '-Ltester/covoar', '-Lrtemstoolkit', '-lccovoar', '-lrld',
>>>>>>>> '-ldwarf', '-lelf', '-liberty', '-Wl,-Bdynamic']
>>>>>>>> tester/covoar/libccovoar.a(DesiredSymbols.cc.1.o): In function
>>>>>>>> `Coverage::DesiredSymbols::determineSourceLines(Coverage::CoverageRanges*,
>>>>>>>> Coverage::ExecutableInfo*)':
>>>>>>>> /home/joel/rtems-work/rtems-tools/build/../tester/covoar/DesiredSymbols.cc:413:
>>>>>>>> undefined reference to `rld::path::__xpg_basename(std::string
>>>>>>>> const&)'
>>>>>>>> /home/joel/rtems-work/rtems-tools/build/../tester/covoar/DesiredSymbols.cc:415:
>>>>>>>> undefined reference to `rld::path::__xpg_basename(std::string
>>>>>>>> const&)'
>>>>>>>> collect2: error: ld returned 1 exit status
>>>>>>>>
>>>>>>>> tester/covoar/libccovoar.a(DesiredSymbols.cc.1.o): In function
>>>>>>>> `Coverage::DesiredSymbols::determineSourceLines(Coverage::CoverageRanges*,
>>>>>>>> Coverage::ExecutableInfo*)':
>>>>>>>> /home/joel/rtems-work/rtems-tools/build/../tester/covoar/DesiredSymbols.cc:413:
>>>>>>>> undefined reference to `rld::path::__xpg_basename(std::string
>>>>>>>> const&)'
>>>>>>>> /home/joel/rtems-work/rtems-tools/build/../tester/covoar/DesiredSymbols.cc:415:
>>>>>>>> undefined reference to `rld::path::__xpg_basename(std::string
>>>>>>>> const&)'
>>>>>>>> collect2: error: ld returned 1 exit status
>>>>>>>>
>>>>>>>> Waf: Leaving directory `/home/joel/rtems-work/rtems-tools/build'
>>>>>>>> Build failed
>>>>>>>> -> task in 'trace-converter' failed with exit status 1:
>>>>>>>> {task 34721616: cxxprogram TraceConverter.cc.2.o,TraceLis
>>>>>>>> t.cc.2.o,TraceReaderBase.cc.2.o,TraceReaderLogQEMU.cc.2.o,Tr
>>>>>>>> aceWriterBase.cc.2.o,TraceWriterQEMU.cc.2.o -> trace-converter}
>>>>>>>> ['/usr/bin/g++', 'tester/covoar/TraceConverter.cc.2.o',
>>>>>>>> 'tester/covoar/TraceList.cc.2.o', 'tester/covoar/TraceReaderBase.cc.2.o',
>>>>>>>> 'tester/covoar/TraceReaderLogQEMU.cc.2.o',
>>>>>>>> 'tester/covoar/TraceWriterBase.cc.2.o',
>>>>>>>> 'tester/covoar/TraceWriterQEMU.cc.2.o',
>>>>>>>> '-o/home/joel/rtems-work/rtems-tools/build/tester/covoar/trace-converter',
>>>>>>>> '-Wl,-Bstatic', '-Ltester/covoar', '-Lrtemstoolkit', '-lccovoar', '-lrld',
>>>>>>>> '-ldwarf', '-lelf', '-liberty', '-Wl,-Bdynamic']
>>>>>>>> -> task in 'covoar' failed with exit status 1:
>>>>>>>> {task 34820256: cxxprogram covoar.cc.3.o -> covoar}
>>>>>>>> ['/usr/bin/g++', 'tester/covoar/covoar.cc.3.o',
>>>>>>>> '-o/home/joel/rtems-work/rtems-tools/build/tester/covoar/covoar',
>>>>>>>> '-Wl,-Bstatic', '-Ltester/covoar', '-Lrtemstoolkit', '-lccovoar', '-lrld',
>>>>>>>> '-ldwarf', '-lelf', '-liberty', '-Wl,-Bdynamic']
>>>>>>>> ====================================================
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---
>>>>>>>>>
>>>>>>>>> diff --git a/tester/covoar/wscript b/tester/covoar/wscript
>>>>>>>>> index 55d5ec9..dd4ad83 100644
>>>>>>>>> --- a/tester/covoar/wscript
>>>>>>>>> +++ b/tester/covoar/wscript
>>>>>>>>> @@ -63,6 +63,7 @@ def build(bld):
>>>>>>>>> rtl_includes = [rtemstoolkit,
>>>>>>>>> rtemstoolkit + '/elftoolchain/libelf',
>>>>>>>>> rtemstoolkit + '/elftoolchain/common',
>>>>>>>>> + rtemstoolkit + '/elftoolchain/libdwarf',
>>>>>>>>> rtemstoolkit + '/libiberty']
>>>>>>>>> if bld.env.DEST_OS == 'win32':
>>>>>>>>> rtl_includes += [rtemstoolkit + '/win32']
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -- vijay
>>>>>>>>>
>>>>>>>>> On 7 May 2018 at 13:30, Vijay Kumar Banerjee <
>>>>>>>>> vijaykumar9597 at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 6 May 2018 at 13:29, Chris Johns <chrisj at rtems.org> wrote:
>>>>>>>>>>
>>>>>>>>>>> On 6/5/18 5:28 pm, Vijay Kumar Banerjee wrote:
>>>>>>>>>>>
>>>>>>>>>>>> On 6 May 2018 at 08:54, Chris Johns <chrisj at rtems.org <mailto:
>>>>>>>>>>>> chrisj at rtems.org>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Would you please try `waf clean build` to see if
>>>>>>>>>>>> rebuilding
>>>>>>>>>>>> everything fixes this?
>>>>>>>>>>>>
>>>>>>>>>>>> still getting the same error .
>>>>>>>>>>>> I'm using g++ 7.3.1 on fedora 27.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> OK
>>>>>>>>>>>
>>>>>>>>>>> If you have changed something in the code, can you please send a
>>>>>>>>>>>> patch for the same ?
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I have not changed anything. I do not have a Linux box to try.
>>>>>>>>>>>
>>>>>>>>>>> I tried to do it afresh as well, it's still failing to build.
>>>>>>>>>> Can someone please try to build it in a linux system ?
>>>>>>>>>>
>>>>>>>>>>> Chris
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> devel mailing list
>>>>>>>>> devel at rtems.org
>>>>>>>>> http://lists.rtems.org/mailman/listinfo/devel
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> devel mailing list
>>>>>>>> devel at rtems.org
>>>>>>>> http://lists.rtems.org/mailman/listinfo/devel
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rtems.org/pipermail/devel/attachments/20180507/d255e5f9/attachment-0002.html>
More information about the devel
mailing list