New validation test suites

Wed Dec 15 16:41:24 UTC 2021

On 15/12/2021 17:01, Joel Sherrill wrote:
> I waited to reply because Chris got most of my questions in his first
> response. :)
> 
> On Tue, Dec 14, 2021 at 11:46 PM Chris Johns <chrisj at rtems.org> wrote:
>>
>> On 14/12/21 6:24 pm, Sebastian Huber wrote:
>>> Hello Chris,
>>>
>>> On 13/12/2021 22:01, Chris Johns wrote:
>>>> On 14/12/21 1:53 am, Sebastian Huber wrote:
>>>>> Hello,
>>>>>
>>>>> the ESA activity to pre-qualify parts of RTEMS according to ECSS requirements is
>>>>> nearly complete. There is a short presentation available here:
>>>>>
>>>>> https://indico.esa.int/event/374/timetable/
>>>>
>>>> Was the change in memory usage for 4.8 of 23812 bytes to 68896 explained?
>>>
>>> The foot print increase has mainly five reasons:
>>>
>>> * General GCC code generation
>>>
>>> * Chip errata workarounds done by GCC
>>>
>>> * More features used from RTEMS (for example uniprocessor synchronization done
>>> via task priorities vs. mutex synchronization)
>>>
>>> * SMP support of RTEMS
>>>
>>> * New RTEMS features such as transitive priority inheritance
>>>
>>
>> Thanks.
> 
> If the 4.8 size was based on the Edisoft version, it was also
> stripped of functionality even within methods in RTEMS itself.
> Combine that with uniprocessor vs SMP and more robust
> algorithms in the current RTEMS and it creeps up.
> 
> It would be interesting to see a straight uniprocessor comparison
> but even that would not have feature parity.

I didn't have time to do a thorough comparison. The above numbers are 
obtain from totally different applications. Anyway, reducing the code 
size would be a difficult task.

[...]
>>>> The header in a test says the regeneration instructions are in the engineering
>>>> manual but I could not quickly find them?
>>>
>>> https://docs.rtems.org/branches/master/eng/req/howto.html#generate-content-after-changes
>>>
>>>
>>> In an earlier version of the header, we had a link which you didn't like:
>>
>> If I need to look at the formatting rules the heading "Software Development
>> Management" is easy to see and then a click on "Coding Standards" gives me what
>> I am looking for.
>>
>> To generate these headers I click on "Software Requirements Engineering" and
>> then do I just guess until I find it in the "How To" section? I am actually
>> asking this be sorted out so it is not left hanging and we are not left guessing
>> what to do. If it can be rearrange into something meaningful it would help. :)
> 
> Have you had anyone not directly involved in the generation process
> try this from the instructions? That always seems to be when the
> holes show up.

Frank was able to specify (and test) the Timer Manager, Clock Manager, 
Rate-Monotonic Manager, and Message Manager. You need a bit of time to 
get accustomed to the work flow. However, the table based specification 
worked really well for the project.

> 
>>> commit a6689fb147649a29ff5533cec8a53635e2c95ec6
>>> Author: Sebastian Huber <sebastian.huber at embedded-brains.de>
>>> Date:   Fri Jan 22 16:01:46 2021 +0100
>>>
>>>      Improve file header comment in generated files
>>>
>>> diff --git a/cpukit/include/rtems/rtems/types.h
>>> b/cpukit/include/rtems/rtems/types.h
>>> index 32b45113c8..a058eedea1 100644
>>> --- a/cpukit/include/rtems/rtems/types.h
>>> +++ b/cpukit/include/rtems/rtems/types.h
>>> @@ -38,11 +38,15 @@
>>>    * worded better please post a report or patch to an RTEMS mailing list
>>>    * or raise a bug report:
>>>    *
>>> - * https://docs.rtems.org/branches/master/user/support/bugs.html
>>> + * https://www.rtems.org/bugs.html
>>>    *
>>> - * For information on updating and regenerating please refer to:
>>> + * For information on updating and regenerating please refer to the How-To
>>> + * section in the Software Requirements Engineering chapter of the
>>> + * RTEMS Software Engineering manual.  The manual is provided as a part of
>>> + * a release.  For development sources please refer to the online
>>> + * documentation at:
>>>    *
>>> - * https://docs.rtems.org/branches/master/eng/req/howto.html
>>> + * https://docs.rtems.org
>>>    */
>>>
>>>   /* Generated from spec:/rtems/type/if/header */
>>>
>>
>> I am fine with the header, as we know links to the documentation is really hard
>> long term.
> 
> Early on, we discussed some annotation maybe like SPDX to
> note that a file had higher rigor with requirements and tests.
> Is this Generated note implying this? Is it sufficient to note that
> if you touch this, we would really like you to address the full
> chain on this?

Yes, this would be great. If you modify API header files through 
specification items, then you get for example an up to date RTEMS 
Classic API Guide automatically.

> 
>>>>> The patch set is organized so that general support code for the validation tests
>>>>> is added first and then there are commits for each pre-qualified Classic API
>>>>> Manager or subsystem.
>>>>>
>>>>> I did build all BSPs with the patch set. The validation tests use only
>>>>> statically allocated resources.
>>>>
>>>> Are the validation tests compatible with rtems-test?
>>>
>>> Yes.
>>
>> Great.
>>
>>>> How many test executables does this add to the testsuite?
>>>
>>> If RTEMS_SMP is disabled, then there are 19 new executables in the patch set. If
>>> RTEMS_SMP is enable, then there are 28 new executables.
>>
>> Nice.
>>
>>>> What hardware have the validation tests been run on? Any tier 1 archs?
>>>
>>> I tested with the sparc/leon3 BSPs and the arm/realview_pbx_a9_qemu.
>>
>> Is the leon3 tested on hardware or simulation?
>>
>>> You need a
>>> full implementation of the new Interrupt Manager directives and a working Cache
>>> Manager implementation.
>>
>> Is this documented?
>>
>> I am sorry I do not know the list of archs and bsps that support the new
>> interrupt manager directives. Maybe it would be good to list them?
> 
> Yes. And since you don't know the bsps which don't have the support,
> what will the tests do?

Some tests fail.

>>> I noticed an issue with the thread restart on aarch64/a53_lp64_qemu.
> 
> I passed this along to Kinsey but it would be helpful if you could start
> a thread/ticket about this so he knows what to look at.
> 
>>>
>>> On powerpc/psim there is an issue in one test case, due to:
>>>
>>> #define CPU_ALL_TASKS_ARE_FP CPU_HARDWARE_FP
>>
>> Sorry, I am not following what the issue is? Does this effect all PPC BSPS?
>>
>>> Another issue is that the tm27 interrupt must be independent of the clock driver
>>> interrupt.  This is not the case for powerpc/psim.
> 
> This is a change in the design from the original assumptions over 30 years
> ago. It will break more than psim. Many boards do not have a second source
> for a counter/timer and it was assumed that the clock hardware could be
> used since the tmtests do not configure a clock driver.
> 
> What exactly broke this?

We don't need a second counter/timer. We just need a software generated 
interrupt. Maybe I can rearrange the test cases so that the tm27 is only 
used if no clock driver is needed.

> 
> 
>>>
>>> There is definitely some work left to cover all edge cases. Some tests are quite
>>> complicated.
>>
>> Sure. I would like to understand the effects this has?
>>
>>>> Is there anything that interprets the new test output format? It looks like lots
>>>> of great info but a little difficult to read.
>>>
>>> EDISOFT worked on a test report generator, however, it is not yet in a
>>> reviewable state.
>>
>> OK. I think something that handles this data would be good to have.
>>
>>>>> Some low memory targets are not able to link all test suites.
>>>>
>>>> Are these excluded in the normal way?
>>>
>>> Yes:
>>>
>>> https://github.com/sebhub/rtems/commit/2feedb9e7805f483e35b7914b9bd7c808e31a8b4#diff-bf7b325198d4133c9e52f49d77447905ecd3c6ac5159d5cd85e7efeffa6da47e
>>
>> Nice.
>>
>> This looks really good and I like what it provides. It is a fantastic baseline
>> of the interfaces we have however to be useful long term it needs to generate
>> regression notices and so that means we need to know and understand the
>> baselines this brings us now and for which arches.
> 
> Somewhere in this discussion was mention of coverage results.
> It was mentioned that branches were covered but I would appreciate
> that being clarified. Is this at the source level or assembly level?
> How is this done? Is the procedure documented?

I still have a long RTEMS integration TODO list. The gcov support is 
just one item. We achieved 100% line and branch coverage at source code 
level (-O0 with gcov) on a system with at least three processors (about 
99.8% on a system with two processors).

> 
> The coverage reports from rtems-test/covoar are at the assembly
> language level and report on branch statements being taken and
> not taken. This is intended to help us exceed MCDC testing.

It would be good to get the coverage data also with -O2 without gcov. So 
far I haven't figured out how to do this with rtems-test/covoar. I think 
-O2 without gcov is the better approach, but ESA wanted -O0 with gcov.

> 
> I know you posted some gcov support patches earlier so am guessing
> that you used that. Some insight would be great.

In general, the gcov support works really nice. After the test run, the 
gcov information is printed out via the serial test interface. On the 
host you can read it and convert it to *.gcda files.

-- 
embedded brains GmbH
Herr Sebastian HUBER
Dornierstr. 4
82178 Puchheim
Germany
email: sebastian.huber at embedded-brains.de
phone: +49-89-18 94 741 - 16
fax:   +49-89-18 94 741 - 08

Registergericht: Amtsgericht München
Registernummer: HRB 157899
Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler
Unsere Datenschutzerklärung finden Sie hier:
https://embedded-brains.de/datenschutzerklaerung/