New validation test suites
Joel Sherrill
joel at rtems.org
Wed Dec 15 16:01:59 UTC 2021
I waited to reply because Chris got most of my questions in his first
response. :)
On Tue, Dec 14, 2021 at 11:46 PM Chris Johns <chrisj at rtems.org> wrote:
>
> On 14/12/21 6:24 pm, Sebastian Huber wrote:
> > Hello Chris,
> >
> > On 13/12/2021 22:01, Chris Johns wrote:
> >> On 14/12/21 1:53 am, Sebastian Huber wrote:
> >>> Hello,
> >>>
> >>> the ESA activity to pre-qualify parts of RTEMS according to ECSS requirements is
> >>> nearly complete. There is a short presentation available here:
> >>>
> >>> https://indico.esa.int/event/374/timetable/
> >>
> >> Was the change in memory usage for 4.8 of 23812 bytes to 68896 explained?
> >
> > The foot print increase has mainly five reasons:
> >
> > * General GCC code generation
> >
> > * Chip errata workarounds done by GCC
> >
> > * More features used from RTEMS (for example uniprocessor synchronization done
> > via task priorities vs. mutex synchronization)
> >
> > * SMP support of RTEMS
> >
> > * New RTEMS features such as transitive priority inheritance
> >
>
> Thanks.
If the 4.8 size was based on the Edisoft version, it was also
stripped of functionality even within methods in RTEMS itself.
Combine that with uniprocessor vs SMP and more robust
algorithms in the current RTEMS and it creeps up.
It would be interesting to see a straight uniprocessor comparison
but even that would not have feature parity.
>
> >>> We finished the specification of the pre-qualified RTEMS feature set. The
> >>> specification is available in an RTEMS Project repository:
> >>>
> >>> https://git.rtems.org/rtems-central/tree/spec
> >>
> >> I had a quick look. Is there a more user friendly view of this data?
> >>
> >> I think the term "specification" is a little bit misleading because the data
> >> files are not easily read by a person. I understand this is the specification
> >> data set however it is not what I am traditionally use to seeing.
> >
> > You can use the "./specview.py" script to get views of the specification. For
> > example, this command displays the transition map for the rtems_signal_send()
> > directive:
>
> Is specview.py part of rtems.git?
>
> If not part of rtems.git how much data is there for all the output? That is it
> is generated and held in the repo with the tests?
>
> >
> > ./specview.py --filter=action-table /rtems/signal/req/send
> > .. table::
> > :class: longtable
> >
> > ===== ========== ===== ======= ======= ======== ====== ====== =============
> > =========
> > Entry Descriptor Task Set Handler ASR Nested Status Handler
> > Recursive
> > ===== ========== ===== ======= ======= ======== ====== ====== =============
> > =========
> > 0 0 NoObj Zero Invalid Enabled Yes InvNum NoCall No
> > 1 0 NoObj Zero Invalid Enabled No InvNum NoCall No
> > 2 0 NoObj Zero Invalid Disabled Yes InvNum NoCall No
> > 3 0 NoObj Zero Invalid Disabled No InvNum NoCall No
> > 4 0 NoObj Zero Valid Enabled Yes InvNum NoCall No
> > 5 0 NoObj Zero Valid Enabled No InvNum NoCall No
> > 6 0 NoObj Zero Valid Disabled Yes InvNum NoCall No
> > 7 0 NoObj Zero Valid Disabled No InvNum NoCall No
> > 8 1 NoObj NonZero Invalid Enabled Yes InvId NoCall No
> > 9 1 NoObj NonZero Invalid Enabled No InvId NoCall No
> > 10 1 NoObj NonZero Invalid Disabled Yes InvId NoCall No
> > 11 1 NoObj NonZero Invalid Disabled No InvId NoCall No
> > 12 1 NoObj NonZero Valid Enabled Yes InvId NoCall No
> > 13 1 NoObj NonZero Valid Enabled No InvId NoCall No
> > 14 1 NoObj NonZero Valid Disabled Yes InvId NoCall No
> > 15 1 NoObj NonZero Valid Disabled No InvId NoCall No
> > 16 0 Self Zero Invalid Enabled Yes InvNum NoCall No
> > 17 0 Self Zero Invalid Enabled No InvNum NoCall No
> > 18 0 Self Zero Invalid Disabled Yes InvNum NoCall No
> > 19 0 Self Zero Invalid Disabled No InvNum NoCall No
> > 20 0 Self Zero Valid Enabled Yes InvNum NoCall No
> > 21 0 Self Zero Valid Enabled No InvNum NoCall No
> > 22 0 Self Zero Valid Disabled Yes InvNum NoCall No
> > 23 0 Self Zero Valid Disabled No InvNum NoCall No
> > 24 2 Self NonZero Invalid Enabled Yes NotDef NoCall No
> > 25 2 Self NonZero Invalid Enabled No NotDef NoCall No
> > 26 2 Self NonZero Invalid Disabled Yes NotDef NoCall No
> > 27 2 Self NonZero Invalid Disabled No NotDef NoCall No
> > 28 6 Self NonZero Valid Enabled Yes Ok DuringSend Yes
> > 29 4 Self NonZero Valid Enabled No Ok DuringSend No
> > 30 3 Self NonZero Valid Disabled Yes Ok AfterEnable No
> > 31 3 Self NonZero Valid Disabled No Ok AfterEnable No
> > 32 0 Other Zero Invalid Enabled Yes InvNum NoCall No
> > 33 0 Other Zero Invalid Enabled No InvNum NoCall No
> > 34 0 Other Zero Invalid Disabled Yes InvNum NoCall No
> > 35 0 Other Zero Invalid Disabled No InvNum NoCall No
> > 36 0 Other Zero Valid Enabled Yes InvNum NoCall No
> > 37 0 Other Zero Valid Enabled No InvNum NoCall No
> > 38 0 Other Zero Valid Disabled Yes InvNum NoCall No
> > 39 0 Other Zero Valid Disabled No InvNum NoCall No
> > 40 2 Other NonZero Invalid Enabled Yes NotDef NoCall No
> > 41 2 Other NonZero Invalid Enabled No NotDef NoCall No
> > 42 2 Other NonZero Invalid Disabled Yes NotDef NoCall No
> > 43 2 Other NonZero Invalid Disabled No NotDef NoCall No
> > 44 7 Other NonZero Valid Enabled Yes Ok AfterDispatch Yes
> > 45 5 Other NonZero Valid Enabled No Ok AfterDispatch No
> > 46 3 Other NonZero Valid Disabled Yes Ok AfterEnable No
> > 47 3 Other NonZero Valid Disabled No Ok AfterEnable No
> > ===== ========== ===== ======= ======= ======== ====== ====== =============
> > =========
>
> Is the
> >
> > Here the same information in a different view, for each post-condition set the
> > pre-condition sets are displayed:
> >
> > ./specview.py --filter=action-list /rtems/signal/req/send
> >
> > Status = Ok, Handler = DuringSend, Recursive = Yes
> >
> > * Task = Self, Set = NonZero, Handler = Valid, ASR = Enabled, Nested = Yes
> >
> > Status = Ok, Handler = DuringSend, Recursive = No
> >
> > * Task = Self, Set = NonZero, Handler = Valid, ASR = Enabled, Nested = No
> >
> > Status = Ok, Handler = AfterDispatch, Recursive = Yes
> >
> > * Task = Other, Set = NonZero, Handler = Valid, ASR = Enabled, Nested = Yes
> >
> > Status = Ok, Handler = AfterDispatch, Recursive = No
> >
> > * Task = Other, Set = NonZero, Handler = Valid, ASR = Enabled, Nested = No
> >
> > Status = Ok, Handler = AfterEnable, Recursive = No
> >
> > * Task = { Self, Other }, Set = NonZero, Handler = Valid, ASR = Disabled,
> > Nested = { Yes, No }
> >
> > Status = InvId, Handler = NoCall, Recursive = No
> >
> > * Task = NoObj, Set = NonZero, Handler = { Invalid, Valid }, ASR = {
> > Enabled, Disabled }, Nested = { Yes, No }
> >
> > Status = NotDef, Handler = NoCall, Recursive = No
> >
> > * Task = { Self, Other }, Set = NonZero, Handler = Invalid, ASR = { Enabled,
> > Disabled }, Nested = { Yes, No }
> >
> > Status = InvNum, Handler = NoCall, Recursive = No
> >
> > * Task = { NoObj, Self, Other }, Set = Zero, Handler = { Invalid, Valid },
> > ASR = { Enabled, Disabled }, Nested = { Yes, No }
> >
> >
> >>
> >>> The validation tests are generated from the specification using the
> >>> "./spec2modules.py" script and end up in the RTEMS sources of a Git submodule. I
> >>> think the specification and the generation tool is now sufficiently stable so
> >>> that the validation test code can be integrated in the RTEMS master branch. The
> >>> patch set is too big for the mailing list, so you can review it here:
> >>>
> >>> https://git.rtems.org/sebh/rtems.git/log/?h=validation
> >>
> >> The link failed.
> >
> > Yes, viewing my personal repository no longer works. I am not sure if this is a
> > temporary issue. This is why I added the github link.
>
> It seems to have been temporary. It is back again.
>
> >
> >>
> >>> https://github.com/sebhub/rtems/tree/validation
> >>
> >> The header in a test says the regeneration instructions are in the engineering
> >> manual but I could not quickly find them?
> >
> > https://docs.rtems.org/branches/master/eng/req/howto.html#generate-content-after-changes
> >
> >
> > In an earlier version of the header, we had a link which you didn't like:
>
> If I need to look at the formatting rules the heading "Software Development
> Management" is easy to see and then a click on "Coding Standards" gives me what
> I am looking for.
>
> To generate these headers I click on "Software Requirements Engineering" and
> then do I just guess until I find it in the "How To" section? I am actually
> asking this be sorted out so it is not left hanging and we are not left guessing
> what to do. If it can be rearrange into something meaningful it would help. :)
Have you had anyone not directly involved in the generation process
try this from the instructions? That always seems to be when the
holes show up.
> > commit a6689fb147649a29ff5533cec8a53635e2c95ec6
> > Author: Sebastian Huber <sebastian.huber at embedded-brains.de>
> > Date: Fri Jan 22 16:01:46 2021 +0100
> >
> > Improve file header comment in generated files
> >
> > diff --git a/cpukit/include/rtems/rtems/types.h
> > b/cpukit/include/rtems/rtems/types.h
> > index 32b45113c8..a058eedea1 100644
> > --- a/cpukit/include/rtems/rtems/types.h
> > +++ b/cpukit/include/rtems/rtems/types.h
> > @@ -38,11 +38,15 @@
> > * worded better please post a report or patch to an RTEMS mailing list
> > * or raise a bug report:
> > *
> > - * https://docs.rtems.org/branches/master/user/support/bugs.html
> > + * https://www.rtems.org/bugs.html
> > *
> > - * For information on updating and regenerating please refer to:
> > + * For information on updating and regenerating please refer to the How-To
> > + * section in the Software Requirements Engineering chapter of the
> > + * RTEMS Software Engineering manual. The manual is provided as a part of
> > + * a release. For development sources please refer to the online
> > + * documentation at:
> > *
> > - * https://docs.rtems.org/branches/master/eng/req/howto.html
> > + * https://docs.rtems.org
> > */
> >
> > /* Generated from spec:/rtems/type/if/header */
> >
>
> I am fine with the header, as we know links to the documentation is really hard
> long term.
Early on, we discussed some annotation maybe like SPDX to
note that a file had higher rigor with requirements and tests.
Is this Generated note implying this? Is it sufficient to note that
if you touch this, we would really like you to address the full
chain on this?
> >>> The patch set is organized so that general support code for the validation tests
> >>> is added first and then there are commits for each pre-qualified Classic API
> >>> Manager or subsystem.
> >>>
> >>> I did build all BSPs with the patch set. The validation tests use only
> >>> statically allocated resources.
> >>
> >> Are the validation tests compatible with rtems-test?
> >
> > Yes.
>
> Great.
>
> >> How many test executables does this add to the testsuite?
> >
> > If RTEMS_SMP is disabled, then there are 19 new executables in the patch set. If
> > RTEMS_SMP is enable, then there are 28 new executables.
>
> Nice.
>
> >> What hardware have the validation tests been run on? Any tier 1 archs?
> >
> > I tested with the sparc/leon3 BSPs and the arm/realview_pbx_a9_qemu.
>
> Is the leon3 tested on hardware or simulation?
>
> > You need a
> > full implementation of the new Interrupt Manager directives and a working Cache
> > Manager implementation.
>
> Is this documented?
>
> I am sorry I do not know the list of archs and bsps that support the new
> interrupt manager directives. Maybe it would be good to list them?
Yes. And since you don't know the bsps which don't have the support,
what will the tests do?
> > I noticed an issue with the thread restart on aarch64/a53_lp64_qemu.
I passed this along to Kinsey but it would be helpful if you could start
a thread/ticket about this so he knows what to look at.
> >
> > On powerpc/psim there is an issue in one test case, due to:
> >
> > #define CPU_ALL_TASKS_ARE_FP CPU_HARDWARE_FP
>
> Sorry, I am not following what the issue is? Does this effect all PPC BSPS?
>
> > Another issue is that the tm27 interrupt must be independent of the clock driver
> > interrupt. This is not the case for powerpc/psim.
This is a change in the design from the original assumptions over 30 years
ago. It will break more than psim. Many boards do not have a second source
for a counter/timer and it was assumed that the clock hardware could be
used since the tmtests do not configure a clock driver.
What exactly broke this?
> >
> > There is definitely some work left to cover all edge cases. Some tests are quite
> > complicated.
>
> Sure. I would like to understand the effects this has?
>
> >> Is there anything that interprets the new test output format? It looks like lots
> >> of great info but a little difficult to read.
> >
> > EDISOFT worked on a test report generator, however, it is not yet in a
> > reviewable state.
>
> OK. I think something that handles this data would be good to have.
>
> >>> Some low memory targets are not able to link all test suites.
> >>
> >> Are these excluded in the normal way?
> >
> > Yes:
> >
> > https://github.com/sebhub/rtems/commit/2feedb9e7805f483e35b7914b9bd7c808e31a8b4#diff-bf7b325198d4133c9e52f49d77447905ecd3c6ac5159d5cd85e7efeffa6da47e
>
> Nice.
>
> This looks really good and I like what it provides. It is a fantastic baseline
> of the interfaces we have however to be useful long term it needs to generate
> regression notices and so that means we need to know and understand the
> baselines this brings us now and for which arches.
Somewhere in this discussion was mention of coverage results.
It was mentioned that branches were covered but I would appreciate
that being clarified. Is this at the source level or assembly level?
How is this done? Is the procedure documented?
The coverage reports from rtems-test/covoar are at the assembly
language level and report on branch statements being taken and
not taken. This is intended to help us exceed MCDC testing.
I know you posted some gcov support patches earlier so am guessing
that you used that. Some insight would be great.
Thanks.
--joel
>
> Chris
> _______________________________________________
> devel mailing list
> devel at rtems.org
> http://lists.rtems.org/mailman/listinfo/devel
More information about the devel
mailing list