New validation test suites
Chris Johns
chrisj at rtems.org
Wed Dec 15 05:46:08 UTC 2021
On 14/12/21 6:24 pm, Sebastian Huber wrote:
> Hello Chris,
>
> On 13/12/2021 22:01, Chris Johns wrote:
>> On 14/12/21 1:53 am, Sebastian Huber wrote:
>>> Hello,
>>>
>>> the ESA activity to pre-qualify parts of RTEMS according to ECSS requirements is
>>> nearly complete. There is a short presentation available here:
>>>
>>> https://indico.esa.int/event/374/timetable/
>>
>> Was the change in memory usage for 4.8 of 23812 bytes to 68896 explained?
>
> The foot print increase has mainly five reasons:
>
> * General GCC code generation
>
> * Chip errata workarounds done by GCC
>
> * More features used from RTEMS (for example uniprocessor synchronization done
> via task priorities vs. mutex synchronization)
>
> * SMP support of RTEMS
>
> * New RTEMS features such as transitive priority inheritance
>
Thanks.
>>> We finished the specification of the pre-qualified RTEMS feature set. The
>>> specification is available in an RTEMS Project repository:
>>>
>>> https://git.rtems.org/rtems-central/tree/spec
>>
>> I had a quick look. Is there a more user friendly view of this data?
>>
>> I think the term "specification" is a little bit misleading because the data
>> files are not easily read by a person. I understand this is the specification
>> data set however it is not what I am traditionally use to seeing.
>
> You can use the "./specview.py" script to get views of the specification. For
> example, this command displays the transition map for the rtems_signal_send()
> directive:
Is specview.py part of rtems.git?
If not part of rtems.git how much data is there for all the output? That is it
is generated and held in the repo with the tests?
>
> ./specview.py --filter=action-table /rtems/signal/req/send
> .. table::
> :class: longtable
>
> ===== ========== ===== ======= ======= ======== ====== ====== =============
> =========
> Entry Descriptor Task Set Handler ASR Nested Status Handler
> Recursive
> ===== ========== ===== ======= ======= ======== ====== ====== =============
> =========
> 0 0 NoObj Zero Invalid Enabled Yes InvNum NoCall No
> 1 0 NoObj Zero Invalid Enabled No InvNum NoCall No
> 2 0 NoObj Zero Invalid Disabled Yes InvNum NoCall No
> 3 0 NoObj Zero Invalid Disabled No InvNum NoCall No
> 4 0 NoObj Zero Valid Enabled Yes InvNum NoCall No
> 5 0 NoObj Zero Valid Enabled No InvNum NoCall No
> 6 0 NoObj Zero Valid Disabled Yes InvNum NoCall No
> 7 0 NoObj Zero Valid Disabled No InvNum NoCall No
> 8 1 NoObj NonZero Invalid Enabled Yes InvId NoCall No
> 9 1 NoObj NonZero Invalid Enabled No InvId NoCall No
> 10 1 NoObj NonZero Invalid Disabled Yes InvId NoCall No
> 11 1 NoObj NonZero Invalid Disabled No InvId NoCall No
> 12 1 NoObj NonZero Valid Enabled Yes InvId NoCall No
> 13 1 NoObj NonZero Valid Enabled No InvId NoCall No
> 14 1 NoObj NonZero Valid Disabled Yes InvId NoCall No
> 15 1 NoObj NonZero Valid Disabled No InvId NoCall No
> 16 0 Self Zero Invalid Enabled Yes InvNum NoCall No
> 17 0 Self Zero Invalid Enabled No InvNum NoCall No
> 18 0 Self Zero Invalid Disabled Yes InvNum NoCall No
> 19 0 Self Zero Invalid Disabled No InvNum NoCall No
> 20 0 Self Zero Valid Enabled Yes InvNum NoCall No
> 21 0 Self Zero Valid Enabled No InvNum NoCall No
> 22 0 Self Zero Valid Disabled Yes InvNum NoCall No
> 23 0 Self Zero Valid Disabled No InvNum NoCall No
> 24 2 Self NonZero Invalid Enabled Yes NotDef NoCall No
> 25 2 Self NonZero Invalid Enabled No NotDef NoCall No
> 26 2 Self NonZero Invalid Disabled Yes NotDef NoCall No
> 27 2 Self NonZero Invalid Disabled No NotDef NoCall No
> 28 6 Self NonZero Valid Enabled Yes Ok DuringSend Yes
> 29 4 Self NonZero Valid Enabled No Ok DuringSend No
> 30 3 Self NonZero Valid Disabled Yes Ok AfterEnable No
> 31 3 Self NonZero Valid Disabled No Ok AfterEnable No
> 32 0 Other Zero Invalid Enabled Yes InvNum NoCall No
> 33 0 Other Zero Invalid Enabled No InvNum NoCall No
> 34 0 Other Zero Invalid Disabled Yes InvNum NoCall No
> 35 0 Other Zero Invalid Disabled No InvNum NoCall No
> 36 0 Other Zero Valid Enabled Yes InvNum NoCall No
> 37 0 Other Zero Valid Enabled No InvNum NoCall No
> 38 0 Other Zero Valid Disabled Yes InvNum NoCall No
> 39 0 Other Zero Valid Disabled No InvNum NoCall No
> 40 2 Other NonZero Invalid Enabled Yes NotDef NoCall No
> 41 2 Other NonZero Invalid Enabled No NotDef NoCall No
> 42 2 Other NonZero Invalid Disabled Yes NotDef NoCall No
> 43 2 Other NonZero Invalid Disabled No NotDef NoCall No
> 44 7 Other NonZero Valid Enabled Yes Ok AfterDispatch Yes
> 45 5 Other NonZero Valid Enabled No Ok AfterDispatch No
> 46 3 Other NonZero Valid Disabled Yes Ok AfterEnable No
> 47 3 Other NonZero Valid Disabled No Ok AfterEnable No
> ===== ========== ===== ======= ======= ======== ====== ====== =============
> =========
Is the
>
> Here the same information in a different view, for each post-condition set the
> pre-condition sets are displayed:
>
> ./specview.py --filter=action-list /rtems/signal/req/send
>
> Status = Ok, Handler = DuringSend, Recursive = Yes
>
> * Task = Self, Set = NonZero, Handler = Valid, ASR = Enabled, Nested = Yes
>
> Status = Ok, Handler = DuringSend, Recursive = No
>
> * Task = Self, Set = NonZero, Handler = Valid, ASR = Enabled, Nested = No
>
> Status = Ok, Handler = AfterDispatch, Recursive = Yes
>
> * Task = Other, Set = NonZero, Handler = Valid, ASR = Enabled, Nested = Yes
>
> Status = Ok, Handler = AfterDispatch, Recursive = No
>
> * Task = Other, Set = NonZero, Handler = Valid, ASR = Enabled, Nested = No
>
> Status = Ok, Handler = AfterEnable, Recursive = No
>
> * Task = { Self, Other }, Set = NonZero, Handler = Valid, ASR = Disabled,
> Nested = { Yes, No }
>
> Status = InvId, Handler = NoCall, Recursive = No
>
> * Task = NoObj, Set = NonZero, Handler = { Invalid, Valid }, ASR = {
> Enabled, Disabled }, Nested = { Yes, No }
>
> Status = NotDef, Handler = NoCall, Recursive = No
>
> * Task = { Self, Other }, Set = NonZero, Handler = Invalid, ASR = { Enabled,
> Disabled }, Nested = { Yes, No }
>
> Status = InvNum, Handler = NoCall, Recursive = No
>
> * Task = { NoObj, Self, Other }, Set = Zero, Handler = { Invalid, Valid },
> ASR = { Enabled, Disabled }, Nested = { Yes, No }
>
>
>>
>>> The validation tests are generated from the specification using the
>>> "./spec2modules.py" script and end up in the RTEMS sources of a Git submodule. I
>>> think the specification and the generation tool is now sufficiently stable so
>>> that the validation test code can be integrated in the RTEMS master branch. The
>>> patch set is too big for the mailing list, so you can review it here:
>>>
>>> https://git.rtems.org/sebh/rtems.git/log/?h=validation
>>
>> The link failed.
>
> Yes, viewing my personal repository no longer works. I am not sure if this is a
> temporary issue. This is why I added the github link.
It seems to have been temporary. It is back again.
>
>>
>>> https://github.com/sebhub/rtems/tree/validation
>>
>> The header in a test says the regeneration instructions are in the engineering
>> manual but I could not quickly find them?
>
> https://docs.rtems.org/branches/master/eng/req/howto.html#generate-content-after-changes
>
>
> In an earlier version of the header, we had a link which you didn't like:
If I need to look at the formatting rules the heading "Software Development
Management" is easy to see and then a click on "Coding Standards" gives me what
I am looking for.
To generate these headers I click on "Software Requirements Engineering" and
then do I just guess until I find it in the "How To" section? I am actually
asking this be sorted out so it is not left hanging and we are not left guessing
what to do. If it can be rearrange into something meaningful it would help. :)
> commit a6689fb147649a29ff5533cec8a53635e2c95ec6
> Author: Sebastian Huber <sebastian.huber at embedded-brains.de>
> Date: Fri Jan 22 16:01:46 2021 +0100
>
> Improve file header comment in generated files
>
> diff --git a/cpukit/include/rtems/rtems/types.h
> b/cpukit/include/rtems/rtems/types.h
> index 32b45113c8..a058eedea1 100644
> --- a/cpukit/include/rtems/rtems/types.h
> +++ b/cpukit/include/rtems/rtems/types.h
> @@ -38,11 +38,15 @@
> * worded better please post a report or patch to an RTEMS mailing list
> * or raise a bug report:
> *
> - * https://docs.rtems.org/branches/master/user/support/bugs.html
> + * https://www.rtems.org/bugs.html
> *
> - * For information on updating and regenerating please refer to:
> + * For information on updating and regenerating please refer to the How-To
> + * section in the Software Requirements Engineering chapter of the
> + * RTEMS Software Engineering manual. The manual is provided as a part of
> + * a release. For development sources please refer to the online
> + * documentation at:
> *
> - * https://docs.rtems.org/branches/master/eng/req/howto.html
> + * https://docs.rtems.org
> */
>
> /* Generated from spec:/rtems/type/if/header */
>
I am fine with the header, as we know links to the documentation is really hard
long term.
>>> The patch set is organized so that general support code for the validation tests
>>> is added first and then there are commits for each pre-qualified Classic API
>>> Manager or subsystem.
>>>
>>> I did build all BSPs with the patch set. The validation tests use only
>>> statically allocated resources.
>>
>> Are the validation tests compatible with rtems-test?
>
> Yes.
Great.
>> How many test executables does this add to the testsuite?
>
> If RTEMS_SMP is disabled, then there are 19 new executables in the patch set. If
> RTEMS_SMP is enable, then there are 28 new executables.
Nice.
>> What hardware have the validation tests been run on? Any tier 1 archs?
>
> I tested with the sparc/leon3 BSPs and the arm/realview_pbx_a9_qemu.
Is the leon3 tested on hardware or simulation?
> You need a
> full implementation of the new Interrupt Manager directives and a working Cache
> Manager implementation.
Is this documented?
I am sorry I do not know the list of archs and bsps that support the new
interrupt manager directives. Maybe it would be good to list them?
> I noticed an issue with the thread restart on aarch64/a53_lp64_qemu.
>
> On powerpc/psim there is an issue in one test case, due to:
>
> #define CPU_ALL_TASKS_ARE_FP CPU_HARDWARE_FP
Sorry, I am not following what the issue is? Does this effect all PPC BSPS?
> Another issue is that the tm27 interrupt must be independent of the clock driver
> interrupt. This is not the case for powerpc/psim.
>
> There is definitely some work left to cover all edge cases. Some tests are quite
> complicated.
Sure. I would like to understand the effects this has?
>> Is there anything that interprets the new test output format? It looks like lots
>> of great info but a little difficult to read.
>
> EDISOFT worked on a test report generator, however, it is not yet in a
> reviewable state.
OK. I think something that handles this data would be good to have.
>>> Some low memory targets are not able to link all test suites.
>>
>> Are these excluded in the normal way?
>
> Yes:
>
> https://github.com/sebhub/rtems/commit/2feedb9e7805f483e35b7914b9bd7c808e31a8b4#diff-bf7b325198d4133c9e52f49d77447905ecd3c6ac5159d5cd85e7efeffa6da47e
Nice.
This looks really good and I like what it provides. It is a fantastic baseline
of the interfaces we have however to be useful long term it needs to generate
regression notices and so that means we need to know and understand the
baselines this brings us now and for which arches.
Chris
More information about the devel
mailing list