New validation test suites
Sebastian Huber
sebastian.huber at embedded-brains.de
Wed Dec 22 15:27:22 UTC 2021
On 17/12/2021 04:34, Chris Johns wrote:
> On 16/12/21 6:36 pm, Sebastian Huber wrote:
>> On 16/12/2021 04:51, Chris Johns wrote:
>>> On 16/12/21 3:27 am, Sebastian Huber wrote:
>>>> On 15/12/2021 06:46, Chris Johns wrote:
>>>>> On 14/12/21 6:24 pm, Sebastian Huber wrote:
>>>>>> Hello Chris,
>>>>>>
>>>>>> On 13/12/2021 22:01, Chris Johns wrote:
>>>>>>> On 14/12/21 1:53 am, Sebastian Huber wrote:
>>>> [...]
>>>>>>>> We finished the specification of the pre-qualified RTEMS feature set. The
>>>>>>>> specification is available in an RTEMS Project repository:
>>>>>>>>
>>>>>>>> https://git.rtems.org/rtems-central/tree/spec
>>>>>>>
>>>>>>> I had a quick look. Is there a more user friendly view of this data?
>>>>>>>
>>>>>>> I think the term "specification" is a little bit misleading because the data
>>>>>>> files are not easily read by a person. I understand this is the specification
>>>>>>> data set however it is not what I am traditionally use to seeing.
>>>>>>
>>>>>> You can use the "./specview.py" script to get views of the specification. For
>>>>>> example, this command displays the transition map for the rtems_signal_send()
>>>>>> directive:
>>>>>
>>>>> Is specview.py part of rtems.git?
>>>>
>>>> No, this script is in rtems-central. This is also the location of the
>>>> specification items.
>>>
>>> I am not sure linking a script from that repo like this is helpful.
>>>
>>>>> If not part of rtems.git how much data is there for all the output? That is it
>>>>> is generated and held in the repo with the tests?
>>>>
>>>> In rtems.git, there are only the generated sources.
>>>>
>>>> [...]
>>>
>>> There should be no reach back to the upstream specs, scripts etc and for good
>>> reasons. The information you posted is nice and useful and I do not wish to
>>> release manage rtems-central to accommodate these tests in a release.
>>>
>>> Would capturing that information with the tests be something worth doing?
>>
>> I don't think it would be useful. If you want to modify the tests you should
>> work with the specification items and the corresponding scripts.
>
> This is not about modifying the tests. As I previously stated the tests provide
> little detail on the verification matrix being solved.
>
> In relation to where is best to make changes, rtems-central may be the best
> place however we will accept patches to the tests as they are in rtems.git. How
> that is pushed back to rtems-central is not a focus here.
In theory, you can change the tests by hand, however, this is very
difficult. I had to compact the preconditions to postconditions table to
reduce the data size of the tests. It looks now like this:
static const ScoreSchedReqYield_Entry
ScoreSchedReqYield_Entries[] = {
#if !defined(RTEMS_SMP)
{ 1, 0, 0, 0, 0, 0, ScoreSchedReqYield_Post_HomeSchedulerState_NA,
ScoreSchedReqYield_Post_AskForHelp_NA },
#else
{ 1, 0, 0, 0, 0, 0, ScoreSchedReqYield_Post_HomeSchedulerState_NA,
ScoreSchedReqYield_Post_AskForHelp_NA },
#endif
{ 1, 0, 0, 0, 0, 0, ScoreSchedReqYield_Post_HomeSchedulerState_NA,
ScoreSchedReqYield_Post_AskForHelp_NA },
[...]
#if !defined(RTEMS_SMP)
{ 1, 0, 0, 0, 0, 0, ScoreSchedReqYield_Post_HomeSchedulerState_NA,
ScoreSchedReqYield_Post_AskForHelp_NA }
#else
{ 0, 0, 0, 0, 0, 0, ScoreSchedReqYield_Post_HomeSchedulerState_Idle,
ScoreSchedReqYield_Post_AskForHelp_No }
#endif
};
static const uint8_t
ScoreSchedReqYield_Map[] = {
0, 0, 2, 2, 3, 8, 10, 11, 0, 0, 2, 2, 4, 4, 1, 1, 5, 5, 1, 1, 5, 5,
1, 1, 0,
0, 6, 6, 12, 8, 13, 14, 0, 0, 6, 6, 4, 4, 9, 9, 3, 15, 7, 7, 3, 3, 7, 7
};
If you want to work with these tests, then you definitely should do this
via rtems-central.
>
>> Adding the
>> tables as comments would blow up the sources considerably. Some tests have about
>> 50000 table entries and the table entries depend on C preprocessor defines.
>
> Ah yes I agree and what I was wanting to understand.
>
> I think we need to understand what a release will contain because the specs used
> are not captured with a release. I appreciate the efforts to make this available
> as a workflow for development and for the pre-qual process but I am now
> critically examining what this means for a release. For example lets say years
> after a release someone questions a test, they will only have the test source
> code in the release package and I think this is a short coming. Adding the
> rtems-central repo as a package to the releases source may be a solution however
> this creates further issues. How do I know the master of the rtems-central and
> the committed sources match when creating the release as separate pieces of
> rtems.git may have been updated at different commit points in rtems-central?
>
> I suggest we get together when we can. Contact me off-line and lets see if we
> can arrange a time :)
We can add a release branch to rtems-central. The RTEMS sources are
included as a Git submodule.
>
>>
>> [...]
>>>>>> In an earlier version of the header, we had a link which you didn't like:
>>>>>
>>>>> If I need to look at the formatting rules the heading "Software Development
>>>>> Management" is easy to see and then a click on "Coding Standards" gives me what
>>>>> I am looking for.
>>>>>
>>>>> To generate these headers I click on "Software Requirements Engineering" and
>>>>> then do I just guess until I find it in the "How To" section? I am actually
>>>>> asking this be sorted out so it is not left hanging and we are not left
>>>>> guessing
>>>>> what to do. If it can be rearrange into something meaningful it would help. :)
>>>>
>>>> Well, if you read the text in the header:
>>>>
>>>> * For information on updating and regenerating please refer to the How-To
>>>> * section in the Software Requirements Engineering chapter of the
>>>> * RTEMS Software Engineering manual. The manual is provided as a part of
>>>> * a release. For development sources please refer to the online
>>>> * documentation at:
>>>> *
>>>> * https://docs.rtems.org
>>>>
>>>> You should read the How-to section or not?
>>>
>>> Yes I should have and thanks for pointing this out but I did not see this and
>>> the manual as it stands did not help. I think it should change. It can be
>>> performed post this patch set but I think the documentation would read better if
>>> changed.
>>
>> Could you please make a suggestions how the text should be changed?
>>
>
> I am not sure. I have not looked at rtems-central, the scripts or commands it
> provides. I think at a minimum a section each set of generated sources in
> rtems.git needs to be covered.
Ok, I will add a short description of the scripts to the How-to section.
>
>>>>>>> What hardware have the validation tests been run on? Any tier 1 archs?
>>>>>>
>>>>>> I tested with the sparc/leon3 BSPs and the arm/realview_pbx_a9_qemu.
>>>>>
>>>>> Is the leon3 tested on hardware or simulation?
>>>>>
>>>>>> You need a
>>>>>> full implementation of the new Interrupt Manager directives and a working
>>>>>> Cache
>>>>>> Manager implementation.
>>>>>
>>>>> Is this documented?
>>>>>
>>>>> I am sorry I do not know the list of archs and bsps that support the new
>>>>> interrupt manager directives. Maybe it would be good to list them?
>>>>
>>>> All BSPs have at least a stub implementation of the new directives. The
>>>> directives are tested in a dedicated test suite. You will notice failures in
>>>> this test suite if the directives are not implemented.
>>>
>>> Are these expected failures?
>>
>> Yes, they would be expected failures. I can add the test information. For which
>> BSPs should I do this?
>
> Any we know the tests will not run on?
I rearranged the test case to test suites distribution. Now, the tm27
support is no longer used if a clock driver is enabled.
>
>>>>>> I noticed an issue with the thread restart on aarch64/a53_lp64_qemu.
This issue is now fixed. It was an error in the test.
>>>>>>
>>>>>> On powerpc/psim there is an issue in one test case, due to:
>>>>>>
>>>>>> #define CPU_ALL_TASKS_ARE_FP CPU_HARDWARE_FP
>>>>>
>>>>> Sorry, I am not following what the issue is? Does this effect all PPC BSPS?
>>>>
>>>> Not all, the newer BSPs have no separate floating-point context.
>>>
>>> Which ones have the issue, the newer BSPs or the older ones?
>>
>> The older ones.
I fixed this issue in the tests. However, a new issue popped up. The
setjmp() doesn't work in interrupt context for psim since the FPU is
disabled.
>>
>>>
>>>> This is something which needs to be fixed in the specification.
>>>
>>> Of?
>>>
>>> < From my point of view this is just a minor issue.
>>>
>>> As in fixing these tests?
>>>
>>>>>> Another issue is that the tm27 interrupt must be independent of the clock
>>>>>> driver
>>>>>> interrupt. This is not the case for powerpc/psim.
>>>>>>
>>>>>> There is definitely some work left to cover all edge cases. Some tests are
>>>>>> quite
>>>>>> complicated.
>>>>>
>>>>> Sure. I would like to understand the effects this has?
>>>>
>>>> Maybe I can rearrange the test cases so that the tm27 support is only used if no
>>>> clock driver is needed. The tm27 support is used to run handlers in interrupt
>>>> context.
>>>
>>> OK.
>>
>> I will try to fix these issues, but this will delay the integration for a couple
>> of weeks.
>
> I am fine with tickets that track the work to be done.
>
>>>>>>> Is there anything that interprets the new test output format? It looks like
>>>>>>> lots
>>>>>>> of great info but a little difficult to read.
>>>>>>
>>>>>> EDISOFT worked on a test report generator, however, it is not yet in a
>>>>>> reviewable state.
>>>>>
>>>>> OK. I think something that handles this data would be good to have.
>>>>
>>>> Yes, maybe we could let a student work on this. In theory, this is not
>>>> difficult. Read the report.yaml generated by the RTEMS Tester and convert it to
>>>> a Python objects representation. Then use this high-level representation to
>>>> generate a report in format X.
>>>
>>> Sounds good.
>>>
>>> And we need to get all the BSPs baselined with 0 failures so we know where we
>>> stand as changes are being made.
>>
>> All BSPs is a bit too much.
>>
>
> Yes sorry I meant all tier 1 BSPs.
I can add tickets for the current tier 1 BSP for things which are
missing. All tier 1 BSPs lack support for the new Interrupt Manager
directives for example.
--
embedded brains GmbH
Herr Sebastian HUBER
Dornierstr. 4
82178 Puchheim
Germany
email: sebastian.huber at embedded-brains.de
phone: +49-89-18 94 741 - 16
fax: +49-89-18 94 741 - 08
Registergericht: Amtsgericht München
Registernummer: HRB 157899
Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler
Unsere Datenschutzerklärung finden Sie hier:
https://embedded-brains.de/datenschutzerklaerung/
More information about the devel
mailing list