Where are results of rtems-tester test results archived?

[Chris Johns]

On 22/9/19 7:58 am, dufault at hda.com wrote:
> 
> 
>> On Sep 21, 2019, at 17:49 , <dufault at hda.com> <dufault at hda.com> wrote:
>>
>>
>>
>>> On Sep 21, 2019, at 16:41 , Chris Johns <chrisj at rtems.org> wrote:
>>>
>>> On 22/9/19 1:18 am, dufault at hda.com wrote:
>>>>> On Sep 21, 2019, at 11:03 , Joel Sherrill <joel at rtems.org> wrote:
>>>>> On Sat, Sep 21, 2019, 9:55 AM Peter Dufault <dufault at hda.com> wrote:
>>>>> I’ve searched but can’t find anywhere.  I’d like to see the results of the tests on all architectures to compare to what I see on PowerPC-beatnik.
>>>>>
>>>>> There is a build@ mailing list and the archives are at https://lists.rtems.org/pipermail/build/
>>>>>
>>>>> There should be results from at least me for psim.
>>>>>
>>>>> You are encouraged to subscribe to the list and post results. Many boards have no results.
>>>>>
>>>> That doesn’t look like what I want.  I’m looking for something like the following (a small snippet of my test run in progress) to see what failures are shared by what board support packages.
>>>>
>>>> [141/597] p:128 f:7   u:2   e:0   I:0   B:3   t:0   i:0   W:0   | powerpc/beatnik: telnetd01.exe
>>>> [142/597] p:129 f:7   u:2   e:0   I:0   B:3   t:0   i:0   W:0   | powerpc/beatnik: termios.exe
>>>> [143/597] p:129 f:7   u:3   e:0   I:0   B:3   t:0   i:0   W:0   | powerpc/beatnik: termios01.exe
>>>> [144/597] p:129 f:8   u:3   e:0   I:0   B:3   t:0   i:0   W:0   | powerpc/beatnik: termios02.exe
>>>> [145/597] p:129 f:9   u:3   e:0   I:0   B:3   t:0   i:0   W:0   | powerpc/beatnik: termios03.exe
>>>> [146/597] p:130 f:9   u:3   e:0   I:0   B:3   t:0   i:0   W:0   | powerpc/beatnik: termios04.exe
>>>> [147/597] p:131 f:9   u:3   e:0   I:0   B:3   t:0   i:0   W:0   | powerpc/beatnik: termios05.exe
>>>
>>> The --log option should place all the test results in a log. Is that what you want?
>>>
>>> There are other options that let you log the console data as well. This is
>>> disabled by default to make the log more compact.
>>>
>>
>> I want a collection of results for all tested BSPs so that I can compare what I’m seeing with “beatnik” with others.
>>
>> I thought there was a collection of systems that went through automated tests, an RTEMS testing lab of sorts.  Am I confused?  I was ready to try to get an MVME5500 and whatever else was needed to the lab.
>>
> Beatnik finished:
> [595/597] p:563 f:19  u:6   e:0   I:0   B:3   t:1   i:2   W:0   | powerpc/beatnik: tmonetoone.exe
> [596/597] p:564 f:19  u:6   e:0   I:0   B:3   t:1   i:2   W:0   | powerpc/beatnik: tmoverhd.exe
> [597/597] p:565 f:19  u:6   e:0   I:0   B:3   t:1   i:2   W:0   | powerpc/beatnik: tmtimer01.exe
> Passed:        565
> Failed:         20
> User Input:      6
> Expected Fail:   0
> Indeterminate:   0
> Benchmark:       3
> Timeout:         1
> Invalid:         2
> Wrong Version:   0
> Wrong Build:     0
> Wrong Tools:     0
> ------------------
> Total:         597
> Average test time: 0:00:20.652608
> Testing time     : 3:25:29.607257

The results are relative to the BSP. It would be nice to say we can expect all
tests to pass on all BSPs however the likelihood of that happening is low. To
manage this so a user of a specific BSP can see if there are regressions tests
can be tagged with the expected result for a specific BSP. If a test is known to
fail the test can be set to expected-fail for a BSP and you should see no
failures, invalids or timeouts. I raised a ticket for this ..

https://devel.rtems.org/ticket/2962

... and while I can do some I cannot do them all. I am wondering if the lack of
feedback is related to the process needed to tag a specific test for a BSP. I am
considering a tool to make the process simpler. It also takes time to examine
the results and to determine if expected-fail is suitable.

The test controls are documented here ....

https://docs.rtems.org/branches/master/user/testing/tests.html#test-controls

Chris