Where are results of rtems-tester test results archived?

Sun Sep 22 00:18:05 UTC 2019

On Sat, Sep 21, 2019, 6:21 PM Chris Johns <chrisj at rtems.org> wrote:

>
>
> On 22/9/19 7:58 am, dufault at hda.com wrote:
> >
> >
> >> On Sep 21, 2019, at 17:49 , <dufault at hda.com> <dufault at hda.com> wrote:
> >>
> >>
> >>
> >>> On Sep 21, 2019, at 16:41 , Chris Johns <chrisj at rtems.org> wrote:
> >>>
> >>> On 22/9/19 1:18 am, dufault at hda.com wrote:
> >>>>> On Sep 21, 2019, at 11:03 , Joel Sherrill <joel at rtems.org> wrote:
> >>>>> On Sat, Sep 21, 2019, 9:55 AM Peter Dufault <dufault at hda.com> wrote:
> >>>>> I’ve searched but can’t find anywhere.  I’d like to see the results
> of the tests on all architectures to compare to what I see on
> PowerPC-beatnik.
> >>>>>
> >>>>> There is a build@ mailing list and the archives are at
> https://lists.rtems.org/pipermail/build/
> >>>>>
> >>>>> There should be results from at least me for psim.
> >>>>>
> >>>>> You are encouraged to subscribe to the list and post results. Many
> boards have no results.
> >>>>>
> >>>> That doesn’t look like what I want.  I’m looking for something like
> the following (a small snippet of my test run in progress) to see what
> failures are shared by what board support packages.
> >>>>
> >>>> [141/597] p:128 f:7   u:2   e:0   I:0   B:3   t:0   i:0   W:0   |
> powerpc/beatnik: telnetd01.exe
> >>>> [142/597] p:129 f:7   u:2   e:0   I:0   B:3   t:0   i:0   W:0   |
> powerpc/beatnik: termios.exe
> >>>> [143/597] p:129 f:7   u:3   e:0   I:0   B:3   t:0   i:0   W:0   |
> powerpc/beatnik: termios01.exe
> >>>> [144/597] p:129 f:8   u:3   e:0   I:0   B:3   t:0   i:0   W:0   |
> powerpc/beatnik: termios02.exe
> >>>> [145/597] p:129 f:9   u:3   e:0   I:0   B:3   t:0   i:0   W:0   |
> powerpc/beatnik: termios03.exe
> >>>> [146/597] p:130 f:9   u:3   e:0   I:0   B:3   t:0   i:0   W:0   |
> powerpc/beatnik: termios04.exe
> >>>> [147/597] p:131 f:9   u:3   e:0   I:0   B:3   t:0   i:0   W:0   |
> powerpc/beatnik: termios05.exe
> >>>
> >>> The --log option should place all the test results in a log. Is that
> what you want?
> >>>
> >>> There are other options that let you log the console data as well.
> This is
> >>> disabled by default to make the log more compact.
> >>>
> >>
> >> I want a collection of results for all tested BSPs so that I can
> compare what I’m seeing with “beatnik” with others.
> >>
> >> I thought there was a collection of systems that went through automated
> tests, an RTEMS testing lab of sorts.  Am I confused?  I was ready to try
> to get an MVME5500 and whatever else was needed to the lab.
> >>
> > Beatnik finished:
> > [595/597] p:563 f:19  u:6   e:0   I:0   B:3   t:1   i:2   W:0   |
> powerpc/beatnik: tmonetoone.exe
> > [596/597] p:564 f:19  u:6   e:0   I:0   B:3   t:1   i:2   W:0   |
> powerpc/beatnik: tmoverhd.exe
> > [597/597] p:565 f:19  u:6   e:0   I:0   B:3   t:1   i:2   W:0   |
> powerpc/beatnik: tmtimer01.exe
> > Passed:        565
> > Failed:         20
> > User Input:      6
> > Expected Fail:   0
> > Indeterminate:   0
> > Benchmark:       3
> > Timeout:         1
> > Invalid:         2
> > Wrong Version:   0
> > Wrong Build:     0
> > Wrong Tools:     0
> > ------------------
> > Total:         597
> > Average test time: 0:00:20.652608
> > Testing time     : 3:25:29.607257
>
> The results are relative to the BSP. It would be nice to say we can expect
> all
> tests to pass on all BSPs however the likelihood of that happening is low.
> To
> manage this so a user of a specific BSP can see if there are regressions
> tests
> can be tagged with the expected result for a specific BSP. If a test is
> known to
> fail the test can be set to expected-fail for a BSP and you should see no
> failures, invalids or timeouts. I raised a ticket for this ..
>
> https://devel.rtems.org/ticket/2962
>
> ... and while I can do some I cannot do them all. I am wondering if the
> lack of
> feedback is related to the process needed to tag a specific test for a
> BSP. I am
> considering a tool to make the process simpler. It also takes time to
> examine
> the results and to determine if expected-fail is suitable.
>

Some of the failures I see on mips are very clearly lack of support for
something on that architecture. This will be common if we get results on
anything which doesn't have TLS, dynamic loading, or debugging support.

Addressing this type of issue at a level higher than individual bsps would
be helpful.

>
> The test controls are documented here ....
>
>
> https://docs.rtems.org/branches/master/user/testing/tests.html#test-controls
>
> Chris
> _______________________________________________
> devel mailing list
> devel at rtems.org
> http://lists.rtems.org/mailman/listinfo/devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rtems.org/pipermail/devel/attachments/20190921/3c396615/attachment-0002.html>