Fwd: [PATCH 000/111] GRLIB/LEON RTEMS RCC patches

[Chris Johns]

On 18/03/2015 12:54 am, Daniel Hellstrom wrote:
> On 03/15/2015 10:29 PM, Chris Johns wrote:
>> On 14/03/2015 12:39 pm, Amar Takhar wrote:
>>>> On 11/03/2015 1:27 am, Karel Gardas wrote:
>>>>> Yes, this is a big patch but if you do not merge it now and later
>>>>> following waf work you do tree/file placement refactoring than the
>>>>> patch
>>>>> would be even harder to merge.
>>>
>>> I was thinking this would land in 4.12 then waf would be based off of
>>> that.
>>> 4.12 would be released weeks after 4.11.1
>>>
>>> There needs to be another branch for any 'bigger' changes to hit
>>> RTEMS after
>>> 4.11.
>>>
>>
>> Sure.
>>
>>>
>>> On 2015-03-14 11:25 +1100, Chris Johns wrote:
>>>>
>>>> Yes I tend to see this as important. It will be less overall work to
>>>> merge before the source is changed.
>>>>
>>>> My support for merging sooner rather than latter is conditional on
>>>> Daniel and team ensuring the SPARC and LEON BSP are tested and there
>>>> are
>>>> no regressions for 4.11.
>>>
>>> As long as it's well tested and we redo all the testing Joel has
>>> already done..
>>>
>>
>> There are no tests in the patch set which is a key concern.
>
> Thank you all for you interest and concerns about this.
>
> I agree with you that it is a problem that there are no tests. As I see
> it the new drivers adds functionality and are well isolated from the
> other code. The existing drivers being patched never had tests and
> continue not to have tests, I'm not sure I've seen driver tests for
> other BSPs?

Sorry I was not more specific. It is not the drivers I was talking 
about, it is the pieces of code under cpukit. It is normal to have tests 
using a dummy driver in the testsuite to make sure the interface to the 
actual drivers is in some sane state.

> The drivers comes from RCC based on RTEMS-4.10.2 which
> includes many fixes on the existing drivers and are being used in many
> hardware set ups.

This is understood. My comments about access to a suitable simulator 
and/or hardware is addressing some concerns I have in the actual driver 
testing. The issue for the RTEMS project is accepting code we cannot 
test. We are responsible but we cannot test it. If we could test then we 
are in a better position to ensure a release we make is working.

> When it comes to the libdrvmgr and libpci it will be
> SPARC only at least.

I understand this but the cpukit is for all archs and I also understand 
it does not make sense to have this generic code in a SPARC specific 
part of the source tree.

> The minimum mandatory drivers for RTEMS are
> IRQCtrl, Timer and Console drivers. The IRQ Ctrl driver is not affected
> by drvmgr or libpci. The main concern would therefore be the Timer
> driver and Console driver, which is tested by the RTEMS tests which
> seems to execute successfully.

The testing is also about regressions and if in time something appears 
in the cpukit part of the code and someone other than the original 
developer makes a change the tests provide some level of confidence the 
changes have not broken something. With this confidence we can assume 
the existing hardware drivers should work, something we cannot test 
without access to a suitable test system. I am fine with the code being 
merged and tests coming later.

I should highlight the rtems-test command and it's testing provides the 
basis for our testing into the future. We support simulator testing 
which is nice because we can run the tests in parallel and in some cases 
we have testing on real hardware, the Zynq chip is an example of this. 
The real hardware tests allow us to build a confidence level the 
simulation is working. I see buildbot running tests on selected 
simulators on a regular basis and then testing on real hardware at 
specific intervals and this increases the value of the simulation 
results by validating them. The output of buildbot will in time be 
filtered to create a view of what is tested and what is not tested in 
RTEMS and I feel this information will be significance. My begging for 
testing support is general and to all RTEMS users because it will make a 
difference. We are not in this position yet but are moving to it and 
once we get 4.11 released this will start to be a focus. This is just a 
heads up about what is coming.

Finally, my simplistic understand is a buildbot slave can run on 
someone's machine remote from the RTEMS server machines and it connects 
to the RTEMS buildbot master making the communications secure. The 
buildbot master can then instruct the slave to build and/or run tests.

Chris