Addition of Rule Checkers

[Chris Johns]

On 30/7/19 4:32 pm, Sebastian Huber wrote:
> On 30/07/2019 01:01, Joel Sherrill wrote:
>>
>>
>> On Mon, Jul 29, 2019 at 5:23 PM Gedare Bloom <gedare at rtems.org
>> <mailto:gedare at rtems.org>> wrote:
>>
>>     On Wed, Jul 24, 2019 at 7:16 AM Joel Sherrill <joel at rtems.org
>>     <mailto:joel at rtems.org>> wrote:
>>      >
>>      > Hi
>>      >
>>      > I just wanted to make sure we followed proper procedures and
>>     policies when considering rule checkers.
>>      >
>>     I guess these comments are germane to the other thread, but I want to
>>     reflect here a bit.
>>
>>      > 1. The license must appropriate.
>>      >
>>     Can you expand what you mean by this?
>>
>>
>> It must be a FLOSS tool -- it can't be proprietary. We use Scan but it has
>> issues.
>>
>> It also must be available on the normal set of commonly used hosts.
>>
>>
>>      > 2. There should be some basic requirements that the tool is
>>     expected to meet to be fit for purpose
>>      >
>>      > 3. The tool must support the range of development hosts used in
>>     the Community.
>>      >
>>     Only if we make it a requirement for developers to check these rules
>>     locally before submitting patches/commits. If we had a server for
>>     example that could accept submissions from developers, this would be
>>     irrelevant?
>>
>>
>> If we relax this rule, then we will end up with a one host tool and have to have
>> a server that runs it. Our infrastructure is FreeBSD so saying it supports the
>> range
>> of hosts is likely saying it supports our servers, Linux, MacOS, Cygwin, and
>> MingW
> 
> I would not make it a requirement that every user must be able to run the tools
> on a wide range of platforms. We should clearly identify the tools (e.g.
> version) and the tool configuration (rules, scope).

I am not sure I understand what this means, specifically ...

> It should enable someone
> else to set up the tool and run it in a configuration used by the RTEMS project.

What does "set up the tool" mean? Do I need to have a suitable host available
that runs the tool if my preferred host is not supported?

> What is the overall goal of this checker stuff? We want that they find bugs for
> us which we can fix. For the checker tools we have at least to categories: 1.
> static analysis and 2. coding rule checkers.

We need to balance this with needing to maintain a low effort bar for patches or
we will not see them.

> 1. For the static analysis tools I think it doesn't matter how they are run or
> by whom. If they find a bug, we fix it.

Is this a precondition for submission or pushing of a patch? I think you are
saying it is not but I cannot tell.

> 2. This is a bit different to the old style coding rule checkers. There aim is
> to write the code with rules so that bugs are prevented in the first place (e.g.
> MISRA). I think these tools must be freely available. 

Agreed which brings us back to Joel's conditions. I support those as they are
based on our long history and being bitten before by not being as strict as we
should have. I am not sure we need to match the host list we support for
development of RTEMS applications however this discussion become circular.

> Rule violations will result in patches which don't fix bugs.

What happens to patches that do not meet the rules but fix a bug? Are the rules
that full proof?

Is such a patch rejected, ignored, or is it expected a developer sorts this out
so the patch can be pushed? I am concerned about an increase in the open source
tax for developers with open commit access.

We need to maintain workable solution for all the unfunded effort and a bar that
is suitably low so patches from the community can be posted and accepted.
Manuel's comment about needing rules that are understandable is important,
knowing there are rules is also important and then being able to check those
rules becomes important.

>>      > As a detail, once selected the tool needs to have an RSB recipe.
>>      >
>>     Only if you require that checkers must be built-from-source.
>>
>>
>> If we can't get source, then they aren't FLOSS and we are depending on
>> them coming with distributions and being provided for every host we care
>> about.
>>
>> We need something like Phabricator so things can be run against the code
>> as part of the review/submission process but I want every tool to be available
>> to every user.
> 
> If you set up a project infrastructure which automatically checks patches, then
> not everyone needs to install the tools used for the automatic checks. My
> experience is that in projects with good automatic checks in the Git pull
> pipeline the developers learn to rely on this and stop to check their patches
> locally. They just wait for the reports from the automatic checkers and if a red
> flag appears, they get active.

I agree and would love to see such a set up.

I feel those who benefit the most from the project having this infrastructure to
aid and improve the quality of RTEMS should consider how they can fund on an
ongoing basis the project's infrastructure. Anyone and any organisation can
contract Joel or me privately if they would like more information.

The reality so far for what we have is:

 - Hardware - Google via GSoC, OAR (support payments)
 - Effort - 100% unfunded, Amar (97%), me (3% and that is generous to me).

Getting suitable people to help with the support effort is not easy. We run a
complex hardware environment with tight control. We have too, it is on a fast
backbone and front facing to a range of hostile things. We welcome help but you
need to show a history of good admin skills.

Chris