Placement of new unit and validation tests?

[Chris Johns]

On 13/12/19 12:56 am, Sebastian Huber wrote:
> I would like to write new test cases and programs using the RTEMS Test Framework:
> 
> https://docs.rtems.org/branches/master/eng/test-framework.html

I have no specific issue with the test framework as a way forward however I am
not yet comfortable about the integration of these new types of tests into the
eco-system plus what happens to the existing tests?

Currently the `rtems-test` command is the only supported way we have to run the
tests on simulators and hardware and I am busy promoting and encouraging users
in our community to integrate to this tool and to use it to validate their
systems. I feel it is important we maintain this command and what it does.

I am concerned I am expected to integrate these new tests into the `rtems-test`
command. It would be nice if you could explain what needs to be done in the
eco-system to integrate these type of tests and how you think this work will be
done?

> There will be two sets of tests. One set are unit tests. The other set are
> validation tests. The validation tests check that the implementation meets its
> requirements. The unit tests don't need a link to requirements. They can be used
> to test internal APIs like the chains, red-black trees, etc. independent of
> requirements.
> 
> I suggest to add "testsuites/unit" for unit tests and "testsuites/validation"
> for validation tests.

I am not sure about these names but I cannot think of anything better. I see
below you will provide another layer of directories in these paths. That is good.

> For validation tests we need a link from test cases to requirements. This needs
> to be discussed separately.

Sure. It would be good if the eco-system aspects are discussed as well.

> With the RTEMS Test Framework test cases can be written independently from the
> test runner program defining the test suite. This could be used for example to
> run a test case in different contexts. Another option is to group test cases
> depending on the available RAM in the target system, e.g. bigger systems can
> execute more test cases at once. This gives raise to a different organization
> compared to the existing tests programs.

Can you please explain how rtems-test is to handle integrated tests?

How do you manage possible cross-talk between tests?

I think the approach is sound as the time to load a test is often much more than
the time the test runs.

> I suggest to add topic directories to "testsuites/unit" and
> "testsuites/validation", e.g. "testsuites/unit/chains". Test case files use a
> naming scheme of "tc-${topic}-${name}.c" and are placed in the corresponding
> topic directory.  Test suite files use a naming scheme of "ts-${name}.c" and are
> placed in "testsuites/unit" or "testsuites/validation".

Is ${name} a name of a C file selected by the developer? Can there be more than
one ${name}s ?

Chris