*.tcfg specification?

[Chris Johns]

On 13/11/19 1:30 am, Sebastian Huber wrote:
> Hello Chris,
> 
> thanks for your comments. I converted now all tests except the Ada tests, 

Nice.

> tests which use the pax tool,

This might help ...

https://git.rtems.org/rtems_waf/tree/rootfs.py

>  and the dl* tests. 

Plus this ...

https://git.rtems.org/rtems_waf/tree/dl.py

> For an approach to address the test
> states please see below.
> 
> On 11/11/2019 23:25, Chris Johns wrote:
>> On 11/11/19 7:13 pm, Sebastian Huber wrote:
>>> what is the purpose of the *.tcfg test configuration files?
>>
>> The tcfg files provide a way to implement the "test controls" ...
>>
>> https://docs.rtems.org/branches/master/user/testing/tests.html#test-controls
>>
>> .. as well as excluding a test. A test executable that is not excluded must
>> build and be loadable on a BSP target and executed.
> 
> There are some *norun* tests.
> 

Ah yes. I seem to remember we gave the tests a different extension so rtems-test
does not find them?

>> Tests can be excluded due to
>> an architecture issue (lacking an architecture feature), BSP issue (not enough
>> memory) or an RTEMS or tools support related issue (missing feature). A build
>> test run on a BSP target uses the "test banners" to publish it's state ...
>>
>> https://docs.rtems.org/branches/master/user/testing/tests.html#
>>
>> The existing implementation is a mix of build system controls to exclude a test
>> and command line defines to control the build in a standard and controlled way.
>> The rtems-test tool understands the test banners and adapts it's control based
>> on what the test reports. For example a `user-input` test is terminated and the
>> next test is run.
> 
> Ok, it seems a test program has exactly one state for a given BSP.

Yes, this is driven by the ABI, hardware profile and similar constraints.

>>> You can disable compilation of test programs with them, e.g.
>>>
>>> exclude: flashdisk01
>>
>> Yes. There is also `rexclude` a regex filter, for example `rexclude:
>> dl[0-9][0-9]`.
>>
>>> The "testsuites/rtems-test-check.py" contains a bit more stuff. What is really
>>> needed?
>>
>> I suspect you will end up needing to match the functionality provided. What
>> rtems-test-check.py supports has grown since I introduced this support. It
>> started off as a simple script however it did not do what we needed and adding
>> features cause the performance to blow out so I switch to python and since then
>> more complexity has been added. The script is currently run once for each test,
>> that can be avoided with waf.
> 
> I don't want to use this script in the new build system. 

That is understandable. It is a purpose built script for the existing build
system. I think the functionality it provided is the important part.

> I want to map the
> *.tcfg functionality to the build specification and use the means that are
> already there as far as possible. I would like to avoid too many special cases.

I am not sure I understand what you mean by special cases? Providing a exact fit
does not give us space to grow and we all change shape over time, it cannot be
avoided. I am a living example of that.

>>> I have to map this to the new build system.
>>
>> Yes you will. The mapping of the tcfg files is more complex than it may appear.
>> The complexity is in what we need to represent and not in the implementation. In
>> a waf build system the configure phase could be a single pass where the tcfg
>> files are loaded in a single pass and then held in memory while the build
>> instance for each test is created.
>>
>> A quick summary:
>>
>> 1. includes
>> 2. excludes and include with regular expression filtering, ie exclude a bunch
>> then include something specific
>> 3. test controls
>>
>> Include support allows a single definition of a test profile and reuse across a
>> number of BSPs. The use of regex filters allow for subtle variations of how this
>> works.
> 
> In theory, we could support regular expressions in the "enabled-by" attribute.
> Is this really necessary? It shouldn't be an issue to enumerate all tests that
> belong to a certain group.
> 
> $ grep --include='*.tcfg' rexclude -r .
> ./testsuites/testdata/disable-libdl-tests.tcfg:rexclude: dl[0-9][0-9]

The support was added because I needed to update the list of excluded libdl
tests when a new test was added and I seemed to always forget until a while
later Joel pings me with a breakage. A user adding a new libdl test would not
know a change was needed.

The functionality lets us name tests of a specific class or category and then
manage that class.

> It is only used for the dl* tests and it would be simple to just give a list of
> tests.

This is a case of something being present and it's use grows as we find more
cases of it being needed. It is about providing a framework and not a specific
fit. I would not place too much on there being only one instance of its use.

>>> One approach is to use the "enabled-by" attribute to disable test programs on
>>> demand, e.g.:
>>>
>>> $ cat spec/build/testsuites/samples/RTEMS-BUILD-TEST-SAMPLE-TICKER.yml
>>
>> Hmm, I had not noticed .yml is being used. By default emacs does not know this
>> is a YAML format file. Why has this been selected over .yaml? Having to teach
>> editors about a new extension is painful.
> 
> Doorstop uses this extension. We would have to adjust doorstop to use another
> extension.

That is a shame.

>>> active: true
>>> build-type: test-program
>>> cflags: []
>>> cppflags: []
>>> cxxflags: []
>>> derived: false
>>> enabled-by:
>>> - not: DISABLE_TEST_SAMPLE_TICKER
>>> features: c cprogram
>>> header: ''
>>> includes: []
>>> level: 1.15
>>> links: []
>>> normative: true
>>> order: 0
>>> ref: ''
>>> reviewed: null
>>> source:
>>> - testsuites/samples/ticker/init.c
>>> - testsuites/samples/ticker/tasks.c
>>> stlib: []
>>> target: testsuites/samples/ticker.exe
>>> text: ''
>>> type: build
>>> use: []
>>>
>>> A BSP can use a "build-type: option" file to disable tests, e.g.
>>>
>>> actions:
>>> - disable-tests:
>>>    - SAMPLE/TICKER
>>>    - LIB/FLASHDISK01
>>
>> I would be careful before heading down this path. Lets assume 150 BSPs and 570
>> tests, that is 85,500 tests and test states. Adding a modest number of test
>> cases could effect the readability of these files. I think you will need to
>> structure this rather than having it expressed as a flat list.
> 
> You will have a flat list per BSP (with includes via item "links") just as you
> have it now with the *.tcfg files.

Perfect. There is no need to have the same implementation in the new build system.

We need to make sure we capture the important pieces of information we have now
that has taken a few years to build up ....

$ grep 'small-memory-testsuite' `find . -name \*.tcfg` | wc -l
      27

We have 27 small memory BSPs (remove the line count and you see the list). If a
new test is added that cannot run on small memory BSPs we need a simple way in
one location to update the list. This applies to each of the top level test
controls we have.

>> There are 77 tcfg files. There are top level definitions that are used by BSPs:
>>
>>   testsuites/testdata/disable-intrcritical-tests.tcfg
>>   testsuites/testdata/disable-libdl-tests.tcfg
>>   testsuites/testdata/disable-mrfs-tests.tcfg
>>   testsuites/testdata/rtems.tcfg
>>   testsuites/testdata/small-memory-testsuite.tcfg
>>   testsuites/testdata/require-tick-isr.tcfg
>>   testsuites/testdata/disable-jffs2-tests.tcfg
>>   testsuites/testdata/disable-iconv-tests.tcfg
> 
> You can do includes through the "links" attribute. The number of test state
> specification items will be roughly the same as the *.tcfg file count.

Nice.

>> The rtems.tcfg defines the state of tests common to all BSPs, for example
>> `user-input` and `benchmark`. These are invariant. Others like
>> small-memory-testsuite.tcfg are more complex including other test profiles like
>> disable-iconv-tests.tcfg etc.
>>
>> I can see us having test profiles based on an architecture or BSP family before
>> you reach the BSP. I have not checked to see if this is already happening.
>>
>> If you could structure the dependency in away that reflected what there is it
>> would help. For example tms570ls3137_hdk_with_loader-testsuite.tcfg includes
>> small-memory-testsuite.tcfg plus it excludes linpack. If that BSPs spec was just
>> `small-memory` and `exclude: linpack` it would be readable and maintainable.
>>
>> I understand wanting to use the spec support however I do not know enough about
>> it to offer a specific solution.
> 
> The "links" attribute is a very important thing in the specification item
> business. 

I agree.

> I guess I have to add some dot images to the software requirements
> engineering chapter.
>
>>> An open issue is how and if we should set other states, e.g. expected to fail.
>>> We could generate a "bsp/teststate.h" file for each BSP which optionally defines
>>> for each test a state other than "pass", e.g.
>>>
>>> #define TEST_SAMPLE_TICKER_STATE RTEMS_TEST_STATE_FAIL
>>
>> Why not stay with defines for this specific case? I understand and agree the
>> issue of figuring out what can happen with a build because you need to
>> understand the command line but in this case the parameters are bounded and
>> known. You need a build instance for each test a BSP has and you can add a
>> specific set of defines to each build instance. Waf will track the flags in a
>> build and any changes will rebuild the dependent targets.
>>
>>> This could be also realized with a special option action:
>>>
>>> actions:
>>> - set-test-states:
>>>    - SAMPLE/TICKER: disable
>>>    - LIB/FLASHDISK01: xfail
>>
>> I have avoided abbreviations so far and used `expected-fail`. I would like it to
>> stay that way if that is OK.
>>
>>> The "set-test-states" action can validate if the state exists. I can use the
>>> "enable-by" for "disable" states and the "bsp/teststate.h" file for other
>>> states.
>>
>> My concerns with a single test state header are:
>>
>> 1. size, we have a lot of tests
> 
> It doesn't matter much where you place the information.  With the CPPFLAGS
> approach the information just moves to the configuration set environment.

I am not sure yet. I suppose in time we will find out.

>> 2. slipping into adding things which should not be there but could be, ie adding
>> C code to control something
>> 3. potential cross-talk between tests, ie I borrow a piece of code from another
>> test and do not change a dependent define.
> 
> I think 2. and 3. are theoretical issues since the content of the header file
> would be determined by the build option item actions.

Item 2 is about a policy for changes in this area and rejecting patches that
step into adding C code type functionality.

I can see 3. happening, it is only a matter of time.

>> I also like the idea of a single place to inspect how tests are configured.
> 
> Currently you have an optional *.tcfg per base BSP variant with an ability to
> include files. You also have a magic rtems.tcfg which is added by default. This
> is not a single place to inspect from my point of view. However, I don't think a
> good alternative exists.

Yes, this is a weakness with the tcfg files that I was going to solve by adding
a command to the top level of the source tree to provide reports. I stopped this
when you started on this build system task.

I have been asking for months, hmmm maybe a year or more for the tcfg files to
be updated with expected-fail states so rtems-test can determine regressions. I
had started to wonder if core maintainers were no sure how to make these changes
and that is reasonable because there was no documentation and no method to
report the effect of the change without a full build and test run. I needed to
make the process easier and simpler for it be used.

> I re-implemented now practically the *.tcfg files and the existing mechanism
> (module the regular expressions). You can find an example here:
> 
> https://git.rtems.org/sebh/rtems.git/tree/spec/build/bsps/RTEMS-BUILD-BSP-TSTFIXME.yml?h=build

Is the test name always in caps?

What about ...

 DL[0-9][0-9]: exclude

? :)

> This is more or less the old rtems.tcfg.
> 
> It is pulled in by:
> 
> https://git.rtems.org/sebh/rtems.git/tree/spec/build/bsps/RTEMS-BUILD-BSP-BSPOPTS.yml?h=build

How are links and their checksum maintained?

> Which in turn is included by every BSP, e.g.
> 
> https://git.rtems.org/sebh/rtems.git/tree/spec/build/bsps/sparc/erc32/RTEMS-BUILD-BSP-SPARC-ERC32-ERC32.yml?h=build

Does this means a change in RTEMS-BUILD-BSP-TSTFIXME.yml changes each file in
the link tree as we work up to the root?

> For the "set-test-state" action see:
> 
> https://git.rtems.org/sebh/rtems.git/tree/wscript?h=build#n593

Nice.

> Each test has a TEST_EXCLUDE_<NAME> enable and a "cppflags" attribute set to "-
> ${TEST_CPPFLAGS_<NAME>}".
> 
> The enable and cppflags are set by the "set-test-state" action and are stored in
> the environment:
> 
> $ grep -r TEST_ build/c4che/sparc/erc32_cache.py
> ENABLE = ['sparc', 'erc32', 'RTEMS_NETWORKING', 'RTEMS_NEWLIB',
> 'RTEMS_POSIX_API', 'RTEMS_SMP', 'BUILD_BENCHMARKS', 'BUILD_FSTESTS',
> 'BUILD_LIBTESTS', 'BUILD_MPTESTS', 'BUILD_PSXTESTS', 'BUILD_PSXTMTESTS',
> 'BUILD_RHEALSTONE', 'BUILD_SAMPLES', 'BUILD_SMPTESTS', 'BUILD_SPTESTS',
> 'BUILD_TMTESTS', 'TEST_EXCLUDE_DL04', 'TEST_EXCLUDE_DL05', 'TEST_EXCLUDE_DL06',
> 'TEST_EXCLUDE_DL07', 'TEST_EXCLUDE_DL01', 'TEST_EXCLUDE_DL02',
> 'TEST_EXCLUDE_DL03', 'TEST_EXCLUDE_TAR01', 'TEST_EXCLUDE_TAR02',
> 'TEST_EXCLUDE_TAR03', 'TEST_EXCLUDE_DL08', 'TEST_EXCLUDE_DL09',
> 'TEST_EXCLUDE_DL10', 'TEST_EXCLUDE_MGHTTPD01']
> TEST_CPPFLAGS_CAPTURE = ['-DTEST_STATE_USER_INPUT=1']
> TEST_CPPFLAGS_DHRYSTONE = ['-DTEST_STATE_BENCHMARK=1']
> TEST_CPPFLAGS_FILEIO = ['-DTEST_STATE_USER_INPUT=1']
> TEST_CPPFLAGS_LINPACK = ['-DTEST_STATE_BENCHMARK=1']
> TEST_CPPFLAGS_MONITOR = ['-DTEST_STATE_USER_INPUT=1']
> TEST_CPPFLAGS_PSXFENV01 = ['-DTEST_STATE_EXPECTED_FAIL=1']
> TEST_CPPFLAGS_TERMIOS = ['-DTEST_STATE_USER_INPUT=1']
> TEST_CPPFLAGS_TOP = ['-DTEST_STATE_USER_INPUT=1']
> TEST_CPPFLAGS_WHETSTONE = ['-DTEST_STATE_BENCHMARK=1']

Looks good.

I think in time we will need a reporting tool for the spec data in the source
tree, for example a way to get the list of tests that are not an expected-pass.
Does Doorstop provide easy to run specialised reports?

Chris