*.tcfg specification?

Tue Nov 12 14:30:24 UTC 2019

Hello Chris,

thanks for your comments. I converted now all tests except the Ada 
tests, tests which use the pax tool, and the dl* tests. For an approach 
to address the test states please see below.

On 11/11/2019 23:25, Chris Johns wrote:
> On 11/11/19 7:13 pm, Sebastian Huber wrote:
>> what is the purpose of the *.tcfg test configuration files?
> 
> The tcfg files provide a way to implement the "test controls" ...
> 
> https://docs.rtems.org/branches/master/user/testing/tests.html#test-controls
> 
> .. as well as excluding a test. A test executable that is not excluded must
> build and be loadable on a BSP target and executed.

There are some *norun* tests.

> Tests can be excluded due to
> an architecture issue (lacking an architecture feature), BSP issue (not enough
> memory) or an RTEMS or tools support related issue (missing feature). A build
> test run on a BSP target uses the "test banners" to publish it's state ...
> 
> https://docs.rtems.org/branches/master/user/testing/tests.html#
> 
> The existing implementation is a mix of build system controls to exclude a test
> and command line defines to control the build in a standard and controlled way.
> The rtems-test tool understands the test banners and adapts it's control based
> on what the test reports. For example a `user-input` test is terminated and the
> next test is run.

Ok, it seems a test program has exactly one state for a given BSP.

> 
>> You can disable compilation of test programs with them, e.g.
>>
>> exclude: flashdisk01
> 
> Yes. There is also `rexclude` a regex filter, for example `rexclude: dl[0-9][0-9]`.
> 
>> The "testsuites/rtems-test-check.py" contains a bit more stuff. What is really
>> needed?
> 
> I suspect you will end up needing to match the functionality provided. What
> rtems-test-check.py supports has grown since I introduced this support. It
> started off as a simple script however it did not do what we needed and adding
> features cause the performance to blow out so I switch to python and since then
> more complexity has been added. The script is currently run once for each test,
> that can be avoided with waf.

I don't want to use this script in the new build system. I want to map 
the *.tcfg functionality to the build specification and use the means 
that are already there as far as possible. I would like to avoid too 
many special cases.

> 
>> I have to map this to the new build system.
> 
> Yes you will. The mapping of the tcfg files is more complex than it may appear.
> The complexity is in what we need to represent and not in the implementation. In
> a waf build system the configure phase could be a single pass where the tcfg
> files are loaded in a single pass and then held in memory while the build
> instance for each test is created.
> 
> A quick summary:
> 
> 1. includes
> 2. excludes and include with regular expression filtering, ie exclude a bunch
> then include something specific
> 3. test controls
> 
> Include support allows a single definition of a test profile and reuse across a
> number of BSPs. The use of regex filters allow for subtle variations of how this
> works.

In theory, we could support regular expressions in the "enabled-by" 
attribute. Is this really necessary? It shouldn't be an issue to 
enumerate all tests that belong to a certain group.

$ grep --include='*.tcfg' rexclude -r .
./testsuites/testdata/disable-libdl-tests.tcfg:rexclude: dl[0-9][0-9]

It is only used for the dl* tests and it would be simple to just give a 
list of tests.

> 
>> One approach is to use the "enabled-by" attribute to disable test programs on
>> demand, e.g.:
>>
>> $ cat spec/build/testsuites/samples/RTEMS-BUILD-TEST-SAMPLE-TICKER.yml
> 
> Hmm, I had not noticed .yml is being used. By default emacs does not know this
> is a YAML format file. Why has this been selected over .yaml? Having to teach
> editors about a new extension is painful.

Doorstop uses this extension. We would have to adjust doorstop to use 
another extension.

> 
>> active: true
>> build-type: test-program
>> cflags: []
>> cppflags: []
>> cxxflags: []
>> derived: false
>> enabled-by:
>> - not: DISABLE_TEST_SAMPLE_TICKER
>> features: c cprogram
>> header: ''
>> includes: []
>> level: 1.15
>> links: []
>> normative: true
>> order: 0
>> ref: ''
>> reviewed: null
>> source:
>> - testsuites/samples/ticker/init.c
>> - testsuites/samples/ticker/tasks.c
>> stlib: []
>> target: testsuites/samples/ticker.exe
>> text: ''
>> type: build
>> use: []
>>
>> A BSP can use a "build-type: option" file to disable tests, e.g.
>>
>> actions:
>> - disable-tests:
>>    - SAMPLE/TICKER
>>    - LIB/FLASHDISK01
> 
> I would be careful before heading down this path. Lets assume 150 BSPs and 570
> tests, that is 85,500 tests and test states. Adding a modest number of test
> cases could effect the readability of these files. I think you will need to
> structure this rather than having it expressed as a flat list.

You will have a flat list per BSP (with includes via item "links") just 
as you have it now with the *.tcfg files.

> 
> There are 77 tcfg files. There are top level definitions that are used by BSPs:
> 
>   testsuites/testdata/disable-intrcritical-tests.tcfg
>   testsuites/testdata/disable-libdl-tests.tcfg
>   testsuites/testdata/disable-mrfs-tests.tcfg
>   testsuites/testdata/rtems.tcfg
>   testsuites/testdata/small-memory-testsuite.tcfg
>   testsuites/testdata/require-tick-isr.tcfg
>   testsuites/testdata/disable-jffs2-tests.tcfg
>   testsuites/testdata/disable-iconv-tests.tcfg

You can do includes through the "links" attribute. The number of test 
state specification items will be roughly the same as the *.tcfg file count.

> 
> The rtems.tcfg defines the state of tests common to all BSPs, for example
> `user-input` and `benchmark`. These are invariant. Others like
> small-memory-testsuite.tcfg are more complex including other test profiles like
> disable-iconv-tests.tcfg etc.
> 
> I can see us having test profiles based on an architecture or BSP family before
> you reach the BSP. I have not checked to see if this is already happening.
> 
> If you could structure the dependency in away that reflected what there is it
> would help. For example tms570ls3137_hdk_with_loader-testsuite.tcfg includes
> small-memory-testsuite.tcfg plus it excludes linpack. If that BSPs spec was just
> `small-memory` and `exclude: linpack` it would be readable and maintainable.
> 
> I understand wanting to use the spec support however I do not know enough about
> it to offer a specific solution.

The "links" attribute is a very important thing in the specification 
item business. I guess I have to add some dot images to the software 
requirements engineering chapter.

> 
>> An open issue is how and if we should set other states, e.g. expected to fail.
>> We could generate a "bsp/teststate.h" file for each BSP which optionally defines
>> for each test a state other than "pass", e.g.
>>
>> #define TEST_SAMPLE_TICKER_STATE RTEMS_TEST_STATE_FAIL
> 
> Why not stay with defines for this specific case? I understand and agree the
> issue of figuring out what can happen with a build because you need to
> understand the command line but in this case the parameters are bounded and
> known. You need a build instance for each test a BSP has and you can add a
> specific set of defines to each build instance. Waf will track the flags in a
> build and any changes will rebuild the dependent targets.
> 
>> This could be also realized with a special option action:
>>
>> actions:
>> - set-test-states:
>>    - SAMPLE/TICKER: disable
>>    - LIB/FLASHDISK01: xfail
> 
> I have avoided abbreviations so far and used `expected-fail`. I would like it to
> stay that way if that is OK.
> 
>> The "set-test-states" action can validate if the state exists. I can use the
>> "enable-by" for "disable" states and the "bsp/teststate.h" file for other states.
> 
> My concerns with a single test state header are:
> 
> 1. size, we have a lot of tests

It doesn't matter much where you place the information.  With the 
CPPFLAGS approach the information just moves to the configuration set 
environment.

> 2. slipping into adding things which should not be there but could be, ie adding
> C code to control something
> 3. potential cross-talk between tests, ie I borrow a piece of code from another
> test and do not change a dependent define.

I think 2. and 3. are theoretical issues since the content of the header 
file would be determined by the build option item actions.

> 
> I also like the idea of a single place to inspect how tests are configured.

Currently you have an optional *.tcfg per base BSP variant with an 
ability to include files. You also have a magic rtems.tcfg which is 
added by default. This is not a single place to inspect from my point of 
view. However, I don't think a good alternative exists.

I re-implemented now practically the *.tcfg files and the existing 
mechanism (module the regular expressions). You can find an example here:

https://git.rtems.org/sebh/rtems.git/tree/spec/build/bsps/RTEMS-BUILD-BSP-TSTFIXME.yml?h=build

This is more or less the old rtems.tcfg.

It is pulled in by:

https://git.rtems.org/sebh/rtems.git/tree/spec/build/bsps/RTEMS-BUILD-BSP-BSPOPTS.yml?h=build

Which in turn is included by every BSP, e.g.

https://git.rtems.org/sebh/rtems.git/tree/spec/build/bsps/sparc/erc32/RTEMS-BUILD-BSP-SPARC-ERC32-ERC32.yml?h=build

For the "set-test-state" action see:

https://git.rtems.org/sebh/rtems.git/tree/wscript?h=build#n593

Each test has a TEST_EXCLUDE_<NAME> enable and a "cppflags" attribute 
set to "- ${TEST_CPPFLAGS_<NAME>}".

The enable and cppflags are set by the "set-test-state" action and are 
stored in the environment:

$ grep -r TEST_ build/c4che/sparc/erc32_cache.py
ENABLE = ['sparc', 'erc32', 'RTEMS_NETWORKING', 'RTEMS_NEWLIB', 
'RTEMS_POSIX_API', 'RTEMS_SMP', 'BUILD_BENCHMARKS', 'BUILD_FSTESTS', 
'BUILD_LIBTESTS', 'BUILD_MPTESTS', 'BUILD_PSXTESTS', 'BUILD_PSXTMTESTS', 
'BUILD_RHEALSTONE', 'BUILD_SAMPLES', 'BUILD_SMPTESTS', 'BUILD_SPTESTS', 
'BUILD_TMTESTS', 'TEST_EXCLUDE_DL04', 'TEST_EXCLUDE_DL05', 
'TEST_EXCLUDE_DL06', 'TEST_EXCLUDE_DL07', 'TEST_EXCLUDE_DL01', 
'TEST_EXCLUDE_DL02', 'TEST_EXCLUDE_DL03', 'TEST_EXCLUDE_TAR01', 
'TEST_EXCLUDE_TAR02', 'TEST_EXCLUDE_TAR03', 'TEST_EXCLUDE_DL08', 
'TEST_EXCLUDE_DL09', 'TEST_EXCLUDE_DL10', 'TEST_EXCLUDE_MGHTTPD01']
TEST_CPPFLAGS_CAPTURE = ['-DTEST_STATE_USER_INPUT=1']
TEST_CPPFLAGS_DHRYSTONE = ['-DTEST_STATE_BENCHMARK=1']
TEST_CPPFLAGS_FILEIO = ['-DTEST_STATE_USER_INPUT=1']
TEST_CPPFLAGS_LINPACK = ['-DTEST_STATE_BENCHMARK=1']
TEST_CPPFLAGS_MONITOR = ['-DTEST_STATE_USER_INPUT=1']
TEST_CPPFLAGS_PSXFENV01 = ['-DTEST_STATE_EXPECTED_FAIL=1']
TEST_CPPFLAGS_TERMIOS = ['-DTEST_STATE_USER_INPUT=1']
TEST_CPPFLAGS_TOP = ['-DTEST_STATE_USER_INPUT=1']
TEST_CPPFLAGS_WHETSTONE = ['-DTEST_STATE_BENCHMARK=1']

-- 
Sebastian Huber, embedded brains GmbH

Address : Dornierstr. 4, D-82178 Puchheim, Germany
Phone   : +49 89 189 47 41-16
Fax     : +49 89 189 47 41-09
E-Mail  : sebastian.huber at embedded-brains.de
PGP     : Public key available on request.

Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG.