*.tcfg specification?

Tue Nov 12 23:09:35 UTC 2019

On Tue, Nov 12, 2019 at 4:42 PM Chris Johns <chrisj at rtems.org> wrote:

> On 13/11/19 1:30 am, Sebastian Huber wrote:
> > Hello Chris,
> >
> > thanks for your comments. I converted now all tests except the Ada
> tests,
>
> Nice.
>
> > tests which use the pax tool,
>
> This might help ...
>
> https://git.rtems.org/rtems_waf/tree/rootfs.py
>
> >  and the dl* tests.
>
> Plus this ...
>
> https://git.rtems.org/rtems_waf/tree/dl.py
>
> > For an approach to address the test
> > states please see below.
> >
> > On 11/11/2019 23:25, Chris Johns wrote:
> >> On 11/11/19 7:13 pm, Sebastian Huber wrote:
> >>> what is the purpose of the *.tcfg test configuration files?
> >>
> >> The tcfg files provide a way to implement the "test controls" ...
> >>
> >>
> https://docs.rtems.org/branches/master/user/testing/tests.html#test-controls
> >>
> >> .. as well as excluding a test. A test executable that is not excluded
> must
> >> build and be loadable on a BSP target and executed.
> >
> > There are some *norun* tests.
> >
>
> Ah yes. I seem to remember we gave the tests a different extension so
> rtems-test
> does not find them?
>
> >> Tests can be excluded due to
> >> an architecture issue (lacking an architecture feature), BSP issue (not
> enough
> >> memory) or an RTEMS or tools support related issue (missing feature). A
> build
> >> test run on a BSP target uses the "test banners" to publish it's state
> ...
> >>
> >> https://docs.rtems.org/branches/master/user/testing/tests.html#
> >>
> >> The existing implementation is a mix of build system controls to
> exclude a test
> >> and command line defines to control the build in a standard and
> controlled way.
> >> The rtems-test tool understands the test banners and adapts it's
> control based
> >> on what the test reports. For example a `user-input` test is terminated
> and the
> >> next test is run.
> >
> > Ok, it seems a test program has exactly one state for a given BSP.
>
> Yes, this is driven by the ABI, hardware profile and similar constraints.
>

This is generally the case but we have a few BSPs where the same executable
runs on real hardware and multiple simulators. The SPARC BSPs are at the
top of this list. The pc386 is another example.

We currently have no way to specify that a test could pass on real hardware,
fail on simulator 1 and pass on simulator 2. For the SPARC BSPs, we now
have real HW, sis, qemu and tsim.

The .tcfg files don't begin to address this and I really don't have any good
ideas myself. It is just a very real life case that it would be nice to
address
so we have 100% expected passes on all execution platforms.

>
> >>> You can disable compilation of test programs with them, e.g.
> >>>
> >>> exclude: flashdisk01
> >>
> >> Yes. There is also `rexclude` a regex filter, for example `rexclude:
> >> dl[0-9][0-9]`.
> >>
> >>> The "testsuites/rtems-test-check.py" contains a bit more stuff. What
> is really
> >>> needed?
> >>
> >> I suspect you will end up needing to match the functionality provided.
> What
> >> rtems-test-check.py supports has grown since I introduced this support.
> It
> >> started off as a simple script however it did not do what we needed and
> adding
> >> features cause the performance to blow out so I switch to python and
> since then
> >> more complexity has been added. The script is currently run once for
> each test,
> >> that can be avoided with waf.
> >
> > I don't want to use this script in the new build system.
>
> That is understandable. It is a purpose built script for the existing build
> system. I think the functionality it provided is the important part.
>
> > I want to map the
> > *.tcfg functionality to the build specification and use the means that
> are
> > already there as far as possible. I would like to avoid too many special
> cases.
>
> I am not sure I understand what you mean by special cases? Providing a
> exact fit
> does not give us space to grow and we all change shape over time, it
> cannot be
> avoided. I am a living example of that.
>
> >>> I have to map this to the new build system.
> >>
> >> Yes you will. The mapping of the tcfg files is more complex than it may
> appear.
> >> The complexity is in what we need to represent and not in the
> implementation. In
> >> a waf build system the configure phase could be a single pass where the
> tcfg
> >> files are loaded in a single pass and then held in memory while the
> build
> >> instance for each test is created.
> >>
> >> A quick summary:
> >>
> >> 1. includes
> >> 2. excludes and include with regular expression filtering, ie exclude a
> bunch
> >> then include something specific
> >> 3. test controls
> >>
> >> Include support allows a single definition of a test profile and reuse
> across a
> >> number of BSPs. The use of regex filters allow for subtle variations of
> how this
> >> works.
> >
> > In theory, we could support regular expressions in the "enabled-by"
> attribute.
> > Is this really necessary? It shouldn't be an issue to enumerate all
> tests that
> > belong to a certain group.
> >
> > $ grep --include='*.tcfg' rexclude -r .
> > ./testsuites/testdata/disable-libdl-tests.tcfg:rexclude: dl[0-9][0-9]
>
> The support was added because I needed to update the list of excluded libdl
> tests when a new test was added and I seemed to always forget until a while
> later Joel pings me with a breakage. A user adding a new libdl test would
> not
> know a change was needed.
>
> The functionality lets us name tests of a specific class or category and
> then
> manage that class.
>
> > It is only used for the dl* tests and it would be simple to just give a
> list of
> > tests.
>
> This is a case of something being present and it's use grows as we find
> more
> cases of it being needed. It is about providing a framework and not a
> specific
> fit. I would not place too much on there being only one instance of its
> use.
>
> >>> One approach is to use the "enabled-by" attribute to disable test
> programs on
> >>> demand, e.g.:
> >>>
> >>> $ cat spec/build/testsuites/samples/RTEMS-BUILD-TEST-SAMPLE-TICKER.yml
> >>
> >> Hmm, I had not noticed .yml is being used. By default emacs does not
> know this
> >> is a YAML format file. Why has this been selected over .yaml? Having to
> teach
> >> editors about a new extension is painful.
> >
> > Doorstop uses this extension. We would have to adjust doorstop to use
> another
> > extension.
>
> That is a shame.
>
> >>> active: true
> >>> build-type: test-program
> >>> cflags: []
> >>> cppflags: []
> >>> cxxflags: []
> >>> derived: false
> >>> enabled-by:
> >>> - not: DISABLE_TEST_SAMPLE_TICKER
> >>> features: c cprogram
> >>> header: ''
> >>> includes: []
> >>> level: 1.15
> >>> links: []
> >>> normative: true
> >>> order: 0
> >>> ref: ''
> >>> reviewed: null
> >>> source:
> >>> - testsuites/samples/ticker/init.c
> >>> - testsuites/samples/ticker/tasks.c
> >>> stlib: []
> >>> target: testsuites/samples/ticker.exe
> >>> text: ''
> >>> type: build
> >>> use: []
> >>>
> >>> A BSP can use a "build-type: option" file to disable tests, e.g.
> >>>
> >>> actions:
> >>> - disable-tests:
> >>>    - SAMPLE/TICKER
> >>>    - LIB/FLASHDISK01
> >>
> >> I would be careful before heading down this path. Lets assume 150 BSPs
> and 570
> >> tests, that is 85,500 tests and test states. Adding a modest number of
> test
> >> cases could effect the readability of these files. I think you will
> need to
> >> structure this rather than having it expressed as a flat list.
> >
> > You will have a flat list per BSP (with includes via item "links") just
> as you
> > have it now with the *.tcfg files.
>
> Perfect. There is no need to have the same implementation in the new build
> system.
>
> We need to make sure we capture the important pieces of information we
> have now
> that has taken a few years to build up ....
>
> $ grep 'small-memory-testsuite' `find . -name \*.tcfg` | wc -l
>       27
>
> We have 27 small memory BSPs (remove the line count and you see the list).
> If a
> new test is added that cannot run on small memory BSPs we need a simple
> way in
> one location to update the list. This applies to each of the top level test
> controls we have.
>
> >> There are 77 tcfg files. There are top level definitions that are used
> by BSPs:
> >>
> >>   testsuites/testdata/disable-intrcritical-tests.tcfg
> >>   testsuites/testdata/disable-libdl-tests.tcfg
> >>   testsuites/testdata/disable-mrfs-tests.tcfg
> >>   testsuites/testdata/rtems.tcfg
> >>   testsuites/testdata/small-memory-testsuite.tcfg
> >>   testsuites/testdata/require-tick-isr.tcfg
> >>   testsuites/testdata/disable-jffs2-tests.tcfg
> >>   testsuites/testdata/disable-iconv-tests.tcfg
> >
> > You can do includes through the "links" attribute. The number of test
> state
> > specification items will be roughly the same as the *.tcfg file count.
>
> Nice.
>
> >> The rtems.tcfg defines the state of tests common to all BSPs, for
> example
> >> `user-input` and `benchmark`. These are invariant. Others like
> >> small-memory-testsuite.tcfg are more complex including other test
> profiles like
> >> disable-iconv-tests.tcfg etc.
> >>
> >> I can see us having test profiles based on an architecture or BSP
> family before
> >> you reach the BSP. I have not checked to see if this is already
> happening.
> >>
> >> If you could structure the dependency in away that reflected what there
> is it
> >> would help. For example tms570ls3137_hdk_with_loader-testsuite.tcfg
> includes
> >> small-memory-testsuite.tcfg plus it excludes linpack. If that BSPs spec
> was just
> >> `small-memory` and `exclude: linpack` it would be readable and
> maintainable.
> >>
> >> I understand wanting to use the spec support however I do not know
> enough about
> >> it to offer a specific solution.
> >
> > The "links" attribute is a very important thing in the specification item
> > business.
>
> I agree.
>
> > I guess I have to add some dot images to the software requirements
> > engineering chapter.
> >
> >>> An open issue is how and if we should set other states, e.g. expected
> to fail.
> >>> We could generate a "bsp/teststate.h" file for each BSP which
> optionally defines
> >>> for each test a state other than "pass", e.g.
> >>>
> >>> #define TEST_SAMPLE_TICKER_STATE RTEMS_TEST_STATE_FAIL
> >>
> >> Why not stay with defines for this specific case? I understand and
> agree the
> >> issue of figuring out what can happen with a build because you need to
> >> understand the command line but in this case the parameters are bounded
> and
> >> known. You need a build instance for each test a BSP has and you can
> add a
> >> specific set of defines to each build instance. Waf will track the
> flags in a
> >> build and any changes will rebuild the dependent targets.
> >>
> >>> This could be also realized with a special option action:
> >>>
> >>> actions:
> >>> - set-test-states:
> >>>    - SAMPLE/TICKER: disable
> >>>    - LIB/FLASHDISK01: xfail
> >>
> >> I have avoided abbreviations so far and used `expected-fail`. I would
> like it to
> >> stay that way if that is OK.
> >>
> >>> The "set-test-states" action can validate if the state exists. I can
> use the
> >>> "enable-by" for "disable" states and the "bsp/teststate.h" file for
> other
> >>> states.
> >>
> >> My concerns with a single test state header are:
> >>
> >> 1. size, we have a lot of tests
> >
> > It doesn't matter much where you place the information.  With the
> CPPFLAGS
> > approach the information just moves to the configuration set environment.
>
> I am not sure yet. I suppose in time we will find out.
>
> >> 2. slipping into adding things which should not be there but could be,
> ie adding
> >> C code to control something
> >> 3. potential cross-talk between tests, ie I borrow a piece of code from
> another
> >> test and do not change a dependent define.
> >
> > I think 2. and 3. are theoretical issues since the content of the header
> file
> > would be determined by the build option item actions.
>
> Item 2 is about a policy for changes in this area and rejecting patches
> that
> step into adding C code type functionality.
>
> I can see 3. happening, it is only a matter of time.
>
> >> I also like the idea of a single place to inspect how tests are
> configured.
> >
> > Currently you have an optional *.tcfg per base BSP variant with an
> ability to
> > include files. You also have a magic rtems.tcfg which is added by
> default. This
> > is not a single place to inspect from my point of view. However, I don't
> think a
> > good alternative exists.
>
> Yes, this is a weakness with the tcfg files that I was going to solve by
> adding
> a command to the top level of the source tree to provide reports. I
> stopped this
> when you started on this build system task.
>
> I have been asking for months, hmmm maybe a year or more for the tcfg
> files to
> be updated with expected-fail states so rtems-test can determine
> regressions. I
> had started to wonder if core maintainers were no sure how to make these
> changes
> and that is reasonable because there was no documentation and no method to
> report the effect of the change without a full build and test run. I
> needed to
> make the process easier and simpler for it be used.
>
> > I re-implemented now practically the *.tcfg files and the existing
> mechanism
> > (module the regular expressions). You can find an example here:
> >
> >
> https://git.rtems.org/sebh/rtems.git/tree/spec/build/bsps/RTEMS-BUILD-BSP-TSTFIXME.yml?h=build
>
> Is the test name always in caps?
>
> What about ...
>
>  DL[0-9][0-9]: exclude
>
> ? :)
>
> > This is more or less the old rtems.tcfg.
> >
> > It is pulled in by:
> >
> >
> https://git.rtems.org/sebh/rtems.git/tree/spec/build/bsps/RTEMS-BUILD-BSP-BSPOPTS.yml?h=build
>
> How are links and their checksum maintained?
>
> > Which in turn is included by every BSP, e.g.
> >
> >
> https://git.rtems.org/sebh/rtems.git/tree/spec/build/bsps/sparc/erc32/RTEMS-BUILD-BSP-SPARC-ERC32-ERC32.yml?h=build
>
> Does this means a change in RTEMS-BUILD-BSP-TSTFIXME.yml changes each file
> in
> the link tree as we work up to the root?
>
> > For the "set-test-state" action see:
> >
> > https://git.rtems.org/sebh/rtems.git/tree/wscript?h=build#n593
>
> Nice.
>
> > Each test has a TEST_EXCLUDE_<NAME> enable and a "cppflags" attribute
> set to "-
> > ${TEST_CPPFLAGS_<NAME>}".
> >
> > The enable and cppflags are set by the "set-test-state" action and are
> stored in
> > the environment:
> >
> > $ grep -r TEST_ build/c4che/sparc/erc32_cache.py
> > ENABLE = ['sparc', 'erc32', 'RTEMS_NETWORKING', 'RTEMS_NEWLIB',
> > 'RTEMS_POSIX_API', 'RTEMS_SMP', 'BUILD_BENCHMARKS', 'BUILD_FSTESTS',
> > 'BUILD_LIBTESTS', 'BUILD_MPTESTS', 'BUILD_PSXTESTS', 'BUILD_PSXTMTESTS',
> > 'BUILD_RHEALSTONE', 'BUILD_SAMPLES', 'BUILD_SMPTESTS', 'BUILD_SPTESTS',
> > 'BUILD_TMTESTS', 'TEST_EXCLUDE_DL04', 'TEST_EXCLUDE_DL05',
> 'TEST_EXCLUDE_DL06',
> > 'TEST_EXCLUDE_DL07', 'TEST_EXCLUDE_DL01', 'TEST_EXCLUDE_DL02',
> > 'TEST_EXCLUDE_DL03', 'TEST_EXCLUDE_TAR01', 'TEST_EXCLUDE_TAR02',
> > 'TEST_EXCLUDE_TAR03', 'TEST_EXCLUDE_DL08', 'TEST_EXCLUDE_DL09',
> > 'TEST_EXCLUDE_DL10', 'TEST_EXCLUDE_MGHTTPD01']
> > TEST_CPPFLAGS_CAPTURE = ['-DTEST_STATE_USER_INPUT=1']
> > TEST_CPPFLAGS_DHRYSTONE = ['-DTEST_STATE_BENCHMARK=1']
> > TEST_CPPFLAGS_FILEIO = ['-DTEST_STATE_USER_INPUT=1']
> > TEST_CPPFLAGS_LINPACK = ['-DTEST_STATE_BENCHMARK=1']
> > TEST_CPPFLAGS_MONITOR = ['-DTEST_STATE_USER_INPUT=1']
> > TEST_CPPFLAGS_PSXFENV01 = ['-DTEST_STATE_EXPECTED_FAIL=1']
> > TEST_CPPFLAGS_TERMIOS = ['-DTEST_STATE_USER_INPUT=1']
> > TEST_CPPFLAGS_TOP = ['-DTEST_STATE_USER_INPUT=1']
> > TEST_CPPFLAGS_WHETSTONE = ['-DTEST_STATE_BENCHMARK=1']
>
> Looks good.
>
> I think in time we will need a reporting tool for the spec data in the
> source
> tree, for example a way to get the list of tests that are not an
> expected-pass.
> Does Doorstop provide easy to run specialised reports?
>
> Chris
> _______________________________________________
> devel mailing list
> devel at rtems.org
> http://lists.rtems.org/mailman/listinfo/devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rtems.org/pipermail/devel/attachments/20191112/d86788e9/attachment-0001.html>