[PATCH 00/33] Test framework improvements
Chris Johns
chrisj at rtems.org
Wed Jul 22 05:04:47 UTC 2020
On 22/7/20 1:04 am, Sebastian Huber wrote:
> This patch set adds a couple of improvements to the test framework:
>
> * The header file changes from <t.h> to <rtems/test.h>.
>
> * Support for a stack of test fixtures. This helps to write test
> building blocks.
>
> * The test check messages are now optional.
>
> * The support for interrupt tests and change all the interrupt critical
> tests to use this new test support. This should address the sporadic
> failures and timeouts.
>
> For the documentation please have a look at:
>
> https://ftp.rtems.org/pub/rtems/people/sebh/eng2.pdf
>
I assume you asking us to look only at section 8?
The doco looks good. The following are some review comment:
8.1
a) Supported languages?
8.1.1
a) The wording ".. outcome all right, ..." seems non-specific for someting I
would assume needs to be specific. What about " ... is not the required outcome
for the test, ..."?
b) As previously discussed there are a few more states a test result can be
other than pass and fail and I thikn the wording here may need tightening. I am
not sure what is needed. Also how do resource leaks effect the result?
8.1.2
a) It test case naming to be specified or can be use anything we like, ie lower
case with `_`, Camelcase, ...?
b) Change "static constructors" to "static C constructors".
8.1.3
a) Change "test case as well as to give thescope for log messags." to " test
case as well as the scope for log messages."
b) Change "function. It can be set within the scope of one .." to "function and
can be set within the scope of one .."
c) Does the framework provide a standard way to export a dynamic test
environment so the test and the results can be reviewed as a complete set of data?
8.1.4
a) Should "Each non-quiet test check fetches.." be "A non-quiet test check
fetches.."?
8.1.5
a) Change "..if various resources are leaked .." to ".. if various resources
have leaked ..".
8.1.7
a) Change "You can add test case destructors with T_add_destructor(). They
are.." to "A test case destructor can be added with T_add_destructor(). The
destructors are..".
b) Maybe add it is best to use statically allocated memory only?
8.1.8
a) "Meet its expecation" ... I am not sure what this means, who expecation and
what does expecation mean? Do you mean " .. test check meets the check's
exepected outcome."?
b) I am not sure what "The actual value ..." is referring to. The value the test
expects to be the input from the test to the check? Ah coming back to here, I
see this is in the next bit. Would adding a little bit in the text before using
the special words like actual and expected be helpful?
c) I think the T_assert_ etc variants of the test checks should be explained
somewhere. I think it is important to know tests can continue even after a fail
or they can be made to stop. I had noted I had not seen this documented and I
have now found it buried in here.
d) I think the word `available` can be removed from the initial sentence on all
sub-sectons?
8.1.8.10
a) I think type variant piece of test should be moved to the top and add to the
initial sentence so you have " are available where the type variant `xyz` must
be ...".
8.1.8.12
a) What about 0 or EINTR as OK? :)
8.1.6
a) Are the printf variants check for type correctness by the compiler? If so I
suggest this is mentioned.
8.1.11
a) Can "You can .." be removed so it is not person specific? For example "You
can convert time into ticks with the" ciuld be "The time can be converted into
ticks with the"? This comment covers all the "You can"s. :)
8.1.13
a) What is the picture drawn in?
I will comment on the patches as well.
Chris
> Sebastian Huber (33):
> libtest: <rtems/test.h> to <rtems/simple-test.h>
> libtest: Move <t.h> to <rtems/test.h>
> libtest: Add T_busy()
> libtest: Add T_get_one_clock_tick_busy()
> libtest: Add T_make_runner()
> libtest: Support custom scope messages via fixture
> libtest: Add push/pop fixture support
> libtest: Add T_get_scope()
> libtest: Split POSIX Keys support
> libtest: Add T_stop()
> libtest: Add T_CHECK_FMT
> libtest: Make check message optional
> libtest: Add T_unreachable()
> libtest: Add quiet assert NULL pointer checks
> libtest: Add rtems_test_run()
> libtest: Add T_interrupt_test()
> psxintrcritical01: Use T_interrupt_test()
> spintrcritical01/2/3/4/5: Use T_interrupt_test()
> spintrcritical06/spintrcritical07: Remove tests
> spintrcritical08: Use T_interrupt_test()
> spintrcritical09: Use T_interrupt_test()
> spintrcritical10: Use T_interrupt_test()
> spintrcritical11/12: Use T_interrupt_test()
> spintrcritical13/14: Use T_interrupt_test()
> spintrcritical15: Use T_interrupt_test()
> spintrcritical16: Use T_interrupt_test()
> spintrcritical18: Use T_interrupt_test()
> spintrcritical20: Use T_interrupt_test()
> spintrcritical21: Use T_interrupt_test()
> spintrcritical22: Use T_interrupt_test()
> spintrcritical23: Use T_interrupt_test()
> spintrcritical24: Use T_interrupt_test()
> spintrcritical_support: Remove
>
> cpukit/Makefile.am | 5 +
> cpukit/headers.am | 3 +-
> cpukit/include/rtems/doxygen-appl-config.h | 3736 +++++++++++++++++
> cpukit/include/rtems/simple-test.h | 327 ++
> cpukit/include/rtems/test.h | 2653 ++++++++++--
> cpukit/include/t.h | 2379 -----------
> cpukit/libtest/t-test-busy-tick.c | 121 +
> cpukit/libtest/t-test-busy.c | 58 +
> cpukit/libtest/t-test-checks-eno.c | 10 +-
> cpukit/libtest/t-test-checks-psx.c | 10 +-
> cpukit/libtest/t-test-checks.c | 194 +-
> cpukit/libtest/t-test-hash-sha256.c | 2 +-
> cpukit/libtest/t-test-interrupt.c | 420 ++
> cpukit/libtest/t-test-malloc.c | 2 +-
> cpukit/libtest/t-test-rtems-context.c | 8 +-
> cpukit/libtest/t-test-rtems-fds.c | 5 +-
> cpukit/libtest/t-test-rtems-heap.c | 6 +-
> cpukit/libtest/t-test-rtems-measure.c | 2 +-
> cpukit/libtest/t-test-rtems-objs.c | 100 +-
> cpukit/libtest/t-test-rtems-posix-keys.c | 116 +
> cpukit/libtest/t-test-rtems.c | 10 +-
> cpukit/libtest/t-test-rtems.h | 62 +
> cpukit/libtest/t-test-time.c | 2 +-
> cpukit/libtest/t-test.c | 279 +-
> cpukit/libtest/testbeginend.c | 2 +-
> cpukit/libtest/testbusy.c | 89 +-
> cpukit/libtest/testextension.c | 2 +-
> cpukit/libtest/testparallel.c | 2 +-
> cpukit/libtest/testrun.c | 87 +
> cpukit/libtest/testwrappers.c | 2 +-
> spec/build/cpukit/librtemstest.yml | 1 +
> testsuites/ada/support/initimpl.h | 2 +-
> testsuites/libtests/Makefile.am | 9 +
> testsuites/libtests/block08/system.h | 2 +-
> testsuites/libtests/block09/init.c | 2 +-
> testsuites/libtests/block10/init.c | 2 +-
> testsuites/libtests/capture01/system.h | 2 +-
> testsuites/libtests/configure.ac | 1 +
> testsuites/libtests/dl01/dl01-o1.c | 2 +-
> testsuites/libtests/dl02/dl02-o1.c | 2 +-
> testsuites/libtests/dl02/dl02-o2.c | 2 +-
> testsuites/libtests/dl05/dl05-o5.cc | 2 +-
> testsuites/libtests/dl06/dl06-o1.c | 2 +-
> testsuites/libtests/dl06/dl06-o2.c | 2 +-
> testsuites/libtests/dl07/dl07-o1.c | 2 +-
> testsuites/libtests/dl07/dl07-o2.c | 2 +-
> testsuites/libtests/dl07/dl07-o3.c | 2 +-
> testsuites/libtests/dl07/dl07-o4.c | 2 +-
> testsuites/libtests/dl07/dl07-o5.c | 2 +-
> testsuites/libtests/dl08/dl08-o1.c | 2 +-
> testsuites/libtests/dl08/dl08-o2.c | 2 +-
> testsuites/libtests/dl08/dl08-o3.c | 2 +-
> testsuites/libtests/dl08/dl08-o4.c | 2 +-
> testsuites/libtests/dl08/dl08-o5.c | 2 +-
> .../dl08/dl08-o6-123456789-123456789.c | 2 +-
> testsuites/libtests/dl09/dl09-o1.c | 2 +-
> testsuites/libtests/dl09/dl09-o2.c | 2 +-
> testsuites/libtests/dl09/dl09-o3.c | 2 +-
> testsuites/libtests/dl09/dl09-o4.c | 2 +-
> testsuites/libtests/dl09/dl09-o5.c | 2 +-
> testsuites/libtests/dl10/dl10-o1.c | 2 +-
> testsuites/libtests/dl10/dl10-o2.c | 2 +-
> testsuites/libtests/dl10/dl10-o3.c | 2 +-
> testsuites/libtests/dl10/dl10-o4.c | 2 +-
> testsuites/libtests/dl10/dl10-o5.c | 2 +-
> testsuites/libtests/dl10/dl10-o6.c | 2 +-
> testsuites/libtests/mathl/init.c | 2 +-
> testsuites/libtests/ttest01/init.c | 6 +-
> testsuites/libtests/ttest01/test-assert.c | 2 +-
> testsuites/libtests/ttest01/test-checks.c | 25 +-
> testsuites/libtests/ttest01/test-destructor.c | 2 +-
> testsuites/libtests/ttest01/test-eno.c | 2 +-
> testsuites/libtests/ttest01/test-example.c | 2 +-
> testsuites/libtests/ttest01/test-fixture.c | 121 +-
> testsuites/libtests/ttest01/test-leak.c | 2 +-
> testsuites/libtests/ttest01/test-log.c | 2 +-
> testsuites/libtests/ttest01/test-malloc.c | 2 +-
> testsuites/libtests/ttest01/test-plan.c | 2 +-
> testsuites/libtests/ttest01/test-psx.c | 2 +-
> testsuites/libtests/ttest01/test-rtems.c | 2 +-
> testsuites/libtests/ttest01/test-simple.c | 2 +-
> testsuites/libtests/ttest01/test-step.c | 2 +-
> .../libtests/ttest01/test-task-context.c | 2 +-
> testsuites/libtests/ttest01/test-time.c | 2 +-
> testsuites/libtests/ttest01/test-verbosity.c | 2 +-
> testsuites/libtests/ttest02/init.c | 176 +
> testsuites/libtests/ttest02/ttest02.doc | 11 +
> testsuites/libtests/ttest02/ttest02.scn | 37 +
> testsuites/psxtests/Makefile.am | 3 +-
> testsuites/psxtests/psxbarrier01/main.c | 2 +-
> testsuites/psxtests/psxconfig01/init.c | 2 +-
> testsuites/psxtests/psxfenv01/init.c | 2 +-
> testsuites/psxtests/psxintrcritical01/init.c | 135 +-
> .../psxintrcritical01/psxintrcritical01.scn | 33 +-
> testsuites/psxtests/psxinttypes01/init.c | 2 +-
> testsuites/psxtests/psxndbm01/init.c | 2 +-
> testsuites/psxtests/psxrwlock01/main.c | 2 +-
> testsuites/samples/base_mp/system.h | 2 +-
> testsuites/samples/base_sp/system.h | 2 +-
> testsuites/samples/capture/system.h | 2 +-
> testsuites/samples/cdtest/main.cc | 2 +-
> testsuites/samples/cdtest/system.h | 2 +-
> testsuites/samples/fileio/system.h | 2 +-
> testsuites/samples/iostream/system.h | 2 +-
> testsuites/samples/nsecs/system.h | 2 +-
> testsuites/samples/paranoia/system.h | 2 +-
> testsuites/samples/pppd/system.h | 2 +-
> testsuites/samples/ticker/system.h | 2 +-
> testsuites/smptests/smpatomic01/init.c | 2 +-
> testsuites/smptests/smplock01/init.c | 2 +-
> testsuites/smptests/smpmulticast01/init.c | 2 +-
> testsuites/smptests/smpscheduler03/init.c | 2 +-
> testsuites/smptests/smpscheduler05/init.c | 2 +-
> testsuites/smptests/smpscheduler06/init.c | 2 +-
> testsuites/smptests/smpscheduler07/init.c | 2 +-
> testsuites/sptests/Makefile.am | 95 +-
> testsuites/sptests/configure.ac | 2 -
> testsuites/sptests/spheapprot/init.c | 2 +-
> .../spintrcritical01/spintrcritical01.scn | 30 +-
> .../spintrcritical01/spintrcritical01impl.h | 121 +-
> .../spintrcritical02/spintrcritical02.scn | 30 +-
> .../spintrcritical03/spintrcritical03.scn | 30 +-
> .../spintrcritical04/spintrcritical04.scn | 30 +-
> .../spintrcritical05/spintrcritical05.scn | 30 +-
> testsuites/sptests/spintrcritical06/init.c | 2 -
> .../spintrcritical06/spintrcritical06.doc | 28 -
> .../spintrcritical06/spintrcritical06.scn | 6 -
> .../spintrcritical06/spintrcritical06impl.h | 145 -
> testsuites/sptests/spintrcritical07/init.c | 2 -
> .../spintrcritical07/spintrcritical07.doc | 28 -
> .../spintrcritical07/spintrcritical07.scn | 6 -
> testsuites/sptests/spintrcritical08/init.c | 165 +-
> .../spintrcritical08/spintrcritical08.scn | 30 +-
> testsuites/sptests/spintrcritical09/init.c | 121 +-
> .../spintrcritical09/spintrcritical09.scn | 28 +-
> testsuites/sptests/spintrcritical10/init.c | 198 +-
> .../spintrcritical10/spintrcritical10.scn | 33 +-
> .../spintrcritical11/spintrcritical11.scn | 30 +-
> .../spintrcritical11/spintrcritical11impl.h | 110 +-
> .../spintrcritical12/spintrcritical12.scn | 30 +-
> .../spintrcritical13/spintrcritical13.scn | 31 +-
> .../spintrcritical13/spintrcritical13impl.h | 150 +-
> .../spintrcritical14/spintrcritical14.scn | 31 +-
> testsuites/sptests/spintrcritical15/init.c | 133 +-
> .../spintrcritical15/spintrcritical15.scn | 34 +-
> testsuites/sptests/spintrcritical16/init.c | 135 +-
> .../spintrcritical16/spintrcritical16.scn | 31 +-
> testsuites/sptests/spintrcritical18/init.c | 146 +-
> .../spintrcritical18/spintrcritical18.scn | 30 +-
> testsuites/sptests/spintrcritical20/init.c | 145 +-
> .../spintrcritical20/spintrcritical20.scn | 32 +-
> testsuites/sptests/spintrcritical21/init.c | 220 +-
> .../spintrcritical21/spintrcritical21.scn | 27 +-
> testsuites/sptests/spintrcritical21/system.h | 39 -
> testsuites/sptests/spintrcritical22/init.c | 103 +-
> .../spintrcritical22/spintrcritical22.scn | 20 +
> testsuites/sptests/spintrcritical23/init.c | 117 +-
> .../spintrcritical23/spintrcritical23.scn | 20 +
> testsuites/sptests/spintrcritical24/init.c | 128 +-
> .../spintrcritical24/spintrcritical24.scn | 32 +
> .../spintrcritical_support/intrcritical.c | 171 -
> .../spintrcritical_support/intrcritical.h | 52 -
> testsuites/sptests/sptimecounter02/init.c | 2 +-
> testsuites/sptests/sptimecounter03/init.c | 2 +-
> testsuites/sptests/sptimecounter04/init.c | 2 +-
> testsuites/support/include/buffer_test_io.h | 2 +-
> testsuites/support/include/tmacros.h | 2 +-
> testsuites/tmtests/tmfine01/init.c | 2 +-
> testsuites/tmtests/tmonetoone/init.c | 2 +-
> 169 files changed, 10028 insertions(+), 4543 deletions(-)
> create mode 100644 cpukit/include/rtems/doxygen-appl-config.h
> create mode 100644 cpukit/include/rtems/simple-test.h
> delete mode 100644 cpukit/include/t.h
> create mode 100644 cpukit/libtest/t-test-busy-tick.c
> create mode 100644 cpukit/libtest/t-test-busy.c
> create mode 100644 cpukit/libtest/t-test-interrupt.c
> create mode 100644 cpukit/libtest/t-test-rtems-posix-keys.c
> create mode 100644 cpukit/libtest/t-test-rtems.h
> create mode 100644 cpukit/libtest/testrun.c
> create mode 100644 testsuites/libtests/ttest02/init.c
> create mode 100644 testsuites/libtests/ttest02/ttest02.doc
> create mode 100644 testsuites/libtests/ttest02/ttest02.scn
> delete mode 100644 testsuites/sptests/spintrcritical06/init.c
> delete mode 100644 testsuites/sptests/spintrcritical06/spintrcritical06.doc
> delete mode 100644 testsuites/sptests/spintrcritical06/spintrcritical06.scn
> delete mode 100644 testsuites/sptests/spintrcritical06/spintrcritical06impl.h
> delete mode 100644 testsuites/sptests/spintrcritical07/init.c
> delete mode 100644 testsuites/sptests/spintrcritical07/spintrcritical07.doc
> delete mode 100644 testsuites/sptests/spintrcritical07/spintrcritical07.scn
> delete mode 100644 testsuites/sptests/spintrcritical21/system.h
> delete mode 100644 testsuites/sptests/spintrcritical_support/intrcritical.c
> delete mode 100644 testsuites/sptests/spintrcritical_support/intrcritical.h
>
More information about the devel
mailing list