[PATCH] Added documentation for rtems-tester in rtems-docs

Tanu Hari Dixit tokencolour at gmail.com
Tue May 30 18:36:13 UTC 2017


Contains the ReST file for documentation for rtems-tester.
Also contains the new format for the ini configuration files.
---
 user/tools/rtems-tester.rst | 639 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 639 insertions(+)
 create mode 100644 user/tools/rtems-tester.rst

diff --git a/user/tools/rtems-tester.rst b/user/tools/rtems-tester.rst
new file mode 100644
index 0000000..24c62dd
--- /dev/null
+++ b/user/tools/rtems-tester.rst
@@ -0,0 +1,639 @@
+RTEMS Tester
+************
+
+.. image:: ../../images/user/rtemswhitebg.jpg
+   :width: 400
+   :align: center
+   
+Chris Johns <chrisj at rtems.org>
+1.0, June 2014
+
+RTEMS Tester
+============
+
+The RTEMS Tester is a test framework. It includes a command line interface to
+run tests on supported targets. The framework provides back-end support for
+common simulators and debuggers. The board support package (BSP) configurations
+for RTEMS are provided and can be used to run all the tests provided with
+RTEMS. The framework is not specific to RTEMS and can be configured to run any
+suitable application.
+
+RTEMS is an embedded operating system and is cross-compiled on a range of host
+machines. The executables run on the target hardware and this can vary widely
+from open source simulators, commercial simulators, debuggers with simulators,
+to debuggers with hardware specific pods and devices. Testing RTEMS requires
+the cross-compiled test executable is transferred to the target hardware,
+executed and the output returned to the host where it is analyzed to determine
+the test result. The RTEMS Tester provides a framework to do this.
+
+Running all the RTEMS tests on your target is very important. It provides you
+with a traceable record showing that your RTEMS version and its tools are
+working at the level the RTEMS development team expect when releasing
+RTEMS. Being able to easily run the tests and verify the results is critical in
+maintaining a high standard.
+
+The RTEMS Tester contains:
+
+* Command line tool (``rtems-test``)
+* BSP Configuration scripts
+* Back-end Configuration scripts
+* Back-end Python classes
+* Python based framework
+
+This document is for users of the RTEMS Tester as well as those wanting to
+develop, extend and investigate the RTEMS Tester.
+
+License
+=======
+
+The RTEMS Tester is part of the RTEMS Tools Project. The code is released under
+the OSI approved "The BSD 2-Clause License". It is free to use and we encourage
+this, including on operating systems other than RTEMS.
+
+The code and command line tools must retain the same names and always reference
+the RTEMS Tools Project.
+
+Quick Start
+===========
+
+The quick start will show you how to run the test suite for a BSP. It will
+explain how to get the RTEMS Tester, set it up and run the tests for the erc32
+BSP. It assumes you have a valid SPARC tool chain and have built the erc32 BSP
+version of RTEMS 4.12.
+
+Setup
+=====
+
+Set up a development work space:
+
+.. code-block:: shell
+
+    $ cd
+    $ mkdir -p development/rtems/test
+    $ cd development/rtems/test
+
+First fetch the RTEMS tester from the RTEMS Tools repository:
+
+.. code-block:: shell
+
+    $ git clone git://git.rtems.org/rtems-tools.git rtems-tools.git
+    $ cd rtems-tools.git/tester
+
+Available BSP testers
+=====================
+
+You can list the available BSP testers with:
+
+.. code-block:: shell
+
+    $ ./rtems-test --list-bsps
+    arm920
+    beagleboardxm
+    jmr3904-run
+    jmr3904
+    mcf5235
+    psim-run
+    psim
+    realview_pbx_a9_qemu
+    sis-run
+    sis
+    xilinx_zynq_a9_qemu
+    xilinx_zynq_a9_qemu_smp
+    xilinx_zynq_zc706
+    xilinx_zynq_zc706_qemu
+
+.. note:: The list is growing all the time and if your BSP is not supported we 
+          encourage you to add it and submit the configuration back to the project.
+
+Some of the BSPs may appear more than once in the list. These are aliased BSP
+configurations that may use a different back-end. An example is the SPARC
+Instruction Simulator (SIS) BSP. There is the 'sis' tester which uses the GDB
+back-end and the 'sis-run' tester which uses the command line version of the SIS
+simulator. We will show how to use ``rtems-test`` command with the SIS BSP
+because it is easy to build an use.
+
+Adding support for a BSP to the tester
+======================================
+
+To add support for a BSP, one should add a configuration file in 
+``rtems-tools.git/tester/rtems/testing/bsps``. The configuration files 
+should be specified in an INI file. The configuration files provide a 
+means to the backend of `rtems-tester` to automate the running of tests.
+There is the ``<bsp>`` tester which uses the GDB back-end and the ``<bsp>-run`` 
+tester which uses the command line version of the respective simulator.
+The general format for specifying an INI configuration file is as follows:
+
+<bsp>.ini:
+
+.. code-block:: shell
+
+    [<bsp_name>]
+    bsp = '<bsp_name>'
+    gdb = '%{_rtscripts}/gdb.cfg'
+    arch = '<arch_name>'
+
+    [gdb_script]
+    gdb_script ='<bsp>_gdb_script'
+    <bsp>_gdb_script =   '<Instructions for gdb
+	    ...
+	    ...>'
+
+<bsp>-run.ini:
+
+.. code-block:: shell
+
+    [<bsp_name>]
+    bsp = '<bsp_name>'
+
+    [run]
+    run = '%{_rtscripts}/run.cfg'
+    arch = '<arch_name>'
+    bsp_run_cmd = '%{rtems_tools}/%{bsp_arch}-rtems%{rtems_version}-run'
+    bsp_run_opts = '<options>'
+
+Here is an example:
+
+arm7tdmi.ini:
+
+.. code-block:: shell
+
+    [arm7tdmi]
+    bsp = 'arm7tdmi'
+    gdb = '%{_rtscripts}/gdb.cfg'
+    arch = 'arm'
+
+    [gdb_script]
+    gdb_script ='arm7tdmi_gdb_script'
+    arm7tdmi_gdb_script =   'target sim
+        load
+        run'
+
+arm7tdmi-run.ini:
+
+.. code-block:: shell
+
+    [arm7tdmi]
+    bsp = 'arm7tdmi'
+
+    [run]
+    run = '%{_rtscripts}/run.cfg'
+    arch = 'arm'
+    bsp_run_cmd = '%{rtems_tools}/%{bsp_arch}-rtems%{rtems_version}-run'
+    bsp_run_opts = '-a -nouartrx'
+
+If the tester should run the tests on qemu for a bsp, the given pattern can 
+be followed:
+
+.. code-block:: shell
+
+    [<bsp_name>]
+    bsp = '<bsp_name>'
+
+    [qemu-script]
+    run = '%{_rtscripts}/qemu.cfg'
+    arch = '<arch_name>'
+    opts = '%{qemu_opts_base} %{qemu_opts_no_net} -m 32M'
+
+For instance,
+
+generic_or1k.ini:
+
+.. code-block:: shell
+
+    [generic_or1k]
+    bsp = 'generic_or1k'
+
+    [qemu-script]
+    run = '%{_rtscripts}/qemu.cfg'
+    arch = 'or32'
+    opts = '%{qemu_opts_base} %{qemu_opts_no_net} -m 32M'
+
+User can also provide a "settings.ini" file in case a configuration file needs 
+a path to, for instance, a first stage bootloader (fsbl) that is placed 
+somewhere in the host machine. The settings.ini file can be passed with 
+``rtems-test --with-settings = /path/to/settings/file`` option. The 
+<bsp_name>.ini can contain defaults and the settings file can contain the 
+user-specified configuration.
+
+
+xilinx_zynq_zc706.ini:
+
+.. code-block:: shell
+
+    [<xilinx_zynq_zc706>]
+    bsp =  'xilinx_zynq_zc706'
+    jobs = '1'
+
+    [gdb-script]
+    gdb =  '%{_rtscripts}/gdb.cfg'
+    arch =  'arm'
+    bsp_tty_dev =  ${settings:bsp_tty_dev}
+    gdb_script =  'xilinx_zynq_zc706_gdb_script'
+    xilinx_zynq_zc706_gdb_script =  'target remote kaka:3333
+    mon load_image ${settings:path_to_fsbl} 0 elf
+        mon resume 0
+        mon sleep 4000
+        mon halt
+        load
+        b bsp_reset
+        continue'
+
+settings.ini:
+
+.. code-block:: shell
+
+    [settings]
+    bsp_tty_dev = '/dev/cu.SLAB_USBtoUART'
+    path_to_fsbl = '/home/chris/development/si/work/hydra/boot/xilinx-zynq-fsbl/build/arm-rtems4.11-xilinx_zynq_zc706/hydra-fsbl.elf'
+
+Building RTEMS Tests
+====================
+
+Build RTEMS with a configuration command line something similar to:
+
+.. note:: The following assumes a Unix-type host and that the tools have been 
+          built with a prefix of ``$HOME/development/rtems/4.12``.
+
+.. code-block:: shell
+
+    $ cd
+    $ export PATH=$HOME/development/rtems/4.12/bin:$PATH
+    $ mkdir -p development/rtems/kernel
+    $ cd development/rtems/kernel
+    $ git clone git://git.rtems.org/rtems.git rtems.git
+    $ cd rtems.git
+    $ ./bootstrap -c && ./bootstrap -p && ./bootstrap
+    $ cd ..
+    $ mkdir -p builds/erc32
+    $ cd builds/erc32
+    $ ../../rtems.git/configure --target=sparc-rtems4.12 \
+                        --enable-tests --enable-rtemsbsp=erc32
+    $ make 
+
+Add the -j option to the make command with the number of cores to run the 
+build parallel where it can.
+
+Building all the tests takes time and it uses more disk so be patient. When
+finished all the tests will be built and ready to run. Before running all the 
+tests it is a good idea to run the ``hello`` test. The ``hello`` test is an 
+RTEMS version of the classic "Hello World" example and running it shows you 
+have a working tool chain and build of RTEMS ready to run the tests. Using the 
+run command:
+
+.. code-block:: shell
+
+    $ sparc-rtems4.12-run sparc-rtems4.12/c/erc32/testsuites/samples/hello/hello.exe
+
+    *** BEGIN OF TEST HELLO WORLD ***
+    Hello World
+    *** END OF TEST HELLO WORLD ***
+
+The run command is the GDB simulator without the GDB part.
+
+Running the example using GDB:
+
+.. code-block:: shell
+
+    $ sparc-rtems4.12-gdb sparc-rtems4.12/c/erc32/testsuites/samples/hello/hello.exe
+    GNU gdb (GDB) 7.12
+    Copyright (C) 2016 Free Software Foundation, Inc.
+    License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
+    This is free software: you are free to change and redistribute it.
+    There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
+    and "show warranty" for details.
+    This GDB was configured as "--host=x86_64-linux-gnu --target=sparc-rtems4.12".
+    Type "show configuration" for configuration details.
+    For bug reporting instructions, please see:
+    <http://www.gnu.org/software/gdb/bugs/>.
+    Find the GDB manual and other documentation resources online at:
+    <http://www.gnu.org/software/gdb/documentation/>.
+    For help, type "help".
+    Type "apropos word" to search for commands related to "word"...
+    Reading symbols from 
+    sparc-rtems4.12/c/erc32/testsuites/samples/hello/hello.exe...done.
+    (gdb) target sim
+    Connected to the simulator.
+    (gdb) load
+    (gdb) r
+    Starting program: sparc-rtems4.12/c/erc32/testsuites/samples/hello/hello.exe 
+
+
+    *** BEGIN OF TEST HELLO WORLD ***
+    Hello World
+    *** END OF TEST HELLO WORLD ***
+    [Inferior 1 (process 42000) exited normally]
+    (gdb) q
+
+The command r is used to debug set break points before issuing the GDB run 
+command.
+
+There are currently close to 500 separate tests and you can run them all with a
+single RTEMS Tester command.
+
+.. note:: Use a recent RSB version to build a SPARC tool chain. A recent patch
+          fixes issues with the GDB simulator and it improves the test results.
+          It is also recommended you use a recent RTEMS 4.11 version as the 
+          tests have been cleaned up to improve the results.
+
+Running the Tests
+=================
+
+The ``rtems-test`` command line accepts a range of options. These are discussed
+later in the manual. Any command line argument without a `--` prefix is a test
+executable. You can pass more than one executable on the command line. If the
+executable is a path to a directory the directories under that path are
+searched for any file with a ``.exe`` extension. This is the default extension
+for RTEMS executables built within RTEMS.
+
+To run the erc32 tests enter the following command from the top of the erc32 BSP
+build tree:
+
+.. code-block:: shell
+
+    $ ~/development/rtems/test/rtems-tools.git/tester/rtems-test \
+             --log=log_erc32_run \
+             --rtems-bsp=erc32-run \
+             --rtems-tools=$HOME/development/rtems/4.12 \
+                 sparc-rtems4.12/c/erc32/testsuites/samples 
+    RTEMS Testing - Tester, 4.12.not_released
+    [ 1/13] p:0  f:0  u:0  e:0  I:0  B:0  t:0  i:0  | sparc/erc32: base_sp.exe
+    [ 2/13] p:0  f:0  u:0  e:0  I:0  B:0  t:0  i:0  | sparc/erc32: capture.exe
+    [ 3/13] p:0  f:0  u:0  e:0  I:0  B:0  t:0  i:0  | sparc/erc32: cdtest.exe
+    [ 4/13] p:0  f:0  u:0  e:0  I:0  B:0  t:0  i:0  | sparc/erc32: fileio.exe
+    [ 5/13] p:2  f:0  u:0  e:0  I:0  B:0  t:0  i:0  | sparc/erc32: hello.exe
+    [ 6/13] p:2  f:0  u:0  e:0  I:0  B:0  t:0  i:0  | sparc/erc32: cxx_iostream.exe
+    [ 8/13] p:2  f:0  u:0  e:0  I:0  B:0  t:2  i:0  | sparc/erc32: minimum.exe
+    [ 7/13] p:2  f:0  u:0  e:0  I:0  B:0  t:2  i:0  | sparc/erc32: loopback.exe
+    [ 9/13] p:3  f:0  u:0  e:0  I:0  B:0  t:3  i:0  | sparc/erc32: nsecs.exe
+    [10/13] p:3  f:0  u:0  e:0  I:0  B:0  t:3  i:0  | sparc/erc32: paranoia.exe
+    [11/13] p:4  f:0  u:0  e:0  I:0  B:0  t:3  i:0  | sparc/erc32: pppd.exe
+    [12/13] p:6  f:0  u:0  e:0  I:0  B:0  t:3  i:0  | sparc/erc32: ticker.exe
+    [13/13] p:6  f:0  u:0  e:0  I:0  B:0  t:3  i:0  | sparc/erc32: unlimited.exe
+    Passed:         7
+    Failed:         0
+    User Input:     0
+    Expected Fail:  0
+    Indeterminate:  0
+    Benchmark:      0
+    Timeout:        5
+    Invalid:        1
+    Total:         13
+    Average test time: 0:00:27.963000
+    Testing time     : 0:06:03.519012
+
+
+
+* The RTEMS Tester's test command. In this example we are using an absolute
+  path.
+* The ``--log`` option sends the output to a log file. By default only failed
+  tests log the complete output.
+* Select the erc32 BSP and use GDB.
+* Path to the RTEMS tools so GDB can be found.
+* Path to the erc32 BSP built with all tests to run. If you add subdirectories
+  to the path specific tests can be run.
+* The output has been shortened so it fits nicely here.
+* The test results shows passes, fails, timeouts, and invalid results. In
+  this run 13 tests passed and 5 tests timed out and 1 is invalid. The 
+  timeouts are probably due to the tests not having enough execute time to 
+  complete. The default timeout is 180 seconds and some of the interrupt tests 
+  need longer. The amount of time depends on the performance of your host CPU 
+  running the simulations.
+* The output shows the average time per test and the total time taken to run 
+  all the tests.
+* If the path to the testsuites was put to 
+  ``sparc-rtems4.12/c/erc32/testsuites`` instead of 
+  ``sparc-rtems4.12/c/erc32/testsuites/samples`` then all the executables 
+  would have been tested and not just those in samples.
+
+This BSP requires the ``--rtems-tools`` option because the SPARC GDB is the
+``sparc-rtems4.11-gdb`` command that is part of the RTEMS tools. Not every BSP
+will require this option so you will need to check the specifics of the BSP
+configuration to determine if it is needed.
+
+The output you see is each test starting to run. The ``rtems-test`` command by
+default runs multiple tests in parallel so you will see a number start quickly
+and then new tests start as others finish. The output shown here is from an
+8 core processor so the first 8 are started in parallel and the status shows
+the order in which they actually started, which is not 1 to 8.
+
+The test start line shows the current status of the tests. The status reported
+is when the test starts and not the result of that test. A fail, timeout or
+invalid count changing means a test running before this test started failed,
+not the starting test. The status here has 7 tests passed, no failures, 5
+timeouts and 1 invalid test.
+
+.. code-block:: shell
+
+    [ 5/13] p:2  f:0  u:0  e:0  I:0  B:0  t:0  i:0  | sparc/erc32: hello.exe
+
+* [ 5/13] indicates the test number, in this case test 5 of 13 tests.
+* p is the passed test count (2 in this case)
+* f is the failed test count (0 in this case)
+* u is the count for test marked as "user-input" as they expect input from user
+* e is the expected-fail count (tests that are expected to fail)
+* I is the count for tests the results of which are indeterminate
+* B is the count for benchmarked tests
+* t is the timeout test count 
+* i is the invalid test count.
+* sparc/erc32 is the architecture and BSP names.
+* hello.exe is the executable name.
+
+The test log records all the tests and results. The reporting mode by default
+only provides the output history if a test fails, times out, or is invalid. The
+time taken by each test is also recorded.
+
+The tests must complete in a specified time or the test is marked as timed
+out. The default timeout is 3 minutes and can be globally changed using the
+``--timeout`` command line option. The time required to complete a test can
+vary. When simulators are run in parallel the time taken depends on the
+specifics of the host machine being used. A test per core is the most stable
+method even though more tests can be run than available cores. If your machine
+needs longer or you are using a VM you may need to lengthen the timeout.
+
+Test Status
+===========
+
+Tests can be marked with one of the following:
+
+* Pass
+* Fail
+* Timeout
+* Invalid
+
+The RTEMS console or stdout output from the test is needed to determine the
+result of the test.
+
+Pass
+----
+A test passes if the start and end markers are seen in the test output. The
+start marker is ``***`` and the end mark is ``*** END OF TEST``. All tests in 
+the RTEMS test suite have these markers.
+
+Fail
+----
+A test fails if the start marker is seen and there is no end marker.
+
+Timeout
+-------
+If the test does not complete within the timeout setting the test is marked as
+having timed out.
+
+Invalid
+-------
+If no start marker is seen the test is marked as invalid. If you are testing on
+real target hardware things can sometimes go wrong and the target may not
+initialize or respond to the debugger in an expected way.
+
+Reporting
+=========
+
+The report written to the log has the following modes:
+
+* All (``all``)
+* Failures (``failures``)
+* None (``none``)
+
+The mode is controlled using the command line option ``--report-mode`` using
+the values listed above.
+
+All
+---
+The output of all tests is written to the log.
+
+Failures
+--------
+The output of the all tests that do not pass is written to the log.
+
+None
+----
+No output is written to the log.
+
+The output is tagged so you can determine where it comes from. The following is
+the complete output for the In Memory File System test ``imfs_fslink.exe``
+running on a Coldfire MCF5235 using GDB and a BDM pod:
+
+.. code-block:: shell
+
+    [ 11/472] p:9   f:0   t:0   i:1   | m68k/mcf5235: imfs_fslink.exe
+    > gdb: ..../bin/m68k-rtems4.11-gdb -i=mi --nx --quiet ..../imfs_fslink.exe 
+    > Reading symbols from ..../fstests/imfs_fslink/imfs_fslink.exe...
+    > done.
+    > target remote | m68k-bdm-gdbserver pipe 003-005
+    > Remote debugging using | m68k-bdm-gdbserver pipe 003-005
+    > m68k-bdm: debug module version 0
+    > m68k-bdm: detected MCF5235
+    > m68k-bdm: architecture CF5235 connected to 003-005
+    > m68k-bdm: Coldfire debug module version is 0 (5206(e)/5235/5272/5282)
+    > Process 003-005 created; pid = 0
+    > 0x00006200 in ?? ()
+    > thb *0xffe254c0
+    > Hardware assisted breakpoint 1 at 0xffe254c0
+    > continue
+    > Continuing.
+    ] 
+    ]
+    ] External Reset
+    ]
+    ] ColdFire MCF5235 on the BCC
+    ] Firmware v3b.1a.1a (Built on Jul 21 2004 17:31:28)
+    ] Copyright 1995-2004 Freescale Semiconductor, Inc.  All Rights Reserved.
+    ]
+    ] Enter 'help' for help.
+    ]
+    > Temporary breakpoint
+    > 1, 0xffe254c0 in ?? ()
+    > load
+    > Loading section .text, size 0x147e0 lma 0x40000
+    > Loading section .data, size 0x5d0 lma 0x547e0
+    > Start address 0x40414, load size 85424
+    > Transfer rate: 10 KB/sec, 1898 bytes/write.
+    > b bsp_reset
+    > Breakpoint 2 at 0x41274: file ..../shared/bspreset_loop.c, line 14.
+    > continue
+    > Continuing.
+    ] dBUG>
+    ]
+    ] *** FILE SYSTEM TEST ( IMFS ) *** 
+    ] Initializing filesystem IMFS
+    ]
+    ]
+    ] *** LINK TEST ***
+    ] link creates hardlinks
+    ] test if the stat is the same
+    ] chmod and chown
+    ] unlink then stat the file
+    ] *** END OF LINK TEST ***
+    ]
+    ]
+    ] Shutting down filesystem IMFS
+    ] *** END OF FILE SYSTEM TEST ( IMFS ) *** 
+    > Breakpoint
+    > 2, bsp_reset () at ..../m68k/mcf5235/../../shared/bspreset_loop.c:14
+    > 14    {
+    Result: passed     Time: 0:00:10.045447 
+
+* GDB command line (Note: paths with \'....' have been shortened)
+* Lines starting with ``>`` are from GDB's console.
+* Line starting with ``]`` are from the target's console.
+* The result with the test time.
+
+Running Tests in Parallel
+=========================
+
+The RTEMS Tester supports parallel execution of tests by default. This only
+makes sense if the test back-end can run in parallel without resulting in
+resource contention. Simulators are an example of back-ends that can run in
+parallel. A hardware debug tool like a BDM or JTAG pod can manage only a
+single test at once so the tests need to be run one at a time.
+
+The test framework manages the test jobs and orders the output in the report
+log in test order. Output is held for completed tests until the next test to be
+reported has finished.
+
+Command Line Help
+=================
+
+The ``rtems-test`` command line accepts a range of options. You can review the
+available option by the ``--help`` option:
+
+.. code-block:: shell
+
+    RTEMS Tools Project (c) 2012-2014 Chris Johns
+    Options and arguments:
+    --always-clean               : Always clean the build tree, even with an error
+    --debug-trace                : Debug trace based on specific flags
+    --dry-run                    : Do everything but actually run the build
+    --force                      : Force the build to proceed
+    --jobs=[0..n,none,half,full] : Run with specified number of jobs, default: num CPUs.
+    --keep-going                 : Do not stop on an error.
+    --list-bsps                  : List the supported BSPs
+    --log file                   : Log file where all build output is written to
+    --macros file[,file]         : Macro format files to load after the defaults
+    --no-clean                   : Do not clean up the build tree
+    --quiet                      : Quiet output (not used)
+    --report-mode                : Reporting modes, failures (default),all,none
+    --rtems-bsp                  : The RTEMS BSP to run the test on
+    --rtems-tools                : The path to the RTEMS tools
+    --target                     : Set the target triplet
+    --timeout                    : Set the test timeout in seconds (default 180 seconds)
+    --trace                      : Trace the execution
+    --warn-all                   : Generate warnings
+
+Development
+===========
+
+The RTEMS Tester framework and command line tool is under active
+development. These are changing, being fixed, broken and generally improved. If
+you want to help please see the Wiki page for open items.
+
+History
+=======
+
+The RTEMS Tester is based on a refactored base of Python code used in the RTEMS
+Source Builder. This code provided a working tested base that has been extended
+and expanded to meet the requirements for the RTEMS Tester. The tester uses the
+specifics found in the various scripts and configurations in the
+rtems-testing.git repo that has been accumulated over many years. The shell
+script implementation is restricted in what it can do and, per BSP script, is a
+maintenance burden. For example the command lines and options vary between each
+script.
-- 
2.1.4



More information about the devel mailing list