Common example

The example provides a simple test suite as described below. After executing the test suite hierarchy, success and failure messages are written to console and to result file.

The following chapters describe the structure of the example, the principles for running a test including a simple test case and the features provided by the command shell test framework.

The test technology used for component and unit tests is based on a few simple procedures and third party tools (Linux tools for MS Windows: find, diff, grep, sed). In order to run tests the following steps are necessary:

1. Before running tests, data and expected results have to be provided in an appropriate structure below a test root directory (see "Preparing data" below). The location of the test root directory does not matter, since all path definitions are relative.
2. When default actions or files do not match the requirements for some tests, those have to be overloaded by appropriate actions and data sub directory in the test suite directory.
3. Test suites or tests have to be executed in test runs and results have to be checked by calling RunTestSuite. In case of test failures, single tests may be corrected and repeated by calling RunTest.

When a test fails, there are always two possibilities: Either the tested component is not correct or the expected result has been defined improperly. Both must taken into account.

In order to manage test suites and test cases more comfortable, a GUI application (TestBrowser) has been provided. TestBrowser created a database from the file system (directory structure). All relevant information is stored in the file system. When starting the TestBrowser the first time, the database will be created from the file system data. Running the TestBrowser, database and file system are automatically synchronized.

Within the example, following directories are used:

• data - contains test set and results
• expectations - contains expected results
• actions - contains test run actions

In order to support inheritance features in a simple way, the example refers to a hierarchical test suite arrangement. The example provides a simple test suite hierarchy below the main suite:

Test suites marked as run are tests and will be executed in a test run. Test suites marked as suite allow providing resources for a test in the work area. In the example, each directory may define a test suite and a test at the same time. The example framework does not support more than one test for a test suite.

It is assumed, that test suites are provided according to a test case hierarchy below a common test root directory (ROOT_DIR). The test root contains common resources as test frame work procedures, tools and executables (binaries) and the work area (work_area) The top level for a test suite hierarchy for testing a component is a directory with the name main_suite below test root. This contains common resources for all the subsequent test cases. Sub directories are numbered (1, ... n).

In order to mark test suites as suite and/or test, two special files may be provided in each test suite directory:

• suite - Each test suite in the hierarchy contains a file named suite. The system is constructed in a way, that it will stop recursion when the first folder is met, which does not contain a suite file. The file may be empty or may contain a short description of the test suite concept (e.g. test intend).

• run - Test suites, which are also tests and have to be executed, contain a file named run. The run file may contain a description or comments for the test run to be executed.

The content of these files is of no relevance for the test framework.

Default resources may be defined in main_suite, which may be replaced or extended by resources defined further down in the test suite hierarchy. main_suite is usually marked as suite but not as run.

Besides some procedures managing complex test runs, the test root directory contains following directories:

• main_suite - top entry for test suite hierarchy (ROOT_DIR/main_suite)
• test_runs - contains a sub directory for each test run
• binaries - binaries or executables to be tested (BIN_DIR)
• binaries/tools - third party tolls used for running tests (TOOLS_DIR)
• work_area - work area (WORK_AREA)

Ea component to be tested, MyTest.cmd is called from the binaries directory.the shell echo command had been selected. The procedure simply calls the echo command for creating an output file containing the test suite pass for the test suite currently tested.

The simple component to be tested does not require input data, but will produce an output file, only. Hence, the data directories are empty.

Except test suite 1/2, all the other test suited marked as run contain a file myTestResult.txt with the expected text data in sub directory expectations. The corresponding file in test suite 1/2 has different content and will, hence, fail when running tests.

After executing the test suite hierarchy, results.out and logfile.out contain more or less the subsequent data. In order to get a more detailled protocol, one may call TestOut.cmd.