Running a test

In order to run a test for one or more test cases, a test run environment (run root) has to be provided, which contains test runs in the same hierarchy as in the main_suite. For running tests, only the test case directory structure is required (without data, actions and expected directory).

• run root (RUN_PATH)

• main_suite
• actions (run, preprocessing, postprocessing, compare, ..)
• data
• binaries (executable files)
• MyTool
• tools (diff, find, grep, sed)
• work_area

• data (in/out)
• actions
• expected

The binaries directory contains tools and programs needed for running the test. work_area provides all test specific resources, i.e. the work area may be used for one test at the time. Before running a test, the work area has to be prepared for the test. This is a task of the test framework.

For running a test, the main_suite/actions directory contains a run action. In case of requested additional actions within a test, the run action has to be updated and stored in the directory of the corresponding test case or test suite. Moreover, the preprocessing action has to be provided in the main_suite/actions directory and is overwritten in some test suites in order to perform actions in the preprocessing step.

After running the test, test results are evaluated by comparing data collected in the expected directory with the data created in data directory (postprocesssing and compare action). In case of differences, result files differing are copied from data to failed.

Since the work area will be recreated for each test, default procedures for actions are stored in main_suite/actions. At least three default actions preprocessing, run and postprocessing have to be provided. Since actions may be replaced by other actions with the same name (overloading), these actions are called dynamic actions.

Common settings for environment variables are defined in settings action, which is part of the test framework.

The preprocessing action allows providing specific actions to be called before running the test. Typical actions are copying additional data or manipulating test data for meeting test case requirements. The default action does nothing, but may be overloaded in subordinated test cases (e.g. generating test suites from a test suite template).

The preprocessing action is called and defined differently for Windows (cmd) and Linux (bash).

Running the test suite depends on the test to be performed. Typically, the run action is implemented on a higher level in the hierarchy (main suite). When, however, considering another hierarchy, where each call gets its own test suite below the main test suite, the run action could be implemented on the next lower level.

As first and only parameter the current test or test suite is passed (path relative to main_suite).

Summary information (start and stop time) is written to a log file (LOGFILE_OUT) in the test run directory.

run calls the table compiler and writes start and stop information to logfile.out. An example for overwriting the run action was the requirement calling the table compiler twice. An example is given in the ODABA test suite under Utilities/CopyDB/actions.

The run action is called and defined differently for Windows (cmd) and Linux (bash).

The default postprocessing action compares all files defined in the expected directory with the files created in the data directory by calling compare after copying all error files (.err) to the test suite directory.

The Compare action compares the file passed with a file having the same file name in the data directory in the work area. When comparing returns differences, those are written to the COMPARE_OUT location, which has been defined in settings (compare.out in test run directory). In case of errors (e.g. missing file in data directory), error messages are written to the error location ERRORS_OUT (errors.out in test run directory).

When comparing a file fails, the file is copied to the failed directory in the test suite (copyFailed) and an error notice is written to compare.out.

More enhanced compare could be easily achieved by calling grep (remove lines not to be compared) and sed (change lines, i.g. for removing comments) before comparing files.

Usually, the postprocessing action has to be provided for each test system ones on top of the hierarchy. But it may also be overloaded for any test suite in the test suite hierarchy.

The postprocessing action is called and defined differently for Windows (cmd) and Linux (bash).