Test evaluation

Evaluating test results is typically done by comparing expected data with data created by running a test. The test framework only expects that test results for a test are summarized in a file compare.out (COMPARE_OUT environment variable). When no differences have been found between test results and expected data, the file is empty (success). When one or more files differ between expected data and test results, the differences should be listed in the compare.out file (failed). When test had not been executed for any other reason, the compare.out file does not exist.

Creating the compare.out file, is task of the postprocessing action. This is not part of the test framework, but has to be provided for a specific test environment. The ODABATest environment provides the procedures listed below:

• postprocessing - Preparing files for compare (GetErrorsFromLog and Compare)

• GetErrorsFromLog - Remove timestamps and directory paths from test error.lst and output.lst files, which are created by all test test runs.
• Compare - Compares updated files with expected results. In case of differences, those are written to COMPARE_OUT (compare.out) file and the test result file differing is written to the test/failed directory (CopyFailed action)
• CopyFailed - Save test result file differing from expected data in failed directory.

From the compare.out file, the results.out file for the complete test run is appended with the test result for the test (success or failed). In addition, the run procedure for ODABATest environment creates a log file logfile.out containing start and stop time for each test.