Test framework actions (Windows)
The cmd test framework consists of a number of procedures managing tests under Windows.
In order to run a test, all required resources are copied to the work area (work_area) which is structured according to the requirements of the component to be tested. When running a series of tests in the test framework, the work area will be overwritten each time, a new test is started. In order to guarantee traceability, the work area has to be copied to the test suite running the test (e.g. as zip file). Here, only differing files are stored to the failed directory in the test suite.
The test framework provides a set of common actions (command line procedures) for managing test preparation and evaluation. In addition, each specific test environment has to provide adopted actions named preprocessing, run and postprocessing (command line procedures with extension .cmd), which are usually stored in main_suite/actions directory, and possibly overloaded in several test cases or test suites. Test framework actions (procedures) are stored in the test root directory.
Framework actions that may be called from command line are RunTest.cmd, RunMany.cmd, SetupWorkArea.cmd and TestOut.cmd. In order to support parallel testing, the work area location may also be passed as parameter to RunTest or RunMany action. Default is the work_area directory below the test root directory.
In order to execute a single or a number test cases, framework actions RunTest or RunMany may be called.
For executing a single test case, RunTest is called. In order to locate the proper test case, the relative path has to be defined like API/Local/Property.
In order to run the complete test suite hierarchy, RunMany may to be called. The procedure executes all test case directories containing a run file. In order to filter test cases for execution, one may rename the run file in test case directories to be excluded.
For each test directory, runSingle will be called. runSingle executes a single test suite after checking the presence of a run file in the directory. When the run file exists, testRun is called. testRun is called via callNormalized, which normalizes the directory path by removing ./ at the beginning of the path name (this type of path names are not supported by older command line functions as type or copy).
// RunTest.cmd
@echo off
rem execute single test
rem 1 - relative test suite path (e.g. 0/1/2)
rem 2 - test run directory (%RUN_PATH%)
rem 3 - main suite directory (%ROOT_PATH%)
rem 4 - test root directory (%cd%)
rem 5 - work directory (%WORK_DIR%)
call %4\settings.cmd %1 %2 %3 %4 %5
pushd %RUN_ROOT%
call %ROOT_DIR%\runSingle.cmd ./%~1
popd
// RunMany.cmd
@echo off
rem execute test run(s)
rem 1 - test run directory (%RUN_PATH%)
rem 2 - main suite directory (%ROOT_PATH%)
rem 3 - test root directory (%cd%)
if "%3" == "" (call %cd%\settings.cmd . %1 %2) else (call %3\settings.cmd . %1 %2 %3)
pushd %2
%TOOLS_DIR%\find . -type d -exec %ROOT_DIR%\runSingle.cmd {} ;
popd
// runSingle.cmd
@echo off
rem run single test in hierarchy
rem 1 - test folder name below main suite ...\main_suite
if exist %MAIN_SUITE%\%1\run %ROOT_DIR%\callNormalized.cmd %ROOT_DIR%\testRun.cmd %1
// CallNormalized.cmd
@echo off
rem convert Linux path to MS Windows path (./a/b... --> a\b...)
rem 1 - procedure to be called
rem 2 - "find" name for directory or file
FOR /F "tokens=2,3,4,5 delims=/" %%G IN ('echo %2') DO call %1 %%G\%%H\%%I\%%J %2
In order to run a test, the testRun action is provided by the test frame work (testRun.cmd). The testRun action updates settings for environment variables, prepares the work area and calls test environment specific actions for the selected work suite. settings and prepare are framework actions, which cannot be overloaded, while preprocessing, run and postprocessing are test environment specific actions, which have to be defined for each specific test environment in top or lower test suite or test case actions directory.
After setting test environment variables (settings action), the prepare action copies test data (actions, data, expected) to corresponding directories in work_area from the test suite hierarchy. Test data from test cases and test suites is copied recursively (copyRecursive) from the test suite hierarchy by copying resources from the test case to be tested and all its parent test suites. A directory is considered as test suite, when it contains a file suite. Copying stops, when the first directory was found, which is not a test suite. Files existing on lower levels in the hierarchy, always will overwrite files with the same name provided on higher levels.
When NO_RUN has been set to YES (SetupWorkArea), the testRun action terminates without executing the test run. Otherwise actions preprocessing, run and postprocessing are called.
Finally the test result is checked by comparing data and writing differences to COMPARE_OUT (compare.out) file in test run directory. When COMPARE_OUT is empty, the test status is success, and failed otherwise. The result is written to console and to results.out in the test run directory by the report action.
// testRun.cmd
@echo off
rem execute test suite
rem 1 - work suite directory name below main_suite
pushd .
rem prepare test work suite ... (global actions)
call %ROOT_DIR%\settings.cmd %1 %RUN_ROOT% %MAIN_SUITE% %ROOT_DIR%
call %ROOT_DIR%\prepare.cmd
if "%NO_RUN%" == "YES" goto end
rem run test ... (actions may be overloaded)
call %TEST_ACT%\preprocessing.cmd %1
call %TEST_ACT%\run.cmd %1
call %TEST_ACT%\postprocessing.cmd %1
rem create test report
call %ROOT_DIR%\report.cmd %1
:end
popd
// settings.cmd
rem Set test environment variables
rem 1 - relative test suite path (e.g. 0/1/2)
rem 2 - test run directory (%RUN_PATH%)
rem 3 - main suite directory (%ROOT_PATH%)
rem 4 - test root directory (%cd%)
rem 5 - work directory (%WORK_DIR%)
rem global test environment
if "%4" == "" (set ROOT_DIR=%cd%) else (set ROOT_DIR=%~4)
if "%3" == "" (set MAIN_SUITE=%ROOT_PATH%) else (set MAIN_SUITE=%~3)
if "%2" == "" (set RUN_ROOT=%RUN_PATH%) else (set RUN_ROOT=%~2)
if "%1" == "" (set TEST_SUITE=%MAIN_SUITE%) else (set TEST_SUITE=%MAIN_SUITE%\%~1)
set TOOLS_DIR=%ROOT_DIR%\binaries\tools
set BIN_DIR=%ROOT_DIR%\binaries
rem test work area settings
if "%~5" == "" (set WORK_AREA=%WORK_DIR%) else (set WORK_AREA=%~5)
if "%WORK_AREA%" == "" set WORK_AREA=%ROOT_DIR%
set WORK_AREA=%WORK_AREA%\work_area
set TEST_ACT=%WORK_AREA%\actions
rem prepare test run directory
if not exist %RUN_ROOT% mkdir %RUN_ROOT%
set TEST_RUN=%RUN_ROOT%\%~1
set LOGFILE_OUT=%RUN_ROOT%\logfile.out
set RESULTS_OUT=%RUN_ROOT%\results.out
set REPORT_OUT=%RUN_ROOT%\report.out
if not exist %TEST_RUN% mkdir %TEST_RUN%
rem error protocols
set ERRORS_OUT=%TEST_RUN%\errors.out
set COMPARE_OUT=%TEST_RUN%\compare.out
:end
// prepare.cmd
rem Initialize test folder
rem delete last test suite results
cd /D %TEST_SUITE%
if exist %COMPARE_OUT% del %COMPARE_OUT%
if exist %ERRORS_OUT% del %ERRORS_OUT%
rem delete work suite data
del /Q /S %WORK_AREA%\data\*.* > nul
del /Q /S %WORK_AREA%\expected\*.* > nul
del /Q /S %WORK_AREA%\actions\*.* > nul
del /Q /S %TEST_RUN%\failed\*.* > nul
rem copy resources
call %ROOT_DIR%\copyRecursive.cmd
// copyRecursive.cmd
rem Copy test data recursively from test case hierarchy
if exist suite (
pushd ..\.
call %ROOT_DIR%\copyRecursive.cmd
popd
if exist data xcopy /Y data\*.* %WORK_AREA%\data\*.* > nul
if exist expected xcopy /Y expected\*.* %WORK_AREA%\expected\*.* > nul
if exist actions xcopy /Y actions\*.* %WORK_AREA%\actions\*.* > nul
)
// report.cmd
rem evaluate termination code
rem 1 - work suite directory name below test root\0
set RESULT=success
if exist "%COMPARE_OUT%" (
%TOOLS_DIR%\diff "%ROOT_DIR%\main_suite\compare.top" "%COMPARE_OUT%" >> nul || set RESULT=failed
) else (
set RESULT=failed
echo ... nothing to compare (expected empty) >%COMPARE_OUT%
)
echo %1; %RESULT% >>%RESULTS_OUT%
echo %1 terminated: %RESULT%
if not exist %TEST_RUN%\run copy %TEST_SUITE%\run %TEST_RUN%\run >>nul
Remarks about the test runs may have been stored in each test run directory in the run file. In order to print out a comprehensive test protocol, one may call TestOut framework action (TestOut.cmd). Via calling action callNormalized in order to change Linux paths to Windows path the outRun is called for each sub directory containing a run file.
More reporting features are provided with ODABA Test Browser application.
// TestOut.cmd
@echo off
rem list test results
rem 1 - relative test suite path (e.g. 0/1/2)
rem 2 - test run directory (%RUN_PATH%)
rem 3 - main suite directory (%ROOT_PATH%)
rem 4 - test root directory (%cd%)
call settings.cmd %1 %2 %3 %4
if exist %REPORT_OUT% del %REPORT_OUT%
pushd %RUN_ROOT%
%TOOLS_DIR%\find . -type d -exec %ROOT_DIR%\outSingle.cmd {} %1 ;
type %RESULTS_OUT%
popd
// outSingle.cmd
@echo off
rem run single test in hierarchy
rem 1 - test folder name below top suite ...\main_suite
call %ROOT_DIR%\callNormalized.cmd %ROOT_DIR%\outRun.cmd %1
// outRun.cmd
@echo off
rem 1 - test folder name below main suite ...\main_suite
call %ROOT_DIR%\settings.cmd %1 %ROOT_DIR%
if exist %TEST_SUITE%\run (
pushd %TEST_RUN%
echo Tested %1>>%REPORT_OUT%
if exist run type run >>%REPORT_OUT%
echo .>>%REPORT_OUT%
echo ----------------------------------------->>%REPORT_OUT%
popd
)