company logo

Test framework actions (Windows)

The cmd test framework consists of a number of procedures managing tests under Windows.

In order to run a test, all required resources are copied to the work area (work_area) which is structured according to the requirements of the component to be tested. When running a series of tests in the test framework, the work area will be overwritten each time, a new test is started. In order to guarantee traceability, the work area has to be copied to the test suite running the test (e.g. as zip file). Here, only differing files are stored to the failed directory in the test suite.

The test framework provides a set of common actions (command line procedures) for managing test preparation and evaluation. In addition, each specific test environment has to provide adopted actions named preprocessing, run and postprocessing (command line procedures with extension .cmd), which are usually stored in main_suite/actions directory, and possibly overloaded in several test cases or test suites. Test framework actions (procedures) are stored in the test root directory.

Framework actions that may be called from command line are RunTest.cmd, RunMany.cmd, SetupWorkArea.cmd and TestOut.cmd. In order to support parallel testing, the work area location may also be passed as parameter to RunTest or RunMany action. Default is the work_area directory below the test root directory.

Starting test run(s)

In order to execute a single or a number test cases, framework actions RunTest or RunMany may be called.

For executing a single test case, RunTest is called. In order to locate the proper test case, the relative path has to be defined like API/Local/Property.

In order to run the complete test suite hierarchy, RunMany may to be called. The procedure executes all test case directories containing a run file. In order to filter test cases for execution, one may rename the run file in test case directories to be excluded.

For each test directory, runSingle will be called. runSingle executes a single test suite after checking the presence of a run file in the directory. When the run file exists, testRun is called. testRun is called via callNormalized, which normalizes the directory path by removing ./ at the beginning of the path name (this type of path names are not supported by older command line functions as type or copy).

// RunTest.cmd

  @echo off

  rem execute single test

  rem 1 - relative test suite path (e.g. 0/1/2)

  rem 2 - test run directory (%RUN_PATH%)

  rem 3 - main suite directory (%ROOT_PATH%)

  rem 4 - test root directory (%cd%)

  rem 5 - work directory (%WORK_DIR%)

  

  call %4\settings.cmd %1 %2 %3 %4 %5

  

  pushd %RUN_ROOT%

  call %ROOT_DIR%\runSingle.cmd ./%~1

  popd

// RunMany.cmd

  @echo off

  rem execute test run(s)

  rem 1 - test run directory (%RUN_PATH%)

  rem 2 - main suite directory (%ROOT_PATH%)

  rem 3 - test root directory (%cd%)

  

  if "%3" == "" (call %cd%\settings.cmd . %1 %2) else (call %3\settings.cmd . %1 %2 %3)

  

  pushd %2

  %TOOLS_DIR%\find . -type d -exec %ROOT_DIR%\runSingle.cmd {} ;        

  popd

// runSingle.cmd

  @echo off

  rem run single test in hierarchy

  rem 1 - test folder name below main suite ...\main_suite

  

  if exist %MAIN_SUITE%\%1\run %ROOT_DIR%\callNormalized.cmd %ROOT_DIR%\testRun.cmd %1

// CallNormalized.cmd

@echo off

rem convert Linux path to MS Windows path (./a/b... --> a\b...)

rem 1 - procedure to be called

rem 2 - "find" name for directory or file

FOR /F "tokens=2,3,4,5 delims=/" %%G IN ('echo %2') DO call %1 %%G\%%H\%%I\%%J %2

Running a test within the frame work

In order to run a test, the testRun action is provided by the test frame work (testRun.cmd). The testRun action updates settings for environment variables, prepares the work area and calls test environment specific actions for the selected work suite. settings and prepare are framework actions, which cannot be overloaded, while preprocessing, run and postprocessing are test environment specific actions, which have to be defined for each specific test environment in top or lower test suite or test case actions directory.

After setting test environment variables (settings action), the prepare action copies test data (actions, data, expected) to corresponding directories in work_area from the test suite hierarchy. Test data from test cases and test suites is copied recursively (copyRecursive) from the test suite hierarchy by copying resources from the test case to be tested and all its parent test suites. A directory is considered as test suite, when it contains a file suite. Copying stops, when the first directory was found, which is not a test suite. Files existing on lower levels in the hierarchy, always will overwrite files with the same name provided on higher levels.

When NO_RUN has been set to YES (SetupWorkArea), the testRun action terminates without executing the test run. Otherwise actions preprocessing, run and postprocessing are called.

Finally the test result is checked by comparing data and writing differences to COMPARE_OUT (compare.out) file in test run directory. When COMPARE_OUT is empty, the test status is success, and failed otherwise. The result is written to console and to results.out in the test run directory by the report action.

// testRun.cmd

  @echo off

  rem execute test suite

  rem 1 - work suite directory name below main_suite

  pushd .

  rem prepare test work suite ... (global actions)

  call %ROOT_DIR%\settings.cmd %1 %RUN_ROOT% %MAIN_SUITE% %ROOT_DIR%

  call %ROOT_DIR%\prepare.cmd

  if "%NO_RUN%" == "YES" goto end

  rem run test ... (actions may be overloaded)

  call %TEST_ACT%\preprocessing.cmd %1

  call %TEST_ACT%\run.cmd %1

  call %TEST_ACT%\postprocessing.cmd %1

  rem create test report

  call %ROOT_DIR%\report.cmd %1

  :end

  popd

// settings.cmd

  rem Set test environment variables

  rem 1 - relative test suite path (e.g. 0/1/2)

  rem 2 - test run directory (%RUN_PATH%)

  rem 3 - main suite directory (%ROOT_PATH%)

  rem 4 - test root directory (%cd%)

  rem 5 - work directory (%WORK_DIR%)

  

  rem global test environment

  if "%4" == "" (set ROOT_DIR=%cd%) else (set ROOT_DIR=%~4)

  if "%3" == "" (set MAIN_SUITE=%ROOT_PATH%) else (set MAIN_SUITE=%~3)

  if "%2" == "" (set RUN_ROOT=%RUN_PATH%) else (set RUN_ROOT=%~2)

  if "%1" == "" (set TEST_SUITE=%MAIN_SUITE%) else (set TEST_SUITE=%MAIN_SUITE%\%~1)

  set TOOLS_DIR=%ROOT_DIR%\binaries\tools

  set BIN_DIR=%ROOT_DIR%\binaries

  

  rem test work area settings

  if "%~5" == "" (set WORK_AREA=%WORK_DIR%) else (set WORK_AREA=%~5)

  if "%WORK_AREA%" == "" set WORK_AREA=%ROOT_DIR%

  set WORK_AREA=%WORK_AREA%\work_area

  set TEST_ACT=%WORK_AREA%\actions

  

  rem prepare test run directory

  if not exist %RUN_ROOT% mkdir %RUN_ROOT%

  set TEST_RUN=%RUN_ROOT%\%~1

  set LOGFILE_OUT=%RUN_ROOT%\logfile.out

  set RESULTS_OUT=%RUN_ROOT%\results.out

  set REPORT_OUT=%RUN_ROOT%\report.out

  if not exist %TEST_RUN% mkdir %TEST_RUN%

  

  rem error protocols

  set ERRORS_OUT=%TEST_RUN%\errors.out

  set COMPARE_OUT=%TEST_RUN%\compare.out

  :end

// prepare.cmd

  rem Initialize test folder

  rem delete last test suite results

  cd /D %TEST_SUITE%

  if exist %COMPARE_OUT% del %COMPARE_OUT%

  if exist %ERRORS_OUT% del %ERRORS_OUT%

  

  rem delete work suite data

  del /Q /S %WORK_AREA%\data\*.* > nul

  del /Q /S %WORK_AREA%\expected\*.* > nul

  del /Q /S %WORK_AREA%\actions\*.* > nul

  del /Q /S %TEST_RUN%\failed\*.* > nul

  

  rem copy resources

  call %ROOT_DIR%\copyRecursive.cmd

// copyRecursive.cmd

  rem Copy test data recursively from test case hierarchy

  if exist suite (

      pushd ..\.

   call %ROOT_DIR%\copyRecursive.cmd

   popd

   if exist data xcopy /Y data\*.* %WORK_AREA%\data\*.* > nul

   if exist expected xcopy /Y expected\*.* %WORK_AREA%\expected\*.* > nul

   if exist actions xcopy /Y actions\*.* %WORK_AREA%\actions\*.* > nul

  )

// report.cmd

  rem evaluate termination code

  rem 1 - work suite directory name below test root\0

  

  set RESULT=success

  if exist "%COMPARE_OUT%" (

   %TOOLS_DIR%\diff "%ROOT_DIR%\main_suite\compare.top" "%COMPARE_OUT%" >> nul || set RESULT=failed

  ) else (

   set RESULT=failed

   echo ... nothing to compare (expected empty) >%COMPARE_OUT%

  )

  echo %1; %RESULT% >>%RESULTS_OUT%

  echo %1 terminated: %RESULT%

  if not exist %TEST_RUN%\run copy %TEST_SUITE%\run %TEST_RUN%\run >>nul

Output test protokol

Remarks about the test runs may have been stored in each test run directory in the run file. In order to print out a comprehensive test protocol, one may call TestOut framework action (TestOut.cmd). Via calling action callNormalized in order to change Linux paths to Windows path the outRun is called for each sub directory containing a run file.

More reporting features are provided with ODABA Test Browser application.

// TestOut.cmd

  @echo off

  rem list test results

  rem 1 - relative test suite path (e.g. 0/1/2)

  rem 2 - test run directory (%RUN_PATH%)

  rem 3 - main suite directory (%ROOT_PATH%)

  rem 4 - test root directory (%cd%)

  

  call settings.cmd %1 %2 %3 %4

  if exist %REPORT_OUT% del %REPORT_OUT%

  

  pushd %RUN_ROOT%

  %TOOLS_DIR%\find . -type d -exec %ROOT_DIR%\outSingle.cmd {} %1 ;

  type %RESULTS_OUT%

  popd

// outSingle.cmd

  @echo off

  rem run single test in hierarchy

  rem 1 - test folder name below top suite ...\main_suite

  call %ROOT_DIR%\callNormalized.cmd %ROOT_DIR%\outRun.cmd %1

// outRun.cmd

  @echo off

  rem 1 - test folder name below main suite ...\main_suite

  call %ROOT_DIR%\settings.cmd %1 %ROOT_DIR%

  if exist %TEST_SUITE%\run (

    pushd %TEST_RUN%

    echo Tested %1>>%REPORT_OUT%

    if exist run type run >>%REPORT_OUT%

    echo .>>%REPORT_OUT%

    echo ----------------------------------------->>%REPORT_OUT%

    popd

  )