Test framework actions (Linux)

The bash test framework consists of a number of procedures for managing tests under Linux.

In order to run a test, all required resources are copied to the work area (work_area) which is structured according to the requirements of the component to be tested. When running a series of tests in the test framework, the work area will be overwritten each time, a new test is started. In order to guarantee traceability, the work area has to be copied to the test suite running the test (e.g. as zip file). Here, only differing files are stored to the failed directory in the test suite.

The test framework provides a set of common actions for managing test preparation and evaluation. In addition, each specific test environment has to provide adopted actions named preprocessing, run and postprocessing (bash file names without extension), which are usually stored in main_suite/actions directory, and possibly overloaded in several test cases or test suites. Test framework actions (procedures) are stored in the test root directory.

Several actions that are called internally by the framework are defined as functions in TestSuite.sh. Framework actions that may be called from command line are RunTest.sh, RunMany.sh, SetupWorkArea.sh and TestOut.sh. In order to support parallel testing, the work area location may also be passed as parameter to RunTest or RunMany action.

Starting test run(s)

In order to execute a single or a number test cases, framework actions RunTest or RunMany may be called.

For executing a single test case, RunTest is called. In order to locate the proper test case, the relative path has to be defined like API/Local/Property.

In order to run the complete test suite hierarchy, RunMany may to be called. The procedure executes all test case directories containing a run file. In order to filter test cases for execution, one may rename the run file in test case directories to be excluded.

For each test directory, runSingle will be called. runSingle executes a single test suite after checking the presence of a run file in the directory. When the run file exists, testRun is called.

# RunTest.sh

#!/bin/bash

source ./TestSuite.sh

# execute single test

# 1 - relative test suite path (e.g. 0/1/2)

test_suite=$1 # CHAR

# 2 - test run directory (${RUN_PATH%})

run_root=$2 # CHAR

# 3 - main suite directory (${ROOT_PATH})

main_suite=$3 # CHAR

# 4 - test root directory ($PWD)

root_dir=$4 # CHAR

# 5 - work area location (${WORK_DIR})

WORK_dir=$4 # CHAR

settings ${test_suite} ${run_root} ${main_suite} ${root_dir} ${work_dir} ;

pushd ${RUN_ROOT} > /dev/null

runSingle ./${test_suite}

popd > /dev/null

# RunMany.sh

#!/bin/bash

source ./TestSuite.sh

# execute single test

# 1 - relative test suite path (e.g. 0/1/2)

test_suite=$1 # CHAR

# 2 - test run directory (${RUN_PATH%})

run_root=$2 # CHAR

# 3 - main suite directory (${ROOT_PATH})

main_suite=$3 # CHAR

# 4 - test root directory ($PWD)

root_dir=$4 # CHAR

# 5 - work area location (${WORK_DIR})

work_dir=$4 # CHAR

settings ${test_suite} ${run_root} ${main_suite} ${root_dir} ${work_dir} ;

pushd %RUN_ROOT%

find ./ -type d -exec ${ROOT_DIR}\runSingle.sh {} $1 ;

popd

# run single test in hierarchy (TestSuite.sh)

runSingle() {

# 1 - test directory below main suite .../main_suite

test_suite=$1 # CHAR

#CODE

if [ -f ${MAIN_SUITE}/$1/run ]; then

testRun ${test_suite};

}

Running a test within the frame work

In order to run a test, the testRun action is provided by the test framework (TestSuite.sh). The testRun action updates settings for environment variables, prepares the work area and calls test environment specific actions for the selected work suite. settings and prepare are framework actions, which cannot be overloaded, while preprocessing, run and postprocessing are test environment specific actions, which have to be defined for each specific test environment in top or lower test suite or test case actions directory.

After setting test environment variables (settings action), the prepare action copies test data (actions, data, expected) to corresponding directories in work_area from the test suite hierarchy. Test data from test cases and test suites is copied recursively (copyRecursive) from the test suite hierarchy by copying resources from the test case to be tested and all its parent test suites. A directory is considered as test suite, when it contains a file suite. Copying stops, when the first directory was found, which is not a test suite. Files existing on lower levels in the hierarchy, always will overwrite files with the same name provided on higher levels.

When NO_RUN has been set to YES (SetupWorkArea), the testRun action terminates without executing the test run. Otherwise actions preprocessing, run and postprocessing are called.

Finally the test result is checked by comparing data and writing differences to COMPARE_OUT (compare.out) file in test run directory. When COMPARE_OUT is empty, the test status is success, and failed otherwise. The result is written to console and to results.out in the test run directory by the report action.

# selected actions called by testRun in TestSuite.sh

# execute singele test

testRun() {

# 1 - test directory below main suite .../main_suite

test_suite=$1 # CHAR

#CODE

pushd . > /dev/null

# prepare test work suite ... (global actions)

settings ${test_suite} ${RUN_ROOT} ${MAIN_SUITE} ${ROOT_DIR}

prepare

source ${TEST_ACT}/preprocessing

if [ ! "${NO_RUN}" == "YES" ]; then

# run test ... (actions may be overloaded)

preprocessing ${test_suite}

${TEST_ACT}/run ${test_suite}

${TEST_ACT}/postprocessing ${test_suite}

# create test report

report ${test_suite}

popd > /dev/null

}

# Set global test frame work variables

settings() {

# 1 - relative test suite path (e.g. 0/1/2)

test_suite=$1 # CHAR

# 2 - test run directory (${RUN_PATH%})

run_root=$2 # CHAR

# 3 - main suite directory (${ROOT_PATH})

main_suite=$3 # CHAR

# 4 - test root directory ($PWD)

root_dir=$4 # CHAR

# 5 - temporary work directory (${WORK_DIR})

work_dir=$5 # CHAR

#CODE

if [ "${root_dir}" == "" ]; then

export ROOT_DIR=$PWD;

else

export ROOT_DIR=${root_dir};

if [ "${main_suite}" == "" ]; then

export MAIN_SUITE=${ROOT_PATH};

else

export MAIN_SUITE=${main_suite};

if [ "${run_root}" == "" ]; then

export RUN_ROOT=${RUN_PATH};

else

export RUN_ROOT=${run_root};

if [ ! -d ${RUN_ROOT} ]; then

mkdir -p ${RUN_ROOT};

if [ "${test_suite}" == "" ]; then

export TEST_SUITE=${MAIN_SUITE};

else

export TEST_SUITE=${MAIN_SUITE}/${test_suite};

if [ "${work_dir}" == "" ]; then

if [ "${WORK_DIR}" == "" ]; then

WORK_AREA=${ROOT_DIR}/work_area;

else

WORK_AREA=${WORK_DIR};

else

WORK_AREA=${work_dir};

if [ ! -d ${WORK_AREA}/actions ]; then

mkdir -p ${RUN_ROOT}/actions;

if [ ! -d ${WORK_AREA}/data ]; then

mkdir -p ${RUN_ROOT}/data;

if [ ! -d ${WORK_AREA}/expected ]; then

mkdir -p ${RUN_ROOT}/expected;

export WORK_AREA

export TEST_ACT=${WORK_AREA}/actions

export BIN_DIR=${ROOT_DIR}/binaries

export TOOLS_DIR=${BIN_DIR}/tools

export TEST_RUN=${RUN_ROOT}/${test_suite}

export LOGFILE_OUT=${RUN_ROOT}/logfile.out

export RESULTS_OUT=${RUN_ROOT}/results.out

export REPORT_OUT=${RUN_ROOT}/report.out

export ERRORS_OUT=${TEST_RUN}/errors.out

export COMPARE_OUT=${TEST_RUN}/compare.out

export LD_LIBRARY_PATH=/usr/local/lib

export ODABA_TOOLS=/usr/local/lib/odaba/tools

export TEST_NAME=$(basename ${TEST_SUITE})

}

# Initialize test folder

prepare() {

#CODE

# delete last test suite results

rm -f ${COMPARE_OUT}

rm -f ${ERRORS_OUT}

# delete work suite data

rm -rf ${WORK_AREA}/data/*

rm -rf ${WORK_AREA}/expected/*

rm -rf ${WORK_AREA}/actions/*

rm -rf ${TEST_RUN}/failed/*

pushd ${TEST_SUITE} > /dev/null

copyRecursive

popd > /dev/null

}

# Hierarchical copies contents of data, expected and actions into the ${work_area}

copyRecursive() {

#CODE

if [ -f suite ]; then

pushd .. > /dev/null

copyRecursive

popd > /dev/null

cp -r data/* ${WORK_AREA}/data 2>/dev/null

cp -r expected/* ${WORK_AREA}/expected 2>/dev/null

cp -r actions/* ${WORK_AREA}/actions 2>/dev/null

}

# evaluate termination code

report() {

# 1 - test directory below main suite .../main_suite

test_suite=$1 # CHAR

test_result=success

#CODE

if [ -f ${COMPARE_OUT} ]; then

diff "${ROOT_DIR}/main_suite/compare.top" "${COMPARE_OUT}" > /dev/null || test_result=failed

else

test_result=failed

echo "... nothing to compare (expected empty)" >${COMPARE_OUT}

echo "${test_suite}; ${test_result}" >>${RESULTS_OUT}

echo "${test_suite} terminated: ${test_result}"

if [ ! -f ${TEST_RUN}/run ]; then

cp ${TEST_SUITE}/run ${TEST_RUN}/run

}

Output test protokol

Remarks about the test runs may have been stored in each test run directory in the run file. In order to print out a comprehensive test protocol, one may call TestOut framework action (TestOut.sh). The action calls outRun for each sub directory containing a run file.

More reporting features are provided with ODABA Test Browser application.

# TestOut.sh

#!/bin/bash

source ./TestSuite.sh

# execute single test

# 1 - relative test suite path (e.g. 0/1/2)

test_suite=$1 # CHAR

# 2 - test run directory (${RUN_PATH%})

run_root=$2 # CHAR

# 3 - main suite directory (${ROOT_PATH})

main_suite=$3 # CHAR

# 4 - test root directory ($PWD)

root_dir=$4 # CHAR

settings ${test_suite} ${run_root} ${main_suite} ${root_dir} ;

rm -f ${REPORT_OUT}

pushd ${RUN_ROOT}

find ./ -type d -exec ${ROOT_DIR}/outRun.sh {} ${test_suite} ;

type ${RESULTS_OUT}

popd

# create test run summary and print to REPORT_OUT (TestSuite.sh)

outRun() {

# 1 - test directory below main suite .../main_suite

test_suite=$1 # CHAR

#CODE

settings %1 %ROOT_DIR%

if [ -f ${TEST_SUITE}/run ]; then

pushd ${TEST_RUN} > /dev/null

echo Tested ${test_suite}>>${REPORT_OUT}

if [ -f run ]; then

type run >>${REPORT_OUT}

echo .>>${REPORT_OUT}

echo ----------------------------------------->>${REPORT_OUT}

popd > /dev/null

}

to top Comment