Principles for arranging tests in test frameworks
The test framework is an appropriate environment, which allows running requested test scenarios. The test framework controls test runs including any number of test suites and manages test resources for running tests. Ideally, the test framework also communicates with the test management in order to return results for executed tests or test runs.
A simple test framework is the command shell (bash, cmd or other). Depending on implementation language and type (GUI or command line components) one may also refer to more advanced test frameworks (as Abbot, OpenCTF, Fitnesse etc.). Common principles can be identified in different frameworks. Because of better transparency, command shell procedures (bash, cmd) are used for providing essential test framework functions.
The task of a test framework is to perform tests conceptually described in test cases. Usually, test cases for component tests describe the way to test the component against specifications in requirements. In case of requirement driven tests, for each requirement any number of test cases may be defined. A test case defines a specific test aspect (for a requirement), e.g. borderline checks for a value or parameter. In order to completely test a test case, several tests may be necessary. A test provides rules (actions) and test data for running the test and expected test results.
Similar test cases are collected in test suites. Tests suites may form hierarchies by defining upper test suites containing a number of similar subordinated test suites. In order to execute automated tests, test suites are provided in a test framework. A test suite is the environment that provides necessary resources for running one or more test cases.
Test suites may be arranged in hierarchies, where each test suite includes all subordinated test suites. TestBrowser as well as the test frame work expect one and only one topmost test suite, which is called main suite.
A test run is the execution of a number of tests defined in one or more test suites. Within a test run any number of tests from different test suites may be collected. Typically, a test run, contains all tests for a test suite and its subordinated test suites.
For supporting statistics on test runs and for comparing test runs, results of each test run need to be stored. In the example, test run results are stored in a directory structure test run identifier/test identifier, but any other systematic including test run and test identifiers might be used. .
In order to support automated tests, naming conventions will help a lot. Usually, it does not matter, which naming conventions are agreed upon, as long as names can be filtered by means of regular expressions (e.g. having a specific prefix).