TMMi Level 2 Managed

In level 2, testing becomes a managed process and is clearly separated from debugging. The process discipline reflected by maturity level 2 helps to ensure that existing practices are retained during times of stress. However, it is by many stakeholders still perceived as being a project phase that follows coding. In the context of improving the test process, a company-wide or programme-wide test strategy is established. Test plan are also being developed. Within the test plan a test approach is defined, whereby the approach is based on the level of risk. Risk management techniques are used to identify the product risks based on documented requirements. The test plan will define what testing is required, when, how and by whom. Commitments are established among relevant stakeholders and revised as needed. Testing is monitored and controlled to ensure it is going according to plan and actions can be taken if deviations occur. The status of the work products and the delivery of testing services are visible to management. For deriving and selecting test cases from specifications test design techniques are applied. However, testing may still start relatively late in the development life cycle, e.g. during the design or even during the coding phase. Testing is multi-leveled: there are unit, integration, system and acceptance test levels. For each identified test level there are specific testing objectives defined in the organization-wide or programme-wide test strategy. The main objective of testing in a TMMi level 2 organizations is to verify that the product satisfies the specified requirements. The purpose is also to clearly differentiate the processes of testing and debugging. Many quality problems at this TMMi level occur because testing occurs late in the development life cycle. Defects are propagated from the requirements and design into code. There are no review programs as yet to address this important issue. Post code, execution based testing is by many stakeholders still considered the primary testing activity.

The process areas at TMMi level 2 are:

• Test Policy and Strategy
• Test Planning
• Test Monitoring and Control
• Test Design and Execution
• Test Environment

Test Policy and Strategy

The purpose of Test Policy and Strategy is to develop and establish a test policy and an organizationwide or programme-wide test strategy in which the test levels are unambiguously defined. To measure test performance test performance indicators are introduced.

When an organization wants to improve its test process, it should first clearly define a test policy. The test policy defines the organization’s overall test objectives, goals and strategic views regarding testing. It is important that the test policy is aligned with the overall business (quality) policy of the organization. A test policy is necessary to attain a common view on testing between all stakeholders within an organization. This common view is indispensable to align test (process improvement) activities. The test policy should address both new development and maintenance testing activities. Within the test policy the objectives for test process improvement should be stated. These objectives will subsequently be translated into a set of key test performance indicators. The test policy and the accompanying performance indicators provide a clear direction, and a means to communicate expected and achieved levels of test performance. The performance indicators have the objective to show the value of testing and test process improvement to stakeholders.

Based upon the test policy a test strategy will be defined. The test strategy covers the generic test requirements for an organization or programme (one or more projects). The test strategy addresses the generic product risks and presents a process for mitigating those risks in line with the testing policy. A typical test strategy will include a description of the test levels that are to be applied, for example: unit, integration, system and acceptance test. For each test level amongst others the objectives, responsibilities, main tasks and entry/exit criteria are defined. The test strategy serves as a starting point for the testing activities within projects. The projects are set up in accordance with the organization-wide or programme-wide test strategy. When a test strategy is defined and followed, less overlap between the test levels is likely to occur leading to a more efficient test process. Also since the test objectives and approach of the various levels is aligned fewer holes are likely to remain leading to a more effective test process.

The process area Test Policy and Strategy involves the definition and deployment of a test policy and test strategy. Within the test strategy, test levels are identified. For each test level, amongst other test objectives, responsibilities and main tasks are defined. To measure test performance and the accomplishment of test (improvement) goals test performance indicators are defined and deployed.

Test Planning

The purpose of Test Planning is to define a test approach based on the identified risks and the defined test strategy, and to establish and maintain well-founded plans for performing and managing the testing activities.

After confirmation of the test assignment, an overall study is carried out regarding the product to be tested, the project organization, the requirements, and the development process. As part of test planning, the test approach is defined by based on the outcome of a product risk assessment. Depending on the level and type of risks, it is decided which requirements of the product will be tested, to what degree and how. The objective is to provide the best possible level and type of coverage to the parts of the system with the highest risk level. Based on the test approach the work to be done is estimated and as a result the proposed test approach is provided with clear cost. The product risks, test approach and estimates are defined in close co-operation with the stakeholders. Testers should not take these decisions themselves. The test plan will either confirm, or explain noncompliance, with the test strategy. Within test planning, the test deliverables that are to be provided are identified, the resources that are needed are determined, and aspects relating to infrastructure are defined. In addition, test project risks regarding testing are identified. As a result the test plan will define what testing is required, when, how and by whom. Finally, the test plan document is developed and committed upon by the stakeholders. The test plan provides the basis for performing and controlling the testing activities. The test plan will usually need to be revised, using a formal change control process, as the project progresses to address changes in the requirements and commitments, inaccurate estimates, corrective actions, and (test) process changes.

The process area Test Planning involves performing a product risk assessment on the test object and defining a differentiated test approach based on the risks identified. It also involved developing estimates for the testing to be performed, establishing necessary commitments, and defining and maintaining the plan to perform and manage the testing. A test plan is required for each identified test level. At level 2 test plans are typically developed per test level.

Test Monitoring and Control

The purpose of Test Monitoring and Control is to provide an understanding of test progress and product quality so that appropriate corrective actions can be taken when test progress deviates significantly from plan or product quality deviates significantly from expectations.

The progress of testing and the quality of the products should both be monitored and controlled. The progress of the testing is monitored by comparing the status of actual test (work) products, tasks (including their attributes), effort, cost, and schedule to the test plan. The quality of the product is monitored by means of indicators such as product risks mitigated, the number of defects found, number of open defects, and status against test exit criteria. Monitoring involves gathering the required (raw) data, e.g. from test log and test incidents reports, reviewing the raw data on their validity and calculating the defined progress and product quality measures. Test summary reports should be written on a periodic and event driven basis as a means to provide a common understanding on test progress and product quality. Since ‘testing is the measurement of product quality’ [Hetzel], especially the practices around product quality reporting are key to the success of this process area. Appropriate corrective actions should be taken in time when the test progress deviates from the plan or product quality deviates from expectations. These actions may require re-planning, which may include revising the original plan or additional mitigations activities with the current plan. Corrective actions that influence the original committed plan should be agreed with the stakeholders.

An essential part of test monitoring and control is test project risk management. Test project risk management is performed to identify and solve major problems that undermine the test plan as early as possible. When performing project risk management, it is also important to identify problems that are beyond the responsibility of testing. For instance, organizational budget cuts, delay of development work products or changed/added functionality can all affect the test process significantly. Building on the test projects risks documented in the test plan, test project risks are monitored and controlled and corrective actions are initiated as needed.

Test Design and Execution

The purpose of Test Design and Execution is to improve test process capability during test design and execution by establishing test design specifications using test design techniques, performing a structured test execution process and managing test incidents to closure.

Structured testing implies that test design techniques are applied, possibly supported by tools. Test design techniques are used to derive and select test conditions and subsequently test cases from requirements and design specifications. The test conditions and test cases are documented in a test specification. A test case consists of the description of the input values, execution preconditions, expected results and execution post conditions. At a later stage, as more information becomes available regarding the implementation, the test cases are translated into test procedures. In a test procedure, also referred to as a manual test script, the specific test actions and checks are arranged in an executable sequence. Specific test data required to be able to run the test procedure is created. The tests will subsequently be executed using these test procedures. The test design and execution activities follow the test approach as defined in the test plan. The specific test design techniques applied (e.g. black-box, white-box or experienced-based) are amongst other based on level and type of product risk identified during test planning.

During the test execution stage, incidents are found and incident reports are written. Incidents are be logged using an incident management system and communication regarding the incidents with stakeholders is established. For incident management a basic incident classification scheme is established, and a procedure is put in place to handle the incident life cycle process and manage them to closure.

The process area Test Design and Execution addresses the test preparation phase including the application of test design techniques to derive and select test conditions and test cases. It also addresses the creation of specific test data, the execution of the tests using documented test procedures and incident management.

Test Environment

The purpose of Test Environment is to establish and maintain an adequate environment, including test data, in which it is possible to execute the tests in a manageable and repeatable way.

A managed and controlled test environment is indispensable for any testing. It is also needed to obtain test results under conditions which are as close as possible to the ‘real-life’ situation. This is especially true for higher level testing, e.g. at system and acceptance test level. Furthermore, at any test level the reproducibility of test results should not be endangered by undesired or unknown changes in the test environment.

Specification of test environment requirements is performed early in the projects. The requirements specification is reviewed to ensure their correctness, suitability, feasibility and its representativeness towards the ‘real-life’ operational environment. Early requirements specification has the advantage that there is more time to acquire and/or develop the required test environment and components such as simulators, stubs or drivers. The type of environment required will depend on the product to be tested and the test types, methods and techniques used.

Availability of a test environment encompasses a number of issues which need to be dealt with: Is it necessary for testing to have an environment per test level? A separate test environment per test team or per test level can be very expensive. Maybe it is possible to have the same environment shared between testers and developers. But then strict management and control is necessary as both testing and development activities are done in the same environment. When poorly managed, this situation can cause many problems ranging from conflicting reservations to people finding the environment in an unknown or undesired state when starting one’s activities.

Finally test environment management also includes amongst other managing access to the test environment by providing log-in details, managing test data, configuration management and providing technical support on progress disturbing issues during test execution.

As part of the test environments process area also the requirements regarding generic test data, and the creation and management of the test data is addressed. Whereas specific test data is defined during the test design and analysis activity, more generic test data is often defined and created as a separate activity. Generic test data is re-used by many testers and provides overall background data that is needed to perform the system functions. Generic test data often consists of master data and some initial content for primary data. Sometimes timing requirements influence this activity.

The process area Test Environment addresses all activities for specifying test environment requirements, implementing the test environment and managing and controlling the test environment. Managing and control of the test environment also includes aspects such as configuration management and ensuring availability. The test environment process area has both the physical test environment and the test data within its scope.