TMMi Maturity Levels |
|
|
Overview TMMi has a staged architecture for process improvement. It contains stages or levels through which an organization passes as its testing process evolves from one that is ad-hoc and unmanaged, to one that is managed, defined, measured, and optimized. Achieving each stage ensures that an adequate improvement has been laid as a foundation for the next stage. The internal structure of the TMMi is rich in testing practices that can be learned and applied in a systematic way to support a quality testing process that improves in incremental steps. There are five levels in the TMMi that prescribe a maturity hierarchy and an evolutionary path to test process improvement. Each level has a set of process areas that and organization needs to focus on to achieve maturity at that level. Experience has shown that organizations do their best when they focus their test improvement process efforts on a manageable  number of process areas at a time, and that those areas require increasing sophistication as the organization improves. Because each maturity level forms a necessary foundation for the next level, trying to skip a maturity level is usually counter productive. At the same time, you must recognize that test process improvement efforts should focus on the needs of the organization in the context of its business environment and the process areas at higher maturity levels may address the current needs of an organization or project. For example, organizations seeking to move from maturity level 1 to maturity level 2 are frequent encouraged to establish a test group, which is addressed by the Test Organization process area that resides are maturity level 3. Although the test group is not a necessary characteristic of a TMMi level 2 organization, it can be a useful part of the organization’s approach to achieve TMMi maturity level 2. The process areas for each maturity level of the TMMi are shown in figure 1. They are fully described later in other chapters and are also listed below along with a brief description of the characteristics of an organization at each TMMi level. The description will introduce the reader to the evolutionary path prescribed in the TMMi for test process improvement. Note that the TMMi does not have a specific process area dedicated to test tools and/or test automation. Within TMMi test tools are treated as a supporting resource (practices) and are therefore part of the process area where they provide support, e.g. applying a test design tool is a supporting test practice within the process area Test Design and Execution at TMMi level 2 and applying a performance testing is tool is a supporting test practice within the process area Non-Functional Testing at TMMi level Level 1 Initial At TMMi level 1, testing is a chaotic, undefined process and is often considered a part of debugging. The organization usually does not provide a stable environment to support the processes. Success in these organizations depends on the competence and heroics of the people in the organization and not the use of proven processes. Tests are developed in an ad-hoc way after coding is completed. Testing and debugging are interleaved to get the bugs out of the system. The objective of testing at this level is to show that the software runs without major failures. Products are released without adequate visibility regarding quality and risks. In the field, the product does often not fulfill its needs, is not stable, or is too slow to work with. Within testing there is a lack of resources, tools and well-educated staff. At TMMi level 1 there are no defined process areas. Maturity level 1 organizations are characterized by a tendency to over commit, abandonment of processes in a time of crises, and an inability to repeat their successes. Also products tend not to be released on time, budgets are overrun and quality is not according to expectations. Level 2 Managed At TMMi level 2, testing becomes a managed process and is clearly separated from debugging. The process discipline reflected by maturity level 2 helps to ensure that existing practices are retained during times of stress. However, testing is by many stakeholders still perceived as being a project phase that follows coding. In the context of improving the test process, a company-wide or programme-wide test strategy is established. Test plans are also being developed. Within the test plan a test approach is defined, whereby the approach is based on the result of a product risk assessment. Risk management techniques are used to identify the product risks based on documented requirements. The test plan defines what testing is required, when, how and by whom. Commitments are established with stakeholders and revised as needed. Testing is monitored and controlled to ensure it is going according to plan and actions can be taken if deviations occur. The status of the work products and the delivery of testing services are visible to management. For deriving and selecting test cases from specifications test design techniques are applied. However, testing may still start relatively late in the development life cycle, e.g. during the design or even during the coding phase. Testing is multileveled: there are unit, integration, system and acceptance test levels. For each identified test level there are specific testing objectives defined in the organization-wide or programme-wide test strategy. The main objective of testing in a TMMi level 2 organizations is to verify that the product satisfies the specified requirements. The purpose is also to clearly differentiate the processes of testing and debugging. Many quality problems at this TMMi level occur because testing occurs late in the development life cycle. Defects are propagated from the requirements and design into code. There are no formal review programs as yet to address this important issue. Post code, execution based testing is by many stakeholders still considered the primary testing activity. The process areas at TMMi level 2 are:
Level 3 Defined At TMMi level 3, testing is no longer a phase that follows coding. It is fully integrated into the development life cycle and the associated milestones. Test planning is done at an early project stage, e.g. during the requirements phase, by means of a master test plan. The development of a master test plan builds on the test planning skills acquired at level 2. The organization’s set of standard processes, which is the basis for maturity level 3, is established and improved over time. A test organization and a specific test training program exist, and testing is perceived as being a profession. Test cases are gathered, stored and managed in a central database for re-use and regression testing. Basic tools support key testing activities. Organizations at this level begin to realize the importance of reviews in quality control; a formal review program is implemented although not yet linked to the dynamic testing process. Reviews take place across the life cycle. Test professionals are involved in reviews on requirements specifications. Whereby the test designs at TMMi level 2 focus mainly on functionality testing, test designs and test techniques are expanded, depending the business objectives, to also include non-functional testing, e.g. on usability and/or reliability. A critical distinction between TMMi maturity level 2 and 3 is the scope of the standards, process descriptions, and procedures. At maturity level 2 these may be quite different in each specific instance, e.g. on a particular project. At maturity level 3 these are tailored from the organization’s set of standard processes to suit a particular project or organization unit and therefore are more consistent except for the differences allowed by the tailoring guidelines. Another critical distinction is that at maturity level, processes are typically described more rigorously than at maturity level 2. As a consequence at maturity level 3, the organization must revisit the maturity level 2 process areas. The process areas at TMMi level 3 are:
Level 4 Management and Measurement In TMMi 4 organizations testing is a thoroughly defined, well-founded and measurable process. At maturity level 4, the organization and projects establish quantitative objectives for product quality and process performance and use them as criteria in managing them. Product quality and process performance is understood in statistical terms and is managed throughout the life cycle. Measures are incorporated into the organization’s measurement repository to support fact-based decision making. Reviews and inspections are considered to be part of testing and used the measure document quality. The static and dynamic testing approach are integrated into one. Reviews are formally used as means © 2008 TMMi Foundation V1.0 10 to control quality gates. Products are evaluated using quantitative criteria for quality attributes such as reliability, usability and maintainability. An organization wide test measurement program provides information and visibility regarding the test process. Testing is perceived as evaluation; it consists of all life cycle activities concerned with checking products and related work products. The process areas at TMMi level 4 are:
Level 5 Optimization On the basis of all results that have been achieved by fulfilling all the improvement goals of the previous maturity levels, testing is now a completely defined process and one is capable of controlling the costs and the testing effectiveness. At TMMi maturity level 5, an organization continually improves it processes based on a quantitative understanding of the common cause of variation inherent in processes. Improving test process performance is carried out through incremental and innovative process and technological improvements. The methods and techniques are optimized and there is a continuous focus on fine-tuning and test process improvement. Defect prevention and quality control are practiced. Statistical sampling, measurements of confidence levels, trustworthiness, and reliability drive the test process. Amongst others “Defect Prevention†and “Quality Control†are introduced as process areas. The test process is characterized by sampling based quality measurements. A detailed procedure exists for selecting and evaluating test tools. Tools support the test process as much as possible during test design, test execution, regression testing, test case management, etc. Process reuse is also practiced at level 5 supported by a process asset library. Testing is a process with the objective to prevent defects. Process areas at level 5 are:
|