Testing process assessment and optimization

Return on Investment Calculator

Here is a checklist for testing process assessment:

Testing process assessment objectives

  • Diagnose testing process weaknesses
  • Recognize a need for new test tools
  • Technical evaluation of test practices
  • Economical evaluation of test practices

Testing process documentation

Testing process should be consistently documented in order to:

  • be communicated to other teams
  • be executed in a repeatable way
  • be effectively improved

Data collected for assessment

  • System complexity
  • Complexity of development stage under test
  • Testing process goals, objectives and strategies
  • Development stage under test cost/budget
  • Corresponding testing process cost/budget - by testing phase/test type
  • Development and testing schedule
  • Testing process inputs/outputs
  • Testing resources:
    • Test team - structure and skills, test planning, execution and automation responsibilities
    • Hardware resources
    • Software resources and testing tools
  • Total number of tests
  • Testing phases and number of tests per phase
  • Test techniques used
  • Number of automated tests - per phase - and tools used
  • Test coverage information:
    • Requirements coverage
    • Acceptance criteria coverage
    • Structural coverage
    • Data path coverage
    • Source code lines coverage
  • Number of defects detected by each testing phase - according to module/severity/test type
  • Number of defects found on beta sites - according to module/severity
  • Number of defects reported by users - according to module/severity
  • Costs of critical defects reported by users
  • Potential cost of defects found by testing - had them not been detected

Testing process prerequisites

  • Goals, objectives, strategies and methodology have been defined
  • Testing process has been documented and reviewed
  • Budget is adequate
  • Testing team size, structure, skills, experience and training are adequate
  • Space, hardware and software resources with technical support are sufficient
  • All the necessary testing tools are accessible - with training and support
  • Work plan is defined, understandable and doable
  • Testing metrics are used
  • Testing starts from requirements/specification phase and goes in parallel to system development
  • Users are involved

Testing metrics

  • Coverage metrics
  • User involvement
  • Relative Testing Costs (testing costs/total costs)
  • Cost per defect found
  • Testing budget conformity (real/planned)
  • Test efficiency (number of defects found/number of tests)
  • Not detected start-up failures (easily detectable major defects)
  • Not detected critical defects
  • Test automation ratio (automated tests/total tests)
  • Testing effectiveness (number of defects found/test cost) - per module/test phase/test type
  • DDP - Defect Detection Percentage (number of defects found by testing/total known defects)

Testing process optimization

Results of assessment are used for testing process improvements.

Testing metrics provide feedback about effectiveness of correcting actions.

Periodical cycles of assessment/modification maximize testing process business value.

Home | Site map | Search | Terms of Use | Privacy Policy | Contact us