Design for Six Sigma Spotlight:
Statistically-based Test Optimization
"There is no way around it — we have to find ways to do more with less. The integrated program use of statistical techniques such as Design of Experiments, have proven themselves to be powerful enablers in our test optimization efforts to reduce cost and cycle time while providing our customers with confidence that our systems will perform."
Dr. Tom Kennedy, President,
Raytheon Integrated Defense Systems
Design for Six Sigma (DFSS) is a methodology used to predict, manage and improve performance, producibility and affordability for the benefit of Raytheon's customers. Figure 1 shows how DFSS practices support every stage of the Integrated Product Development System (IPDS), Raytheon's structured development process. These practices include:
- "Voice of the customer" modeling and evaluation as an integral part of the requirements analysis process.
- Statistical performance modeling and optimization.
- Focused application of Design for Manufacture and Assembly (DFMA) principles and best practices.
- Statistically-based test optimization techniques enablement of more effective and efficient testing.
- Predictable acceleration of product development cycle time using lean and critical chain concepts.
- Improved supplier performance using DFSS methodologies.
Raytheon is being challenged by customers to develop and deliver increasingly complex systems that meet user requirements in the shortest time, with the highest reliability, at the lowest cost. Given this challenge, there is increased pressure on integration, verification and validation (IV&V) activities to optimize performance. Toward this end, this article discusses the enabling application of use case modeling and analysis, and test case optimization strategies in IV&V.
Use Case Modeling and Analysis
A powerful enabler for validating that a system is achieving its intended mission is use case modeling and analysis. The application of use cases for modeling and analysis is, of course, standard practice in the realm of systems engineering. It is uncommon, however, for these use case models to be fully leveraged in the development of system test cases.
Given the complexities of modern systems, the testing of all possible scenarios quickly becomes unfeasible. The decision, then, is not just how to test but also what to test. The most common approach is for system and test engineers to consider the requirements and technologies involved in determination of the representative tests to be conducted. This typically results in requirements based testing that may not adequately consider whether the test conditions are statistically likely, or of high criticality, in operational scenarios. The integration of usage based probabilistic models with domain expertise has been shown to improve upon up-front understanding of requirements and result in greater system test efficiencies and test effectiveness.
For those unfamiliar with state diagrams and transition probabilities, let's discuss an application from everyday life — use of an ATM. Users who visit an ATM are interested in different use-case scenarios. They may be interested in withdrawing cash, checking their balance, depositing money, reviewing the status of a loan, or any combination or sequence of these (or other) banking activities. Capturing this information in the form of a state diagram (Figure 2), along with the expected (or transition) probability of each path enables development of a model. This model is used to simulate operational use, generating likely combinatorial paths (user scenarios) and their relative frequency of occurrence. Based on a Pareto analysis of these frequencies, likely paths are identified for further exploration and development of representative use cases. Critical but statistically unlikely paths are then typically added to the listing in order to ensure test completeness. A Raytheon project employing this usage-based model approach has demonstrated a 50 percent reduction in test associated costs while significantly improving upon its previously experienced quality levels during system certification testing.
Test Case Optimization
Another industry best practice has emerged with the motivation of further integrating domain expertise and statistical methods in order to most effectively cover the test space at the minimum cost and cycle time. Combinatorial design methods are employed to statistically assess the test coverage of existing and under-development test plans. This is accomplished by determining the percentage of n-way combinations between identified input test parameters that are covered by alternative test plans. Specific interest is given to those n-way combinations that are of technical interest from a domain and/or use-case perspective. The rdExpert™ statistical package designates individual requirements and two-way combinations to represent "critical test coverage" and higher-order combinations (typically including three- and four-way combinations) as part of an overall test coverage assessment. If three-way or higher combinations are of specific interest, they should be given priority over two-way combinations of lesser priority in the assessment of test coverage risk. Once the key individual and interoperability requirements have been identified from a technical and use case perspective, an optimized test plan is developed using Design of Experiments algorithms. The resulting experimental designs are mathematically orthogonal, thereby enabling resulting analysis and root causal analysis. Figure 3 is an example of a resulting rdExpert test coverage analysis diagram for an optimized plan. Benefits from the integrated program application of this test case optimization approach include an average 30 percent reduction in test costs while maintaining or improving upon existing test coverage.
The application of statistical methods (including those of use-case modelling and analysis, Combinatorial Design Methods and Design of Experiments) have been cited as an aerospace industry best practice that enables the achievement of higher levels of mission assurance, reduced cycle time, increased productivity and reduced cost. Raytheon's integrated program application of these techniques has delivered on this potential through enablement of our identification of risk and the optimization of our test planning and execution.
Neal Mackertich, Peter Kraus