1st Edition
Introduction to Combinatorial Testing
Combinatorial testing of software analyzes interactions among variables using a very small number of tests. This advanced approach has demonstrated success in providing strong, low-cost testing in real-world situations. Introduction to Combinatorial Testing presents a complete self-contained tutorial on advanced combinatorial testing methods for real-world software.
The book introduces key concepts and procedures of combinatorial testing, explains how to use software tools for generating combinatorial tests, and shows how this approach can be integrated with existing practice. Detailed explanations and examples clarify how and why to use various techniques. Sections on cost and practical considerations describe tradeoffs and limitations that may impact resources or funding. While the authors introduce some of the theory and mathematics of combinatorial methods, readers can use the methods without in-depth knowledge of the underlying mathematics.
Accessible to undergraduate students and researchers in computer science and engineering, this book illustrates the practical application of combinatorial methods in software testing. Giving pointers to freely available tools and offering resources on a supplementary website, the book encourages readers to apply these methods in their own testing projects.
Combinatorial Methods in Testing
Software Failures and the Interaction Rule
Two Forms of Combinatorial Testing
Covering Arrays
The Test Oracle Problem
Quick Start: How to Use the Basics of Combinatorial Methods Right Away
Combinatorial Testing Applied
Document Object Model, Carmelo Montanez-Rivera, D. Richard Kuhn, Mary Brady, Rick Rivello, Jenise Reyes Rodriguez, and Michael Powers
Rich Web Applications, Chad M. Maughan
Configuration Testing
Runtime Environment Configurations
Highly Configurable Systems and Software Product Lines
Invalid Combinations and Constraints
Cost and Practical Considerations
Input Testing
Partitioning the Input Space
Input Variables versus Test Parameters
Fault Type and Detectability
Building Tests to Match an Operational Profile
Scaling Considerations
Cost and Practical Considerations
Test Parameter Analysis, Eduardo Miranda
What Should Be Included as a Test Parameter
Combination Anomalies
Classification Tree Method
Modeling Methodology
Selecting the System under Test
Combinatorial Testing and Boundary Value Analysis
Managing System State, George Sherwood
Test Fact or Partitions with State
Test Fact or Simplifications
Sequence Unit Replay Model
Single Region State Models
Multiple Region State Models
Measuring Combinatorial Coverage
Software Test Coverage
Combinatorial Coverage
Using Combinatorial Coverage
Cost and Practical Considerations
Analysis of (t + 1)-Way Coverage
Test Suite Prioritization by Combinatorial Coverage, Renee Bryce and Sreedevi Sampath
Combinatorial Coverage for Test Suite Prioritization
Algorithm
Prioritization Criteria
Review of Empirical Studies
Tool: Combinatorial-Based Prioritization for User-Session-Based Testing
Other Approaches to Test Suite Prioritization Using Combinatorial Interactions
Cost and Practical Considerations
Combinatorial Testing and Random Test Generation
Coverage of Random Tests
Adaptive Random Testing
Tradeoffs: Covering Arrays and Random Generation
Cost and Practical Considerations
Sequence-Covering Arrays
Sequence-Covering Array Definition
Size and Construct ion of Sequence-Covering Arrays
Using Sequence-Covering Arrays
Cost and Practical Considerations
Assertion-Based Testing
Basic Assertions for Testing
Stronger Assertion-Based Testing
Cost and Practical Considerations
Model-Based Testing
Overview
Access Control System Example
SMV Model
Integrating Combinatorial Tests into the Model
Generating Tests from Counterexamples
Cost and Practical Considerations
Fault Localization
Fault Localization Process
Locating Faults: Example
Cost and Practical Considerations
Evolution from Design of Experiments
Background
Pairwise (Two-Way) Testing of Software Systems
Combinatorial t-Way Testing of Software Systems
Algorithms for Covering Array Construction, Linbin Yu and Yu Lei
Overview
Algorithm AETG
Algorithm IPOG
Cost and Practical Considerations
Appendix A: Mathematics Review
Appendix B: Empirical Data on Software Failures
Appendix C: Resources for Combinatorial Testing
Appendix D: Test Tools
References
Index
A Summary and Review appear at the end of each chapter.
Biography
D. Richard Kuhn, Raghu N. Kacker, Yu Lei
"I was prepared at the outset to dislike this book. I expected a different subject. It was written by researchers who may or may not have been able to relate to practitioners. It was about a subject that seemed virtually impossible to make practical. But in the end, it turned out to be on a quite different subject, and the authors overcame all my concerns. I thoroughly recommend it to anyone involved in the practice of software testing."
—Computing Reviews, March 2014