1st Edition

Introduction to Combinatorial Testing

By D. Richard Kuhn, Raghu N. Kacker, Yu Lei Copyright 2013
    341 Pages 92 B/W Illustrations
    by Chapman & Hall

    Combinatorial testing of software analyzes interactions among variables using a very small number of tests. This advanced approach has demonstrated success in providing strong, low-cost testing in real-world situations. Introduction to Combinatorial Testing presents a complete self-contained tutorial on advanced combinatorial testing methods for real-world software.

    The book introduces key concepts and procedures of combinatorial testing, explains how to use software tools for generating combinatorial tests, and shows how this approach can be integrated with existing practice. Detailed explanations and examples clarify how and why to use various techniques. Sections on cost and practical considerations describe tradeoffs and limitations that may impact resources or funding. While the authors introduce some of the theory and mathematics of combinatorial methods, readers can use the methods without in-depth knowledge of the underlying mathematics.

    Accessible to undergraduate students and researchers in computer science and engineering, this book illustrates the practical application of combinatorial methods in software testing. Giving pointers to freely available tools and offering resources on a supplementary website, the book encourages readers to apply these methods in their own testing projects.

    Combinatorial Methods in Testing
    Software Failures and the Interaction Rule
    Two Forms of Combinatorial Testing
    Covering Arrays
    The Test Oracle Problem
    Quick Start: How to Use the Basics of Combinatorial Methods Right Away

    Combinatorial Testing Applied
    Document Object Model, Carmelo Montanez-Rivera, D. Richard Kuhn, Mary Brady, Rick Rivello, Jenise Reyes Rodriguez, and Michael Powers
    Rich Web Applications, Chad M. Maughan

    Configuration Testing
    Runtime Environment Configurations
    Highly Configurable Systems and Software Product Lines
    Invalid Combinations and Constraints
    Cost and Practical Considerations

    Input Testing
    Partitioning the Input Space
    Input Variables versus Test Parameters
    Fault Type and Detectability
    Building Tests to Match an Operational Profile
    Scaling Considerations
    Cost and Practical Considerations

    Test Parameter Analysis, Eduardo Miranda
    What Should Be Included as a Test Parameter
    Combination Anomalies
    Classification Tree Method
    Modeling Methodology
    Selecting the System under Test
    Combinatorial Testing and Boundary Value Analysis

    Managing System State, George Sherwood
    Test Fact or Partitions with State
    Test Fact or Simplifications
    Sequence Unit Replay Model
    Single Region State Models
    Multiple Region State Models

    Measuring Combinatorial Coverage
    Software Test Coverage
    Combinatorial Coverage
    Using Combinatorial Coverage
    Cost and Practical Considerations
    Analysis of (t + 1)-Way Coverage

    Test Suite Prioritization by Combinatorial Coverage, Renee Bryce and Sreedevi Sampath
    Combinatorial Coverage for Test Suite Prioritization
    Prioritization Criteria
    Review of Empirical Studies
    Tool: Combinatorial-Based Prioritization for User-Session-Based Testing
    Other Approaches to Test Suite Prioritization Using Combinatorial Interactions
    Cost and Practical Considerations

    Combinatorial Testing and Random Test Generation
    Coverage of Random Tests
    Adaptive Random Testing
    Tradeoffs: Covering Arrays and Random Generation
    Cost and Practical Considerations

    Sequence-Covering Arrays
    Sequence-Covering Array Definition
    Size and Construct ion of Sequence-Covering Arrays
    Using Sequence-Covering Arrays
    Cost and Practical Considerations

    Assertion-Based Testing
    Basic Assertions for Testing
    Stronger Assertion-Based Testing
    Cost and Practical Considerations

    Model-Based Testing
    Access Control System Example
    SMV Model
    Integrating Combinatorial Tests into the Model
    Generating Tests from Counterexamples
    Cost and Practical Considerations

    Fault Localization
    Fault Localization Process
    Locating Faults: Example
    Cost and Practical Considerations

    Evolution from Design of Experiments
    Pairwise (Two-Way) Testing of Software Systems
    Combinatorial t-Way Testing of Software Systems

    Algorithms for Covering Array Construction, Linbin Yu and Yu Lei
    Algorithm AETG
    Algorithm IPOG
    Cost and Practical Considerations

    Appendix A: Mathematics Review
    Appendix B: Empirical Data on Software Failures
    Appendix C: Resources for Combinatorial Testing
    Appendix D: Test Tools



    A Summary and Review appear at the end of each chapter.


    D. Richard Kuhn, Raghu N. Kacker, Yu Lei

    "I was prepared at the outset to dislike this book. I expected a different subject. It was written by researchers who may or may not have been able to relate to practitioners. It was about a subject that seemed virtually impossible to make practical. But in the end, it turned out to be on a quite different subject, and the authors overcame all my concerns. I thoroughly recommend it to anyone involved in the practice of software testing."
    —Computing Reviews, March 2014