1st Edition
Quasi-Experimentation A Guide to Design and Analysis
By Charles S. Reichardt
Copyright 2019
361 Pages
by
Guilford Press
361 Pages
by
Guilford Press
Featuring engaging examples from diverse disciplines, this book explains how to use modern approaches to quasi-experimentation to derive credible estimates of treatment effects under the demanding constraints of field settings. Foremost expert Charles S. Reichardt provides an in-depth examination of the design and statistical analysis of pretest–posttest, nonequivalent groups, regression discontinuity, and interrupted time-series designs. He details their relative strengths and weaknesses and offers practical advice about their use. Comparing quasi-experiments to randomized experiments, Reichardt discusses when and why the former might be a better choice than the latter in the face of the contingencies that are likely to arise in practice. Modern methods for elaborating a research design to remove bias from estimates of treatment effects are described, as are tactics for dealing with missing data and noncompliance with treatment assignment. Throughout, mathematical equations are translated into words to enhance accessibility. Adding to its discussion of prototypical quasi-experiments, the book also provides a complete typology of quasi-experimental design options to help the reader craft the best research design to fit the circumstances of a given study.
1. Introduction
Overview
1.1 Introduction
1.2 The Definition of Quasi-Experiment
1.3 Why Study Quasi-Experiments
1.4 Overview of the Volume
1.5 Conclusions
1.6 Suggested Reading
2. Cause and Effect
Overview
2.1 Introduction
2.2 Practical Comparisons and Confounds
2.3 The Counterfactual Definition
2.4 The Stable-Unit-Treatment-Value Assumption (SUTVA)
2.5 The Causal Question Being Addressed
2.6 Conventions
2.7 Conclusions
2.8 Suggested Reading
3. Threats to Validity
Overview
3.1 Introduction
3.2 The Size of an Effect
3.3 Construct Validity
3.4 Internal Validity
3.5 Statistical Conclusion Validity
3.6 External Validity
3.7 Trade-offs among Types of Validity
3.8 A Focus on Internal and Statistical Conclusion Validity
3.9 Conclusions
3.10 Suggested Reading
4. Randomized Experiments
Overview
4.1 Introduction
4.2 Between-Groups Randomized Experiments
4.3 Examples of Randomized Experiments Conducted in the Field
4.4 Selection Differences
4.5 Analysis of Data from the Posttest-Only Randomized Experiment
4.6 Analysis of Data from the Pretest–Posttest Randomized Experiment
4.7 Noncompliance with Treatment Assignment
4.8 Missing Data and Attrition
4.9 Cluster-Randomized Experiments
4.10 Other Threats to Validity in Randomized Experiments
4.11 Strengths and Weaknesses
4.12 Conclusions
4.13 Suggested Reading
5. One-Group Posttest-Only Designs
Overview
5.1 Introduction
5.2 Examples of One-Group Posttest-Only Designs
5.3 Strengths and Weaknesses
5.4 Conclusions
5.5 Suggested Reading
6. Pretest–Posttest Designs
Overview
6.1 Introduction
6.2 Examples of Pretest–Posttest Designs
6.3 Threats to Internal Validity
6.4 Design Variations
6.5 Strengths and Weaknesses
6.6 Conclusions
6.7 Suggested Reading
7. Nonequivalent Group Designs
Overview
7.1 Introduction
7.2 Two Basic Nonequivalent Group Designs
7.3 Change-Score Analysis
7.4 Analysis of Covariance
7.5 Matching and Blocking
7.6 Propensity Scores
7.7 Instrumental Variables
7.8 Selection Models
7.9 Sensitivity Analyses and Tests of Ignorability
7.10 Other Threats to Internal Validity besides Selection Differences
7.11 Alternative Nonequivalent Group Designs
7.12 Empirical Evaluations and Best Practices
7.13 Strengths and Weaknesses
7.14 Conclusions
7.15 Suggested Reading
8. Regression Discontinuity Designs
Overview
8.1 Introduction
8.2 The Quantitative Assignment Variable
8.3 Statistical Analysis
8.4 Fuzzy Regression Discontinuity
8.5 Threats to Internal Validity
8.6 Supplemented Designs
8.7 Cluster Regression Discontinuity Designs
8.8 Strengths and Weaknesses
8.9 Conclusions
8.10 Suggested Reading
9. Interrupted Time-Series Designs
Overview
9.1 Introduction
9.2 The Temporal Pattern of the Treatment Effect
9.3 Two Versions of the Design
9.4 The Statistical Analysis of Data When N = 1
9.5 The Statistical Analysis of Data When N Is Large
9.6 Threats to Internal Validity
9.7 Design Supplements I: Multiple Interventions
9.8 Design Supplements II: Basic Comparative ITS Designs
9.9 Design Supplements III: Comparative ITS Designs with Multiple Treatments
9.10 Single-Case Designs
9.11 Strengths and Weaknesses
9.12 Conclusions
9.13 Suggested Reading
10. A Typology of Comparisons
Overview
10.1 Introduction
10.2 The Principle of Parallelism
10.3 Comparisons across Participants
10.4 Comparisons across Times
10.5 Comparisons across Settings
10.6 Comparisons across Outcome Measures
10.7 Within- and Between-Subject Designs
10.8 A Typology of Comparisons
10.9 Random Assignment to Treatment Conditions
10.10 Assignment to Treatment Conditions Based on an Explicit Quantitative Ordering
10.11 Nonequivalent Assignment to Treatment Conditions
10.12 Credibility and Ease of Implementation
10.13 The Most Commonly Used Comparisons
10.14 Conclusions
10.15 Suggested Reading
11. Methods of Design Elaboration
Overview
11.1 Introduction
11.2 Three Methods of Design Elaboration
11.3 The Four Size-of-Effect Factors as Sources for the Two Estimates in Design Elaboration
11.4 Conclusions
11.5 Suggested Reading
12. Unfocused Design Elaboration and Pattern Matching
Overview
12.1 Introduction
12.2 Four Examples of Unfocused Design Elaboration
12.3 Pattern Matching
12.4 Conclusions
12.5 Suggested Reading
13. Principles of Design and Analysis for Estimating Effects
Overview
13.1 Introduction
13.2 Design Trumps Statistics
13.3 Customized Designs
13.4 Threats to Validity
13.5 The Principle of Parallelism
13.6 The Typology of Simple Comparisons
13.7 Pattern Matching and Design Elaborations
13.8 Size of Effects
13.9 Bracketing Estimates of Effects
13.10 Critical Multiplism
13.11 Mediation
13.12 Moderation
13.13 Implementation
13.14 Qualitative Research Methods
13.15 Honest and Open Reporting of Results
13.16 Conclusions
13.17 Suggested Reading
Appendix: The Problems of Overdetermination and Preemption
A.1 The Problem of Overdetermination
A.2 The Problem of Preemption
References
Glossary
Author Index
Subject Index
About the Author
Overview
1.1 Introduction
1.2 The Definition of Quasi-Experiment
1.3 Why Study Quasi-Experiments
1.4 Overview of the Volume
1.5 Conclusions
1.6 Suggested Reading
2. Cause and Effect
Overview
2.1 Introduction
2.2 Practical Comparisons and Confounds
2.3 The Counterfactual Definition
2.4 The Stable-Unit-Treatment-Value Assumption (SUTVA)
2.5 The Causal Question Being Addressed
2.6 Conventions
2.7 Conclusions
2.8 Suggested Reading
3. Threats to Validity
Overview
3.1 Introduction
3.2 The Size of an Effect
3.3 Construct Validity
3.4 Internal Validity
3.5 Statistical Conclusion Validity
3.6 External Validity
3.7 Trade-offs among Types of Validity
3.8 A Focus on Internal and Statistical Conclusion Validity
3.9 Conclusions
3.10 Suggested Reading
4. Randomized Experiments
Overview
4.1 Introduction
4.2 Between-Groups Randomized Experiments
4.3 Examples of Randomized Experiments Conducted in the Field
4.4 Selection Differences
4.5 Analysis of Data from the Posttest-Only Randomized Experiment
4.6 Analysis of Data from the Pretest–Posttest Randomized Experiment
4.7 Noncompliance with Treatment Assignment
4.8 Missing Data and Attrition
4.9 Cluster-Randomized Experiments
4.10 Other Threats to Validity in Randomized Experiments
4.11 Strengths and Weaknesses
4.12 Conclusions
4.13 Suggested Reading
5. One-Group Posttest-Only Designs
Overview
5.1 Introduction
5.2 Examples of One-Group Posttest-Only Designs
5.3 Strengths and Weaknesses
5.4 Conclusions
5.5 Suggested Reading
6. Pretest–Posttest Designs
Overview
6.1 Introduction
6.2 Examples of Pretest–Posttest Designs
6.3 Threats to Internal Validity
6.4 Design Variations
6.5 Strengths and Weaknesses
6.6 Conclusions
6.7 Suggested Reading
7. Nonequivalent Group Designs
Overview
7.1 Introduction
7.2 Two Basic Nonequivalent Group Designs
7.3 Change-Score Analysis
7.4 Analysis of Covariance
7.5 Matching and Blocking
7.6 Propensity Scores
7.7 Instrumental Variables
7.8 Selection Models
7.9 Sensitivity Analyses and Tests of Ignorability
7.10 Other Threats to Internal Validity besides Selection Differences
7.11 Alternative Nonequivalent Group Designs
7.12 Empirical Evaluations and Best Practices
7.13 Strengths and Weaknesses
7.14 Conclusions
7.15 Suggested Reading
8. Regression Discontinuity Designs
Overview
8.1 Introduction
8.2 The Quantitative Assignment Variable
8.3 Statistical Analysis
8.4 Fuzzy Regression Discontinuity
8.5 Threats to Internal Validity
8.6 Supplemented Designs
8.7 Cluster Regression Discontinuity Designs
8.8 Strengths and Weaknesses
8.9 Conclusions
8.10 Suggested Reading
9. Interrupted Time-Series Designs
Overview
9.1 Introduction
9.2 The Temporal Pattern of the Treatment Effect
9.3 Two Versions of the Design
9.4 The Statistical Analysis of Data When N = 1
9.5 The Statistical Analysis of Data When N Is Large
9.6 Threats to Internal Validity
9.7 Design Supplements I: Multiple Interventions
9.8 Design Supplements II: Basic Comparative ITS Designs
9.9 Design Supplements III: Comparative ITS Designs with Multiple Treatments
9.10 Single-Case Designs
9.11 Strengths and Weaknesses
9.12 Conclusions
9.13 Suggested Reading
10. A Typology of Comparisons
Overview
10.1 Introduction
10.2 The Principle of Parallelism
10.3 Comparisons across Participants
10.4 Comparisons across Times
10.5 Comparisons across Settings
10.6 Comparisons across Outcome Measures
10.7 Within- and Between-Subject Designs
10.8 A Typology of Comparisons
10.9 Random Assignment to Treatment Conditions
10.10 Assignment to Treatment Conditions Based on an Explicit Quantitative Ordering
10.11 Nonequivalent Assignment to Treatment Conditions
10.12 Credibility and Ease of Implementation
10.13 The Most Commonly Used Comparisons
10.14 Conclusions
10.15 Suggested Reading
11. Methods of Design Elaboration
Overview
11.1 Introduction
11.2 Three Methods of Design Elaboration
11.3 The Four Size-of-Effect Factors as Sources for the Two Estimates in Design Elaboration
11.4 Conclusions
11.5 Suggested Reading
12. Unfocused Design Elaboration and Pattern Matching
Overview
12.1 Introduction
12.2 Four Examples of Unfocused Design Elaboration
12.3 Pattern Matching
12.4 Conclusions
12.5 Suggested Reading
13. Principles of Design and Analysis for Estimating Effects
Overview
13.1 Introduction
13.2 Design Trumps Statistics
13.3 Customized Designs
13.4 Threats to Validity
13.5 The Principle of Parallelism
13.6 The Typology of Simple Comparisons
13.7 Pattern Matching and Design Elaborations
13.8 Size of Effects
13.9 Bracketing Estimates of Effects
13.10 Critical Multiplism
13.11 Mediation
13.12 Moderation
13.13 Implementation
13.14 Qualitative Research Methods
13.15 Honest and Open Reporting of Results
13.16 Conclusions
13.17 Suggested Reading
Appendix: The Problems of Overdetermination and Preemption
A.1 The Problem of Overdetermination
A.2 The Problem of Preemption
References
Glossary
Author Index
Subject Index
About the Author
Biography
Charles S. Reichardt, PhD, is Professor of Psychology at the University of Denver. He is an elected fellow of the American Psychological Society, an elected member of the Society of Multivariate Experimental Psychology, and a recipient of the Robert Perloff President’s Prize from the Evaluation Research Society and the Jeffrey S. Tanaka Award from the Society of Multivariate Experimental Psychology. Dr. Reichardt’s research focuses on quasi-experimentation.
"This book represents an important contribution to the literature on research designs that may be implemented when randomized experiments are not feasible or are limited. Covering the full range of design alternatives, this is the first text that fuses important recent advances from statistics and econometrics into Campbell’s pioneering approach. A notable feature is Reichardt’s careful attention to issues that arise in each research design; he offers innovative design and analysis strategies that can minimize these issues and permit the strongest possible conclusions from research. Clearly written, this text is an outstanding choice for courses focusing on key issues of research design, and is suitable for graduate students with only a basic background in statistics. Established researchers will find it to be a valuable reference that offers new insights for strengthening research designs so that they yield the most credible possible evidence."--Stephen G. West, PhD, Department of Psychology, Arizona State University-
"This book not only compiles a comprehensive list of methods on quasi-causal design, but also problematizes causes of biases even in perfectly executed experimental and quasi-experimental designs. The author’s take on the ways in which quasi-experiments could potentially render better results than randomized experiments is refreshing and important. Professors will want to discuss this book in their classes. I highly recommend it for students and even more experienced researchers--the author highlights the fundamentals of each approach along with its strengths and limitations.”--Manuel González Canché, PhD, Higher Education, Quantitative Methods, and Education Policy Divisions, Graduate School of Education, University of Pennsylvania
"A 'must read.' After a thorough presentation of the strengths of randomized experiments, Reichardt provides a remarkably up-to-date review and synthesis of current thinking on the best, most useful alternatives. Notably, he uses simple and direct language to explain key concepts of the 'counterfactual outcomes' approach for estimating causal effects. While written for a graduate and professional audience, the book does not require advanced statistical knowledge. It is ideal as a supplemental text for a graduate course on experimental design and the analysis of variance, or as the primary source for a seminar on quasi-experimental design and analysis. Practicing scientists will want to own this book to understand how best to confront analytic issues in their empirical research and interpret their results."--Keith F. Widaman, PhD, Distinguished Professor, Graduate School of Education, University of California, Riverside
"Reichardt provides an expansive treatment of quasi-experimental designs, in the tradition of Shadish, Cook, and Campbell. This book includes up-to-date discussions of propensity scores, modern missing data procedures, and instrumental variables. Students will appreciate the numerous examples that help clarify the concepts. I would recommend this book for any graduate research methods class--I will certainly use it myself.”--Felix J. Thoemmes, PhD, Department of Human Development and Department of Psychology, Cornell University
"Ever wonder how to best design a quasi-experimental study? This book will help you figure out which research questions best lend themselves to this type of experimental design. Have you collected data from a quasi-experiment and now want to make sure that you correctly analyze and interpret the results? This book addresses the assumptions that must be met, potential pitfalls, and statistical considerations. As an educational psychologist who teaches students across disciplines, I recommend this book as an up-to-date reference on quasi-experimental designs. Randomized controlled trials are not always feasible, for many reasons, so the way this text is framed is actually more useful for fields like education and the social sciences."--Meagan C. Arrastia-Chisholm, PhD, Department of Psychology, Counseling, and Family Therapy, Valdosta State University
