Replication and Evidence Factors in Observational Studies

I Background: Aspects of Causal Inference

1. Causal Inference in Randomized Experiments

A Randomized Experiment

Structure and Notation

Covariates and Outcomes

Causal Effects with Two Treatment Groups

Inference with Random Assignment

Randomization Tests for Continuous Outcomes

Confidence Sets for Causal Effects

The General Situation

Summary: Randomization Simplifies Causal Inference

Using R

How Are Observational Studies Different From Experiments?

Sensitivity Analysis

*Another Example of Sensitivity Analysis

Design Sensitivity

Summary: Biased Treatment Assignment

Using R

Exercises

Biases Can Replicate

Some Perspectives

Replications that Disrupt Some Potential Biases

Instruments and Replication

Summary: Replication is Not Repetition

Smoking and Periodontal Disease

Antineoplastic Drugs and DNA Damage

Lead Absorption in Children

Minimum Wages and Employment

Benzene and Chromosome Aberrations

Summary: Mutually Supporting, Unrelated Comparisons

Using R

Exercises

Structure of the Simple Analyses

Antineoplastic Drugs and DNA Damage

Smoking and Periodontal Disease

Factors that Do Not Concur

Summary: Strengthen Evidence of Cause and Effect

Using R

Exercises

Closed Testing with Three Factors

Confidence Intervals for Magnitudes of Effect

Evidence Factors Plus An Incompatible Comparison

Summary: Planned Analyses Can Accomplish More

Using R

Exercises

Dependent P-values Larger than Uniform

Jointly Larger Than Uniform

Creating Jointly Valid, Possibly Dependent P-values

Combining Jointly Valid, Possibly Dependent P-values

Summary: Dependent P-values Jointly Larger than Uniform

Exercises

Formalizing Intuition About Unrelated Pieces of Evidence

Individuals, Strata and Treatment Positions

Permutation Matrices

Pick Matrices

Direct Sums of Permutation Matrices

Subpick Matrices

Treatments with Doses

Permuting Strata of the Same Size

Permuting Several Permutation Matrices

Doing Several Things at Once

Summary: A Treatment Assignment is a Permutation

Complement: Split Matrices

Exercises

Sets of Permutation Matrices

Products of Sets

Unique Representation As a Product of Two Factors

*Closure

Summary: Factoring Sets of Treatment Assignments

Exercises

One Distribution

A Set of Distributions

*Some Technical Remarks and Definitions

Sensitivity Analysis

Summary: Probability on a Set of Treatment Assignments

Complement: Sensitivity Analysis with Doses

Exercises

Marginal and Conditional Distributions

Joint Distribution of Two Sensitivity Analyses

Sets of Marginal and Conditional Distributions

Ignoring a Factor

Conditioning on a Factor

Combining Two Sensitivity Analyses

Summary: Combining Two Sensitivity Analyses

Complement: More Than Two Factors

Exercises

Why Groups?

Groups

Groups in Evidence Factors: Some Examples

Group Products

Summary: Groups Provide the Needed Factors

Aspects of Design

Nearly Optimal Complete Blocks

Optimal Incomplete Block Designs

Variation in Treatment Within and Between Institutions

Comparing Study Designs: Which Design is Best?

Summary: Build Evidence Factors into the Design

Using R

Some Common Design Elements

*Symmetric Sets of Biases

Biography

Author

Paul R. Rosenbaum is the Robert G. Putzel Professor of Statistics at the Wharton School of the University of Pennsylvania. For contributions to causal inference, he received the R. A. Fisher Award in 2019 and the George W. Snedecor Award in 2003, both from the Committee of Presidents of Statistical Societies (COPSS). He delivered an IMS Medallion Lecture on the topic of this book in 2020. Dr. Rosenbaum is the author of several other books including Observational Studies (Springer 1995, 2002), Design of Observational Studies (Springer 2010, 2020), and Observation and Experiment: An Introduction to Causal Inference (Harvard University Press 2017).

"In summary, this book provides clear descriptions to explain analysis of evidence factors in observational studies. In addition, the author also provides the rigorous theory to support the validity of methodologies in this book. For the implementation of methodologies, the author develops a package ‘evident’, which makes readers reproduce and implement the methods easily. In general, this book is an amazing reference for those who are interested in causal inference or observational studies."
-Li-Pang Chen in Journal of the Royal Statistical Society Series A, March 2022

"(...) the book sets high standards for the analysis of those observational studies that fit within its purview. A wide range of examples and associated data are discussed, all with important public health implications. ... As the author suggests, practically minded readers who skip the detailed mathematics can nevertheless gain important insights by following the motivation, applications and examples. The issues raised and points made are important whenever associations found in observational data are used as a basis for claims of causation..."
-John H. Maindonald in International Statistical Review, March 2022

"Overall, I consider the book to be a rich resource for introducing this relatively new yet highly impactful area of research. The book is organized into four Sections. … Section II sets the tone for the rest of the book by collecting carefully chosen examples.  … Chapter 4 provides five real studies to elicit aspects of evidence factors from a variety of representative examples. … The chapter is supplemented through R codes for the examples covered and hands-on exercises for the interested reader. Concepts are well elucidated through concrete running examples. … Section III of the book lends a mathematically rigorous lens to the intuitions gathered from the data analyses and numerical examples in Section II. … This logic is beautifully explained.”
-Rajarshi Mukherjee in Biometrics, June 2021

"This book is the first to discuss evidence factors and is a valuable contribution. Statisticians working on observational studies would find the book useful. Empirical researchers who conduct observational studies would find Chapters 1-6 useful. I would say the book serves more as a reference than a textbook although the book is as lucidly written as any good textbook…I would strongly recommend publication. The book will be of wide interest to causal inference practitioners."
-Ted Westling, University of Massachusetts, Amherst

"This book will be of wide interest to causal inference practitioners."
-Joel Greenhouse, Carnegie Mellon University

"This book not only brings much of the discussion around the topic of replicability in causal inference in one place, it does it in a way accessible to most. I would absolutely recommend this book for publication. (i) A big strength is that the book is conscious of the balance it needs to keep among motivating the concept, providing technical exposure and demonstrating the application of the method. (ii) This book is self-contained. (iii) The R codes in the footnotes and more references to specific R packages to implement the methods is a huge plus for the book. (iv) Of course, more on this topic exists that is not covered in the book. This book gives necessary references to papers for curious readers. There is a clear distinction of the focus of chapters through 6 and the latter chapters… While the earlier chapters consider Evidence Factors in Practice, the later chapters are about the Theory of Evidence Factors. This distinction is important to illustrate ideas. It is also nice that the book rounds up the discussion at the end in Chapter 13 with many practical tools…This book not only brings much of the discussion around the topic of replicability in causal inference in one place, it does it in a way accessible to most. I would absolutely recommend this book for publication."
-Bikram Karmakar, University of Florida

"Paul Rosenbaum is a gifted expositor of complex statistical concepts and methods. His books on analyzing data from observational studies are not only a pleasure to read and to learn from but are scholarly and erudite in ways that are not typical of writings in statistics…The proposed manuscript is in the same style as Rosenbaum’s earlier books and therefore promises to be popular as a reference for research workers or as a textbook for advanced undergraduate or graduate students, i.e., readers with sufficient statistical maturity. There is a lot of conceptual and technical machinery required to understand and use statistical methods for causal inference. In this book Rosenbaum is taking a step back. His goal is to explicate the informal steps that lead to a consensus about a causal relationship in practice and to provide formal methods for interrogating and weighing evidence from studies to help the scientific community reach consensus about causal relationships. This book will be a valuable addition to the causal inference literature."
-Dylan Small, University of Pennsylvania