1st Edition

Theory of Statistical Inference

By Anthony Almudevar Copyright 2022
    470 Pages 26 B/W Illustrations
    by Chapman & Hall

    470 Pages 26 B/W Illustrations
    by Chapman & Hall

    Theory of Statistical Inference is designed as a reference on statistical inference for researchers and students at the graduate or advanced undergraduate level. It presents a unified treatment of the foundational ideas of modern statistical inference, and would be suitable for a core course in a graduate program in statistics or biostatistics. The emphasis is on the application of mathematical theory to the problem of inference, leading to an optimization theory allowing the choice of those statistical methods yielding the most efficient use of data. The book shows how a small number of key concepts, such as sufficiency, invariance, stochastic ordering, decision theory and vector space algebra play a recurring and unifying role.

    The volume can be divided into four sections. Part I provides a review of the required distribution theory. Part II introduces the problem of statistical inference. This includes the definitions of the exponential family, invariant and Bayesian models. Basic concepts of estimation, confidence intervals and hypothesis testing are introduced here. Part III constitutes the core of the volume, presenting a formal theory of statistical inference. Beginning with decision theory, this section then covers uniformly minimum variance unbiased (UMVU) estimation, minimum risk equivariant (MRE) estimation and the Neyman-Pearson test. Finally, Part IV introduces large sample theory. This section begins with stochastic limit theorems, the δ-method, the Bahadur representation theorem for sample quantiles, large sample U-estimation, the Cramér-Rao lower bound and asymptotic efficiency. A separate chapter is then devoted to estimating equation methods. The volume ends with a detailed development of large sample hypothesis testing, based on the likelihood ratio test (LRT), Rao score test and the Wald test.


    • This volume includes treatment of linear and nonlinear regression models, ANOVA models, generalized linear models (GLM) and generalized estimating equations (GEE).
    • An introduction to decision theory (including risk, admissibility, classification, Bayes and minimax decision rules) is presented. The importance of this sometimes overlooked topic to statistical methodology is emphasized.
    • The volume emphasizes throughout the important role that can be played by group theory and invariance in statistical inference.
    • Nonparametric (rank-based) methods are derived by the same principles used for parametric models and are therefore presented as solutions to well-defined mathematical problems, rather than as robust heuristic alternatives to parametric methods.
    • Each chapter ends with a set of theoretical and applied exercises integrated with the main text. Problems involving R programming are included.
    • Appendices summarize the necessary background in analysis, matrix algebra and group theory.


    1 Distribution Theory

    1.1 Introduction

    1.2 Probability Measures

    1.3 Some Important Theorems of Probability

    1.4 Commonly Used Distributions

    1.5 Stochastic Order Relations

    1.6 Quantiles

    1.7 Inversion of the CDF

    1.8 Transformations of Random Variables

    1.9 Moment Generating Functions

    1.10 Moments and Cumulants

    1.11 Problems

    2 Multivariate Distributions

    2.1 Introduction

    2.2 Parametric Classes of Multivariate Distributions

    2.3 Multivariate Transformations

    2.4 Order Statistics

    2.5 Quadratic Forms, Idempotent Matrices and Cochran’s Theorem

    2.6 MGF and CGF of Independent Sums

    2.7 Multivariate Extensions of the MGF

    2.8 Problems

    3 Statistical Models

    3.1 Introduction

    3.2 Parametric Families for Statistical Inference

    3.3 Location-Scale Parameter Models

    3.4 Regular Families

    3.5 Fisher Information

    3.6 Exponential Families

    3.7 Sufficiency

    3.8 Complete and Ancillary Statistics

    3.9 Conditional Models and Contingency Tables

    3.10 Bayesian Models

    3.11 Indifference, Invariance and Bayesian Prior Distributions

    3.12 Nuisance Parameters

    3.13 Principles of Inference

    3.14 Problems

    4 Methods of Estimation

    4.1 Introduction

    4.2 Unbiased Estimators

    4.3 Method of Moments Estimators

    4.4 Sample Quantiles and Percentiles

    4.5 Maximum Likelihood Estimation

    4.6 Confidence Sets

    4.7 Equivariant Versus Shrinkage Estimation

    4.8 Bayesian Estimation

    4.9 Problems

    5 Hypothesis Testing

    5.1 Introduction

    5.2 Basic Definitions

    5.3 Principles of Hypothesis Tests

    5.4 The Observed Level of Significance (P-Values)

    5.5 One and Two Sided Tests

    5.6 Hypothesis Tests and Pivots

    5.7 Likelihood Ratio Tests

    5.8 Similar Tests

    5.9 Problems

    6 Linear Models

    6.1 Introduction

    6.2 Linear Models - Definition

    6.3 Best Linear Unbiased Estimators (BLUE)

    6.4 Least-squares Estimators, BLUEs and Projection Matrices

    6.5 Ordinary and Generalized Least-Squares Estimators

    6.6 ANOVA Decomposition and the F Test for Linear Models

    6.7 The F Test for One-Way ANOVA

    6.8 Simultaneous Confidence Intervals

    6.9 Multiple Linear Regression

    6.10 Problems

    7 Decision Theory

    7.1 Introduction

    7.2 Ranking Estimators by MSE

    7.3 Prediction

    7.4 The Structure of Decision Theoretic Inference

    7.5 Loss and Risk

    7.6 Uniformly Minimum Risk Estimators (The Location-Scale Model

    7.7 Some Principles of Admissibility

    7.8 Admissibility for Exponential Families (Karlin’s Theorem)

    7.9 Bayes Decision Rules

    7.10 Admissibility and Optimality

    7.11 Problems

    8 Uniformly Minimum Variance Unbiased (UMVU) Estimation

    8.1 Introduction

    8.2 Definition of UMVUE’s

    8.3 UMVUE’s and Sufficiency

    8.4 Methods of Deriving UMVUEs

    8.5 Nonparametric Estimation and U-statistics

    8.6 Rank Based Measures of Correlation

    8.7 Problems

    9 Group Structure and Invariant Inference

    9.1 Introduction

    9.2 MRE Estimators for Location Parameters

    9.3 MRE Estimators for Scale Parameters

    9.4 Invariant Density Families

    9.5 Some Applications of Invariance

    9.6 Invariant Hypothesis Tests

    9.7 Problems

    10 The Neyman-Pearson Lemma

    10.1 Introduction

    10.2 Hypothesis Test as Decision Rules

    10.3 Neyman-Pearson (NP) Tests

    10.4 Monotone Likelihood Ratios (MLR)

    10.5 The Generalized Neyman-Pearson Lemma

    10.6 Invariant Hypothesis Tests

    10.7 Permutation Invariant Tests

    10.8 Problems

    11 Limit Theorems

    11.1 Introduction

    11.2 Limits of Sequences of Random Variables

    11.3 Limits of Expected Values

    11.4 Uniform Integrability

    11.5 The Law of Large Numbers

    11.6 Weak Convergence

    11.7 Multivariate Extensions of Limit Theorems

    11.8 The Continuous Mapping Theorem

    11.9 MGFs, CGFs and Weak Convergence

    11.10 The Central Limit Theorem for Triangular Arrays

    11.11 Weak Convergence of Random Vectors

    11.12 Problems

    12 Large Sample Estimation - Basic Principles

    12.1 Introduction

    12.2 The _-Method

    12.3 Variance Stabilizing Transformations

    12.4 The _-Method and Higher Order Approximations

    12.5 The Multivariate _-Method

    12.6 Approximating the Distributions of Sample Quantiles: The Bahadur Representation Theorem

    12.7 A Central Limit Theorem for U-statistics

    12.8 The Information Inequality

    12.9 Asymptotic Efficiency

    12.10 Problems

    13 Asymptotic Theory for Estimating Equations

    13.1 Introduction

    13.2 Consistency and Asymptotic Normality of M-estimators

    13.3 Asymptotic Theory of MLEs

    13.4 A General Form for Regression Models

    13.5 Nonlinear Regression

    13.6 Generalized Linear Models (GLMs)

    13.7 Generalized Estimating Equations (GEE)

    13.8 Consistency of M-estimators

    13.9 Asymptotic Distribution of ˆ_n

    13.10 Regularity Conditions for Estimating Equations

    13.11 Problems

    14 Large Sample Hypothesis Testing

    14.1 Introduction

    14.2 Model Assumptions

    14.3 Large Sample Tests for Simple Hypotheses

    14.4 Nuisance Parameters and Composite Null Hypotheses

    14.5 A Comparison of the LR, Wald and Score Tests

    14.6 Pearson’s _2 Test for Independence in Contingency Tables

    14.7 Estimating Power for Approximate _2 Tests

    14.8 Problems

    A Parametric Classes of Densities

    B Topics in Linear Algebra

    B.1 Numbers

    B.2 Equivalence Relations

    B.3 Vector Spaces

    B.4 Matrices

    B.5 Dimension of a Subset of Rd

    C Topics in Real Analysis and Measure Theory

    C.1 Metric spaces

    C.2 Measure Theory

    C.3 Integration

    C.4 Exchange of Integration and Differentiation

    C.5 The Gamma and Beta Functions

    C.6 Stirling’s Approximation of the Factorial

    C.7 The Gradient Vector and the Hessian Matrix

    C.8 Normed Vector Spaces

    C.9 Taylor’s Theorem

    D Group Theory

    D.1 Definition of a Group

    D.2 Subgroups

    D.3 Group Homomorphisms

    D.4 Transformation Groups

    D.5 Orbits and Maximal Invariants




    Anthony Almudevar is an Associate Professor of Biostatistics and Computational Biology at the University of Rochester. His research interests include statistical methodology, graphical models, bioinformatics, optimization and control theory. Other published volumes include Almudevar A (2014) Approximate Iterative Algorithms, CRC Press, and Statistical Modeling for Biological Systems: In Memory of Andrei Yakovlev, Anthony Almudevar, David Oakes and Jack Hall, editors (2020), Springer.

    "The large variety of topics covered in the current version allows students at all level, be it bachelor, master or Ph.D. to find topics suitable for their own study. On one hand, the book contains fairly advanced topics to help researchers in their own research areas, and on the other hand, it provides enough materials for lecturers to prepare lecture notes and design an entire course on theoretical statistics. Overall, I am extremely excited about Theory of Statistical Inference and eagerly waiting for numerous students, researchers, lecturers and even working professionals to enjoy the immense benefit coming out of its publication."

    Somabha MukherjeeNational University of Singapore, Singapore, Journal of the American Statistical Association, February 2024.