1st Edition
Philosophies, Puzzles and Paradoxes A Statistician’s Search for Truth
Unlike mathematics, statistics deals with real-world data and involves a higher degree of subjectivity due to the role of interpretation. Interpretation is shaped by context as well as the knowledge, preferences, assumptions and preconceptions of the interpreter, leading to a variety of interpretations of concepts as well as results. Philosophies, Puzzles and Paradoxes: A Statistician’s Search for Truth thoroughly examines the distinct philosophical approaches to statistics – Bayesian, frequentist and likelihood – arising from different interpretations of probability and uncertainty. These differences are highlighted through numerous puzzles and paradoxes and illuminated by extensive discussions of the background philosophy of science.
Features:
- Exploration of the philosophy of knowledge and truth and how they relate to deductive and inductive reasoning, and ultimately scientific and statistical thinking
- Discussion of the philosophical theories of probability that are wider than the standard Bayesian and frequentist views
- Exposition and examination of Savage’s axioms as the basis of subjective probability and Bayesian statistics
- Explanation of likelihood and likelihood-based inference, including the controversy surrounding the likelihood principle
- Discussion of fiducial probability and its evolution to confidence procedure
- Introduction of extended and hierarchical likelihood for random parameters, with the recognition of confidence as extended likelihood, leading to epistemic confidence as an objective measure of uncertainty for single events
- Detailed analyses and new variations of classic paradoxes, such as the Monty Hall puzzle, the paradox of the ravens, the exchange paradox, and more
- Substantive yet non-technical, catering to readers with only introductory exposure to the theory of probability and statistics
This book primarily targets statisticians in general, including both undergraduate and graduate students, as well as researchers interested in the philosophical basis of probability and statistics. It is also suitable for philosophers of science and general readers intrigued by puzzles and paradoxes.
1 Philosophical Theories of Knowledge and Truth
2 Deduction and Induction
3 Hilbert’s Broken Dream: Limitations of Deductive Reasoning
4 ‘Real’ Scientific Process
5 The Rise of Probability
6 Philosophical Theories of Probability
7 Rereading Savage
8 Inverse Probability Method
9 What Prior?
10 Likelihood
11 P-value and Confidence
12 Extended Likelihood
13 Epistemic Confidence
14 Paradoxes of Savage’s Axioms
15 Fallacious Fallacies
16 Monty Hall Puzzle and the Three Prisoners Paradox
17 Lottery Paradox and the Cold Suspect Puzzle
18 Paradox of the Ravens
19 Exchange Paradox
Biography
Yudi Pawitan graduated with a PhD in statistics in 1987 from the University of California at Davis and has been a professor of biostatistics since 2001 at the Department of Medical Epidemiology and Biostatistics, Karolinska Institutet, Stockholm, Sweden. He has worked in many areas of statistical applications, including time series analyses and medical imaging, and for the last 20 years in the modelling and analysis of high-throughput genetic and molecular data with applications in cancer. He has published more than 200 peer-reviewed research papers, split about equally between methodology and applied publications. He is the author of the monograph In All Likelihood (2001) and co-author of Generalized Linear Models with Random Effects (2017) together with Youngjo Lee and John Nelder, both covering likelihood-based statistical modelling and inference. Philosophy of science, statistical puzzles and paradoxes have been lifelong interests.
Youngjo Lee graduated with a PhD in statistics in 1983 from Iowa State University. He is currently a professor emeritus of statistics at Seoul National University, an endowed-chair professor of data and knowledge service engineering at Dankook University, and a vice president of the Korean Academy of Science and Technology. Alongside the late John Asworth Nelder, he is an originator of hierarchical generalized models and h-likelihood, having co-authored over 200 peer-reviewed research papers on the application of h-likelihood in various statistical areas. He is also a co-author of monographs on h-likelihood theory and applications. Furthermore, he has developed related software and is currently extending h-likelihood procedures to deep neural networks.
“This beautiful book is a tour de force, guiding the reader from early efforts to extract understanding from observations right up to the basis of modern cutting-edge analytic tools. Statisticians, data scientists, and even philosophers will gain new insights from the light shed by this book, and it is certainly one I shall return to repeatedly. Its style is accessible and engrossing, although it does become more mathematical in later sections. Early chapters focus on philosophical aspects (what is knowledge? what is truth?) and it develops through discussion of the challenges of deduction and induction, via the rise of probability, to likelihood and the authors’ developments of H-likelihood and extended likelihood. The final section deconstructs several classical inferential paradoxes in an entertaining and deeply informative way.”
~Professor David J. Hand, Imperial College, London“This is a remarkable book: wide-ranging, ambitious, challenging and profound but also intriguing, fascinating and original. Despite the fact that I have been thinking about the foundations of statistics for a long time, in particular as to how those foundations relate to what we want to do and achieve as practising statisticians, I learned a lot from reading it and also unlearned some things I thought I knew.”
~Stephen Senn, from the Foreword of the book"It takes the reader on a sweeping conceptual journey covering the foundations of statistical inference, epistemology, philosophy of science, probability, decision-making, and more. The book begins with a whirlwind tour of epistemology and philosophy, carefully tailored to its audience and establishing the critical issues at stake in the journey ahead. The book progressively becomes more original and provocative, culminating in Pawitan and Lee's original work on extended likelihood and epistemic confidence and their use in resolving conceptual issues -- the 'paradoxes' of the title -- in the foundations of statistical and scientific inference. A real highlight of the book is the consistently careful and wise treatment of complex and oft-misunderstood foundational and conceptual issues. I highly recommend this book to anyone looking to begin or continue their own statistically- and philosophically-informed journey and 'Search for Truth'."
~Oliver Maclaren, University of Auckland“A fun and enjoyable read, highly recommended. Mathematicians, statisticians, philosophers, and the inquiring user of data will all learn a lot from it, although probably in very different ways! I especially liked the book’s interweaving of the history and foundations of the subject, all the while drawing on many of the classical conundrums that help make it at once fascinating and challenging. And in particular, surely almost everyone will be intrigued by the final part, which discusses some of the classical paradoxes of probability such as the Allais and Ellsberg paradoxes of utility, old chestnuts such as the Monty Hall, lottery, and exchange paradoxes, and Carl Hempel’s paradox of the ravens.”
~Sandy L Zabell, Northwestern University“Yudi Pawitan and Youngjo Lee's book is a must-read for anyone interested in the philosophy of scientific inference. It is the first to offer a systematic treatment of a wide range of paradoxes and puzzles in epistemology and philosophy of science from a likelihoodist perspective. Breaking from the usual focus on debates and tensions, the book's likelihoodist stance seeks to cooperate with some frequentist and Bayesian ideas. This effort to bridge the gap between these viewpoints deserves more attention. Moreover, the presentation even makes sophisticated advances in likelihoodism accessible to a broader audience outside of statistics, including philosophers. We are in urgent need of more works like this to foster collaboration between philosophy-minded statisticians and statistics-minded philosophers.”
~Hanti Lin, University of California, Davis"Pawitan and Lee take us on an idiosyncratic and engaging journey through the logical foundations of probability and statistics, often forking out in unexpected, but always rewarding, directions. Original accounts of the major approaches and tools of statistical inference—frequentist, Bayesian, confidence, and, with a special emphasis, likelihood—are firmly located in their broader historical and philosophical context. A particularly valuable feature is the extensive collection of puzzles and paradoxes, whose detailed analysis is both amusing and revealing, and constitutes a refreshing new approach to comparative statistical inference. There is much nourishing food for thought here, both for the professional statistician and for the interested lay person."
~Philip Dawid, University of Cambridge“I disagree with much of this book, but it's an entertaining and thought-provoking introduction to some challenging questions.”
~ Andrew Gelman, Department of Statistics and Department of Political Science, Columbia University
