Mathematical Theory of Bayesian Statistics: 1st Edition (Hardback) book cover

Mathematical Theory of Bayesian Statistics

1st Edition

By Sumio Watanabe

Chapman and Hall/CRC

320 pages | 50 B/W Illus.

Purchasing Options:$ = USD
Hardback: 9781482238068
pub: 2018-04-23
eBook (VitalSource) : 9781315373010
pub: 2018-04-27
from $101.97

FREE Standard Shipping!


Mathematical Theory of Bayesian Statistics introduces the mathematical foundation of Bayesian inference which is well-known to be more accurate in many real-world problems than the maximum likelihood method. Recent research has uncovered several mathematical laws in Bayesian statistics, by which both the generalization loss and the marginal likelihood are estimated even if the posterior distribution cannot be approximated by any normal distribution.


  • Explains Bayesian inference not subjectively but objectively.
  • Provides a mathematical framework for conventional Bayesian theorems.
  • Introduces and proves new theorems.
  • Cross validation and information criteria of Bayesian statistics are studied from the mathematical point of view.
  • Illustrates applications to several statistical problems, for example, model selection, hyperparameter optimization, and hypothesis tests.

This book provides basic introductions for students, researchers, and users of Bayesian statistics, as well as applied mathematicians.


Sumio Watanabe is a professor of Department of Mathematical and Computing Science at Tokyo Institute of Technology. He studies the relationship between algebraic geometry and mathematical statistics.


"Information criteria are introduced from the two viewpoints, model selection and hyperparameter optimization. In each viewpoint, the properties of the generalization loss and the free energy or the minus log marginal likelihood are investigated. The book is very nicely written with well-defined concepts and contexts. I recommend to all students and researchers." ~Rozsa Horvath-Bokor, Zentralblatt MATH

Table of Contents

Definition of Bayesian Statistics

    Bayesian Statistics

    Probability distribution

    True Distribution

    Statistical model, prior, and posterior

    Examples of Posterior Distributions

    Estimation and Generalization

    Marginal Likelihood or Partition Function

    Conditional Independent Cases

Statistical Models

    Normal Distribution

    Multinomial Distribution

    Linear regression

    Neural Network

    Finite Normal Mixture

    Nonparametric Mixture

Basic Formula of Bayesian Observables

    Formal Relation between True and Model

    Normalized Observables

    Cumulant Generating Functions

    Basic Bayesian Theory

Regular Posterior Distribution

    Division of Partition Function

    Asymptotic Free Energy

    Asymptotic Losses

    Proof of Asymptotic Expansions

    Point Estimators

Standard Posterior Distribution

    Standard Form

    State Density Function

    Asymptotic Free Energy

    Renormalized Posterior Distribution

    Conditionally Independent Case

General Posterior Distribution

    Bayesian Decomposition

    Resolution of Singularities

    General Asymptotic Theory

    Maximum A Posteriori Method

Markov Chain Monte Carlo

    Metropolis Method

    Basic Metropolis Method

    Hamiltonian Monte Carlo

    Parallel Tempering

    Gibbs Sampler

    Gibbs Sampler for Normal Mixture

    Nonparametric Bayesian Sampler

    Numerical Approximation of Bayesian Observables

    Generalization and Cross Validation Losses

    Numerical Free Energy

Information Criteria

    Model Selection

    Criteria for Generalization Loss

    Comparison of ISCV with WAIC

    Criteria for Free Energy

    Discussion for Model Selection

    Hyperparameter Optimization

    Criteria for Generalization Loss

    Criterion for Free energy

    Discussion for Hyperparameter Optimization

Topics in Bayesian Statistics

    Formal Optimality

    Bayesian Hypothesis Test

    Bayesian Model Comparison

    Phase Transition

    Discovery Process

    Hierarchical Bayes

Basic Probability Theory

Delta Function

Kullback-Leibler Distance

Probability Space

Empirical Process

Convergence of Expected Values

Mixture by Dirichlet Process

About the Author

Sumio Watanabe is a professor in the Department of Computational Intelligence and Systems Science at Tokyo Institute of Technology, Japan.

Subject Categories

BISAC Subject Codes/Headings:
MATHEMATICS / Probability & Statistics / General