340 pages | 23 B/W Illus.
Clear, rigorous, and intuitive, Markov Processes provides a bridge from an undergraduate probability course to a course in stochastic processes and also as a reference for those that want to see detailed proofs of the theorems of Markov processes. It contains copious computational examples that motivate and illustrate the theorems. The text is designed to be understandable to students who have taken an undergraduate probability course without needing an instructor to fill in any gaps.
The book begins with a review of basic probability, then covers the case of finite state, discrete time Markov processes. Building on this, the text deals with the discrete time, infinite state case and provides background for continuous Markov processes with exponential random variables and Poisson processes. It presents continuous Markov processes which include the basic material of Kolmogorov’s equations, infinitesimal generators, and explosions. The book concludes with coverage of both discrete and continuous reversible Markov chains.
While Markov processes are touched on in probability courses, this book offers the opportunity to concentrate on the topic when additional study is required. It discusses how Markov processes are applied in a number of fields, including economics, physics, and mathematical biology. The book fills the gap between a calculus based probability course, normally taken as an upper level undergraduate course, and a course in stochastic processes, which is typically a graduate course.
"All chapters are followed by exercises that render this text-book attractive for teachers…"
"Kirkwood…has published another significant mathematics monograph."
"Suitable for audiences who strive to grasp the fundamental concepts of various types of Markov processes or to prepare for learning advanced stochastic processes… the monograph can serve as a textbook since it provides essential examples and exercise problems applied in economics, finance, engineering, physics, and biology."
—S-T. Kim, North Carolina A&T State University
Review of Probability
Review of Basic Probability Definitions
Some Common Probability Distributions
Properties of a Probability Distribution
Properties of the Expected Value
Expected Value of a Random Variable with Common Distributions
Moment Generating Functions
Discrete-Time, Finite-State Markov Chains
Directed Graphs: Examples of Markov Chains
Random Walk with Reflecting Boundaries
Central Problem of Markov Chains
Condition to Ensure a Unique Equilibrium State
Finding the Equilibrium State
Transient and Recurrent States
Absorbing Markov Chains
Mean First Passage Time
Mean Recurrence Time and the Equilibrium State
Fundamental Matrix for Regular Markov Chains
Dividing a Markov Chain into Equivalence Classes
Periodic Markov Chains
Reducible Markov Chains
Discrete-Time, Infinite-State Markov Chains
Delayed Renewal Processes
Equilibrium State for Countable Markov Chains
Physical Interpretation of the Equilibrium State
Null Recurrent versus Positive Recurrent States
Random Walk in
Exponential Distribution and Poisson Process
Continuous Random Variables
Cumulative Distribution Function (Continuous Case)
Exponential Distribution as a Model for Arrivals
Memoryless Random Variables
Poisson Processes with Occurrences of Two Types
Continuous-Time Markov Chains
Generators of Continuous Markov Chains: The Kolmogorov Forward and Backward Equations
Connection Between the Steady State of a Continuous Markov Chain and the Steady State of the Embedded Matrix
Birth and Birth-Death Processes
Birth and Death Processes
Detailed Balance Equations
Reversible Markov Chains
Random Walks on Weighted Graphs
Discrete-Time Birth-Death Process as a Reversible Markov Chain
Continuous-Time Reversible Markov Chains