Markov Processes: 1st Edition (Hardback) book cover

Markov Processes

1st Edition

By James R. Kirkwood

CRC Press

340 pages | 23 B/W Illus.

Purchasing Options:$ = USD
Hardback: 9781482240733
pub: 2015-02-09
eBook (VitalSource) : 9780429069116
pub: 2015-02-09
from $28.98

FREE Standard Shipping!


Clear, rigorous, and intuitive, Markov Processes provides a bridge from an undergraduate probability course to a course in stochastic processes and also as a reference for those that want to see detailed proofs of the theorems of Markov processes. It contains copious computational examples that motivate and illustrate the theorems. The text is designed to be understandable to students who have taken an undergraduate probability course without needing an instructor to fill in any gaps.

The book begins with a review of basic probability, then covers the case of finite state, discrete time Markov processes. Building on this, the text deals with the discrete time, infinite state case and provides background for continuous Markov processes with exponential random variables and Poisson processes. It presents continuous Markov processes which include the basic material of Kolmogorov’s equations, infinitesimal generators, and explosions. The book concludes with coverage of both discrete and continuous reversible Markov chains.

While Markov processes are touched on in probability courses, this book offers the opportunity to concentrate on the topic when additional study is required. It discusses how Markov processes are applied in a number of fields, including economics, physics, and mathematical biology. The book fills the gap between a calculus based probability course, normally taken as an upper level undergraduate course, and a course in stochastic processes, which is typically a graduate course.


"All chapters are followed by exercises that render this text-book attractive for teachers…"

Zentralblatt MATH

"Kirkwood…has published another significant mathematics monograph."

"Suitable for audiences who strive to grasp the fundamental concepts of various types of Markov processes or to prepare for learning advanced stochastic processes… the monograph can serve as a textbook since it provides essential examples and exercise problems applied in economics, finance, engineering, physics, and biology."

—S-T. Kim, North Carolina A&T State University

Table of Contents

Review of Probability

Short History

Review of Basic Probability Definitions

Some Common Probability Distributions

Properties of a Probability Distribution

Properties of the Expected Value

Expected Value of a Random Variable with Common Distributions

Generating Functions

Moment Generating Functions


Discrete-Time, Finite-State Markov Chains



Transition Matrices

Directed Graphs: Examples of Markov Chains

Random Walk with Reflecting Boundaries

Gambler’s Ruin

Ehrenfest Model

Central Problem of Markov Chains

Condition to Ensure a Unique Equilibrium State

Finding the Equilibrium State

Transient and Recurrent States

Indicator Functions

Perron-Frobenius Theorem

Absorbing Markov Chains

Mean First Passage Time

Mean Recurrence Time and the Equilibrium State

Fundamental Matrix for Regular Markov Chains

Dividing a Markov Chain into Equivalence Classes

Periodic Markov Chains

Reducible Markov Chains



Discrete-Time, Infinite-State Markov Chains

Renewal Processes

Delayed Renewal Processes

Equilibrium State for Countable Markov Chains

Physical Interpretation of the Equilibrium State

Null Recurrent versus Positive Recurrent States

Difference Equations

Branching Processes

Random Walk in


Exponential Distribution and Poisson Process

Continuous Random Variables

Cumulative Distribution Function (Continuous Case)

Exponential Distribution

o(h) Functions

Exponential Distribution as a Model for Arrivals

Memoryless Random Variables

Poisson Process

Poisson Processes with Occurrences of Two Types


Continuous-Time Markov Chains


Generators of Continuous Markov Chains: The Kolmogorov Forward and Backward Equations

Connection Between the Steady State of a Continuous Markov Chain and the Steady State of the Embedded Matrix


Birth and Birth-Death Processes

Birth and Death Processes

Queuing Models

Detailed Balance Equations


Reversible Markov Chains

Random Walks on Weighted Graphs

Discrete-Time Birth-Death Process as a Reversible Markov Chain

Continuous-Time Reversible Markov Chains



About the Originator


About the Series

Advances in Applied Mathematics

Learn more…

Subject Categories

BISAC Subject Codes/Headings:
BUSINESS & ECONOMICS / Operations Research
MATHEMATICS / Probability & Statistics / Bayesian Analysis