Essentials of Stochastic Processes

Contents

Review of Probability

1. Probabilities, Independence 1
2. Random Variables, Distributions 8
3. Expected Value, Moments 18

1. Markov Chains

1. Definitions and Examples 28
2. Multistep Transition Probabilities 34
3. Classification of States 39
4. Limit Behavior 48
5. Some Special Examples 59
6. One-Step Calculations 66
7. Infinite State Spaces 73
8. Proofs of the Convergence Theorems 81
9. Exercises 88 

2. Martingales

1. Conditional Expectation 100
2. Examples of Martingales 102
3. Optional Stopping Theorem 110
4. Applications 114
5. Exercises 121

3. Poisson Processes

1. Exponential Distribution 126
2. Defining the Poisson Process 130
3. Compund Poisson Processes 137
4. Thinning and Superposition 140
5. Conditioning 142
6. Spatial Poisson Processes 145
7. Exercises 152 

4. Continuous-Time Markov Chains

1. Definitions and Examples 159
2. Computing the Transition Probability 164
3. Limiting Behavior 169
4. Queueing Chains 176
5. Reversibility 181
6. Queueing Networks 185
7. Closed Queueing Networks 195
8. Exercises 200

5. Renewal Theory

1. Basic Definitions 209
2. Laws of Large Numbers 214
3. Applications to Queueing Theory 221
4. Age and Residual Life 228
5. Exercises 234

6. Brownian Motion

1. Basic Definitions 242
2. Markov Property, Reflection Principle 246
3. Martingales, Hitting Times 250
4. Option Pricing in Discrete Time 257
5. The Black--Scholes Formula 261
6. Exercises 265

References 271

Answers to Selected Exercises 273

Index 279