IE 203 • Final • Operations Research II

What is a Markov Chain?

One-Step Transition Probabilities

Example

n-Step Transition Probabilities

Chapman-Kolmogorov Equations

Example

Unconditional State Probabilities

Example

Steady State Distribution

Example

Classes and State Properties

Periodicity and Ergodic Markov Chains

Example

Example

Example

First Passage Times

Example

Expected First Passage Time

Expected Recurrence Time

Example

Absorbing Chains

Absorption Probabilities

Example

Probability of Ever Visiting a State

Example

Expected Number of State Visits

Example

Expected Time Until Absoption

Example

From Discrete to Continuous

Example 1

Example 2

Example 3

Steady State Distribution

Q Matrix

Example 4

Example 5

Example 6

Introduction

Policy Improvement Algorithm: Step 1

Policy Improvement Algorithm: Step 2

Example 1

LP Solution

LP Solution with Discount

Counting Processes

Assumptions of Poisson Processes

Example 1

Memoryless Property

Example 2

Minimum of Exponential Random Variables

Example 3

Thinning

Example 4

Superposition

Example 5

Non-homogenous Poisson Processes

Example 6

A subproblem of CTMC

Random Rates, Finite Chain

Example 1

Random Rates, Infinite Chain

Example 2

Equal Rates, Infinite Chain, Single Server

Example 3

Equal Rates, Finite Chain, Single Server

Example 4

Equal Rates, Finite Chain, Multiple Servers

Example 5

Kendall's Notation

Terminology

M/M/1 Queue

Example 1

M/M/s Queue

Example 2

M/M/1/K Queue

Example 3

M/M/s/K Queue

Example 4

Markov Chains - Discrete 1

Markov Chains - Discrete 2

Markov Chains - Discrete 3

Markov Chains - Discrete 4

Markov Chains - Discrete 5

Markov Chains - Discrete 6

Markov Chains - Discrete 7

Markov Chains - Discrete 8

Markov Chains - Discrete 9

Markov Chains - Discrete 10

Markov Chains - Discrete 11

Markov Chains - Discrete 12

Markov Chains - Continuous 1

Markov Chains - Continuous 2

Markov Chains - Continuous 3

Markov Decision Process 1

Poisson Processes 1

Poisson Processes 2

Poisson Processes 3

Poisson Processes 4

Queueing Theory 1

Queueing Theory 2

Queueing Theory 3

Queueing Theory 4