6.262 Discrete Stochastic Processes (Spring 2011, MIT OCW): Lecture 09

6.262 Discrete Stochastic Processes

6.262 Discrete Stochastic Processes (Spring 2011, MIT OCW). Instructor: Professor Robert Gallager. Discrete stochastic processes are essentially probabilistic systems that evolve in time via random changes occurring at discrete fixed or random intervals. This course aims to help students acquire both the mathematical principles and the intuition necessary to create, analyze, and understand insightful models for a broad range of these processes. The range of areas for which discrete stochastic-process models are useful is constantly expanding, and includes many applications in engineering, physics, biology, operations research and finance. (from ocw.mit.edu)

Lecture 09 - Markov Rewards and Dynamic Programming

This lecture covers rewards for Markov chains, expected first passage time, and aggregate rewards with a final reward. The professor then moves on to discuss dynamic programming and the dynamic programming algorithm.

Go to the Course Home or watch other lectures:

Lecture 01 - Introduction and Probability Review

Lecture 02 - More Review; The Bernoulli Process

Lecture 03 - Law of Large Numbers, Convergence

Lecture 04 - Poisson (the Perfect Arrival Process)

Lecture 05 - Poisson Combining and Splitting

Lecture 06 - From Poisson to Markov

Lecture 07 - Finite-state Markov Chains; The Matrix Approach

Lecture 08 - Markov Eigenvalues and Eigenvectors

Lecture 09 - Markov Rewards and Dynamic Programming

Lecture 10 - Renewals and the Strong Law of Large Numbers

Lecture 11 - Renewals: Strong Law and Rewards