Summary of Week 12 -- Capsule 4 -- Solving MDPs

00:00:00 - 00:15:00

This video discusses how to solve an MDP using two different algorithms – policy iteration and value duration. Policy iteration is faster per iteration, but it converges in fewer iterations.

• 00:00:00 In this capsule, the author discusses how to find the optimal policy for an mdp. Three well-known techniques for doing this are value duration, policy iteration, and linear programming. The author introduces term, value function, and explains that vfst is the expected sum of rewards of being an s. Next, the author explains how to calculate an expectation over rewards, and how to take the max when choosing a policy.
• 00:05:00 The bellman equation is a recursive equation that calculates the value of a state, as well as the rewards received by its neighbors, based on the state's reward and the rewards received by its neighbors. Dynamic programming is a method for solving problems that decomposes the problem into a series of sub-problems that are solved in turn, until the final solution is reached.
• 00:10:00 The video discusses value iteration and policy iteration, two algorithms used to solve MDPs. Value iteration allows for the optimization of an andp, while policy iteration improves the policy explicitly. Once the policy does not change, the algorithm declares convergence.
• 00:15:00 In this video, the presenter discusses the two algorithms used for finding an optimal policy – policy iteration and value duration. Policy iteration is faster per iteration, but it converges in fewer iterations.