Summary of Jessica Hamrick - Mental Simulation, Imagination, and Model-Based Deep RL @ UCL DARK

00:00:00 - 00:35:00

Jessica Hamrick discusses the role of mental simulation in human cognition, and how it is predictive, compositional, and adaptive. She then discusses recent work on model-based reinforcement learning, focusing on how planning can benefit the performance of agents.

• 00:00:00 In her keynote at the UCL DARK conference, Jessica Hamrick discusses how humans are the ultimate problem solvers and use their imagination and creativity to find solutions. She also discusses how mental stimulation, or the ability to simulate future events, is predictive and allows for better decision-making. Hamrick says that although we are not yet close to understanding how mental stimulation works in humans completely, there are ways to strive to close this gap using compositionality.
• 00:05:00 In this talk, Jessica Hamrick discusses the different types of mental simulations that humans engage in, including physical simulation, compositional simulation, and causal simulation. She also discusses the role of cognition in these activities and the ways that memory and imagination are highly compositional.
• 00:10:00 This video discusses the role of mental simulation in human cognition, and how it is predictive, compositional, and adaptive. It then discusses recent work on model-based reinforcement learning, focusing on how planning can benefit the performance of agents.
• 00:15:00 In this video, Jessica Hamrick explains how model-based deep reinforcement learning (RL) can be improved by using deeper search strategies. The results of this analysis suggest that, while deeper search does improve performance in some domains, it is primarily helpful in constructing targets for learning and obtaining useful data distributions.
• 00:20:00 In this video, Jessica Hamrick discusses the effects of deep RL on model-based decision-making. She finds that in most cases, a shallow tree of two or three nodes is sufficient, and complex planning is not usually necessary. The only exception is the game of nine by nine go, where deeper planning leads to better performance. Generalization is also important, and more planning at test time often fails to improve performance.
• 00:25:00 Jessica Hamrick discusses her work on mental simulation, imagination, and model-based deep rl at UCL. She shows how simple shallow forms of planning may be sufficient in many popular model-based rl environments, and concludes that effective planning requires good representations for multiple components. Finally, she discusses how a new type of agent, gndqn for graph network dqm, is designed to deal with changing action spaces in environments like block stacking.
• 00:30:00 In this video, Jessica Hamrick discusses how graph-based neural networks can be used to model and simulate mental processes such as imagination and simulation. She also demonstrates how these networks are able to generalize beyond the training data.
• 00:35:00 Jessica Hamrick discusses the limitations of model-based reinforcement learning, and how using more sophisticated planning techniques can help improve performance. She also mentions a tutorial she gave on the subject with Igor Mordatcha.