Q-learning the learning rate

Abstract:

In reinforcement learning, the learning rate is a fundamental parameter that determines how past prediction errors affect future predictions. Traditionally, the learning rate is kept constant, yet behaviorally, the learning rate is known to change both within and across contexts. Here, we propose a model-free approach to setting the learning rate. Key to our proposal is that one think about the learning rate as an action to be chosen to minimize a loss (prediction risk). This differentiates our proposal from learning models where the learning rate is adapted in a mechanistic way or based on a model of the task at hand. We use Q-learning to achieve prediction risk minimization and to simultaneously learn the prediction risk. Our algorithm produces learning rates that are a function of both risk (uncertainty in stable environments) and volatility (likelihood of changes in the environment). Learning rates decrease with risk and increase in volatility. The same functional dependence emerges in model-based approaches to setting the learning rate. We discuss behavioral evidence that these results parallel human and animal behavior. Imaging of the dopaminergic system, insula and anterior cingulate cortex of the primate brain supports the premise behind our account, namely, that reward learning is uncertainty-sensitive.

back