Meta Reinforcement learning. A huge drawback of scores of RL algorithms is the fact that they are intimately tied to the environment they are trained and tested in. The upside is that you may overfit an algorithm to do well for a single task, such as an Atari game. But the consequence is that these agents do not generalize at all to slightly different tasks. This goes so far, that agents learn different behaviours pending the random seed in an otherwise identical environment [B30]. Meta RL tries the seemingly impossible, namely, to train agents that generalize to different environments that have never been seen during training. This is accomplished with a limited amount of finetuning, where a meta model adapts its internal configuration to the new environment. Early work from [B31] uses an LSTM cell for adaptation to new Markov Decision Processes (MDP), which was further developed in [B32] and [B33]. They train their model over a set of MDP’s. These tasks are somewhat different though similar in nature. Such as a robot with slightly different physical parameters, or a maze that differs. The main difference to traditional RL is the fact that the policy not only observes the state, but also the last reward and the last action. This mechanism is used so that the agent may Version Status Date Page 2.0 Non-Confidential 2024.05.1172022.03.1 70/100 learn from a history of states, actions and rewards and adjust the dynamics when needed. Key components are: • Deploying a recurrent model with a memory state. The hidden state is used to encapsulate knowledge on the current task. It is updated during roll outs • A meta-learning algorithm. In [B32], [B33], this can be gradient descent to update an LSTM next to a reset of the hidden state, the moment a new MDP is encountered. • A distribution of MDP’s. Work from [B34] treats the hyperparameters as learnable parameters: specifically, the discount factor and bootstrapping parameter are learned. These are optimized via a second (meta) objective function and using cross correlation over a sequence of consecutive episodes. As stated earlier, the exploitation versus exploration dilemma is central to RL. Common solutions include epsilon-greedy action selection, adding random noise to actions, or using some type of stochastic policy. Work from [B36] aims to learn structured action noise by conditioning it on a pre-task (latent) random variable. The variable is sampled per episode and should determine the exploration behaviour best for this particular roll ...
