M step. Maximizing Q with respect to θ is generally a straightforward task once the conditional expectations have been approximated. This is particularly true when the assumed distri- bution of the outcome is a member of the exponential family, for which there are numerous maximization functions available in software packages (e.g. SAS, R).
M step. The second step is to fit a parametric policy to the improved policy calculated in the first step using supervised learning. The fitting of the next policy parameter θi+1 is done by solving a weighted maximum likelihood problem while requiring the policy change from one iteration to the next is constrained. In this step, the hybrid policy is optimized in a decoupled form for its discrete and continuous parts separately. The adaptation of hybrid policy optimization using the actor-critic framework for the UUV use case is depicted in Figure 19. Version Status Date Page 2.0 Non-Confidential 2024.05.1172022.03.1 43/100 Figure 19 Architecture for the hybrid RL approach (actor-critic) in the UUV use case, inspired by [A2] and [A5]. Boxes denote components, arrows depict data flow. The layers on the right represent a set of distributed actor instances. In each iteration step, the actor instances (stack of layers) comprising the RL agent gather experience in episodes of a fixed length (number of steps) through interacting with the respectively instantiated version of the environment (= DT; see Section 4.2.2) while following the current policy πθi constructed by the learner. At each layer, the discrete and continuous actions, i.e., scenario parameters, are fed into the DT simultaneously by the RL agent and based on the current state of the DT. The experience data (trajectories of selected actions and their log probabilities, and transitions, i.e., successor states and obtained rewards) from all actor instances are stored into the replay buffer and fed to the learner. Then, after some stopping criterion is fulfilled, the learner updates the policy based on the gathered experience data according to the above-described procedure. In the distributed actor framework ACME (see [A9]) also applied by Xxxxxxx et al. [A2], it is designated that the actors gain experience by interacting with their own instance of the environment (=DT) in parallel and pull policy parameter updates from the learner asynchronously to accelerate the data generation process. In addition, this distributed architecture allows for the learning process to proceed as quickly as possible regardless of the speed of data gathering. However, to start in a simple, less error-prone way, we aim to set up the learning architecture first with a single actor instance and consider extending this to a distributed actor setup at a later point in time.
