Integration and verification. The above-mentioned algorithm has been implemented into the toolchain and tested. A step-by-step implementation of MPO can be found in D1.3.1. In short, as shown in the diagram 20, the training gets initiated, and first trajectories/episodes are sampled. Each step in the episode corresponds to a loop in the toolchain. If a predetermined number of episodes has been gathered and stored in an experience buffer, then the critic and actor are updated using samples from that buffer. Samples can be re-used multiple times. However, over-fitting can become an issue here. In continuation, the sampling and updating is repeated. Figure 20 Process diagram of the RLA’s training , it's essential to isolate any errors caused by the environment (e.g., Commented [v(66]: Do I understand correctly that you're referring to potential bugs in other components of the pipeline? Commented [M(67R66]: You are right... I rephrased it. Is it clearer now? Commented [v(68R66]: Yes! To ensure the algorithm's accurate performance and other components in the pipeline), rather than from the RL algorithm’s implementation first conduct tests during early development in a controlled and well-understood setting. This is especially important as the training environment in this use case is extensive and feedback data is expensive. To achieve this, we employed the standard [A20] to validate the training process. However, since this library doesn't offer an environment capable of handling both discrete and continuous actions simultaneously as required by the use case, we utilized a superposition of a discrete and continuous environment as it can be seen in Figure 21. Specifically, we employed the "LunarLander" environment, which has both discrete and continuous action versions. Commented [v(69]: Might be nice to include something like a figure here to illustrate this concept Commented [M(70R69]: Please review =) Commented [v(71R69]: Very nice! In this hybrid setup, we concatenated observations, while actions were separated based on their affiliation to the discrete or continuous environment. The reward from the two environments was added together. Initially, we used default values as a baseline. Nevertheless, we also explored variations in the training process and adjustments to the actor's neural network. Version Status Date Page 2.0 Non-Confidential 2024.05.1172022.03.1 46/100 Figure 21 Visualization of the adapted LunarLander gym environment for early algorithm verification and the actor network ...
