Functional view. Commented [A(42]: something went wrong here, I believe it is Section 3 From an abstract, functional perspective the training process is shown as an optimization process in Figure 13 (using the IDEF0 format [55]: input arrows impinging from left into the function box, outputs exiting at right, control inputs into the top; arrow from below indicates the system performing the function). From this perspective three major functions can be distinguished: the controller function, the modelled behaviour function, and the ‘AI-function’. The controller (Optimization Control System) decides how well the training proceeds, and if the process should be stopped. The function F1 (performed by the Digital Twin) is responsible for creating the (modelled/simulated) system behaviour (expressed as ‘system output’). Function F2 is responsible for learning from the behaviour provided to it, and for suggesting new settings in order to learn more. This is obviously performed by the AI system. The diagram also indicates on the arrows the essential information needed to enable the training. To realize a real solution for industrial application, these functions and information streams must be mapped to an implementation, see section 2.4.2. Version Status Date Page 2.0 Non-Confidential 2024.05.1172022.03.1 28/100 Figure 13 Functional diagram of the training process.
Functional view. In the case of the AI subsystem, we decompose the functional view into the training phase and the operational or deployment phase. For XXXXXX the training phase comprises the typical RL blocks, which means that the agent interacts with the own defined environment. That environment is based on the gym-framework, which enables the agent to go to through the sequential steps of the problem. The steps followed are defined by us following the logic of the problem. The RL algorithm used will update its policy and learn a (desired) behaviour in the training process. Depending on the RL algorithm used, the method of learning and updating the policy will differ. In the next section the differences when updating the policy will be highlighted. In this section, through the graph, the learning process can be visualized in Figure 71. Version Status Date Page 2.0 Non-Confidential 2024.05.1172022.03.1 85/100 Commented [M(126]: Is this the latest version? This seems to have fewer loops in it than previous versions. Commented [M(127]: What is the "policy update threshold"? Figure 71 Flow diagram of main functions and decisions during the training phase. Decision points contain exemplary numbers. The training phase generally consists of three nested loops. The lowest loop is the obtaining a state. predicting the next action, obtaining the next state and so on. This loop is terminated when an optimal state is reached. The sequence of steps is often referred to as an episode. The superordinate loop iterates through episodes. After a certain number of episodes or states across the performed episodes, the policy is updated. The highest order loop then assesses the state of the training which consists of iteratively updating the policy (decision: “Policy update threshold?”). It is generally stopped after a predefined number of policy updates or the evaluation of the training progress on a validation test set. Commented [M(128]: I believe this is not the case for actor-critic. You don't set an exploration rate in this case. Commented [A(129R128]: You are right, how you do this for actor-critic then? I guess there is exploration/exploitation in place but somehow implicitly encoded? In the operational phase no more learning is done, the learned behaviour is the one that will be deployed on the machine. For this, the trained policy encoded into e.g., a neural net, is used as is without any further updating. To do so, one either can decouple the artefact from the training algorithm (...
