Methods of special interest to XXXXXX. In the scope of XXXXXX the idea is to design first a digital twin of the process (e.g. Electron Microscope) and then use learning tools to design policies based on simulations from this digital twin. This description fits very well within the methods described above that fall into the second and third categories of model- based reinforcement learning, i.e., model learned, planning/simulation based (referred to as Explicit Planning on Learned Transitions in [21]; and model learned, experiments/simulation based (referred to as Explicit Planning on Learned Transitions with Hybrid Model-Free/Model-Based Imagination in [21]). All the methods described under these categories can be used for XXXXXX. Of special interest for models where real data can be combined with synthetic data obtained with the digital twin is Dyna-Q, the simple architecture proposed in [12,13] that integrates models and experience as summarized in Fig. 11. Experience can both be used to make better models, which can then be used to plan decisions, or to apply direct decisions based on experimental data based methods. This is a fundamental paradigm useful in the XXXXXX use-cases that have very limited experimental data and can leverage this framework for sample efficiency. Another important method is PILCOS [6] for problems with low dimension. In essence, it is a policy gradient method which uses smart approximation to estimate the influence of changing the policy in terms of performance through a gradient based numerical method. The main bottleneck is the curse of dimensionality. For higher dimensional problems iLQG [Xxxxx et al., 2012] and Guided Policy Search (GPS) are very strong options. Commented [K(29]: Monte Carlo While the full model is often not available in the use-cases of XXXXXX, the methods that assume that the model is known are still powerful methods such as stochastic rollout and Monte Carlo tree search and can either be directly useful or adjusted for the problems at hand. Stochastic rollout [10,11] is an approximate method for optimal control problems with stochastic disturbances in which (Xxxxx-Xxxxx) simulations are used to approximate the costs-to-go when a so-called based policy is used. The consequence of each action at a given stage is evaluated based on these simulations and best decisions are selected. Xxxxx-Xxxxx tree search methods [7,8,9] operate similarly but consider larger depths or lookahead horizons for decisions and prune which decisions to consider based o...
