Reinforcement Learning in Cellular Networks. In this section, we will comprehensively evaluate conventional RL and DRL, and touch upon some of the latest literature in the field. Typically, the wireless environment is random and mostly unknown, with no accurate information regarding its model. RL offers model-free frameworks that are practical to solve various issues such as resource allocation, stochastic optimization and association. As a markovian process, agents interact with an unknown environment to explore and exploit it to find an optimal policy in RL. Q-learning Reinforcement Learning Mathematically, MDP can be denoted by 4-tuple < S, A, P, R >, where S is the state space, A is the action space, P is the state transition probability, with P(s′|s, a) specifying the probability of transitioning to the next state s′ ∈ S given the current state s ∈ S after applying the action a ∈ A, and R is the immediate reward received by the agent, usually denoted by Rn for reward received at time step n or R(s, a) to show its general dependency on s and a. The agent’s actions are bound by its policy π : S ×A → [0, 1], where π(a|s) gives the probability of performing action a ∈ A when in state s ∈ S. The agent thrives to improve its policy π based on its experience to maximize its long-term expected k=0 return E[Gn], where the return Gn ≜ ∑∞ γkRn+k is the accumulated discounted reward from time step n onwards with a discount factor 0 ≤ γ ≤ 1. See appendix A. In cellular networks, environments are highly dynamic and variant, making some decision-making problems such as resource allocation, interference mitigation and cell association computationally expensive. The utilization of Q-learning provides a reliable solution to these traditional challenges. Q-learning is based on finite MDP; thus the states and actions are quantized into discrete levels. In recent years, RL-based approaches have been of great interest to many researchers in the wireless communications field, where the topic of Q-learning has been studied extensively in the context of resource optimization [55–59]. In [55], authors have proposed a generalized Q-learning scheme for CR networks aimed at simplifying network management and introducing the concept of self- organized networks. The objectives are to offload the balance between different cells and to optimize handover performance. Results have shown that the framework has performed and learned the best Time To Trigger (TTT) in the mobility robustness optimization (MRO). They concluded that ...
