View source for Reinforcement learning

{{Short description|Field of machine learning}}
{{For|reinforcement learning in psychology|Reinforcement|Operant conditioning}}
[[File: Reinforcement learning diagram.svg|thumb|right|250px| The typical framing of a Reinforcement Learning (RL) scenario: an agent takes actions in an environment, which is interpreted into a reward and a state representation, which are fed back to the agent.]]
{{Machine learning|Reinforcement learning}}

'''Reinforcement learning''' ('''RL''') is an interdisciplinary area of [[machine learning]] and [[optimal control]] concerned with how an [[intelligent agent]] should [[Action selection|take actions]] in a dynamic environment in order to [[Reward-based selection|maximize a reward]] signal. Reinforcement learning is one of the [[Machine learning#Approaches|three basic machine learning paradigms]], alongside [[supervised learning]] and [[unsupervised learning]].

[[Q-learning]] at its simplest stores data in tables. This approach becomes [[Computational complexity theory|infeasible]] as the number of states/actions increases (e.g., if the state space or action space were continuous), as the probability of the agent visiting a particular state and performing a particular action diminishes. 

Reinforcement learning differs from supervised learning in not needing labelled input-output pairs to be presented, and in not needing sub-optimal actions to be explicitly corrected. Instead, the focus is on finding a balance between exploration (of uncharted territory) and exploitation (of current knowledge) with the goal of maximizing the cumulative reward (the feedback of which might be incomplete or delayed).<ref name="kaelbling">{{cite journal|last1=Kaelbling|first1=Leslie P.|last2=Littman|first2=Michael L.|author-link2=Michael L. Littman|last3=Moore|first3=Andrew W.|author-link3=Andrew W. Moore|year=1996|title=Reinforcement Learning: A Survey|url=http://www.cs.washington.edu/research/jair/abstracts/kaelbling96a.html|url-status=dead|journal=Journal of Artificial Intelligence Research|volume=4|pages=237–285|doi=10.1613/jair.301|archive-url=http://webarchive.loc.gov/all/20011120234539/http://www.cs.washington.edu/research/jair/abstracts/kaelbling96a.html|archive-date=2001-11-20 |author-link1=Leslie P. Kaelbling|arxiv=cs/9605103|s2cid=1708582}}</ref> The search for this balance is known as the [[exploration–exploitation dilemma]].

The environment is typically stated in the form of a [[Markov decision process]] (MDP), as many reinforcement learning algorithms use [[dynamic programming]] techniques.<ref>{{Cite book|author1=van Otterlo, M.|author2=Wiering, M.|title=Reinforcement Learning |chapter=Reinforcement Learning and Markov Decision Processes |volume=12|pages=3–42 |year=2012 |doi=10.1007/978-3-642-27645-3_1|series=Adaptation, Learning, and Optimization|isbn=978-3-642-27644-6}}</ref> The main difference between classical dynamic programming methods and reinforcement learning algorithms is that the latter do not assume knowledge of an exact mathematical model of the Markov decision process, and they target large MDPs where exact methods become infeasible.<ref name="Li-2023">{{cite book |last1=Li |first1=Shengbo |title= Reinforcement Learning for Sequential Decision and Optimal Control |date=2023 |location=Springer Verlag, Singapore |isbn=978-9-811-97783-1 |pages=1–460 |doi=10.1007/978-981-19-7784-8 |s2cid=257928563 |edition=First | url=https://link.springer.com/book/10.1007/978-981-19-7784-8}}</ref> {{toclimit|3}}

== Principles ==
Due to its generality, reinforcement learning is studied in many disciplines, such as [[game theory]], [[control theory]], [[operations research]], [[information theory]], [[simulation-based optimization]], [[multi-agent system]]s, [[swarm intelligence]], and [[statistics]]. In the operations research and control literature, RL is called ''approximate dynamic programming'', or ''neuro-dynamic programming.'' The problems of interest in RL have also been studied in the [[optimal control theory|theory of optimal control]], which is concerned mostly with the existence and characterization of optimal solutions, and algorithms for their exact computation, and less with learning or approximation (particularly in the absence of a mathematical model of the environment).

Basic reinforcement learning is modeled as a [[Markov decision process]]:

* A set of environment and agent states (the state space), <math>\mathcal{S}</math>;
* A set of actions (the action space), <math>\mathcal{A}</math>, of the agent;
* <math>P_a(s,s')=\Pr(S_{t+1}=s'\mid S_t=s, A_t=a)</math>, the transition probability (at time <math>t</math>) from state <math>s</math> to state <math>s'</math> under action <math>a</math>.
* <math>R_a(s,s')</math>, the immediate reward after transition from <math>s</math> to <math>s'</math> under action <math>a</math>.

The purpose of reinforcement learning is for the agent to learn an optimal (or near-optimal) policy that maximizes the reward function or other user-provided reinforcement signal that accumulates from immediate rewards. This is similar to [[Reinforcement|processes]] that appear to occur in animal psychology. For example, biological brains are hardwired to interpret signals such as pain and hunger as negative reinforcements, and interpret pleasure and food intake as positive reinforcements. In some circumstances, animals learn to adopt behaviors that optimize these rewards. This suggests that animals are capable of reinforcement learning.<ref>{{cite book |last1=Russell |first1=Stuart J. |last2=Norvig |first2=Peter |title=Artificial intelligence : a modern approach |date=2010 |location=Upper Saddle River, New Jersey |publisher=[[Prentice Hall]] |isbn=978-0-13-604259-4 |pages=830, 831 |edition=Third}}</ref><ref>{{cite journal |last1=Lee |first1=Daeyeol |last2=Seo |first2=Hyojung |last3=Jung |first3=Min Whan |title=Neural Basis of Reinforcement Learning and Decision Making |journal=Annual Review of Neuroscience |date=21 July 2012 |volume=35 |issue=1 |pages=287–308 |doi=10.1146/annurev-neuro-062111-150512|pmid=22462543 |pmc=3490621 }}</ref>

A basic reinforcement learning agent interacts with its environment in discrete time steps. At each time step {{mvar|t}}, the agent receives the current state <math>S_t</math> and reward <math>R_t</math>. It then chooses an action <math>A_t</math> from the set of available actions, which is subsequently sent to the environment. The environment moves to a new state <math>S_{t+1}</math> and the reward <math>R_{t+1}</math> associated with the ''transition'' <math>(S_t,A_t,S_{t+1})</math> is determined. The goal of a reinforcement learning agent is to learn a ''policy'': 

<math>\pi: \mathcal{S} \times \mathcal{A} \rightarrow [0,1] </math>, <math>\pi(s,a) = \Pr(A_t = a\mid S_t =s)</math>

that maximizes the expected cumulative reward.

Formulating the problem as a Markov decision process assumes the agent directly observes the current environmental state; in this case, the problem is said to have ''full observability''. If the agent only has access to a subset of states, or if the observed states are corrupted by noise, the agent is said to have ''partial observability'', and formally the problem must be formulated as a [[partially observable Markov decision process]]. In both cases, the set of actions available to the agent can be restricted. For example, the state of an account balance could be restricted to be positive; if the current value of the state is 3 and the state transition attempts to reduce the value by 4, the transition will not be allowed.

When the agent's performance is compared to that of an agent that acts optimally, the difference in performance yields the notion of [[Regret (decision theory)|regret]]. In order to act near optimally, the agent must reason about long-term consequences of its actions (i.e., maximize future rewards), although the immediate reward associated with this might be negative.

Thus, reinforcement learning is particularly well-suited to problems that include a long-term versus short-term reward trade-off. It has been applied successfully to various problems, including [[energy storage]],<ref>{{cite journal | doi=10.1016/j.epsr.2022.108515 | title=Community energy storage operation via reinforcement learning with eligibility traces | date=2022 | last1=Salazar Duque | first1=Edgar Mauricio | last2=Giraldo | first2=Juan S. | last3=Vergara | first3=Pedro P. | last4=Nguyen | first4=Phuong | last5=Van Der Molen | first5=Anne | last6=Slootweg | first6=Han | journal=Electric Power Systems Research | volume=212 | s2cid=250635151 | doi-access=free | bibcode=2022EPSR..21208515S }}</ref> [[robot control]],<ref>{{cite arXiv | eprint=2005.04323 | last1=Xie | first1=Zhaoming | author2=Hung Yu Ling | author3=Nam Hee Kim | author4=Michiel van de Panne | title=ALLSTEPS: Curriculum-driven Learning of Stepping Stone Skills | date=2020 | class=cs.GR }}</ref> [[Photovoltaic system|photovoltaic generators]],<ref>{{cite journal | doi=10.1016/j.ijepes.2021.107628 | title=Optimal dispatch of PV inverters in unbalanced distribution systems using Reinforcement Learning | date=2022 | last1=Vergara | first1=Pedro P. | last2=Salazar | first2=Mauricio | last3=Giraldo | first3=Juan S. | last4=Palensky | first4=Peter | journal=International Journal of Electrical Power & Energy Systems | volume=136 | s2cid=244099841 | doi-access=free | bibcode=2022IJEPE.13607628V }}</ref> [[backgammon]], [[checkers]],{{Sfn|Sutton|Barto|2018|p=|loc=Chapter 11}} [[Go (game)|Go]] ([[AlphaGo]]), and [[Self-driving car|autonomous driving systems]].<ref name="Ren-2022">{{cite journal | url=https://ieeexplore.ieee.org/document/9857655 | title=Self-Learned Intelligence for Integrated Decision and Control of Automated Vehicles at Signalized Intersections | date=2022 | doi=10.1109/TITS.2022.3196167 | last1=Ren | first1=Yangang | last2=Jiang | first2=Jianhua | last3=Zhan | first3=Guojian | last4=Li | first4=Shengbo Eben | last5=Chen | first5=Chen | last6=Li | first6=Keqiang | last7=Duan | first7=Jingliang | journal=IEEE Transactions on Intelligent Transportation Systems | volume=23 | issue=12 | pages=24145–24156 | arxiv=2110.12359 }}</ref>

Two elements make reinforcement learning powerful: the use of samples to optimize performance, and the use of [[Neural network (machine learning)|function approximation]] to deal with large environments. Thanks to these two key components, RL can be used in large environments in the following situations:

* A model of the environment is known, but an [[Closed-form expression|analytic solution]] is not available;
* Only a simulation model of the environment is given (the subject of [[simulation-based optimization]]);<ref>{{cite book|url = https://www.springer.com/mathematics/applications/book/978-1-4020-7454-7|title = Simulation-based Optimization: Parametric Optimization Techniques and Reinforcement|last = Gosavi|first = Abhijit|publisher = Springer|year = 2003|isbn = 978-1-4020-7454-7|author-link = Abhijit Gosavi|series = Operations Research/Computer Science Interfaces Series}}</ref> 
* The only way to collect information about the environment is to interact with it.

The first two of these problems could be considered planning problems (since some form of model is available), while the last one could be considered to be a genuine learning problem. However, reinforcement learning converts both planning problems to [[machine learning]] problems.

== Exploration ==
The exploration vs. exploitation trade-off has been most thoroughly studied through the [[multi-armed bandit]] problem and for finite state space Markov decision processes in Burnetas and Katehakis (1997).<ref name="Optimal adaptive policies for Marko">{{citation | last1 = Burnetas|first1 = Apostolos N.|last2 = Katehakis|first2 = Michael N.|author-link2 = Michael N. Katehakis|year = 1997|title = Optimal adaptive policies for Markov Decision Processes|journal = [[Mathematics of Operations Research]] |volume = 22 | issue=1 |pages = 222–255 |doi=10.1287/moor.22.1.222 | jstor=3690147}}</ref>

Reinforcement learning requires clever exploration mechanisms; randomly selecting actions, without reference to an estimated probability distribution, shows poor performance. The case of (small) finite Markov decision processes is relatively well understood. However, due to the lack of algorithms that scale well with the number of states (or scale to problems with infinite state spaces), simple exploration methods are the most practical.

One such method is <math>\varepsilon</math>-greedy, where <math>0 < \varepsilon < 1</math> is a parameter controlling the amount of exploration vs. exploitation.  With probability <math>1-\varepsilon</math>, exploitation is chosen, and the agent chooses the action that it believes has the best long-term effect (ties between actions are broken uniformly at random). Alternatively, with probability <math>\varepsilon</math>, exploration is chosen, and the action is chosen uniformly at random. <math>\varepsilon</math> is usually a fixed parameter but can be adjusted either according to a schedule (making the agent explore progressively less), or adaptively based on heuristics.<ref>{{citation | last1 = Tokic | first1 = Michel | last2 = Palm | first2 = Günther | chapter   = Value-Difference Based Exploration: Adaptive Control Between Epsilon-Greedy and Softmax | pages = 335–346 | publisher = Springer | series = Lecture Notes in Computer Science | title = KI 2011: Advances in Artificial Intelligence | volume = 7006 | year = 2011 | chapter-url = http://www.tokic.com/www/tokicm/publikationen/papers/KI2011.pdf | isbn = 978-3-642-24455-1}}</ref>

== Algorithms for control learning ==
Even if the issue of exploration is disregarded and even if the state was observable (assumed hereafter), the problem remains to use past experience to find out which actions lead to higher cumulative rewards.

=== Criterion of optimality ===
==== Policy ====
The agent's action selection is modeled as a map called ''policy'':
:<math>\pi: \mathcal{A} \times \mathcal{S} \rightarrow [0,1]</math>
:<math>\pi(a,s) = \Pr(A_t = a \mid S_t = s)</math>

The policy map gives the probability of taking action <math>a</math> when in state <math>s</math>.<ref name=":0">{{Cite web|url=http://people.inf.elte.hu/lorincz/Files/RL_2006/SuttonBook.pdf|title=Reinforcement learning: An introduction|access-date=2017-07-23|archive-date=2017-07-12|archive-url=https://web.archive.org/web/20170712170739/http://people.inf.elte.hu/lorincz/Files/RL_2006/SuttonBook.pdf|url-status=dead}}</ref>{{Rp|61}} There are also deterministic policies.

==== State-value function ====
The state-value function <math>V_\pi(s)</math> is defined as, ''expected discounted return'' starting with state <math>s</math>, i.e. <math>S_0 = s</math>, and successively following policy <math>\pi</math>. Hence, roughly speaking, the value function estimates "how good" it is to be in a given state.<ref name=":0" />{{Rp|60}}

:<math>V_\pi(s) = \operatorname \mathbb{E}[G\mid S_0 = s] = \operatorname \mathbb{E}\left[\sum_{t=0}^\infty \gamma^t R_{t+1}\mid S_0 = s\right],</math>

where the random variable <math>G</math> denotes the '''discounted return''', and is defined as the sum of future discounted rewards:

:<math>G=\sum_{t=0}^\infty \gamma^t R_{t+1}=R_1 + \gamma R_2 + \gamma^2 R_3 + \dots,</math>

where <math>R_{t+1}</math> is the reward for transitioning from state <math>S_t</math> to <math>S_{t+1}</math>, <math>0 \le \gamma<1</math> is the [[Q-learning#Discount factor|discount rate]]. <math>\gamma</math> is less than 1, so rewards in the distant future are weighted less than rewards in the immediate future.

The algorithm must find a policy with maximum expected discounted return. From the theory of Markov decision processes it is known that, without loss of generality, the search can be restricted to the set of so-called ''stationary'' policies. A policy is ''stationary'' if the action-distribution returned by it depends only on the last state visited (from the observation agent's history). The search can be further restricted to ''deterministic'' stationary policies. A ''deterministic stationary'' policy deterministically selects actions based on the current state. Since any such policy can be identified with a mapping from the set of states to the set of actions, these policies can be identified with such mappings with no loss of generality.

=== Brute force ===
The [[brute-force search|brute force]] approach entails two steps:
* For each possible policy, sample returns while following it
* Choose the policy with the largest expected discounted return

One problem with this is that the number of policies can be large, or even infinite. Another is that the variance of the returns may be large, which requires many samples to accurately estimate the discounted return of each policy.

These problems can be ameliorated if we assume some structure and allow samples generated from one policy to influence the estimates made for others. The two main approaches for achieving this are [[#Value function|value function estimation]] and [[#Direct policy search|direct policy search]].

=== Value function ===
{{see also|Value function}}
Value function approaches attempt to find a policy that maximizes the discounted return by maintaining a set of estimates of expected discounted returns <math>\operatorname \mathbb{E}[G]</math> for some policy (usually either the "current" [on-policy] or the optimal [off-policy] one).

These methods rely on the theory of Markov decision processes, where optimality is defined in a sense stronger than the one above: A policy is optimal if it achieves the best-expected discounted return from ''any'' initial state (i.e., initial distributions play no role in this definition). Again, an optimal policy can always be found among stationary policies.

To define optimality in a formal manner, define the state-value of a policy <math>\pi</math> by

:<math> V^{\pi} (s) = \operatorname \mathbb{E}[G\mid s,\pi],</math>

where <math>G</math> stands for the discounted return associated with following <math>\pi</math> from the initial state <math>s</math>. Defining <math>V^*(s)</math> as the maximum possible state-value of <math>V^\pi(s)</math>, where <math>\pi</math> is allowed to change,

:<math>V^*(s) = \max_\pi V^\pi(s).</math>

A policy that achieves these optimal state-values in each state is called ''optimal''. Clearly, a policy that is optimal in this sense is also optimal in the sense that it maximizes the expected discounted return, since <math>V^*(s) = \max_\pi \mathbb{E}[G\mid s,\pi]</math>, where <math>s</math> is a state randomly sampled from the distribution <math>\mu</math> of initial states (so <math>\mu(s) = \Pr(S_0 = s)</math>).

Although state-values suffice to define optimality, it is useful to define action-values. Given a state <math>s</math>, an action <math>a</math> and a policy <math>\pi</math>, the action-value of the pair <math>(s,a)</math> under <math>\pi</math> is defined by

:<math>Q^\pi(s,a) = \operatorname \mathbb{E}[G\mid s,a,\pi],\,</math>

where <math>G</math> now stands for the random discounted return associated with first taking action <math>a</math> in state <math>s</math> and following <math>\pi</math>, thereafter.

The theory of Markov decision processes states that if <math>\pi^*</math> is an optimal policy, we act optimally (take the optimal action) by choosing the action from <math>Q^{\pi^*}(s,\cdot)</math> with the highest action-value at each state, <math>s</math>. The ''action-value function'' of such an optimal policy (<math>Q^{\pi^*}</math>) is called the ''optimal action-value function'' and is commonly denoted by <math>Q^*</math>. In summary, the knowledge of the optimal action-value function alone suffices to know how to act optimally.

Assuming full knowledge of the Markov decision process, the two basic approaches to compute the optimal action-value function are [[value iteration]] and [[policy iteration]]. Both algorithms compute a sequence of functions <math>Q_k</math> (<math>k=0,1,2,\ldots</math>) that converge to <math>Q^*</math>. Computing these functions involves computing expectations over the whole state-space, which is impractical for all but the smallest (finite) Markov decision processes. In reinforcement learning methods, expectations are approximated by averaging over samples and using function approximation techniques to cope with the need to represent value functions over large state-action spaces.

====Monte Carlo methods====
[[Monte Carlo sampling|Monte Carlo methods]]<ref>{{Cite journal |last1=Singh |first1=Satinder P. |last2=Sutton |first2=Richard S. |date=1996-03-01 |title=Reinforcement learning with replacing eligibility traces |url=https://link.springer.com/article/10.1007/BF00114726 |journal=Machine Learning |language=en |volume=22 |issue=1 |pages=123–158 |doi=10.1007/BF00114726 |issn=1573-0565}}</ref> are used to solve reinforcement learning problems by averaging sample returns. Unlike methods that require full knowledge of the environment’s dynamics, Monte Carlo methods rely solely on actual or [[Simulation|simulated]] experience—sequences of states, actions, and rewards obtained from interaction with an environment. This makes them applicable in situations where the complete dynamics are unknown. Learning from actual experience does not require prior knowledge of the environment and can still lead to optimal behavior. When using simulated experience, only a model capable of generating sample transitions is required, rather than a full specification of [[Markov chain|transition probabilities]], which is necessary for [[dynamic programming]] methods.

Monte Carlo methods apply to episodic tasks, where experience is divided into episodes that eventually terminate. Policy and value function updates occur only after the completion of an episode, making these methods incremental on an episode-by-episode basis, though not on a step-by-step (online) basis. The term "Monte Carlo" generally refers to any method involving [[random sampling]]; however, in this context, it specifically refers to methods that compute averages from ''complete'' returns, rather than ''partial'' returns.

These methods function similarly to the [[Multi-armed bandit|bandit algorithms]], in which returns are averaged for each state-action pair. The key difference is that actions taken in one state affect the returns of subsequent states within the same episode, making the problem [[non-stationary]]. To address this non-stationarity, Monte Carlo methods use the framework of general policy iteration (GPI). While dynamic programming computes [[Value function|value functions]] using full knowledge of the [[Markov decision process]] (MDP), Monte Carlo methods learn these functions through sample returns. The value functions and policies interact similarly to dynamic programming to achieve [[Mathematical optimization|optimality]], first addressing the prediction problem and then extending to policy improvement and control, all based on sampled experience.<ref name=":0" />
====Temporal difference methods ====
{{Main|Temporal difference learning}}
The first problem is corrected by allowing the procedure to change the policy (at some or all states) before the values settle. This too may be problematic as it might prevent convergence. Most current algorithms do this, giving rise to the class of ''generalized policy iteration'' algorithms. Many [[Actor-critic algorithm|''actor-critic'' methods]] belong to this category.

The second issue can be corrected by allowing trajectories to contribute to any state-action pair in them. This may also help to some extent with the third problem, although a better solution when returns have high variance is Sutton's [[temporal difference]] (TD) methods that are based on the recursive [[Bellman equation]].<ref>{{cite thesis|last = Sutton|first = Richard S.|title = Temporal Credit Assignment in Reinforcement Learning|degree = PhD|publisher = University of Massachusetts, Amherst, MA|url = http://incompleteideas.net/sutton/publications.html#PhDthesis|author-link = Richard S. Sutton|year = 1984|access-date = 2017-03-29|archive-date = 2017-03-30|archive-url = https://web.archive.org/web/20170330002227/http://incompleteideas.net/sutton/publications.html#PhDthesis|url-status = dead}}</ref>{{sfn|Sutton|Barto|2018|loc=[http://incompleteideas.net/sutton/book/ebook/node60.html §6. Temporal-Difference Learning]}} The computation in TD methods can be incremental (when after each transition the memory is changed and the transition is thrown away), or batch (when the transitions are batched and the estimates are computed once based on the batch). Batch methods, such as the least-squares temporal difference method,<ref>{{cite journal
  | doi = 10.1023/A:1018056104778
  | last1 = Bradtke | first1 = Steven J. | author-link1 = Steven J. Bradtke
  | last2 = Barto | first2 = Andrew G. | author-link2 = Andrew G. Barto
  | title = Learning to predict by the method of temporal differences
  | journal = Machine Learning
  | volume = 22
  | pages = 33–57
  | year = 1996
  | citeseerx = 10.1.1.143.857 | s2cid = 20327856 }}</ref> may use the information in the samples better, while incremental methods are the only choice when batch methods are infeasible due to their high computational or memory complexity. Some methods try to combine the two approaches. Methods based on temporal differences also overcome the fourth issue.

Another problem specific to TD comes from their reliance on the recursive Bellman equation. Most TD methods have a so-called <math>\lambda</math> parameter <math>(0\le \lambda\le 1)</math> that can continuously interpolate between Monte Carlo methods that do not rely on the Bellman equations and the basic TD methods that rely entirely on the Bellman equations. This can be effective in palliating this issue.

==== Function approximation methods ====
In order to address the fifth issue, ''function approximation methods'' are used. ''Linear function approximation'' starts with a mapping <math>\phi</math> that assigns a finite-dimensional vector to each state-action pair. Then, the action values of a state-action pair <math>(s,a)</math> are obtained by linearly combining the components of <math>\phi(s,a)</math> with some ''weights'' <math>\theta</math>:

:<math>Q(s,a) = \sum_{i=1}^d \theta_i \phi_i(s,a).</math>

The algorithms then adjust the weights, instead of adjusting the values associated with the individual state-action pairs. Methods based on ideas from [[nonparametric statistics]] (which can be seen to construct their own features) have been explored.

Value iteration can also be used as a starting point, giving rise to the [[Q-learning]] algorithm and its many variants.<ref>{{cite thesis
  | last = Watkins | first = Christopher J.C.H. | author-link = Christopher J.C.H. Watkins
  | degree= PhD
  | title= Learning from Delayed Rewards
  | year= 1989
  | publisher = King’s College, Cambridge, UK
  | url= http://www.cs.rhul.ac.uk/~chrisw/new_thesis.pdf}}</ref> Including Deep Q-learning methods when a neural network is used to represent Q, with various applications in stochastic search problems.<ref name="MBK">{{Cite journal |title = Detection of Static and Mobile Targets by an Autonomous Agent with Deep Q-Learning Abilities | journal=Entropy | year=2022 | volume=24 | issue=8 | page=1168  | doi=10.3390/e24081168 | pmid=36010832 | pmc=9407070 | bibcode=2022Entrp..24.1168M | doi-access=free | last1=Matzliach | first1=Barouch | last2=Ben-Gal | first2=Irad | last3=Kagan | first3=Evgeny }}</ref>

The problem with using action-values is that they may need highly precise estimates of the competing action values that can be hard to obtain when the returns are noisy, though this problem is mitigated to some extent by temporal difference methods. Using the so-called compatible function approximation method compromises generality and efficiency.

=== Direct policy search ===
An alternative method is to search directly in (some subset of) the policy space, in which case the problem becomes a case of [[stochastic optimization]]. The two approaches available are gradient-based and gradient-free methods.

[[Gradient]]-based methods (''policy gradient methods'') start with a mapping from a finite-dimensional (parameter) space to the space of policies: given the parameter vector <math>\theta</math>, let <math>\pi_\theta</math> denote the policy associated to <math>\theta</math>. Defining the performance function by <math>\rho(\theta) = \rho^{\pi_\theta}</math> under mild conditions this function will be differentiable as a function of the parameter vector <math>\theta</math>. If the gradient of <math>\rho</math> was known, one could use [[gradient descent|gradient ascent]]. Since an analytic expression for the gradient is not available, only a noisy estimate is available. Such an estimate can be constructed in many ways, giving rise to algorithms such as Williams's REINFORCE method<ref>{{cite conference
  | last = Williams | first = Ronald J. | author-link = Ronald J. Williams
  | title = A class of gradient-estimating algorithms for reinforcement learning in neural networks
  | book-title = Proceedings of the IEEE First International Conference on Neural Networks
  | year = 1987| citeseerx = 10.1.1.129.8871 }}</ref> (which is known as the likelihood ratio method in the [[simulation-based optimization]] literature).<ref>{{cite conference |last1=Peters |first1=Jan |author-link1=Jan Peters (computer scientist) |last2=Vijayakumar |first2=Sethu |author-link2=Sethu Vijayakumar |last3=Schaal |first3=Stefan |author-link3=Stefan Schaal |title=Reinforcement Learning for Humanoid Robotics |conference=IEEE-RAS International Conference on Humanoid Robots | year = 2003 | url = http://www-clmc.usc.edu/publications/p/peters-ICHR2003.pdf |url-status=dead |archive-url=http://web.archive.org/web/20130512223911/http://www-clmc.usc.edu/publications/p/peters-ICHR2003.pdf |archive-date=2013-05-12}}</ref>

A large class of methods avoids relying on gradient information. These include [[simulated annealing]], [[cross-entropy method|cross-entropy search]] or methods of [[evolutionary computation]]. Many gradient-free methods can achieve (in theory and in the limit) a global optimum.

Policy search methods may converge slowly given noisy data. For example, this happens in episodic problems when the trajectories are long and the variance of the returns is large. Value-function based methods that rely on temporal differences might help in this case. In recent years, ''actor–critic methods'' have been proposed and performed well on various problems.<ref>{{Cite web|url=https://medium.com/emergent-future/simple-reinforcement-learning-with-tensorflow-part-8-asynchronous-actor-critic-agents-a3c-c88f72a5e9f2|title=Simple Reinforcement Learning with Tensorflow Part 8: Asynchronous Actor-Critic Agents (A3C)|last=Juliani|first=Arthur|date=2016-12-17|website=Medium|access-date=2018-02-22}}</ref>

Policy search methods have been used in the [[robotics]] context.<ref>{{Cite book|title=A Survey on Policy Search for Robotics|last1=Deisenroth|first1=Marc Peter|last2=Neumann|first2=Gerhard|last3=Peters|first3=Jan|publisher=NOW Publishers|year=2013|series=Foundations and Trends in Robotics|volume=2|issue=1–2|pages=1–142 |author-link1=Marc Peter Deisenroth|author-link2=Gerhard Neumann|author-link3=Jan Peters (computer scientist)|hdl=10044/1/12051|doi=10.1561/2300000021|url=http://eprints.lincoln.ac.uk/28029/1/PolicySearchReview.pdf}}</ref> Many policy search methods may get stuck in local optima (as they are based on [[Local search (optimization)|local search]]).

=== Model-based algorithms ===
Finally, all of the above methods can be combined with algorithms that first learn a model of the [[Markov decision process]], the probability of each next state given an action taken from an existing state. For instance, the Dyna algorithm learns a model from experience, and uses that to provide more modelled transitions for a value function, in addition to the real transitions.<ref>{{Cite conference |last1=Sutton |first1=Richard| title=Integrated Architectures for Learning, Planning and Reacting based on Dynamic Programming |year=1990 |book-title=Machine Learning: Proceedings of the Seventh International Workshop}}</ref> Such methods can sometimes be extended to use of non-parametric models, such as when the transitions are simply stored and "replayed" to the learning algorithm.<ref>{{Cite conference | first1 = Long-Ji | last1 = Lin | title = Self-improving reactive agents based on reinforcement learning, planning and teaching | book-title = Machine Learning volume 8 | year = 1992 | doi = 10.1007/BF00992699 |url=https://link.springer.com/content/pdf/10.1007/BF00992699.pdf}}</ref>

Model-based methods can be more computationally intensive than model-free approaches, and their utility can be limited by the extent to which the Markov decision process can be learnt.<ref>{{Citation |last=Zou |first=Lan |title=Chapter 7 - Meta-reinforcement learning |date=2023-01-01 |url=https://www.sciencedirect.com/science/article/pii/B9780323899314000110 |work=Meta-Learning |pages=267–297 |editor-last=Zou |editor-first=Lan |access-date=2023-11-08 |publisher=Academic Press |doi=10.1016/b978-0-323-89931-4.00011-0 |isbn=978-0-323-89931-4}}</ref>

There are other ways to use models than to update a value function.<ref>{{Cite conference
    | last1 = van Hasselt | first1 = Hado
    | last2 = Hessel | first2 = Matteo
    | last3 = Aslanides | first3 = John
    | title = When to use parametric models in reinforcement learning?
    | year = 2019
    | book-title = Advances in Neural Information Processing Systems 32
    | url = https://proceedings.neurips.cc/paper/2019/file/1b742ae215adf18b75449c6e272fd92d-Paper.pdf
}}</ref> For instance, in [[model predictive control]] the model is used to update the behavior directly.

== Theory ==
Both the asymptotic and finite-sample behaviors of most algorithms are well understood. Algorithms with provably good online performance (addressing the exploration issue) are known.

Efficient exploration of Markov decision processes is given in  Burnetas and Katehakis (1997).<ref name="Optimal adaptive policies for Marko"/> Finite-time performance bounds have also appeared for many algorithms, but these bounds are expected to be rather loose and thus more work is needed to better understand the relative advantages and limitations.

For incremental algorithms, asymptotic convergence issues have been settled.{{Clarify|reason=What are the issues that have been settled?|date=January 2020}} Temporal-difference-based algorithms converge under a wider set of conditions than was previously possible (for example, when used with arbitrary, smooth function approximation).

== Research ==
{{More citations needed section|date=October 2022}}
Research topics include:
* actor-critic architecture<ref>{{Cite journal |last1=Grondman |first1=Ivo |last2=Vaandrager |first2=Maarten |last3=Busoniu |first3=Lucian |last4=Babuska |first4=Robert |last5=Schuitema |first5=Erik |date=2012-06-01 |title=Efficient Model Learning Methods for Actor–Critic Control |url=https://dl.acm.org/doi/10.1109/TSMCB.2011.2170565 |journal= IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics)|volume=42 |issue=3 |pages=591–602 |doi=10.1109/TSMCB.2011.2170565 |pmid=22156998 |issn=1083-4419}}</ref>
* actor-critic-scenery architecture <ref name="Li-2023"/>
* adaptive methods that work with fewer (or no) parameters under a large number of conditions
* bug detection in software projects<ref>{{Cite web |title=On the Use of Reinforcement Learning for Testing Game Mechanics : ACM - Computers in Entertainment |url=https://cie.acm.org/articles/use-reinforcements-learning-testing-game-mechanics/ |access-date=2018-11-27 |website=cie.acm.org |language=en}}</ref>
* continuous learning
* combinations with logic-based frameworks<ref>{{Cite journal|last1=Riveret|first1=Regis|last2=Gao|first2=Yang|date=2019|title=A probabilistic argumentation framework for reinforcement learning agents|journal=Autonomous Agents and Multi-Agent Systems|language=en|volume=33|issue=1–2|pages=216–274|doi=10.1007/s10458-019-09404-2|s2cid=71147890}}</ref>
* exploration in large Markov decision processes
* [[reinforcement learning from human feedback|human feedback]]<ref>{{cite arXiv |last1=Yamagata |first1=Taku |last2=McConville |first2=Ryan |last3=Santos-Rodriguez |first3=Raul |date=2021-11-16 |title=Reinforcement Learning with Feedback from Multiple Humans with Diverse Skills |class=cs.LG |eprint=2111.08596 }}</ref>
* interaction between implicit and explicit learning in skill acquisition
* [[Intrinsic motivation (artificial intelligence)|intrinsic motivation]] which differentiates information-seeking, curiosity-type behaviours from task-dependent goal-directed behaviours large-scale empirical evaluations
* large (or continuous) action spaces
* modular and hierarchical reinforcement learning<ref>{{Cite journal|last1=Kulkarni|first1=Tejas D.|last2=Narasimhan|first2=Karthik R.|last3=Saeedi|first3=Ardavan|last4=Tenenbaum|first4=Joshua B.|date=2016|title=Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation|url=http://dl.acm.org/citation.cfm?id=3157382.3157509|journal=Proceedings of the 30th International Conference on Neural Information Processing Systems|series=NIPS'16|location=USA|publisher=Curran Associates Inc.|pages=3682–3690|isbn=978-1-5108-3881-9|bibcode=2016arXiv160406057K|arxiv=1604.06057}}</ref>
* multiagent/distributed reinforcement learning is a topic of interest. Applications are expanding.<ref>{{Cite web |title=Reinforcement Learning / Successes of Reinforcement Learning |url=http://umichrl.pbworks.com/Successes-of-Reinforcement-Learning/ |access-date=2017-08-06 |website=umichrl.pbworks.com}}</ref>
* occupant-centric control
* optimization of computing resources<ref>{{Cite book |last1=Dey |first1=Somdip |last2=Singh |first2=Amit Kumar |last3=Wang |first3=Xiaohang |last4=McDonald-Maier |first4=Klaus |title=2020 Design, Automation & Test in Europe Conference & Exhibition (DATE) |chapter=User Interaction Aware Reinforcement Learning for Power and Thermal Efficiency of CPU-GPU Mobile MPSoCs |date=March 2020 |chapter-url=https://ieeexplore.ieee.org/document/9116294 |pages=1728–1733 |doi=10.23919/DATE48585.2020.9116294 |isbn=978-3-9819263-4-7 |s2cid=219858480|url=http://repository.essex.ac.uk/27546/1/User%20Interaction%20Aware%20Reinforcement%20Learning.pdf }}</ref><ref>{{Cite web |last=Quested |first=Tony |title=Smartphones get smarter with Essex innovation |work=Business Weekly |url=https://www.businessweekly.co.uk/news/academia-research/smartphones-get-smarter-essex-innovation |access-date=2021-06-17}}</ref><ref>{{Cite web |last=Williams |first=Rhiannon |date=2020-07-21 |title=Future smartphones 'will prolong their own battery life by monitoring owners' behaviour' |url=https://inews.co.uk/news/technology/future-smartphones-prolong-battery-life-monitoring-behaviour-558689 |access-date=2021-06-17 |website=[[i (British newspaper)|i]] |language=en}}</ref>
*[[Partially observable Markov decision process|partial information]] (e.g., using [[predictive state representation]])
* reward function based on maximising novel information<ref name="kaplan2004">{{cite book |last1=Kaplan |first1=F. |title=Embodied Artificial Intelligence |last2=Oudeyer |first2=P. |chapter=Maximizing Learning Progress: An Internal Reward System for Development |publisher=Springer |year=2004 |isbn=978-3-540-22484-6 |editor-last=Iida |editor-first=F. |series=Lecture Notes in Computer Science |volume=3139 |location=Berlin; Heidelberg |pages=259–270 |doi=10.1007/978-3-540-27833-7_19 |s2cid=9781221 |editor2-last=Pfeifer |editor2-first=R. |editor3-last=Steels |editor3-first=L. |editor4-last=Kuniyoshi |editor4-first=Y.}}</ref><ref name="klyubin2008">{{cite journal |last1=Klyubin |first1=A. |last2=Polani |first2=D. |last3=Nehaniv |first3=C. |year=2008 |title=Keep your options open: an information-based driving principle for sensorimotor systems |journal=PLOS ONE |volume=3 |issue=12 |pages=e4018 |bibcode=2008PLoSO...3.4018K |doi=10.1371/journal.pone.0004018 |pmc=2607028 |pmid=19107219 |doi-access=free}}</ref><ref name="barto2013">{{cite book |last=Barto |first=A. G. |url=https://people.cs.umass.edu/~barto/IMCleVer-chapter-totypeset2.pdf |title=Intrinsically Motivated Learning in Natural and Artificial Systems |publisher=Springer |year=2013 |location=Berlin; Heidelberg |pages=17–47 |chapter=Intrinsic motivation and reinforcement learning}}</ref>
* sample-based planning (e.g., based on [[Monte Carlo tree search]]).
*securities trading<ref>{{cite journal |last1=Dabérius |first1=Kevin |last2=Granat |first2=Elvin |last3=Karlsson |first3=Patrik |date=2020 |title=Deep Execution - Value and Policy Based Reinforcement Learning for Trading and Beating Market Benchmarks |ssrn=3374766 |journal=The Journal of Machine Learning in Finance |volume=1}}</ref>
* [[transfer learning]]<ref>{{Cite journal|last1=George Karimpanal|first1=Thommen|last2=Bouffanais|first2=Roland|date=2019|title=Self-organizing maps for storage and transfer of knowledge in reinforcement learning|journal=Adaptive Behavior|language=en|volume=27|issue=2|pages=111–126|doi=10.1177/1059712318818568|issn=1059-7123|arxiv=1811.08318|s2cid=53774629}}</ref>
* TD learning modeling [[dopamine]]-based learning in the brain. [[Dopaminergic]] projections from the [[substantia nigra]] to the [[basal ganglia]] function are the prediction error.
* value-function and policy search methods

== Comparison of key algorithms ==
{| class="wikitable sortable"
|-
! Algorithm !! Description !!Policy !! Action space !! State space !! Operator
|-
| [[Monte Carlo method|Monte Carlo]] || Every visit to Monte Carlo ||  Either || Discrete || Discrete || Sample-means of state-values or action-values
|-
| [[Temporal difference learning|TD learning]] || State–action–reward–state ||  Off-policy || Discrete || Discrete || State-value
|-
| [[Q-learning]] || State–action–reward–state ||  Off-policy || Discrete || Discrete || Action-value
|-
| [[State–action–reward–state–action|SARSA]] || State–action–reward–state–action  || On-policy || Discrete || Discrete || Action-value
|-
| [[Q-learning#Deep Q-learning|DQN]] || Deep Q Network  || Off-policy || Discrete || Continuous || Action-value
|-
| DDPG || Deep Deterministic Policy Gradient || Off-policy || Continuous || Continuous || Action-value
|-
| A3C || Asynchronous Advantage Actor-Critic Algorithm || On-policy || Discrete || Continuous || Advantage (=action-value - state-value)
|-
| TRPO || Trust Region Policy Optimization ||  On-policy || Continuous or Discrete || Continuous || Advantage
|-
| [[Proximal Policy Optimization|PPO]] || Proximal Policy Optimization ||  On-policy || Continuous or Discrete || Continuous || Advantage
|-
|TD3
|Twin Delayed Deep Deterministic Policy Gradient
|Off-policy
|Continuous
|Continuous
|Action-value
|-
|SAC
|Soft Actor-Critic
|Off-policy
|Continuous
|Continuous
|Advantage
|-
|[[Distributional Soft Actor Critic|DSAC]]<ref>{{cite journal|author1=J Duan |author2=Y Guan| author3=S Li| title= Distributional Soft Actor-Critic: Off-policy reinforcement learning for addressing value estimation errors| journal=  IEEE Transactions on Neural Networks and Learning Systems |volume=33 | issue=11 |year= 2021 |pages= 6584–6598 |doi=10.1109/TNNLS.2021.3082568 |pmid=34101599 |arxiv=2001.02811 |s2cid=211259373 |url= https://ieeexplore.ieee.org/document/9448360 }}</ref><ref>{{cite book|author1=Y Ren |author2=J Duan| author3=S Li|title=2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC) |chapter=Improving Generalization of Reinforcement Learning with Minimax Distributional Soft Actor-Critic |year= 2020 |pages=1–6 |doi=10.1109/ITSC45102.2020.9294300 |arxiv=2002.05502 |isbn=978-1-7281-4149-7 |s2cid=211096594 |chapter-url= https://ieeexplore.ieee.org/document/9294300 }}</ref><ref>{{cite journal |last1=Duan |first1=J |last2=Wang |first2=W |last3=Xiao |first3=L |date=2025 |title=Distributional Soft Actor-Critic with Three Refinements |journal=IEEE Transactions on Pattern Analysis and Machine Intelligence |pages=1–12 |doi=10.1109/TPAMI.2025.3537087 |arxiv=2310.05858 }}</ref> ||Distributional Soft Actor Critic ||Off-policy ||Continuous ||Continuous ||Action-value distribution
|}

=== Associative reinforcement learning ===
Associative reinforcement learning tasks combine facets of stochastic learning automata tasks and supervised learning pattern classification tasks. In associative reinforcement learning tasks, the learning system interacts in a closed loop with its environment.<ref>{{cite book |last1=Soucek |first1=Branko |title=Dynamic, Genetic and Chaotic Programming: The Sixth-Generation Computer Technology Series |date=6 May 1992 |publisher=John Wiley & Sons, Inc |isbn=0-471-55717-X |page=38}}</ref>

=== Deep reinforcement learning ===
This approach extends reinforcement learning by using a deep neural network and without explicitly designing the state space.<ref name="intro_deep_RL">{{cite journal |first= Vincent|display-authors=etal|last= Francois-Lavet |year=2018 |title= An Introduction to Deep Reinforcement Learning |journal=Foundations and Trends in Machine Learning|volume=11 |issue=3–4 |pages=219–354 |doi=10.1561/2200000071|arxiv= 1811.12560 |bibcode=2018arXiv181112560F|s2cid=54434537}}</ref> The work on learning ATARI games by Google [[DeepMind]] increased attention to [[deep reinforcement learning]] or [[end-to-end reinforcement learning]].<ref name="DQN2">{{cite journal |first= Volodymyr|display-authors=etal|last= Mnih |year=2015 |title= Human-level control through deep reinforcement learning |journal=Nature|volume=518 |issue=7540 |pages=529–533 |doi=10.1038/nature14236|pmid= 25719670 |bibcode=2015Natur.518..529M |s2cid=205242740}}</ref>

=== Adversarial deep reinforcement learning ===
Adversarial deep reinforcement learning is an active area of research in reinforcement learning focusing on vulnerabilities of learned policies. In this research area some studies initially showed that reinforcement learning policies are susceptible to imperceptible adversarial manipulations.<ref>{{cite journal |last1= Goodfellow|first1=Ian  |last2=Shlens |first2= Jonathan|last3=Szegedy|first3=Christian|title= Explaining and Harnessing Adversarial Examples |journal= International Conference on Learning Representations |date= 2015 |arxiv=1412.6572 }}</ref><ref>{{cite book |last1= Behzadan|first1=Vahid  |last2=Munir |first2= Arslan|title=Machine Learning and Data Mining in Pattern Recognition |chapter=Vulnerability of Deep Reinforcement Learning to Policy Induction Attacks |series=Lecture Notes in Computer Science |date= 2017 |volume=10358 |pages=262–275 |doi=10.1007/978-3-319-62416-7_19 |arxiv=1701.04143|isbn=978-3-319-62415-0 |s2cid=1562290 }}</ref><ref>{{Cite book |last1=Huang |first1=Sandy |last2=Papernot |first2=Nicolas |last3=Goodfellow |first3=Ian |last4=Duan |first4=Yan |last5=Abbeel |first5=Pieter |url=http://worldcat.org/oclc/1106256905 |title=Adversarial Attacks on Neural Network Policies |date=2017-02-07 |oclc=1106256905}}</ref> While some methods have been proposed to overcome these susceptibilities, in the most recent studies it has been shown that these proposed solutions are far from providing an accurate representation of current vulnerabilities of deep reinforcement learning policies.<ref>{{cite journal |last1=Korkmaz |first1=Ezgi |date=2022 |title=Deep Reinforcement Learning Policies Learn Shared Adversarial Features Across MDPs. |journal=Thirty-Sixth AAAI Conference on Artificial Intelligence (AAAI-22) |volume=36 |issue=7 |pages=7229–7238 |doi=10.1609/aaai.v36i7.20684 |arxiv=2112.09025|s2cid=245219157 |doi-access=free }}</ref>

=== Fuzzy reinforcement learning ===
By introducing [[Fuzzy control system|fuzzy inference]] in reinforcement learning,<ref>{{Cite book |last=Berenji |first=H.R. |title=Proceedings of 1994 IEEE 3rd International Fuzzy Systems Conference |chapter=Fuzzy Q-learning: A new approach for fuzzy dynamic programming |date=1994 |chapter-url=https://ieeexplore.ieee.org/document/343737 |location=Orlando, FL, USA |publisher=IEEE |pages=486–491 |doi=10.1109/FUZZY.1994.343737|isbn=0-7803-1896-X |s2cid=56694947 }}</ref> approximating the state-action value function with [[fuzzy rule]]s in continuous space becomes possible. The IF - THEN form of fuzzy rules make this approach suitable for expressing the results in a form close to natural language. Extending FRL with Fuzzy Rule Interpolation<ref>{{Cite book |last=Vincze |first=David |title=2017 IEEE 15th International Symposium on Applied Machine Intelligence and Informatics (SAMI) |date=2017 |chapter=Fuzzy rule interpolation and reinforcement learning |chapter-url=http://users.iit.uni-miskolc.hu/~vinczed/research/vinczed_sami2017_author_draft.pdf |publisher=IEEE |pages=173–178 |doi=10.1109/SAMI.2017.7880298|isbn=978-1-5090-5655-2 |s2cid=17590120 }}</ref> allows the use of reduced size sparse fuzzy rule-bases to emphasize cardinal rules (most important state-action values).

=== Inverse reinforcement learning ===
In inverse reinforcement learning (IRL), no reward function is given. Instead, the reward function is inferred given an observed behavior from an expert. The idea is to mimic observed behavior, which is often optimal or close to optimal.<ref>{{cite book |last1=Ng |first1=A. Y. |last2=Russell |first2=S. J. |year=2000 |chapter=Algorithms for Inverse Reinforcement Learning |title=Proceeding ICML '00 Proceedings of the Seventeenth International Conference on Machine Learning |pages=663–670 |isbn=1-55860-707-2 |chapter-url=https://ai.stanford.edu/~ang/papers/icml00-irl.pdf }}</ref> One popular IRL paradigm is named maximum entropy inverse reinforcement learning (MaxEnt IRL). <ref>{{Cite journal |last1=Ziebart |first1=Brian D. |last2=Maas |first2=Andrew |last3=Bagnell |first3=J. Andrew |last4=Dey |first4=Anind K. |date=2008-07-13 |title=Maximum entropy inverse reinforcement learning |url=https://dl.acm.org/doi/10.5555/1620270.1620297 |journal=Proceedings of the 23rd National Conference on Artificial Intelligence - Volume 3 |series=AAAI'08 |location=Chicago, Illinois |publisher=AAAI Press |pages=1433–1438 |isbn=978-1-57735-368-3 |s2cid=336219}}</ref> MaxEnt IRL estimates the parameters of a linear model of the reward function by maximizing the entropy of the probability distribution of observed trajectories subject to constraints related to matching expected feature counts. Recently it has been shown that MaxEnt IRL is a particular case of a more general framework named random utility inverse reinforcement learning (RU-IRL). <ref>{{Cite journal |last1=Pitombeira-Neto |first1=Anselmo R. |last2=Santos |first2=Helano P. |last3=Coelho da Silva |first3=Ticiana L. |last4=de Macedo |first4=José Antonio F. |date=March 2024 |title=Trajectory modeling via random utility inverse reinforcement learning |url=https://doi.org/10.1016/j.ins.2024.120128 |journal=Information Sciences |volume=660 |pages=120128 |doi=10.1016/j.ins.2024.120128 |issn=0020-0255 |s2cid=235187141|arxiv=2105.12092 }}</ref> RU-IRL is based on [[Random utility model|random utility theory]] and Markov decision processes. While prior IRL approaches assume that the apparent random behavior of an observed agent is due to it following a random policy, RU-IRL assumes that the observed agent follows a deterministic policy but randomness in observed behavior is due to the fact that an observer only has partial access to the features the observed agent uses in decision making. The utility function is modeled as a random variable to account for the ignorance of the observer regarding the features the observed agent actually considers in its utility function.

===Multi-objective reinforcement learning===
Multi-objective reinforcement learning (MORL) is a form of reinforcement learning concerned with conflicting alternatives. It is distinct from multi-objective optimization in that it is concerned with agents acting in environments.<ref>{{cite journal |vauthors=Hayes C, Radulescu R, Bargiacchi E, et al |date= 2022 |title=A practical guide to multi-objective reinforcement learning and planning|journal= Autonomous Agents and Multi-Agent Systems |volume= 36|doi= 10.1007/s10458-022-09552-y|s2cid= 254235920 |doi-access= free|arxiv= 2103.09568}},</ref><ref>{{cite book |title=Multiple Attribute Decision Making: Methods and Applications |edition=1st |first1=Gwo-Hshiung |last1=Tzeng |first2=Jih-Jeng |last2=Huang |date=2011 |publisher=CRC Press |isbn=9781439861578}}</ref>

=== Safe reinforcement learning ===
Safe reinforcement learning (SRL) can be defined as the process of learning policies that maximize the expectation of the return in problems in which it is important to ensure reasonable system performance and/or respect safety constraints during the learning and/or deployment processes.<ref>{{cite journal |last1=García |first1=Javier |last2=Fernández |first2=Fernando |title=A comprehensive survey on safe reinforcement learning |url=https://jmlr.org/papers/volume16/garcia15a/garcia15a.pdf |journal=The Journal of Machine Learning Research |date=1 January 2015 |volume=16 |issue=1 |pages=1437–1480 }}</ref> An alternative approach is risk-averse reinforcement learning, where instead of the ''expected'' return, a ''risk-measure'' of the return is optimized, such as the [[Expected shortfall|conditional value at risk]] (CVaR).<ref>{{Cite journal |last1=Dabney |first1=Will |last2=Ostrovski |first2=Georg |last3=Silver |first3=David |last4=Munos |first4=Remi |date=2018-07-03 |title=Implicit Quantile Networks for Distributional Reinforcement Learning |url=https://proceedings.mlr.press/v80/dabney18a.html |journal=Proceedings of the 35th International Conference on Machine Learning |language=en |publisher=PMLR |pages=1096–1105|arxiv=1806.06923 }}</ref> In addition to mitigating risk, the CVaR objective increases robustness to model uncertainties.<ref>{{Cite journal |last1=Chow |first1=Yinlam |last2=Tamar |first2=Aviv |last3=Mannor |first3=Shie |last4=Pavone |first4=Marco |date=2015 |title=Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approach |url=https://proceedings.neurips.cc/paper/2015/hash/64223ccf70bbb65a3a4aceac37e21016-Abstract.html |journal=Advances in Neural Information Processing Systems |publisher=Curran Associates, Inc. |volume=28|arxiv=1506.02188 }}</ref><ref>{{Cite web |title=Train Hard, Fight Easy: Robust Meta Reinforcement Learning |url=https://scholar.google.com/citations?view_op=view_citation&hl=en&user=LnwyFkkAAAAJ&citation_for_view=LnwyFkkAAAAJ:eQOLeE2rZwMC |access-date=2024-06-21 |website=scholar.google.com}}</ref> However, CVaR optimization in risk-averse RL requires special care, to prevent gradient bias<ref>{{Cite journal |last1=Tamar |first1=Aviv |last2=Glassner |first2=Yonatan |last3=Mannor |first3=Shie |date=2015-02-21 |title=Optimizing the CVaR via Sampling |url=https://ojs.aaai.org/index.php/AAAI/article/view/9561 |journal=Proceedings of the AAAI Conference on Artificial Intelligence |language=en |volume=29 |issue=1 |doi=10.1609/aaai.v29i1.9561 |issn=2374-3468|arxiv=1404.3862 }}</ref> and blindness to success.<ref>{{Cite journal |last1=Greenberg |first1=Ido |last2=Chow |first2=Yinlam |last3=Ghavamzadeh |first3=Mohammad |last4=Mannor |first4=Shie |date=2022-12-06 |title=Efficient Risk-Averse Reinforcement Learning |url=https://proceedings.neurips.cc/paper_files/paper/2022/hash/d2511dfb731fa336739782ba825cd98c-Abstract-Conference.html |journal=Advances in Neural Information Processing Systems |language=en |volume=35 |pages=32639–32652|arxiv=2205.05138 }}</ref>

=== Self-reinforcement learning ===
Self-reinforcement learning (or self-learning), is a learning paradigm which does not use the concept of immediate reward <math>R_a(s,s')</math> after transition from <math>s</math> to <math>s'</math> with action <math>a</math>. It does not use an external reinforcement, it only uses the agent internal self-reinforcement. The internal self-reinforcement is provided by mechanism of feelings and emotions. In the learning process emotions are backpropagated by a mechanism of secondary reinforcement. The learning equation does not include the immediate reward, it only includes the state evaluation. 

The self-reinforcement algorithm updates a memory matrix <math>W=||w(a,s)||</math> such that in each iteration executes the following machine learning routine:
# In situation <math>s</math> perform action <math>a</math>.
# Receive a consequence situation <math>s'</math>.
# Compute state evaluation <math>v(s')</math> of how good is to be in the consequence situation <math>s'</math>.
# Update crossbar memory <math>w'(a,s) = w(a,s) + v(s')</math>.

Initial conditions of the memory are received as input from the genetic environment. It is a system with only one input (situation), and only one output (action, or behavior). 

Self-reinforcement (self-learning) was introduced in 1982 along with a neural network capable of self-reinforcement learning, named Crossbar Adaptive Array (CAA).<ref> Bozinovski, S. (1982). "A self-learning system using secondary reinforcement". In Trappl, Robert (ed.). Cybernetics and Systems Research: Proceedings of the Sixth European Meeting on Cybernetics and Systems Research. North-Holland. pp. 397–402. ISBN 978-0-444-86488-8 </ref><ref> Bozinovski S. (1995) "Neuro genetic agents and structural theory of self-reinforcement learning systems". CMPSCI Technical Report 95-107, University of Massachusetts at Amherst [https://web.cs.umass.edu/publication/docs/1995/UM-CS-1995-107.pdf]</ref> The CAA computes, in a crossbar fashion, both decisions about actions and emotions (feelings) about consequence states. The system is driven by the interaction between cognition and emotion. <ref> Bozinovski, S. (2014) "Modeling mechanisms of cognition-emotion interaction in artificial neural networks, since 1981." Procedia Computer Science p. 255-263 </ref>

== Statistical comparison of reinforcement learning algorithms ==
Efficient comparison of RL algorithms is essential for research, deployment and monitoring of RL systems. To compare different algorithms on a given environment, an agent can be trained for each algorithm. Since the performance is sensitive to implementation details, all algorithms should be implemented as closely as possible to each other.<ref>{{Cite journal |last1=Engstrom |first1=Logan |last2=Ilyas |first2=Andrew |last3=Santurkar |first3=Shibani |last4=Tsipras |first4=Dimitris |last5=Janoos |first5=Firdaus |last6=Rudolph |first6=Larry |last7=Madry |first7=Aleksander |date=2019-09-25 |title=Implementation Matters in Deep RL: A Case Study on PPO and TRPO |url=https://openreview.net/forum?id=r1etN1rtPB |journal=ICLR |language=en}}</ref> After the training is finished, the agents can be run on a sample of test episodes, and their scores (returns) can be compared. Since episodes are typically assumed to be [[i.i.d]], standard statistical tools can be used for hypothesis testing, such as [[Student's t-test|T-test]] and [[permutation test]].<ref>{{Cite journal |last=Colas |first=Cédric |date=2019-03-06 |title=A Hitchhiker's Guide to Statistical Comparisons of Reinforcement Learning Algorithms |url=https://openreview.net/forum?id=ryx0N3IaIV |journal=International Conference on Learning Representations |arxiv=1904.06979 |language=en}}</ref> This requires to accumulate all the rewards within an episode into a single number - the episodic return. However, this causes a loss of information, as different time-steps are averaged together, possibly with different levels of noise. Whenever the noise level varies across the episode, the statistical power can be improved significantly, by weighting the rewards according to their estimated noise.<ref>{{Cite journal |last1=Greenberg |first1=Ido |last2=Mannor |first2=Shie |date=2021-07-01 |title=Detecting Rewards Deterioration in Episodic Reinforcement Learning |url=https://proceedings.mlr.press/v139/greenberg21a.html |journal=Proceedings of the 38th International Conference on Machine Learning |language=en |publisher=PMLR |pages=3842–3853|arxiv=2010.11660 }}</ref>

== See also ==
{{Div col|colwidth=20em}}
* [[Temporal difference learning]]
* [[Q-learning]]
* [[State–action–reward–state–action]] (SARSA)
* [[Reinforcement learning from human feedback]]
* [[Optimal control]]
* [[Error-driven learning]]
* [[Multi-agent reinforcement learning]]
* [[Apprenticeship learning]]
* [[Model-free (reinforcement learning)]]
* [[active learning (machine learning)]]
{{Div col end}}

== References ==
{{reflist}}

== Further reading ==
* {{cite journal |last1=Annaswamy |first1=Anuradha M. |title=Adaptive Control and Intersections with Reinforcement Learning |journal=Annual Review of Control, Robotics, and Autonomous Systems |date=3 May 2023 |volume=6 |issue=1 |pages=65–93 |doi=10.1146/annurev-control-062922-090153 |s2cid=255702873 |language=en |issn=2573-5144|doi-access=free }}
* {{cite journal|last1 = Auer|first1 = Peter|last2 = Jaksch|first2 = Thomas |last3 = Ortner|first3 = Ronald |year = 2010|title = Near-optimal regret bounds for reinforcement learning|url = http://jmlr.csail.mit.edu/papers/v11/jaksch10a.html|journal = Journal of Machine Learning Research|volume = 11|pages = 1563–1600 |author-link1 = Peter Auer}}
* {{cite book|url=http://www.mit.edu/~dimitrib/RLbook.html|last1=Bertsekas |first1=Dimitri P.|title= REINFORCEMENT LEARNING AND OPTIMAL CONTROL |date= 2023 |publisher=Athena Scientific |orig-date=2019 |isbn=978-1-886-52939-7|edition=1st}}
* {{cite book|url = http://www.dcsc.tudelft.nl/rlbook/|title = Reinforcement Learning and Dynamic Programming using Function Approximators|last1 = Busoniu|first1 = Lucian|last2 = Babuska|first2 = Robert|last3 = De Schutter|first3 = Bart|last4 = Ernst|first4 = Damien|publisher = Taylor & Francis CRC Press|year = 2010|isbn = 978-1-4398-2108-4|author-link3 = Bart De Schutter }}
* {{cite journal
  | doi = 10.1561/2200000071
  | last1 = François-Lavet | first1 = Vincent
  | last2 = Henderson | first2 = Peter
  | last3 = Islam | first3 = Riashat
  | last4 = Bellemare | first4 = Marc G.
  | last5 = Pineau | first5 = Joelle
  | title = An Introduction to Deep Reinforcement Learning
  | journal = Foundations and Trends in Machine Learning
  | volume = 11
  | issue = 3–4 | pages = 219–354
  | year = 2018
  | arxiv = 1811.12560 | bibcode = 2018arXiv181112560F | s2cid = 54434537 }}
* {{cite book|url=https://link.springer.com/book/10.1007/978-981-19-7784-8|title=Reinforcement Learning for Sequential Decision and Optimal Control|last1=Li |first1=Shengbo Eben |publisher= Springer Verlag, Singapore |year=2023 |doi=10.1007/978-981-19-7784-8 |isbn=978-9-811-97783-1 |edition=1st }}
* {{cite book |last=Powell |first=Warren |title=Approximate dynamic programming: solving the curses of dimensionality |year=2011 |publisher=Wiley-Interscience |isbn= |url=http://www.castlelab.princeton.edu/adp.htm |access-date=2010-09-08 |archive-date=2016-07-31 |archive-url=https://web.archive.org/web/20160731230325/http://castlelab.princeton.edu/adp.htm |url-status=dead }}
* {{cite journal
  | doi = 10.1007/BF00115009
  | last = Sutton
  | first = Richard S.
  | author-link = Richard S. Sutton
  | title = Learning to predict by the method of temporal differences
  | journal = Machine Learning
  | volume = 3
  | pages = 9–44
  | year = 1988
  | doi-access = free
  }}
* {{cite book|url=http://incompleteideas.net/sutton/book/the-book.html|title=Reinforcement Learning: An Introduction|last1=Sutton|first1=Richard S.|last2=Barto|first2=Andrew G.|publisher=MIT Press|year=2018 |orig-date=1998 |isbn=978-0-262-03924-6|edition=2nd |author-link1=Richard S. Sutton|author-link2=Andrew Barto}}
* {{cite conference|last1 = Szita|first1 = Istvan|last2 = Szepesvari|first2 = Csaba |year = 2010|title = Model-based Reinforcement Learning with Nearly Tight Exploration Complexity Bounds|url = http://www.icml2010.org/papers/546.pdf|publisher = Omnipress|pages = 1031–1038  |book-title = ICML 2010|url-status = dead|archive-url = https://web.archive.org/web/20100714095438/http://www.icml2010.org/papers/546.pdf|archive-date = 2010-07-14}}

== External links ==
* [https://mpatacchiola.github.io/blog/2016/12/09/dissecting-reinforcement-learning.html Dissecting Reinforcement Learning] Series of blog post on reinforcement learning with Python code
* [https://lilianweng.github.io/posts/2018-02-19-rl-overview/ A (Long) Peek into Reinforcement Learning]

{{Artificial intelligence navbox}}
{{Computer science}}

[[Category:Reinforcement learning| ]]
[[Category:Markov models]]
[[Category:Belief revision]]