View source for Optimal control

{{Short description|Mathematical way of attaining a desired output from a dynamic system}}
[[File:Optimal Control Luus.png|thumb|Optimal control problem benchmark (Luus) with an integral objective, inequality, and differential constraint]]

'''Optimal control theory''' is a branch of [[control theory]] that deals with finding a [[Control (optimal control theory)|control]] for a [[dynamical system]] over a period of time such that an [[objective function]] is optimized.<ref name=":0">{{Cite book|last=Ross|first=Isaac |title=A primer on Pontryagin's principle in optimal control|publisher=Collegiate Publishers|year=2015|isbn=978-0-9843571-0-9|location=San Francisco|oclc=625106088}}</ref> It has numerous applications in science, engineering and operations research. For example, the dynamical system might be a [[spacecraft]] with controls corresponding to rocket thrusters, and the objective might be to reach the [[Moon]] with minimum fuel expenditure.<ref>{{cite book |first=David G. |last=Luenberger |author-link=David Luenberger |title=Introduction to Dynamic Systems |url=https://archive.org/details/introductiontody00luen_582 |url-access=limited |location=New York |publisher=John Wiley & Sons |year=1979 |isbn=0-471-02594-1 |chapter=Optimal Control |pages=[https://archive.org/details/introductiontody00luen_582/page/n406 393]–435 }}</ref> Or the dynamical system could be a nation's [[economy]], with the objective to minimize [[unemployment]]; the controls in this case could be [[Fiscal policy|fiscal]] and [[monetary policy]].<ref>{{Cite book|author=Kamien, Morton I.| url=http://worldcat.org/oclc/869522905 |title=Dynamic Optimization: the Calculus of Variations and Optimal Control in Economics and Management|date=2013 |publisher=Dover Publications |isbn=978-1-306-39299-0|oclc=869522905}}</ref> A dynamical system may also be introduced to embed [[Operations research|operations research problems]] within the framework of optimal control theory.<ref>{{cite arXiv |last1=Ross|first1=I. M. |last2=Proulx|first2=R. J. |last3=Karpenko|first3=M. |date=2020-05-06 | title=An Optimal Control Theory for the Traveling Salesman Problem and Its Variants |class=math.OC|eprint=2005.03186}}</ref><ref>{{Cite journal|last1=Ross|first1=Isaac M.|last2=Karpenko|first2=Mark|last3=Proulx|first3=Ronald J.|date=2016-01-01|title=A Nonsmooth Calculus for Solving Some Graph-Theoretic Control Problems**This research was sponsored by the U.S. Navy.| journal=IFAC-PapersOnLine |series=10th IFAC Symposium on Nonlinear Control Systems NOLCOS 2016|language=en|volume=49| issue=18|pages=462–467| doi=10.1016/j.ifacol.2016.10.208| issn=2405-8963| doi-access=free}}</ref>

Optimal control is an extension of the [[calculus of variations]], and is a mathematical optimization method for deriving [[control theory|control policies]].<ref>{{cite journal |first=R. W. H. |last=Sargent |author-link=Roger W. H. Sargent |title=Optimal Control |journal=Journal of Computational and Applied Mathematics |volume=124 |issue=1–2 |year=2000 |pages=361–371 |doi=10.1016/S0377-0427(00)00418-0 |bibcode=2000JCoAM.124..361S |doi-access=free }}</ref> The method is largely due to the work of [[Lev Pontryagin]] and [[Richard Bellman]] in the 1950s, after contributions to calculus of variations by [[Edward J. McShane]].<ref>{{cite journal |first=A. E. |last=Bryson |author-link=Arthur E. Bryson |year=1996 |title=Optimal Control—1950 to 1985 |journal= IEEE Control Systems Magazine |volume=16 |issue=3 |pages=26–33 |doi=10.1109/37.506395 }}</ref> Optimal control can be seen as a [[control strategy]] in [[control theory]].<ref name=":0" />

==General method==
Optimal control deals with the problem of finding a control law for a given system such that a certain [[optimality criterion]] is achieved. A control problem includes a [[cost functional]] that is a [[function (mathematics)|function]] of state and control variables. An '''optimal control''' is a set of [[differential equation]]s describing the paths of the control variables that minimize the cost function. The optimal control can be derived using [[Pontryagin's maximum principle]] (a [[necessary condition]] also known as Pontryagin's minimum principle or simply Pontryagin's principle),<ref>{{cite book |first=I. M. |last=Ross |author-link=I. Michael Ross |year=2009 |title=A Primer on Pontryagin's Principle in Optimal Control |publisher=Collegiate Publishers |isbn=978-0-9843571-0-9 }}</ref> or by solving the [[Hamilton–Jacobi–Bellman equation]] (a [[sufficient condition]]).

We begin with a simple example. Consider a car traveling in a straight line on a hilly road. The question is, how should the driver press the accelerator pedal in order to ''minimize'' the total traveling time? In this example, the term ''control law'' refers specifically to the way in which the driver presses the accelerator and shifts the gears. The ''system'' consists of both the car and the road, and the ''optimality criterion'' is the minimization of the total traveling time. Control problems usually include ancillary [[Constraint (mathematics)|constraint]]s. For example, the amount of available fuel might be limited, the accelerator pedal cannot be pushed through the floor of the car, speed limits, etc.

A proper cost function will be a mathematical expression giving the traveling time as a function of the speed, geometrical considerations, and [[initial condition]]s of the system. [[constraint (mathematics)|Constraint]]s are often interchangeable with the cost function.

Another related optimal control problem may be to find the way to drive the car so as to minimize its fuel consumption, given that it must complete a given course in a time not exceeding some amount. Yet another related control problem may be to minimize the total monetary cost of completing the trip, given assumed monetary prices for time and fuel.

A more abstract framework goes as follows.<ref name=":0" />  Minimize the continuous-time cost functional
<math display="block">J[\textbf{x}(\cdot), \textbf{u}(\cdot), t_0, t_f] := E\,[\textbf{x}(t_0),t_0,\textbf{x}(t_f),t_f] + \int_{t_0}^{t_f} F\,[\textbf{x}(t),\textbf{u}(t),t] \,\mathrm dt</math>
subject to the first-order dynamic constraints (the '''state equation''')
<math display="block"> \dot{\textbf{x}}(t) = \textbf{f}\,[\,\textbf{x}(t), \textbf{u}(t), t],</math>
the algebraic ''path constraints''
<math display="block"> \textbf{h}\,[\textbf{x}(t),\textbf{u}(t),t] \leq \textbf{0},</math>
and the [[boundary condition|endpoint condition]]s
<math display="block">\textbf{e}[\textbf{x}(t_0),t_0,\textbf{x}(t_f),t_f] = 0</math>
where <math>\textbf{x}(t)</math> is the ''state'', <math>\textbf{u}(t)</math> is the ''control'', <math>t</math> is the independent variable (generally speaking, time), <math>t_0</math> is the initial time, and <math>t_f</math> is the terminal time.  The terms <math>E</math> and <math>F</math> are called the ''endpoint cost '' and the ''running cost'' respectively. In the calculus of variations, <math>E</math> and <math>F</math> are referred to as the Mayer term and the ''[[Lagrange multiplier|Lagrangian]]'', respectively.  Furthermore, it is noted that the path constraints are in general ''inequality'' constraints and thus may not be active (i.e., equal to zero) at the optimal solution.  It is also noted that the optimal control problem as stated above may have multiple solutions (i.e., the solution may not be unique).  Thus, it is most often the case that any solution <math>[\textbf{x}^*(t),\textbf{u}^*(t),t_0^*, t_f^*]</math> to the optimal control problem is ''locally minimizing''.

==Linear quadratic control==
A special case of the general nonlinear optimal control problem given in the previous section is the [[Linear-quadratic regulator|''linear quadratic'' (LQ) optimal control problem]].  The LQ problem is stated as follows.  Minimize the ''quadratic'' continuous-time cost functional
<math display="block">J=\tfrac{1}{2} \mathbf{x}^{\mathsf{T}}(t_f)\mathbf{S}_f\mathbf{x}(t_f) + \tfrac{1}{2} \int_{t_0}^{t_f} [\,\mathbf{x}^{\mathsf{T}}(t)\mathbf{Q}(t)\mathbf{x}(t) + \mathbf{u}^{\mathsf{T}}(t)\mathbf{R}(t) \mathbf{u}(t)]\, \mathrm dt</math>

Subject to the ''linear'' first-order dynamic constraints
<math display="block">\dot{\mathbf{x}}(t)= \mathbf{A}(t) \mathbf{x}(t) + \mathbf{B}(t) \mathbf{u}(t), </math>
and the initial condition
<math display="block"> \mathbf{x}(t_0) = \mathbf{x}_0</math>

A particular form of the LQ problem that arises in many control system problems is that of the ''linear quadratic regulator'' (LQR) where all of the matrices (i.e., <math>\mathbf{A}</math>, <math>\mathbf{B}</math>, <math>\mathbf{Q}</math>, and <math>\mathbf{R}</math>) are ''constant'', the initial time is arbitrarily set to zero, and the terminal time is taken in the limit <math>t_f\rightarrow\infty</math> (this last assumption is what is known as ''infinite horizon'').  The LQR problem is stated as follows.  Minimize the infinite horizon quadratic continuous-time cost functional
<math display="block">J= \tfrac{1}{2} \int_{0}^{\infty}[\mathbf{x}^{\mathsf{T}}(t)\mathbf{Q}\mathbf{x}(t) + \mathbf{u}^{\mathsf{T}}(t)\mathbf{R}\mathbf{u}(t)]\, \mathrm dt</math>

Subject to the ''linear time-invariant'' first-order dynamic constraints
<math display="block">\dot{\mathbf{x}}(t) = \mathbf{A} \mathbf{x}(t) + \mathbf{B} \mathbf{u}(t), </math>
and the initial condition
<math display="block"> \mathbf{x}(t_0) = \mathbf{x}_0</math>

In the finite-horizon case the matrices are restricted in that <math>\mathbf{Q}</math> and <math>\mathbf{R}</math> are positive semi-definite and positive definite, respectively.  In the infinite-horizon case, however, the [[matrix (mathematics)|matrices]] <math>\mathbf{Q}</math> and <math>\mathbf{R}</math> are not only positive-semidefinite and positive-definite, respectively, but are also ''constant''.  These additional restrictions on
<math>\mathbf{Q}</math> and <math>\mathbf{R}</math> in the infinite-horizon case are enforced to ensure that the cost functional remains positive.  Furthermore, in order to ensure that the cost function is ''bounded'', the additional restriction is imposed that the pair <math>(\mathbf{A},\mathbf{B})</math> is ''[[Controllability|controllable]]''.  Note that the LQ or LQR cost functional can be thought of physically as attempting to minimize the ''control energy'' (measured as a quadratic form).

The infinite horizon problem (i.e., LQR) may seem overly restrictive and essentially useless because it assumes that the operator is driving the system to zero-state and hence driving the output of the system to zero. This is indeed correct. However the problem of driving the output to a desired nonzero level can be solved ''after'' the zero output one is. In fact, it can be proved that this secondary LQR problem can be solved in a very straightforward manner.  It has been shown in classical optimal control theory that the LQ (or LQR) optimal control has the feedback form
<math display="block">\mathbf{u}(t) = -\mathbf{K}(t)\mathbf{x}(t)</math>
where <math>\mathbf{K}(t)</math> is a properly dimensioned matrix, given as
<math display="block">\mathbf{K}(t) = \mathbf{R}^{-1}\mathbf{B}^{\mathsf{T}}\mathbf{S}(t),</math>
and <math>\mathbf{S}(t)</math> is the solution of the differential [[Riccati equation]].  The differential Riccati equation is given as
<math display="block">\dot{\mathbf{S}}(t) = -\mathbf{S}(t)\mathbf{A}-\mathbf{A}^{\mathsf{T}} \mathbf{S}(t) +\mathbf{S}(t)\mathbf{B}\mathbf{R}^{-1}\mathbf{B}^{\mathsf{T}}\mathbf{S}(t) - \mathbf{Q}</math>

For the finite horizon LQ problem, the Riccati equation is integrated backward in time using the terminal boundary condition
<math display="block">\mathbf{S}(t_f) = \mathbf{S}_f</math>

For the infinite horizon LQR problem, the differential Riccati equation is replaced with the ''algebraic'' Riccati equation (ARE) given as
<math display="block">\mathbf{0} = -\mathbf{S}\mathbf{A}-\mathbf{A}^{\mathsf{T}}\mathbf{S}+\mathbf{S}\mathbf{B}\mathbf{R}^{-1}\mathbf{B}^{\mathsf{T}}\mathbf{S}-\mathbf{Q}</math>

Understanding that the ARE arises from infinite horizon problem, the matrices <math>\mathbf{A}</math>, <math>\mathbf{B}</math>, <math>\mathbf{Q}</math>, and <math>\mathbf{R}</math> are all ''constant''.  It is noted that there are in general multiple solutions to the algebraic Riccati equation and the ''positive definite'' (or positive semi-definite) solution is the one that is used to compute the feedback gain.  The LQ (LQR) problem was elegantly solved by [[Rudolf E. Kálmán]].<ref>Kalman, Rudolf. ''A new approach to linear filtering and prediction problems''. Transactions of the ASME, Journal of Basic Engineering, 82:34–45, 1960</ref>

==Numerical methods for optimal control==
Optimal control problems are generally nonlinear and therefore, generally do not have analytic solutions (e.g., like the linear-quadratic optimal control problem).  As a result, it is necessary to employ numerical methods to solve optimal control problems.  In the early years of optimal control ({{abbr|c.|circa}} 1950s to 1980s) the favored approach for solving optimal control problems was that of ''indirect methods''.  In an indirect method, the calculus of variations is employed to obtain the first-order optimality conditions.  These conditions result in a two-point (or, in the case of a complex problem, a multi-point) [[boundary-value problem]].  This boundary-value problem actually has a special structure because it arises from taking the derivative of a [[Hamiltonian (control theory)|Hamiltonian]].  Thus, the resulting [[dynamical system]] is a [[Hamiltonian system]] of the form<ref name=":0" />
<math display="block">\begin{align}
\dot{\textbf{x}} & = \frac{\partial H}{\partial\boldsymbol{\lambda}} \\[1.2ex]
\dot{\boldsymbol{\lambda}} & = -\frac{\partial H}{\partial\textbf{x}}
\end{align}</math>
where
<math display="block">H= F +\boldsymbol{\lambda}^{\mathsf{T}}\textbf{f}- \boldsymbol{\mu}^{\mathsf{T}}\textbf{h}</math>
is the ''augmented Hamiltonian'' and in an indirect method, the boundary-value problem is solved (using the appropriate boundary or ''transversality'' conditions).  The beauty of using an indirect method is that the state and adjoint (i.e., <math>\boldsymbol{\lambda}</math>) are solved for and the resulting solution is readily verified to be an extremal trajectory.  The disadvantage of indirect methods is that the boundary-value problem is often extremely difficult to solve (particularly for problems that span large time intervals or problems with interior point constraints).  A well-known software program that implements indirect methods is BNDSCO.<ref>Oberle, H. J. and Grimm, W., "BNDSCO-A Program for the Numerical Solution of Optimal Control Problems," Institute for Flight Systems Dynamics, DLR, Oberpfaffenhofen, 1989</ref>

The approach that has risen to prominence in numerical optimal control since the 1980s is that of so-called ''direct methods''.  In a direct method, the state or the control, or both, are approximated using an appropriate function approximation (e.g., polynomial approximation or piecewise constant parameterization).  Simultaneously, the cost functional is approximated as a ''cost function''.  Then, the coefficients of the function approximations are treated as optimization variables and the problem is "transcribed" to a nonlinear optimization problem of the form:

Minimize
<math display="block"> F(\mathbf{z})</math>
subject to the algebraic constraints
<math display="block"> \begin{align}
\mathbf{g}(\mathbf{z}) & = \mathbf{0} \\
\mathbf{h}(\mathbf{z}) & \leq \mathbf{0}
\end{align} </math>

Depending upon the type of direct method employed, the size of the nonlinear optimization problem can be quite small (e.g., as in a direct shooting or [[quasilinearization]] method), moderate (e.g. [[pseudospectral optimal control]]<ref name="ReviewPOC">{{cite journal |author-link=I. Michael Ross |first1=I. M. |last1=Ross |first2=M. |last2=Karpenko |title=A Review of Pseudospectral Optimal Control: From Theory to Flight |journal=Annual Reviews in Control |volume=36 |issue=2 |pages=182–197 |year=2012 |doi=10.1016/j.arcontrol.2012.09.002 |title-link=Pseudospectral optimal control }}</ref>) or may be quite large (e.g., a direct [[collocation method]]<ref>{{cite book |last=Betts |first=J. T. |title=Practical Methods for Optimal Control Using Nonlinear Programming |publisher=SIAM Press |location=Philadelphia, Pennsylvania |edition=2nd |year=2010 |isbn=978-0-89871-688-7 }}</ref>). In the latter case (i.e., a collocation method), the nonlinear optimization problem may be literally thousands to tens of thousands of variables and constraints. Given the size of many NLPs arising from a direct method, it may appear somewhat counter-intuitive that solving the nonlinear optimization problem is easier than solving the boundary-value problem. It is, however, the fact that the NLP is easier to solve than the boundary-value problem. The reason for the relative ease of computation, particularly of a direct collocation method, is that the NLP is ''sparse'' and many well-known software programs exist (e.g., [[SNOPT]]<ref>Gill, P. E., Murray, W. M., and Saunders, M. A., ''User's Manual for SNOPT Version 7: Software for Large-Scale Nonlinear Programming'', University of California, San Diego Report, 24 April 2007</ref>) to solve large sparse NLPs. As a result, the range of problems that can be solved via direct methods (particularly direct ''collocation methods'' which are very popular these days) is significantly larger than the range of problems that can be solved via indirect methods. In fact, direct methods have become so popular these days that many people have written elaborate software programs that employ these methods. In particular, many such programs include ''DIRCOL'',<ref>von Stryk, O., ''User's Guide for DIRCOL (version 2.1): A Direct Collocation Method for the Numerical Solution of Optimal Control Problems'', Fachgebiet Simulation und Systemoptimierung (SIM), Technische Universität Darmstadt (2000, Version of November 1999).</ref> SOCS,<ref>Betts, J.T. and Huffman, W. P., ''Sparse Optimal Control Software, SOCS'', Boeing Information and Support Services, Seattle, Washington, July 1997</ref> OTIS,<ref>{{cite journal |last1=Hargraves |first1=C. R. |last2=Paris |first2=S. W. |title=Direct Trajectory Optimization Using Nonlinear Programming and Collocation |journal=Journal of Guidance, Control, and Dynamics |volume=10 |issue=4 |year=1987 |pages=338–342 |doi=10.2514/3.20223 |bibcode=1987JGCD...10..338H }}</ref> GESOP/[[ASTOS]],<ref>Gath, P.F., Well, K.H., "Trajectory Optimization Using a Combination of Direct Multiple Shooting and Collocation", AIAA 2001–4047, AIAA Guidance, Navigation, and Control Conference, Montréal, Québec, Canada, 6–9 August 2001</ref> DITAN.<ref>Vasile M., Bernelli-Zazzera F., Fornasari N., Masarati P., "Design of Interplanetary and Lunar Missions Combining Low-Thrust and Gravity Assists", Final Report of the ESA/ESOC Study Contract No. 14126/00/D/CS, September 2002</ref> and PyGMO/PyKEP.<ref>Izzo, Dario. "PyGMO and PyKEP: open source tools for massively parallel optimization in astrodynamics (the case of interplanetary trajectory optimization)." Proceed. Fifth International Conf. Astrodynam. Tools and Techniques, ICATT. 2012.</ref> In recent years, due to the advent of the [[MATLAB]] programming language, optimal control software in MATLAB has become more common. Examples of academically developed MATLAB software tools implementing direct methods include ''RIOTS'',<ref>[http://www.schwartz-home.com/RIOTS/ RIOTS] {{Webarchive| url=https://web.archive.org/web/20110716014630/http://www.schwartz-home.com/RIOTS/ |date=16 July 2011 }}, based on {{cite thesis |type=Ph.D. |last=Schwartz |first=Adam |date=1996 |title=Theory and Implementation of Methods based on Runge–Kutta Integration for Solving Optimal Control Problems |publisher=University of California at Berkeley |oclc=35140322 }}</ref> ''[[DIDO (optimal control)|DIDO]]'',<ref>Ross, I. M., ''Enhancements to the DIDO Optimal Control Toolbox, arXiv 2020.'' https://arxiv.org/abs/2004.13112</ref> ''DIRECT'',<ref>Williams, P., ''User's Guide to DIRECT, Version 2.00,'' Melbourne, Australia, 2008</ref> FALCON.m,<ref>[http://www.falcon-m.com/ FALCON.m], described in Rieck, M., Bittner, M., Grüter, B., Diepolder, J., and Piprek, P., ''FALCON.m - User Guide'', Institute of Flight System Dynamics, Technical University of Munich, October 2019</ref> and ''GPOPS,''<ref>[http://gpops.sourceforge.net GPOPS] {{Webarchive|url=https://web.archive.org/web/20110724074641/http://gpops.sourceforge.net/ |date=24 July 2011 }}, described in Rao, A. V., Benson, D. A., Huntington, G. T., Francolin, C., Darby, C. L., and Patterson, M. A., ''User's Manual for GPOPS: A MATLAB Package for Dynamic Optimization Using the [[Gauss pseudospectral method|Gauss Pseudospectral Method]]'', University of Florida Report, August 2008.</ref> while an example of an industry developed MATLAB tool is ''[[PROPT]]''.<ref>Rutquist, P. and Edvall, M. M, ''PROPT – MATLAB Optimal Control Software," 1260 S.E. Bishop Blvd Ste E, Pullman, WA 99163, USA: Tomlab Optimization, Inc.</ref> These software tools have increased significantly the opportunity for people to explore complex optimal control problems both for academic research and industrial problems.<ref>I.M. Ross, [https://nps.edu/documents/103424443/116151573/Ross.pdf/2c85d1a1-ff5b-4f60-9700-2ee5e1f3f65f?t=1580766209000 Computational Optimal Control], 3rd Workshop in Computational Issues in Nonlinear Control, October 8th, 2019, Monterey, CA</ref> Finally, it is noted that general-purpose MATLAB optimization environments such as [[TOMLAB]] have made coding complex optimal control problems significantly easier than was previously possible in languages such as C and [[FORTRAN]].

==Discrete-time optimal control==
The examples thus far have shown [[continuous time]] systems and control solutions. In fact, as optimal control solutions are now often implemented [[Digital data|digital]]ly, contemporary control theory is now primarily concerned with [[discrete time]] systems and solutions.  The Theory of [[Consistent Approximations]]<ref>E. Polak, ''On the use of consistent approximations in the solution of semi-infinite optimization and optimal control problems'' Math. Prog. 62 pp. 385–415 (1993).</ref><ref>{{Cite journal|last=Ross|first=I M. |date=2005-12-01|title=A Roadmap for Optimal Control: The Right Way to Commute |url=http://dx.doi.org/10.1196/annals.1370.015 |journal=Annals of the New York Academy of Sciences|volume=1065 |issue=1 |pages=210–231 |doi=10.1196/annals.1370.015 |pmid=16510411 |bibcode=2005NYASA1065..210R |s2cid=7625851 |issn=0077-8923}}</ref> provides conditions under which solutions to a series of increasingly accurate discretized optimal control problem converge to the solution of the original, continuous-time problem.  Not all discretization methods have this property, even seemingly obvious ones.<ref>{{Cite journal|last1=Fahroo|first1=Fariba |last2=Ross|first2=I. Michael |date=September 2008|title=Convergence of the Costates Does Not Imply Convergence of the Control|url=http://dx.doi.org/10.2514/1.37331|journal=Journal of Guidance, Control, and Dynamics|volume=31|issue=5 |pages=1492–1497|doi=10.2514/1.37331 |bibcode=2008JGCD...31.1492F |s2cid=756939 |issn=0731-5090|hdl=10945/57005|hdl-access=free}}</ref>  For instance, using a variable step-size routine to integrate the problem's dynamic equations may generate a gradient which does not converge to zero (or point in the right direction) as the solution is approached.   The direct method ''[http://www.schwartz-home.com/RIOTS RIOTS]'' is based on the Theory of Consistent Approximation.

==Examples==
{{unreferenced section|date=April 2018}}
A common solution strategy in many optimal control problems is to solve for the costate (sometimes called the [[shadow price]]) <math>\lambda(t)</math>. The costate summarizes in one number the marginal value of expanding or contracting the state variable next turn. The marginal value is not only the gains accruing to it next turn but associated with the duration of the program. It is nice when <math>\lambda(t)</math> can be solved analytically, but usually, the most one can do is describe it sufficiently well that the intuition can grasp the character of the solution and an equation solver can solve numerically for the values.

Having obtained <math>\lambda(t)</math>, the turn-t optimal value for the control can usually be solved as a differential equation conditional on knowledge of <math>\lambda(t)</math>. Again it is infrequent, especially in continuous-time problems, that one obtains the value of the control or the state explicitly. Usually, the strategy is to solve for thresholds and regions that characterize the optimal control and use a numerical solver to isolate the actual choice values in time.

===Finite time===
{{confusing|section|reason=the law of evolution mentioned in the example is not mentioned in the article and is probably not the same as [[evolution]]|date=October 2018}}
Consider the problem of a mine owner who must decide at what rate to extract ore from their mine. They own rights to the ore from date <math>0</math> to date <math>T</math>. At date <math>0</math> there is <math>x_0</math> ore in the ground, and the time-dependent amount of ore <math>x(t)</math> left in the ground declines at the rate of <math>u(t)</math> that the mine owner extracts it. The mine owner extracts ore at cost <math>u(t)^2/x(t)</math> (the cost of extraction increasing with the square of the extraction speed and the inverse of the amount of ore left) and sells ore at a constant price <math>p</math>. Any ore left in the ground at time <math>T</math> cannot be sold and has no value (there is no "scrap value"). The owner chooses the rate of extraction varying with time <math>u(t)</math> to maximize profits over the period of ownership with no time discounting.

{{ordered list
| 1 = Discrete-time version
{{pb}}
The manager maximizes profit <math>\Pi</math>:
<math display="block">\Pi = \sum_{t=0}^{T-1} \left[ pu_t - \frac{u_t^2}{x_t} \right] </math>
subject to the law of motion for the state variable <math>x_t</math>
<math display="block">x_{t+1} - x_t = - u_t</math>

Form the Hamiltonian and differentiate:
<math display="block">\begin{align}
H &= pu_t - \frac{u_t^2}{x_t} - \lambda_{t+1} u_t \\
\frac{\partial H}{\partial u_t} &= p - \lambda_{t+1} - 2\frac{u_t}{x_t} = 0 \\
\lambda_{t+1} - \lambda_t &= -\frac{\partial H}{\partial x_t} = -\left( \frac{u_t}{x_t} \right)^2
\end{align}</math>

As the mine owner does not value the ore remaining at time <math>T</math>,
<math display="block">\lambda_T = 0</math>

Using the above equations, it is easy to solve for the <math>x_t</math> and <math>\lambda_t</math> series
<math display="block">\begin{align}
\lambda_t &= \lambda_{t+1} + \frac{\left(p-\lambda_{t+1}\right)^2}{4} \\
x_{t+1} &= x_t \frac{2 - p + \lambda_{t+1}}{2}
\end{align}</math>
and using the initial and turn-T conditions, the <math>x_t</math> series can be solved explicitly, giving <math>u_t</math>.

| 2 = Continuous-time version
{{pb}}
The manager maximizes profit <math>\Pi</math>:
<math display="block">\Pi = \int_0^T \left[ pu(t) - \frac{u(t)^2}{x(t)} \right] dt </math>
where the state variable <math>x(t)</math> evolves as follows:
<math display="block"> \dot x(t) = - u(t) </math>

Form the Hamiltonian and differentiate:
<math display="block">\begin{align}
H &= pu(t) - \frac{u(t)^2}{x(t)} - \lambda(t) u(t) \\
\frac{\partial H}{\partial u} &= p - \lambda(t) - 2\frac{u(t)}{x(t)} = 0 \\
\dot\lambda(t) &= -\frac{\partial H}{\partial x} = -\left( \frac{u(t)}{x(t)} \right)^2
\end{align}</math>

As the mine owner does not value the ore remaining at time <math>T</math>,
<math display="block">\lambda(T) = 0</math>

Using the above equations, it is easy to solve for the differential equations governing <math>u(t)</math> and <math>\lambda(t)</math>
<math display="block">\begin{align}
\dot\lambda(t) &= -\frac{(p-\lambda(t))^2}{4} \\
u(t) &= x(t) \frac{p- \lambda(t)}{2}
\end{align}</math>
and using the initial and turn-T conditions, the functions can be solved to yield
<math display="block">x(t) = \frac{\left(4-pt+pT\right)^2}{\left(4+pT\right)^2} x_0  </math>
}}

==See also==
{{colbegin|colwidth=22em}}
* [[Active inference]]
* [[Bellman equation]]
* [[Bellman pseudospectral method]]
* [[Brachistochrone]]
* [[DIDO (optimal control)|DIDO]]
* [[DNSS point]]
* [[Dynamic programming]]
* [[Gauss pseudospectral method]]
* [[Generalized filtering]]
* [[GPOPS-II]]
* [[CasADi]]
* [[JModelica.org]] (Modelica-based open source platform for dynamic optimization)
* [[Kalman filter]]
* [[Linear-quadratic regulator]]
* [[Model Predictive Control]]
* [[Overtaking criterion]]
* [[PID controller]]
* [[PROPT| PROPT (Optimal Control Software for MATLAB)]]
* [[Pseudospectral optimal control]]
* [[Pursuit-evasion]] games
* [[Sliding mode control]]
* [[SNOPT]]
* [[Stochastic control]]
* [[Trajectory optimization]]
{{colend}}

==References==
{{Reflist|30em}}

==Further reading==
* {{cite book |first=D. P. |last=Bertsekas |author-link=Dimitri Bertsekas |title=Dynamic Programming and Optimal Control |location=Belmont |publisher=Athena |year=1995 |isbn=1-886529-11-6 }}
* {{cite book |last1=Bryson |first1=A. E. |author-link=Arthur E. Bryson |last2=Ho |first2=Y.-C. |author-link2=Yu-Chi Ho |title=Applied Optimal Control: Optimization, Estimation and Control |edition=Revised |publisher=John Wiley and Sons |location=New York |year=1975 |isbn=0-470-11481-9 |url=https://books.google.com/books?id=P4TKxn7qW5kC }}
* {{cite book |first1=W. H. |last1=Fleming |author-link=Wendell Fleming |first2=R. W. |last2=Rishel |author-link2=Raymond Rishel |title=Deterministic and Stochastic Optimal Control |location=New York |publisher=Springer |year=1975 |isbn=0-387-90155-8 |url=https://books.google.com/books?id=qJDbBwAAQBAJ }}
* {{cite book |first1=M. I. |last1=Kamien |author-link=Morton Kamien |first2=N. L. |last2=Schwartz |author-link2=Nancy Schwartz |title=Dynamic Optimization: The Calculus of Variations and Optimal Control in Economics and Management |location=New York |publisher=Elsevier |edition=Second |year=1991 |isbn=0-444-01609-0 |url=https://books.google.com/books?id=0IoGUn8wjDQC }}
* {{cite book |last=Kirk |first=D. E. |author-link=Donald E. Kirk |year=1970 |title=Optimal Control Theory: An Introduction |location=Englewood Cliffs |publisher=Prentice-Hall |isbn=0-13-638098-0 |url=https://books.google.com/books?id=onuH0PnZwV4C }}

==External links==
*{{Scholarpedia|title=Optimal control|curator=Victor M. Becerra|urlname=Optimal_control|date=2008|access-date=31 December 2022}}
*[https://nps.edu/documents/103424443/116151573/Ross.pdf/2c85d1a1-ff5b-4f60-9700-2ee5e1f3f65f?t=1580766209000 Computational Optimal Control]
* Dr. Benoît CHACHUAT: [https://web.archive.org/web/20100716102304/http://lawww.epfl.ch/page4234.html Automatic Control Laboratory] – Nonlinear Programming, Calculus of Variations and Optimal Control.
* [http://www.mathworks.com/products/connections/product_detail/product_61633.html DIDO - MATLAB tool for optimal control]
* [https://gekko.readthedocs.io/en/latest/ GEKKO - Python package for optimal control] 
* [https://web.archive.org/web/20101031195713/http://www.astos.de/products/gesop GESOP – Graphical Environment for Simulation and OPtimization]
{{Use dmy dates|date=April 2020}}
* [http://gpops2.com/ GPOPS-II – General-Purpose MATLAB Optimal Control Software]
* [https://web.casadi.org/ CasADi –  Free and open source symbolic framework for optimal control]
* [http://tomdyn.com/ PROPT – MATLAB Optimal Control Software]
* [https://openocl.org/ OpenOCL – Open Optimal Control Library] {{Webarchive|url=https://web.archive.org/web/20190420200752/https://openocl.org/ |date=20 April 2019 }}
* Elmer G. Wiens: [https://web.archive.org/web/20100325135634/http://www.egwald.ca/optimalcontrol/index.php Optimal Control] – Applications of Optimal Control Theory Using the Pontryagin Maximum Principle with interactive models.
* [http://blog.sciencenet.cn/home.php?mod=space&uid=1565&do=blog&id=209522 On Optimal Control] by Yu-Chi Ho
* [https://www.youtube.com/watch?v=faQeCI1IgoQ Pseudospectral optimal control: Part 1]
* [https://www.youtube.com/watch?v=jRmJwQI_JZw Pseudospectral optimal control: Part 2]
* Lecture Recordings and Script by Prof. Moritz Diehl, University of Freiburg on [https://www.syscop.de/teaching/ss2020/numerical-optimal-control-online Numerical Optimal Control]

{{Authority control}}

{{DEFAULTSORT:Optimal Control}}
[[Category:Optimal control| ]]
[[Category:Mathematical optimization]]