Optimal Control Theory

The Simplest Problem of Optimal Control

We may state the simplest problem of optimal control as:

\[ \begin{align} & Max \ V=\int_0^T{f(t,y(t),u(t))dt} \\ & s.t. \ \frac{dy}{dt} = g(t,y(t),u(t)) \\ & \qquad y(0) = y_0, \ y(T) \ free \ (y_0, T \ given) \\ & \qquad t \in [0,T] \end{align} \tag{1} \]

The Maximum Principle

The most important result in optimal control theory-a first-order necessary condition-is known as the maximum principle.

Three types of variables were already presented in the problem statement(1): \(t\) (time), \(y\) (state), and \(u\) (control). It turns out that in the solution process, yet another type of variable will emerge. It is called the costate variable (or auxiliary variable), to be denoted by \(\mu\). As we shall see, a costate variable is akin to a Lagrange multiplier and, as such, it is in the nature of a valuation variable, measuring the shadow price of an associated state variable. Like \(y\) and \(u\), the variable \(\mu\) can take different values at different points of time. Thus the symbol \(\mu\) is really a short version of \(\mu(t)\).

The vehicle through which the costate variable gains entry into the optimal control problem is the Hamiltonian function , or simply the Hamiltonian, which figures very prominently in the solution process. Denoted by \(H\), the Hamiltonian is defined as:

\[ H(t,y,u,\mu) = f(t,y(t),u(t)) + \mu(t)g(t,y(t),u(t)) \tag{2} \]

the maximum principle involves two first-order differential equations in the state variable \(y\) and the costate variable \(\mu\). Besides, there is a requirement that the Hamiltonian \(H\) be maximized with respect to the control variable \(u\) at every point of time. For pedagogical effectiveness, we shall first state and discuss the conditions involved, before providing the rationale for the maximum principle.

For the problem in (1), and with the Hamiltonian defined in (2), the maximum principle conditions are

\[ \begin{align} & \underset{u}{Max} \ H(t,y,u,\mu) \ \bigl( \Leftrightarrow H(t,y,u^*,\mu) \ge H(t,y,u,\mu) \bigr) \\ & \frac{\partial H}{\partial y} = - \frac{d\mu}{dt} \\ & \frac{dy}{dt} = g(t,y,u) \\ & y(0) = y_0 \end{align} \tag{3} \]

It might appear on first thought that the requirement in (3) could have been more succinctly embodied in the first-order condition \(\frac{\partial H}{\partial u} = 0\) (properly supported by an appropriate second order condition). The truth, however, is that the requirement \(\underset{u}{Max} \ H\) is a much broader statement of the requirement. In Figure 1, we have drawn three curves, each indicating a possible plot of the Hamiltonian \(H\) against the control variable \(u\) at a specific point of time, for specific values of \(y\) and \(\mu\).

\[ \begin{align} \mathscr{L} & = \int_0^T{\Bigl( f(t,y(t),u(t)) - \mu(t) \bigl(\frac{dy}{dt} - g(t,y(t),u(t)) \bigr) \Bigr)dt} \\ & = \int_0^T{\bigl( H - \mu(t) \frac{dy}{dt} \bigr)dt} \\ & = \int_0^T{Hdt} - \int_0^T{\mu(t)dy} \\ & = \int_0^T{Hdt} - \mu(t) y(t)|_{0}^{T} + \int_0^T{y(t) \frac{d\mu}{dt}dt} \\ & = \int_0^T{\bigl( H + \frac{d\mu}{dt} y(t) \bigr)dt} - \mu(T) y(T) + \mu(0) y(0) \end{align} \tag{4} \]

\[ \begin{align} & y(t) = y^{*}(t) + \epsilon p(t) \\ & u(t) = u^{*}(t) + \epsilon q(t) \\ & T = T^{*} + \epsilon \Delta T \\ & y(T) = y^{*}(T) + \epsilon \Delta y(T) \end{align} \tag{5} \]

\[ \begin{align} & \frac{d\mathscr{L}}{d\epsilon} =0 \Rightarrow \\ & \int_0^{T(\epsilon)}{\Bigl( \frac{\partial H}{\partial y}p(t) + \frac{\partial H}{\partial u}q(t) + \frac{d\mu}{dt} p(t) \Big)dt} + \bigl( H + \frac{d\mu}{dt} y(t) \bigr)|_{t=T} \cdot \Delta T - \mu(T) \cdot \Delta y(T) - y(T) \frac{d\mu}{dT} \cdot \Delta T = 0 \Rightarrow \\ & \int_0^{T(\epsilon)}{\Bigl( \bigl( \frac{\partial H}{\partial y } + \frac{d\mu}{dt} \bigr)p(t) + \frac{\partial H}{\partial u}q(t) \Bigr)dt} + H|_{t=T} \cdot \Delta T - \mu(T) \cdot \Delta y(T) = 0 \end{align} \tag{6} \]

From the arbitrariness of \(p(t)\), \(q(t)\), \(\Delta T\), and \(\Delta y_T\), we can get

\[ \begin{align} & \frac{\partial H}{\partial y} + \frac{d\mu}{dt} = 0 \\ & \frac{\partial H}{\partial u} = 0 \\ & H|_{t=T} \cdot \Delta T - \mu(T) \cdot \Delta y(T) = 0 \end{align} \tag{7} \]

\[ \begin{align} & \frac{d^2\mathscr{L}}{d\epsilon^2} \le 0 \Rightarrow \\ & \end{align} \tag{8} \]

\[ \begin{align} & (1) \ optimal \ condition: \underset{u}{Max} \ H \\ & \qquad or \ \Bigl(\frac{\partial H}{\partial u} = 0 \ (interior \ solution) \ or \ \frac{du}{dt} = 0 \ (boundary \ solution) \Bigr)\\ & (2) \ costate \ equation: \frac{\partial H}{\partial y} = - \frac{d\mu}{dt} \\ & (3) \ transition \ equation: \frac{dy}{dt} = g(t,y,u) \\ & (4) \ initial \ condition: y(0) = y_0 \\ & (5) \ transversality \ condition: H|_{t=T} \cdot \Delta T -\mu(T) \cdot \Delta y(T) = 0 \\ & (6) \ second \ order \ condition: \frac{\partial^2 H}{\partial u^2} < 0 \end{align} \tag{8} \]

In economic applications of optimal control theory, the integrand function \(f\) often contains a discount factor \(e^{-\rho t}\). The optimal control problem is

\[ \begin{align} & Max \ V=\int_0^T{e^{-\rho t}f(t,y(t),u(t))dt} \\ & s.t.\ \frac{dy}{dt} = g(t,y(t),u(t)) \\ & \qquad y(0) = y_0, \ y(T) \ free \ (y_0, T \ given) \\ & \qquad t \in [0,T] \end{align} \tag{10} \]

\[ \begin{align} H(t,y,u,\mu) & = e^{-\rho t}f(t,y(t),u(t)) + \mu(t)g(t,y(t),u(t)) \\ & = e^{-\rho t} \bigl( f(t,y(t),u(t)) + e^{\rho t}\mu(t)g(t,y(t),u(t)) \bigr) \\ & = e^{-\rho t}\overline H \end{align} \tag{11} \]

\[ \overline H = f(t,y(t),u(t)) + e^{\rho t}\mu(t)g(t,y(t),u(t)) \\ \tag{12} \]

\[ \begin{align} & (1) \ optimal \ condition: \underset{u}{Max} \ \overline H, \\ & \qquad or \ \Bigl( \frac{\partial \overline H}{\partial u} = 0 \ (interior \ solution) \ or \ \frac{du}{dt} = 0 \ (boundary \ solution) \Bigr) \\ & (2) \ costate \ equation: \frac{\partial \overline H}{\partial y} = - e^{\rho t}\frac{d\mu}{dt} \\ & (3) \ transition \ equation: \frac{dy}{dt} = g(t,y,u) \\ & (4) \ initial \ condition: y(0) = y_0 \\ & (5) \ transversality \ condition: \overline H|_{t=T} \cdot \Delta T - e^{\rho T}\mu(T) \cdot \Delta y(T) = 0 \\ & (6) \ second \ order \ condition: \frac{\partial^2 \overline H}{\partial u^2} < 0 \end{align} \tag{13} \]

\[ \begin{align} & Max \ V=\int_0^T{f(t,y_1(t),\cdots,y_m(t),u_1(t),\cdots,u_n(t))dt} \\ & s.t. \ \frac{dy_1}{dt} = g_1(t,y_1(t),\cdots,y_m(t),u_1(t),\cdots,u_n(t)) \\ & \qquad \cdots \\ & \qquad \frac{dy_m}{dt} = g_m(t,y_1(t),\cdots,y_m(t),u_1(t),\cdots,u_n(t)) \\ & \qquad y_1(0) = y_{10},\cdots,y_m(0) = y_{m0} \\ & \qquad y_1(T),\cdots,y_m(T) \ free \\ & \qquad u_1(t) \in \mathscr{U}_1,\cdots,u_n(t) \in \mathscr{U}_n \\ & \qquad t \in [0,T] \ (y_{10},\cdots,y_{m0}, T \ given) \end{align} \tag{14} \]

\[ \begin{align} & Max \ V=\int_0^T{f(t,y_1(t),\cdots,y_m(t),u_1(t),\cdots,u_n(t))dt} \\ & s.t. \ \frac{dy_i}{dt} = g_i(t,y_1(t),\cdots,y_m(t),u_1(t),\cdots,u_n(t)) \\ & \qquad y_i(0) = y_{i0} \\ & \qquad y_i(T) \ free \\ & \qquad u_j(t) \in \mathscr{U}_j \\ & \qquad t \in [0,T] \ (y_{i0}, T \ given) \\ & \qquad (i=1,\cdots,m;j=1,\cdots,n) \end{align} \tag{15} \]

\[ \begin{align} & H(t,y_1,\cdots,y_m,u_1,\cdots,u_n,\mu_1,\cdots,\mu_m) \\ & = f(t,y_1(t),\cdots,y_m(t),u_1(t),\cdots,u_n(t)) + \sum_{i=1}^m \mu_i(t)g_i(t,y_1(t),\cdots,y_m(t),u_1(t),\cdots,u_n(t)) \end{align} \tag{16} \]

\[ \begin{align} & (1) \ optimal \ condition: \underset{u_j}{Max} \ H, \\ & \qquad or \ \Big( \frac{\partial H}{\partial u_j} = 0 \ (interior \ solution) \ or \ \frac{du_j}{dt} = 0 \ (boundary \ solution) \Big) \\ & (2) \ costate \ equation: \frac{\partial H}{\partial y_i} = - \frac{d\mu_i}{dt} \\ & (3) \ transition \ equation: \frac{dy_i}{dt} = g_i(t,y_1(t),\cdots,y_m(t),u_1(t),\cdots,u_n(t)) \\ & (4) \ initial \ condition: y_i(0) = y_{i0} \\ & (5) \ transversality \ condition: H|_{t=T} \cdot \Delta T - \sum_{i=1}^m \mu_i(T) \cdot \Delta y_i(T) = 0 \\ & (6) \ second \ order \ condition: \frac{\partial^2 H}{\partial u_j^2} < 0 \end{align} \tag{17} \]

\[ \begin{align} & Max \ V=\int_0^T{f(t,y(t),u_1(t),u_2(t))dt} \\ & s.t. \ \frac{dy}{dt} = g(t,y(t),u_1(t),u_2(t)) \\ & \qquad h(t,y(t),u_1(t),u_2(t)) = c \\ & \qquad y(0) = y_0, \ y(T) \ free \ (y_0, T \ given) \\ & \qquad t \in [0,T] \end{align} \tag{18} \]

\[ \begin{align} & L(t,y,u_1,u_2,\mu,\theta) \\ & = H(t,y,u_1,u_2,\mu) - \theta(t) (h(t,y(t),u_1(t),u_2(t)) - c) \\ & = f(t,y(t),u(t)) + \mu(t)g(t,y(t),u(t)) - \theta(t) (h(t,y(t),u_1(t),u_2(t)) - c) \end{align} \tag{19} \]

\[ \begin{align} & (1) \ \underset{u}{Max} \ L \\ & \qquad or \ \Bigl(\frac{\partial L}{\partial u_j} = 0 \ (interior \ solution) \ or \ \frac{du_j}{dt} = 0 \ (boundary \ solution) \Bigr)\\ & (2) \ \frac{\partial L}{\partial y} = - \frac{d\mu}{dt} \\ & (3) \ \frac{dy}{dt} = g(t,y,u_1,u_2) \\ & (4) \ h(t,y(t),u_1(t),u_2(t)) - c = 0 \\ & (5) \ y(0) = y_0 \\ & (6) \ H|_{t=T} \cdot \Delta T -\mu(T) \cdot \Delta y(T) = 0 \\ & (7) \ \frac{\partial^2 H}{\partial u_j^2} < 0 \end{align} \tag{9} \]

\[ \mathscr{X} \ \mathbb{X} \ \mathcal{X} \ \Biggl(\biggl(\Bigl(\bigl((x)\bigr)\Bigr)\biggr)\Biggr) \]