Optimal Control Notes

Optimal Control

Solution to Optimal Control Problem

The general form of the optimal control problem and its solution involves minimizing the cost function $J$ over a time horizon $T$. Given the system dynamics and initial state, the control inputs $u_t$ are optimized.

The cost function to be minimized is:

$\min<em>{u</em>0, …, u<em>{T-1}} \sum</em>{t=0}^{T-1} x<em>t^T Q x</em>t + u<em>t^T R u</em>t + x<em>T^T Q</em>f x_T$

Subject to the constraints:

$x<em>{t+1} = A x</em>t + B u_t$

$x_0$ is given.

The solution often takes the form of a linear state feedback:

$u<em>t = F</em>t x_t$

where $F<em>t$ is a gain matrix. This control policy makes the closed-loop system robust. The emergence of this gain $F</em>t$ is a key feature.

Open-Loop Alternative

Consider a case where $T = 2$ and the initial state $x_0$ is given.

$x<em>1 = A x</em>0 + B u_0$

$x<em>2 = A x</em>1 + B u<em>1 = A^2 x</em>0 + A B u<em>0 + B u</em>1$

This approach assumes that the system evolves exactly according to the nominal, noise-free dynamics. The cost function can be written as:

$J(u<em>0, u</em>1) = x<em>0^T Q x</em>0 + u<em>0^T R u</em>0 + x<em>1^T Q x</em>1 + u<em>1^T R u</em>1 + x<em>2^T Q</em>f x_2$

Expanding this, we have:

$J(u<em>0, u</em>1) = x<em>0^T Q x</em>0 + u<em>0^T R u</em>0 + (A x<em>0 + B u</em>0)^T Q (A x<em>0 + B u</em>0) + u<em>1^T R u</em>1 + (A^2 x<em>0 + A B u</em>0 + B u<em>1)^T Q</em>f (A^2 x<em>0 + A B u</em>0 + B u_1)$

In this formulation, all terms are constant except for $u<em>0$ and $u</em>1$.

$J(u<em>0, u</em>1)$ can be expressed in a quadratic form:

$J(u<em>0, u</em>1) = \begin{bmatrix} u<em>0 \ u</em>1 \end{bmatrix}^T H \begin{bmatrix} u<em>0 \ u</em>1 \end{bmatrix} + 2 \begin{bmatrix} x<em>0 \ u</em>0 \ u_1 \end{bmatrix}^T h + c$

Where:

$H = \begin{bmatrix} R + B^T Q B + B^T A^T Q<em>f A B & B^T A^T Q</em>f B \ B^T A^T Q<em>f B & R + B^T Q</em>f B \end{bmatrix}$

$h = \begin{bmatrix} B^T Q A x<em>0 + B^T A^T Q</em>f A^2 x<em>0 \ B^T Q</em>f A^2 x_0 \end{bmatrix}$

$c = x<em>0^T (Q + A^T Q A + A^T A^T Q</em>f A^2) x_0$

Thus, $J(u)$ is quadratic in $u$ and linear in $x_0$.

To minimize $J(u)$, we calculate the gradient and set it to zero:

$\nabla J(u) = 2 H u + 2 h = 0$

$u^* = -H^{-1} h$

This open-loop control lacks robustness because there is no feedback or dependence on $x$ or time $t$.

A MATLAB demo might visualize this optimization.

Beyond LQR

The general form of the optimal control problem is:

$\min<em>{u</em>0, …, u<em>{T-1}} \sum</em>{t=0}^{T-1} l(x<em>t, u</em>t, t)$

Subject to:

$x<em>{t+1} = f(x</em>t, u_t, t)$

$x_0$ is given.

In general, an analytic solution to this problem is unknown, so we no longer have the worry of a feedback emerging from the analytic solution.

We can still solve this as an open-loop optimization problem:

$x<em>1 = f(x</em>0, u_0, 0)$

$x<em>2 = f(f(x</em>0, u<em>0, 0), u</em>1, 1)$

$x<em>3 = f(f(f(x</em>0, u<em>0, 0), u</em>1, 1), u_2, 2)$

and so on. Plug these into the cost:

$J(u<em>0, …, u</em>{T-1}; x<em>0) = \sum</em>{t=0}^{T-1} l(x<em>t, u</em>t, t) = l(x<em>0, u</em>0, 0) + l(f(x<em>0, u</em>0, 0), u<em>1, 1) + … + l(f(…f(x</em>0, u<em>0, 0), …), u</em>{T-1}, T-1)$

This can be fed into a general-purpose optimization solver. Failure is possible.

Problems with the Open-Loop Approach

This open-loop approach has at least two problems:

1. Lack of Robustness: It is sensitive to disturbances and model inaccuracies.
2. High Computational Complexity: The complexity can be very high for large $T$.

Model Predictive Control (MPC)

One solution is Model Predictive Control (MPC).

MPC involves solving an optimal control problem over a moving window (a.k.a. Receding Horizon Control):

$\min<em>{u</em>{t|t}, …, u<em>{t+T</em>{win}-1 | t}} \sum<em>{k=0}^{T</em>{win}-1} l(x<em>{t+k|t}, u</em>{t+k|t})$

Subject to:

$x<em>{k+1} = f(x</em>k, u_k, k)$

$x_0$ is given.

Apply $u<em>{t|t}$ to the real system (not the model), measure $x</em>{t+1}$ (including noise), and repeat. The window recedes as new measurements are incorporated.

Even though MPC solves an open-loop problem for each window, it creates closed-loop (feedback) behavior. This imparts robustness to the system.

MPC vs. Open-Loop