Including Integral Action

A standard LQR controller minimizes a cost function and provides state feedback, but it does not guarantee that the output $y(t)$ will converge to a desired constant reference $r_y$.

This leads to steady-state error.

By augmenting the system with a new state $z(t)$ that integrates the error between $y(t)$ and $r_y(t)$, and then feeding this into the controller, we can:

- Drive the integral of the output error to zero.

- Ensure that the system output tracks the reference exactly at steady state.

- Achieve zero steady-state error even in the presence of disturbances.

To put simply, imagine we included a state into our system that integrated the error. Then we include that integral in our LQR cost function so that the input penalises the error integral. However, our integral is non-linear and the LQR only works with linear systems - we must linearise.

What does it mean to linearise? It means choosing a point around which we decide to model our system. Intuitively, we would choose to linearise our system around the steady-state values, so that we can then describe any changes as “deviations” from the target value. This is not much different to how we normally linearise a system, where instead of writing $x_p()$ for perturbations, we write $\delta x(t)$ for deviations.

Our final system will look like the following:

$\frac{d}{dt} \begin{bmatrix} \delta x(t) \\ \delta z(t) \end{bmatrix} = \begin{bmatrix} A & 0 \\ -C & 0 \end{bmatrix} \begin{bmatrix} \delta x(t) \\ \delta z(t) \end{bmatrix} + \begin{bmatrix} B \\ 0 \end{bmatrix} \delta u(t),$

We finally feed our new states into an LQR. and minimise!

The following is a derivation for this system. We start with determining a few values before we jump into the linearisation.

We should go from this:

$\dot{x}(t)=Ax(t)+Bu(t)\quad,\quad y(t)=Cx(t)$

To this:

$\frac{d}{dt} \begin{bmatrix} \delta x(t) \\ \delta z(t) \end{bmatrix} = \begin{bmatrix} A & 0 \\ -C & 0 \end{bmatrix} \begin{bmatrix} \delta x(t) \\ \delta z(t) \end{bmatrix} + \begin{bmatrix} B \\ 0 \end{bmatrix} \delta u(t)$

$\delta y(t) = [C \quad 0] \begin{bmatrix} \delta x(t) \\ \delta z(t) \end{bmatrix}$

---

1. Standard LQR Feedback Law

We start with the standard state feedback law:

$u(t) = Kx(t) + r(t)$

- $u(t)$: control input vector.

- $x(t)$: state vector.

- $K$: state feedback gain matrix.

- $r(t)$: external input or reference signal.

In Linear Quadratic Gaussian (LQG), a similar structure is used but with state estimates.

---

2. State-Space System Without Noise

We simplify the system by assuming no noise and using a state-space model with $D = 0$, leading to:

$\dot{x}(t) = Ax(t) + Bu(t) \quad,\quad y(t) = Cx(t)$

- $A$, $B$, and $C$: system matrices.

- $y(t)$: output of the system.

---

3. Adding Integral Action

To ensure the output $y(t)$ tracks the desired output $r_y(t)$, we introduce an integral of the output error:

$z(t) = \int_0^t \left( r_y(\tau) - y(\tau) \right) d\tau$

- $z(t)$: integral of the error (used for integral action).

- $r_y(t)$: desired reference output.

- This leads to:

$\dot{z}(t) = r_y(t) - y(t) = r_y(t) - Cx(t)$

---

4. Steady-State Analysis (we need a few values before the linearisation)

For a constant reference $r_y$, steady state is reached when $\dot{x}(t) = 0$, hence:

$0 = A\bar{x} + B\bar{u}$

- $\bar{x}$, $\bar{u}$: steady-state values of $x$ and $u$.

$0 = A\bar{x} + B\bar{u} \Rightarrow \bar{u} = -B^{-1}A\bar{x}$ (this result will be useful at the end)

Also, the steady-state output becomes:

$\bar{y} = C\bar{x}$

At steady state, we also require $\dot{z}(t) = 0$ (since if not, $z(t)$ would diverge):

$0 = r_y - \bar{y} \Rightarrow r_y = C\bar{x}$

---

5. Deviation Variables (where we actually linearise)

We now linearize the system around the steady-state by defining deviation variables. Note how we’ve also linearised the input, which we will use to create a new feedback law:

$x(t) = \bar{x} + \delta x(t), \quad u(t) = \bar{u} + \delta u(t), \quad y(t) = \bar{y} + \delta y(t)$

Substituting these into the dynamics:

a. For the state equation:

$\dot{x}(t) = A(\bar{x} + \delta x(t)) + B(\bar{u} + \delta u(t))$

Expanding:

$= A\bar{x} + A\delta x(t) + B\bar{u} + B\delta u(t)$

Since $A\bar{x} + B\bar{u} = 0$ from steady-state,

$\dot{x}(t) = A\delta x(t) + B\delta u(t) \Rightarrow \dot{\delta x}(t) = A\delta x(t) + B\delta u(t)$

---

b. For the output (y - y_steady becomes y_deviation):

$y(t) = C(\bar{x} + \delta x(t)) = C\bar{x} + C\delta x(t) \Rightarrow \delta y(t) = C\delta x(t)$

---

c. For the integral state:

$\dot{z}(t) = r_y - Cx(t) = r_y - C(\bar{x} + \delta x(t)) = r_y - C\bar{x} - C\delta x(t)$

But recall that $r_y = C\bar{x} \Rightarrow \dot{z}(t) = -C\delta x(t)$

$\Rightarrow \dot{\delta z}(t) = -C\delta x(t)$

---

6. State-Space Model with Integral Action

Now, we combine the deviations into an augmented state vector:

$\delta \xi(t) = \begin{bmatrix} \delta x(t) \\ \delta z(t) \end{bmatrix}$

The dynamics are:

$\frac{d}{dt} \begin{bmatrix} \delta x(t) \\ \delta z(t) \end{bmatrix} = \begin{bmatrix} A & 0 \\ -C & 0 \end{bmatrix} \begin{bmatrix} \delta x(t) \\ \delta z(t) \end{bmatrix} + \begin{bmatrix} B \\ 0 \end{bmatrix} \delta u(t)$

Output equation:

$\delta y(t) = [C \quad 0] \begin{bmatrix} \delta x(t) \\ \delta z(t) \end{bmatrix}$

---

Define a state feedback law for this system as

$\delta u(t) = K \begin{bmatrix} \delta x(t) \\ \delta z(t) \end{bmatrix}$

The way to think about this, is let’s say you used an observer system to help formulate an input to the main system. Here instead, we use an integral system we’ve constructed to make the input (this is only part of it for now). We then use an LQR to choose our gain, just like in an LQG.

The $K$ can be found by solving the infinite time LQR that minimises

$J = \int_0^\infty \left\{ \begin{bmatrix} \delta x(t) \\ \delta z(t) \end{bmatrix}^\top \begin{bmatrix} Q_c & 0 \\ 0 & \eta I \end{bmatrix} \begin{bmatrix} \delta x(t) \\ \delta z(t) \end{bmatrix} + \delta u^\top(t) R_c \delta u(t) \right\} dt$

where $\eta \in \mathbb{R}$ is a positive scalar which adjusts the relative weighting that is applied to integral action in the controller.

The feedback gain matrix $K$ is partitioned as:

$K = \begin{bmatrix} K_P & K_I \end{bmatrix},$

where:

- $K_P$ is the proportional feedback gain acting on the state deviation $\delta x(t),$ and

- $K_I$ is the integral feedback gain acting on the integral of the error $\delta z(t).$

Defining the feedback law

The initial law is:

$u(t) = Kx(t) + r(t)$

This is pure proportional control (state feedback) plus a reference input.

With Integral Action

To include integral action, the controller is adapted as follows:

$\delta u(t) = K \begin{bmatrix} \delta x(t) \\ \delta z(t) \end{bmatrix},$

$u(t) = \bar{u} + \delta u(t) = \bar{u} + K_P \delta x(t) + K_I \delta z(t)$

Here’s how the terms break down:

• $\bar{u}$: The steady-state input, i.e., the value of the input required to keep the system at the desired steady-state operating point, $0 = A\bar{x} + B\bar{u} \Rightarrow \bar{u} = -B^{-1}A\bar{x}$

• $\delta x(t)$, $\delta z(t)$: Deviations from steady state for the state and the integrated error, respectively.

• $K_P$: The gain on the state deviation (like the original feedback).

$K_I$: The gain on the integrated error (the new integral term)

Interpretation of $\bar{u}$

• $\bar{u}$ is a constant determined from solving the steady-state equations of your system.

• It replaces the term $r(t)$ in the original law. Now, instead of having a time-varying reference term $r(t)$, you embed the reference through deviation variables and design around that.

The structure of the control law is a P+I controller:

- $K_P \delta x(t)$ is Proportional control on the state error.

- $K_I \delta z(t)$ is Integral control on the accumulated output error.

The proportional-plus-integral (P+I) controller is tuned by adjusting $\eta$, which controls the weight of the integral term $\delta z(t)$ in the cost function.

- A larger $\eta$ increases the emphasis on integral action, reducing steady-state error but potentially increasing control effort.

- A smaller $\eta$ reduces the emphasis on integral action, prioritizing proportional control and potentially faster system response.