Many time series process is expressed in difference equation. So, what is difference equation?
Let's have a look in the following difference equation:
\[ z_t - \phi_1 z_{t-1} - \phi_2 z_{t-2} = e_t \]
The difference equation itself is a time series process that the present value depends on its historical values. By using the lag operator \( L \), we can rewrite the difference equation by defining \( \phi(L)=1-\phi_1 L - \phi_2 L^2 - \cdots - \phi_p L^p \), while \( L y_t = y_{t-1} \).
Now, for difference euqation \( \phi(L)z_t = e_t \), we have the following to remember:
If \( z_{t}^{(1)} \) and \( z_{t}^{(2)} \) are the homogeneous solution (which mean \( e_t = 0 \)),
then \( (b_{1}z_{t}^{(1)}+b_2 z_{t}^{(2)}) \) is also a solution for arbitrary constant \( b_1 \) and \( b_2 \).
If \( z_{t}^{(P)} \) is the particular solution and \( z_{t}^{(H)} \) is the homogeneous solution,
then \( (z_{t}^{(P)}+ z_{t}^{(H)}) \) is the general solution.
If \( (1-L)^m z_{t}= 0 \),
then a solution is: \( z_t = b t^j \)
where \( b \) is an arbitrary constant and \( 0 \leqq j < m \).
The general solution is: \( z_t = \sum_{j=0}^{m-1}b_j t^j \)
If \( (1-RL)^m z_t = 0 \)
then a solution is: \( z_t = t^j R^t \)
The general solution is \( z_t = (\sum_{j=0}^{m-1}b_j t^j) R^t \)
Now, it's time to talk about the conditions for the time series difference equations to become stationarity.
Consider the following AR(2) time series process:
\[ z_t = \phi_1 z_{t-1} + \phi_2 z{t-2} + e_t \] \[ (1-\phi_1 L - \phi_2 L^2)z_t = e_t \]
If \( e_t \) is a random error with mean zero, we take the expectation for both side, the process becomes:
\[ (1-\phi_1 L - \phi_2 L^2)z_t = 0 \]
Which is a difference equation.
If \( z_t \) is not equal to zero, the first factor should be:
\[ (1-\phi_1 L-\phi_2 L^2) = 0 \] \[ (\phi_2 L^2 + \phi_1 L - 1) = 0 \]
The roots of the equation \( \phi(L) \) are:
\[ L = \frac{-\phi_1 \pm \sqrt{\phi_{1}^2 - 4 \phi_2}}{2\phi_2} \]
let the roots be \( r_1 \) and \( r_2 \), then: \[ (\phi_2 L^2 + \phi_1 L - 1) = (L-r_1)(L-r_2)=0 \]
Dividing both side by \( r_1 \) and \( r_2 \): \[ (\frac{L}{r_1}-1)(\frac{L}{r_2}-1)=0 \] \[ (1-\frac{L}{r_1})(1-\frac{L}{r_2})=0 \]
From lemma 4, if the difference equation expressed in the form of \( (1-RL)^m z_t = 0 \), then a solution is: \( z_t = t^j R^t \)
Now, let's think about stationarity, if \( R>1 \), then the series is explosive as \( z_t = t^j R^t \).
So the necessary condition for a stationary series is \( R<1 \), which means the inverse of the roots in the polynominal \( \phi(L) \) should be smaller than 1; put it into another way; the roots should be greater than 1.
Let two roots be \( r_1 \) and \( r_2 \).
If the roots are complex:
\[ r_1 = a + bi \] \[ r_2 = a - bi \]
We can express the complex number in polar form:
\[ a + bi = r(cos(\theta)+isin(\theta)) \] \[ a - bi = r(cos(\theta)-isin(\theta)) \]
Graphically,
If the roots are complex, from lemma 1 and 4:
\[ r_1 = a + bi = r(cos(\theta)+isin(\theta)) \] \[ r_2 = a - bi = r(cos(\theta)-isin(\theta)) \]
\[ z_t = b_{01}\frac{1}{r_1^t} + b_{02}\frac{1}{r_2^t} \]
Because \( (r(cos(\theta)-isin(\theta)))^k= r^k(cos(k\theta)-isin(k\theta)) \)
So, \( r \) must large than 1 in order to make the process stationary, so we often refers the stationary conditions is the roots of the equation \( \phi(L) \) must be lie outside the unit circle.
Now, consider the roots again: \[ L = \frac{-\phi_1 \pm \sqrt{\phi_{1}^2 - 4 \phi_2}}{2\phi_2} \]
as \( |r_1| \) and \( |r_2| \) should greater than 1.
\[ |\frac{1}{r_1} \times \frac{1}{r_2}| = |\phi_2| < 1 \]
and
\[ |\frac{1}{r_1} + \frac{1}{r_2}| = |\phi_1| < 2 \]
So the conditions for stationarity regardless of the roots are real or complex are:
\[ \frac{-\phi_1 + \sqrt{\phi_{1}^2 - 4 \phi_2}}{2\phi_2} < 1 \] \[ \sqrt{\phi_{1}^2 - 4 \phi_2} < 2 + \phi_1 \] \[ \phi_2 < 1 + \phi_1 \]
Also:
\[ \frac{-\phi_1 - \sqrt{\phi_{1}^2 - 4 \phi_2}}{2\phi_2} > -1 \] \[ -\sqrt{\phi_{1}^2 - 4 \phi_2} < \phi_1 - 2 \] \[ \phi_2 > 1 - \phi_1 \]
For real roots, \( \phi_{1}^2 - 4 \phi_2 > 0 \) For complex roots, \( \phi_{1}^2 - 4 \phi_2 < 0 \)
The following examples are extracted from Wei's textbook:
\[ z_t-2 z_{t-1}+z_{t-2}=0 \] \[ (1-2L+L^2)z_t = 0 \] \[ (1-L)^2 z_t = 0 \]
From lemma 3, the general solution is:
\[ z_t = b_0 + b_1 t \]
\[ z_t - 2z_{t-1}+1.5z_{t-2}-0.5z_{t-3}=0 \] \[ (1-2L+1.5L^2 -0.5L^3)z_t = 0 \] \[ (1-L+0.5L^2)(1-L)=0 \]
The roots of \( (1-L+0.5L^2)=0 \) are: \[ r_1 = 1 + 1i \] \[ r_2 = 1 - 1i \]
Inverse of the roots are:
\[ R = 0.5 \pm 0.5 i \]
\[ r = \sqrt(0.5^2 + 0.5^2) = \sqrt(0.5) \] \[ \theta = tan^{-1}(0.5/0.5) = \frac{\pi}{4} \]
The general solution is:
\[ (1-L+0.5L^2)(1-L)z_t=0 \]
From lemma 3 and lemma 4:
\[ z_t = b_0 + b_1 \sqrt(0.5)^t cos(\frac{\pi}{4}t) + b_1 \sqrt(0.5)^t sin(\frac{\pi}{4}t) \]
Using companion matrix is yet another convenience way to identify a non-stationary time series.
Consider the following AR(p) series:
\[ y_t = \phi_1 y_{t-1} + \phi_2 y_{t-2} + \cdots + \phi_p y_{t-p} + e_t \]
We can rewrite the above AR(p) series to AR(1) series by using the companion matrix \( F \).
Let \( z \) be the column vector containing the time series \( y_t \) with dimension \( p\times 1 \), \( F \) be the companion matrix with dimension \( p \times p \), and \( v \) be the column vector containing the shock \( e_t \) with dimension \( p \times 1 \).
\[ z = \left[ \begin{array}{c} y_{t} \\ y_{t-1} \\ \vdots \\ y_{t-(p-1)} \end{array} \right] \]
\[ F = \left[ \begin{array}{ccccc} \phi_1 & \phi_2 & \cdots & \phi_{p-1} & \phi_{p} \\ 1 & 0 & \cdots & 0 & 0 \\ 0 & 1 & \cdots & 0 & 0 \\ \vdots & \vdots & \vdots & \vdots & \vdots \\ 0 & 0 & \cdots & 1 & 0 \end{array} \right] \]
\[ v = \left[ \begin{array}{c} a_{t} \\ a_{t-1} \\ \vdots \\ a_{t-(p-1)} \end{array} \right] \]
The AR(P) process can be expressed as AR(1) by using the companion matrix \( F \):
\[ z_t = F z_{t-1} + v_t \]
\[ \left[ \begin{array}{c} y_{t} \\ y_{t-1} \\ \vdots \\ y_{t-(p-2)} \\ y_{t-(p-1)} \end{array} \right] = \left[ \begin{array}{ccccc} \phi_1 & \phi_2 & \cdots & \phi_{p-1} & \phi_{p} \\ 1 & 0 & \cdots & 0 & 0 \\ 0 & 1 & \cdots & 0 & 0 \\ \vdots & \vdots & \vdots & \vdots & \vdots \\ 0 & 0 & \cdots & 1 & 0 \end{array} \right] \left[ \begin{array}{c} y_{t-1} \\ y_{t-2} \\ \vdots \\ y_{t-(p-3)} \\ y_{t-(p-2)} \end{array} \right] + \left[ \begin{array}{c} a_{t} \\ a_{t-1} \\ \vdots \\ a_{t-(p-2)} \\ a_{t-(p-1)} \end{array} \right] \]
When we recursivly subsitute \( z_{t} \):
\[ z_t = F (F z_{t-2} + v_{t-1}) + v_t \] \[ z_t = F^2 z_{t-2} + F v_{t-1} + v_t \] \[ z_t = F^2 (F z_{t-3} + v_{t-2}) + F v_{t-1} + v_t \] \[ z_t = F^3 z_{t-3} + F^2 v_{t-2} + F v_{t-1} + v_t \] \[ z_t = F^t z_{0} + F^{t-1} v_{1} + \cdots + F v_{t-1} + v_t \] \[ z_t = F^{t+1} z_{-1} + F^t v_0 + F^{t-1} v_{1} + \cdots + F v_{t-1} + v_t \]
So, \( z_{t+j} \) can be expressed in:
\[ z_{t+j} = F^{j+1} z_{t-1} + F^j v_t + F^{j-1} v_{t+1} + \cdots + F v_{t+j-1} + v_{t+j} \]
Let's us now considering the 1st row of the above matrix
\[ y_{t+j} = f_{11}^{(j+1)} y_{t-1} + f_{12}^{(j+1)} y_{t-2} + f_{13}^{(j+1)} y_{t-3} + \cdots + f_{1p}^{(j+1)} y_{t-p} + f_{11}^{(j)} v_{t} + f_{11}^{(j-1)} v_{t+1} + \cdots + + f_{11}^{(1)} v_{t+j-1} + f_{11}^{(0)} v_{t+j} \]
where \( f_{ij}^(k) \) is the element that in the \( i \) row and \( j \) column in the matrix \( F^k \).
Now, consider the effect of a random shock \( v_t \) affect the time series in long term.
\[ \frac{d y_{t+j}}{d v_t} = f_{11}^{(j)} \]
If the shock is permanent (wont die out), the value \( f_{11}^{(j)} \) is larger than 1. So, for a stationary series, the value of the companion matrix \( f_{11}^{(j)} \) should be smaller than 1.
Calcuating the companion matrix \( F^j \) can be complex, but if the eigenvalue of a matrix \( F \) are distinct, we can decompose the matrix \( F \) to:
\[ F = T\Lambda T^{-1} \]
where
\( T \) is a nonsingluar matrix
\( \Lambda \) is (\( p \times p \)) matrix with eigenvalues of \( F \) be the principal diagonal \[ \Lambda = \left[ \begin{array}{llll}\lambda_1 & 0 & \cdots &0 \\ 0 & \lambda_2 & \dots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \cdots & \lambda_p \end{array} \right] \]
\[ F^2 = T \Lambda T^{-1} T \Lambda T^{-1} = T \Lambda^2 T^{-1} \]
\[ F^j = T \left[ \begin{array}{llll}\lambda_1^j & 0 & \cdots &0 \\ 0 & \lambda_2^j & \dots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \cdots & \lambda_p^j \end{array} \right] T^{-1} \]
Now, let's consider \( f_{11}^{(j)} \) again:
\[ f_{11}^{(j)} = [t_{11} t^{11}] \lambda_1^j + [t_{12} t^{21}] \lambda_2^j + \cdots + [t_{1p} t^{p1}] \lambda_p^j \]
\[ f_{11}^{(j)} = c_1 \lambda_1^j + c_2 \lambda_2^j + \cdots + c_p \lambda_p^j \]
Since \( T T^{-1} \) is an identity matrix, thus
\[ c_1 + c_2 + \cdots + c_p = 1 \]
The effect of a random shock \( v_j \) to the series is a weighted average of eigenvalues to the jth power:
\[ \frac{d y_{t+j}}{d v_t} = c_1 \lambda_1^j + c_2 \lambda_2^j + \cdots + c_p \lambda_p^j \]
So, the necessary condition for a time series to be stationary, all the eigenvalues \( \lambda_i \) must be smaller than one.
Now, we have two methods for checking whether a time series process is stationary or not. 1. All the roots in the equation \( \phi(L) \) should lie outside the unit circle. 2. The eigenvalues of its companion matrix \( F \) is smaller than one.
As finding the roots of a high degree polynomial equation can be tedious, checking the eigenvalues \( \lambda_i < 1 \) is an easier method to check stationary.