We need to model/create/represent/mimic, a stationary TS
We start by building the simplest possible TS from our main building block, the WN.
The, we build bigger models
We study the conditions that make our TS, stationary
Why? analytically tractable/statistical inference of their estimators are standard.
We will denote our white noise \(\varepsilon_t\) for any \(t\). It is an iid random variable with zero mean and constant variance i.e. \(\varepsilon_t \sim WN(0,\sigma^2)\)
Thus, we get a white noise time series of size \(T\): \(\{\varepsilon_1,\varepsilon_2,...,\varepsilon_T\}\)
Naive example:
Imagine that you plan to draw a well-behaved dice 10 times, each draw is independent.
Now substract 3.5 to positively sign wins (above average) from the value of the dice.
This is a white noise time series of size 10 (a uniformly distributed one)
Moving average process (MA)
Let’s model/create/build a TS noted \(\{Y_t\}\) using our building block \(\{\varepsilon_t\}\). Is \(\{Y_t\}\)WN? Stationary?
Naive model
\(Y_t = \varepsilon_t\)
\(Y_t\) is obviously WN and stationary
Model 1
\(Y_t = a + \varepsilon_t\) where \(a\) is a fixed value (a constant, non random)
\(E(Y_t) = a\) ; so it is not white anymore.
Yet, \(V(Y_t) = V(\varepsilon) = \sigma^2\)
\(Y_t\) autocovariance (or covariance) is the same as \(\varepsilon_t\) so it should be independent across time \(t\): \(\gamma_Y(h) = \gamma_{\varepsilon}(h) = 0\)
Note that adding a constant to \(\varepsilon\) keeps the normality assumption. So \(Y_t \sim N(a,\sigma^2)\)
Is model 1 (\(Y_t\)) stationary ?
Indeed, its expectation is constant across \(t\), so is the variance and the (auto)covariance.
This is just noise around a mean \(a\):
Simulation
setobs 100 // Time periods scalar sigma2 = 2 // Sigma2scalar a = 5 // agen E = rnormal(0,sqrt(sigma2)) // our Gaussian WN, we assumed normalitygeny = a + Egen time = _nli * ** Plottsset timetsliney
\(Y_t = 5 + \varepsilon_t\) where \(\varepsilon_t \sim N(0,2)\), stationary process
\(Y_t\) is ii normally distributed (iid)
Model 2
Now let’s add one WN from the immediate past, most relevant past \(\varepsilon_{t-1}\) but weighted by \(\theta_1\) :
\[Y_t = a + \varepsilon_t + \theta_1 \varepsilon_{t-1}\]
Is it WN?
\(E(Y_t) = a\). If \(a = 0\) this could eventually be WN given that \(V(Y_t)= (1+\theta_1^2)\sigma^2\).
But, since \(Y_t\) and \(Y_{t-1}\) share \(\varepsilon_t\), they are correlated: \(\gamma(1)\) cannot be 0. Note however that \(\gamma(h) = 0\) for h>1.
So this cannot be WN even if \(a\) is 0
Is it stationary?
The mean and variance are fixed/constant (across t).
what about time dependence \(\gamma(h)\)? \[\gamma(1) = E(Y_t - a) (Y_{t-1} - a )\]
\(\gamma(1) = \theta_1\sigma^2\) this does not depend on \(t\)
Further covariances \(\gamma(h)\) for lags (\(h\)) larger than 1 must be 0, since \(Y_t\) and \(Y_{t-h}\) don’t share any WN \(\varepsilon_t\) term. Thus, the process is stationary since 0 is independent from \(t\).
As you notice, the (auto)covariance functions \(\gamma_Y(h)\) is key,
No covariance term \(\gamma(h)\) depends on \(t\). Given a constant mean, (weak) stationarity builds solely on \(\gamma(h)\). This gave rise to the term ‘covariance stationarity’.
Simulation
clearallsetobs 100 // Time periods scalar sigma2 = 2 // Sigma2scalar a = 5 // ascalar theta1 = 3gen E = rnormal(0,sqrt(sigma2)) // Gaussian WNgen time = _ntsset timegeny = . // Just to initialize our variable* Simulationdyngen{update y = a + E + theta1*l.E if time > 1}li * in 1/10** Plottsset timetsliney
Number of observations (_N) was 0, now 100.
Time variable: time, 1 to 100
Delta: 1 unit
(100 missing values generated)
+------------------------------+
| E time y |
|------------------------------|
1. | 2.942765 1 . |
2. | -.9301596 2 12.89813 |
3. | -.0054639 3 2.204057 |
4. | -.4752548 4 4.508354 |
5. | -3.196665 5 .3775709 |
|------------------------------|
6. | -.0644086 6 -4.654403 |
7. | .6207599 7 5.427534 |
8. | -1.469483 8 5.392797 |
9. | -.9865208 9 -.3949695 |
10. | 2.38448 10 4.424917 |
+------------------------------+
Time variable: time, 1 to 100
Delta: 1 unit
\(Y_t = 5 + \varepsilon_t + 3\varepsilon_{t-1}\) where \(\varepsilon_t \sim N(0,2)\), covariance stationary process
\(Y_t\) is not independently distributed, yet it is still normally distributed (sum of 2 normal WN)
Model 3
Now let’s add many WN from the past, all weighted by \(\theta_j\) coefficients :
Recall the definition of a moving average, it is a rolling window average with arbitrary weights.
The terms \(\theta_1 \varepsilon_{t-1} + \theta_2 \varepsilon_{t-2} + \theta_3 \varepsilon_{t-3} + ... +\theta_q \varepsilon_{t-q}\) are a \(\theta\)-weighted sum of past information (WM) over a q-sized rolling window, from \(t-1\) to \(t-q\). Thus, it is a (weigthed) moving average of past WN.
Exercise
Identify the \(\gamma(h)\) function for a MA(2) process: \(Y_t = a + \varepsilon_t + \theta_1\varepsilon_{t-1} + \theta_2 \varepsilon_{t-2}\)
Tip. See Hamilton, Ch.3
MA(q)
\(Y_t = a + \varepsilon_t + \sum_{j=1}^{q} \theta_j \varepsilon_{t-j}\)
A general finite (q) order moving average model shares the previous properties
Indeed, it has constant mean (\(a\)), and \(\gamma(h)\) does not depend on \(t\).
Thus, any MA(q) process is stationary
Exercise
Given \(\gamma(h)\), the (auto)covariance function of a MA(q) process, provide the definition of the autocorrelation function \(\rho(h)\)
Draw the autocorrelation function for a MA(3) where \(\theta_1 =3\) , \(\theta_2=-2\), \(\theta_3=0.1\) and \(\sigma^2 = 2\).
Autoregresions (AR)
Many time series in social sciences exhibit some inertia. A past behavior \(Y_{t-h}\) may affect today’s i.e. \(Y_{t}\). Many non-systematic environmental factors may also alter \(Y_{t}\). Simply put, while using our main building block \(\varepsilon_t\):
\(Y_t = a + \delta Y_{t-1} + \varepsilon_t\)
This is an AUTOregression or first-order autoregresive model - AR(1)
Since there is a constant \(a\) this will not be WN even if we cancel our newly introduced lagged term by setting \(\delta = 0\). So what matters is to indentify whether this is stationary.
To identify its expectation and \(\gamma(h)\) we need to rewrite it as a function of our building blocks \(\{\varepsilon_t\}\)
AR stationarity
We can forward or backward replace the lagged dependent variable to write it as a function of our WN process.
From a backward replacement of the stochastic series \(\{Y_1, Y_2,...,Y_t\}\) : \(Y_{t-1} = a + \delta Y_{t-2} + \varepsilon_{t-1}\) into \(Y_t\), replacing everything, even \(Y_1\) as a function of an initial condition \(Y_0\):
\(Y_t = a \sum_{j=0}^{t-1}\delta^j + \delta^t Y_0 + \sum_{j=0}^{t-1}\delta^j\varepsilon_{t-j}\)
Applying the expectation operator to identify the population mean : \(E (Y_t) = a \sum_{j=0}^{t-1}\delta^j + \delta^t Y_0 + \sum_{j=0}^{t-1}\delta^j\underbrace{E(\varepsilon_{t-j})}_0\)
This depends on \(t\) from the first and second term.
If our time series is long, there is hope. Assume \(t \rightarrow \infty\):
First term is the sum of geometric series. If \(|\delta| < 1\), then: \(\lim_{t\rightarrow \infty} a \sum_{j=0}^{t-1}\delta^{t} = a (\frac{1}{1-\delta})\)
Clearly, the second term will tend to zero if \(|\delta| < 1\) as \(t \rightarrow \infty\)
Hence, a long TS that follows a AR(1) process whose \(|\delta| < 1\), has constant mean (time invariant).
Short process or \(|\delta| >1\) are doomed non-stationary
Let’s introduce some intuition regarding stationarity by running a simulation
clearallsetobs 1000 // Long Time Seriesscalar sigma2 = 30 // Sigma2scalar a = 5 // ascalar y0 = 100 // initial condition in Y_tscalar delta_I = 0.7scalar delta_II = 1scalar delta_III = 1.3scalar lrm = a*(1/(1-delta_I)) // Long run mean Idi lrmgen E = rnormal(0,sqrt(sigma2)) // Gaussian WNgen time = _ntsset timegen yI = y0 in 1 //gen yII = y0 in 1 // Just to initialize our AR processgen yIII = y0 in 1 //* Simulationdyngen{update yI = a + E + delta_I*l.yI if time > 1update yII = a + E + delta_II*l.yII if time > 1update yIII = a + E + delta_III*l.yIII if time > 1}li * in 950/1000** Plottsset timetsline yI in 950/1000 // Long ts
From our formulas, \(|\theta| >= 1\) can not have a constant mean, not even with high \(t\). It makes things worst:
** Plottsset timetsline yII // everything
Time variable: time, 1 to 1000
Delta: 1 unit
The whole TS for \(\delta = 1\)
Clearly time dependent. It even seems to have a (deterministic) trend: \(Y_t = \beta_0 + \beta_1 t + \varepsilon_t\)
** Plottsset timetsline yIII // everything
Time variable: time, 1 to 1000
Delta: 1 unit
The whole TS for \(\delta > 1\)
AR(1) and MA relationship
If a long TS brings the two first terms of: \(Y_t = a \sum_{j=0}^{t-1}\delta^j + \delta^t Y_0 + \sum_{j=0}^{t-1}\delta^j\varepsilon_{t-j}\) to zero given \(|\delta|<1\), then what?
So given a long TS, an AR(1) \(\equiv\) MA(\(\infty\)) !
AR(1) Covariance function
Mean stationarity is only available given a long TS and \(\delta < 1\), so there is no point of studying the \(\gamma(h)\) function without these assumptions.
So instead of this solution for \(Y_t\): \(Y_t = a \sum_{j=0}^{t-1}\delta^j + \delta^t Y_0 + \sum_{j=0}^{t-1}\delta^j\varepsilon_{t-j}\)
Stationarity of a more general autoregressice process AR(p) could be cumbersome to check using the previous approach
\(Y_t = a + \sum_{j=1}^p \alpha_j Y_{t-j} + \varepsilon_t\)
An alternative way to check for stationarity (in mean and covariance) is:
\(\sum_j \alpha_j < 1\), this is a necessary (not sufficient condition)
\(\sum_j |\alpha_j| < 1\), the signs of the coefficients may cancel out from the previous expression, so this is a sufficient condition.
If \(\sum_j |\alpha_j| = 1\), there will not be stationarity. The implied difference equations solution will have a unit root solution.
The caveat of this approach is the dismissal of informative \(\gamma(h)\), but we will come back to this while studying the Box-Jenkins specification approach.
ARMA(p,q)
Stationarity of a more general autoregressive process ARMA(p,q) depends solely on the AR(p) process
