Statistical Foundations

Statistical Foundations

Inference procedures often critically rely on an estimate for \(\sigma^2\)

Statistical Foundations

In classical situations, we typically have an optimal estimator!

\[\hat{\sigma}^2 = \frac{1}{n-1} \sum^n_{i = 1}(x_i - \overline{x})^2\]

Johnson & Johnson Quarterly Earnings

What do you notice? What do you wonder?

Global Temperature Changes

Analyze Speech Patterns

New York Stock Exchange

El Nino and New Fish

Cardiovascular Mortality in LA

Variance Estimation in a General Setting

• Dependence structure

• Multivariate data

• Focus on positively correlated data

An Alternative CLT

Long Run Variance

\[ \begin{align*} \Omega=& \sum_{h = -\infty}^\infty \Gamma(h)\\ =& \sum_{h = -\infty}^\infty Cov(u_{t}, u_{t-h})\\ =& \sum_{h = -\infty}^\infty E\left [(u_t - \mu)(u_{t-h}-\mu)' \right]\\ \end{align*} \]

When do we have this type of data?

• Econometrics

• Time Series Models

• Spectral Analysis

• Markov Chain Monte Carlo

Example

So…what is the problem?

\[ \begin{align*} \hat{\Omega}^{(N)}_T =& \sum_{h = -\infty}^\infty \hat{\Gamma}(h)\\ = & \sum_{h = -(T - 1)}^{T - 1} \frac{1}{T} \sum_{t=1}^{T - h} (u_t)(u_{t+h} )'\\ = & \sum_{h = -(T - 1)}^{T - 1} \frac{1}{T} \sum_{t=1}^{T - h} (u_t - \overline{u})(u_{t+h} - \overline{u})' \end{align*} \]

• Behaves erratically.

• Infinite variance.

• Lack of convergence to standard limiting distributions.

• May not be positive semi-definite.

Current Estimators ¹

\[ \begin{align*} \hat{\Omega}^{(SV)}_T =& \sum_{ h = -(T-1)}^{T-1} \kappa \left (\frac{h}{bT}\right ) \hat{\Gamma}(h) \\ =& \frac{1}{T}\sum_{h= -(T-1)}^{T-1} \kappa \left (\frac{h}{bT} \right )\sum_{t=1}^{T-h}(u_t - \overline{u})(u_{t+h} - \overline{u})'\\ =& \frac{1}{T} \sum_{t=1}^T \sum_{s=1}^T \kappa\left( \frac{t-s}{bT} \right) (u_t - \overline{u})(u_{s} - \overline{u})' \end{align*} \]

Current Estimators

Typical class kernels for the SV estimator \(\mathscr{K}_1:\)

• \(\kappa:\mathbb{R} \rightarrow [-1, 1]\)

• \(\kappa(x) = \kappa(-x)\)

• \(\kappa(0) =1\)

• \(c_2< \infty\)

• \(\int_0^\infty \kappa(x) x dx < \infty\)

• piece wise smooth

• \(\frac{1}{2 \pi} \int^\infty_{-\infty} \kappa(u) \exp(-iu\theta) du>0; \forall \theta>0\)

where \(c_i = \int^\infty_{-\infty} \kappa^i(u) du\)

Common Kernel Functions

Current Estimators: Bias and Variance

\[ \begin{align} \Omega = & \sum_{h = -\infty}^\infty \Gamma(h) & \text{}\\ \geq & \sum_{h=-(T-1)}^{T-1} \Gamma(h) & \text{Finite Sampling} \label{omega_bias:finite}\\ \approx & \sum_{h=-(T-1)}^{T-1} \sum_{t=1}^{T - h}\frac{1}{T-|h|}E[\hat{u}_t \hat{u}_{t-h}'] & \text{Estimation} \label{omega_bias:demeaning}\\ \geq& \sum_{h=-(T-1)}^{T-1} \sum_{t=1}^{T - h}\frac{1}{T}E[\hat{u}_t \hat{u}_{t-h}'] & \text{Autocov. Estimator}\label{omega_bias:autocov}\\ \geq & \sum_{h = -\left \lfloor bT\right \rfloor}^{\left \lfloor bT\right \rfloor}\sum_{t=1}^{\left \lfloor bT\right \rfloor - h}\frac{1}{T} E[\hat{u}_t, \hat{u}_{t-h}'] & \text{Bandwidth} \label{omega_bias:bandwidth}\\ \geq & \sum_{h=-\left \lfloor bT\right \rfloor}^{\left \lfloor bT\right \rfloor} \kappa \left(\frac{h}{bT} \right )E[\hat{u}_t \hat{u}_{t-h}'] & \text{ Kernel Function} \label{omega_bias:kernel} \end{align} \]

Current Estimators: Bias and Variability

Main Issues:

• Finite Sampling Bias

• Kernel Bias

• De-meaning Bias

We must further consider:

• Dimensionality Bias

• Biases are dependent on underlying correlations structure.

• Relationships between bias and variability.

Current Estimators

Current Estimators¹

• Tuning parameters: \(b\), kernel function.

  • \(b\) estimated by AMSE

  • Several standard kernels recommended.

• Positive semi-definite estimate.

• Negative bias.

• Consistent.

• Called Spectral Variance, Kernel, Newey-West and HAC estimators.

Lugsail Estimator

\[ \hat{\Omega}^{(L)}_T = \frac{1}{1-c} \hat{\Omega}^{(SV)}_{T,b} - \frac{c}{1-c}\hat{\Omega}^{(SV)}_{T,b/r } \]

Lugsail Estimator

Lugsail class of kernels for the SV estimator \(\mathscr{K}_2:\)

• \(\kappa: \mathbb{R} \rightarrow (-\infty, \infty)\)

• \(\kappa^* \in \mathscr{K}_1\)

• \(\kappa\left (x \right ) = \frac{1}{1-c} \kappa^* \left (x \right) - \frac{c}{1-c}\kappa^* \left ( xr \right)\)

• \(c \in [0, 1), r\geq 1\)

Observe \(\mathscr{K}_1 \subset \mathscr{K}_2\).

Picking Lugsail Parameters

where \(q\) is the characteristic component of the kernel function.

Picking Lugsail Parameters

Lugsail Estimator ¹

• Offset the negative bias.

• Tuning parameters: \(r\), \(c\), kernel function, \(b\)

  • \(r\) and \(c\) determined by amount of negative bias suspected

  • Use mother kernel \(\kappa^* \in \mathscr{K}_1\)

• Computationally efficient.

• Might not be positive semi-definite.

• Consistent.

• Higher variance.

• What about \(b\)?

Bias and Variance

Let \(\kappa \in \mathscr{K}_2\), and assume that \(bT \rightarrow \infty\) as \(T \rightarrow \infty\), but at a slower rate

\[\begin{align*} Bias\left (\hat{\Omega}^{(L)}_T\right ) = & \left ( \frac{1-cr^q}{1-c} \right ) g_q^*h_q(bT)^{-q} + o\left ((bT)^{-q} \right) \\ Var\left (vec \left (\hat{\Omega}^{(L)}_T\right )\right ) = & (I_{d^2} + K_{dd})(\Omega\otimes \Omega)c_2b + o(b) \end{align*}\]

• \(I_{d^2}\) is the \(d^2 \times d^2\) identity matrix

• \(K_{dd}\) is the \(d^2 \times d^2\) commutation matrix

• \(\otimes\) is the Kronecker product

• \(h_q = \sum_{j=-\infty}^\infty |j|^q \Gamma_j\), the Parzen generalized derivative

Loss Functions

• \(Bias^2(\hat{\Omega}_T) + Var(\hat{\Omega}_T)\)

• \(\tau |Bias(\hat{\Omega}_T)| + (1-\tau)Var(\hat{\Omega}_T)\)

• \(\tau |e_I(b)-\alpha| + (1-\tau)(\Delta_p^{max}(b))\)

• \(\tau|e_{I}(b)-\alpha| + (1-\tau)\left (\int \Delta_p(b) d\prod_{\delta}(\delta)\right )\)

• \(\tau(e_{I}(b)-\alpha)^2 + (1-\tau)(\Delta^{max}_p(b))^2\)

• \(\tau(e_{I}(b)) + (1-\tau)(e_{II}(b))\)

• \(\text{argmin}\left \{e_{II}(b)\right \} \text{ ; s.t. }e_I(b) \leq \tau\alpha\)

Fixed-\(b\) Asymptotic Theory

Fixed-\(b\) Asymptotic Theory

Fixed-\(b\) Probabilities¹

• Accounts for variability.

• Does not account for biases.

An Additional Asssumption

Approximate Finite Sample Bias

\[u_t = \rho u_{t-1}+ \epsilon_t\]

with \(\epsilon_t \overset{iid}{\sim} N(0, \Omega_\epsilon)\). Then

\[ \begin{align*} \sum_{|s|> \left \lfloor bT \right \rfloor} \Gamma(s) = & \sum_{s = \infty}^\infty \Gamma(s) - \sum_{s = -\left \lfloor bT \right \rfloor}^{\left \lfloor bT \right \rfloor} \Gamma(s)\\ = & 2 \sum_{s = \left \lfloor bT \right \rfloor + 1}^\infty \Gamma(s)\\ = & 2 \Gamma(0) \left( \sum_{s = \left \lfloor bT \right \rfloor + 1}^\infty \rho^s \right) \\ = & 2 \Gamma(0) \left (\frac{\rho}{1-\rho}\rho^{\left \lfloor bT \right \rfloor + 1}\right) \\ = & \Omega \frac{2\rho^2}{1+\rho}\rho^{\left \lfloor bT \right \rfloor } \end{align*} \]

Therefore \(c_b \approx \frac{2\rho^2}{1+\rho}\rho^{\left \lfloor bT \right \rfloor }\).

Finite Sample Probabilities

Same terms as fixed-\(b\), but now includes bias.

Finding Error Rates

• Type 1 error distortion due to bias.

• Lugsail Type 1 error rate is smaller.

• Type 1 error arm shape.

• Type 2 error is concave as a function of \(b\).

Optimal \(b\)

Optimal \(b\)

Optimal \(b\)

recommended \(\tau = \frac{-\alpha^{1/(2d)}}{T \log(\rho)}\)

Optimal \(b\)

Size Distortion and Power

Regression Models

\[ y_t =\beta_0 + \beta_1x_{t,1} + \beta_2x_{t,2}+ \beta_3x_{t,3}+ \beta_4x_{t,4} + u_t \] where

• \(x_{t,i} = \rho x_{t-1,i} + e_t\) for \(i = 1, 2, 3, 4\)

• \(\beta_j =0\) for \(j = 0, 1, 2, 3, 4\)

• \(u_t\) is a dependent process depending on parameter \(\rho\)

\[ \begin{matrix} H_0:& \beta_{1} = 0 \\ H_A:& \beta_{1} \neq 0 \end{matrix} \]

AR(1) - HOMO

AR(1) - HET

ARMA(1, 1) - Gaussian

ARMA(1, 1) - Laplace

Summary

• Testing optimal \(b\) for zero lugsail estimators.

• Fixed-\(b\) asymptotic address variability, and zero lugsail estimators address bias.

• Previously, bias was often addressed through data filtering.

• Converges to Type 1 error rate faster.

• Bandwidth rule follows the curve.

• Procedure is works well with complex models.

Future Directions

• Third order fixed-\(b\) CV approximation

• MCMC

• Loss functions: Coverage probability error, other testing focused loss functions.

• Usage: Forecasting.

• Determining lugsail parameters.

• Software development.