BootStrapReport

Bootstrap Introduction

We want to draw inference abouta parameter \(\theta\), which can be a vector.

• How do we conxtruct confidence intervals/ test?
• Parameter estimates
• Distributions of these based on hypethetical resampling

Some notation:

• Parameter \(\theta = \theta(F)\)
• Sample statistic/ estimator \(T(X_1,...,X_n)\)
• Define \(\hat{\theta_B}=\theta(\hat{F})\), PLUG IN ESTIMATOR

NB: \(T(X_1,...,X_n)\) can equal \(\theta(\hat{F})\), in which case it is a plug-in estimator

Example

• Population mean: \(\mu = \int x.f(x)dx = \theta(F)\)
• Plug-in estimator: \(\hat{\theta_B} = \theta(\hat{F}) = \int x.f(x) dx = \sum x_i \frac{1}{n} = \bar{x}\)

• Alternative estimator: \(\hat{\theta_2} = T(\hat{F})\) = sample median

• \(F \rightarrow\) population distribution

• \(\hat{F} \rightarrow\) empirical distribution based on sample \(x_1,...,x_n\), a discrete probability distribution with mass \(\frac{1}{n}\) for each observation

Question:

How do we sample from this?

• Bootstrap Sample: \(X_1^*,..,X_n^*\)

Plug in principle:

A method to estimate a parameter from a sample

\[\theta=\theta(F)\]

\[\hat{\theta_B}=\theta(\hat{F})\]

Bootstrap makes use of the principle of plugging in this estimator.

Bootstrap principle:

\(\hat{F}\) is an empirical distribution function, which serves as an estimate of \(F\)

We treat \(\hat{F}\) as if it is the population and draw rebeated samples from it

Assume that the behaviour of \(T(X_1^{*i},...,T_n^{*i})\) relative to \(\hat{\theta_B}\) is the same as that of \(T(X_1,...,X_n)\) relative to \(\theta\)

Further notation:

\(T(X_1^{*i},...,T_n^{*i})\) is the estimate from the \(i^{th}\) Bootstrap Sample

\(B\) is the number of bootstrap samples

\[\bar{T^*}=\frac{1}{B} \sum_{i=1}^B T(X_1^{*i},...,X_n^{*i})\]

Bias:

\[E(\hat{\theta})-\theta\]

Bootstrap estimate of bias:

\[\bar{T^*} - \hat{\theta_B}\]

Plug in an estimate, which may differ from \(T(X_1,...,X_n)\)

Standard error:

\[\widehat{SE}(T(X_1,...,X_n))=\Big[\frac{1}{B} \sum_{i=1}^{B} (T(X_1^{*i},...,X_n^{*i})-\hat{T^*})^2\Big]^{\frac{1}{2}}\]

Basic Bootstrap confidence interval:

Define:

\(c_1=B(\frac{\alpha}{2})^{th} \space order \space statistic\) (from bootstrap distribution}

\(c_2=B(\frac{1- \alpha}{2})^{th} \space order \space statistic\)

if \(T(X_1,...,X_n)\) is the plug-in estimator,

\[L = 2 \hat{\theta_B}-c_2\]

\[U = 2 \hat{\theta_B}-c_1\]

else:

\[L = \hat{\theta_B}+T(X_1,...,X_n)\]

\[U = \hat{\theta_B}+T(X_1,...,X_n)\]

Bootstrap Percentile Interval:

If the bootstrap is distributed symmetrically around \(T(X_1,...,X^n)\), then:

\[T(X_1,...,X_n)-c_2 = -(T(X_1,...,X_n)-c_1)\]

and the confidence interval simplifies to \((c_1,c_2)\)

Non-parametric vs. Parametric Bootstrap:

Regression Example:

• NON-PARAMETRIC

No assumptions about model, distribution, or sample observations

Regression is done on sample pairs: \((x_i,y_i)\)

• PARAMETRIC

Assume model is correct, only parameter values unknown, sample from:

\[F_Y(y;\hat{\theta})\]

Regression is done by sampling residuals from \(N(0,\hat{\sigma}^2)\), calculate:

\[y_i^*=\hat{\alpha}+\hat{\beta}x_i+e_i^*\]

• SEMI-PARAMETRIC

Assume residuals are iid, but do not assume a distribution

Regression is done by resampling from observed residuals, bootstrapping data as before,

\[y_i^*=\hat{\alpha}+\hat{\beta}x_i+e_i^*\]