When does bootstrap Fail

When Does bootstrap fail? Select a statistic of interest.

The purpose of the bootstrap is to construct an approximate sampling distribution for the statistic of interest. So, if the statistic of interest \(X_n\) follows certain distribution, we’d like our bootstrap distribution \(X_n^*\) to converge to the same distribution. If we don’t have this, then we can’t trust the inferences made.

Extreme Value

One example is the minimum value of a distribution. Here is an illustration why the bootstrap fails.

sampling distribution

Let \(X_1, X_2...X_n\) from Uniform[0,1] and \(M_n=min\{X_1, X_2...X_n\}\) be the minimum of the sample.
use order statistics, we can get \[F_{M_n}(X)=P(M_n <X)=1-(1-X)^n\] let X=t/n, Then we can get \[F_{M_n}(X)=P(M_n <t/n)=1-(1-t/n)^n\] Based on e limitation, We can get \(P(n*M_n <t)=1-(1-t/n)^n=1-e^{-t}\) Then, it is known that n*M_n converges to exp(1). ### Bootstrap distribution Let \(X^*_1, X^*_2...X^*_n\) from the original observations and \(M^*_n=min\{X^*_1, X^*_2...X^*_n\}\) \[P(M_n^* =Mn)=1-(1-1/n)^n=1-e^{-1}=0.632\] This implies one of the observation in the bootstrap sample will select the minimum value of the original sample Mn with a probability 0.632.

Thus, \(M^*_n\) has a huge probability mass at the value \(M_n\), meaning that the distribution of M*n will not be close to an exponential distribution.

Simulation of sampling distribution and bootstrap distribution

Here I do the simulation to construct a sampling distribution and bootstrap distribution. From the histogram, It is obvious to see the difference between two distributions.

N=1000
samp=numeric(N)

for(i in 1:N)
{
  x=runif(500,0,1)
  samp[i] =min(x)
}

da=runif(500,0,1)

my.boot = numeric(N)
for(i in 1:N)
{
  x=sample(da, 500, TRUE)
  my.boot[i] =min(x)
}
par(mfrow = c(1,2))
hist(samp,prob=T)
curve(1000*(1-x)^999,col = "red", lwd = 2,add = TRUE,main='') #add density curve for the sampling distribution of  minimum value
hist(my.boot,prob=T)


curve(1000*(1-x)^999,col = "red", lwd = 2,add = TRUE,main='') #add density curve for the sampling distribution of  minimum value 

legend('topright','True distr of Xmin',lty=1,col='red')

Confidence Interval for bootstrap minimum

However, our population minimum of the uniform distribution is 0.

Clearly, the 95% bootstrap CI does not cover the true population minimum — 0.

#Efron's percentile method
bs_min = mean(my.boot)
quantile(my.boot,c(0.025,0.975))

##         2.5%        97.5% 
## 0.0003610309 0.0046329985

d = density(my.boot)
d

## 
## Call:
##  density.default(x = my.boot)
## 
## Data: my.boot (1000 obs.);   Bandwidth 'bw' = 0.0003941
## 
##        x                    y           
##  Min.   :-0.0008211   Min.   :  0.0000  
##  1st Qu.: 0.0027792   1st Qu.:  0.0267  
##  Median : 0.0063796   Median :  2.0018  
##  Mean   : 0.0063796   Mean   : 69.3152  
##  3rd Qu.: 0.0099800   3rd Qu.: 75.7161  
##  Max.   : 0.0135804   Max.   :641.8153

plot(d)

# The 95% CI from bootstrap never cover the true population min.

# Use bootstrap t 
min0 <- min(da)
Shat <- sd(da)/sqrt(500)
thetas <- NULL
tstar <- NULL

for (i in 1:N) {
  x_sample <- sample(da,500, TRUE) #new resample
  min <- min(x_sample) #estimate
  thetas[i] <- min #save
  tstar[i] <- (min - min0)/(sd(x_sample)/sqrt(500)) #pivotal quantity
}

c(min0, min(thetas)) #compare sample min to min of bootstrap sampling distribution

## [1] 0.0003610309 0.0003610309

c(Shat, sd(thetas)) #compare standard error from sample to standard error estimate from bootstrap distribution

## [1] 0.012781386 0.001826993

quantile(tstar, probs = c(0.025,0.975)) #quantiles from bootstrap percentile t

##      2.5%     97.5% 
## 0.0000000 0.3396792

tstar

##    [1] 0.3007891 0.3046506 0.0000000 0.3365403 0.0000000 0.0000000
##    [7] 0.0000000 0.3102549 0.2366950 0.0000000 0.0000000 0.0000000
##   [13] 0.0000000 0.0000000 0.0000000 0.2399109 0.0000000 0.0000000
##   [19] 0.3058522 0.0000000 0.2402506 0.2355374 0.0000000 0.3556324
##   [25] 0.0000000 0.0000000 0.2334619 0.0000000 0.2276214 0.2343929
##   [31] 0.0000000 0.2257894 0.0000000 0.2337754 0.3049631 0.0000000
##   [37] 0.3360490 0.2338475 0.3062385 0.2360428 0.0000000 0.0000000
##   [43] 0.0000000 0.2337010 0.0000000 0.0000000 0.3325927 0.0000000
##   [49] 0.0000000 0.3001840 0.0000000 0.0000000 0.0000000 0.0000000
##   [55] 0.2279094 0.0000000 0.0000000 0.3223322 0.0000000 0.2405891
##   [61] 0.0000000 0.0000000 0.0000000 0.2318210 0.3066050 0.0000000
##   [67] 0.0000000 0.0000000 0.2410776 0.3007199 0.0000000 0.2931222
##   [73] 0.2314144 0.0000000 0.2283061 0.0000000 0.2986364 0.0000000
##   [79] 0.0000000 0.0000000 0.2371909 0.0000000 0.0000000 0.2371025
##   [85] 0.2420246 0.0000000 0.3125234 0.0000000 0.2412874 0.2367644
##   [91] 0.0000000 0.0000000 0.0000000 0.0000000 0.2434111 0.0000000
##   [97] 0.2436123 0.0000000 0.0000000 0.2389403 0.0000000 0.0000000
##  [103] 0.2351032 0.2397261 0.0000000 0.2404266 0.0000000 0.0000000
##  [109] 0.0000000 0.3144480 0.0000000 0.3047961 0.3396585 0.0000000
##  [115] 0.0000000 0.2342852 0.0000000 0.0000000 0.0000000 0.0000000
##  [121] 0.0000000 0.0000000 0.0000000 0.0000000 0.2304896 0.0000000
##  [127] 0.2398178 0.0000000 0.0000000 0.3265176 0.0000000 0.2347908
##  [133] 0.0000000 0.2262918 0.0000000 0.2972921 0.0000000 0.0000000
##  [139] 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.2363506
##  [145] 0.0000000 0.2383708 0.0000000 0.0000000 0.0000000 0.0000000
##  [151] 0.0000000 0.2379400 0.2333630 0.0000000 0.0000000 0.0000000
##  [157] 0.0000000 0.0000000 0.2368067 0.0000000 0.2403549 0.0000000
##  [163] 0.0000000 0.0000000 0.2379602 0.2356158 0.3422777 0.0000000
##  [169] 0.0000000 0.2372771 0.2278690 0.0000000 0.2311899 0.0000000
##  [175] 0.3335934 0.3034049 0.0000000 0.0000000 0.3004079 0.0000000
##  [181] 0.2341683 0.2305744 0.0000000 0.0000000 0.0000000 0.0000000
##  [187] 0.3034490 0.2388261 0.0000000 0.0000000 0.2323948 0.0000000
##  [193] 0.2323578 0.2349967 0.3128262 0.0000000 0.0000000 0.2380258
##  [199] 0.0000000 0.0000000 0.0000000 0.0000000 0.2963809 0.0000000
##  [205] 0.0000000 0.0000000 0.3426762 0.0000000 0.0000000 0.0000000
##  [211] 0.3134110 0.3015966 0.2407359 0.3276169 0.2382773 0.0000000
##  [217] 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.3087007
##  [223] 0.0000000 0.0000000 0.0000000 0.3498512 0.2327708 0.0000000
##  [229] 0.0000000 0.0000000 0.0000000 0.2376965 0.2422311 0.2386627
##  [235] 0.0000000 0.0000000 0.0000000 0.2346718 0.3044198 0.0000000
##  [241] 0.0000000 0.0000000 0.3076258 0.2504732 0.0000000 0.0000000
##  [247] 0.3132985 0.3445830 0.0000000 0.2373671 0.2437966 0.3440412
##  [253] 0.0000000 0.0000000 0.2418859 0.0000000 0.0000000 0.0000000
##  [259] 0.0000000 0.2400195 0.2290154 0.3113158 0.0000000 0.0000000
##  [265] 0.2329900 0.0000000 0.0000000 0.0000000 0.2324035 0.2328734
##  [271] 0.0000000 0.0000000 0.0000000 0.2308010 0.0000000 0.0000000
##  [277] 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
##  [283] 0.2325941 0.0000000 0.2317274 0.2350957 0.3088299 0.0000000
##  [289] 0.2396127 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
##  [295] 0.2990055 0.0000000 0.3129095 0.0000000 0.0000000 0.3548551
##  [301] 0.2383269 0.0000000 0.0000000 0.0000000 0.0000000 0.3357655
##  [307] 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.2398526
##  [313] 0.2455896 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
##  [319] 0.0000000 0.2274637 0.0000000 0.2447835 0.0000000 0.0000000
##  [325] 0.2342584 0.3100203 0.2316598 0.0000000 0.0000000 0.2339497
##  [331] 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
##  [337] 0.0000000 0.0000000 0.0000000 0.0000000 0.3178448 0.0000000
##  [343] 0.2404082 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
##  [349] 0.0000000 0.0000000 0.3526245 0.0000000 0.0000000 0.0000000
##  [355] 0.0000000 0.0000000 0.3428148 0.3226848 0.2385567 0.0000000
##  [361] 0.3073567 0.0000000 0.3184853 0.2415067 0.2322208 0.2398102
##  [367] 0.0000000 0.2329015 0.0000000 0.2340838 0.0000000 0.0000000
##  [373] 0.2352357 0.2373862 0.0000000 0.2332062 0.0000000 0.0000000
##  [379] 0.2366422 0.2363616 0.0000000 0.0000000 1.2911169 0.0000000
##  [385] 0.0000000 0.0000000 0.0000000 0.0000000 0.2373943 0.3207914
##  [391] 0.2409995 0.0000000 0.0000000 0.3373873 0.0000000 0.0000000
##  [397] 0.3270491 0.2353170 0.0000000 0.2394071 0.0000000 0.2486592
##  [403] 0.2410812 0.2372433 0.0000000 0.0000000 0.2430568 0.0000000
##  [409] 0.0000000 0.0000000 0.3029533 0.0000000 0.3030773 0.0000000
##  [415] 1.2719829 0.0000000 0.3324652 0.0000000 0.0000000 0.0000000
##  [421] 0.0000000 0.2282662 0.2343867 0.0000000 0.0000000 0.3022462
##  [427] 0.2317956 0.0000000 0.0000000 0.2343216 0.0000000 0.0000000
##  [433] 0.2300250 0.0000000 0.0000000 0.2373465 0.0000000 0.2325509
##  [439] 0.3025454 0.0000000 0.0000000 0.0000000 0.0000000 0.3008939
##  [445] 0.2347506 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
##  [451] 0.0000000 0.2344844 0.3032256 0.0000000 0.2392315 0.0000000
##  [457] 0.2278963 0.0000000 0.3289395 0.0000000 0.3308472 0.3439788
##  [463] 0.0000000 0.3050503 0.0000000 0.0000000 0.2388282 0.0000000
##  [469] 0.3539239 0.0000000 0.3046302 0.0000000 0.2406805 0.0000000
##  [475] 0.0000000 0.2373577 0.0000000 0.0000000 0.0000000 0.0000000
##  [481] 0.0000000 0.0000000 0.0000000 0.0000000 0.3128313 0.3022812
##  [487] 0.3024533 0.2303329 0.0000000 0.0000000 0.0000000 0.2375743
##  [493] 0.2427422 0.2361536 0.0000000 0.2383535 0.0000000 0.0000000
##  [499] 0.0000000 0.2338332 0.0000000 0.0000000 0.0000000 0.2453773
##  [505] 0.0000000 0.0000000 0.0000000 0.2411378 0.0000000 0.2435576
##  [511] 0.0000000 0.0000000 0.2445927 0.0000000 0.0000000 0.0000000
##  [517] 0.3101598 0.0000000 0.2912052 0.2321391 0.0000000 0.0000000
##  [523] 0.3106585 0.0000000 0.0000000 0.0000000 0.2327042 0.0000000
##  [529] 0.3099228 0.0000000 0.0000000 0.3311501 0.2356802 0.0000000
##  [535] 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
##  [541] 0.0000000 0.0000000 0.2384725 0.0000000 0.2313133 0.0000000
##  [547] 0.2365042 0.0000000 0.2371825 0.2337824 0.0000000 0.0000000
##  [553] 0.2331091 0.0000000 0.0000000 0.2341411 0.2400109 0.0000000
##  [559] 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.2972268
##  [565] 0.2395007 0.0000000 0.0000000 0.0000000 0.2905825 0.0000000
##  [571] 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.2284650
##  [577] 0.0000000 0.2395413 0.0000000 0.2324201 0.2335112 0.0000000
##  [583] 0.0000000 0.0000000 0.3083818 0.0000000 0.0000000 0.2998447
##  [589] 0.2321272 0.0000000 0.0000000 0.0000000 0.2420254 0.0000000
##  [595] 0.2395367 0.0000000 0.0000000 0.2326966 0.0000000 0.0000000
##  [601] 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
##  [607] 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
##  [613] 0.2340107 0.0000000 0.0000000 0.0000000 0.2404961 0.0000000
##  [619] 0.0000000 0.0000000 0.0000000 0.0000000 0.3421591 0.2364260
##  [625] 0.0000000 0.0000000 0.3404857 0.3099111 0.3055776 0.0000000
##  [631] 0.2259435 0.2376598 0.0000000 0.0000000 0.2326062 0.0000000
##  [637] 0.0000000 0.0000000 0.2282448 0.0000000 0.0000000 0.0000000
##  [643] 0.2297970 0.2380620 0.3105358 0.0000000 0.0000000 0.0000000
##  [649] 0.2426454 0.0000000 0.0000000 0.2393866 0.0000000 0.0000000
##  [655] 0.0000000 0.2315307 0.0000000 0.0000000 0.0000000 0.2315402
##  [661] 0.0000000 0.3508373 0.0000000 0.0000000 0.2350348 0.2356978
##  [667] 0.0000000 0.0000000 0.0000000 0.2396560 0.2422477 0.0000000
##  [673] 0.0000000 0.2368770 0.0000000 0.0000000 0.2343043 0.0000000
##  [679] 0.0000000 0.0000000 0.2290680 0.2340846 0.2392083 0.0000000
##  [685] 0.0000000 0.2284017 0.2394313 0.0000000 0.2410606 0.0000000
##  [691] 0.0000000 0.2398813 0.3634473 0.0000000 0.2448673 0.0000000
##  [697] 0.0000000 0.0000000 0.0000000 0.2410213 0.0000000 0.2392115
##  [703] 0.0000000 0.2386332 0.0000000 0.3425811 0.2339073 0.0000000
##  [709] 0.2348299 0.2450114 0.0000000 0.0000000 0.0000000 0.0000000
##  [715] 0.2472837 0.0000000 0.0000000 0.2285489 0.0000000 0.0000000
##  [721] 0.0000000 0.0000000 0.2250631 0.0000000 0.0000000 0.0000000
##  [727] 0.2330584 0.2274785 0.2382074 0.0000000 0.2348630 0.0000000
##  [733] 0.3059731 0.0000000 0.2320073 0.0000000 0.0000000 0.2335655
##  [739] 0.0000000 0.0000000 0.0000000 0.2347381 0.0000000 0.2382706
##  [745] 0.0000000 0.0000000 0.3021332 0.0000000 0.2335313 0.2338547
##  [751] 0.0000000 0.2407879 0.2446166 0.2341018 0.0000000 0.0000000
##  [757] 0.2458768 0.3117787 0.0000000 0.2418939 0.0000000 0.2372294
##  [763] 0.0000000 0.3392328 0.2996532 0.3341463 0.2437322 0.3312630
##  [769] 0.0000000 0.0000000 0.0000000 0.3456346 0.2349346 0.3363687
##  [775] 0.2308813 0.0000000 0.2294583 0.0000000 0.3682878 0.0000000
##  [781] 0.2481315 0.2991509 0.2371095 0.0000000 0.3104393 0.0000000
##  [787] 0.0000000 0.0000000 0.2302768 0.0000000 0.0000000 0.0000000
##  [793] 0.2328286 0.0000000 0.0000000 0.2349360 0.0000000 0.0000000
##  [799] 0.2361035 0.0000000 0.0000000 0.3012764 0.0000000 0.0000000
##  [805] 0.0000000 0.0000000 0.0000000 0.0000000 0.2370763 0.2381971
##  [811] 0.2298175 0.0000000 0.0000000 0.0000000 0.3103551 0.2357680
##  [817] 0.0000000 0.0000000 0.0000000 0.0000000 0.2309550 0.0000000
##  [823] 0.2453492 0.0000000 0.0000000 0.0000000 0.0000000 0.3082185
##  [829] 0.2334523 0.2375652 0.3132978 0.0000000 0.0000000 0.0000000
##  [835] 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
##  [841] 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
##  [847] 0.2393787 0.0000000 0.0000000 0.2299968 0.3270783 0.0000000
##  [853] 0.2387933 0.0000000 0.0000000 0.0000000 0.3082195 0.0000000
##  [859] 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.3034599
##  [865] 0.0000000 0.0000000 0.0000000 0.2322190 0.0000000 0.2382402
##  [871] 0.0000000 0.0000000 0.2230292 0.0000000 0.0000000 0.0000000
##  [877] 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.3041591
##  [883] 0.3492170 0.0000000 0.3029938 0.0000000 0.0000000 0.0000000
##  [889] 0.0000000 0.0000000 0.3588391 0.0000000 0.0000000 0.2296545
##  [895] 0.2359075 0.0000000 0.0000000 0.2333274 0.0000000 0.0000000
##  [901] 0.0000000 0.3296150 0.2455761 0.2261234 0.2448812 0.0000000
##  [907] 0.0000000 0.2294508 0.0000000 0.0000000 0.0000000 0.2279881
##  [913] 0.0000000 0.3084293 0.0000000 0.0000000 0.2393034 0.0000000
##  [919] 0.0000000 0.0000000 0.0000000 0.3115913 0.0000000 0.0000000
##  [925] 0.0000000 0.2366802 0.0000000 0.3117024 0.2388551 0.0000000
##  [931] 0.2342683 0.0000000 1.2136167 0.0000000 0.0000000 0.2445990
##  [937] 0.2372952 0.2350983 0.0000000 0.2411300 0.2302412 0.0000000
##  [943] 0.2369203 0.2402425 0.0000000 0.0000000 0.0000000 0.2376959
##  [949] 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.2297283
##  [955] 0.2343139 0.0000000 0.3461149 0.2382014 0.0000000 0.0000000
##  [961] 0.0000000 0.2326090 0.3490861 0.0000000 0.0000000 0.2353681
##  [967] 0.0000000 0.2314771 0.0000000 0.0000000 0.0000000 0.0000000
##  [973] 0.0000000 0.2369199 0.0000000 0.0000000 0.0000000 0.0000000
##  [979] 0.0000000 0.3364205 0.0000000 0.0000000 0.0000000 0.0000000
##  [985] 0.0000000 0.0000000 0.2305273 0.2331896 0.2312781 0.0000000
##  [991] 0.0000000 0.2394251 0.0000000 0.0000000 0.0000000 0.2309535
##  [997] 0.2376830 0.0000000 0.0000000 0.0000000

qt(c(0.025,0.975), 500 - 1) #quantiles from student t distribution

## [1] -1.964729  1.964729

min0 + quantile(tstar, probs = c(0.025,0.975))*Shat #bootstrap percentile t confidence

##         2.5%        97.5% 
## 0.0003610309 0.0047026021

min0 + qt(c(0.025,0.975), 500 - 1) * Shat #student t confidence interval for comparison

## [1] -0.02475093  0.02547300

par(mfrow = c(1,2))
plot(density(tstar),main = 'Boostrap pivotal quantity')

t.values <- seq(-2,2,.1)
plot(x = t.values,y = dt(t.values,499), type = "l", lty = "dotted", ylim = c(0,.4), xlab = "t", ylab = "f(t)",main = 't-dsitrbution ')