1 Significance and False Discovery

In the starter script, I’ve extracted a set of p-values from (independent) marginal regression of review stars on presence in review text for each of 5000 words. That is, we fit

\[stars_i = \alpha + \beta 1 [ x_{ji} > 0 ] + \epsilon_{ji}\]

for each term \(j\) with count \(x_{ji}\) in review \(i\), and return the p-value associated with a test of \(\beta_j \neq 0\). We’ll use these 5000 independent regressions to screen words.

1.1  

Plot the p-values and comment on their distribution.

library(tidyverse)
── Attaching packages ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── tidyverse 1.2.1 ──
✔ ggplot2 2.2.1     ✔ purrr   0.2.4
✔ tibble  1.4.2     ✔ dplyr   0.7.4
✔ tidyr   0.8.0     ✔ stringr 1.2.0
✔ readr   1.1.1     ✔ forcats 0.2.0
package ‘tibble’ was built under R version 3.4.3package ‘tidyr’ was built under R version 3.4.3── Conflicts ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::collapse() masks distrom::collapse()
✖ tidyr::expand()   masks Matrix::expand()
✖ dplyr::filter()   masks stats::filter()
✖ dplyr::lag()      masks stats::lag()
pvals =as.data.frame(mrgpvals)
ggplot(pvals, aes(x = pvals$mrgpvals)) + geom_histogram()

1.2  

What is the p-value rejection region associated with a 1% False Discovery Rate? How many p-values are in this rejection region?

rej_region = fdr_cut(pvals$mrgpvals, q = .01, plotit = TRUE)

The rejection region associated with a 1% False Discovery Rate is 0.0016078. With this in mind, we would reject anything less than this value, which would mean we would reject 812 values.

1.3

Suppose you just select the words with the 250 smallest p-values as significant. How many of these do you expect are false discoveries?

pvals = pvals[order(pvals$mrgpvals), , drop = FALSE]
pvals_250 = pvals[1:250, , drop = FALSE]
pvals_250_cut = fdr_cut(pvals = pvals_250$mrgpvals, q = .1, plotit = TRUE)

for the smallest 250 P-values at a q (i.e. alpha) of .01, we would need to cut anything below 1.790372310^{-8} and that would mean that 249 would be false discoveries.

2 Least-Squares and Bootstrapping

For this question you will fit a lasso path of Gaussian regressions for stars onto x. We will focus on the AICc selected slice of \(\hat\beta\) along this path.

2.1

What are the in-sample SSE and R2 for this regression?

df = data.frame(as.matrix(yhat))
df$stars = stars
is.r2 = R2(df$stars,df$seg66, family = "gaussian")
sse = deviance(df$stars, df$seg66, family = 'gaussian')

In Sample \(R^2\): 0.5000111

\(SSE\): 6632.7653746

2.2  

Describe what ratings we’d expect for Bowling businesses and Airports.

3 Model Choice and Logistic Regression

Define a ‘bad review’ as one with fewer than 4 stars. We’ll regress this binary outcome on x.

rev$bad.review = ifelse(rev$stars < 4, 1, 0)

3.1   Use cv.gamlr to fit a cross-validated lasso regularization path for this regression. Plot the in-sample fit.

cv.badreview = cv.gamlr(x, rev$bad.review, lmr = 1e-4, verb=F, family = "binomial")
numerically perfect fit for some observations.
plot(cv.stars)

plot(linfit)

3.2

Describe the criteria used to choose models under select=“1se” and select=“min” rules. What are estimated out-of-sample R2 for models fit using these \(\lambda\)?

The “min” lambda is minimum average out of sample deviance, represented by the left most vertical line, while the “1se” lambda value/oos deviance is at least 1 standard deviation away from the minimum value and still penalizes the large number of coefficeints.

r2_min <- summary(cv.badreview)[cv.badreview$seg.min,"oos.r2"]

5-fold binomial cv.gamlr object
r2_1se <- summary(cv.badreview)[cv.badreview$seg.1se,"oos.r2"]

5-fold binomial cv.gamlr object
r2_val_min <- cor((predict(cv.badreview, x, select = "min")[,1]),rev$bad.review) 
r2_val_1se <- cor((predict(cv.badreview, x, select = "1se")[,1]),rev$bad.review)

\(R^2\) for min lambda: 0.6254965

\(R^2\) for 1 se lambda: 0.6095646

3.3  

Compare AICc, AIC, and BIC selection to each other and to the CV rules.

library(knitr)
cv_min = cor((predict(cv.stars, x, select = "min")[,1]), (stars))
cv_1se = cor((predict(cv.stars, x, select = "1se")[,1]), (stars))
aicc = cor((predict(cv.stars, x, select=NULL)[,1]), (stars))
aic = cor((predict(linfit, 
                      x, 
                      select=NULL, 
                      corrected=FALSE)[,1]),(stars))
bic = cor(exp(predict(linfit, 
                      x, 
                      select=NULL,
                      k=log(linfit$nobs),
                      corrected=FALSE)[,1]), stars)
table1 = data.frame("Method" = c("Minimum Cross Validation",
                                 "1se Cross Validation",
                                 "AICc","AIC","BIC"),
                    "Value" = c(cv_min, 
                                cv_1se, 
                                aicc,
                                aic, 
                                bic))
kable(table1)
Method Value
Minimum Cross Validation -0.6700239
1se Cross Validation -0.6570002
AICc -0.6570002
AIC 0.7076896
BIC 0.5185542

Print the top ten best and worst review words under your preferred IC selection rule. Do the results make sense?

words = data.frame("coef" = as.matrix(coef(cv.badreview)),
                   "abs.val" = abs(as.matrix(coef(cv.badreview))),
                   "word" = rownames(coef(cv.badreview))
                   )
words = arrange(words, desc(words$seg33.1))
colnames(words) = c("Coefficient Value", "Coefficient Absolute Value", "Word")
kable(head(words, 10))
Coefficient Value Coefficient Absolute Value Word
1.366706 1.366706 shitty
1.350759 1.350759 stale
1.350303 1.350303 overpriced
1.299702 1.299702 tasteless
1.189143 1.189143 disappointing
1.119422 1.119422 Mobile Phones
1.083031 1.083031 disgusting
1.081217 1.081217 bland
1.069535 1.069535 poisoning
1.009014 1.009014 horrible

Yes, these words all make sense they are all negative. Additionall all of their coefficients are greater than 1 meaning that any one of these words independently increase the odds of bad.review being 1. ***

4 Multinomial Regression

Now, we’ll consider stars as an unordered 5-level factor. Use the code in your starter script to run a multinomial logistic regression.

4.1

What is the change in odds of 1 vs 2-3 stars for an extra count of word crap in the review?

multi.coef = as.data.frame(as.matrix(Bmn)) %>%rownames_to_column() %>% filter(rowname == 'crap' |rowname == 'Crap')
kable(multi.coef)

rowname 1 2 3 4 5
crap 0.108569 0 0 -0.1969228 0 

one_star = exp(multi.coef$`1`)
four_star = exp(multi.coef$`4`)

if the word ‘crap’ is in the review, the change in odds is 0.7819642, while it is zero for both 2 and 3, and then it changes again for 4 stars where the likelihood ratio is 0.9711562. This means that if the word crap appeasr in a review, it is 11% more likely to be a one-star review, and approximately 18% less likely to be a four star review.

4.2

What are the changes in odds of 1 star vs 2 and 1 star vs 5 star ratings for an extra count of word fantastic in the review?

multi.coef2 = as.data.frame(as.matrix(Bmn)) %>%rownames_to_column() %>% filter(rowname == 'fantastic' |rowname == 'Fantastic' | rowname == 'FANTASTIC')
kable(multi.coef2)

rowname 1 2 3 4 5
fantastic -0.2459463 -0.4714128 -0.2779925 -0.029268 0.173123 

one_star = exp(multi.coef2$`1`)
two_star = exp(multi.coef2$`2`)
three_star = exp(multi.coef2$`3`)
four_star = exp(multi.coef2$`4`)
five_star = exp(multi.coef2$`5`)
stars = data.frame("Rating" = c("One Star", "Two Star","Three Star","Four Star","Five Star"),
                   "Likelihood Ratio" = c(one_star, two_star, three_star, four_star, five_star)
                   )
Rating Likelihood.Ratio
One Star 0.7819642
Two Star 0.6241199
Three Star 0.7573025
Four Star 0.9711562
Five Star 1.1890124

We can see here that using the word fantastic in a review leads to an 18% more likely to be a five star review, while it means it is 3% less likely to be a four star review, 25% less likely to be a three star review, 48% less likely to be a two star review and interestly it is 22% less likely to be a 1 star review.

LS0tCnRpdGxlOiAiTWFjaGluZSBMZWFybmluZyBQcm9ibGVtIFNldCA0IgphdXRob3I6ICJUb20gQ3VycmFuIgpkYXRlOiAiTWF5IDIyLCAyMDE4IgpvdXRwdXQ6CiAgcGRmX2RvY3VtZW50OiBkZWZhdWx0CiAgaHRtbF9ub3RlYm9vazogZGVmYXVsdAotLS0KCmBgYHtyIHN0YXJ0ZXJfc2NyaXB0cywgbWVzc2FnZT1GQUxTRSwgd2FybmluZz1GQUxTRSwgaW5jbHVkZT1GQUxTRSwgcGFnZWQucHJpbnQ9RkFMU0V9CmZkcl9jdXQgPC0gZnVuY3Rpb24ocHZhbHMsIHEsIHBsb3RpdD1GQUxTRSwgLi4uKXsKICBwdmFscyA8LSBwdmFsc1shaXMubmEocHZhbHMpXQogIE4gPC0gbGVuZ3RoKHB2YWxzKQogIAogIGsgPC0gcmFuayhwdmFscywgdGllcy5tZXRob2Q9Im1pbiIpCiAgYWxwaGEgPC0gbWF4KHB2YWxzWyBwdmFsczw9IChxKmsvTikgXSkKICAKICBpZihwbG90aXQpewogICAgc2lnIDwtIGZhY3RvcihwdmFsczw9YWxwaGEpCiAgICBvIDwtIG9yZGVyKHB2YWxzKQogICAgcGxvdChwdmFsc1tvXSwgY29sPWMoImdyZXk2MCIsInJlZCIpW3NpZ1tvXV0sIHBjaD0yMCwgLi4uLCAKICAgICAgIHlsYWI9InAtdmFsdWVzIiwgeGxhYj0idGVzdHMgb3JkZXJlZCBieSBwLXZhbHVlIiwgbWFpbiA9IHBhc3RlKCdGRFIgPScscSkpCiAgICBsaW5lcygxOk4sIHEqKDE6TikvTikKICB9CiAgCiAgcmV0dXJuKGFscGhhKQp9CgojIyBwcmVkIG11c3QgYmUgcHJvYmFiaWxpdGllcyAoMDxwcmVkPDEpIGZvciBiaW5vbWlhbApkZXZpYW5jZSA8LSBmdW5jdGlvbih5LCBwcmVkLCBmYW1pbHk9YygiZ2F1c3NpYW4iLCJiaW5vbWlhbCIpKXsKCWZhbWlseSA8LSBtYXRjaC5hcmcoZmFtaWx5KQoJaWYoZmFtaWx5PT0iZ2F1c3NpYW4iKXsKCQlyZXR1cm4oIHN1bSggKHktcHJlZCleMiApICkKCX1lbHNlewoJCWlmKGlzLmZhY3Rvcih5KSkgeSA8LSBhcy5udW1lcmljKHkpPjEKCQlyZXR1cm4oIC0yKnN1bSggeSpsb2cocHJlZCkgKyAoMS15KSpsb2coMS1wcmVkKSApICkKCX0KfQoKCiMjIGdldCBudWxsIGRldmFpbmNlIHRvbywgYW5kIHJldHVybiBSMgpSMiA8LSBmdW5jdGlvbih5LCBwcmVkLCBmYW1pbHk9YygiZ2F1c3NpYW4iLCJiaW5vbWlhbCIpKXsKCWZhbSA8LSBtYXRjaC5hcmcoZmFtaWx5KQoJaWYoZmFtPT0iYmlub21pYWwiKXsKCQlpZihpcy5mYWN0b3IoeSkpeyB5IDwtIGFzLm51bWVyaWMoeSk+MSB9Cgl9CglkZXYgPC0gZGV2aWFuY2UoeSwgcHJlZCwgZmFtaWx5PWZhbSkKCWRldjAgPC0gZGV2aWFuY2UoeSwgbWVhbih5KSwgZmFtaWx5PWZhbSkKCXJldHVybigxLWRldi9kZXYwKQp9CgoKIyMjIE1JRFRFUk06IHllbHAgYnVzaW5lc3MgcmV2aWV3cwoKIyMgcmVhZCB0aGUgZGF0YQpiaXogPC0gcmVhZC5jc3YoIkJpelR5cGUuY3N2IikgIAp0eHQgPC0gcmVhZC5jc3YoIldvcmRGcmVxLmNzdiIpCnJldiA8LSByZWFkLmNzdigiUmV2aWV3U3RhdHMuY3N2IikgIyBud3JkPXJldmlldyBsZW5ndGgsIG5yZXY9ICMgcmV2aWV3cyBieSB0aGlzIHVzZXIKd29yZHMgPC0gc2Nhbigid29yZHMudHh0Iiwgd2hhdD0iY2hhcmFjdGVyIiwgc2VwPSJcbiIpICNcbiBpcyBgZW5kIG9mIGxpbmUnCmNhdGVnb3JpZXMgPC0gc2NhbigiY2F0ZWdvcmllcy50eHQiLCB3aGF0PSJjaGFyYWN0ZXIiLCBzZXA9IlxuIikgIyB0aGVzZSBhcmUgdGhlIHllbHAgY2F0ZWdvcmllcwoKIyMgY3JlYXRlIHNvbWUgc3BhcnNlIG1hdHJpY2VzCmxpYnJhcnkoZ2FtbHIpCkJpeiA8LSBzcGFyc2VNYXRyaXgoaT1iaXokaSxqPWJpeiRqLHg9Yml6JHgsCglkaW1uYW1lcz1saXN0KHJldj0xOm5yb3cocmV2KSxjYXQ9Y2F0ZWdvcmllcykpCgpDIDwtIHNwYXJzZU1hdHJpeChpPXR4dCRpLGo9dHh0JGosCgkJeD10eHQkeCwKCQlkaW1uYW1lcz1saXN0KHJldj0xOm5yb3cocmV2KSxjYXQ9d29yZHMpKQoKIyMgd2hhdCBkbyB0aGVzZSBsb29rIGxpa2U/CkJpelsxOjIsYygxMiwxMDMsOTYsMTI0LDEyOSwxMzApXSAjIHJldmlld3MgaW4gcm93cywgYnVzaW5lc3MgdHlwZXMgY29uY2VybmVkIGluIGNvbHVtbnMKQ1sxOjUsMTo1XSAjIHJldmlld3MgaW4gcm93cywgd29yZCBpbmRleGVzIGluIGNvbHVtbnMsIHZhbHVlcyBhcmUgd29yZCBjb3VudHMKIyBqdXN0IG1hdHJpY2VzLCBidXQgd2UgaGF2ZW4ndCBzdG9yZWQgdGhlIHplcm9zCgojIyBbMV0gTWFyZ2luYWwgUmVncmVzc2lvbiBTY3JlZW5pbmcgCiMjCiMjIEFub3RoZXIgZXhhbXBsZSBvZiBhbiBhbGdvcml0aG0gdGhhdCBjYW4gYmUgRGlzdHJpYnV0ZWQhIAojIyBXZSdsbCBkbyA1MDAwIHVuaXZhcmlhdGUgcmVncmVzc2lvbnMgb2YgCiMjIHN0YXIgcmF0aW5nIG9uIHdvcmQgcHJlc2VuY2UsIG9uZSBmb3IgZWFjaCB3b3JkLgojIyBFYWNoIHJlZ3Jlc3Npb24gd2lsbCByZXR1cm4gYSBwLXZhbHVlLCBhbmQgd2UgY2FuCiMjIHVzZSB0aGlzIGFzIGFuIGluaXRpYWwgc2NyZWVuIGZvciB1c2VmdWwgd29yZHMuCgojIGNyZWF0ZSBhIGRlbnNlIG1hdHJpeCBvZiB3b3JkIHByZXNlbmNlClAgPC0gYXMuZGF0YS5mcmFtZShhcy5tYXRyaXgoQz4wKSkKbGlicmFyeShwYXJhbGxlbCkKbWFyZ3JlZyA8LSBmdW5jdGlvbihwKXsKCWZpdCA8LSBsbShzdGFyc35wKQoJc2YgPC0gc3VtbWFyeShmaXQpCglyZXR1cm4oc2YkY29lZlsyLDRdKSAKfQoKY2wgPC0gbWFrZUNsdXN0ZXIoZGV0ZWN0Q29yZXMoKSkKIyBwdWxsIG91dCBzdGFycyBhbmQgZXhwb3J0IHRvIGNvcmVzCnN0YXJzIDwtIHJldiRzdGFycwpjbHVzdGVyRXhwb3J0KGNsLCJzdGFycyIpIAojIHJ1biB0aGUgcmVncmVzc2lvbnMgaW4gcGFyYWxsZWwKbXJncHZhbHMgPC0gdW5saXN0KHBhckxhcHBseShjbCxQLG1hcmdyZWcpKQojIyBJZiBwYXJhbGxlbCBzdHVmZiBpcyBub3Qgd29ya2luZywgCiMjIHlvdSBjYW4gYWxzbyBqdXN0IGRvIChpbiBzZXJpYWwpOgojIG1yZ3B2YWxzIDwtIGMoKQojIGZvcihqIGluIDE6NTAwMCl7CiMgCXByaW50KGopCiMgCW1yZ3B2YWxzIDwtIGMobXJncHZhbHMsbWFyZ3JlZyhQWyxqXSkpCiMgfQojIyBtYWtlIHN1cmUgd2UgaGF2ZSBuYW1lcwpuYW1lcyhtcmdwdmFscykgPC0gY29sbmFtZXMoUCkKCiMjIyMjICBTZXQgZGF0YSBmb3IgcmVzdCBvZiBleGFtICMjIyMjCgojIFJlc3RyaWN0IHRleHQgZGF0YSB0byB0b3AgMjUwIGJ5IGFib3ZlIHB2YWx1ZXMKQ2N1dCA8LSBDWyxuYW1lcyhzb3J0KG1yZ3B2YWxzKVsxOjI1MF0pXQojIGNyZWF0ZSBtYXRyaXggb2YgaW50ZXJhY3Rpb25zIGJldHdlZW4gbGVuZ3RoIGFuZCBjYXRlZ29yeQpCaXpOV1JEIDwtIHJldiRud3JkKkJpegpjb2xuYW1lcyhCaXpOV1JEKSA8LSBwYXN0ZSgiTldSRDoiLGNvbG5hbWVzKEJpeiksc2VwPSIiKSAKIyBjcmVhdGUgb3VyIHggbWF0cml4LCB1c2VkIHRocm91Z2hvdXQuCnggPC0gY0JpbmQoQml6LEJpek5XUkQsQ2N1dCkgIyBtYWtlIHN1cmUgeW91IHVuZGVyc3RhbmQgd2hhdCdzIGluIHJvd3MsIHdoYXQncyBpbiBjb2x1bW5zLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICMgYW5kIHdoYXQgZG8gdmFsdWVzIG1lYW4KCiMjIFsyXSBsaW5lYXIgcmVncmVzc2lvbgojIG5vdGU6IHlvdSdsbCB3YW50IHRvIHVzZSBsYW1iZGEubWluLnJhdGlvPTFlLTMgaW4geW91ciBsYXNzb3MuCiMgbGluZWFyIHJlZ3Jlc3Npb24gZm9yIDEtNSBzdGFycyBvbiB4LiAKbGluZml0IDwtIGdhbWxyKHgsIHk9c3RhcnMsIGxhbWJkYS5taW4ucmF0aW89MWUtMykKIyB0aGUgY29lZmZpY2llbnRzIChkcm9wIHR1cm5zIGl0IGludG8gc2ltcGxlIHZlY3RvcikKQmxpbiA8LSBkcm9wKGNvZWYobGluZml0KSkgIyBBSUNjIGRlZmF1bHQgc2VsZWN0aW9uCiMgcHJlZGljdGVkIHZhbHVlcwp5aGF0IDwtIHByZWRpY3QobGluZml0LHgpCiMgbm90ZSBsaW5maXQkZGV2aWFuY2UgY29udGFpbnMgdGhlIGluIHNhbXBsZSBkZXZpYW5jZSBmb3IgZWFjaCBsYW1iZGEKCiMjIGJvb3RzdHJhcCBjb2RlIGZvciAyLjMKIyB0aGUgc2FtcGxpbmcgZGlzdHJpYnV0aW9uIG9mIEFJQ2Mgb3B0aW1hbCBkZi4KIyBmdW5jdGlvbiB0YWtlcyBhbiBpbmRleCBvZiBkYXRhIGFuZCB3aWxsIHJldHVybiBmaXR0ZWQgQUlDYyBvcHRpbWFsIGRmCmJvb3RkZiA8LSBmdW5jdGlvbihpYil7CglyZXF1aXJlKGdhbWxyKQoJZml0IDwtIGdhbWxyKHhbaWIsXSx5PXN0YXJzW2liXSwgbGFtYmRhLm1pbi5yYXRpbz0xZS0zKQoJc3VtKGNvZWYoZml0KSE9MCktMQp9CiMgZXhwb3J0IHRoZSBkYXRhIHRvIHRoZSBjbHVzdGVycyAKY2x1c3RlckV4cG9ydChjbCwieCIpCiMgcnVuIDEwMCBib290c3RyYXAgcmVzYW1wbGUgZml0cwpib290cyA8LSAxMDAKbiA8LSBucm93KHgpCnJlc2FtcCA8LSBhcy5kYXRhLmZyYW1lKG1hdHJpeChzYW1wbGUoMTpuLGJvb3RzKm4scmVwbGFjZT1UUlVFKSxuY29sPWJvb3RzKSkKZGZzYW1wIDwtIHVubGlzdChwYXJMYXBwbHkoY2wscmVzYW1wLGJvb3RkZikpCgojIyBhZ2FpbiwgdGhpcyBpcyBhbGwgcHJldHR5IGZhc3Qgc28gaXQgY291bGQgaGF2ZSBoYXBwZW5lZCBpbiBzZXJpYWw6CiMgZGZzYW1wIDwtIGMoKQojIGZvcihiIGluIDE6MTAwKXsKIyAJZGZzYW1wIDwtIGMoZGZzYW1wLCBib290ZGYocmVzYW1wWyxiXSkpCiMgCXByaW50KGIpCiMgfQoKIyMgWzNdIGxvZ2lzdGljIHJlZ3Jlc3Npb24gYW5kIG1vZGVsIHNlbGVjdGlvbgp5IDwtIHJldiRzdGFyczw0ICMgb3VyIGBiYWQgcmV2aWV3JyBiaW5hcnkgcmVzcG9uc2UKIyBTYXkgYmluZml0IGlzIHRoZSBlc3RpbWF0ZWQgY3YuZ2FtbHIgb2JqZWN0LgojIFRoZW4gbm90ZSBiaW5maXQkY3ZtIGNvbnRhaW5zICBhdmVyYWdlIE9PUyBkZXZpYW5jZSBmb3IgZWFjaCBsYW1iZGEsIGFuZAojIGJpbmZpdCRzZWcubWluIGFuZCBiaW5maXQkc2VnLjFzZSBpZGVudGlmeSBzZWxlY3RlZCBzZWdtZW50cyB1bmRlciBlYWNoIHJ1bGUKCiMjIFs0XSBUcmVhdG1lbnQgZWZmZWN0cyBlc3RpbWF0aW9uIGZvciBgcmV2aWV3ZXIgZXhwZXJpZW5jZScKCiMgY29uc2lkZXIgdHJlYXRtZW50IGFjdGluZyBvbiBsb2cgc2NhbGUKZCA8LSBsb2cocmV2JG5yZXYpCiMgZml0IGEgbW9kZWwgZm9yIGQgb24geAp0cmVhdGZpdCA8LSBnYW1scih4LGQpCgojIyBbNV0gbXVsdGlub21pYWwgcmVncmVzc2lvbgojIHZlcnkgc3RyYWlnaHQgZm9yd2FyZCwgeW91IGp1c3QgbmVlZCB0byByZWFkIHRoZSBjb2VmZmljaWVudHMKbGlicmFyeShkaXN0cm9tKQp5bW4gPC0gZmFjdG9yKHN0YXJzKQptbmZpdCA8LSBkbXIoY2wseCx5bW4pCkJtbiA8LSBjb2VmKG1uZml0KQpCbW5bYygiY3JhcCIsImZhbnRhc3RpYyIpLF0KCmBgYAoKIyMjIDEgU2lnbmlmaWNhbmNlIGFuZCBGYWxzZSBEaXNjb3ZlcnkKCkluIHRoZSBzdGFydGVyIHNjcmlwdCwgSeKAmXZlIGV4dHJhY3RlZCBhIHNldCBvZiBwLXZhbHVlcyBmcm9tIChpbmRlcGVuZGVudCkgbWFyZ2luYWwgcmVncmVzc2lvbiBvZiByZXZpZXcgc3RhcnMgb24gcHJlc2VuY2UgaW4gcmV2aWV3IHRleHQgZm9yIGVhY2ggb2YgNTAwMCB3b3Jkcy4gVGhhdCBpcywgd2UgZml0CgokJHN0YXJzX2kgPSBcYWxwaGEgKyBcYmV0YSAxIFsgeF97aml9ID4gMCBdICsgXGVwc2lsb25fe2ppfSQkCgpmb3IgZWFjaCB0ZXJtICRqJCB3aXRoIGNvdW50ICR4X3tqaX0kIGluIHJldmlldyAkaSQsIGFuZCByZXR1cm4gdGhlIHAtdmFsdWUgYXNzb2NpYXRlZCB3aXRoIGEgdGVzdCBvZiAkXGJldGFfaiBcbmVxIDAkLiBXZeKAmWxsIHVzZSB0aGVzZSA1MDAwIGluZGVwZW5kZW50IHJlZ3Jlc3Npb25zIHRvIHNjcmVlbiB3b3Jkcy4KCioqMS4xKiogXCAKCioqUGxvdCB0aGUgcC12YWx1ZXMgYW5kIGNvbW1lbnQgb24gdGhlaXIgZGlzdHJpYnV0aW9uLioqCgpgYGB7ciBtZXNzYWdlPUZBTFNFLCB3YXJuaW5nPUZBTFNFLCBpbmNsdWRlPUZBTFNFLCBwYWdlZC5wcmludD1GQUxTRX0KbGlicmFyeSh0aWR5dmVyc2UpCmBgYAoKYGBge3IgbWVzc2FnZT1GQUxTRSwgd2FybmluZz1GQUxTRSwgcGFnZWQucHJpbnQ9RkFMU0V9CgpwdmFscyA9YXMuZGF0YS5mcmFtZShtcmdwdmFscykKCmdncGxvdChwdmFscywgYWVzKHggPSBwdmFscyRtcmdwdmFscykpICsgZ2VvbV9oaXN0b2dyYW0oKQpgYGAKCioqMS4yKiogXCAKCioqV2hhdCBpcyB0aGUgcC12YWx1ZSByZWplY3Rpb24gcmVnaW9uIGFzc29jaWF0ZWQgd2l0aCBhIDElIEZhbHNlIERpc2NvdmVyeSBSYXRlPyBIb3cgbWFueSBwLXZhbHVlcyBhcmUgaW4gdGhpcyByZWplY3Rpb24gcmVnaW9uPyoqCgpgYGB7ciBtZXNzYWdlPUZBTFNFLCB3YXJuaW5nPUZBTFNFfQoKcmVqX3JlZ2lvbiA9IGZkcl9jdXQocHZhbHMkbXJncHZhbHMsIHEgPSAuMDEsIHBsb3RpdCA9IFRSVUUpCgpgYGAKVGhlIHJlamVjdGlvbiByZWdpb24gYXNzb2NpYXRlZCB3aXRoIGEgMSUgRmFsc2UgRGlzY292ZXJ5IFJhdGUgaXMgYHIgcmVqX3JlZ2lvbmAuIFdpdGggdGhpcyBpbiBtaW5kLCB3ZSB3b3VsZCByZWplY3QgYW55dGhpbmcgbGVzcyB0aGFuIHRoaXMgdmFsdWUsIHdoaWNoIHdvdWxkIG1lYW4gd2Ugd291bGQgcmVqZWN0IGByIGxlbmd0aChwdmFsc1twdmFscyRtcmdwdmFscyA8IHJlal9yZWdpb24sXSlgIHZhbHVlcy4KCgoqKjEuMyoqIFwKCioqU3VwcG9zZSB5b3UganVzdCBzZWxlY3QgdGhlIHdvcmRzIHdpdGggdGhlIDI1MCBzbWFsbGVzdCBwLXZhbHVlcyBhcyBzaWduaWZpY2FudC4gSG93IG1hbnkgb2YgdGhlc2UgZG8geW91IGV4cGVjdCBhcmUgZmFsc2UgZGlzY292ZXJpZXM/KioKYGBge3J9CnB2YWxzID0gcHZhbHNbb3JkZXIocHZhbHMkbXJncHZhbHMpLCAsIGRyb3AgPSBGQUxTRV0KCnB2YWxzXzI1MCA9IHB2YWxzWzE6MjUwLCAsIGRyb3AgPSBGQUxTRV0KCnB2YWxzXzI1MF9jdXQgPSBmZHJfY3V0KHB2YWxzID0gcHZhbHNfMjUwJG1yZ3B2YWxzLCBxID0gLjEsIHBsb3RpdCA9IFRSVUUpCmBgYAoKZm9yIHRoZSBzbWFsbGVzdCAyNTAgUC12YWx1ZXMgYXQgYSBxIChpLmUuIGFscGhhKSBvZiAuMDEsIHdlIHdvdWxkIG5lZWQgdG8gY3V0IGFueXRoaW5nIGJlbG93IGByIHB2YWxzXzI1MF9jdXRgIGFuZCB0aGF0IHdvdWxkIG1lYW4gdGhhdCBgciBsZW5ndGgocHZhbHNfMjUwW3B2YWxzXzI1MCA8IHB2YWxzXzI1MF9jdXQsXSlgIHdvdWxkIGJlIGZhbHNlIGRpc2NvdmVyaWVzLgoKKioqCgojIyMgMiBMZWFzdC1TcXVhcmVzIGFuZCBCb290c3RyYXBwaW5nCgpGb3IgdGhpcyBxdWVzdGlvbiB5b3Ugd2lsbCBmaXQgYSBsYXNzbyBwYXRoIG9mIEdhdXNzaWFuIHJlZ3Jlc3Npb25zIGZvciBzdGFycyBvbnRvIHguIFdlIHdpbGwgZm9jdXMgb24gdGhlIEFJQ2Mgc2VsZWN0ZWQgc2xpY2Ugb2YgJFxoYXRcYmV0YSQgYWxvbmcgdGhpcyBwYXRoLgoKCgoqKjIuMSoqIFwKCioqV2hhdCBhcmUgdGhlIGluLXNhbXBsZSBTU0UgYW5kIFIyIGZvciB0aGlzIHJlZ3Jlc3Npb24/KioKCmBgYHtyfQoKZGYgPSBkYXRhLmZyYW1lKGFzLm1hdHJpeCh5aGF0KSkKCmRmJHN0YXJzID0gc3RhcnMKCmlzLnIyID0gUjIoZGYkc3RhcnMsZGYkc2VnNjYsIGZhbWlseSA9ICJnYXVzc2lhbiIpCgpzc2UgPSBkZXZpYW5jZShkZiRzdGFycywgZGYkc2VnNjYsIGZhbWlseSA9ICdnYXVzc2lhbicpCgpgYGAKCkluIFNhbXBsZSAkUl4yJDogYHIgaXMucjJgCgokU1NFJDogYHIgc3NlYAoKKioyLjIqKiBcIAoKKipEZXNjcmliZSB3aGF0IHJhdGluZ3Mgd2XigJlkIGV4cGVjdCBmb3IgQm93bGluZyBidXNpbmVzc2VzIGFuZCBBaXJwb3J0cy4qKgoKCioqKgoKIyMjIDMgTW9kZWwgQ2hvaWNlIGFuZCBMb2dpc3RpYyBSZWdyZXNzaW9uCgpEZWZpbmUgYSDigJhiYWQgcmV2aWV34oCZIGFzIG9uZSB3aXRoIGZld2VyIHRoYW4gNCBzdGFycy4gV2XigJlsbCByZWdyZXNzIHRoaXMgYmluYXJ5IG91dGNvbWUgb24geC4gCmBgYHtyfQpyZXYkYmFkLnJldmlldyA9IGlmZWxzZShyZXYkc3RhcnMgPCA0LCAxLCAwKQpgYGAKCioqMy4xKiogXCAKKipVc2UgY3YuZ2FtbHIgdG8gZml0IGEgY3Jvc3MtdmFsaWRhdGVkIGxhc3NvIHJlZ3VsYXJpemF0aW9uIHBhdGggZm9yIHRoaXMgcmVncmVzc2lvbi4gUGxvdCB0aGUgaW4tc2FtcGxlIGZpdC4qKiAKCmBgYHtyfQpjdi5iYWRyZXZpZXcgPSBjdi5nYW1scih4LCByZXYkYmFkLnJldmlldywgbG1yID0gMWUtNCwgdmVyYj1GLCBmYW1pbHkgPSAiYmlub21pYWwiKQoKcGxvdChjdi5zdGFycykKCmBgYAoKYGBge3J9CnBsb3QobGluZml0KQpgYGAKCioqMy4yKipcCgoqKkRlc2NyaWJlIHRoZSBjcml0ZXJpYSB1c2VkIHRvIGNob29zZSBtb2RlbHMgdW5kZXIgc2VsZWN0PSIxc2UiIGFuZCBzZWxlY3Q9Im1pbiIgcnVsZXMuIFdoYXQgYXJlIGVzdGltYXRlZCBvdXQtb2Ytc2FtcGxlIFIyIGZvciBtb2RlbHMgZml0IHVzaW5nIHRoZXNlICRcbGFtYmRhJD8qKiAKClRoZSDigJxtaW7igJ0gbGFtYmRhIGlzIG1pbmltdW0gYXZlcmFnZSBvdXQgb2Ygc2FtcGxlIGRldmlhbmNlLCByZXByZXNlbnRlZCBieSB0aGUgbGVmdCBtb3N0IHZlcnRpY2FsIGxpbmUsIHdoaWxlIHRoZSDigJwxc2XigJ0gbGFtYmRhIHZhbHVlL29vcyBkZXZpYW5jZSBpcyBhdCBsZWFzdCAxIHN0YW5kYXJkIGRldmlhdGlvbiBhd2F5IGZyb20gdGhlIG1pbmltdW0gdmFsdWUgYW5kIHN0aWxsIHBlbmFsaXplcyB0aGUgbGFyZ2UgbnVtYmVyIG9mIGNvZWZmaWNlaW50cy4KCmBgYHtyfQpyMl9taW4gPC0gc3VtbWFyeShjdi5iYWRyZXZpZXcpW2N2LmJhZHJldmlldyRzZWcubWluLCJvb3MucjIiXQoKcjJfMXNlIDwtIHN1bW1hcnkoY3YuYmFkcmV2aWV3KVtjdi5iYWRyZXZpZXckc2VnLjFzZSwib29zLnIyIl0KCnIyX3ZhbF9taW4gPC0gY29yKChwcmVkaWN0KGN2LmJhZHJldmlldywgeCwgc2VsZWN0ID0gIm1pbiIpWywxXSkscmV2JGJhZC5yZXZpZXcpIAoKcjJfdmFsXzFzZSA8LSBjb3IoKHByZWRpY3QoY3YuYmFkcmV2aWV3LCB4LCBzZWxlY3QgPSAiMXNlIilbLDFdKSxyZXYkYmFkLnJldmlldykKYGBgCgokUl4yJCBmb3IgbWluIGxhbWJkYTogYHIgcjJfdmFsX21pbmAKCiRSXjIkIGZvciAxIHNlIGxhbWJkYTogYHIgcjJfdmFsXzFzZWAKCioqMy4zKiogXCAKCioqQ29tcGFyZSBBSUNjLCBBSUMsIGFuZCBCSUMgc2VsZWN0aW9uIHRvIGVhY2ggb3RoZXIgYW5kIHRvIHRoZSBDViBydWxlcy4qKgoKYGBge3J9CmxpYnJhcnkoa25pdHIpCmN2X21pbiA9IGNvcigocHJlZGljdChjdi5zdGFycywgeCwgc2VsZWN0ID0gIm1pbiIpWywxXSksIChzdGFycykpCgpjdl8xc2UgPSBjb3IoKHByZWRpY3QoY3Yuc3RhcnMsIHgsIHNlbGVjdCA9ICIxc2UiKVssMV0pLCAoc3RhcnMpKQoKYWljYyA9IGNvcigocHJlZGljdChjdi5zdGFycywgeCwgc2VsZWN0PU5VTEwpWywxXSksIChzdGFycykpCgphaWMgPSBjb3IoKHByZWRpY3QobGluZml0LCAKICAgICAgICAgICAgICAgICAgICAgIHgsIAogICAgICAgICAgICAgICAgICAgICAgc2VsZWN0PU5VTEwsIAogICAgICAgICAgICAgICAgICAgICAgY29ycmVjdGVkPUZBTFNFKVssMV0pLChzdGFycykpCgpiaWMgPSBjb3IoZXhwKHByZWRpY3QobGluZml0LCAKICAgICAgICAgICAgICAgICAgICAgIHgsIAogICAgICAgICAgICAgICAgICAgICAgc2VsZWN0PU5VTEwsCiAgICAgICAgICAgICAgICAgICAgICBrPWxvZyhsaW5maXQkbm9icyksCiAgICAgICAgICAgICAgICAgICAgICBjb3JyZWN0ZWQ9RkFMU0UpWywxXSksIHN0YXJzKQoKCgp0YWJsZTEgPSBkYXRhLmZyYW1lKCJNZXRob2QiID0gYygiTWluaW11bSBDcm9zcyBWYWxpZGF0aW9uIiwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIjFzZSBDcm9zcyBWYWxpZGF0aW9uIiwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIkFJQ2MiLCJBSUMiLCJCSUMiKSwKICAgICAgICAgICAgICAgICAgICAiVmFsdWUiID0gYyhjdl9taW4sIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGN2XzFzZSwgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgYWljYywKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBhaWMsIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGJpYykpCgprYWJsZSh0YWJsZTEpCmBgYAoqKlByaW50IHRoZSB0b3AgdGVuIGJlc3QgYW5kIHdvcnN0IHJldmlldyB3b3JkcyB1bmRlciB5b3VyIHByZWZlcnJlZCBJQyBzZWxlY3Rpb24gcnVsZS4gRG8gdGhlIHJlc3VsdHMgbWFrZSBzZW5zZT8qKgpgYGB7cn0Kd29yZHMgPSBkYXRhLmZyYW1lKCJjb2VmIiA9IGFzLm1hdHJpeChjb2VmKGN2LmJhZHJldmlldykpLAogICAgICAgICAgICAgICAgICAgImFicy52YWwiID0gYWJzKGFzLm1hdHJpeChjb2VmKGN2LmJhZHJldmlldykpKSwKICAgICAgICAgICAgICAgICAgICJ3b3JkIiA9IHJvd25hbWVzKGNvZWYoY3YuYmFkcmV2aWV3KSkKICAgICAgICAgICAgICAgICAgICkKCndvcmRzID0gYXJyYW5nZSh3b3JkcywgZGVzYyh3b3JkcyRzZWczMy4xKSkKY29sbmFtZXMod29yZHMpID0gYygiQ29lZmZpY2llbnQgVmFsdWUiLCAiQ29lZmZpY2llbnQgQWJzb2x1dGUgVmFsdWUiLCAiV29yZCIpCgprYWJsZShoZWFkKHdvcmRzLCAxMCkpCmBgYAoKWWVzLCB0aGVzZSB3b3JkcyBhbGwgbWFrZSBzZW5zZSB0aGV5IGFyZSBhbGwgbmVnYXRpdmUuIEFkZGl0aW9uYWxsIGFsbCBvZiB0aGVpciBjb2VmZmljaWVudHMgYXJlIGdyZWF0ZXIgdGhhbiAxIG1lYW5pbmcgdGhhdCBhbnkgb25lIG9mIHRoZXNlIHdvcmRzIGluZGVwZW5kZW50bHkgaW5jcmVhc2UgdGhlIG9kZHMgb2YgYGJhZC5yZXZpZXdgIGJlaW5nIDEuCioqKgoKIyMjIDQgTXVsdGlub21pYWwgUmVncmVzc2lvbgpOb3csIHdl4oCZbGwgY29uc2lkZXIgc3RhcnMgYXMgYW4gdW5vcmRlcmVkIDUtbGV2ZWwgZmFjdG9yLiBVc2UgdGhlIGNvZGUgaW4geW91ciBzdGFydGVyIHNjcmlwdCB0byBydW4gYSBtdWx0aW5vbWlhbCBsb2dpc3RpYyByZWdyZXNzaW9uLgoKKio0LjEqKlwKCioqV2hhdCBpcyB0aGUgY2hhbmdlIGluIG9kZHMgb2YgMSB2cyAyLTMgc3RhcnMgZm9yIGFuIGV4dHJhIGNvdW50IG9mIHdvcmQgY3JhcCBpbiB0aGUgcmV2aWV3PyoqCgpgYGB7ciBtZXNzYWdlPUZBTFNFLCB3YXJuaW5nPUZBTFNFLCBwYWdlZC5wcmludD1GQUxTRX0KbXVsdGkuY29lZiA9IGFzLmRhdGEuZnJhbWUoYXMubWF0cml4KEJtbikpICU+JXJvd25hbWVzX3RvX2NvbHVtbigpICU+JSBmaWx0ZXIocm93bmFtZSA9PSAnY3JhcCcgfHJvd25hbWUgPT0gJ0NyYXAnKQoKa2FibGUobXVsdGkuY29lZikKCm9uZV9zdGFyID0gZXhwKG11bHRpLmNvZWYkYDFgKQoKZm91cl9zdGFyID0gZXhwKG11bHRpLmNvZWYkYDRgKQpgYGAKCmlmIHRoZSB3b3JkICdjcmFwJyBpcyBpbiB0aGUgcmV2aWV3LCB0aGUgY2hhbmdlIGluIG9kZHMgaXMgYHIgb25lX3N0YXJgLCB3aGlsZSBpdCBpcyB6ZXJvIGZvciBib3RoIDIgYW5kIDMsIGFuZCB0aGVuIGl0IGNoYW5nZXMgYWdhaW4gZm9yIDQgc3RhcnMgd2hlcmUgdGhlIGxpa2VsaWhvb2QgcmF0aW8gaXMgYHIgZm91cl9zdGFyYC4gVGhpcyBtZWFucyB0aGF0IGlmIHRoZSB3b3JkIGNyYXAgYXBwZWFzciBpbiBhIHJldmlldywgaXQgaXMgMTElIG1vcmUgbGlrZWx5IHRvIGJlIGEgb25lLXN0YXIgcmV2aWV3LCBhbmQgYXBwcm94aW1hdGVseSAxOCUgbGVzcyBsaWtlbHkgdG8gYmUgYSBmb3VyIHN0YXIgcmV2aWV3LiAKCioqNC4yKipcCgoqKldoYXQgYXJlIHRoZSBjaGFuZ2VzIGluIG9kZHMgb2YgMSBzdGFyIHZzIDIgYW5kIDEgc3RhciB2cyA1IHN0YXIgcmF0aW5ncyBmb3IgYW4gZXh0cmEgY291bnQgb2Ygd29yZCBmYW50YXN0aWMgaW4gdGhlIHJldmlldz8qKgoKYGBge3IgbWVzc2FnZT1GQUxTRSwgd2FybmluZz1GQUxTRSwgcGFnZWQucHJpbnQ9RkFMU0V9Cm11bHRpLmNvZWYyID0gYXMuZGF0YS5mcmFtZShhcy5tYXRyaXgoQm1uKSkgJT4lcm93bmFtZXNfdG9fY29sdW1uKCkgJT4lIGZpbHRlcihyb3duYW1lID09ICdmYW50YXN0aWMnIHxyb3duYW1lID09ICdGYW50YXN0aWMnIHwgcm93bmFtZSA9PSAnRkFOVEFTVElDJykKCmthYmxlKG11bHRpLmNvZWYyKQoKb25lX3N0YXIgPSBleHAobXVsdGkuY29lZjIkYDFgKQoKdHdvX3N0YXIgPSBleHAobXVsdGkuY29lZjIkYDJgKQoKdGhyZWVfc3RhciA9IGV4cChtdWx0aS5jb2VmMiRgM2ApCgpmb3VyX3N0YXIgPSBleHAobXVsdGkuY29lZjIkYDRgKQoKZml2ZV9zdGFyID0gZXhwKG11bHRpLmNvZWYyJGA1YCkKCnN0YXJzID0gZGF0YS5mcmFtZSgiUmF0aW5nIiA9IGMoIk9uZSBTdGFyIiwgIlR3byBTdGFyIiwiVGhyZWUgU3RhciIsIkZvdXIgU3RhciIsIkZpdmUgU3RhciIpLAogICAgICAgICAgICAgICAgICAgIkxpa2VsaWhvb2QgUmF0aW8iID0gYyhvbmVfc3RhciwgdHdvX3N0YXIsIHRocmVlX3N0YXIsIGZvdXJfc3RhciwgZml2ZV9zdGFyKQogICAgICAgICAgICAgICAgICAgKQpgYGAKCmBgYHtyIG1lc3NhZ2U9RkFMU0UsIHdhcm5pbmc9RkFMU0UsIHBhZ2VkLnByaW50PUZBTFNFfQprYWJsZShzdGFycykKYGBgCgpXZSBjYW4gc2VlIGhlcmUgdGhhdCB1c2luZyB0aGUgd29yZCBmYW50YXN0aWMgaW4gYSByZXZpZXcgbGVhZHMgdG8gYW4gMTglIG1vcmUgbGlrZWx5IHRvIGJlIGEgZml2ZSBzdGFyIHJldmlldywgd2hpbGUgaXQgbWVhbnMgaXQgaXMgMyUgbGVzcyBsaWtlbHkgdG8gYmUgYSBmb3VyIHN0YXIgcmV2aWV3LCAyNSUgbGVzcyBsaWtlbHkgdG8gYmUgYSB0aHJlZSBzdGFyIHJldmlldywgNDglIGxlc3MgbGlrZWx5IHRvIGJlIGEgdHdvIHN0YXIgcmV2aWV3IGFuZCBpbnRlcmVzdGx5IGl0IGlzIDIyJSBsZXNzIGxpa2VseSB0byBiZSBhIDEgc3RhciByZXZpZXcuICA=