R Notebook
The point of this notebook it to show the usage of Lasso Ridge and Elastic Net modeling techniques. The least squares method reduces error with no bias to reduce variance. The ridge elastic net and ridge regression method introduce bias with the effort to reduce variance. Although this model is not the best use case for these methods this is just a example to be shared.
First we will read in the data. Below this we will build a basic OLS model using many predictors which I have already looked at.
# summary stats
df_training = read.csv(ENTER FILE LOCATION HERE)
str(df_training)First we will build the ols model.Examine its residuals and if we have highly correlated variables. This is where the shrinkage occurs mostly. I will run a quick correlation matrix then get into the coefficient then run the ols model before trying ridge, lasso and elastic net methods.
CLICK NEW WINDOW NEXT TO X for the correlation plt. Much better to see. 1 (high correlation) and 0 (no correlation)
library("corrplot")
idCol = grep("Id",names(df_training)) # find column "id"
df_no_id = df_training[, - idCol] # take out column
nums = sapply(df_no_id, is.numeric) # filter to numerical columns
df_num = df_no_id[ , nums] # create dataframe with numerical columns
df_num[is.na(df_num)] = 0.001 #fill nulls with 0.001
correlations = cor(df_num)
corrplot(correlations,order = "hclust")
As you can see there are some high correlated variables.Running the model talked about above. Note the coefficients.
area_model_streets2 = lm(log(df_training$SalePrice)~log(df_training$LotArea) + log(df_training$GrLivArea) + df_training$Neighborhood + df_training$TotRmsAbvGrd + df_training$GarageCars + df_training$HouseStyle + df_training$OverallCond + df_training$OverallQual)
summary(area_model_streets2)
Call:
lm(formula = log(df_training$SalePrice) ~ log(df_training$LotArea) +
log(df_training$GrLivArea) + df_training$Neighborhood + df_training$TotRmsAbvGrd +
df_training$GarageCars + df_training$HouseStyle + df_training$OverallCond +
df_training$OverallQual)
Residuals:
Min 1Q Median 3Q Max
-1.03949 -0.06932 0.01143 0.08313 0.57066
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 6.5050226 0.1828764 35.571 < 0.0000000000000002 ***
log(df_training$LotArea) 0.0842876 0.0109357 7.708 0.0000000000000239 ***
log(df_training$GrLivArea) 0.5341450 0.0277032 19.281 < 0.0000000000000002 ***
df_training$NeighborhoodBlueste -0.1001010 0.1086708 -0.921 0.357134
df_training$NeighborhoodBrDale -0.1608692 0.0524268 -3.068 0.002192 **
df_training$NeighborhoodBrkSide -0.1722051 0.0435822 -3.951 0.0000815499300887 ***
df_training$NeighborhoodClearCr -0.0102975 0.0495381 -0.208 0.835360
df_training$NeighborhoodCollgCr 0.0026949 0.0391760 0.069 0.945167
df_training$NeighborhoodCrawfor -0.0440432 0.0437473 -1.007 0.314220
df_training$NeighborhoodEdwards -0.1990913 0.0415739 -4.789 0.0000018525736891 ***
df_training$NeighborhoodGilbert -0.0257946 0.0419100 -0.615 0.538340
df_training$NeighborhoodIDOTRR -0.3236899 0.0462723 -6.995 0.0000000000040613 ***
df_training$NeighborhoodMeadowV -0.1413827 0.0524647 -2.695 0.007126 **
df_training$NeighborhoodMitchel -0.0972273 0.0436133 -2.229 0.025949 *
df_training$NeighborhoodNAmes -0.1360661 0.0397123 -3.426 0.000629 ***
df_training$NeighborhoodNoRidge 0.1074856 0.0446045 2.410 0.016090 *
df_training$NeighborhoodNPkVill -0.0718950 0.0600549 -1.197 0.231446
df_training$NeighborhoodNridgHt 0.1406981 0.0404075 3.482 0.000513 ***
df_training$NeighborhoodNWAmes -0.1355513 0.0418501 -3.239 0.001227 **
df_training$NeighborhoodOldTown -0.2991952 0.0410953 -7.281 0.0000000000005483 ***
df_training$NeighborhoodSawyer -0.1359081 0.0424771 -3.200 0.001407 **
df_training$NeighborhoodSawyerW -0.0620460 0.0421185 -1.473 0.140937
df_training$NeighborhoodSomerst 0.0614261 0.0397448 1.546 0.122445
df_training$NeighborhoodStoneBr 0.1536365 0.0465722 3.299 0.000995 ***
df_training$NeighborhoodSWISU -0.1834667 0.0495657 -3.701 0.000223 ***
df_training$NeighborhoodTimber 0.0006565 0.0450941 0.015 0.988386
df_training$NeighborhoodVeenker 0.0218231 0.0583467 0.374 0.708442
df_training$TotRmsAbvGrd -0.0099024 0.0043559 -2.273 0.023157 *
df_training$GarageCars 0.0699860 0.0071809 9.746 < 0.0000000000000002 ***
df_training$HouseStyle1.5Unf 0.0586424 0.0419147 1.399 0.162004
df_training$HouseStyle1Story 0.1004665 0.0156245 6.430 0.0000000001737351 ***
df_training$HouseStyle2.5Fin -0.0489341 0.0554757 -0.882 0.377881
df_training$HouseStyle2.5Unf -0.0359028 0.0461714 -0.778 0.436935
df_training$HouseStyle2Story 0.0006316 0.0155231 0.041 0.967551
df_training$HouseStyleSFoyer 0.1775357 0.0297419 5.969 0.0000000030060245 ***
df_training$HouseStyleSLvl 0.0834466 0.0232612 3.587 0.000345 ***
df_training$OverallCond 0.0540869 0.0038305 14.120 < 0.0000000000000002 ***
df_training$OverallQual 0.0877702 0.0048165 18.223 < 0.0000000000000002 ***
---
Signif. codes: 0 *** 0.001 ** 0.01 * 0.05 . 0.1 1
Residual standard error: 0.144 on 1422 degrees of freedom
Multiple R-squared: 0.8733, Adjusted R-squared: 0.87
F-statistic: 264.9 on 37 and 1422 DF, p-value: < 0.00000000000000022Please note the coefficients that are being shown as you will how the ridge, lasso and elastic net responds when trying to shrink coefficients. THe key difference here is increasing bias for less variance.
We must first make a matrix of all of our variables, then use the elmnet function to look at the ridge, lasso and elastic net coefficient penalizations. Then we will plot the 3 forms; view them from right to left in terms of the penalization factor. Also the axis ticks at the top are the ‘number of non-zero’ variables highlighting the usage of lasso and elastic net for variable selection.
xfactors = model.matrix(~log(df_training$LotArea) + log(df_training$GrLivArea) + df_training$Neighborhood + df_training$TotRmsAbvGrd + df_training$GarageCars + df_training$HouseStyle + df_training$OverallCond + df_training$OverallQual,df_training)
x = as.matrix(data.frame(xfactors))
y=log(df_training$SalePrice)
scaled_y = scale(y, center = TRUE, scale = FALSE)
# run ridge,lasso and elastic net
glmod1 = glmnet(x,y=scaled_y,alpha=0) # ridge
glmod2 = glmnet(x,y=scaled_y,alpha=1) # lasso
glmod3 = glmnet(x,y=scaled_y,alpha=0.2) #elastic net
# plot all 3 side by side
par(mfrow=c(1,3))
plot(glmod1,main = "Ridge Trace",xlab = "Penalization Scale")
plot(glmod2,main = "Lasso Trace",xlab = "Penalization Scale")plot(glmod3,main = "Elastic Net Trace",xlab = "Penalization Scale")
Ridge shrinks the coefficients of the low variance predictor (alpha of 1). Lasso shrinks and does variable selection (alpha of 0) while Elastic net straddles the two (between 1 and 0). Look at this right to left.
Posting the stats of the first coefficients against the cross validated fit of within 1 standard error, you can see the the shrinkage of the coefficients that occured as well as the feature selection for lasso.
cvfitR = cv.glmnet(x, y=log(df_training$SalePrice),alpha=0)
cvfitL = cv.glmnet(x, y=log(df_training$SalePrice),alpha=1)
cvfitEL = cv.glmnet(x, y=log(df_training$SalePrice),alpha=0.2)
summary(area_model_streets2)
Call:
lm(formula = log(df_training$SalePrice) ~ log(df_training$LotArea) +
log(df_training$GrLivArea) + df_training$Neighborhood + df_training$TotRmsAbvGrd +
df_training$GarageCars + df_training$HouseStyle + df_training$OverallCond +
df_training$OverallQual)
Residuals:
Min 1Q Median 3Q Max
-1.03949 -0.06932 0.01143 0.08313 0.57066
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 6.5050226 0.1828764 35.571 < 0.0000000000000002 ***
log(df_training$LotArea) 0.0842876 0.0109357 7.708 0.0000000000000239 ***
log(df_training$GrLivArea) 0.5341450 0.0277032 19.281 < 0.0000000000000002 ***
df_training$NeighborhoodBlueste -0.1001010 0.1086708 -0.921 0.357134
df_training$NeighborhoodBrDale -0.1608692 0.0524268 -3.068 0.002192 **
df_training$NeighborhoodBrkSide -0.1722051 0.0435822 -3.951 0.0000815499300887 ***
df_training$NeighborhoodClearCr -0.0102975 0.0495381 -0.208 0.835360
df_training$NeighborhoodCollgCr 0.0026949 0.0391760 0.069 0.945167
df_training$NeighborhoodCrawfor -0.0440432 0.0437473 -1.007 0.314220
df_training$NeighborhoodEdwards -0.1990913 0.0415739 -4.789 0.0000018525736891 ***
df_training$NeighborhoodGilbert -0.0257946 0.0419100 -0.615 0.538340
df_training$NeighborhoodIDOTRR -0.3236899 0.0462723 -6.995 0.0000000000040613 ***
df_training$NeighborhoodMeadowV -0.1413827 0.0524647 -2.695 0.007126 **
df_training$NeighborhoodMitchel -0.0972273 0.0436133 -2.229 0.025949 *
df_training$NeighborhoodNAmes -0.1360661 0.0397123 -3.426 0.000629 ***
df_training$NeighborhoodNoRidge 0.1074856 0.0446045 2.410 0.016090 *
df_training$NeighborhoodNPkVill -0.0718950 0.0600549 -1.197 0.231446
df_training$NeighborhoodNridgHt 0.1406981 0.0404075 3.482 0.000513 ***
df_training$NeighborhoodNWAmes -0.1355513 0.0418501 -3.239 0.001227 **
df_training$NeighborhoodOldTown -0.2991952 0.0410953 -7.281 0.0000000000005483 ***
df_training$NeighborhoodSawyer -0.1359081 0.0424771 -3.200 0.001407 **
df_training$NeighborhoodSawyerW -0.0620460 0.0421185 -1.473 0.140937
df_training$NeighborhoodSomerst 0.0614261 0.0397448 1.546 0.122445
df_training$NeighborhoodStoneBr 0.1536365 0.0465722 3.299 0.000995 ***
df_training$NeighborhoodSWISU -0.1834667 0.0495657 -3.701 0.000223 ***
df_training$NeighborhoodTimber 0.0006565 0.0450941 0.015 0.988386
df_training$NeighborhoodVeenker 0.0218231 0.0583467 0.374 0.708442
df_training$TotRmsAbvGrd -0.0099024 0.0043559 -2.273 0.023157 *
df_training$GarageCars 0.0699860 0.0071809 9.746 < 0.0000000000000002 ***
df_training$HouseStyle1.5Unf 0.0586424 0.0419147 1.399 0.162004
df_training$HouseStyle1Story 0.1004665 0.0156245 6.430 0.0000000001737351 ***
df_training$HouseStyle2.5Fin -0.0489341 0.0554757 -0.882 0.377881
df_training$HouseStyle2.5Unf -0.0359028 0.0461714 -0.778 0.436935
df_training$HouseStyle2Story 0.0006316 0.0155231 0.041 0.967551
df_training$HouseStyleSFoyer 0.1775357 0.0297419 5.969 0.0000000030060245 ***
df_training$HouseStyleSLvl 0.0834466 0.0232612 3.587 0.000345 ***
df_training$OverallCond 0.0540869 0.0038305 14.120 < 0.0000000000000002 ***
df_training$OverallQual 0.0877702 0.0048165 18.223 < 0.0000000000000002 ***
---
Signif. codes: 0 *** 0.001 ** 0.01 * 0.05 . 0.1 1
Residual standard error: 0.144 on 1422 degrees of freedom
Multiple R-squared: 0.8733, Adjusted R-squared: 0.87
F-statistic: 264.9 on 37 and 1422 DF, p-value: < 0.00000000000000022coef(cvfitR, s = "lambda.1se")39 x 1 sparse Matrix of class "dgCMatrix"
1
(Intercept) 8.132080816
X.Intercept. .
log.df_training.LotArea. 0.079494718
log.df_training.GrLivArea. 0.300715348
df_training.NeighborhoodBlueste -0.031144599
df_training.NeighborhoodBrDale -0.119438899
df_training.NeighborhoodBrkSide -0.089869274
df_training.NeighborhoodClearCr 0.083528415
df_training.NeighborhoodCollgCr 0.064687130
df_training.NeighborhoodCrawfor 0.058794169
df_training.NeighborhoodEdwards -0.107009316
df_training.NeighborhoodGilbert 0.028626443
df_training.NeighborhoodIDOTRR -0.230954796
df_training.NeighborhoodMeadowV -0.073360662
df_training.NeighborhoodMitchel -0.017346520
df_training.NeighborhoodNAmes -0.045550789
df_training.NeighborhoodNoRidge 0.184343095
df_training.NeighborhoodNPkVill -0.020817353
df_training.NeighborhoodNridgHt 0.192304750
df_training.NeighborhoodNWAmes -0.025322881
df_training.NeighborhoodOldTown -0.181426510
df_training.NeighborhoodSawyer -0.050123950
df_training.NeighborhoodSawyerW 0.015125198
df_training.NeighborhoodSomerst 0.113176343
df_training.NeighborhoodStoneBr 0.215020322
df_training.NeighborhoodSWISU -0.085009088
df_training.NeighborhoodTimber 0.079561392
df_training.NeighborhoodVeenker 0.124861534
df_training.TotRmsAbvGrd 0.015224126
df_training.GarageCars 0.086870637
df_training.HouseStyle1.5Unf -0.027687205
df_training.HouseStyle1Story 0.042931920
df_training.HouseStyle2.5Fin -0.022953865
df_training.HouseStyle2.5Unf -0.056561633
df_training.HouseStyle2Story -0.002595278
df_training.HouseStyleSFoyer 0.068751030
df_training.HouseStyleSLvl 0.031023650
df_training.OverallCond 0.034487148
df_training.OverallQual 0.085115687coef(cvfitL, s = "lambda.1se")39 x 1 sparse Matrix of class "dgCMatrix"
1
(Intercept) 7.2251860724
X.Intercept. .
log.df_training.LotArea. 0.0838157108
log.df_training.GrLivArea. 0.4096969283
df_training.NeighborhoodBlueste .
df_training.NeighborhoodBrDale -0.0572475613
df_training.NeighborhoodBrkSide -0.0490462294
df_training.NeighborhoodClearCr 0.0491690423
df_training.NeighborhoodCollgCr 0.0449165810
df_training.NeighborhoodCrawfor 0.0219733815
df_training.NeighborhoodEdwards -0.0772366682
df_training.NeighborhoodGilbert 0.0003304377
df_training.NeighborhoodIDOTRR -0.2016898903
df_training.NeighborhoodMeadowV .
df_training.NeighborhoodMitchel .
df_training.NeighborhoodNAmes -0.0196379404
df_training.NeighborhoodNoRidge 0.1205339049
df_training.NeighborhoodNPkVill .
df_training.NeighborhoodNridgHt 0.1456391505
df_training.NeighborhoodNWAmes -0.0196388196
df_training.NeighborhoodOldTown -0.1857902021
df_training.NeighborhoodSawyer -0.0018019383
df_training.NeighborhoodSawyerW .
df_training.NeighborhoodSomerst 0.0759860392
df_training.NeighborhoodStoneBr 0.1500389133
df_training.NeighborhoodSWISU -0.0480826294
df_training.NeighborhoodTimber 0.0244285381
df_training.NeighborhoodVeenker 0.0517944794
df_training.TotRmsAbvGrd .
df_training.GarageCars 0.0849268594
df_training.HouseStyle1.5Unf .
df_training.HouseStyle1Story 0.0589529459
df_training.HouseStyle2.5Fin .
df_training.HouseStyle2.5Unf -0.0096063727
df_training.HouseStyle2Story .
df_training.HouseStyleSFoyer 0.0690475234
df_training.HouseStyleSLvl 0.0180100119
df_training.OverallCond 0.0378317065
df_training.OverallQual 0.1101194653coef(cvfitEL, s = "lambda.1se")39 x 1 sparse Matrix of class "dgCMatrix"
1
(Intercept) 7.450183459
X.Intercept. .
log.df_training.LotArea. 0.086232221
log.df_training.GrLivArea. 0.382340264
df_training.NeighborhoodBlueste .
df_training.NeighborhoodBrDale -0.066101517
df_training.NeighborhoodBrkSide -0.057383815
df_training.NeighborhoodClearCr 0.053691031
df_training.NeighborhoodCollgCr 0.049130519
df_training.NeighborhoodCrawfor 0.031838653
df_training.NeighborhoodEdwards -0.079892447
df_training.NeighborhoodGilbert 0.004096141
df_training.NeighborhoodIDOTRR -0.206288558
df_training.NeighborhoodMeadowV .
df_training.NeighborhoodMitchel .
df_training.NeighborhoodNAmes -0.021301000
df_training.NeighborhoodNoRidge 0.135360044
df_training.NeighborhoodNPkVill .
df_training.NeighborhoodNridgHt 0.159483920
df_training.NeighborhoodNWAmes -0.013440287
df_training.NeighborhoodOldTown -0.180660670
df_training.NeighborhoodSawyer -0.007899022
df_training.NeighborhoodSawyerW .
df_training.NeighborhoodSomerst 0.082831010
df_training.NeighborhoodStoneBr 0.165397113
df_training.NeighborhoodSWISU -0.048579899
df_training.NeighborhoodTimber 0.036708680
df_training.NeighborhoodVeenker 0.065168011
df_training.TotRmsAbvGrd .
df_training.GarageCars 0.088167317
df_training.HouseStyle1.5Unf .
df_training.HouseStyle1Story 0.046779943
df_training.HouseStyle2.5Fin .
df_training.HouseStyle2.5Unf -0.010776480
df_training.HouseStyle2Story .
df_training.HouseStyleSFoyer 0.052114997
df_training.HouseStyleSLvl 0.011812849
df_training.OverallCond 0.035481693
df_training.OverallQual 0.104093140Below I predict using the updated coefficients and the run the RMSE calculation to see the respective merit of each model.
Testing the Root mean squared error you see that the RMSE is still better on the original model. In some cases this model will work with many highly correlated variables. In my specific case it might not have been useful.
# predict
predict_ridge = predict(cvfitR,newx = xfactors,s = "lambda.1se")
predict_lasso = predict(cvfitL,newx = xfactors,s = "lambda.1se")
predict_elnet = predict(cvfitEL,newx = xfactors,s = "lambda.1se")
# run RMSE calc
RMSE_streets = sqrt(mean((area_model_streets2$fitted.values - log(df_training$SalePrice))^2))
RMSE_lasso = sqrt(mean((predict_lasso - log(df_training$SalePrice))^2))
RMSE_ridge = sqrt(mean((predict_ridge - log(df_training$SalePrice))^2))
RMSE_elnet = sqrt(mean((predict_elnet- log(df_training$SalePrice))^2))
RMSE_streets[1] 0.1421347RMSE_lasso[1] 0.1475909RMSE_ridge[1] 0.1504958RMSE_elnet[1] 0.1482253LS0tDQp0aXRsZTogIlIgTm90ZWJvb2siDQpvdXRwdXQ6IGh0bWxfbm90ZWJvb2sNCi0tLQ0KDQpUaGUgcG9pbnQgb2YgdGhpcyBub3RlYm9vayBpdCB0byBzaG93IHRoZSB1c2FnZSBvZiBMYXNzbyBSaWRnZSBhbmQgRWxhc3RpYyBOZXQgbW9kZWxpbmcgdGVjaG5pcXVlcy4gVGhlIGxlYXN0IHNxdWFyZXMgbWV0aG9kIHJlZHVjZXMgZXJyb3Igd2l0aCBubyBiaWFzIHRvIHJlZHVjZSB2YXJpYW5jZS4gVGhlIHJpZGdlIGVsYXN0aWMgbmV0IGFuZCByaWRnZSByZWdyZXNzaW9uIG1ldGhvZCBpbnRyb2R1Y2UgYmlhcyB3aXRoIHRoZSBlZmZvcnQgdG8gcmVkdWNlIHZhcmlhbmNlLiBBbHRob3VnaCB0aGlzIG1vZGVsIGlzIG5vdCB0aGUgYmVzdCB1c2UgY2FzZSBmb3IgdGhlc2UgbWV0aG9kcyB0aGlzIGlzIGp1c3QgYSBleGFtcGxlIHRvIGJlIHNoYXJlZC4NCg0KRmlyc3Qgd2Ugd2lsbCByZWFkIGluIHRoZSBkYXRhLiBCZWxvdyB0aGlzIHdlIHdpbGwgYnVpbGQgYSBiYXNpYyBPTFMgbW9kZWwgdXNpbmcgbWFueSBwcmVkaWN0b3JzIHdoaWNoIEkgaGF2ZSBhbHJlYWR5IGxvb2tlZCBhdC4NCg0KDQpgYGB7cn0NCiMgc3VtbWFyeSBzdGF0cw0KZGZfdHJhaW5pbmcgPSByZWFkLmNzdihFTlRFUiBGSUxFIExPQ0FUSU9OIEhFUkUpDQpzdHIoZGZfdHJhaW5pbmcpDQpgYGANCg0KRmlyc3Qgd2Ugd2lsbCBidWlsZCB0aGUgb2xzIG1vZGVsLkV4YW1pbmUgaXRzIHJlc2lkdWFscyBhbmQgaWYgd2UgaGF2ZSBoaWdobHkgY29ycmVsYXRlZCB2YXJpYWJsZXMuIFRoaXMgaXMgd2hlcmUgdGhlIHNocmlua2FnZSBvY2N1cnMgbW9zdGx5LiBJIHdpbGwgcnVuIGEgcXVpY2sgY29ycmVsYXRpb24gbWF0cml4IHRoZW4gZ2V0IGludG8gdGhlIGNvZWZmaWNpZW50IHRoZW4gcnVuIHRoZSBvbHMgbW9kZWwgYmVmb3JlIHRyeWluZyByaWRnZSwgbGFzc28gYW5kIGVsYXN0aWMgbmV0IG1ldGhvZHMuDQoNCkNMSUNLIE5FVyBXSU5ET1cgTkVYVCBUTyBYIGZvciB0aGUgY29ycmVsYXRpb24gcGx0LiBNdWNoIGJldHRlciB0byBzZWUuIDEgKGhpZ2ggY29ycmVsYXRpb24pIGFuZCAwIChubyBjb3JyZWxhdGlvbikNCg0KYGBge3J9DQpsaWJyYXJ5KCJjb3JycGxvdCIpDQoNCmlkQ29sID0gZ3JlcCgiSWQiLG5hbWVzKGRmX3RyYWluaW5nKSkgIyBmaW5kIGNvbHVtbiAiaWQiDQpkZl9ub19pZCA9IGRmX3RyYWluaW5nWywgLSBpZENvbF0gIyB0YWtlIG91dCBjb2x1bW4NCm51bXMgPSAgc2FwcGx5KGRmX25vX2lkLCBpcy5udW1lcmljKSAjIGZpbHRlciB0byBudW1lcmljYWwgY29sdW1ucw0KZGZfbnVtID0gZGZfbm9faWRbICwgbnVtc10gIyBjcmVhdGUgZGF0YWZyYW1lIHdpdGggbnVtZXJpY2FsIGNvbHVtbnMNCmRmX251bVtpcy5uYShkZl9udW0pXSA9IDAuMDAxICNmaWxsIG51bGxzIHdpdGggMC4wMDENCg0KY29ycmVsYXRpb25zID0gY29yKGRmX251bSkNCmNvcnJwbG90KGNvcnJlbGF0aW9ucyxvcmRlciA9ICJoY2x1c3QiKQ0KDQpgYGANCg0KQXMgeW91IGNhbiBzZWUgdGhlcmUgYXJlIHNvbWUgaGlnaCBjb3JyZWxhdGVkIHZhcmlhYmxlcy5SdW5uaW5nIHRoZSBtb2RlbCB0YWxrZWQgYWJvdXQgYWJvdmUuIE5vdGUgdGhlIGNvZWZmaWNpZW50cy4NCg0KYGBge3J9DQphcmVhX21vZGVsX3N0cmVldHMyID0gbG0obG9nKGRmX3RyYWluaW5nJFNhbGVQcmljZSl+bG9nKGRmX3RyYWluaW5nJExvdEFyZWEpICsgbG9nKGRmX3RyYWluaW5nJEdyTGl2QXJlYSkgICsgZGZfdHJhaW5pbmckTmVpZ2hib3Job29kICsgZGZfdHJhaW5pbmckVG90Um1zQWJ2R3JkICsgZGZfdHJhaW5pbmckR2FyYWdlQ2FycyArIGRmX3RyYWluaW5nJEhvdXNlU3R5bGUgKyBkZl90cmFpbmluZyRPdmVyYWxsQ29uZCArIGRmX3RyYWluaW5nJE92ZXJhbGxRdWFsKQ0KDQpzdW1tYXJ5KGFyZWFfbW9kZWxfc3RyZWV0czIpDQoNCmBgYA0KUGxlYXNlIG5vdGUgdGhlIGNvZWZmaWNpZW50cyB0aGF0IGFyZSBiZWluZyBzaG93biBhcyB5b3Ugd2lsbCBob3cgdGhlIHJpZGdlLCBsYXNzbyBhbmQgZWxhc3RpYyBuZXQgcmVzcG9uZHMgd2hlbiB0cnlpbmcgdG8gc2hyaW5rIGNvZWZmaWNpZW50cy4gVEhlIGtleSBkaWZmZXJlbmNlIGhlcmUgaXMgaW5jcmVhc2luZyBiaWFzIGZvciBsZXNzIHZhcmlhbmNlLg0KDQpXZSBtdXN0IGZpcnN0IG1ha2UgYSBtYXRyaXggb2YgYWxsIG9mIG91ciB2YXJpYWJsZXMsIHRoZW4gdXNlIHRoZSBlbG1uZXQgZnVuY3Rpb24gdG8gbG9vayBhdCB0aGUgcmlkZ2UsIGxhc3NvIGFuZCBlbGFzdGljIG5ldCBjb2VmZmljaWVudCBwZW5hbGl6YXRpb25zLiBUaGVuIHdlIHdpbGwgcGxvdCB0aGUgMyBmb3JtczsgdmlldyB0aGVtIGZyb20gcmlnaHQgdG8gbGVmdCBpbiB0ZXJtcyBvZiB0aGUgcGVuYWxpemF0aW9uIGZhY3Rvci4gQWxzbyB0aGUgYXhpcyB0aWNrcyBhdCB0aGUgdG9wIGFyZSB0aGUgJ251bWJlciBvZiBub24temVybycgdmFyaWFibGVzIGhpZ2hsaWdodGluZyB0aGUgdXNhZ2Ugb2YgbGFzc28gYW5kIGVsYXN0aWMgbmV0IGZvciB2YXJpYWJsZSBzZWxlY3Rpb24uDQoNCmBgYHtyfQ0KDQp4ZmFjdG9ycyA9IG1vZGVsLm1hdHJpeCh+bG9nKGRmX3RyYWluaW5nJExvdEFyZWEpICsgbG9nKGRmX3RyYWluaW5nJEdyTGl2QXJlYSkgKyBkZl90cmFpbmluZyROZWlnaGJvcmhvb2QgKyBkZl90cmFpbmluZyRUb3RSbXNBYnZHcmQgKyBkZl90cmFpbmluZyRHYXJhZ2VDYXJzICsgZGZfdHJhaW5pbmckSG91c2VTdHlsZSAgKyBkZl90cmFpbmluZyRPdmVyYWxsQ29uZCArIGRmX3RyYWluaW5nJE92ZXJhbGxRdWFsLGRmX3RyYWluaW5nKQ0KeCA9IGFzLm1hdHJpeChkYXRhLmZyYW1lKHhmYWN0b3JzKSkNCnk9bG9nKGRmX3RyYWluaW5nJFNhbGVQcmljZSkNCnNjYWxlZF95ID0gc2NhbGUoeSwgY2VudGVyID0gVFJVRSwgc2NhbGUgPSBGQUxTRSkNCg0KIyBydW4gcmlkZ2UsbGFzc28gYW5kIGVsYXN0aWMgbmV0DQoNCmdsbW9kMSA9IGdsbW5ldCh4LHk9c2NhbGVkX3ksYWxwaGE9MCkgIyByaWRnZQ0KZ2xtb2QyID0gZ2xtbmV0KHgseT1zY2FsZWRfeSxhbHBoYT0xKSAjIGxhc3NvDQpnbG1vZDMgPSBnbG1uZXQoeCx5PXNjYWxlZF95LGFscGhhPTAuMikgI2VsYXN0aWMgbmV0DQoNCiMgcGxvdCBhbGwgMyBzaWRlIGJ5IHNpZGUNCg0KcGFyKG1mcm93PWMoMSwzKSkNCg0KcGxvdChnbG1vZDEsbWFpbiA9ICJSaWRnZSBUcmFjZSIseGxhYiA9ICJQZW5hbGl6YXRpb24gU2NhbGUiKQ0KcGxvdChnbG1vZDIsbWFpbiA9ICJMYXNzbyBUcmFjZSIseGxhYiA9ICJQZW5hbGl6YXRpb24gU2NhbGUiKQ0KcGxvdChnbG1vZDMsbWFpbiA9ICJFbGFzdGljIE5ldCBUcmFjZSIseGxhYiA9ICJQZW5hbGl6YXRpb24gU2NhbGUiKQ0KDQoNCmBgYA0KDQpSaWRnZSBzaHJpbmtzIHRoZSBjb2VmZmljaWVudHMgb2YgdGhlIGxvdyB2YXJpYW5jZSBwcmVkaWN0b3IgKGFscGhhIG9mIDEpLiBMYXNzbyBzaHJpbmtzIGFuZCBkb2VzIHZhcmlhYmxlIHNlbGVjdGlvbiAoYWxwaGEgb2YgMCkgd2hpbGUgRWxhc3RpYyBuZXQgc3RyYWRkbGVzIHRoZSB0d28gKGJldHdlZW4gMSBhbmQgMCkuIExvb2sgYXQgdGhpcyByaWdodCB0byBsZWZ0Lg0KDQpQb3N0aW5nIHRoZSBzdGF0cyBvZiB0aGUgZmlyc3QgY29lZmZpY2llbnRzIGFnYWluc3QgdGhlIGNyb3NzIHZhbGlkYXRlZCBmaXQgb2Ygd2l0aGluIDEgc3RhbmRhcmQgZXJyb3IsIHlvdSBjYW4gc2VlIHRoZSB0aGUgc2hyaW5rYWdlIG9mIHRoZSBjb2VmZmljaWVudHMgdGhhdCBvY2N1cmVkIGFzIHdlbGwgYXMgdGhlIGZlYXR1cmUgc2VsZWN0aW9uIGZvciBsYXNzby4NCg0KYGBge3J9DQpjdmZpdFIgPSBjdi5nbG1uZXQoeCwgeT1sb2coZGZfdHJhaW5pbmckU2FsZVByaWNlKSxhbHBoYT0wKQ0KY3ZmaXRMID0gY3YuZ2xtbmV0KHgsIHk9bG9nKGRmX3RyYWluaW5nJFNhbGVQcmljZSksYWxwaGE9MSkNCmN2Zml0RUwgPSBjdi5nbG1uZXQoeCwgeT1sb2coZGZfdHJhaW5pbmckU2FsZVByaWNlKSxhbHBoYT0wLjIpDQoNCnN1bW1hcnkoYXJlYV9tb2RlbF9zdHJlZXRzMikNCmNvZWYoY3ZmaXRSLCBzID0gImxhbWJkYS4xc2UiKQ0KY29lZihjdmZpdEwsIHMgPSAibGFtYmRhLjFzZSIpDQpjb2VmKGN2Zml0RUwsIHMgPSAibGFtYmRhLjFzZSIpDQpgYGANCg0KQmVsb3cgSSBwcmVkaWN0IHVzaW5nIHRoZSB1cGRhdGVkIGNvZWZmaWNpZW50cyBhbmQgdGhlIHJ1biB0aGUgUk1TRSBjYWxjdWxhdGlvbiB0byBzZWUgdGhlIHJlc3BlY3RpdmUgbWVyaXQgb2YgZWFjaCBtb2RlbC4NCg0KVGVzdGluZyB0aGUgUm9vdCBtZWFuIHNxdWFyZWQgZXJyb3IgeW91IHNlZSB0aGF0IHRoZSBSTVNFIGlzIHN0aWxsIGJldHRlciBvbiB0aGUgb3JpZ2luYWwgbW9kZWwuIEluIHNvbWUgY2FzZXMgdGhpcyBtb2RlbCB3aWxsIHdvcmsgd2l0aCBtYW55IGhpZ2hseSBjb3JyZWxhdGVkIHZhcmlhYmxlcy4gSW4gbXkgc3BlY2lmaWMgY2FzZSBpdCBtaWdodCBub3QgaGF2ZSBiZWVuIHVzZWZ1bC4NCg0KYGBge3J9DQoNCiMgcHJlZGljdA0KDQpwcmVkaWN0X3JpZGdlID0gcHJlZGljdChjdmZpdFIsbmV3eCA9IHhmYWN0b3JzLHMgPSAibGFtYmRhLjFzZSIpDQpwcmVkaWN0X2xhc3NvID0gcHJlZGljdChjdmZpdEwsbmV3eCA9IHhmYWN0b3JzLHMgPSAibGFtYmRhLjFzZSIpDQpwcmVkaWN0X2VsbmV0ID0gcHJlZGljdChjdmZpdEVMLG5ld3ggPSB4ZmFjdG9ycyxzID0gImxhbWJkYS4xc2UiKQ0KDQojIHJ1biBSTVNFIGNhbGMNCg0KUk1TRV9zdHJlZXRzID0gc3FydChtZWFuKChhcmVhX21vZGVsX3N0cmVldHMyJGZpdHRlZC52YWx1ZXMgLSBsb2coZGZfdHJhaW5pbmckU2FsZVByaWNlKSleMikpDQpSTVNFX2xhc3NvID0gc3FydChtZWFuKChwcmVkaWN0X2xhc3NvIC0gbG9nKGRmX3RyYWluaW5nJFNhbGVQcmljZSkpXjIpKQ0KUk1TRV9yaWRnZSA9IHNxcnQobWVhbigocHJlZGljdF9yaWRnZSAtIGxvZyhkZl90cmFpbmluZyRTYWxlUHJpY2UpKV4yKSkNClJNU0VfZWxuZXQgPSBzcXJ0KG1lYW4oKHByZWRpY3RfZWxuZXQtIGxvZyhkZl90cmFpbmluZyRTYWxlUHJpY2UpKV4yKSkNCg0KDQpSTVNFX3N0cmVldHMNClJNU0VfbGFzc28NClJNU0VfcmlkZ2UNClJNU0VfZWxuZXQNCg0KDQpgYGANCg0KDQoNCg0K