Predicting housing price near Moscow
The data is from sberbank housing price competition from kaggle.
We will need to download the data from kaggle and read in the data. (I have already download it in my local directory. )
Read data in R and preprocess it for training the model.
# Load the package for read in data.
library(data.table)
# Read in the data in the local directory
sberbank=fread("train.csv",header=TRUE)
Read 98.5% of 30471 rows
Read 30471 rows and 292 (of 292) columns from 0.043 GB file in 00:00:03# Select rows of data with no missing value (For simplicity, we do not deal with missing value in the data here)
sberbank=sberbank[complete.cases(sberbank),]
sberbank$timestamp=as.Date(sberbank$timestamp)Some data cleaning
change character values into factors and delete some values for simplicity of demonstration.
library(tidyverse)
# sberbank=sberbank%>%select(-water_1line,-culture_objects_top_25,-thermal_power_plant_raion,-incineration_raion ,-oil_chemistry_raion,-radiation_raion ,-railroad_terminal_raion,-big_market_raion,-nuclear_reactor_raion , -detention_facility_raion)
sberbank$product_type=as.factor(sberbank$product_type)
sberbank$sub_area=as.factor(sberbank$product_type)
# To decrease the time for trainig the model, we only select several variables for training.
sberbank=sberbank%>%select(price_doc,timestamp,full_sq,life_sq,floor,max_floor,material,build_year,num_room,kitch_sq,state,product_type,sub_area)More data preprocessing
##############################################################################################
# Record the time
t1=proc.time()
# Load libraries
library(caret) # Package for machine learning algorithms
library(doParallel) # Package for parallel computing
library(MLmetrics) # Package for calculating model performance
library(tidyverse) # Package for data manipulation
# PreProcess the dependent variable data: center,scale,zv,nzv
data1=sberbank%>%select(-price_doc)
data1=data1%>%preProcess("center","scale","zv")%>%predict(newdata=data1)
data1=cbind(data1,price_doc=sberbank$price_doc)Split the data into training and testing data to measure model performance.
# Split the data
data=data1
set.seed(3) # For reproducible purpose
index=createDataPartition(data$price_doc,p=0.9,list=FALSE)
train=data[index,]
test=data[-index,]train the random forest model with parallel backend using caret package
# Define number of cores to use in training the model
cl_num=3
# Register parallel backend
cl <- makeCluster(cl_num)
registerDoParallel(cl)
# Set the training control
ctrl=trainControl(method="repeatedcv",number=5,repeats = 1)
# Set the parameter search grid
max_split=length(colnames(data))-1
rf_grid=expand.grid(mtry=2:max_split)
set.seed(3) # For reproducible purpose
rf=train(log(price_doc)~., data=train, method="rf", trControl=ctrl,metric="RMSE", tuneGrid=rf_grid,importance=TRUE)Loading required package: randomForest
randomForest 4.6-12
Type rfNews() to see new features/changes/bug fixes.
Attaching package: 愼㸱愼㹥randomForest愼㸱愼㹦
The following object is masked from 愼㸱愼㹥package:dplyr愼㸱愼㹦:
combine
The following object is masked from 愼㸱愼㹥package:ggplot2愼㸱愼㹦:
marginmodel performance for random forest
varImp(rf)
rf variable importance
Overall
full_sq 100.00
build_year 90.35
kitch_sq 75.45
max_floor 68.76
life_sq 66.39
state 47.93
num_room 47.04
material 33.41
floor 32.25
product_typeOwnerOccupier 25.94
sub_areaOwnerOccupier 20.70
timestamp 0.00Train the famous XGboost model invented by Tianqi Chen used in the kaggle competition to find Higgs Boson. The God particle if you remembered.
First, we will use the xgbLinear model
gc() # grabage collection used (Mb) gc trigger (Mb) max used (Mb)
Ncells 2360247 126.1 3886542 207.6 3886542 207.6
Vcells 9427190 72.0 25856534 197.3 37429521 285.6# Define number of cores to use in training the model
cl_num=3
# Register parallel backend
cl <- makeCluster(cl_num)
registerDoParallel(cl)
# Set the training control
ctrl=trainControl(method="repeatedcv",number=5,repeats = 1)
# Set the parameter search grid
max_split=length(colnames(data))-1
rf_grid=expand.grid(mtry=2:max_split)
set.seed(3) # For reproducible purpose
xgbLinear=train(log(price_doc)~., data=train, method="xgbLinear", trControl=ctrl,metric="RMSE", tuneLength=10,importance=TRUE)1 package is needed for this model and is not installed. (xgboost). Would you like to try to install it now?
1: yes
2: noyesInstalling package into 愼㸱愼㹥C:/Users/Edward cooper/Documents/R/win-library/3.4愼㸱愼㹦
(as 愼㸱愼㹥lib愼㸱愼㹦 is unspecified)
cannot open URL 'http://www.stats.ox.ac.uk/pub/RWin/src/contrib/PACKAGES.rds': HTTP status was '404 Not Found'cannot open URL 'http://www.stats.ox.ac.uk/pub/RWin/bin/windows/contrib/3.4/PACKAGES.rds': HTTP status was '404 Not Found'trying URL 'https://cran.rstudio.com/bin/windows/contrib/3.4/xgboost_0.6-4.zip'
Content type 'application/zip' length 1693795 bytes (1.6 MB)
downloaded 1.6 MBpackage ‘xgboost’ successfully unpacked and MD5 sums checked
The downloaded binary packages are in
C:\Users\Edward cooper\AppData\Local\Temp\Rtmp4w3lRX\downloaded_packagesLoading required package: xgboost
Attaching package: 愼㸱愼㹥xgboost愼㸱愼㹦
The following object is masked from 愼㸱愼㹥package:dplyr愼㸱愼㹦:
sliceModel performance of xgbLinear
varImp(xgbLinear)
xgbLinear variable importance
Overall
full_sq 100.0000
timestamp 27.5204
build_year 19.0411
life_sq 11.8901
floor 10.9568
max_floor 10.0016
kitch_sq 7.4839
state 6.7344
material 4.3589
num_room 1.7140
product_typeOwnerOccupier 0.5323
sub_areaOwnerOccupier 0.0000Use the xgbtree model
gc() # grabage collection used (Mb) gc trigger (Mb) max used (Mb)
Ncells 2489206 133.0 3886542 207.6 3886542 207.6
Vcells 9648471 73.7 25856534 197.3 37429521 285.6# Define number of cores to use in training the model
cl_num=3
# Register parallel backend
cl <- makeCluster(cl_num)
registerDoParallel(cl)
# Set the training control
ctrl=trainControl(method="repeatedcv",number=5,repeats = 1)
set.seed(3) # For reproducible purpose
xgbTree=train(log(price_doc)~., data=train, method="xgbTree", trControl=ctrl,metric="RMSE", tuneLength=10,importance=TRUE)Loading required package: plyr
---------------------------------------------------------------------
You have loaded plyr after dplyr - this is likely to cause problems.
If you need functions from both plyr and dplyr, please load plyr first, then dplyr:
library(plyr); library(dplyr)
---------------------------------------------------------------------
Attaching package: 愼㸱愼㹥plyr愼㸱愼㹦
The following objects are masked from 愼㸱愼㹥package:dplyr愼㸱愼㹦:
arrange, count, desc, failwith, id, mutate, rename,
summarise, summarize
The following object is masked from 愼㸱愼㹥package:purrr愼㸱愼㹦:
compactModel performance of xgbTree.
rmsle3
[1] 0.6226297The fourth model is the “shallow” neural network.
What I meant by shallow here is that the number of level of neural networks do not exceed 10 layers.
gc() # grabage collection used (Mb) gc trigger (Mb) max used (Mb)
Ncells 2537445 135.6 3886542 207.6 3886542 207.6
Vcells 10631261 81.2 25856534 197.3 37429521 285.6# Define number of cores to use in training the model
cl_num=2
# Register parallel backend
cl <- makeCluster(cl_num)
registerDoParallel(cl)
# Set the training control
ctrl=trainControl(method="repeatedcv",number=5,repeats = 1)
# parameter grid
nnet_grid=expand.grid(size=c(0.001,0.01,0.1,1,10,100),decay=c(100,10,1,0.1))
set.seed(3) # For reproducible purpose
nnet=train(log(price_doc)~., data=train, method="nnet", trControl=ctrl,metric="RMSE", tuneGrid=nnet_grid,importance=TRUE)There were missing values in resampled performance measures.missing values found in aggregated results# weights: 14
initial value 1190018.505689
iter 10 value 1189999.352664
iter 10 value 1189999.344892
iter 10 value 1189999.339918
final value 1189999.339918
convergedt2=proc.time()
(t2-t1)/60 用户 系统 流逝
0.3941667 0.1545000 416.2190000 gc() used (Mb) gc trigger (Mb) max used (Mb)
Ncells 2537385 135.6 3886542 207.6 3886542 207.6
Vcells 10588913 80.8 25856534 197.3 37429521 285.6nnetNeural Network
5440 samples
12 predictor
No pre-processing
Resampling: Cross-Validated (5 fold, repeated 1 times)
Summary of sample sizes: 4351, 4351, 4351, 4354, 4353
Resampling results across tuning parameters:
size decay RMSE Rsquared
1e-03 0.1 14.79000 NaN
1e-03 1.0 14.79000 NaN
1e-03 10.0 14.79000 NaN
1e-03 100.0 14.79020 7.553303e-05
1e-02 0.1 14.79020 1.115801e-04
1e-02 1.0 14.79020 1.115796e-04
1e-02 10.0 14.79020 1.115801e-04
1e-02 100.0 14.79000 NaN
1e-01 0.1 14.79020 1.115801e-04
1e-01 1.0 14.79020 1.115801e-04
1e-01 10.0 14.79020 4.219260e-04
1e-01 100.0 14.79000 2.496738e-04
1e+00 0.1 14.79002 1.115801e-04
1e+00 1.0 14.79008 1.731596e-03
1e+00 10.0 14.79058 NaN
1e+00 100.0 14.79429 NaN
1e+01 0.1 14.79003 1.778511e-02
1e+01 1.0 14.79007 6.189270e-03
1e+01 10.0 14.79020 7.685211e-03
1e+01 100.0 14.79119 2.651527e-03
1e+02 0.1 NaN NaN
1e+02 1.0 NaN NaN
1e+02 10.0 NaN NaN
1e+02 100.0 NaN NaN
RMSE was used to select the optimal model using the smallest value.
The final values used for the model were size = 0.001 and decay = 10.rmse4=MLmetrics::RMSE(y_pred=nnet%>%predict(test),y_true = test$price_doc)
rmsle4=RMSLE(y_pred = exp(nnet%>%predict(test)),y_true = test$price_doc)
rmse4[1] 11192253rmsle4[1] 14.47386stopImplicitCluster()
#stopCluster(cl)
gc() used (Mb) gc trigger (Mb) max used (Mb)
Ncells 2690094 143.7 4703850 251.3 3886542 207.6
Vcells 11436289 87.3 25856534 197.3 37429521 285.6#varImp(nnet)It seems that nnet considerably underestimated the price compared to other models. And it does not perform so well with only 5000 data points.
cordata=data%>%select(price_doc,full_sq,life_sq,kitch_sq,floor,max_floor,state,material,build_year,num_room)
library(corrplot)
corrplot(cor(cordata, use="complete.obs"))
This is the website for the package which has about 233 machine learning algorithms.
This is my blog. Take a look.
LS0tDQp0aXRsZTogIlByZWRpY3RpbmcgaG91c2luZyBwcmljZSBuZWFyIE1vc2NvdyINCm91dHB1dDogaHRtbF9ub3RlYm9vaw0KLS0tDQoNClRoZSBkYXRhIGlzIGZyb20gc2JlcmJhbmsgaG91c2luZyBwcmljZSBjb21wZXRpdGlvbiBmcm9tIFtrYWdnbGVdKGh0dHBzOi8vd3d3LmthZ2dsZS5jb20vYy9zYmVyYmFuay1ydXNzaWFuLWhvdXNpbmctbWFya2V0KS4gDQoNCldlIHdpbGwgbmVlZCB0byBkb3dubG9hZCB0aGUgZGF0YSBmcm9tIGthZ2dsZSBhbmQgcmVhZCBpbiB0aGUgZGF0YS4gKEkgaGF2ZSBhbHJlYWR5IGRvd25sb2FkIGl0IGluIG15IGxvY2FsIGRpcmVjdG9yeS4gKQ0KDQoNCiMjIFJlYWQgZGF0YSBpbiBSIGFuZCBwcmVwcm9jZXNzIGl0IGZvciB0cmFpbmluZyB0aGUgbW9kZWwuIA0KDQoNCmBgYHtyfQ0KIyBMb2FkIHRoZSBwYWNrYWdlIGZvciByZWFkIGluIGRhdGEuDQpsaWJyYXJ5KGRhdGEudGFibGUpDQojIFJlYWQgaW4gdGhlIGRhdGEgaW4gdGhlIGxvY2FsIGRpcmVjdG9yeQ0Kc2JlcmJhbms9ZnJlYWQoInRyYWluLmNzdiIsaGVhZGVyPVRSVUUpDQoNCiMgU2VsZWN0IHJvd3Mgb2YgZGF0YSB3aXRoIG5vIG1pc3NpbmcgdmFsdWUgKEZvciBzaW1wbGljaXR5LCB3ZSBkbyBub3QgZGVhbCB3aXRoIG1pc3NpbmcgdmFsdWUgaW4gdGhlIGRhdGEgaGVyZSkNCnNiZXJiYW5rPXNiZXJiYW5rW2NvbXBsZXRlLmNhc2VzKHNiZXJiYW5rKSxdDQoNCnNiZXJiYW5rJHRpbWVzdGFtcD1hcy5EYXRlKHNiZXJiYW5rJHRpbWVzdGFtcCkNCmBgYA0KDQojIyBTb21lIGRhdGEgY2xlYW5pbmcNCg0KY2hhbmdlIGNoYXJhY3RlciB2YWx1ZXMgaW50byBmYWN0b3JzIGFuZCBkZWxldGUgc29tZSB2YWx1ZXMgZm9yIHNpbXBsaWNpdHkgb2YgZGVtb25zdHJhdGlvbi4gDQoNCmBgYHtyfQ0KbGlicmFyeSh0aWR5dmVyc2UpDQoNCiMgc2JlcmJhbms9c2JlcmJhbmslPiVzZWxlY3QoLXdhdGVyXzFsaW5lLC1jdWx0dXJlX29iamVjdHNfdG9wXzI1LC10aGVybWFsX3Bvd2VyX3BsYW50X3JhaW9uLC1pbmNpbmVyYXRpb25fcmFpb24gLC1vaWxfY2hlbWlzdHJ5X3JhaW9uLC1yYWRpYXRpb25fcmFpb24gLC1yYWlscm9hZF90ZXJtaW5hbF9yYWlvbiwtYmlnX21hcmtldF9yYWlvbiwtbnVjbGVhcl9yZWFjdG9yX3JhaW9uICAsICAtZGV0ZW50aW9uX2ZhY2lsaXR5X3JhaW9uKQ0KDQoNCnNiZXJiYW5rJHByb2R1Y3RfdHlwZT1hcy5mYWN0b3Ioc2JlcmJhbmskcHJvZHVjdF90eXBlKQ0Kc2JlcmJhbmskc3ViX2FyZWE9YXMuZmFjdG9yKHNiZXJiYW5rJHByb2R1Y3RfdHlwZSkNCg0KIyBUbyBkZWNyZWFzZSB0aGUgdGltZSBmb3IgdHJhaW5pZyB0aGUgbW9kZWwsIHdlIG9ubHkgc2VsZWN0IHNldmVyYWwgdmFyaWFibGVzIGZvciB0cmFpbmluZy4gDQpzYmVyYmFuaz1zYmVyYmFuayU+JXNlbGVjdChwcmljZV9kb2MsdGltZXN0YW1wLGZ1bGxfc3EsbGlmZV9zcSxmbG9vcixtYXhfZmxvb3IsbWF0ZXJpYWwsYnVpbGRfeWVhcixudW1fcm9vbSxraXRjaF9zcSxzdGF0ZSxwcm9kdWN0X3R5cGUsc3ViX2FyZWEpDQoNCmBgYA0KDQpNb3JlIGRhdGEgcHJlcHJvY2Vzc2luZw0KDQoNCmBgYHtyfQ0KDQoNCiMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMgDQojIFJlY29yZCB0aGUgdGltZQ0KdDE9cHJvYy50aW1lKCkNCg0KIyBMb2FkIGxpYnJhcmllcw0KbGlicmFyeShjYXJldCkgICAgICAjIFBhY2thZ2UgZm9yIG1hY2hpbmUgbGVhcm5pbmcgYWxnb3JpdGhtcw0KbGlicmFyeShkb1BhcmFsbGVsKSAjIFBhY2thZ2UgZm9yIHBhcmFsbGVsIGNvbXB1dGluZw0KbGlicmFyeShNTG1ldHJpY3MpICAjIFBhY2thZ2UgZm9yIGNhbGN1bGF0aW5nIG1vZGVsIHBlcmZvcm1hbmNlIA0KbGlicmFyeSh0aWR5dmVyc2UpICAjIFBhY2thZ2UgZm9yIGRhdGEgbWFuaXB1bGF0aW9uDQoNCg0KIyBQcmVQcm9jZXNzIHRoZSBkZXBlbmRlbnQgdmFyaWFibGUgZGF0YTogY2VudGVyLHNjYWxlLHp2LG56dg0KZGF0YTE9c2JlcmJhbmslPiVzZWxlY3QoLXByaWNlX2RvYykNCmRhdGExPWRhdGExJT4lcHJlUHJvY2VzcygiY2VudGVyIiwic2NhbGUiLCJ6diIpJT4lcHJlZGljdChuZXdkYXRhPWRhdGExKQ0KZGF0YTE9Y2JpbmQoZGF0YTEscHJpY2VfZG9jPXNiZXJiYW5rJHByaWNlX2RvYykNCmBgYA0KDQoNCg0KDQojIyBTcGxpdCB0aGUgZGF0YSBpbnRvIHRyYWluaW5nIGFuZCB0ZXN0aW5nIGRhdGEgdG8gbWVhc3VyZSBtb2RlbCBwZXJmb3JtYW5jZS4NCg0KYGBge3J9DQojIFNwbGl0IHRoZSBkYXRhIA0KZGF0YT1kYXRhMQ0Kc2V0LnNlZWQoMykgIyBGb3IgcmVwcm9kdWNpYmxlIHB1cnBvc2UgDQppbmRleD1jcmVhdGVEYXRhUGFydGl0aW9uKGRhdGEkcHJpY2VfZG9jLHA9MC45LGxpc3Q9RkFMU0UpIA0KdHJhaW49ZGF0YVtpbmRleCxdDQp0ZXN0PWRhdGFbLWluZGV4LF0NCmBgYA0KDQoNCiMjIHRyYWluIHRoZSByYW5kb20gZm9yZXN0IG1vZGVsIHdpdGggcGFyYWxsZWwgYmFja2VuZCB1c2luZyBjYXJldCBwYWNrYWdlIA0KDQpgYGB7cn0NCg0KIyBEZWZpbmUgbnVtYmVyIG9mIGNvcmVzIHRvIHVzZSBpbiB0cmFpbmluZyB0aGUgbW9kZWwNCmNsX251bT0zDQoNCiMgUmVnaXN0ZXIgcGFyYWxsZWwgYmFja2VuZA0KY2wgPC0gbWFrZUNsdXN0ZXIoY2xfbnVtKSAgDQpyZWdpc3RlckRvUGFyYWxsZWwoY2wpIA0KDQojIFNldCB0aGUgdHJhaW5pbmcgY29udHJvbA0KY3RybD10cmFpbkNvbnRyb2wobWV0aG9kPSJyZXBlYXRlZGN2IixudW1iZXI9NSxyZXBlYXRzID0gMSkNCg0KIyBTZXQgdGhlIHBhcmFtZXRlciBzZWFyY2ggZ3JpZA0KbWF4X3NwbGl0PWxlbmd0aChjb2xuYW1lcyhkYXRhKSktMQ0KcmZfZ3JpZD1leHBhbmQuZ3JpZChtdHJ5PTI6bWF4X3NwbGl0KSANCg0KDQpzZXQuc2VlZCgzKSAjIEZvciByZXByb2R1Y2libGUgcHVycG9zZSANCg0KcmY9dHJhaW4obG9nKHByaWNlX2RvYyl+LiwgZGF0YT10cmFpbiwgbWV0aG9kPSJyZiIsIHRyQ29udHJvbD1jdHJsLG1ldHJpYz0iUk1TRSIsIHR1bmVHcmlkPXJmX2dyaWQsaW1wb3J0YW5jZT1UUlVFKQ0KDQp0Mj1wcm9jLnRpbWUoKQ0KKHQyLXQxKS82MA0KZ2MoKQ0KYGBgDQoNCiMjIG1vZGVsIHBlcmZvcm1hbmNlIGZvciByYW5kb20gZm9yZXN0IA0KDQpgYGB7cn0NCnJtc2UxPU1MbWV0cmljczo6Uk1TRSh5X3ByZWQ9cmYlPiVwcmVkaWN0KHRlc3QpLHlfdHJ1ZSA9IHRlc3QkcHJpY2VfZG9jKQ0Kcm1zbGUxPVJNU0xFKHlfcHJlZCA9IGV4cChyZiU+JXByZWRpY3QodGVzdCkpLHlfdHJ1ZSA9IHRlc3QkcHJpY2VfZG9jKQ0KDQpybXNlMQ0Kcm1zbGUxDQoNCnN0b3BJbXBsaWNpdENsdXN0ZXIoKQ0Kc3RvcENsdXN0ZXIoY2wpDQpnYygpDQpyZg0KdmFySW1wKHJmKQ0KYGBgDQoNCiMjIFRyYWluIHRoZSBmYW1vdXMgWEdib29zdCBtb2RlbCBpbnZlbnRlZCBieSBUaWFucWkgQ2hlbiB1c2VkIGluIHRoZSBrYWdnbGUgY29tcGV0aXRpb24gdG8gZmluZCBbSGlnZ3MgQm9zb25dKGh0dHBzOi8vd3d3LmthZ2dsZS5jb20vYy9oaWdncy1ib3NvbikuIFRoZSBHb2QgcGFydGljbGUgaWYgeW91IHJlbWVtYmVyZWQuIA0KDQojIyBGaXJzdCwgd2Ugd2lsbCB1c2UgdGhlIHhnYkxpbmVhciBtb2RlbA0KDQpgYGB7cn0NCmdjKCkgIyBncmFiYWdlIGNvbGxlY3Rpb24gDQoNCiMgRGVmaW5lIG51bWJlciBvZiBjb3JlcyB0byB1c2UgaW4gdHJhaW5pbmcgdGhlIG1vZGVsDQpjbF9udW09Mw0KDQojIFJlZ2lzdGVyIHBhcmFsbGVsIGJhY2tlbmQNCmNsIDwtIG1ha2VDbHVzdGVyKGNsX251bSkgIA0KcmVnaXN0ZXJEb1BhcmFsbGVsKGNsKSANCg0KIyBTZXQgdGhlIHRyYWluaW5nIGNvbnRyb2wNCmN0cmw9dHJhaW5Db250cm9sKG1ldGhvZD0icmVwZWF0ZWRjdiIsbnVtYmVyPTUscmVwZWF0cyA9IDEpDQoNCnNldC5zZWVkKDMpICMgRm9yIHJlcHJvZHVjaWJsZSBwdXJwb3NlIA0KDQp4Z2JMaW5lYXI9dHJhaW4obG9nKHByaWNlX2RvYyl+LiwgZGF0YT10cmFpbiwgbWV0aG9kPSJ4Z2JMaW5lYXIiLCB0ckNvbnRyb2w9Y3RybCxtZXRyaWM9IlJNU0UiLCB0dW5lTGVuZ3RoPTEwKQ0KDQp0Mj1wcm9jLnRpbWUoKQ0KKHQyLXQxKS82MA0KZ2MoKQ0KYGBgDQoNCg0KIyMgTW9kZWwgcGVyZm9ybWFuY2Ugb2YgeGdiTGluZWFyDQoNCmBgYHtyfQ0Kcm1zZTI9TUxtZXRyaWNzOjpSTVNFKHlfcHJlZD14Z2JMaW5lYXIlPiVwcmVkaWN0KHRlc3QpLHlfdHJ1ZSA9IHRlc3QkcHJpY2VfZG9jKQ0Kcm1zbGUyPVJNU0xFKHlfcHJlZCA9IGV4cCh4Z2JMaW5lYXIlPiVwcmVkaWN0KHRlc3QpKSx5X3RydWUgPSB0ZXN0JHByaWNlX2RvYykNCg0Kcm1zZTINCnJtc2xlMg0KDQpzdG9wSW1wbGljaXRDbHVzdGVyKCkNCnN0b3BDbHVzdGVyKGNsKQ0KZ2MoKQ0KDQp2YXJJbXAoeGdiTGluZWFyKQ0KYGBgDQoNCg0KDQojIyBVc2UgdGhlIHhnYnRyZWUgbW9kZWwgDQoNCg0KYGBge3J9DQpnYygpICMgZ3JhYmFnZSBjb2xsZWN0aW9uIA0KDQojIERlZmluZSBudW1iZXIgb2YgY29yZXMgdG8gdXNlIGluIHRyYWluaW5nIHRoZSBtb2RlbA0KY2xfbnVtPTMNCg0KIyBSZWdpc3RlciBwYXJhbGxlbCBiYWNrZW5kDQpjbCA8LSBtYWtlQ2x1c3RlcihjbF9udW0pICANCnJlZ2lzdGVyRG9QYXJhbGxlbChjbCkgDQoNCiMgU2V0IHRoZSB0cmFpbmluZyBjb250cm9sDQpjdHJsPXRyYWluQ29udHJvbChtZXRob2Q9InJlcGVhdGVkY3YiLG51bWJlcj01LHJlcGVhdHMgPSAxKQ0KDQpzZXQuc2VlZCgzKSAjIEZvciByZXByb2R1Y2libGUgcHVycG9zZSANCg0KeGdiVHJlZT10cmFpbihsb2cocHJpY2VfZG9jKX4uLCBkYXRhPXRyYWluLCBtZXRob2Q9InhnYlRyZWUiLCB0ckNvbnRyb2w9Y3RybCxtZXRyaWM9IlJNU0UiLCB0dW5lTGVuZ3RoPTEwLGltcG9ydGFuY2U9VFJVRSkNCg0KdDI9cHJvYy50aW1lKCkNCih0Mi10MSkvNjANCmdjKCkNCmBgYA0KDQojIyBNb2RlbCBwZXJmb3JtYW5jZSBvZiB4Z2JUcmVlLiANCg0KYGBge3J9DQpybXNlMz1NTG1ldHJpY3M6OlJNU0UoeV9wcmVkPXhnYlRyZWUlPiVwcmVkaWN0KHRlc3QpLHlfdHJ1ZSA9IHRlc3QkcHJpY2VfZG9jKQ0Kcm1zbGUzPVJNU0xFKHlfcHJlZCA9IGV4cCh4Z2JUcmVlJT4lcHJlZGljdCh0ZXN0KSkseV90cnVlID0gdGVzdCRwcmljZV9kb2MpDQoNCnJtc2UzDQpybXNsZTMNCg0Kc3RvcEltcGxpY2l0Q2x1c3RlcigpDQpzdG9wQ2x1c3RlcihjbCkNCmdjKCkNCg0KdmFySW1wKHhnYkxpbmVhcikNCmBgYA0KDQoNCiMjIFRoZSBmb3VydGggbW9kZWwgaXMgdGhlICJzaGFsbG93IiBuZXVyYWwgbmV0d29yay4NCg0KV2hhdCBJIG1lYW50IGJ5IHNoYWxsb3cgaGVyZSBpcyB0aGF0IHRoZSBudW1iZXIgb2YgbGV2ZWwgb2YgbmV1cmFsIG5ldHdvcmtzIGRvIG5vdCBleGNlZWQgMTAgbGF5ZXJzLiANCg0KYGBge3J9DQpnYygpICMgZ3JhYmFnZSBjb2xsZWN0aW9uIA0KDQojIERlZmluZSBudW1iZXIgb2YgY29yZXMgdG8gdXNlIGluIHRyYWluaW5nIHRoZSBtb2RlbA0KY2xfbnVtPTINCg0KIyBSZWdpc3RlciBwYXJhbGxlbCBiYWNrZW5kDQpjbCA8LSBtYWtlQ2x1c3RlcihjbF9udW0pICANCnJlZ2lzdGVyRG9QYXJhbGxlbChjbCkgDQoNCiMgU2V0IHRoZSB0cmFpbmluZyBjb250cm9sDQpjdHJsPXRyYWluQ29udHJvbChtZXRob2Q9InJlcGVhdGVkY3YiLG51bWJlcj01LHJlcGVhdHMgPSAxKQ0KIyBwYXJhbWV0ZXIgZ3JpZCANCm5uZXRfZ3JpZD1leHBhbmQuZ3JpZChzaXplPWMoMC4wMDEsMC4wMSwwLjEsMSwxMCwxMDApLGRlY2F5PWMoMTAwLDEwLDEsMC4xKSkNCg0KDQpzZXQuc2VlZCgzKSAjIEZvciByZXByb2R1Y2libGUgcHVycG9zZSANCg0Kbm5ldD10cmFpbihsb2cocHJpY2VfZG9jKX4uLCBkYXRhPXRyYWluLCBtZXRob2Q9Im5uZXQiLCB0ckNvbnRyb2w9Y3RybCxtZXRyaWM9IlJNU0UiLCB0dW5lR3JpZD1ubmV0X2dyaWQsaW1wb3J0YW5jZT1UUlVFKQ0KDQp0Mj1wcm9jLnRpbWUoKQ0KKHQyLXQxKS82MA0KZ2MoKQ0KDQpubmV0DQpgYGANCg0KDQoNCmBgYHtyfQ0Kcm1zZTQ9TUxtZXRyaWNzOjpSTVNFKHlfcHJlZD1ubmV0JT4lcHJlZGljdCh0ZXN0KSx5X3RydWUgPSB0ZXN0JHByaWNlX2RvYykNCnJtc2xlND1STVNMRSh5X3ByZWQgPSBleHAobm5ldCU+JXByZWRpY3QodGVzdCkpLHlfdHJ1ZSA9IHRlc3QkcHJpY2VfZG9jKQ0KDQpybXNlNA0Kcm1zbGU0DQoNCnN0b3BJbXBsaWNpdENsdXN0ZXIoKQ0KI3N0b3BDbHVzdGVyKGNsKQ0KZ2MoKQ0KDQojdmFySW1wKG5uZXQpDQpgYGANCg0KSXQgc2VlbXMgdGhhdCBubmV0IGNvbnNpZGVyYWJseSB1bmRlcmVzdGltYXRlZCB0aGUgcHJpY2UgY29tcGFyZWQgdG8gb3RoZXIgbW9kZWxzLiBBbmQgaXQgZG9lcyBub3QgcGVyZm9ybSBzbyB3ZWxsIHdpdGggb25seSA1MDAwIGRhdGEgcG9pbnRzLiANCg0KYGBge3J9DQpjb3JkYXRhPWRhdGElPiVzZWxlY3QocHJpY2VfZG9jLGZ1bGxfc3EsbGlmZV9zcSxraXRjaF9zcSxmbG9vcixtYXhfZmxvb3Isc3RhdGUsbWF0ZXJpYWwsYnVpbGRfeWVhcixudW1fcm9vbSkNCg0KbGlicmFyeShjb3JycGxvdCkNCmNvcnJwbG90KGNvcihjb3JkYXRhLCB1c2U9ImNvbXBsZXRlLm9icyIpKQ0KYGBgDQoNCg0KVGhpcyBpcyB0aGUgW3dlYnNpdGVdKGh0dHBzOi8vdG9wZXBvLmdpdGh1Yi5pby9jYXJldC9hdmFpbGFibGUtbW9kZWxzLmh0bWwpIGZvciB0aGUgcGFja2FnZSB3aGljaCBoYXMgYWJvdXQgMjMzIG1hY2hpbmUgbGVhcm5pbmcgYWxnb3JpdGhtcy4gDQoNCg0KVGhpcyBpcyBteSBbYmxvZ10oaHR0cHM6Ly9waGRzdGF0c3BoeXMud29yZHByZXNzLmNvbS8pLiBUYWtlIGEgbG9vay4g