GLM h2o
start h2o
library(h2o)
library(data.table)
h2o.init() Connection successful!
R is connected to the H2O cluster:
H2O cluster uptime: 2 hours 31 minutes
H2O cluster version: 3.14.0.3
H2O cluster version age: 9 days
H2O cluster name: H2O_started_from_R_r631758_ajy644
H2O cluster total nodes: 1
H2O cluster total memory: 3.15 GB
H2O cluster total cores: 8
H2O cluster allowed cores: 8
H2O cluster healthy: TRUE
H2O Connection ip: localhost
H2O Connection port: 54321
H2O Connection proxy: NA
H2O Internal Security: FALSE
H2O API Extensions: Algos, AutoML, Core V3, Core V4
R Version: R version 3.4.2 (2017-09-28) h2o.removeAll()[1] 0load system dataset
path=system.file("extdata","prostate.csv", package="h2o")
h2o.df<-h2o.importFile(path)
|
| | 0%
|
|=======================================================================| 100%df.R<-as.data.frame(h2o.df)fit linear regression model for prostate cancer data
predictors=c("AGE","RACE", "PSA", "GLEASON")
response="VOL"
model.gaussian<-h2o.glm(y=response, x=predictors, training_frame = h2o.df, family="gaussian")
|
| | 0%
|
|=======================================================================| 100%model.gaussianModel Details:
==============
H2ORegressionModel: glm
Model ID: GLM_model_R_1506968312150_58
GLM Model: summary
family link regularization
1 gaussian identity Elastic Net (alpha = 0.5, lambda = 0.003889 )
number_of_predictors_total number_of_active_predictors number_of_iterations
1 4 4 1
training_frame
1 prostate.hex_sid_b80f_20
Coefficients: glm coefficients
names coefficients standardized_coefficients
1 Intercept -2.285527 15.812921
2 AGE 0.310435 2.026232
3 RACE 4.680541 1.445226
4 PSA 0.029275 0.585439
5 GLEASON -1.243792 -1.358163
H2ORegressionMetrics: glm
** Reported on training data. **
MSE: 328.1284
RMSE: 18.11432
MAE: 14.5371
RMSLE: 1.920943
Mean Residual Deviance : 328.1284
R^2 : 0.02269798
Null Deviance :127584.7
Null D.o.F. :379
Residual Deviance :124688.8
Residual D.o.F. :375
AIC :3291.887fit logistic regression for another response
is.factor(h2o.df$CAPSULE)[1] FALSEh2o.df$CAPSULE<-as.factor(h2o.df$CAPSULE)
model.logistic<-h2o.glm(y="CAPSULE", x=predictors, training_frame = h2o.df, family="binomial")
|
| | 0%
|
|=======================================================================| 100%model.logisticModel Details:
==============
H2OBinomialModel: glm
Model ID: GLM_model_R_1506968312150_59
GLM Model: summary
family link regularization
1 binomial logit Elastic Net (alpha = 0.5, lambda = 4.445E-4 )
number_of_predictors_total number_of_active_predictors number_of_iterations
1 4 4 4
training_frame
1 RTMP_sid_b80f_21
Coefficients: glm coefficients
names coefficients standardized_coefficients
1 Intercept -5.867604 -0.467143
2 AGE -0.020819 -0.135888
3 RACE -0.457165 -0.141160
4 PSA 0.027882 0.557565
5 GLEASON 1.071800 1.170356
H2OBinomialMetrics: glm
** Reported on training data. **
MSE: 0.1786802
RMSE: 0.4227058
LogLoss: 0.5289
Mean Per-Class Error: 0.2598975
AUC: 0.7927644
Gini: 0.5855288
R^2: 0.2571069
Residual Deviance: 401.964
AIC: 411.964
Confusion Matrix (vertical: actual; across: predicted) for F1-optimal threshold:
0 1 Error Rate
0 155 72 0.317181 =72/227
1 31 122 0.202614 =31/153
Totals 186 194 0.271053 =103/380
Maximum Metrics: Maximum metrics at their respective thresholds
metric threshold value idx
1 max f1 0.298968 0.703170 191
2 max f2 0.205231 0.803080 294
3 max f0point5 0.531385 0.687606 108
4 max accuracy 0.531385 0.736842 108
5 max precision 0.997166 1.000000 0
6 max recall 0.087936 1.000000 359
7 max specificity 0.997166 1.000000 0
8 max absolute_mcc 0.298968 0.471116 191
9 max min_per_class_accuracy 0.408870 0.713656 172
10 max mean_per_class_accuracy 0.298968 0.740103 191
Gains/Lift Table: Extract with `h2o.gainsLift(<model>, <data>)` or `h2o.gainsLift(<model>, valid=<T/F>, xval=<T/F>)`multinomial modelling example
my understanding is that it is kind of ordinal logistic reg
how to explain the result, the coefficient looks different for each category?
The larger lambda is, the more the coefficients are shrunk toward zero
iris.h2o<-as.h2o(iris)
|
| | 0%
|
|=======================================================================| 100%is.factor(iris.h2o$Species)[1] TRUEmodel.mul<-h2o.glm(y="Species", x=1:4, training_frame = iris.h2o,family="multinomial")
|
| | 0%
|
|=========================== | 38%
|
|=======================================================================| 100%model.mulModel Details:
==============
H2OMultinomialModel: glm
Model ID: GLM_model_R_1506968312150_61
GLM Model: summary
family link regularization
1 multinomial multinomial Elastic Net (alpha = 0.5, lambda = 8.671E-4 )
number_of_predictors_total number_of_active_predictors number_of_iterations
1 15 11 26
training_frame
1 iris
Coefficients: glm multinomial coefficients
names coefs_class_0 coefs_class_1 coefs_class_2 std_coefs_class_0
1 Intercept 6.496784 -0.408705 -18.679065 -3.969984
2 Sepal.Length -1.522256 1.109945 0.000000 -1.260529
3 Sepal.Width 3.757638 -0.684351 -3.575675 1.637828
4 Petal.Length -2.059447 -0.596759 3.326552 -3.635537
5 Petal.Width -4.436355 -1.466536 7.098211 -3.381557
std_coefs_class_1 std_coefs_class_2
1 -0.016699 -8.596793
2 0.919108 0.000000
3 -0.298285 -1.558516
4 -1.053457 5.872356
5 -1.117849 5.410524
H2OMultinomialMetrics: glm
** Reported on training data. **
Training Set Metrics:
=====================
Extract training frame with `h2o.getFrame("iris")`
MSE: (Extract with `h2o.mse`) 0.01469843
RMSE: (Extract with `h2o.rmse`) 0.1212371
Logloss: (Extract with `h2o.logloss`) 0.05728744
Mean Per-Class Error: 0.02
Null Deviance: (Extract with `h2o.nulldeviance`) 329.5837
Residual Deviance: (Extract with `h2o.residual_deviance`) 17.18623
R^2: (Extract with `h2o.r2`) 0.9779524
AIC: (Extract with `h2o.aic`) NaN
Confusion Matrix: Extract with `h2o.confusionMatrix(<model>,train = TRUE)`)
=========================================================================
Confusion Matrix: Row labels: Actual class; Column labels: Predicted class
setosa versicolor virginica Error Rate
setosa 50 0 0 0.0000 = 0 / 50
versicolor 0 48 2 0.0400 = 2 / 50
virginica 0 1 49 0.0200 = 1 / 50
Totals 50 49 51 0.0200 = 3 / 150
Hit Ratio Table: Extract with `h2o.hit_ratio_table(<model>,train = TRUE)`
=======================================================================
Top-3 Hit Ratios:
k hit_ratio
1 1 0.980000
2 2 1.000000
3 3 1.000000possion model for insurance claim data
library(MASS)
data(Insurance)
#need change Ord.factor to factor in class
class(Insurance$Group) <- "factor"
class(Insurance$Age) <- "factor"
insurance.h2o<-as.h2o(Insurance)
|
| | 0%
|
|=======================================================================| 100%model.poisson<-h2o.glm(y="Claims", x=c("District", "Group","Age"), training_frame = insurance.h2o,family="poisson", lambda=0,compute_p_values = T )
|
| | 0%
|
|=======================================================================| 100%model.poissonModel Details:
==============
H2ORegressionModel: glm
Model ID: GLM_model_R_1506968312150_62
GLM Model: summary
family link regularization number_of_predictors_total
1 poisson log None 9
number_of_active_predictors number_of_iterations training_frame
1 9 6 Insurance
Coefficients: glm coefficients
names coefficients std_error z_value p_value
1 Intercept 4.400347 0.057046 77.136215 0.000000
2 District.2 -0.438219 0.042970 -10.198175 0.000000
3 District.3 -0.915205 0.050323 -18.186521 0.000000
4 District.4 -1.443666 0.061576 -23.445286 0.000000
5 Group.1.5-2l -0.518904 0.042993 -12.069485 0.000000
6 Group.<1l -0.989603 0.050447 -19.616527 0.000000
7 Group.>2l -1.578875 0.063515 -24.858362 0.000000
8 Age.30-35 0.114477 0.068431 1.672894 0.094348
9 Age.<25 -0.567693 0.082717 -6.863095 0.000000
10 Age.>35 1.631471 0.054401 29.989514 0.000000
standardized_coefficients
1 4.400347
2 -0.438219
3 -0.915205
4 -1.443666
5 -0.518904
6 -0.989603
7 -1.578875
8 0.114477
9 -0.567693
10 1.631471
H2ORegressionMetrics: glm
** Reported on training data. **
MSE: 69.76565
RMSE: 8.352583
MAE: 5.888571
RMSLE: 0.3498097
Mean Residual Deviance : 1.895504
R^2 : 0.9860048
Null Deviance :4236.679
Null D.o.F. :63
Residual Deviance :121.3123
Residual D.o.F. :54
AIC :458.6257#without P value
model.poisson<-h2o.glm(y="Claims", x=c("District", "Group","Age"), training_frame = insurance.h2o,family="poisson" )
|
| | 0%
|
|=======================================================================| 100%model.poissonModel Details:
==============
H2ORegressionModel: glm
Model ID: GLM_model_R_1506968312150_63
GLM Model: summary
family link regularization
1 poisson log Elastic Net (alpha = 0.5, lambda = 3.9914 )
number_of_predictors_total number_of_active_predictors number_of_iterations
1 12 9 6
training_frame
1 Insurance
Coefficients: glm coefficients
names coefficients standardized_coefficients
1 Intercept 3.316726 3.316726
2 District.1 0.464501 0.464501
3 District.2 0.014646 0.014646
4 District.3 -0.082078 -0.082078
5 District.4 -0.397069 -0.397069
6 Group.1-1.5l 0.530945 0.530945
7 Group.1.5-2l 0.000000 0.000000
8 Group.<1l -0.084988 -0.084988
9 Group.>2l -0.427080 -0.427080
10 Age.25-29 0.000000 0.000000
11 Age.30-35 0.000000 0.000000
12 Age.<25 -0.243187 -0.243187
13 Age.>35 1.277417 1.277417
H2ORegressionMetrics: glm
** Reported on training data. **
MSE: 784.9197
RMSE: 28.01642
MAE: 16.6272
RMSLE: 0.7233781
Mean Residual Deviance : 8.976903
R^2 : 0.8425425
Null Deviance :4236.679
Null D.o.F. :63
Residual Deviance :574.5218
Residual D.o.F. :54
AIC :911.8352gamma model
gamma.inverse <- h2o.glm(y = "DPROS", x = c("AGE","RACE","CAPSULE","DCAPS","PSA","VOL"), training_frame = h2o.df, family = "gamma", link = "inverse"
)
|
| | 0%
|
|=======================================================================| 100%gamma.inverseModel Details:
==============
H2ORegressionModel: glm
Model ID: GLM_model_R_1506968312150_64
GLM Model: summary
family link regularization
1 gamma inverse Elastic Net (alpha = 0.5, lambda = 4.894E-4 )
number_of_predictors_total number_of_active_predictors number_of_iterations
1 7 7 4
training_frame
1 RTMP_sid_b80f_21
Coefficients: glm coefficients
names coefficients standardized_coefficients
1 Intercept 0.486080 0.439020
2 CAPSULE.0 0.052828 0.052828
3 CAPSULE.1 -0.047643 -0.047643
4 AGE 0.001049 0.006845
5 RACE -0.029714 -0.009175
6 DCAPS -0.068706 -0.021344
7 PSA -0.000641 -0.012816
8 VOL 0.000125 0.002291
H2ORegressionMetrics: glm
** Reported on training data. **
MSE: 0.850493
RMSE: 0.9222218
MAE: 0.7474297
RMSLE: 0.2947758
Mean Residual Deviance : 0.1909795
R^2 : 0.1474464
Null Deviance :82.5134
Null D.o.F. :379
Residual Deviance :72.5722
Residual D.o.F. :372
AIC :NaNgamma.log <- h2o.glm(y="DPROS", x = c("AGE","RACE","CAPSULE","DCAPS","PSA","VOL"), training_frame =
h2o.df, family = "gamma", link = "log")
|
| | 0%
|
|=======================================================================| 100%gamma.inverseModel Details:
==============
H2ORegressionModel: glm
Model ID: GLM_model_R_1506968312150_64
GLM Model: summary
family link regularization
1 gamma inverse Elastic Net (alpha = 0.5, lambda = 4.894E-4 )
number_of_predictors_total number_of_active_predictors number_of_iterations
1 7 7 4
training_frame
1 RTMP_sid_b80f_21
Coefficients: glm coefficients
names coefficients standardized_coefficients
1 Intercept 0.486080 0.439020
2 CAPSULE.0 0.052828 0.052828
3 CAPSULE.1 -0.047643 -0.047643
4 AGE 0.001049 0.006845
5 RACE -0.029714 -0.009175
6 DCAPS -0.068706 -0.021344
7 PSA -0.000641 -0.012816
8 VOL 0.000125 0.002291
H2ORegressionMetrics: glm
** Reported on training data. **
MSE: 0.850493
RMSE: 0.9222218
MAE: 0.7474297
RMSLE: 0.2947758
Mean Residual Deviance : 0.1909795
R^2 : 0.1474464
Null Deviance :82.5134
Null D.o.F. :379
Residual Deviance :72.5722
Residual D.o.F. :372
AIC :NaNtweedie model
library(HDtweedie)
data(auto)
dim(auto$x)[1] 2812 56hist(auto$y)
auto.h2o<-as.h2o(auto)
|
| | 0%
|
|=======================================================================| 100%vars=paste("x.", colnames(auto$x), sep="")
vars [1] "x.CAR_TYPE_2" "x.CAR_TYPE_3" "x.CAR_TYPE_4" "x.CAR_TYPE_5" "x.CAR_TYPE_6"
[6] "x.JOBCLASS_3" "x.JOBCLASS_4" "x.JOBCLASS_5" "x.JOBCLASS_6" "x.JOBCLASS_7"
[11] "x.JOBCLASS_8" "x.JOBCLASS_9" "x.MAX_EDUC_2" "x.MAX_EDUC_3" "x.MAX_EDUC_4"
[16] "x.MAX_EDUC_5" "x.KIDSDRIV" "x.KIDSDRIV2" "x.KIDSDRIV3" "x.TRAVTIME"
[21] "x.TRAVTIME2" "x.TRAVTIME3" "x.BLUEBOOK" "x.BLUEBOOK2" "x.BLUEBOOK3"
[26] "x.NPOLICY" "x.NPOLICY2" "x.NPOLICY3" "x.MVR_PTS" "x.MVR_PTS2"
[31] "x.MVR_PTS3" "x.AGE" "x.AGE2" "x.AGE3" "x.HOMEKIDS"
[36] "x.HOMEKIDS2" "x.HOMEKIDS3" "x.YOJ" "x.YOJ2" "x.YOJ3"
[41] "x.INCOME" "x.INCOME2" "x.INCOME3" "x.HOME_VAL" "x.HOME_VAL2"
[46] "x.HOME_VAL3" "x.SAMEHOME" "x.SAMEHOME2" "x.SAMEHOME3" "x.CAR_USE"
[51] "x.RED_CAR" "x.REVOLKED" "x.GENDER" "x.MARRIED" "x.PARENT1"
[56] "x.AREA" model.tweedie=h2o.glm(y="y", x=vars, training_frame = auto.h2o, family="tweedie")
|
| | 0%
|
|========= | 12%
|
|=======================================================================| 100%model.tweedieModel Details:
==============
H2ORegressionModel: glm
Model ID: GLM_model_R_1506968312150_66
GLM Model: summary
family link regularization
1 tweedie tweedie Elastic Net (alpha = 0.5, lambda = 0.00707 )
number_of_predictors_total number_of_active_predictors number_of_iterations
1 56 52 4
training_frame
1 auto
Coefficients: glm coefficients
names coefficients standardized_coefficients
1 Intercept 0.863057 4.112271
2 x.CAR_TYPE_2 0.189050 0.071024
3 x.CAR_TYPE_3 0.226833 0.102243
4 x.CAR_TYPE_4 1.018826 0.330952
5 x.CAR_TYPE_5 -0.147410 -0.066913
---
names coefficients standardized_coefficients
52 x.RED_CAR 0.063559 0.028627
53 x.REVOLKED 9.622193 3.315487
54 x.GENDER -0.359685 -0.178980
55 x.MARRIED -0.307056 -0.149801
56 x.PARENT1 -0.324900 -0.110097
57 x.AREA 1.570891 0.642906
H2ORegressionMetrics: glm
** Reported on training data. **
MSE: 59.19431
RMSE: 7.693784
MAE: 4.462433
RMSLE: NaN
Mean Residual Deviance : 59.19431
R^2 : 0.2469894
Null Deviance :221051.9
Null D.o.F. :2811
Residual Deviance :166454.4
Residual D.o.F. :2759
AIC :NaNparameter adjust in binomial model
# airline.h2o<-h2o.importFile("http://s3.amazonaws.com/h2o-public-test-data/smallldata/airlines/allyears2k_headers.zip")
airlines<-fread("C:\\Users\\r631758\\Desktop\\r631758\\R codes\\H2O\\exercise\\allyears2k_headers.csv")
airlines$IsDepDelayed<-as.factor(airlines$IsDepDelayed)
airlines$Origin<-as.factor(airlines$Origin)
airline.h2o<-as.h2o(airlines)
|
| | 0%
|
|=======================================================================| 100%#stops the model wehn we reach 10 active predictors
model.bin<-h2o.glm(y="IsDepDelayed", x=c("Year", "Origin"), training_frame=airline.h2o,family="binomial", lambda_search=TRUE, max_active_predictors = 10 )
|
| | 0%
|
|========== | 14%
|
|=======================================================================| 100%print(model.bin)Model Details:
==============
H2OBinomialModel: glm
Model ID: GLM_model_R_1506968312150_67
GLM Model: summary
family link regularization
1 binomial logit Elastic Net (alpha = 0.5, lambda = 0.01586 )
lambda_search
1 nlambda = 100, lambda.max = 0.05832, lambda.min = 0.01586, lambda.1se = -1.0
number_of_predictors_total number_of_active_predictors number_of_iterations
1 133 1 15
training_frame
1 airlines
Coefficients: glm coefficients
names coefficients standardized_coefficients
1 Intercept 26.109626 0.100487
2 Origin.ABE 0.000000 0.000000
3 Origin.ABQ 0.000000 0.000000
4 Origin.ACY 0.000000 0.000000
5 Origin.ALB 0.000000 0.000000
---
names coefficients standardized_coefficients
129 Origin.TRI 0.000000 0.000000
130 Origin.TUL 0.000000 0.000000
131 Origin.TUS 0.000000 0.000000
132 Origin.TYS 0.000000 0.000000
133 Origin.UCA 0.000000 0.000000
134 Year -0.013021 -0.082609
H2OBinomialMetrics: glm
** Reported on training data. **
MSE: 0.2485974
RMSE: 0.4985955
LogLoss: 0.6903323
Mean Per-Class Error: 0.5
AUC: 0.5337234
Gini: 0.06744684
R^2: 0.003106494
Residual Deviance: 60718.86
AIC: 60722.86
Confusion Matrix (vertical: actual; across: predicted) for F1-optimal threshold:
NO YES Error Rate
NO 0 20887 1.000000 =20887/20887
YES 0 23091 0.000000 =0/23091
Totals 0 43978 0.474942 =20887/43978
Maximum Metrics: Maximum metrics at their respective thresholds
metric threshold value idx
1 max f1 0.490943 0.688574 21
2 max f2 0.490943 0.846804 21
3 max f0point5 0.520228 0.583758 12
4 max accuracy 0.526724 0.562781 10
5 max precision 0.559025 0.604302 0
6 max recall 0.490943 1.000000 21
7 max specificity 0.559025 0.962130 0
8 max absolute_mcc 0.497452 0.160897 19
9 max min_per_class_accuracy 0.526724 0.559785 10
10 max mean_per_class_accuracy 0.526724 0.562939 10
Gains/Lift Table: Extract with `h2o.gainsLift(<model>, <data>)` or `h2o.gainsLift(<model>, valid=<T/F>, xval=<T/F>)`example to remove collinear variables
a=runif(100)
b=2*a
c=-3*a+10
df=data.frame(a,b,c)
df.h2o<-as.h2o(df)
|
| | 0%
|
|=======================================================================| 100%h2o.fit=h2o.glm(y="c", x=c("a","b"), training_frame = df.h2o, lambda=0,remove_collinear_columns = TRUE)
|
| | 0%
|
|=======================================================================| 100%h2o.fitModel Details:
==============
H2ORegressionModel: glm
Model ID: GLM_model_R_1506968312150_69
GLM Model: summary
family link regularization number_of_predictors_total
1 gaussian identity None 2
number_of_active_predictors number_of_iterations training_frame
1 1 1 df
Coefficients: glm coefficients
names coefficients standardized_coefficients
1 Intercept 10.000000 8.537735
2 a -3.000000 -0.875174
3 b 0.000000 0.000000
H2ORegressionMetrics: glm
** Reported on training data. **
MSE: 5.808223e-14
RMSE: 2.410025e-07
MAE: 2.02423e-07
RMSLE: 2.46524e-08
Mean Residual Deviance : 5.808223e-14
R^2 : 1
Null Deviance :75.82696
Null D.o.F. :99
Residual Deviance :5.808223e-12
Residual D.o.F. :98
AIC :-2757.904#show P value
h2o.fit=h2o.glm(y="c", x=c("a","b"), training_frame = df.h2o, lambda=0,remove_collinear_columns = TRUE, compute_p_values = TRUE)
|
| | 0%
|
|=======================================================================| 100%h2o.fitModel Details:
==============
H2ORegressionModel: glm
Model ID: GLM_model_R_1506968312150_70
GLM Model: summary
family link regularization number_of_predictors_total
1 gaussian identity None 2
number_of_active_predictors number_of_iterations training_frame
1 1 1 df
Coefficients: glm coefficients
names coefficients std_error z_value p_value
1 Intercept 10.000000 0.000000 16693726276680328.000000 0.000000
2 a -3.000000 0.000000 -2841117515105541.000000 0.000000
3 b 0.000000
standardized_coefficients
1 8.537735
2 -0.875174
3 0.000000
H2ORegressionMetrics: glm
** Reported on training data. **
MSE: 5.808223e-14
RMSE: 2.410025e-07
MAE: 2.02423e-07
RMSLE: 2.46524e-08
Mean Residual Deviance : 5.808223e-14
R^2 : 1
Null Deviance :75.82696
Null D.o.F. :99
Residual Deviance :5.808223e-12
Residual D.o.F. :98
AIC :-2757.904CROSS VALIDATION
path=system.file("extdata","prostate.csv", package="h2o")
h2o.df<-h2o.importFile(path)
|
| | 0%
|
|=======================================================================| 100%df.R<-as.data.frame(h2o.df)
is.factor(h2o.df$CAPSULE)[1] FALSEh2o.df$CAPSULE<-as.factor(h2o.df$CAPSULE)
binomial.fit<-h2o.glm(y = "CAPSULE", x = c("AGE", "RACE", "PSA", "GLEASON"), training_frame = h2o.df, family = "binomial", nfolds = 5)
|
| | 0%
|
|=======================================================================| 100%binomial.fitModel Details:
==============
H2OBinomialModel: glm
Model ID: GLM_model_R_1506968312150_71
GLM Model: summary
family link regularization
1 binomial logit Elastic Net (alpha = 0.5, lambda = 4.445E-4 )
number_of_predictors_total number_of_active_predictors number_of_iterations
1 4 4 4
training_frame
1 RTMP_sid_b80f_28
Coefficients: glm coefficients
names coefficients standardized_coefficients
1 Intercept -5.867604 -0.467143
2 AGE -0.020819 -0.135888
3 RACE -0.457165 -0.141160
4 PSA 0.027882 0.557565
5 GLEASON 1.071800 1.170356
H2OBinomialMetrics: glm
** Reported on training data. **
MSE: 0.1786802
RMSE: 0.4227058
LogLoss: 0.5289
Mean Per-Class Error: 0.2598975
AUC: 0.7927644
Gini: 0.5855288
R^2: 0.2571069
Residual Deviance: 401.964
AIC: 411.964
Confusion Matrix (vertical: actual; across: predicted) for F1-optimal threshold:
0 1 Error Rate
0 155 72 0.317181 =72/227
1 31 122 0.202614 =31/153
Totals 186 194 0.271053 =103/380
Maximum Metrics: Maximum metrics at their respective thresholds
metric threshold value idx
1 max f1 0.298968 0.703170 191
2 max f2 0.205231 0.803080 294
3 max f0point5 0.531385 0.687606 108
4 max accuracy 0.531385 0.736842 108
5 max precision 0.997166 1.000000 0
6 max recall 0.087936 1.000000 359
7 max specificity 0.997166 1.000000 0
8 max absolute_mcc 0.298968 0.471116 191
9 max min_per_class_accuracy 0.408870 0.713656 172
10 max mean_per_class_accuracy 0.298968 0.740103 191
Gains/Lift Table: Extract with `h2o.gainsLift(<model>, <data>)` or `h2o.gainsLift(<model>, valid=<T/F>, xval=<T/F>)`
H2OBinomialMetrics: glm
** Reported on cross-validation data. **
** 5-fold cross-validation on training data (Metrics computed for combined holdout predictions) **
MSE: 0.1840359
RMSE: 0.428994
LogLoss: 0.5414469
Mean Per-Class Error: 0.2805707
AUC: 0.7805707
Gini: 0.5611413
R^2: 0.2348397
Residual Deviance: 411.4997
AIC: 421.4997
Confusion Matrix (vertical: actual; across: predicted) for F1-optimal threshold:
0 1 Error Rate
0 156 71 0.312775 =71/227
1 38 115 0.248366 =38/153
Totals 194 186 0.286842 =109/380
Maximum Metrics: Maximum metrics at their respective thresholds
metric threshold value idx
1 max f1 0.326248 0.678466 185
2 max f2 0.209593 0.797342 290
3 max f0point5 0.506358 0.669711 125
4 max accuracy 0.506358 0.728947 125
5 max precision 0.997722 1.000000 0
6 max recall 0.081435 1.000000 362
7 max specificity 0.997722 1.000000 0
8 max absolute_mcc 0.326248 0.430552 185
9 max min_per_class_accuracy 0.430166 0.712418 172
10 max mean_per_class_accuracy 0.326248 0.719429 185
Gains/Lift Table: Extract with `h2o.gainsLift(<model>, <data>)` or `h2o.gainsLift(<model>, valid=<T/F>, xval=<T/F>)`
Cross-Validation Metrics Summary:
mean sd cv_1_valid cv_2_valid cv_3_valid cv_4_valid
accuracy 0.74392027 0.031016229 0.7671233 0.75 0.67105263 0.8030303
auc 0.7891558 0.016868321 0.81 0.81121284 0.7586207 0.8043685
err 0.25607973 0.031016229 0.23287672 0.25 0.32894737 0.1969697
err_count 19.6 2.9529645 17.0 21.0 25.0 13.0
f0point5 0.6795532 0.04458115 0.6535948 0.7 0.5895197 0.7826087
cv_5_valid
accuracy 0.72839504
auc 0.76157695
err 0.27160493
err_count 22.0
f0point5 0.672043
---
mean sd cv_1_valid cv_2_valid cv_3_valid
precision 0.6602908 0.06375785 0.625 0.6603774 0.54
r2 0.23255497 0.03136122 0.2676437 0.24247195 0.16870648
recall 0.81080955 0.07294119 0.8 0.92105263 0.9310345
residual_deviance 82.299934 7.252882 72.21169 91.974174 87.61202
rmse 0.4279206 0.010554106 0.40609625 0.43320233 0.44290596
specificity 0.6980348 0.09284596 0.75 0.6086956 0.5106383
cv_4_valid cv_5_valid
precision 0.8181818 0.65789473
r2 0.28808242 0.19587028
recall 0.6666667 0.7352941
residual_deviance 67.79964 91.902145
rmse 0.41484427 0.44255415
specificity 0.8974359 0.7234042see one more logistic regression
path = system.file("extdata", "prostate.csv", package= "h2o")
h2o_df = h2o.importFile(path)
|
| | 0%
|
|=======================================================================| 100% h2o_df$CAPSULE = as.factor(h2o_df$CAPSULE)
rand_vec <- h2o.runif(h2o_df, seed = 1234)
train <- h2o_df[rand_vec <= 0.8,]
valid <- h2o_df[rand_vec > 0.8,]
binomial.fit = h2o.glm(y = "CAPSULE", x = c("AGE", "RACE", "PSA", "GLEASON"), training_frame = train,validation_frame = valid, family ="binomial")
|
| | 0%
|
|=======================================================================| 100% print(binomial.fit)Model Details:
==============
H2OBinomialModel: glm
Model ID: GLM_model_R_1506968312150_89
GLM Model: summary
family link regularization
1 binomial logit Elastic Net (alpha = 0.5, lambda = 4.906E-4 )
number_of_predictors_total number_of_active_predictors number_of_iterations
1 4 4 4
training_frame
1 RTMP_sid_b80f_32
Coefficients: glm coefficients
names coefficients standardized_coefficients
1 Intercept -6.763082 -0.351132
2 AGE -0.014531 -0.092497
3 RACE -0.534172 -0.161001
4 PSA 0.025826 0.528141
5 GLEASON 1.174225 1.168650
H2OBinomialMetrics: glm
** Reported on training data. **
MSE: 0.1767783
RMSE: 0.4204501
LogLoss: 0.524557
Mean Per-Class Error: 0.2580786
AUC: 0.7993424
Gini: 0.5986848
R^2: 0.2744021
Residual Deviance: 322.078
AIC: 332.078
Confusion Matrix (vertical: actual; across: predicted) for F1-optimal threshold:
0 1 Error Rate
0 122 56 0.314607 =56/178
1 26 103 0.201550 =26/129
Totals 148 159 0.267101 =82/307
Maximum Metrics: Maximum metrics at their respective thresholds
metric threshold value idx
1 max f1 0.293579 0.715278 157
2 max f2 0.220635 0.804749 240
3 max f0point5 0.559468 0.717256 87
4 max accuracy 0.559468 0.742671 87
5 max precision 0.997360 1.000000 0
6 max recall 0.087348 1.000000 293
7 max specificity 0.997360 1.000000 0
8 max absolute_mcc 0.293579 0.477947 157
9 max min_per_class_accuracy 0.428468 0.724719 141
10 max mean_per_class_accuracy 0.293579 0.741921 157
Gains/Lift Table: Extract with `h2o.gainsLift(<model>, <data>)` or `h2o.gainsLift(<model>, valid=<T/F>, xval=<T/F>)`
H2OBinomialMetrics: glm
** Reported on validation data. **
MSE: 0.1900406
RMSE: 0.4359365
LogLoss: 0.5537088
Mean Per-Class Error: 0.2759354
AUC: 0.7704082
Gini: 0.5408163
R^2: 0.138838
Residual Deviance: 80.84149
AIC: 90.84149
Confusion Matrix (vertical: actual; across: predicted) for F1-optimal threshold:
0 1 Error Rate
0 24 25 0.510204 =25/49
1 1 23 0.041667 =1/24
Totals 25 48 0.356164 =26/73
Maximum Metrics: Maximum metrics at their respective thresholds
metric threshold value idx
1 max f1 0.246031 0.638889 47
2 max f2 0.231766 0.805369 52
3 max f0point5 0.530000 0.585938 25
4 max accuracy 0.539814 0.726027 23
5 max precision 0.985975 1.000000 0
6 max recall 0.231766 1.000000 52
7 max specificity 0.985975 1.000000 0
8 max absolute_mcc 0.246031 0.443625 47
9 max min_per_class_accuracy 0.410158 0.693878 31
10 max mean_per_class_accuracy 0.246031 0.724065 47
Gains/Lift Table: Extract with `h2o.gainsLift(<model>, <data>)` or `h2o.gainsLift(<model>, valid=<T/F>, xval=<T/F>)`Coefficients
binomial.fit@model$coefficients Intercept AGE RACE PSA GLEASON
-6.76308189 -0.01453091 -0.53417212 0.02582645 1.17422540 binomial.fit@model$coefficients_tableCoefficients: glm coefficients
names coefficients standardized_coefficients
1 Intercept -6.763082 -0.351132
2 AGE -0.014531 -0.092497
3 RACE -0.534172 -0.161001
4 PSA 0.025826 0.528141
5 GLEASON 1.174225 1.168650 h2o.confusionMatrix(binomial.fit, valid = FALSE)Confusion Matrix (vertical: actual; across: predicted) for max f1 @ threshold = 0.293579041147606:
0 1 Error Rate
0 122 56 0.314607 =56/178
1 26 103 0.201550 =26/129
Totals 148 159 0.267101 =82/307predict function
valid<-h2o_df[(rand_vec>0.8)& (rand_vec<=0.9),]
test<-h2o_df[rand_vec>0.9,]
binomial.fit = h2o.glm(y = "CAPSULE", x = c("AGE", "RACE", "PSA", "GLEASON"), training_frame = train,validation_frame = valid, family = "binomial")
|
| | 0%
|
|=======================================================================| 100%print(binomial.fit)Model Details:
==============
H2OBinomialModel: glm
Model ID: GLM_model_R_1506968312150_97
GLM Model: summary
family link regularization
1 binomial logit Elastic Net (alpha = 0.5, lambda = 4.906E-4 )
number_of_predictors_total number_of_active_predictors number_of_iterations
1 4 4 4
training_frame
1 RTMP_sid_b80f_32
Coefficients: glm coefficients
names coefficients standardized_coefficients
1 Intercept -6.763082 -0.351132
2 AGE -0.014531 -0.092497
3 RACE -0.534172 -0.161001
4 PSA 0.025826 0.528141
5 GLEASON 1.174225 1.168650
H2OBinomialMetrics: glm
** Reported on training data. **
MSE: 0.1767783
RMSE: 0.4204501
LogLoss: 0.524557
Mean Per-Class Error: 0.2580786
AUC: 0.7993424
Gini: 0.5986848
R^2: 0.2744021
Residual Deviance: 322.078
AIC: 332.078
Confusion Matrix (vertical: actual; across: predicted) for F1-optimal threshold:
0 1 Error Rate
0 122 56 0.314607 =56/178
1 26 103 0.201550 =26/129
Totals 148 159 0.267101 =82/307
Maximum Metrics: Maximum metrics at their respective thresholds
metric threshold value idx
1 max f1 0.293579 0.715278 157
2 max f2 0.220635 0.804749 240
3 max f0point5 0.559468 0.717256 87
4 max accuracy 0.559468 0.742671 87
5 max precision 0.997360 1.000000 0
6 max recall 0.087348 1.000000 293
7 max specificity 0.997360 1.000000 0
8 max absolute_mcc 0.293579 0.477947 157
9 max min_per_class_accuracy 0.428468 0.724719 141
10 max mean_per_class_accuracy 0.293579 0.741921 157
Gains/Lift Table: Extract with `h2o.gainsLift(<model>, <data>)` or `h2o.gainsLift(<model>, valid=<T/F>, xval=<T/F>)`
H2OBinomialMetrics: glm
** Reported on validation data. **
MSE: 0.1575926
RMSE: 0.3969793
LogLoss: 0.4660339
Mean Per-Class Error: 0.1904762
AUC: 0.8441558
Gini: 0.6883117
R^2: 0.3014077
Residual Deviance: 29.82617
AIC: 39.82617
Confusion Matrix (vertical: actual; across: predicted) for F1-optimal threshold:
0 1 Error Rate
0 13 8 0.380952 =8/21
1 0 11 0.000000 =0/11
Totals 13 19 0.250000 =8/32
Maximum Metrics: Maximum metrics at their respective thresholds
metric threshold value idx
1 max f1 0.259624 0.733333 18
2 max f2 0.259624 0.873016 18
3 max f0point5 0.512972 0.727273 10
4 max accuracy 0.512972 0.812500 10
5 max precision 0.897733 1.000000 0
6 max recall 0.259624 1.000000 18
7 max specificity 0.897733 1.000000 0
8 max absolute_mcc 0.259624 0.598662 18
9 max min_per_class_accuracy 0.298221 0.761905 13
10 max mean_per_class_accuracy 0.259624 0.809524 18
Gains/Lift Table: Extract with `h2o.gainsLift(<model>, <data>)` or `h2o.gainsLift(<model>, valid=<T/F>, xval=<T/F>)`#make and export predictions
pred=h2o.predict(binomial.fit, test)
|
| | 0%
|
|=======================================================================| 100%test.R<-as.data.frame(test)
pred predict p0 p1
1 1 0.3529455 0.6470545
2 0 0.8082117 0.1917883
3 1 0.7239732 0.2760268
4 1 0.5898419 0.4101581
5 1 0.4947335 0.5052665
6 1 0.2206710 0.7793290
[41 rows x 3 columns] pred.R<-as.data.frame(pred)
# h2o.exportFile(pred, "C:\\Users\\r631758\\Desktop\\r631758\\R codes\\H2O\\exercise\\pred.csv")
#calculate metrics
pref=h2o.performance(binomial.fit,test)
prefH2OBinomialMetrics: glm
MSE: 0.2153659
RMSE: 0.4640754
LogLoss: 0.622138
Mean Per-Class Error: 0.2912088
AUC: 0.7335165
Gini: 0.467033
R^2: 0.005411726
Residual Deviance: 51.01532
AIC: 61.01532
Confusion Matrix (vertical: actual; across: predicted) for F1-optimal threshold:
0 1 Error Rate
0 16 12 0.428571 =12/28
1 2 11 0.153846 =2/13
Totals 18 23 0.341463 =14/41
Maximum Metrics: Maximum metrics at their respective thresholds
metric threshold value idx
1 max f1 0.276532 0.611111 22
2 max f2 0.231766 0.755814 33
3 max f0point5 0.702216 0.612245 8
4 max accuracy 0.702216 0.756098 8
5 max precision 0.985975 1.000000 0
6 max recall 0.231766 1.000000 33
7 max specificity 0.985975 1.000000 0
8 max absolute_mcc 0.702216 0.398422 8
9 max min_per_class_accuracy 0.546377 0.615385 15
10 max mean_per_class_accuracy 0.276532 0.708791 22
Gains/Lift Table: Extract with `h2o.gainsLift(<model>, <data>)` or `h2o.gainsLift(<model>, valid=<T/F>, xval=<T/F>)`whole example
airlines.h2o<-h2o.importFile("http://s3.amazonaws.com/h2o-public-test-data/smalldata/airlines/allyears2k_headers.zip")
|
| | 0%
|
|=======================================================================| 100%model = h2o.glm(y = "IsDepDelayed", x = c("Year","Origin"), training_frame = airlines.h2o,family = "binomial", nfolds = 5)
|
| | 0%
|
|============================================================== | 87%
|
|=======================================================================| 100%print(paste("full model training auc:",model@model$training_metrics@metrics$AUC))[1] "full model training auc: 0.634873567120227"print(paste("full model cv auc:", model@model$cross_validation_metrics@metrics$AUC))[1] "full model cv auc: 0.628851458142331"for ( i in 1:5){
cv_model_name=model@model$cross_validation_models[[i]]$name
cv_model=h2o.getModel(cv_model_name)
print (paste ("cv fold", i, "training auc:", cv_model@model$training_metrics@metrics$AUC, "validation auc: ", cv_model@model$validation_metrics@metrics$AUC))
}[1] "cv fold 1 training auc: 0.636602101975204 validation auc: 0.623484510803242"
[1] "cv fold 2 training auc: 0.634553986745846 validation auc: 0.62953249244805"
[1] "cv fold 3 training auc: 0.635125978081935 validation auc: 0.628089759202146"
[1] "cv fold 4 training auc: 0.632851269882599 validation auc: 0.635450456795434"
[1] "cv fold 5 training auc: 0.63518039191565 validation auc: 0.628495508054523"LS0tDQp0aXRsZTogIkdMTSBoMm8iDQpvdXRwdXQ6IGh0bWxfbm90ZWJvb2sNCi0tLQ0KDQojc3RhcnQgaDJvDQpgYGB7cn0NCmxpYnJhcnkoaDJvKQ0KbGlicmFyeShkYXRhLnRhYmxlKQ0KaDJvLmluaXQoKQ0KaDJvLnJlbW92ZUFsbCgpDQpgYGANCg0KI2xvYWQgc3lzdGVtIGRhdGFzZXQNCmBgYHtyfQ0KcGF0aD1zeXN0ZW0uZmlsZSgiZXh0ZGF0YSIsInByb3N0YXRlLmNzdiIsIHBhY2thZ2U9ImgybyIpDQpoMm8uZGY8LWgyby5pbXBvcnRGaWxlKHBhdGgpDQpkZi5SPC1hcy5kYXRhLmZyYW1lKGgyby5kZikNCmBgYA0KDQojZml0IGxpbmVhciByZWdyZXNzaW9uIG1vZGVsIGZvciBwcm9zdGF0ZSBjYW5jZXIgZGF0YQ0KYGBge3J9DQpwcmVkaWN0b3JzPWMoIkFHRSIsIlJBQ0UiLCAiUFNBIiwgIkdMRUFTT04iKQ0KcmVzcG9uc2U9IlZPTCINCm1vZGVsLmdhdXNzaWFuPC1oMm8uZ2xtKHk9cmVzcG9uc2UsIHg9cHJlZGljdG9ycywgdHJhaW5pbmdfZnJhbWUgPSBoMm8uZGYsIGZhbWlseT0iZ2F1c3NpYW4iKQ0KbW9kZWwuZ2F1c3NpYW4NCmBgYA0KDQojZml0IGxvZ2lzdGljIHJlZ3Jlc3Npb24gZm9yIGFub3RoZXIgcmVzcG9uc2UNCmBgYHtyfQ0KaXMuZmFjdG9yKGgyby5kZiRDQVBTVUxFKQ0KaDJvLmRmJENBUFNVTEU8LWFzLmZhY3RvcihoMm8uZGYkQ0FQU1VMRSkNCm1vZGVsLmxvZ2lzdGljPC1oMm8uZ2xtKHk9IkNBUFNVTEUiLCB4PXByZWRpY3RvcnMsIHRyYWluaW5nX2ZyYW1lID0gaDJvLmRmLCBmYW1pbHk9ImJpbm9taWFsIikNCm1vZGVsLmxvZ2lzdGljDQpgYGANCg0KI211bHRpbm9taWFsIG1vZGVsbGluZyBleGFtcGxlDQojbXkgdW5kZXJzdGFuZGluZyBpcyB0aGF0IGl0IGlzIGtpbmQgb2Ygb3JkaW5hbCBsb2dpc3RpYyByZWcNCiNob3cgdG8gZXhwbGFpbiB0aGUgcmVzdWx0LCB0aGUgY29lZmZpY2llbnQgbG9va3MgZGlmZmVyZW50IGZvciBlYWNoIGNhdGVnb3J5Pw0KI1RoZSBsYXJnZXIgbGFtYmRhIGlzLCB0aGUgbW9yZSB0aGUgY29lZmZpY2llbnRzIGFyZSBzaHJ1bmsgdG93YXJkIHplcm8NCmBgYHtyfQ0KaXJpcy5oMm88LWFzLmgybyhpcmlzKQ0KaXMuZmFjdG9yKGlyaXMuaDJvJFNwZWNpZXMpDQptb2RlbC5tdWw8LWgyby5nbG0oeT0iU3BlY2llcyIsIHg9MTo0LCB0cmFpbmluZ19mcmFtZSA9IGlyaXMuaDJvLGZhbWlseT0ibXVsdGlub21pYWwiKQ0KbW9kZWwubXVsDQpgYGANCg0KI3Bvc3Npb24gbW9kZWwgZm9yIGluc3VyYW5jZSBjbGFpbSBkYXRhDQpgYGB7cn0NCmxpYnJhcnkoTUFTUykNCmRhdGEoSW5zdXJhbmNlKQ0KI25lZWQgY2hhbmdlIE9yZC5mYWN0b3IgdG8gZmFjdG9yIGluIGNsYXNzDQpjbGFzcyhJbnN1cmFuY2UkR3JvdXApIDwtICJmYWN0b3IiDQpjbGFzcyhJbnN1cmFuY2UkQWdlKSA8LSAiZmFjdG9yIg0KDQppbnN1cmFuY2UuaDJvPC1hcy5oMm8oSW5zdXJhbmNlKQ0KbW9kZWwucG9pc3NvbjwtaDJvLmdsbSh5PSJDbGFpbXMiLCB4PWMoIkRpc3RyaWN0IiwgIkdyb3VwIiwiQWdlIiksIHRyYWluaW5nX2ZyYW1lID0gaW5zdXJhbmNlLmgybyxmYW1pbHk9InBvaXNzb24iLCBsYW1iZGE9MCxjb21wdXRlX3BfdmFsdWVzID0gVCApDQptb2RlbC5wb2lzc29uDQoNCiN3aXRob3V0IFAgdmFsdWUNCm1vZGVsLnBvaXNzb248LWgyby5nbG0oeT0iQ2xhaW1zIiwgeD1jKCJEaXN0cmljdCIsICJHcm91cCIsIkFnZSIpLCB0cmFpbmluZ19mcmFtZSA9IGluc3VyYW5jZS5oMm8sZmFtaWx5PSJwb2lzc29uIiApDQptb2RlbC5wb2lzc29uDQpgYGANCg0KI2dhbW1hIG1vZGVsDQpgYGB7cn0NCmdhbW1hLmludmVyc2UgPC0gaDJvLmdsbSh5ID0gIkRQUk9TIiwgeCA9IGMoIkFHRSIsIlJBQ0UiLCJDQVBTVUxFIiwiRENBUFMiLCJQU0EiLCJWT0wiKSwgdHJhaW5pbmdfZnJhbWUgPSBoMm8uZGYsIGZhbWlseSA9ICJnYW1tYSIsIGxpbmsgPSAiaW52ZXJzZSINCikNCmdhbW1hLmludmVyc2UNCmdhbW1hLmxvZyA8LSBoMm8uZ2xtKHk9IkRQUk9TIiwgeCA9IGMoIkFHRSIsIlJBQ0UiLCJDQVBTVUxFIiwiRENBUFMiLCJQU0EiLCJWT0wiKSwgdHJhaW5pbmdfZnJhbWUgPQ0KaDJvLmRmLCBmYW1pbHkgPSAiZ2FtbWEiLCBsaW5rID0gImxvZyIpDQpnYW1tYS5pbnZlcnNlDQpgYGANCg0KI3R3ZWVkaWUgbW9kZWwNCmBgYHtyfQ0KbGlicmFyeShIRHR3ZWVkaWUpDQpkYXRhKGF1dG8pDQpkaW0oYXV0byR4KQ0KaGlzdChhdXRvJHkpDQoNCmF1dG8uaDJvPC1hcy5oMm8oYXV0bykNCnZhcnM9cGFzdGUoInguIiwgY29sbmFtZXMoYXV0byR4KSwgc2VwPSIiKQ0KdmFycw0KbW9kZWwudHdlZWRpZT1oMm8uZ2xtKHk9InkiLCB4PXZhcnMsIHRyYWluaW5nX2ZyYW1lID0gYXV0by5oMm8sIGZhbWlseT0idHdlZWRpZSIpDQptb2RlbC50d2VlZGllDQpgYGANCg0KI3BhcmFtZXRlciBhZGp1c3QgaW4gYmlub21pYWwgbW9kZWwNCmBgYHtyfQ0KIyBhaXJsaW5lLmgybzwtaDJvLmltcG9ydEZpbGUoImh0dHA6Ly9zMy5hbWF6b25hd3MuY29tL2gyby1wdWJsaWMtdGVzdC1kYXRhL3NtYWxsbGRhdGEvYWlybGluZXMvYWxseWVhcnMya19oZWFkZXJzLnppcCIpDQoNCmFpcmxpbmVzPC1mcmVhZCgiQzpcXFVzZXJzXFxyNjMxNzU4XFxEZXNrdG9wXFxyNjMxNzU4XFxSIGNvZGVzXFxIMk9cXGV4ZXJjaXNlXFxhbGx5ZWFyczJrX2hlYWRlcnMuY3N2IikNCg0KYWlybGluZXMkSXNEZXBEZWxheWVkPC1hcy5mYWN0b3IoYWlybGluZXMkSXNEZXBEZWxheWVkKQ0KYWlybGluZXMkT3JpZ2luPC1hcy5mYWN0b3IoYWlybGluZXMkT3JpZ2luKQ0KYWlybGluZS5oMm88LWFzLmgybyhhaXJsaW5lcykNCiNzdG9wcyB0aGUgbW9kZWwgd2VobiB3ZSByZWFjaCAxMCBhY3RpdmUgcHJlZGljdG9ycw0KbW9kZWwuYmluPC1oMm8uZ2xtKHk9IklzRGVwRGVsYXllZCIsIHg9YygiWWVhciIsICJPcmlnaW4iKSwgdHJhaW5pbmdfZnJhbWU9YWlybGluZS5oMm8sZmFtaWx5PSJiaW5vbWlhbCIsIGxhbWJkYV9zZWFyY2g9VFJVRSwgbWF4X2FjdGl2ZV9wcmVkaWN0b3JzID0gMTAgKQ0KcHJpbnQobW9kZWwuYmluKQ0KDQpgYGANCg0KI2V4YW1wbGUgdG8gcmVtb3ZlIGNvbGxpbmVhciB2YXJpYWJsZXMNCmBgYHtyfQ0KYT1ydW5pZigxMDApDQpiPTIqYQ0KYz0tMyphKzEwDQpkZj1kYXRhLmZyYW1lKGEsYixjKQ0KZGYuaDJvPC1hcy5oMm8oZGYpDQpoMm8uZml0PWgyby5nbG0oeT0iYyIsIHg9YygiYSIsImIiKSwgdHJhaW5pbmdfZnJhbWUgPSBkZi5oMm8sIGxhbWJkYT0wLHJlbW92ZV9jb2xsaW5lYXJfY29sdW1ucyA9IFRSVUUpDQpoMm8uZml0DQoNCiNzaG93IFAgdmFsdWUNCmgyby5maXQ9aDJvLmdsbSh5PSJjIiwgeD1jKCJhIiwiYiIpLCB0cmFpbmluZ19mcmFtZSA9IGRmLmgybywgbGFtYmRhPTAscmVtb3ZlX2NvbGxpbmVhcl9jb2x1bW5zID0gVFJVRSwgY29tcHV0ZV9wX3ZhbHVlcyA9IFRSVUUpDQpoMm8uZml0DQpgYGANCiNDUk9TUyBWQUxJREFUSU9ODQpgYGB7cn0NCnBhdGg9c3lzdGVtLmZpbGUoImV4dGRhdGEiLCJwcm9zdGF0ZS5jc3YiLCBwYWNrYWdlPSJoMm8iKQ0KaDJvLmRmPC1oMm8uaW1wb3J0RmlsZShwYXRoKQ0KZGYuUjwtYXMuZGF0YS5mcmFtZShoMm8uZGYpDQppcy5mYWN0b3IoaDJvLmRmJENBUFNVTEUpDQpoMm8uZGYkQ0FQU1VMRTwtYXMuZmFjdG9yKGgyby5kZiRDQVBTVUxFKQ0KYmlub21pYWwuZml0PC1oMm8uZ2xtKHkgPSAiQ0FQU1VMRSIsIHggPSBjKCJBR0UiLCAiUkFDRSIsICJQU0EiLCAiR0xFQVNPTiIpLCB0cmFpbmluZ19mcmFtZSA9IGgyby5kZiwgZmFtaWx5ID0gImJpbm9taWFsIiwgbmZvbGRzID0gNSkNCmJpbm9taWFsLmZpdA0KYGBgDQoNCiNzZWUgb25lIG1vcmUgbG9naXN0aWMgcmVncmVzc2lvbg0KYGBge3J9DQpwYXRoID0gc3lzdGVtLmZpbGUoImV4dGRhdGEiLCAicHJvc3RhdGUuY3N2IiwgcGFja2FnZT0gImgybyIpDQogaDJvX2RmID0gaDJvLmltcG9ydEZpbGUocGF0aCkNCiBoMm9fZGYkQ0FQU1VMRSA9IGFzLmZhY3RvcihoMm9fZGYkQ0FQU1VMRSkNCiByYW5kX3ZlYyA8LSBoMm8ucnVuaWYoaDJvX2RmLCBzZWVkID0gMTIzNCkNCiB0cmFpbiA8LSBoMm9fZGZbcmFuZF92ZWMgPD0gMC44LF0NCiB2YWxpZCA8LSBoMm9fZGZbcmFuZF92ZWMgPiAwLjgsXQ0KIGJpbm9taWFsLmZpdCA9IGgyby5nbG0oeSA9ICJDQVBTVUxFIiwgeCA9IGMoIkFHRSIsICJSQUNFIiwgIlBTQSIsICJHTEVBU09OIiksIHRyYWluaW5nX2ZyYW1lID0gdHJhaW4sdmFsaWRhdGlvbl9mcmFtZSA9IHZhbGlkLCBmYW1pbHkgPSJiaW5vbWlhbCIpDQogcHJpbnQoYmlub21pYWwuZml0KQ0KYGBgDQoNCiNDb2VmZmljaWVudHMNCmBgYHtyfQ0KIGJpbm9taWFsLmZpdEBtb2RlbCRjb2VmZmljaWVudHMNCiBiaW5vbWlhbC5maXRAbW9kZWwkY29lZmZpY2llbnRzX3RhYmxlDQogaDJvLmNvbmZ1c2lvbk1hdHJpeChiaW5vbWlhbC5maXQsIHZhbGlkID0gRkFMU0UpDQpgYGANCg0KI3ByZWRpY3QgZnVuY3Rpb24NCmBgYHtyfQ0KdmFsaWQ8LWgyb19kZlsocmFuZF92ZWM+MC44KSYgKHJhbmRfdmVjPD0wLjkpLF0NCnRlc3Q8LWgyb19kZltyYW5kX3ZlYz4wLjksXQ0KYmlub21pYWwuZml0ID0gaDJvLmdsbSh5ID0gIkNBUFNVTEUiLCB4ID0gYygiQUdFIiwgIlJBQ0UiLCAiUFNBIiwgIkdMRUFTT04iKSwgdHJhaW5pbmdfZnJhbWUgPSB0cmFpbix2YWxpZGF0aW9uX2ZyYW1lID0gdmFsaWQsIGZhbWlseSA9ICJiaW5vbWlhbCIpDQpwcmludChiaW5vbWlhbC5maXQpDQojbWFrZSBhbmQgZXhwb3J0IHByZWRpY3Rpb25zDQpwcmVkPWgyby5wcmVkaWN0KGJpbm9taWFsLmZpdCwgdGVzdCkNCnRlc3QuUjwtYXMuZGF0YS5mcmFtZSh0ZXN0KQ0KcHJlZA0KcHJlZC5SPC1hcy5kYXRhLmZyYW1lKHByZWQpDQojIGgyby5leHBvcnRGaWxlKHByZWQsICJDOlxcVXNlcnNcXHI2MzE3NThcXERlc2t0b3BcXHI2MzE3NThcXFIgY29kZXNcXEgyT1xcZXhlcmNpc2VcXHByZWQuY3N2IikNCiNjYWxjdWxhdGUgbWV0cmljcw0KcHJlZj1oMm8ucGVyZm9ybWFuY2UoYmlub21pYWwuZml0LHRlc3QpDQpwcmVmDQpgYGANCg0KI3dob2xlIGV4YW1wbGUNCmBgYHtyfQ0KYWlybGluZXMuaDJvPC1oMm8uaW1wb3J0RmlsZSgiaHR0cDovL3MzLmFtYXpvbmF3cy5jb20vaDJvLXB1YmxpYy10ZXN0LWRhdGEvc21hbGxkYXRhL2FpcmxpbmVzL2FsbHllYXJzMmtfaGVhZGVycy56aXAiKQ0KbW9kZWwgPSBoMm8uZ2xtKHkgPSAiSXNEZXBEZWxheWVkIiwgeCA9IGMoIlllYXIiLCJPcmlnaW4iKSwgdHJhaW5pbmdfZnJhbWUgPSBhaXJsaW5lcy5oMm8sZmFtaWx5ID0gImJpbm9taWFsIiwgbmZvbGRzID0gNSkNCnByaW50KHBhc3RlKCJmdWxsIG1vZGVsIHRyYWluaW5nIGF1YzoiLG1vZGVsQG1vZGVsJHRyYWluaW5nX21ldHJpY3NAbWV0cmljcyRBVUMpKQ0KcHJpbnQocGFzdGUoImZ1bGwgbW9kZWwgY3YgYXVjOiIsIG1vZGVsQG1vZGVsJGNyb3NzX3ZhbGlkYXRpb25fbWV0cmljc0BtZXRyaWNzJEFVQykpDQoNCmZvciAoIGkgaW4gMTo1KXsNCiAgY3ZfbW9kZWxfbmFtZT1tb2RlbEBtb2RlbCRjcm9zc192YWxpZGF0aW9uX21vZGVsc1tbaV1dJG5hbWUNCiAgY3ZfbW9kZWw9aDJvLmdldE1vZGVsKGN2X21vZGVsX25hbWUpDQogIHByaW50IChwYXN0ZSAoImN2IGZvbGQiLCBpLCAidHJhaW5pbmcgYXVjOiIsIGN2X21vZGVsQG1vZGVsJHRyYWluaW5nX21ldHJpY3NAbWV0cmljcyRBVUMsICJ2YWxpZGF0aW9uIGF1YzogIiwgY3ZfbW9kZWxAbW9kZWwkdmFsaWRhdGlvbl9tZXRyaWNzQG1ldHJpY3MkQVVDKSkNCn0NCmBgYA0KDQo=