Introduction to machine learning
Daniel He
2026-02-12
- Machine
learning
- Machine learning workflow
- Loading packages and datasets
- Spliting the dataset into training and test data sets
- Implement data imputation
- One-hot-endcoding
- Normalizing features
- Plot features
- Recursive feature elimination (rfe)
- Training
a model
Multivariate Adaptive Regression Splines (MARS) - Prepare the test data set
- Prediction uisng testdata
- Compute confusion matrix
- Tuning hyperparameter to optimize the model
- Other marchine learning algorithms
- Comparisons of different models
- Plot comparisons of models
- Ensemble predictions from multiple models
- KNN Classifier
- KNN regression
- Machine learning workflow
Machine learning
Machine learning (ML) is a collection of methods that learn patterns from data and use those patterns to make predictions or classifications. In practice, ML is most useful when (1) the relationship between predictors and outcome is complex, (2) there are many predictors and potential interactions, or (3) prediction performance is the primary goal.
This chapter focuses on a practical, end-to-end workflow using the
caret ecosystem. The emphasis is on reproducible
steps that appear repeatedly in real work:
1) load data and inspect structure,
2) split data into training and test sets,
3) perform preprocessing (imputation, encoding, normalization),
4) explore features visually,
5) train models and tune hyperparameters,
6) evaluate performance using a confusion matrix and cross-validation,
and
7) compare/ensemble multiple models.
We will use the Pima Indians Diabetes dataset
(PimaIndiansDiabetes) as a standard binary classification
example. The outcome variable is diabetes, and the
predictors are clinical measurements. The goal is to predict whether a
subject has diabetes.
Machine learning workflow
A common mistake in ML is to “peek” at the test data during preprocessing or tuning. In a strict workflow, the test set is held out until the end. All preprocessing models (imputation, encoding, scaling) should be built on training data only, then applied to the test data using the same transformation objects. This prevents information leakage and gives a more honest estimate of performance.
Loading packages and datasets
We first load the dataset. It is good practice to keep dataset names short and consistent, especially in teaching code or iterative modeling scripts.
# load the Pima Indians dataset from the mlbench dataset
library(mlbench)## Warning: package 'mlbench' was built under R version 4.4.3data(PimaIndiansDiabetes)
# rename dataset to have shorter name because lazy
diabetes <- PimaIndiansDiabetes- look at the data set Before modeling, always inspect the dataset: check variable types, outcome coding, and whether any variables have unexpected distributions. This early check often prevents downstream errors.
# install.packages(c('caret', 'skimr', 'RANN', 'randomForest', 'gbm', 'xgboost', 'caretEnsemble', 'C50', 'earth'))
# Load the caret package
library(caret)## Loading required package: ggplot2## Warning: package 'ggplot2' was built under R version 4.4.3## Loading required package: lattice# Structure of the dataframe
str(diabetes)## 'data.frame': 768 obs. of 9 variables:
## $ pregnant: num 6 1 8 1 0 5 3 10 2 8 ...
## $ glucose : num 148 85 183 89 137 116 78 115 197 125 ...
## $ pressure: num 72 66 64 66 40 74 50 0 70 96 ...
## $ triceps : num 35 29 0 23 35 0 32 0 45 0 ...
## $ insulin : num 0 0 0 94 168 0 88 0 543 0 ...
## $ mass : num 33.6 26.6 23.3 28.1 43.1 25.6 31 35.3 30.5 0 ...
## $ pedigree: num 0.627 0.351 0.672 0.167 2.288 ...
## $ age : num 50 31 32 21 33 30 26 29 53 54 ...
## $ diabetes: Factor w/ 2 levels "neg","pos": 2 1 2 1 2 1 2 1 2 2 ...# See top 6 rows
head(diabetes )## pregnant glucose pressure triceps insulin mass pedigree age diabetes
## 1 6 148 72 35 0 33.6 0.627 50 pos
## 2 1 85 66 29 0 26.6 0.351 31 neg
## 3 8 183 64 0 0 23.3 0.672 32 pos
## 4 1 89 66 23 94 28.1 0.167 21 neg
## 5 0 137 40 35 168 43.1 2.288 33 pos
## 6 5 116 74 0 0 25.6 0.201 30 negSpliting the dataset into training and test data sets
Splitting data is essential for evaluating generalization. The training set is used to learn model parameters and tuning, and the test set is used for final evaluation.
Here we use an 80/20 split. createDataPartition()
performs a stratified split when possible, which is important for
classification tasks because it helps preserve the class proportions in
the training and test sets.
# Create the training and test datasets
set.seed(100)
# Step 1: Get row numbers for the training data
trainRowNumbers <- createDataPartition(diabetes$diabetes, p=0.8, list=FALSE)
# Step 2: Create the training dataset
trainData <- diabetes[trainRowNumbers,]
# Step 3: Create the test dataset
testData <- diabetes[-trainRowNumbers,]
# Store X and Y for later use.
# x = trainData[, -1]
y = trainData$diabetes- have a look training data set A quick summary of missingness,
distributions, and variable types helps decide which preprocessing steps
are needed.
skimr::skim()is a convenient way to get a compact overview.
library(skimr)## Warning: package 'skimr' was built under R version 4.4.3skimmed <- skim (trainData)
skimmed| Name | trainData |
| Number of rows | 615 |
| Number of columns | 9 |
| _______________________ | |
| Column type frequency: | |
| factor | 1 |
| numeric | 8 |
| ________________________ | |
| Group variables | None |
Variable type: factor
| skim_variable | n_missing | complete_rate | ordered | n_unique | top_counts |
|---|---|---|---|---|---|
| diabetes | 0 | 1 | FALSE | 2 | neg: 400, pos: 215 |
Variable type: numeric
| skim_variable | n_missing | complete_rate | mean | sd | p0 | p25 | p50 | p75 | p100 | hist |
|---|---|---|---|---|---|---|---|---|---|---|
| pregnant | 0 | 1 | 3.88 | 3.42 | 0.00 | 1.00 | 3.00 | 6.00 | 17.00 | ▇▃▂▁▁ |
| glucose | 0 | 1 | 120.87 | 32.58 | 0.00 | 99.00 | 116.00 | 140.00 | 199.00 | ▁▁▇▅▂ |
| pressure | 0 | 1 | 69.33 | 19.44 | 0.00 | 62.00 | 72.00 | 80.00 | 122.00 | ▁▁▇▇▁ |
| triceps | 0 | 1 | 19.97 | 15.87 | 0.00 | 0.00 | 22.00 | 32.00 | 99.00 | ▇▇▂▁▁ |
| insulin | 0 | 1 | 78.00 | 114.39 | 0.00 | 0.00 | 18.00 | 125.00 | 846.00 | ▇▁▁▁▁ |
| mass | 0 | 1 | 32.08 | 7.97 | 0.00 | 27.50 | 32.00 | 36.80 | 67.10 | ▁▂▇▂▁ |
| pedigree | 0 | 1 | 0.46 | 0.33 | 0.08 | 0.24 | 0.36 | 0.61 | 2.29 | ▇▃▁▁▁ |
| age | 0 | 1 | 33.41 | 11.77 | 21.00 | 24.00 | 29.00 | 41.00 | 81.00 | ▇▃▁▁▁ |
Implement data imputation
Missing data are common in real-world clinical and EHR datasets. Many ML algorithms cannot handle missing values directly, so imputation is a practical preprocessing step.
Important practice note: the imputation model must be fit on the training data only, then applied to both training and test sets using the same fitted object.
- compiling knnimpute model Here we use k-nearest neighbors imputation
(
knnImpute). This method imputes missing values based on similar observations in the feature space.
# Create the knn imputation model on the training data
preProcess_missingdata_model <- preProcess(trainData, method='knnImpute')
preProcess_missingdata_model## Created from 615 samples and 9 variables
##
## Pre-processing:
## - centered (8)
## - ignored (1)
## - 5 nearest neighbor imputation (8)
## - scaled (8)- check missingness After applying imputation, we check if any missing
values remain. If
anyNA(trainData)is still TRUE, we need to investigate which variables were not imputed and why.
# Use the imputation model to predict the values of missing data points
library(RANN) # required for knnInpute## Warning: package 'RANN' was built under R version 4.4.3trainData <- predict(preProcess_missingdata_model, newdata = trainData)
anyNA(trainData)## [1] FALSEOne-hot-endcoding
Many ML algorithms require numeric inputs. Categorical predictors must be converted into numeric features. One-hot encoding (dummy variables) is a standard approach: it converts a categorical variable with K levels into K (or K-1) binary indicators.
- Y (dependent) will not be encoded as one-hot-encoding A practical rule: do not one-hot encode the outcome in typical classification workflows. Keep the outcome as a factor with levels representing classes.
This code creates a dummy-variable model based on the training data, then applies it to generate a numeric feature matrix.
# One-Hot Encoding
# Creating dummy variables is converting a categorical variable to as many binary variables as here are categories.
dummies_model <- dummyVars(diabetes ~ ., data=trainData)
# Create the dummy variables using predict. The Y variable (Purchase) will not be present in trainData_mat.
trainData_mat <- predict(dummies_model, newdata = trainData)## Warning in model.frame.default(Terms, newdata, na.action = na.action, xlev =
## object$lvls): variable 'diabetes' is not a factor# # Convert to dataframe
trainData <- data.frame(trainData_mat)
# # See the structure of the new dataset
str(trainData)## 'data.frame': 615 obs. of 8 variables:
## $ pregnant: num -0.843 1.205 -0.843 -1.136 0.327 ...
## $ glucose : num -1.101 1.907 -0.978 0.495 -0.15 ...
## $ pressure: num -0.171 -0.274 -0.171 -1.508 0.24 ...
## $ triceps : num 0.569 -1.258 0.191 0.947 -1.258 ...
## $ insulin : num -0.682 -0.682 0.14 0.787 -0.682 ...
## $ mass : num -0.687 -1.101 -0.499 1.382 -0.812 ...
## $ pedigree: num -0.349 0.637 -0.915 5.601 -0.81 ...
## $ age : num -0.205 -0.1201 -1.0548 -0.0351 -0.29 ...Normalizing features
Many ML algorithms are sensitive to feature scales (e.g., KNN, SVM). Normalization ensures predictors are on comparable scales. Here we use range scaling to transform features to [0, 1].
Key practice note: scaling parameters must be learned on training data only, then applied to test data using the same fitted object.
preProcess_range_model <- preProcess(trainData, method='range')
trainData <- predict(preProcess_range_model, newdata = trainData)
# Append the Y variable instead of normalized data
trainData$diabetes <- y
# Look the dataset
apply(trainData[, -1], 2, FUN=function(x){c('min'=min(x), 'max'=max(x))})## glucose pressure triceps insulin mass pedigree
## min "0.0000000" "0.0000000" "0.00000000" "0.00000000" "0.0000000" "0.000000000"
## max "1.0000000" "1.0000000" "1.00000000" "1.00000000" "1.0000000" "1.000000000"
## age diabetes
## min "0.00000000" "neg"
## max "1.00000000" "pos"After preprocessing, always confirm the structure. This helps catch unexpected type changes or column ordering issues.
str(trainData)## 'data.frame': 615 obs. of 9 variables:
## $ pregnant: num 0.0588 0.4706 0.0588 0 0.2941 ...
## $ glucose : num 0.427 0.92 0.447 0.688 0.583 ...
## $ pressure: num 0.541 0.525 0.541 0.328 0.607 ...
## $ triceps : num 0.293 0 0.232 0.354 0 ...
## $ insulin : num 0 0 0.111 0.199 0 ...
## $ mass : num 0.396 0.347 0.419 0.642 0.382 ...
## $ pedigree: num 0.1235 0.2688 0.0403 1 0.0557 ...
## $ age : num 0.167 0.183 0 0.2 0.15 ...
## $ diabetes: Factor w/ 2 levels "neg","pos": 1 2 1 2 1 2 1 2 2 2 ...Plot features
Exploratory data analysis (EDA) is still valuable in ML workflows. Even when prediction is the goal, feature plots can reveal outliers, skewness, separation between classes, and variables that may be uninformative.
Boxplots show distribution and outliers by class.
featurePlot(x = trainData[, 1:8],
y = trainData$diabetes,
plot = "box",
strip=strip.custom(par.strip.text=list(cex=.7)),
scales = list(x = list(relation="free"),
y = list(relation="free")))
Density plots show the entire distribution and allow visual comparison between classes (e.g., “pos” vs “neg”).
featurePlot(x = trainData[, 1:8],
y = trainData$diabetes,
plot = "density",
strip=strip.custom(par.strip.text=list(cex=.7)),
scales = list(x = list(relation="free"),
y = list(relation="free")))
Correlation plots help detect highly correlated predictors. Strong correlations can affect some models (e.g., linear models) and can also reduce interpretability. In tree-based models, correlation is less problematic, but it still matters for feature importance interpretation.
library(corrplot)## corrplot 0.92 loadedcorrplot(cor((trainData[,-9] )))
Recursive feature elimination (rfe)
Feature selection is sometimes used to reduce dimensionality, improve interpretability, and possibly improve generalization. However, removing predictors can also discard useful signals. In many applied settings, we use feature selection as a diagnostic tool rather than a hard rule.
In some scenarios, we just have to include the significant features into the following model. A good choice of selecting the important features is the recursive feature elimination (RFE). RFE iteratively trains a model and removes the least important predictors, then evaluates performance for different subset sizes.
the final subset model is marked with a
stariskin the last column, here it is 8th. In the printed RFE output, the chosen subset size is marked. This indicates which number of predictors produced the best resampling performance under the chosen control settings.though it is not wise to neglect the other predictors. This is a practical warning: “important” features can change depending on resampling splits and model type. If the goal is prediction, do not remove variables unless you have a clear reason (e.g., cost, feasibility, regulatory constraints, or strong collinearity concerns).
Below we run RFE using random forest functions (rfFuncs)
with repeated cross-validation.
set.seed(100)
options(warn=-1)
subsets <- c(1:8)
ctrl <- rfeControl(functions = rfFuncs, #random forest algorithm
method = "repeatedcv", #k fold cross validation repeated 5 times
repeats = 5,
verbose = FALSE)
lmProfile <- rfe(x=trainData[, 1:8], y=trainData$diabetes,
sizes = subsets,
rfeControl = ctrl)
lmProfile##
## Recursive feature selection
##
## Outer resampling method: Cross-Validated (10 fold, repeated 5 times)
##
## Resampling performance over subset size:
##
## Variables Accuracy Kappa AccuracySD KappaSD Selected
## 1 0.6952 0.2841 0.04915 0.10605
## 2 0.7275 0.3815 0.04359 0.09609
## 3 0.7642 0.4673 0.04216 0.09490
## 4 0.7620 0.4631 0.04762 0.10945
## 5 0.7571 0.4534 0.05152 0.11813
## 6 0.7627 0.4679 0.04949 0.11218
## 7 0.7620 0.4619 0.05210 0.12019
## 8 0.7682 0.4728 0.04620 0.10576 *
##
## The top 5 variables (out of 8):
## glucose, mass, age, pregnant, insulinlook up features of all models in R caret
supports many model types through a unified interface. You can list all
available model methods, and you can look up details for a specific
model. This is helpful when selecting candidate algorithms for a given
problem.
# See available algorithms in caret
modelnames <- paste(names(getModelInfo()), collapse=', ')This shows tuning parameters and required packages for the
xgbTree method.
modelLookup('xgbTree')## model parameter label forReg forClass
## 1 xgbTree nrounds # Boosting Iterations TRUE TRUE
## 2 xgbTree max_depth Max Tree Depth TRUE TRUE
## 3 xgbTree eta Shrinkage TRUE TRUE
## 4 xgbTree gamma Minimum Loss Reduction TRUE TRUE
## 5 xgbTree colsample_bytree Subsample Ratio of Columns TRUE TRUE
## 6 xgbTree min_child_weight Minimum Sum of Instance Weight TRUE TRUE
## 7 xgbTree subsample Subsample Percentage TRUE TRUE
## probModel
## 1 TRUE
## 2 TRUE
## 3 TRUE
## 4 TRUE
## 5 TRUE
## 6 TRUE
## 7 TRUETraining a model
Multivariate Adaptive Regression Splines (MARS)
MARS is a flexible regression/classification approach that can model nonlinear relationships using piecewise linear basis functions. It is often more interpretable than many black-box models while still capturing nonlinearity.
We train the model using caret::train(). The formula
interface diabetes ~ . uses all predictors in
trainData to predict diabetes.
# Set the seed for reproducibility
set.seed(100)
# Train the model using randomForest and predict on the training data itself.
model_mars = train(diabetes ~ ., data=trainData, method='earth')## Loading required package: earth## Loading required package: Formula## Loading required package: plotmo## Loading required package: plotrixfitted <- predict(model_mars)- the default of resampling (Bootstrapped) is 25 reps The
train()function uses resampling to estimate performance during training. If you do not specifytrControl, caret uses a default resampling method (often bootstrap). This matters because reported accuracy during training is a resampling estimate, not the final test performance.
model_mars## Multivariate Adaptive Regression Spline
##
## 615 samples
## 8 predictor
## 2 classes: 'neg', 'pos'
##
## No pre-processing
## Resampling: Bootstrapped (25 reps)
## Summary of sample sizes: 615, 615, 615, 615, 615, 615, ...
## Resampling results across tuning parameters:
##
## nprune Accuracy Kappa
## 2 0.7451922 0.4023855
## 8 0.7680686 0.4748261
## 14 0.7603116 0.4581491
##
## Tuning parameter 'degree' was held constant at a value of 1
## Accuracy was used to select the optimal model using the largest value.
## The final values used for the model were nprune = 8 and degree = 1.- plot the Accuracy of various combinations of the hyper parameters -
interaction.depth and n.trees. In practice, always plot performance across tuning parameters to see stability. Even when a “best” tuning is selected, nearby settings may perform similarly.
plot(model_mars, main="Model Accuracies with MARS")
- calculate the importance of variable Variable importance provides a rough summary of which predictors the model relies on. Interpret importance with caution, especially when predictors are correlated.
varimp_mars <- varImp(model_mars)
plot(varimp_mars, main="Variable Importance with MARS")
Prepare the test data set
imputation,dummy, and normalizationThis is a critical step: we must apply the same transformations learned from training data to the test data. We do not refit preprocessing on test data.
The following code applies (1) imputation, (2) dummy encoding, and (3) range normalization to the test set, using the objects already fitted on training data.
# Step 1: Impute missing values
testData2 <- predict(preProcess_missingdata_model, testData)
# Step 2: Create one-hot encodings (dummy variables)
testData3 <- predict(dummies_model, testData2)
# Step 3: Transform the features to range between 0 and 1
testData4 <- predict(preProcess_range_model, testData3)
# View
head(testData4 )## pregnant glucose pressure triceps insulin mass pedigree
## 1 0.35294118 0.7437186 0.5901639 0.3535354 0.0000000 0.5007452 0.24841629
## 11 0.23529412 0.5527638 0.7540984 0.0000000 0.0000000 0.5603577 0.05113122
## 21 0.17647059 0.6331658 0.7213115 0.4141414 0.2777778 0.5856930 0.28325792
## 24 0.52941176 0.5979899 0.6557377 0.3535354 0.0000000 0.4321908 0.08371041
## 28 0.05882353 0.4874372 0.5409836 0.1515152 0.1654846 0.3457526 0.18506787
## 37 0.64705882 0.6934673 0.6229508 0.0000000 0.0000000 0.4947839 0.15475113
## age
## 1 0.48333333
## 11 0.15000000
## 21 0.10000000
## 24 0.13333333
## 28 0.01666667
## 37 0.23333333Prediction uisng testdata
Now we generate predictions on the final test feature matrix. In classification, predictions are typically class labels; in other settings, we may predict probabilities as well.
# Predict on testData
predicted <- predict(model_mars, testData4)
head(predicted)## [1] pos neg neg neg neg pos
## Levels: neg posCompute confusion matrix
The confusion matrix summarizes classification performance by comparing predicted labels with true labels. It includes sensitivity, specificity, and overall accuracy. In medical or clinical contexts, sensitivity/specificity are often more informative than accuracy alone, especially when classes are imbalanced.
# Compute the confusion matrix
confusionMatrix(reference = as.factor(testData$diabetes), data = as.factor(predicted ))## Confusion Matrix and Statistics
##
## Reference
## Prediction neg pos
## neg 86 22
## pos 14 31
##
## Accuracy : 0.7647
## 95% CI : (0.6894, 0.8294)
## No Information Rate : 0.6536
## P-Value [Acc > NIR] : 0.001988
##
## Kappa : 0.4613
##
## Mcnemar's Test P-Value : 0.243345
##
## Sensitivity : 0.8600
## Specificity : 0.5849
## Pos Pred Value : 0.7963
## Neg Pred Value : 0.6889
## Prevalence : 0.6536
## Detection Rate : 0.5621
## Detection Prevalence : 0.7059
## Balanced Accuracy : 0.7225
##
## 'Positive' Class : neg
## Tuning hyperparameter to optimize the model
Hyperparameter tuning searches over model settings to improve performance. The key is to tune using cross-validation on the training set, then evaluate final tuned performance on the test set.
- setting up hyper parameter
tuneLength, tuneGridHere we define cross-validation settings and specify that class probabilities and ROC-based summary metrics should be computed. ROC/AUC is often preferred when class imbalance exists or when threshold choice matters.
# Define the training control
fitControl <- trainControl(
method = 'cv', # k-fold cross validation
number = 5, # number of folds
savePredictions = 'final', # saves predictions for optimal tuning parameter
classProbs = T, # should class probabilities be returned
summaryFunction=twoClassSummary # results summary function
)We define a tuning grid for MARS: - nprune controls the
number of terms retained (model complexity), - degree
controls interaction degree (nonlinearity/interaction flexibility). Then
we train using ROC as the optimization metric.
# Step 1: Define the tuneGrid
marsGrid <- expand.grid(nprune = c(2, 4, 6, 8, 10),
degree = c(1, 2, 3))
# Step 2: Tune hyper parameters by setting tuneGrid
set.seed(100)
model_mars3 = train(diabetes ~ ., data=trainData, method='earth', metric='ROC', tuneGrid = marsGrid, trControl = fitControl)
model_mars3## Multivariate Adaptive Regression Spline
##
## 615 samples
## 8 predictor
## 2 classes: 'neg', 'pos'
##
## No pre-processing
## Resampling: Cross-Validated (5 fold)
## Summary of sample sizes: 492, 492, 492, 492, 492
## Resampling results across tuning parameters:
##
## degree nprune ROC Sens Spec
## 1 2 0.7962500 0.8850 0.4976744
## 1 4 0.8400581 0.8725 0.6046512
## 1 6 0.8410465 0.8825 0.6046512
## 1 8 0.8471512 0.8850 0.5860465
## 1 10 0.8437209 0.8775 0.6093023
## 2 2 0.7962500 0.8850 0.4976744
## 2 4 0.8284593 0.8775 0.6000000
## 2 6 0.8224419 0.8725 0.5488372
## 2 8 0.8237209 0.8700 0.5395349
## 2 10 0.8212209 0.8650 0.5395349
## 3 2 0.7962500 0.8850 0.4976744
## 3 4 0.8245058 0.8825 0.6000000
## 3 6 0.8205814 0.8750 0.5627907
## 3 8 0.8191860 0.8725 0.5581395
## 3 10 0.8195349 0.8650 0.5627907
##
## ROC was used to select the optimal model using the largest value.
## The final values used for the model were nprune = 8 and degree = 1.After tuning, evaluate on the test data. This gives a more realistic estimate of out-of-sample performance than training resampling metrics.
# Step 3: Predict on testData and Compute the confusion matrix
predicted3 <- predict(model_mars3, testData4)
confusionMatrix(reference = as.factor(testData$diabetes), data = as.factor(predicted3 ))## Confusion Matrix and Statistics
##
## Reference
## Prediction neg pos
## neg 86 22
## pos 14 31
##
## Accuracy : 0.7647
## 95% CI : (0.6894, 0.8294)
## No Information Rate : 0.6536
## P-Value [Acc > NIR] : 0.001988
##
## Kappa : 0.4613
##
## Mcnemar's Test P-Value : 0.243345
##
## Sensitivity : 0.8600
## Specificity : 0.5849
## Pos Pred Value : 0.7963
## Neg Pred Value : 0.6889
## Prevalence : 0.6536
## Detection Rate : 0.5621
## Detection Prevalence : 0.7059
## Balanced Accuracy : 0.7225
##
## 'Positive' Class : neg
## Other marchine learning algorithms
It is good practice to compare multiple algorithms. Different models have different bias-variance profiles and handle nonlinearities and interactions differently. Here we train several common classifiers using the same training control settings.
adaboost algorithm
AdaBoost is an ensemble method that combines weak learners into a stronger classifier. It can perform well but may be sensitive to noise and outliers.
set.seed(100)
# Train the model using adaboost
model_adaboost = train(diabetes ~ ., data=trainData, method='gbm', tuneLength=2, trControl = fitControl)## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2599 nan 0.1000 0.0169
## 2 1.2278 nan 0.1000 0.0134
## 3 1.2078 nan 0.1000 0.0072
## 4 1.1866 nan 0.1000 0.0087
## 5 1.1705 nan 0.1000 0.0046
## 6 1.1464 nan 0.1000 0.0113
## 7 1.1214 nan 0.1000 0.0102
## 8 1.1036 nan 0.1000 0.0054
## 9 1.0877 nan 0.1000 0.0052
## 10 1.0734 nan 0.1000 0.0057
## 20 0.9787 nan 0.1000 0.0010
## 40 0.8784 nan 0.1000 0.0001
## 60 0.8447 nan 0.1000 -0.0005
## 80 0.8169 nan 0.1000 -0.0004
## 100 0.8024 nan 0.1000 -0.0001
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2486 nan 0.1000 0.0170
## 2 1.2092 nan 0.1000 0.0164
## 3 1.1697 nan 0.1000 0.0178
## 4 1.1363 nan 0.1000 0.0128
## 5 1.1186 nan 0.1000 0.0038
## 6 1.0927 nan 0.1000 0.0100
## 7 1.0706 nan 0.1000 0.0089
## 8 1.0505 nan 0.1000 0.0092
## 9 1.0343 nan 0.1000 0.0066
## 10 1.0159 nan 0.1000 0.0061
## 20 0.9048 nan 0.1000 0.0033
## 40 0.8153 nan 0.1000 0.0000
## 60 0.7671 nan 0.1000 -0.0001
## 80 0.7276 nan 0.1000 -0.0026
## 100 0.6973 nan 0.1000 -0.0027
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2510 nan 0.1000 0.0195
## 2 1.2166 nan 0.1000 0.0141
## 3 1.1949 nan 0.1000 0.0073
## 4 1.1637 nan 0.1000 0.0120
## 5 1.1425 nan 0.1000 0.0089
## 6 1.1173 nan 0.1000 0.0081
## 7 1.1000 nan 0.1000 0.0071
## 8 1.0825 nan 0.1000 0.0075
## 9 1.0680 nan 0.1000 0.0027
## 10 1.0547 nan 0.1000 0.0067
## 20 0.9599 nan 0.1000 0.0016
## 40 0.8620 nan 0.1000 0.0003
## 60 0.8179 nan 0.1000 0.0004
## 80 0.7907 nan 0.1000 -0.0012
## 100 0.7692 nan 0.1000 -0.0005
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2431 nan 0.1000 0.0252
## 2 1.1980 nan 0.1000 0.0182
## 3 1.1599 nan 0.1000 0.0175
## 4 1.1316 nan 0.1000 0.0121
## 5 1.1033 nan 0.1000 0.0144
## 6 1.0786 nan 0.1000 0.0107
## 7 1.0549 nan 0.1000 0.0103
## 8 1.0338 nan 0.1000 0.0086
## 9 1.0127 nan 0.1000 0.0069
## 10 0.9940 nan 0.1000 0.0054
## 20 0.8781 nan 0.1000 0.0001
## 40 0.7842 nan 0.1000 -0.0007
## 60 0.7319 nan 0.1000 -0.0010
## 80 0.6971 nan 0.1000 -0.0017
## 100 0.6628 nan 0.1000 -0.0013
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2596 nan 0.1000 0.0164
## 2 1.2257 nan 0.1000 0.0165
## 3 1.1929 nan 0.1000 0.0136
## 4 1.1669 nan 0.1000 0.0109
## 5 1.1449 nan 0.1000 0.0097
## 6 1.1257 nan 0.1000 0.0060
## 7 1.1063 nan 0.1000 0.0088
## 8 1.0893 nan 0.1000 0.0054
## 9 1.0755 nan 0.1000 0.0035
## 10 1.0600 nan 0.1000 0.0061
## 20 0.9688 nan 0.1000 0.0026
## 40 0.8880 nan 0.1000 0.0006
## 60 0.8515 nan 0.1000 0.0002
## 80 0.8267 nan 0.1000 -0.0010
## 100 0.8095 nan 0.1000 -0.0009
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2421 nan 0.1000 0.0236
## 2 1.2006 nan 0.1000 0.0192
## 3 1.1640 nan 0.1000 0.0151
## 4 1.1288 nan 0.1000 0.0122
## 5 1.1009 nan 0.1000 0.0111
## 6 1.0821 nan 0.1000 0.0048
## 7 1.0589 nan 0.1000 0.0067
## 8 1.0360 nan 0.1000 0.0082
## 9 1.0192 nan 0.1000 0.0060
## 10 1.0000 nan 0.1000 0.0033
## 20 0.9025 nan 0.1000 0.0006
## 40 0.8193 nan 0.1000 -0.0020
## 60 0.7742 nan 0.1000 -0.0029
## 80 0.7374 nan 0.1000 -0.0013
## 100 0.7035 nan 0.1000 -0.0022
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2564 nan 0.1000 0.0180
## 2 1.2215 nan 0.1000 0.0174
## 3 1.1923 nan 0.1000 0.0132
## 4 1.1682 nan 0.1000 0.0107
## 5 1.1476 nan 0.1000 0.0084
## 6 1.1324 nan 0.1000 0.0050
## 7 1.1168 nan 0.1000 0.0055
## 8 1.1026 nan 0.1000 0.0046
## 9 1.0898 nan 0.1000 0.0031
## 10 1.0779 nan 0.1000 0.0034
## 20 0.9881 nan 0.1000 0.0027
## 40 0.9101 nan 0.1000 -0.0013
## 60 0.8682 nan 0.1000 -0.0014
## 80 0.8384 nan 0.1000 -0.0009
## 100 0.8252 nan 0.1000 -0.0007
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2388 nan 0.1000 0.0215
## 2 1.2024 nan 0.1000 0.0156
## 3 1.1711 nan 0.1000 0.0131
## 4 1.1427 nan 0.1000 0.0098
## 5 1.1151 nan 0.1000 0.0098
## 6 1.0944 nan 0.1000 0.0073
## 7 1.0744 nan 0.1000 0.0070
## 8 1.0547 nan 0.1000 0.0044
## 9 1.0374 nan 0.1000 0.0068
## 10 1.0213 nan 0.1000 0.0051
## 20 0.9192 nan 0.1000 0.0002
## 40 0.8387 nan 0.1000 -0.0021
## 60 0.7839 nan 0.1000 -0.0013
## 80 0.7466 nan 0.1000 -0.0009
## 100 0.7136 nan 0.1000 -0.0015
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2598 nan 0.1000 0.0154
## 2 1.2303 nan 0.1000 0.0121
## 3 1.2058 nan 0.1000 0.0086
## 4 1.1813 nan 0.1000 0.0078
## 5 1.1665 nan 0.1000 0.0065
## 6 1.1488 nan 0.1000 0.0081
## 7 1.1355 nan 0.1000 0.0055
## 8 1.1229 nan 0.1000 0.0029
## 9 1.1063 nan 0.1000 0.0064
## 10 1.0920 nan 0.1000 0.0036
## 20 1.0023 nan 0.1000 0.0017
## 40 0.9170 nan 0.1000 0.0007
## 60 0.8692 nan 0.1000 -0.0020
## 80 0.8420 nan 0.1000 -0.0010
## 100 0.8251 nan 0.1000 -0.0005
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2506 nan 0.1000 0.0196
## 2 1.2109 nan 0.1000 0.0119
## 3 1.1762 nan 0.1000 0.0156
## 4 1.1454 nan 0.1000 0.0147
## 5 1.1148 nan 0.1000 0.0117
## 6 1.0950 nan 0.1000 0.0079
## 7 1.0755 nan 0.1000 0.0062
## 8 1.0537 nan 0.1000 0.0065
## 9 1.0392 nan 0.1000 0.0045
## 10 1.0262 nan 0.1000 0.0036
## 20 0.9198 nan 0.1000 -0.0006
## 40 0.8385 nan 0.1000 -0.0014
## 60 0.7937 nan 0.1000 -0.0018
## 80 0.7585 nan 0.1000 -0.0037
## 100 0.7255 nan 0.1000 -0.0014
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2535 nan 0.1000 0.0181
## 2 1.2276 nan 0.1000 0.0138
## 3 1.2007 nan 0.1000 0.0118
## 4 1.1755 nan 0.1000 0.0090
## 5 1.1572 nan 0.1000 0.0086
## 6 1.1394 nan 0.1000 0.0074
## 7 1.1300 nan 0.1000 -0.0000
## 8 1.1129 nan 0.1000 0.0063
## 9 1.0970 nan 0.1000 0.0067
## 10 1.0863 nan 0.1000 0.0052
## 20 0.9944 nan 0.1000 0.0006
## 40 0.9047 nan 0.1000 0.0000
## 50 0.8821 nan 0.1000 0.0005model_adaboost## Stochastic Gradient Boosting
##
## 615 samples
## 8 predictor
## 2 classes: 'neg', 'pos'
##
## No pre-processing
## Resampling: Cross-Validated (5 fold)
## Summary of sample sizes: 492, 492, 492, 492, 492
## Resampling results across tuning parameters:
##
## interaction.depth n.trees ROC Sens Spec
## 1 50 0.8378198 0.8950 0.5441860
## 1 100 0.8275872 0.8825 0.5720930
## 2 50 0.8332558 0.8725 0.5674419
## 2 100 0.8297093 0.8675 0.5860465
##
## Tuning parameter 'shrinkage' was held constant at a value of 0.1
##
## Tuning parameter 'n.minobsinnode' was held constant at a value of 10
## ROC was used to select the optimal model using the largest value.
## The final values used for the model were n.trees = 50, interaction.depth =
## 1, shrinkage = 0.1 and n.minobsinnode = 10.random forest
Random forest is a robust tree-based ensemble method. It handles nonlinearities and interactions well and is often a strong baseline model.
set.seed(100)
# Train the model using rf
model_rf = train(diabetes ~ ., data=trainData, method='rf', tuneLength=5, trControl = fitControl)
model_rf## Random Forest
##
## 615 samples
## 8 predictor
## 2 classes: 'neg', 'pos'
##
## No pre-processing
## Resampling: Cross-Validated (5 fold)
## Summary of sample sizes: 492, 492, 492, 492, 492
## Resampling results across tuning parameters:
##
## mtry ROC Sens Spec
## 2 0.8210756 0.8600 0.5906977
## 3 0.8217733 0.8575 0.6046512
## 5 0.8145640 0.8550 0.5906977
## 6 0.8152616 0.8575 0.6093023
## 8 0.8145349 0.8500 0.6000000
##
## ROC was used to select the optimal model using the largest value.
## The final value used for the model was mtry = 3.xgbDART algorithm
This section shows an example template for XGBoost DART, which is a boosted-tree approach with dropout. The code is commented out, which is a common way to keep a placeholder for future extensions.
# set.seed(100)
#
# # Train the model using MARS
# model_xgbDART = train(Purchase ~ ., data=trainData, method='xgbDART', tuneLength=5, trControl = fitControl, verbose=F)
# model_xgbDARTSupport Vector Machines (SVM)
SVMs can be very effective in classification, especially with nonlinear kernels. They are sensitive to feature scaling, which is why normalization earlier is important.
set.seed(100)
# Train the model using MARS
model_svmRadial = train(diabetes ~ ., data=trainData, method='svmRadial', tuneLength=15, trControl = fitControl)
model_svmRadial## Support Vector Machines with Radial Basis Function Kernel
##
## 615 samples
## 8 predictor
## 2 classes: 'neg', 'pos'
##
## No pre-processing
## Resampling: Cross-Validated (5 fold)
## Summary of sample sizes: 492, 492, 492, 492, 492
## Resampling results across tuning parameters:
##
## C ROC Sens Spec
## 0.25 0.8306395 0.8650 0.5813953
## 0.50 0.8308140 0.8800 0.5720930
## 1.00 0.8279070 0.8750 0.5348837
## 2.00 0.8216860 0.8825 0.4976744
## 4.00 0.8204070 0.8925 0.4883721
## 8.00 0.8080814 0.8800 0.4790698
## 16.00 0.7892442 0.8825 0.4651163
## 32.00 0.7677326 0.8875 0.4093023
## 64.00 0.7430814 0.8925 0.3674419
## 128.00 0.7165698 0.8825 0.3255814
## 256.00 0.7062209 0.8975 0.3255814
## 512.00 0.7051163 0.9100 0.2930233
## 1024.00 0.7005814 0.9025 0.3023256
## 2048.00 0.6955233 0.9000 0.3162791
## 4096.00 0.6948837 0.8925 0.3348837
##
## Tuning parameter 'sigma' was held constant at a value of 0.1161195
## ROC was used to select the optimal model using the largest value.
## The final values used for the model were sigma = 0.1161195 and C = 0.5.K-Nearest Neighbors
KNN is a simple and intuitive method: classify based on the majority class of the nearest neighbors. It is strongly affected by scaling and by the choice of K.
set.seed(100)
# Train the model using MARS
model_knn = train(diabetes ~ ., data=trainData, method='knn', tuneLength=15, trControl = fitControl)
model_knn## k-Nearest Neighbors
##
## 615 samples
## 8 predictor
## 2 classes: 'neg', 'pos'
##
## No pre-processing
## Resampling: Cross-Validated (5 fold)
## Summary of sample sizes: 492, 492, 492, 492, 492
## Resampling results across tuning parameters:
##
## k ROC Sens Spec
## 5 0.7401744 0.8250 0.4651163
## 7 0.7655523 0.8425 0.4744186
## 9 0.7707849 0.8500 0.4930233
## 11 0.7797384 0.8800 0.4976744
## 13 0.7876744 0.8725 0.4790698
## 15 0.7951163 0.8800 0.4837209
## 17 0.7933721 0.8775 0.4651163
## 19 0.7997965 0.8825 0.4465116
## 21 0.8001163 0.8975 0.4418605
## 23 0.8024709 0.9050 0.4651163
## 25 0.8037791 0.9050 0.4744186
## 27 0.8082267 0.9050 0.4790698
## 29 0.8069767 0.9150 0.4697674
## 31 0.8083430 0.9100 0.4418605
## 33 0.8064244 0.9175 0.4372093
##
## ROC was used to select the optimal model using the largest value.
## The final value used for the model was k = 31.Comparisons of different models
A common workflow is to train multiple models under the same
resampling settings and then compare their performance distributions.
resamples() collects resampling results across models for
summary and visualization.
# Compare model performances using resample()
models_compare <- resamples(list(ADABOOST=model_adaboost, RF=model_rf, knn=model_knn, MARS=model_mars3, SVM=model_svmRadial))
# Summary of the models performances
summary(models_compare)##
## Call:
## summary.resamples(object = models_compare)
##
## Models: ADABOOST, RF, knn, MARS, SVM
## Number of resamples: 5
##
## ROC
## Min. 1st Qu. Median Mean 3rd Qu. Max. NA's
## ADABOOST 0.7909884 0.8353198 0.8433140 0.8378198 0.8572674 0.8622093 0
## RF 0.7697674 0.8155523 0.8170058 0.8217733 0.8497093 0.8568314 0
## knn 0.7646802 0.7680233 0.8209302 0.8083430 0.8297965 0.8582849 0
## MARS 0.8238372 0.8430233 0.8450581 0.8471512 0.8613372 0.8625000 0
## SVM 0.7648256 0.8279070 0.8436047 0.8308140 0.8441860 0.8735465 0
##
## Sens
## Min. 1st Qu. Median Mean 3rd Qu. Max. NA's
## ADABOOST 0.8250 0.9000 0.9125 0.8950 0.9125 0.9250 0
## RF 0.8250 0.8375 0.8625 0.8575 0.8750 0.8875 0
## knn 0.8875 0.9000 0.9000 0.9100 0.9125 0.9500 0
## MARS 0.8500 0.8625 0.8875 0.8850 0.9125 0.9125 0
## SVM 0.8625 0.8750 0.8750 0.8800 0.8875 0.9000 0
##
## Spec
## Min. 1st Qu. Median Mean 3rd Qu. Max. NA's
## ADABOOST 0.4883721 0.5116279 0.5348837 0.5441860 0.5581395 0.6279070 0
## RF 0.5581395 0.5581395 0.5813953 0.6046512 0.6511628 0.6744186 0
## knn 0.4186047 0.4418605 0.4418605 0.4418605 0.4418605 0.4651163 0
## MARS 0.5348837 0.5348837 0.5813953 0.5860465 0.6046512 0.6744186 0
## SVM 0.5116279 0.5581395 0.5813953 0.5720930 0.6046512 0.6046512 0Plot comparisons of models
Visualization helps compare stability and variability across resampling folds. A model with slightly lower mean performance but more stability might be preferable in production settings.
# Draw box plots to compare models
scales <- list(x=list(relation="free"), y=list(relation="free"))
bwplot(models_compare, scales=scales)
Ensemble predictions from multiple models
Ensembling combines multiple models to potentially improve performance and robustness. The idea is that different models capture different aspects of the signal; combining them can reduce variance and improve generalization.
- create multiple models Here we use
caretEnsembleto train multiple algorithms under repeated cross-validation and store predictions for stacking.
library(caretEnsemble)
# Stacking Algorithms - Run multiple algos in one call.
trainControl <- trainControl(method="repeatedcv",
number=10,
repeats=3,
savePredictions=TRUE,
classProbs=TRUE)
algorithmList <- c('rf', 'gbm', 'earth', 'knn', 'svmRadial')
set.seed(100)
models <- caretList(diabetes ~ ., data=trainData, trControl=trainControl, methodList=algorithmList)## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2519 nan 0.1000 0.0187
## 2 1.2217 nan 0.1000 0.0176
## 3 1.1936 nan 0.1000 0.0143
## 4 1.1705 nan 0.1000 0.0091
## 5 1.1511 nan 0.1000 0.0075
## 6 1.1330 nan 0.1000 0.0077
## 7 1.1169 nan 0.1000 0.0053
## 8 1.1010 nan 0.1000 0.0052
## 9 1.0831 nan 0.1000 0.0073
## 10 1.0682 nan 0.1000 0.0049
## 20 0.9673 nan 0.1000 0.0010
## 40 0.8727 nan 0.1000 0.0000
## 60 0.8242 nan 0.1000 -0.0013
## 80 0.8026 nan 0.1000 -0.0005
## 100 0.7860 nan 0.1000 -0.0016
## 120 0.7768 nan 0.1000 -0.0022
## 140 0.7668 nan 0.1000 -0.0014
## 150 0.7608 nan 0.1000 -0.0003
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2431 nan 0.1000 0.0194
## 2 1.2044 nan 0.1000 0.0131
## 3 1.1708 nan 0.1000 0.0156
## 4 1.1420 nan 0.1000 0.0151
## 5 1.1122 nan 0.1000 0.0136
## 6 1.0858 nan 0.1000 0.0097
## 7 1.0663 nan 0.1000 0.0083
## 8 1.0485 nan 0.1000 0.0068
## 9 1.0268 nan 0.1000 0.0067
## 10 1.0106 nan 0.1000 0.0055
## 20 0.8977 nan 0.1000 0.0014
## 40 0.8001 nan 0.1000 0.0005
## 60 0.7571 nan 0.1000 -0.0012
## 80 0.7230 nan 0.1000 -0.0021
## 100 0.6942 nan 0.1000 -0.0017
## 120 0.6647 nan 0.1000 -0.0016
## 140 0.6397 nan 0.1000 -0.0014
## 150 0.6292 nan 0.1000 -0.0018
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2360 nan 0.1000 0.0247
## 2 1.1932 nan 0.1000 0.0173
## 3 1.1558 nan 0.1000 0.0174
## 4 1.1266 nan 0.1000 0.0121
## 5 1.0870 nan 0.1000 0.0180
## 6 1.0562 nan 0.1000 0.0127
## 7 1.0285 nan 0.1000 0.0107
## 8 1.0040 nan 0.1000 0.0093
## 9 0.9841 nan 0.1000 0.0060
## 10 0.9690 nan 0.1000 0.0039
## 20 0.8568 nan 0.1000 0.0021
## 40 0.7521 nan 0.1000 -0.0020
## 60 0.6947 nan 0.1000 -0.0039
## 80 0.6479 nan 0.1000 -0.0029
## 100 0.6101 nan 0.1000 -0.0025
## 120 0.5714 nan 0.1000 -0.0001
## 140 0.5446 nan 0.1000 -0.0030
## 150 0.5324 nan 0.1000 -0.0021
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2644 nan 0.1000 0.0150
## 2 1.2323 nan 0.1000 0.0126
## 3 1.2055 nan 0.1000 0.0119
## 4 1.1840 nan 0.1000 0.0087
## 5 1.1645 nan 0.1000 0.0082
## 6 1.1442 nan 0.1000 0.0065
## 7 1.1267 nan 0.1000 0.0076
## 8 1.1097 nan 0.1000 0.0048
## 9 1.0940 nan 0.1000 0.0068
## 10 1.0815 nan 0.1000 0.0030
## 20 0.9853 nan 0.1000 -0.0008
## 40 0.9007 nan 0.1000 0.0001
## 60 0.8631 nan 0.1000 0.0003
## 80 0.8379 nan 0.1000 -0.0002
## 100 0.8178 nan 0.1000 -0.0012
## 120 0.8039 nan 0.1000 -0.0003
## 140 0.7929 nan 0.1000 -0.0014
## 150 0.7882 nan 0.1000 -0.0006
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2479 nan 0.1000 0.0222
## 2 1.2101 nan 0.1000 0.0167
## 3 1.1763 nan 0.1000 0.0150
## 4 1.1440 nan 0.1000 0.0126
## 5 1.1180 nan 0.1000 0.0104
## 6 1.0981 nan 0.1000 0.0052
## 7 1.0754 nan 0.1000 0.0050
## 8 1.0577 nan 0.1000 0.0072
## 9 1.0420 nan 0.1000 0.0060
## 10 1.0246 nan 0.1000 0.0070
## 20 0.9215 nan 0.1000 0.0013
## 40 0.8318 nan 0.1000 -0.0007
## 60 0.7928 nan 0.1000 -0.0036
## 80 0.7576 nan 0.1000 -0.0017
## 100 0.7280 nan 0.1000 -0.0003
## 120 0.7126 nan 0.1000 -0.0022
## 140 0.6887 nan 0.1000 -0.0017
## 150 0.6789 nan 0.1000 -0.0017
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2418 nan 0.1000 0.0252
## 2 1.1979 nan 0.1000 0.0196
## 3 1.1548 nan 0.1000 0.0198
## 4 1.1200 nan 0.1000 0.0130
## 5 1.0931 nan 0.1000 0.0113
## 6 1.0685 nan 0.1000 0.0121
## 7 1.0541 nan 0.1000 0.0016
## 8 1.0320 nan 0.1000 0.0101
## 9 1.0051 nan 0.1000 0.0096
## 10 0.9884 nan 0.1000 0.0040
## 20 0.8836 nan 0.1000 0.0002
## 40 0.7863 nan 0.1000 -0.0008
## 60 0.7273 nan 0.1000 -0.0047
## 80 0.6823 nan 0.1000 -0.0017
## 100 0.6396 nan 0.1000 -0.0020
## 120 0.6051 nan 0.1000 -0.0019
## 140 0.5713 nan 0.1000 -0.0024
## 150 0.5585 nan 0.1000 -0.0004
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2596 nan 0.1000 0.0184
## 2 1.2263 nan 0.1000 0.0171
## 3 1.1945 nan 0.1000 0.0154
## 4 1.1663 nan 0.1000 0.0110
## 5 1.1498 nan 0.1000 0.0062
## 6 1.1311 nan 0.1000 0.0093
## 7 1.1156 nan 0.1000 0.0058
## 8 1.1021 nan 0.1000 0.0067
## 9 1.0852 nan 0.1000 0.0062
## 10 1.0734 nan 0.1000 0.0047
## 20 0.9770 nan 0.1000 0.0037
## 40 0.8854 nan 0.1000 -0.0008
## 60 0.8413 nan 0.1000 -0.0007
## 80 0.8164 nan 0.1000 -0.0016
## 100 0.7971 nan 0.1000 -0.0008
## 120 0.7851 nan 0.1000 -0.0012
## 140 0.7735 nan 0.1000 -0.0004
## 150 0.7691 nan 0.1000 -0.0016
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2479 nan 0.1000 0.0200
## 2 1.2065 nan 0.1000 0.0183
## 3 1.1682 nan 0.1000 0.0142
## 4 1.1365 nan 0.1000 0.0150
## 5 1.1123 nan 0.1000 0.0093
## 6 1.0863 nan 0.1000 0.0074
## 7 1.0667 nan 0.1000 0.0030
## 8 1.0485 nan 0.1000 0.0069
## 9 1.0327 nan 0.1000 0.0064
## 10 1.0166 nan 0.1000 0.0075
## 20 0.9062 nan 0.1000 0.0024
## 40 0.8216 nan 0.1000 -0.0011
## 60 0.7742 nan 0.1000 -0.0020
## 80 0.7422 nan 0.1000 -0.0012
## 100 0.7151 nan 0.1000 -0.0020
## 120 0.6933 nan 0.1000 -0.0012
## 140 0.6732 nan 0.1000 -0.0020
## 150 0.6607 nan 0.1000 -0.0024
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2441 nan 0.1000 0.0226
## 2 1.1944 nan 0.1000 0.0225
## 3 1.1560 nan 0.1000 0.0147
## 4 1.1161 nan 0.1000 0.0180
## 5 1.0876 nan 0.1000 0.0133
## 6 1.0621 nan 0.1000 0.0093
## 7 1.0397 nan 0.1000 0.0074
## 8 1.0163 nan 0.1000 0.0103
## 9 0.9937 nan 0.1000 0.0071
## 10 0.9715 nan 0.1000 0.0064
## 20 0.8631 nan 0.1000 -0.0002
## 40 0.7714 nan 0.1000 -0.0024
## 60 0.7156 nan 0.1000 -0.0017
## 80 0.6754 nan 0.1000 -0.0011
## 100 0.6291 nan 0.1000 -0.0028
## 120 0.5903 nan 0.1000 -0.0016
## 140 0.5571 nan 0.1000 -0.0012
## 150 0.5426 nan 0.1000 -0.0011
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2579 nan 0.1000 0.0177
## 2 1.2286 nan 0.1000 0.0158
## 3 1.2023 nan 0.1000 0.0132
## 4 1.1764 nan 0.1000 0.0105
## 5 1.1570 nan 0.1000 0.0082
## 6 1.1391 nan 0.1000 0.0062
## 7 1.1222 nan 0.1000 0.0060
## 8 1.1046 nan 0.1000 0.0061
## 9 1.0894 nan 0.1000 0.0080
## 10 1.0763 nan 0.1000 0.0051
## 20 0.9818 nan 0.1000 0.0003
## 40 0.8877 nan 0.1000 0.0014
## 60 0.8440 nan 0.1000 -0.0010
## 80 0.8197 nan 0.1000 -0.0015
## 100 0.8059 nan 0.1000 -0.0003
## 120 0.7905 nan 0.1000 -0.0010
## 140 0.7786 nan 0.1000 -0.0008
## 150 0.7749 nan 0.1000 -0.0006
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2493 nan 0.1000 0.0210
## 2 1.2086 nan 0.1000 0.0183
## 3 1.1710 nan 0.1000 0.0172
## 4 1.1446 nan 0.1000 0.0100
## 5 1.1116 nan 0.1000 0.0130
## 6 1.0853 nan 0.1000 0.0114
## 7 1.0648 nan 0.1000 0.0074
## 8 1.0437 nan 0.1000 0.0094
## 9 1.0319 nan 0.1000 0.0014
## 10 1.0163 nan 0.1000 0.0048
## 20 0.9054 nan 0.1000 -0.0001
## 40 0.8206 nan 0.1000 -0.0004
## 60 0.7708 nan 0.1000 -0.0013
## 80 0.7396 nan 0.1000 -0.0017
## 100 0.7154 nan 0.1000 -0.0004
## 120 0.6910 nan 0.1000 -0.0001
## 140 0.6671 nan 0.1000 -0.0005
## 150 0.6555 nan 0.1000 -0.0010
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2391 nan 0.1000 0.0217
## 2 1.1956 nan 0.1000 0.0198
## 3 1.1573 nan 0.1000 0.0144
## 4 1.1169 nan 0.1000 0.0129
## 5 1.0910 nan 0.1000 0.0054
## 6 1.0654 nan 0.1000 0.0101
## 7 1.0364 nan 0.1000 0.0125
## 8 1.0119 nan 0.1000 0.0095
## 9 0.9909 nan 0.1000 0.0061
## 10 0.9713 nan 0.1000 0.0082
## 20 0.8630 nan 0.1000 -0.0002
## 40 0.7765 nan 0.1000 -0.0016
## 60 0.7169 nan 0.1000 -0.0018
## 80 0.6732 nan 0.1000 -0.0012
## 100 0.6318 nan 0.1000 -0.0021
## 120 0.5923 nan 0.1000 -0.0001
## 140 0.5608 nan 0.1000 -0.0016
## 150 0.5449 nan 0.1000 -0.0023
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2607 nan 0.1000 0.0156
## 2 1.2315 nan 0.1000 0.0119
## 3 1.2011 nan 0.1000 0.0117
## 4 1.1808 nan 0.1000 0.0064
## 5 1.1661 nan 0.1000 0.0058
## 6 1.1494 nan 0.1000 0.0061
## 7 1.1337 nan 0.1000 0.0069
## 8 1.1218 nan 0.1000 0.0046
## 9 1.1100 nan 0.1000 0.0044
## 10 1.0922 nan 0.1000 0.0058
## 20 1.0063 nan 0.1000 0.0025
## 40 0.9209 nan 0.1000 0.0008
## 60 0.8805 nan 0.1000 -0.0004
## 80 0.8567 nan 0.1000 -0.0006
## 100 0.8430 nan 0.1000 -0.0011
## 120 0.8302 nan 0.1000 -0.0012
## 140 0.8198 nan 0.1000 -0.0015
## 150 0.8137 nan 0.1000 -0.0006
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2580 nan 0.1000 0.0169
## 2 1.2161 nan 0.1000 0.0166
## 3 1.1880 nan 0.1000 0.0135
## 4 1.1580 nan 0.1000 0.0130
## 5 1.1296 nan 0.1000 0.0093
## 6 1.1087 nan 0.1000 0.0047
## 7 1.0835 nan 0.1000 0.0101
## 8 1.0659 nan 0.1000 0.0068
## 9 1.0448 nan 0.1000 0.0085
## 10 1.0283 nan 0.1000 0.0054
## 20 0.9370 nan 0.1000 0.0023
## 40 0.8489 nan 0.1000 -0.0011
## 60 0.8072 nan 0.1000 -0.0015
## 80 0.7667 nan 0.1000 -0.0003
## 100 0.7412 nan 0.1000 -0.0017
## 120 0.7174 nan 0.1000 -0.0014
## 140 0.6960 nan 0.1000 -0.0023
## 150 0.6863 nan 0.1000 -0.0010
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2436 nan 0.1000 0.0231
## 2 1.1981 nan 0.1000 0.0194
## 3 1.1557 nan 0.1000 0.0167
## 4 1.1221 nan 0.1000 0.0145
## 5 1.0925 nan 0.1000 0.0121
## 6 1.0701 nan 0.1000 0.0097
## 7 1.0497 nan 0.1000 0.0075
## 8 1.0310 nan 0.1000 0.0050
## 9 1.0162 nan 0.1000 0.0013
## 10 0.9990 nan 0.1000 0.0048
## 20 0.8851 nan 0.1000 -0.0016
## 40 0.8004 nan 0.1000 -0.0022
## 60 0.7334 nan 0.1000 -0.0013
## 80 0.6892 nan 0.1000 -0.0021
## 100 0.6605 nan 0.1000 -0.0032
## 120 0.6244 nan 0.1000 -0.0018
## 140 0.5940 nan 0.1000 -0.0035
## 150 0.5807 nan 0.1000 -0.0008
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2626 nan 0.1000 0.0124
## 2 1.2337 nan 0.1000 0.0127
## 3 1.2106 nan 0.1000 0.0121
## 4 1.1827 nan 0.1000 0.0112
## 5 1.1626 nan 0.1000 0.0092
## 6 1.1435 nan 0.1000 0.0074
## 7 1.1259 nan 0.1000 0.0063
## 8 1.1106 nan 0.1000 0.0061
## 9 1.0936 nan 0.1000 0.0061
## 10 1.0801 nan 0.1000 0.0032
## 20 0.9932 nan 0.1000 0.0014
## 40 0.9129 nan 0.1000 -0.0002
## 60 0.8699 nan 0.1000 -0.0013
## 80 0.8444 nan 0.1000 -0.0002
## 100 0.8280 nan 0.1000 -0.0009
## 120 0.8156 nan 0.1000 -0.0004
## 140 0.8031 nan 0.1000 -0.0027
## 150 0.7966 nan 0.1000 -0.0003
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2493 nan 0.1000 0.0187
## 2 1.2040 nan 0.1000 0.0185
## 3 1.1696 nan 0.1000 0.0131
## 4 1.1404 nan 0.1000 0.0109
## 5 1.1126 nan 0.1000 0.0096
## 6 1.0894 nan 0.1000 0.0101
## 7 1.0718 nan 0.1000 0.0040
## 8 1.0549 nan 0.1000 0.0080
## 9 1.0359 nan 0.1000 0.0078
## 10 1.0186 nan 0.1000 0.0050
## 20 0.9239 nan 0.1000 -0.0009
## 40 0.8334 nan 0.1000 -0.0009
## 60 0.7887 nan 0.1000 -0.0012
## 80 0.7562 nan 0.1000 -0.0025
## 100 0.7273 nan 0.1000 -0.0006
## 120 0.7002 nan 0.1000 -0.0018
## 140 0.6803 nan 0.1000 -0.0013
## 150 0.6713 nan 0.1000 -0.0011
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2421 nan 0.1000 0.0203
## 2 1.1898 nan 0.1000 0.0224
## 3 1.1486 nan 0.1000 0.0158
## 4 1.1140 nan 0.1000 0.0091
## 5 1.0860 nan 0.1000 0.0129
## 6 1.0579 nan 0.1000 0.0102
## 7 1.0336 nan 0.1000 0.0082
## 8 1.0171 nan 0.1000 0.0045
## 9 1.0008 nan 0.1000 0.0019
## 10 0.9842 nan 0.1000 0.0022
## 20 0.8864 nan 0.1000 -0.0018
## 40 0.7876 nan 0.1000 -0.0028
## 60 0.7342 nan 0.1000 -0.0027
## 80 0.6882 nan 0.1000 -0.0008
## 100 0.6498 nan 0.1000 -0.0019
## 120 0.6123 nan 0.1000 -0.0019
## 140 0.5851 nan 0.1000 -0.0020
## 150 0.5708 nan 0.1000 -0.0019
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2509 nan 0.1000 0.0184
## 2 1.2173 nan 0.1000 0.0129
## 3 1.1857 nan 0.1000 0.0158
## 4 1.1604 nan 0.1000 0.0121
## 5 1.1407 nan 0.1000 0.0103
## 6 1.1210 nan 0.1000 0.0071
## 7 1.1028 nan 0.1000 0.0070
## 8 1.0873 nan 0.1000 0.0066
## 9 1.0731 nan 0.1000 0.0054
## 10 1.0614 nan 0.1000 0.0052
## 20 0.9700 nan 0.1000 0.0012
## 40 0.8776 nan 0.1000 0.0001
## 60 0.8367 nan 0.1000 0.0004
## 80 0.8130 nan 0.1000 -0.0005
## 100 0.7955 nan 0.1000 -0.0017
## 120 0.7786 nan 0.1000 -0.0014
## 140 0.7671 nan 0.1000 -0.0016
## 150 0.7612 nan 0.1000 -0.0014
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2434 nan 0.1000 0.0183
## 2 1.1989 nan 0.1000 0.0206
## 3 1.1655 nan 0.1000 0.0151
## 4 1.1313 nan 0.1000 0.0125
## 5 1.1031 nan 0.1000 0.0110
## 6 1.0745 nan 0.1000 0.0097
## 7 1.0525 nan 0.1000 0.0075
## 8 1.0366 nan 0.1000 0.0060
## 9 1.0172 nan 0.1000 0.0073
## 10 1.0076 nan 0.1000 0.0012
## 20 0.8946 nan 0.1000 0.0035
## 40 0.8070 nan 0.1000 -0.0003
## 60 0.7662 nan 0.1000 -0.0004
## 80 0.7345 nan 0.1000 -0.0006
## 100 0.7074 nan 0.1000 -0.0004
## 120 0.6797 nan 0.1000 -0.0023
## 140 0.6554 nan 0.1000 -0.0009
## 150 0.6467 nan 0.1000 -0.0014
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2389 nan 0.1000 0.0239
## 2 1.1879 nan 0.1000 0.0247
## 3 1.1467 nan 0.1000 0.0171
## 4 1.1089 nan 0.1000 0.0140
## 5 1.0809 nan 0.1000 0.0104
## 6 1.0519 nan 0.1000 0.0118
## 7 1.0293 nan 0.1000 0.0074
## 8 1.0020 nan 0.1000 0.0111
## 9 0.9833 nan 0.1000 0.0076
## 10 0.9687 nan 0.1000 0.0043
## 20 0.8555 nan 0.1000 0.0021
## 40 0.7502 nan 0.1000 -0.0031
## 60 0.6995 nan 0.1000 -0.0027
## 80 0.6536 nan 0.1000 -0.0018
## 100 0.6127 nan 0.1000 -0.0023
## 120 0.5803 nan 0.1000 -0.0017
## 140 0.5508 nan 0.1000 -0.0009
## 150 0.5374 nan 0.1000 -0.0023
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2588 nan 0.1000 0.0157
## 2 1.2228 nan 0.1000 0.0155
## 3 1.1940 nan 0.1000 0.0133
## 4 1.1731 nan 0.1000 0.0094
## 5 1.1552 nan 0.1000 0.0078
## 6 1.1384 nan 0.1000 0.0074
## 7 1.1218 nan 0.1000 0.0083
## 8 1.1063 nan 0.1000 0.0055
## 9 1.0929 nan 0.1000 0.0046
## 10 1.0784 nan 0.1000 0.0042
## 20 0.9887 nan 0.1000 0.0010
## 40 0.9025 nan 0.1000 -0.0007
## 60 0.8600 nan 0.1000 0.0005
## 80 0.8355 nan 0.1000 -0.0014
## 100 0.8190 nan 0.1000 -0.0008
## 120 0.8092 nan 0.1000 -0.0007
## 140 0.7993 nan 0.1000 -0.0009
## 150 0.7941 nan 0.1000 -0.0003
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2491 nan 0.1000 0.0222
## 2 1.2070 nan 0.1000 0.0149
## 3 1.1695 nan 0.1000 0.0177
## 4 1.1432 nan 0.1000 0.0113
## 5 1.1183 nan 0.1000 0.0107
## 6 1.0921 nan 0.1000 0.0090
## 7 1.0657 nan 0.1000 0.0113
## 8 1.0485 nan 0.1000 0.0062
## 9 1.0334 nan 0.1000 0.0046
## 10 1.0153 nan 0.1000 0.0042
## 20 0.9111 nan 0.1000 0.0007
## 40 0.8269 nan 0.1000 -0.0015
## 60 0.7868 nan 0.1000 -0.0006
## 80 0.7544 nan 0.1000 -0.0023
## 100 0.7215 nan 0.1000 -0.0019
## 120 0.6952 nan 0.1000 -0.0010
## 140 0.6726 nan 0.1000 -0.0030
## 150 0.6634 nan 0.1000 -0.0026
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2451 nan 0.1000 0.0229
## 2 1.1942 nan 0.1000 0.0193
## 3 1.1563 nan 0.1000 0.0172
## 4 1.1259 nan 0.1000 0.0158
## 5 1.0974 nan 0.1000 0.0115
## 6 1.0699 nan 0.1000 0.0078
## 7 1.0482 nan 0.1000 0.0080
## 8 1.0299 nan 0.1000 0.0036
## 9 1.0069 nan 0.1000 0.0069
## 10 0.9918 nan 0.1000 0.0030
## 20 0.8870 nan 0.1000 -0.0018
## 40 0.7869 nan 0.1000 -0.0014
## 60 0.7314 nan 0.1000 -0.0019
## 80 0.6910 nan 0.1000 -0.0013
## 100 0.6485 nan 0.1000 -0.0037
## 120 0.6130 nan 0.1000 -0.0020
## 140 0.5764 nan 0.1000 -0.0003
## 150 0.5596 nan 0.1000 -0.0023
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2613 nan 0.1000 0.0168
## 2 1.2306 nan 0.1000 0.0142
## 3 1.2036 nan 0.1000 0.0116
## 4 1.1828 nan 0.1000 0.0082
## 5 1.1619 nan 0.1000 0.0088
## 6 1.1449 nan 0.1000 0.0071
## 7 1.1310 nan 0.1000 0.0055
## 8 1.1148 nan 0.1000 0.0063
## 9 1.0999 nan 0.1000 0.0051
## 10 1.0881 nan 0.1000 0.0049
## 20 0.9994 nan 0.1000 0.0033
## 40 0.9152 nan 0.1000 0.0010
## 60 0.8683 nan 0.1000 -0.0003
## 80 0.8443 nan 0.1000 -0.0008
## 100 0.8287 nan 0.1000 -0.0014
## 120 0.8142 nan 0.1000 -0.0021
## 140 0.8037 nan 0.1000 -0.0012
## 150 0.7987 nan 0.1000 -0.0014
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2519 nan 0.1000 0.0202
## 2 1.2090 nan 0.1000 0.0155
## 3 1.1798 nan 0.1000 0.0152
## 4 1.1551 nan 0.1000 0.0102
## 5 1.1278 nan 0.1000 0.0126
## 6 1.1020 nan 0.1000 0.0103
## 7 1.0871 nan 0.1000 0.0018
## 8 1.0653 nan 0.1000 0.0067
## 9 1.0472 nan 0.1000 0.0058
## 10 1.0315 nan 0.1000 0.0046
## 20 0.9270 nan 0.1000 0.0000
## 40 0.8350 nan 0.1000 -0.0020
## 60 0.7952 nan 0.1000 -0.0000
## 80 0.7645 nan 0.1000 -0.0010
## 100 0.7340 nan 0.1000 -0.0019
## 120 0.7130 nan 0.1000 -0.0026
## 140 0.6899 nan 0.1000 -0.0011
## 150 0.6769 nan 0.1000 -0.0028
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2389 nan 0.1000 0.0215
## 2 1.1962 nan 0.1000 0.0168
## 3 1.1624 nan 0.1000 0.0130
## 4 1.1274 nan 0.1000 0.0147
## 5 1.0980 nan 0.1000 0.0108
## 6 1.0749 nan 0.1000 0.0069
## 7 1.0483 nan 0.1000 0.0099
## 8 1.0271 nan 0.1000 0.0080
## 9 1.0052 nan 0.1000 0.0078
## 10 0.9919 nan 0.1000 0.0022
## 20 0.8821 nan 0.1000 0.0013
## 40 0.7869 nan 0.1000 -0.0009
## 60 0.7353 nan 0.1000 -0.0023
## 80 0.6886 nan 0.1000 -0.0010
## 100 0.6475 nan 0.1000 -0.0026
## 120 0.6205 nan 0.1000 -0.0018
## 140 0.5986 nan 0.1000 -0.0021
## 150 0.5811 nan 0.1000 -0.0019
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2585 nan 0.1000 0.0195
## 2 1.2265 nan 0.1000 0.0123
## 3 1.1977 nan 0.1000 0.0111
## 4 1.1718 nan 0.1000 0.0114
## 5 1.1505 nan 0.1000 0.0061
## 6 1.1282 nan 0.1000 0.0073
## 7 1.1132 nan 0.1000 0.0057
## 8 1.0977 nan 0.1000 0.0066
## 9 1.0833 nan 0.1000 0.0044
## 10 1.0701 nan 0.1000 0.0053
## 20 0.9761 nan 0.1000 0.0017
## 40 0.8967 nan 0.1000 0.0010
## 60 0.8567 nan 0.1000 -0.0011
## 80 0.8280 nan 0.1000 -0.0003
## 100 0.8069 nan 0.1000 -0.0010
## 120 0.7954 nan 0.1000 -0.0015
## 140 0.7837 nan 0.1000 -0.0007
## 150 0.7812 nan 0.1000 -0.0013
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2463 nan 0.1000 0.0204
## 2 1.2060 nan 0.1000 0.0178
## 3 1.1756 nan 0.1000 0.0147
## 4 1.1440 nan 0.1000 0.0123
## 5 1.1142 nan 0.1000 0.0110
## 6 1.0906 nan 0.1000 0.0110
## 7 1.0741 nan 0.1000 0.0054
## 8 1.0559 nan 0.1000 0.0093
## 9 1.0386 nan 0.1000 0.0061
## 10 1.0215 nan 0.1000 0.0063
## 20 0.9113 nan 0.1000 -0.0001
## 40 0.8246 nan 0.1000 -0.0006
## 60 0.7780 nan 0.1000 -0.0003
## 80 0.7429 nan 0.1000 -0.0008
## 100 0.7206 nan 0.1000 -0.0008
## 120 0.7002 nan 0.1000 -0.0018
## 140 0.6790 nan 0.1000 -0.0018
## 150 0.6677 nan 0.1000 -0.0009
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2443 nan 0.1000 0.0213
## 2 1.1920 nan 0.1000 0.0194
## 3 1.1546 nan 0.1000 0.0166
## 4 1.1212 nan 0.1000 0.0161
## 5 1.0925 nan 0.1000 0.0088
## 6 1.0660 nan 0.1000 0.0089
## 7 1.0410 nan 0.1000 0.0092
## 8 1.0208 nan 0.1000 0.0078
## 9 1.0016 nan 0.1000 0.0042
## 10 0.9847 nan 0.1000 0.0060
## 20 0.8639 nan 0.1000 0.0006
## 40 0.7668 nan 0.1000 -0.0022
## 60 0.7150 nan 0.1000 -0.0021
## 80 0.6626 nan 0.1000 -0.0020
## 100 0.6227 nan 0.1000 -0.0022
## 120 0.5900 nan 0.1000 -0.0026
## 140 0.5617 nan 0.1000 -0.0018
## 150 0.5428 nan 0.1000 -0.0020
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2591 nan 0.1000 0.0134
## 2 1.2288 nan 0.1000 0.0126
## 3 1.2016 nan 0.1000 0.0112
## 4 1.1776 nan 0.1000 0.0111
## 5 1.1575 nan 0.1000 0.0087
## 6 1.1406 nan 0.1000 0.0072
## 7 1.1231 nan 0.1000 0.0082
## 8 1.1066 nan 0.1000 0.0065
## 9 1.0979 nan 0.1000 0.0031
## 10 1.0851 nan 0.1000 0.0039
## 20 0.9857 nan 0.1000 0.0009
## 40 0.8954 nan 0.1000 -0.0000
## 60 0.8566 nan 0.1000 -0.0007
## 80 0.8281 nan 0.1000 -0.0003
## 100 0.8137 nan 0.1000 -0.0012
## 120 0.8043 nan 0.1000 -0.0013
## 140 0.7942 nan 0.1000 -0.0010
## 150 0.7900 nan 0.1000 -0.0012
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2491 nan 0.1000 0.0217
## 2 1.2030 nan 0.1000 0.0182
## 3 1.1703 nan 0.1000 0.0120
## 4 1.1441 nan 0.1000 0.0125
## 5 1.1189 nan 0.1000 0.0095
## 6 1.0963 nan 0.1000 0.0080
## 7 1.0732 nan 0.1000 0.0068
## 8 1.0538 nan 0.1000 0.0068
## 9 1.0347 nan 0.1000 0.0066
## 10 1.0199 nan 0.1000 0.0060
## 20 0.9178 nan 0.1000 0.0024
## 40 0.8319 nan 0.1000 -0.0018
## 60 0.7867 nan 0.1000 -0.0007
## 80 0.7519 nan 0.1000 -0.0012
## 100 0.7239 nan 0.1000 -0.0014
## 120 0.7022 nan 0.1000 -0.0021
## 140 0.6807 nan 0.1000 -0.0027
## 150 0.6699 nan 0.1000 -0.0011
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2363 nan 0.1000 0.0217
## 2 1.1923 nan 0.1000 0.0192
## 3 1.1510 nan 0.1000 0.0203
## 4 1.1220 nan 0.1000 0.0113
## 5 1.0901 nan 0.1000 0.0126
## 6 1.0616 nan 0.1000 0.0127
## 7 1.0379 nan 0.1000 0.0091
## 8 1.0214 nan 0.1000 0.0044
## 9 1.0059 nan 0.1000 0.0057
## 10 0.9864 nan 0.1000 0.0062
## 20 0.8855 nan 0.1000 0.0018
## 40 0.7918 nan 0.1000 -0.0028
## 60 0.7272 nan 0.1000 -0.0003
## 80 0.6825 nan 0.1000 -0.0016
## 100 0.6495 nan 0.1000 -0.0016
## 120 0.6155 nan 0.1000 -0.0039
## 140 0.5801 nan 0.1000 -0.0010
## 150 0.5667 nan 0.1000 -0.0017
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2543 nan 0.1000 0.0159
## 2 1.2212 nan 0.1000 0.0157
## 3 1.1907 nan 0.1000 0.0112
## 4 1.1682 nan 0.1000 0.0105
## 5 1.1508 nan 0.1000 0.0073
## 6 1.1364 nan 0.1000 0.0056
## 7 1.1235 nan 0.1000 0.0044
## 8 1.1093 nan 0.1000 0.0054
## 9 1.0966 nan 0.1000 0.0059
## 10 1.0838 nan 0.1000 0.0035
## 20 0.9974 nan 0.1000 0.0016
## 40 0.9195 nan 0.1000 0.0009
## 60 0.8783 nan 0.1000 -0.0002
## 80 0.8537 nan 0.1000 -0.0016
## 100 0.8416 nan 0.1000 -0.0018
## 120 0.8267 nan 0.1000 -0.0020
## 140 0.8158 nan 0.1000 -0.0037
## 150 0.8093 nan 0.1000 -0.0015
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2440 nan 0.1000 0.0250
## 2 1.2051 nan 0.1000 0.0150
## 3 1.1739 nan 0.1000 0.0143
## 4 1.1527 nan 0.1000 0.0089
## 5 1.1241 nan 0.1000 0.0124
## 6 1.1017 nan 0.1000 0.0077
## 7 1.0787 nan 0.1000 0.0089
## 8 1.0649 nan 0.1000 0.0045
## 9 1.0469 nan 0.1000 0.0074
## 10 1.0336 nan 0.1000 0.0040
## 20 0.9290 nan 0.1000 0.0018
## 40 0.8393 nan 0.1000 -0.0002
## 60 0.7926 nan 0.1000 -0.0028
## 80 0.7530 nan 0.1000 0.0002
## 100 0.7258 nan 0.1000 -0.0020
## 120 0.7023 nan 0.1000 -0.0021
## 140 0.6822 nan 0.1000 -0.0008
## 150 0.6712 nan 0.1000 -0.0006
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2433 nan 0.1000 0.0227
## 2 1.2019 nan 0.1000 0.0200
## 3 1.1682 nan 0.1000 0.0117
## 4 1.1322 nan 0.1000 0.0136
## 5 1.1060 nan 0.1000 0.0112
## 6 1.0813 nan 0.1000 0.0100
## 7 1.0610 nan 0.1000 0.0069
## 8 1.0391 nan 0.1000 0.0069
## 9 1.0213 nan 0.1000 0.0083
## 10 1.0047 nan 0.1000 0.0051
## 20 0.8940 nan 0.1000 0.0008
## 40 0.8002 nan 0.1000 -0.0009
## 60 0.7404 nan 0.1000 -0.0003
## 80 0.7030 nan 0.1000 -0.0004
## 100 0.6634 nan 0.1000 -0.0013
## 120 0.6280 nan 0.1000 0.0006
## 140 0.5969 nan 0.1000 -0.0009
## 150 0.5791 nan 0.1000 -0.0008
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2526 nan 0.1000 0.0172
## 2 1.2213 nan 0.1000 0.0181
## 3 1.1934 nan 0.1000 0.0125
## 4 1.1690 nan 0.1000 0.0114
## 5 1.1509 nan 0.1000 0.0080
## 6 1.1340 nan 0.1000 0.0066
## 7 1.1189 nan 0.1000 0.0051
## 8 1.1038 nan 0.1000 0.0067
## 9 1.0898 nan 0.1000 0.0063
## 10 1.0746 nan 0.1000 0.0048
## 20 0.9832 nan 0.1000 0.0036
## 40 0.8975 nan 0.1000 0.0012
## 60 0.8589 nan 0.1000 -0.0012
## 80 0.8438 nan 0.1000 -0.0024
## 100 0.8260 nan 0.1000 -0.0004
## 120 0.8101 nan 0.1000 -0.0006
## 140 0.8006 nan 0.1000 -0.0005
## 150 0.7950 nan 0.1000 -0.0006
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2461 nan 0.1000 0.0189
## 2 1.2045 nan 0.1000 0.0160
## 3 1.1668 nan 0.1000 0.0137
## 4 1.1335 nan 0.1000 0.0135
## 5 1.1094 nan 0.1000 0.0091
## 6 1.0851 nan 0.1000 0.0098
## 7 1.0667 nan 0.1000 0.0062
## 8 1.0483 nan 0.1000 0.0089
## 9 1.0297 nan 0.1000 0.0048
## 10 1.0132 nan 0.1000 0.0034
## 20 0.9042 nan 0.1000 0.0024
## 40 0.8231 nan 0.1000 0.0004
## 60 0.7836 nan 0.1000 -0.0008
## 80 0.7543 nan 0.1000 -0.0022
## 100 0.7279 nan 0.1000 -0.0010
## 120 0.6952 nan 0.1000 -0.0025
## 140 0.6726 nan 0.1000 -0.0011
## 150 0.6607 nan 0.1000 -0.0010
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2404 nan 0.1000 0.0232
## 2 1.1966 nan 0.1000 0.0202
## 3 1.1561 nan 0.1000 0.0158
## 4 1.1182 nan 0.1000 0.0152
## 5 1.0898 nan 0.1000 0.0093
## 6 1.0594 nan 0.1000 0.0110
## 7 1.0391 nan 0.1000 0.0040
## 8 1.0211 nan 0.1000 0.0021
## 9 1.0038 nan 0.1000 0.0026
## 10 0.9845 nan 0.1000 0.0050
## 20 0.8722 nan 0.1000 0.0006
## 40 0.7864 nan 0.1000 -0.0019
## 60 0.7344 nan 0.1000 -0.0026
## 80 0.6937 nan 0.1000 -0.0018
## 100 0.6504 nan 0.1000 -0.0033
## 120 0.6194 nan 0.1000 -0.0013
## 140 0.5825 nan 0.1000 -0.0009
## 150 0.5678 nan 0.1000 -0.0016
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2627 nan 0.1000 0.0149
## 2 1.2354 nan 0.1000 0.0142
## 3 1.2040 nan 0.1000 0.0125
## 4 1.1838 nan 0.1000 0.0079
## 5 1.1613 nan 0.1000 0.0079
## 6 1.1434 nan 0.1000 0.0072
## 7 1.1238 nan 0.1000 0.0060
## 8 1.1133 nan 0.1000 0.0049
## 9 1.1009 nan 0.1000 0.0043
## 10 1.0911 nan 0.1000 0.0025
## 20 0.9978 nan 0.1000 0.0020
## 40 0.9137 nan 0.1000 0.0009
## 60 0.8723 nan 0.1000 -0.0007
## 80 0.8435 nan 0.1000 -0.0005
## 100 0.8273 nan 0.1000 -0.0015
## 120 0.8136 nan 0.1000 -0.0011
## 140 0.8044 nan 0.1000 -0.0009
## 150 0.7991 nan 0.1000 -0.0004
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2519 nan 0.1000 0.0200
## 2 1.2177 nan 0.1000 0.0163
## 3 1.1825 nan 0.1000 0.0158
## 4 1.1595 nan 0.1000 0.0067
## 5 1.1322 nan 0.1000 0.0108
## 6 1.1093 nan 0.1000 0.0083
## 7 1.0904 nan 0.1000 0.0064
## 8 1.0700 nan 0.1000 0.0077
## 9 1.0540 nan 0.1000 0.0058
## 10 1.0365 nan 0.1000 0.0077
## 20 0.9274 nan 0.1000 0.0001
## 40 0.8399 nan 0.1000 -0.0002
## 60 0.8015 nan 0.1000 -0.0019
## 80 0.7691 nan 0.1000 -0.0013
## 100 0.7397 nan 0.1000 -0.0012
## 120 0.7122 nan 0.1000 -0.0013
## 140 0.6859 nan 0.1000 -0.0015
## 150 0.6760 nan 0.1000 -0.0016
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2413 nan 0.1000 0.0183
## 2 1.2020 nan 0.1000 0.0147
## 3 1.1590 nan 0.1000 0.0168
## 4 1.1310 nan 0.1000 0.0106
## 5 1.0946 nan 0.1000 0.0122
## 6 1.0648 nan 0.1000 0.0110
## 7 1.0440 nan 0.1000 0.0081
## 8 1.0239 nan 0.1000 0.0074
## 9 1.0055 nan 0.1000 0.0043
## 10 0.9867 nan 0.1000 0.0037
## 20 0.8876 nan 0.1000 -0.0015
## 40 0.8011 nan 0.1000 -0.0025
## 60 0.7390 nan 0.1000 -0.0031
## 80 0.6999 nan 0.1000 -0.0020
## 100 0.6601 nan 0.1000 -0.0008
## 120 0.6270 nan 0.1000 -0.0029
## 140 0.6009 nan 0.1000 -0.0012
## 150 0.5835 nan 0.1000 -0.0005
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2532 nan 0.1000 0.0173
## 2 1.2216 nan 0.1000 0.0141
## 3 1.1937 nan 0.1000 0.0118
## 4 1.1703 nan 0.1000 0.0094
## 5 1.1525 nan 0.1000 0.0069
## 6 1.1332 nan 0.1000 0.0081
## 7 1.1166 nan 0.1000 0.0054
## 8 1.1021 nan 0.1000 0.0060
## 9 1.0905 nan 0.1000 0.0050
## 10 1.0725 nan 0.1000 0.0061
## 20 0.9803 nan 0.1000 0.0001
## 40 0.8805 nan 0.1000 0.0004
## 60 0.8383 nan 0.1000 -0.0018
## 80 0.8108 nan 0.1000 -0.0004
## 100 0.7938 nan 0.1000 -0.0009
## 120 0.7786 nan 0.1000 -0.0006
## 140 0.7685 nan 0.1000 -0.0014
## 150 0.7627 nan 0.1000 -0.0011
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2410 nan 0.1000 0.0217
## 2 1.2042 nan 0.1000 0.0162
## 3 1.1722 nan 0.1000 0.0127
## 4 1.1430 nan 0.1000 0.0121
## 5 1.1134 nan 0.1000 0.0136
## 6 1.0893 nan 0.1000 0.0092
## 7 1.0685 nan 0.1000 0.0074
## 8 1.0455 nan 0.1000 0.0083
## 9 1.0231 nan 0.1000 0.0077
## 10 1.0091 nan 0.1000 0.0038
## 20 0.9041 nan 0.1000 0.0041
## 40 0.8104 nan 0.1000 -0.0010
## 60 0.7694 nan 0.1000 -0.0022
## 80 0.7367 nan 0.1000 -0.0012
## 100 0.7060 nan 0.1000 -0.0010
## 120 0.6810 nan 0.1000 -0.0012
## 140 0.6597 nan 0.1000 -0.0014
## 150 0.6468 nan 0.1000 -0.0003
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2369 nan 0.1000 0.0250
## 2 1.1950 nan 0.1000 0.0165
## 3 1.1517 nan 0.1000 0.0163
## 4 1.1153 nan 0.1000 0.0141
## 5 1.0779 nan 0.1000 0.0166
## 6 1.0516 nan 0.1000 0.0104
## 7 1.0288 nan 0.1000 0.0081
## 8 1.0081 nan 0.1000 0.0064
## 9 0.9877 nan 0.1000 0.0074
## 10 0.9689 nan 0.1000 0.0064
## 20 0.8531 nan 0.1000 -0.0008
## 40 0.7586 nan 0.1000 -0.0044
## 60 0.7063 nan 0.1000 -0.0022
## 80 0.6674 nan 0.1000 -0.0027
## 100 0.6348 nan 0.1000 -0.0016
## 120 0.5973 nan 0.1000 -0.0010
## 140 0.5666 nan 0.1000 -0.0004
## 150 0.5549 nan 0.1000 -0.0027
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2595 nan 0.1000 0.0186
## 2 1.2324 nan 0.1000 0.0147
## 3 1.2031 nan 0.1000 0.0150
## 4 1.1788 nan 0.1000 0.0118
## 5 1.1586 nan 0.1000 0.0086
## 6 1.1400 nan 0.1000 0.0082
## 7 1.1276 nan 0.1000 0.0050
## 8 1.1130 nan 0.1000 0.0037
## 9 1.0983 nan 0.1000 0.0060
## 10 1.0841 nan 0.1000 0.0065
## 20 0.9872 nan 0.1000 0.0028
## 40 0.9026 nan 0.1000 0.0006
## 60 0.8691 nan 0.1000 -0.0019
## 80 0.8391 nan 0.1000 -0.0016
## 100 0.8197 nan 0.1000 -0.0016
## 120 0.8085 nan 0.1000 -0.0007
## 140 0.7985 nan 0.1000 -0.0010
## 150 0.7933 nan 0.1000 -0.0017
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2443 nan 0.1000 0.0205
## 2 1.2058 nan 0.1000 0.0151
## 3 1.1755 nan 0.1000 0.0133
## 4 1.1483 nan 0.1000 0.0084
## 5 1.1201 nan 0.1000 0.0101
## 6 1.0959 nan 0.1000 0.0083
## 7 1.0722 nan 0.1000 0.0051
## 8 1.0525 nan 0.1000 0.0070
## 9 1.0329 nan 0.1000 0.0071
## 10 1.0178 nan 0.1000 0.0057
## 20 0.9238 nan 0.1000 0.0018
## 40 0.8307 nan 0.1000 -0.0005
## 60 0.7900 nan 0.1000 -0.0027
## 80 0.7543 nan 0.1000 -0.0018
## 100 0.7196 nan 0.1000 -0.0011
## 120 0.6958 nan 0.1000 -0.0010
## 140 0.6734 nan 0.1000 -0.0026
## 150 0.6626 nan 0.1000 -0.0022
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2359 nan 0.1000 0.0225
## 2 1.1974 nan 0.1000 0.0136
## 3 1.1527 nan 0.1000 0.0201
## 4 1.1164 nan 0.1000 0.0141
## 5 1.0820 nan 0.1000 0.0124
## 6 1.0578 nan 0.1000 0.0067
## 7 1.0395 nan 0.1000 0.0058
## 8 1.0183 nan 0.1000 0.0062
## 9 1.0017 nan 0.1000 0.0064
## 10 0.9875 nan 0.1000 0.0028
## 20 0.8765 nan 0.1000 0.0005
## 40 0.7793 nan 0.1000 -0.0004
## 60 0.7237 nan 0.1000 -0.0010
## 80 0.6777 nan 0.1000 -0.0018
## 100 0.6360 nan 0.1000 -0.0022
## 120 0.5939 nan 0.1000 -0.0015
## 140 0.5607 nan 0.1000 -0.0010
## 150 0.5484 nan 0.1000 -0.0026
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2519 nan 0.1000 0.0190
## 2 1.2144 nan 0.1000 0.0133
## 3 1.1889 nan 0.1000 0.0126
## 4 1.1669 nan 0.1000 0.0098
## 5 1.1443 nan 0.1000 0.0095
## 6 1.1239 nan 0.1000 0.0071
## 7 1.1080 nan 0.1000 0.0078
## 8 1.0931 nan 0.1000 0.0062
## 9 1.0773 nan 0.1000 0.0070
## 10 1.0655 nan 0.1000 0.0023
## 20 0.9704 nan 0.1000 0.0021
## 40 0.8781 nan 0.1000 0.0008
## 60 0.8334 nan 0.1000 -0.0002
## 80 0.8062 nan 0.1000 -0.0013
## 100 0.7927 nan 0.1000 -0.0014
## 120 0.7776 nan 0.1000 -0.0016
## 140 0.7638 nan 0.1000 -0.0000
## 150 0.7620 nan 0.1000 -0.0020
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2396 nan 0.1000 0.0249
## 2 1.1953 nan 0.1000 0.0171
## 3 1.1574 nan 0.1000 0.0158
## 4 1.1263 nan 0.1000 0.0102
## 5 1.1012 nan 0.1000 0.0111
## 6 1.0758 nan 0.1000 0.0098
## 7 1.0556 nan 0.1000 0.0064
## 8 1.0358 nan 0.1000 0.0055
## 9 1.0182 nan 0.1000 0.0069
## 10 1.0074 nan 0.1000 0.0005
## 20 0.9018 nan 0.1000 0.0020
## 40 0.8041 nan 0.1000 -0.0002
## 60 0.7581 nan 0.1000 -0.0016
## 80 0.7276 nan 0.1000 -0.0013
## 100 0.6904 nan 0.1000 -0.0020
## 120 0.6641 nan 0.1000 -0.0008
## 140 0.6401 nan 0.1000 -0.0009
## 150 0.6324 nan 0.1000 -0.0011
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2364 nan 0.1000 0.0254
## 2 1.1889 nan 0.1000 0.0184
## 3 1.1474 nan 0.1000 0.0170
## 4 1.1151 nan 0.1000 0.0128
## 5 1.0743 nan 0.1000 0.0172
## 6 1.0521 nan 0.1000 0.0076
## 7 1.0278 nan 0.1000 0.0085
## 8 1.0075 nan 0.1000 0.0091
## 9 0.9907 nan 0.1000 0.0050
## 10 0.9742 nan 0.1000 0.0036
## 20 0.8570 nan 0.1000 0.0018
## 40 0.7517 nan 0.1000 -0.0004
## 60 0.6954 nan 0.1000 -0.0030
## 80 0.6457 nan 0.1000 -0.0020
## 100 0.6042 nan 0.1000 -0.0004
## 120 0.5698 nan 0.1000 -0.0020
## 140 0.5388 nan 0.1000 -0.0019
## 150 0.5266 nan 0.1000 -0.0026
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2521 nan 0.1000 0.0178
## 2 1.2323 nan 0.1000 0.0070
## 3 1.1983 nan 0.1000 0.0137
## 4 1.1731 nan 0.1000 0.0116
## 5 1.1534 nan 0.1000 0.0087
## 6 1.1346 nan 0.1000 0.0085
## 7 1.1222 nan 0.1000 0.0064
## 8 1.1093 nan 0.1000 0.0047
## 9 1.0905 nan 0.1000 0.0043
## 10 1.0795 nan 0.1000 0.0032
## 20 0.9956 nan 0.1000 -0.0005
## 40 0.9101 nan 0.1000 0.0008
## 60 0.8701 nan 0.1000 -0.0010
## 80 0.8415 nan 0.1000 -0.0003
## 100 0.8230 nan 0.1000 -0.0002
## 120 0.8070 nan 0.1000 -0.0010
## 140 0.8007 nan 0.1000 -0.0017
## 150 0.7973 nan 0.1000 -0.0011
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2524 nan 0.1000 0.0214
## 2 1.2094 nan 0.1000 0.0190
## 3 1.1790 nan 0.1000 0.0124
## 4 1.1508 nan 0.1000 0.0134
## 5 1.1226 nan 0.1000 0.0108
## 6 1.1014 nan 0.1000 0.0097
## 7 1.0822 nan 0.1000 0.0068
## 8 1.0646 nan 0.1000 0.0048
## 9 1.0438 nan 0.1000 0.0079
## 10 1.0243 nan 0.1000 0.0062
## 20 0.9279 nan 0.1000 0.0021
## 40 0.8368 nan 0.1000 -0.0013
## 60 0.7948 nan 0.1000 -0.0007
## 80 0.7623 nan 0.1000 -0.0008
## 100 0.7322 nan 0.1000 -0.0013
## 120 0.7077 nan 0.1000 -0.0006
## 140 0.6776 nan 0.1000 -0.0007
## 150 0.6672 nan 0.1000 -0.0011
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2452 nan 0.1000 0.0172
## 2 1.1980 nan 0.1000 0.0210
## 3 1.1606 nan 0.1000 0.0151
## 4 1.1280 nan 0.1000 0.0123
## 5 1.0933 nan 0.1000 0.0102
## 6 1.0661 nan 0.1000 0.0098
## 7 1.0433 nan 0.1000 0.0056
## 8 1.0263 nan 0.1000 0.0015
## 9 1.0072 nan 0.1000 0.0050
## 10 0.9880 nan 0.1000 0.0073
## 20 0.8745 nan 0.1000 -0.0021
## 40 0.7756 nan 0.1000 -0.0022
## 60 0.7180 nan 0.1000 -0.0030
## 80 0.6726 nan 0.1000 -0.0011
## 100 0.6327 nan 0.1000 -0.0008
## 120 0.5973 nan 0.1000 -0.0012
## 140 0.5683 nan 0.1000 -0.0024
## 150 0.5564 nan 0.1000 -0.0025
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2534 nan 0.1000 0.0161
## 2 1.2223 nan 0.1000 0.0149
## 3 1.1958 nan 0.1000 0.0104
## 4 1.1714 nan 0.1000 0.0117
## 5 1.1534 nan 0.1000 0.0056
## 6 1.1382 nan 0.1000 0.0052
## 7 1.1192 nan 0.1000 0.0081
## 8 1.1063 nan 0.1000 0.0059
## 9 1.0914 nan 0.1000 0.0064
## 10 1.0749 nan 0.1000 0.0058
## 20 0.9763 nan 0.1000 0.0026
## 40 0.8825 nan 0.1000 0.0000
## 60 0.8390 nan 0.1000 -0.0005
## 80 0.8142 nan 0.1000 -0.0013
## 100 0.7948 nan 0.1000 -0.0006
## 120 0.7807 nan 0.1000 -0.0010
## 140 0.7662 nan 0.1000 -0.0021
## 150 0.7607 nan 0.1000 -0.0019
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2422 nan 0.1000 0.0218
## 2 1.2076 nan 0.1000 0.0157
## 3 1.1742 nan 0.1000 0.0086
## 4 1.1385 nan 0.1000 0.0157
## 5 1.1056 nan 0.1000 0.0144
## 6 1.0830 nan 0.1000 0.0099
## 7 1.0589 nan 0.1000 0.0094
## 8 1.0410 nan 0.1000 0.0076
## 9 1.0212 nan 0.1000 0.0077
## 10 1.0061 nan 0.1000 0.0044
## 20 0.9001 nan 0.1000 0.0011
## 40 0.8036 nan 0.1000 -0.0004
## 60 0.7627 nan 0.1000 -0.0029
## 80 0.7324 nan 0.1000 -0.0014
## 100 0.6989 nan 0.1000 -0.0002
## 120 0.6746 nan 0.1000 -0.0020
## 140 0.6529 nan 0.1000 -0.0027
## 150 0.6420 nan 0.1000 -0.0017
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2390 nan 0.1000 0.0240
## 2 1.1886 nan 0.1000 0.0200
## 3 1.1479 nan 0.1000 0.0174
## 4 1.1121 nan 0.1000 0.0147
## 5 1.0788 nan 0.1000 0.0144
## 6 1.0528 nan 0.1000 0.0126
## 7 1.0288 nan 0.1000 0.0069
## 8 1.0100 nan 0.1000 0.0094
## 9 0.9898 nan 0.1000 0.0082
## 10 0.9696 nan 0.1000 0.0040
## 20 0.8595 nan 0.1000 0.0013
## 40 0.7579 nan 0.1000 -0.0007
## 60 0.7014 nan 0.1000 -0.0030
## 80 0.6532 nan 0.1000 -0.0016
## 100 0.6146 nan 0.1000 -0.0023
## 120 0.5753 nan 0.1000 -0.0025
## 140 0.5439 nan 0.1000 -0.0010
## 150 0.5333 nan 0.1000 -0.0017
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2615 nan 0.1000 0.0187
## 2 1.2315 nan 0.1000 0.0137
## 3 1.2037 nan 0.1000 0.0136
## 4 1.1816 nan 0.1000 0.0112
## 5 1.1616 nan 0.1000 0.0061
## 6 1.1407 nan 0.1000 0.0081
## 7 1.1221 nan 0.1000 0.0058
## 8 1.1061 nan 0.1000 0.0050
## 9 1.0931 nan 0.1000 0.0046
## 10 1.0785 nan 0.1000 0.0062
## 20 0.9857 nan 0.1000 0.0038
## 40 0.8982 nan 0.1000 0.0013
## 60 0.8592 nan 0.1000 -0.0002
## 80 0.8346 nan 0.1000 -0.0023
## 100 0.8214 nan 0.1000 -0.0006
## 120 0.8101 nan 0.1000 -0.0016
## 140 0.7961 nan 0.1000 -0.0012
## 150 0.7919 nan 0.1000 -0.0007
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2525 nan 0.1000 0.0194
## 2 1.2172 nan 0.1000 0.0138
## 3 1.1819 nan 0.1000 0.0179
## 4 1.1517 nan 0.1000 0.0145
## 5 1.1263 nan 0.1000 0.0102
## 6 1.1059 nan 0.1000 0.0063
## 7 1.0875 nan 0.1000 0.0064
## 8 1.0683 nan 0.1000 0.0056
## 9 1.0453 nan 0.1000 0.0112
## 10 1.0280 nan 0.1000 0.0068
## 20 0.9213 nan 0.1000 -0.0009
## 40 0.8356 nan 0.1000 -0.0025
## 60 0.7958 nan 0.1000 -0.0014
## 80 0.7601 nan 0.1000 0.0003
## 100 0.7258 nan 0.1000 -0.0008
## 120 0.6983 nan 0.1000 -0.0011
## 140 0.6735 nan 0.1000 -0.0016
## 150 0.6594 nan 0.1000 -0.0017
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2375 nan 0.1000 0.0252
## 2 1.1858 nan 0.1000 0.0209
## 3 1.1433 nan 0.1000 0.0165
## 4 1.1113 nan 0.1000 0.0139
## 5 1.0847 nan 0.1000 0.0127
## 6 1.0602 nan 0.1000 0.0088
## 7 1.0373 nan 0.1000 0.0074
## 8 1.0156 nan 0.1000 0.0079
## 9 0.9994 nan 0.1000 0.0070
## 10 0.9851 nan 0.1000 0.0048
## 20 0.8852 nan 0.1000 0.0009
## 40 0.7878 nan 0.1000 -0.0001
## 60 0.7164 nan 0.1000 -0.0009
## 80 0.6771 nan 0.1000 -0.0012
## 100 0.6410 nan 0.1000 -0.0017
## 120 0.6073 nan 0.1000 -0.0017
## 140 0.5742 nan 0.1000 -0.0019
## 150 0.5582 nan 0.1000 -0.0021
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2598 nan 0.1000 0.0172
## 2 1.2280 nan 0.1000 0.0148
## 3 1.1980 nan 0.1000 0.0135
## 4 1.1765 nan 0.1000 0.0089
## 5 1.1613 nan 0.1000 0.0069
## 6 1.1417 nan 0.1000 0.0079
## 7 1.1234 nan 0.1000 0.0079
## 8 1.1072 nan 0.1000 0.0064
## 9 1.0932 nan 0.1000 0.0060
## 10 1.0764 nan 0.1000 0.0052
## 20 0.9747 nan 0.1000 0.0039
## 40 0.8952 nan 0.1000 -0.0007
## 60 0.8539 nan 0.1000 0.0003
## 80 0.8259 nan 0.1000 -0.0009
## 100 0.8073 nan 0.1000 -0.0005
## 120 0.7947 nan 0.1000 -0.0013
## 140 0.7807 nan 0.1000 -0.0001
## 150 0.7753 nan 0.1000 -0.0002
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2542 nan 0.1000 0.0203
## 2 1.2114 nan 0.1000 0.0192
## 3 1.1783 nan 0.1000 0.0136
## 4 1.1411 nan 0.1000 0.0155
## 5 1.1154 nan 0.1000 0.0105
## 6 1.0929 nan 0.1000 0.0081
## 7 1.0679 nan 0.1000 0.0107
## 8 1.0468 nan 0.1000 0.0080
## 9 1.0298 nan 0.1000 0.0059
## 10 1.0125 nan 0.1000 0.0074
## 20 0.9066 nan 0.1000 0.0010
## 40 0.8228 nan 0.1000 -0.0010
## 60 0.7801 nan 0.1000 -0.0002
## 80 0.7429 nan 0.1000 -0.0018
## 100 0.7174 nan 0.1000 -0.0023
## 120 0.6958 nan 0.1000 -0.0029
## 140 0.6712 nan 0.1000 -0.0013
## 150 0.6575 nan 0.1000 -0.0007
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2331 nan 0.1000 0.0297
## 2 1.1814 nan 0.1000 0.0219
## 3 1.1412 nan 0.1000 0.0121
## 4 1.1106 nan 0.1000 0.0116
## 5 1.0805 nan 0.1000 0.0112
## 6 1.0573 nan 0.1000 0.0078
## 7 1.0319 nan 0.1000 0.0090
## 8 1.0079 nan 0.1000 0.0113
## 9 0.9847 nan 0.1000 0.0095
## 10 0.9723 nan 0.1000 0.0052
## 20 0.8604 nan 0.1000 0.0012
## 40 0.7698 nan 0.1000 -0.0017
## 60 0.7181 nan 0.1000 -0.0015
## 80 0.6704 nan 0.1000 -0.0018
## 100 0.6312 nan 0.1000 -0.0009
## 120 0.5998 nan 0.1000 -0.0021
## 140 0.5744 nan 0.1000 -0.0010
## 150 0.5604 nan 0.1000 -0.0026
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2546 nan 0.1000 0.0196
## 2 1.2195 nan 0.1000 0.0173
## 3 1.1934 nan 0.1000 0.0107
## 4 1.1662 nan 0.1000 0.0089
## 5 1.1469 nan 0.1000 0.0093
## 6 1.1319 nan 0.1000 0.0061
## 7 1.1158 nan 0.1000 0.0059
## 8 1.1008 nan 0.1000 0.0059
## 9 1.0881 nan 0.1000 0.0062
## 10 1.0763 nan 0.1000 0.0048
## 20 0.9820 nan 0.1000 0.0012
## 40 0.8917 nan 0.1000 0.0007
## 60 0.8588 nan 0.1000 -0.0013
## 80 0.8323 nan 0.1000 -0.0016
## 100 0.8125 nan 0.1000 -0.0008
## 120 0.8004 nan 0.1000 -0.0025
## 140 0.7912 nan 0.1000 -0.0012
## 150 0.7846 nan 0.1000 -0.0007
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2404 nan 0.1000 0.0213
## 2 1.2008 nan 0.1000 0.0206
## 3 1.1672 nan 0.1000 0.0143
## 4 1.1300 nan 0.1000 0.0145
## 5 1.1037 nan 0.1000 0.0111
## 6 1.0828 nan 0.1000 0.0104
## 7 1.0625 nan 0.1000 0.0075
## 8 1.0433 nan 0.1000 0.0071
## 9 1.0227 nan 0.1000 0.0075
## 10 1.0106 nan 0.1000 0.0027
## 20 0.9130 nan 0.1000 0.0044
## 40 0.8299 nan 0.1000 -0.0018
## 60 0.7841 nan 0.1000 -0.0007
## 80 0.7486 nan 0.1000 -0.0013
## 100 0.7206 nan 0.1000 -0.0025
## 120 0.6975 nan 0.1000 -0.0040
## 140 0.6756 nan 0.1000 -0.0015
## 150 0.6642 nan 0.1000 -0.0011
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2352 nan 0.1000 0.0268
## 2 1.1842 nan 0.1000 0.0193
## 3 1.1437 nan 0.1000 0.0163
## 4 1.1123 nan 0.1000 0.0096
## 5 1.0837 nan 0.1000 0.0133
## 6 1.0568 nan 0.1000 0.0080
## 7 1.0377 nan 0.1000 0.0076
## 8 1.0174 nan 0.1000 0.0064
## 9 0.9957 nan 0.1000 0.0033
## 10 0.9810 nan 0.1000 0.0051
## 20 0.8745 nan 0.1000 0.0032
## 40 0.7867 nan 0.1000 -0.0042
## 60 0.7365 nan 0.1000 -0.0013
## 80 0.6853 nan 0.1000 -0.0012
## 100 0.6480 nan 0.1000 -0.0016
## 120 0.6199 nan 0.1000 -0.0018
## 140 0.5842 nan 0.1000 -0.0018
## 150 0.5691 nan 0.1000 -0.0019
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2604 nan 0.1000 0.0159
## 2 1.2280 nan 0.1000 0.0162
## 3 1.1965 nan 0.1000 0.0104
## 4 1.1801 nan 0.1000 0.0053
## 5 1.1564 nan 0.1000 0.0103
## 6 1.1374 nan 0.1000 0.0079
## 7 1.1224 nan 0.1000 0.0070
## 8 1.1039 nan 0.1000 0.0066
## 9 1.0903 nan 0.1000 0.0052
## 10 1.0803 nan 0.1000 0.0032
## 20 0.9859 nan 0.1000 0.0009
## 40 0.9030 nan 0.1000 -0.0002
## 60 0.8700 nan 0.1000 -0.0021
## 80 0.8438 nan 0.1000 -0.0006
## 100 0.8259 nan 0.1000 -0.0007
## 120 0.8110 nan 0.1000 -0.0006
## 140 0.7992 nan 0.1000 -0.0012
## 150 0.7944 nan 0.1000 -0.0001
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2456 nan 0.1000 0.0214
## 2 1.2054 nan 0.1000 0.0171
## 3 1.1700 nan 0.1000 0.0146
## 4 1.1385 nan 0.1000 0.0123
## 5 1.1140 nan 0.1000 0.0085
## 6 1.0959 nan 0.1000 0.0069
## 7 1.0763 nan 0.1000 0.0082
## 8 1.0560 nan 0.1000 0.0084
## 9 1.0376 nan 0.1000 0.0064
## 10 1.0233 nan 0.1000 0.0039
## 20 0.9259 nan 0.1000 0.0025
## 40 0.8378 nan 0.1000 -0.0017
## 60 0.7989 nan 0.1000 -0.0018
## 80 0.7658 nan 0.1000 -0.0014
## 100 0.7419 nan 0.1000 -0.0018
## 120 0.7156 nan 0.1000 -0.0010
## 140 0.6929 nan 0.1000 -0.0040
## 150 0.6848 nan 0.1000 -0.0014
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2406 nan 0.1000 0.0242
## 2 1.1943 nan 0.1000 0.0184
## 3 1.1554 nan 0.1000 0.0168
## 4 1.1234 nan 0.1000 0.0129
## 5 1.0962 nan 0.1000 0.0134
## 6 1.0695 nan 0.1000 0.0089
## 7 1.0411 nan 0.1000 0.0079
## 8 1.0247 nan 0.1000 0.0046
## 9 1.0060 nan 0.1000 0.0074
## 10 0.9913 nan 0.1000 0.0024
## 20 0.8833 nan 0.1000 -0.0008
## 40 0.7917 nan 0.1000 -0.0026
## 60 0.7324 nan 0.1000 -0.0021
## 80 0.6870 nan 0.1000 -0.0018
## 100 0.6470 nan 0.1000 -0.0037
## 120 0.6094 nan 0.1000 -0.0019
## 140 0.5750 nan 0.1000 -0.0013
## 150 0.5593 nan 0.1000 -0.0009
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2560 nan 0.1000 0.0176
## 2 1.2277 nan 0.1000 0.0124
## 3 1.2055 nan 0.1000 0.0108
## 4 1.1887 nan 0.1000 0.0066
## 5 1.1686 nan 0.1000 0.0090
## 6 1.1497 nan 0.1000 0.0082
## 7 1.1355 nan 0.1000 0.0068
## 8 1.1212 nan 0.1000 0.0062
## 9 1.1065 nan 0.1000 0.0060
## 10 1.0954 nan 0.1000 0.0053
## 20 1.0096 nan 0.1000 0.0019
## 40 0.9211 nan 0.1000 -0.0002
## 60 0.8791 nan 0.1000 -0.0010
## 80 0.8556 nan 0.1000 -0.0003
## 100 0.8357 nan 0.1000 -0.0006
## 120 0.8224 nan 0.1000 0.0000
## 140 0.8108 nan 0.1000 -0.0016
## 150 0.8071 nan 0.1000 -0.0016
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2467 nan 0.1000 0.0185
## 2 1.2093 nan 0.1000 0.0179
## 3 1.1788 nan 0.1000 0.0110
## 4 1.1456 nan 0.1000 0.0121
## 5 1.1174 nan 0.1000 0.0125
## 6 1.0949 nan 0.1000 0.0082
## 7 1.0753 nan 0.1000 0.0073
## 8 1.0572 nan 0.1000 0.0062
## 9 1.0435 nan 0.1000 0.0056
## 10 1.0266 nan 0.1000 0.0048
## 20 0.9311 nan 0.1000 -0.0016
## 40 0.8483 nan 0.1000 -0.0011
## 60 0.8018 nan 0.1000 -0.0027
## 80 0.7721 nan 0.1000 -0.0043
## 100 0.7434 nan 0.1000 -0.0019
## 120 0.7219 nan 0.1000 -0.0020
## 140 0.6976 nan 0.1000 -0.0013
## 150 0.6878 nan 0.1000 -0.0014
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2412 nan 0.1000 0.0211
## 2 1.2001 nan 0.1000 0.0204
## 3 1.1612 nan 0.1000 0.0176
## 4 1.1278 nan 0.1000 0.0124
## 5 1.0928 nan 0.1000 0.0150
## 6 1.0686 nan 0.1000 0.0062
## 7 1.0477 nan 0.1000 0.0085
## 8 1.0251 nan 0.1000 0.0095
## 9 1.0116 nan 0.1000 0.0007
## 10 0.9931 nan 0.1000 0.0055
## 20 0.8930 nan 0.1000 0.0022
## 40 0.8021 nan 0.1000 0.0001
## 60 0.7469 nan 0.1000 -0.0009
## 80 0.7001 nan 0.1000 -0.0011
## 100 0.6628 nan 0.1000 -0.0016
## 120 0.6245 nan 0.1000 -0.0018
## 140 0.5924 nan 0.1000 -0.0009
## 150 0.5791 nan 0.1000 -0.0026
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2564 nan 0.1000 0.0173
## 2 1.2216 nan 0.1000 0.0138
## 3 1.1933 nan 0.1000 0.0117
## 4 1.1692 nan 0.1000 0.0063
## 5 1.1525 nan 0.1000 0.0083
## 6 1.1352 nan 0.1000 0.0070
## 7 1.1201 nan 0.1000 0.0063
## 8 1.1053 nan 0.1000 0.0065
## 9 1.0889 nan 0.1000 0.0020
## 10 1.0785 nan 0.1000 0.0032
## 20 0.9909 nan 0.1000 0.0001
## 40 0.9145 nan 0.1000 -0.0007
## 60 0.8718 nan 0.1000 -0.0010
## 80 0.8477 nan 0.1000 0.0006
## 100 0.8304 nan 0.1000 -0.0009
## 120 0.8206 nan 0.1000 -0.0009
## 140 0.8094 nan 0.1000 -0.0020
## 150 0.8064 nan 0.1000 -0.0009
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2476 nan 0.1000 0.0220
## 2 1.2090 nan 0.1000 0.0165
## 3 1.1742 nan 0.1000 0.0175
## 4 1.1414 nan 0.1000 0.0133
## 5 1.1159 nan 0.1000 0.0088
## 6 1.0902 nan 0.1000 0.0086
## 7 1.0723 nan 0.1000 0.0056
## 8 1.0588 nan 0.1000 0.0039
## 9 1.0404 nan 0.1000 0.0066
## 10 1.0218 nan 0.1000 0.0052
## 20 0.9285 nan 0.1000 0.0007
## 40 0.8475 nan 0.1000 -0.0020
## 60 0.7991 nan 0.1000 -0.0014
## 80 0.7733 nan 0.1000 -0.0016
## 100 0.7434 nan 0.1000 -0.0012
## 120 0.7182 nan 0.1000 -0.0013
## 140 0.6966 nan 0.1000 -0.0016
## 150 0.6825 nan 0.1000 -0.0014
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2330 nan 0.1000 0.0200
## 2 1.1826 nan 0.1000 0.0225
## 3 1.1472 nan 0.1000 0.0132
## 4 1.1118 nan 0.1000 0.0135
## 5 1.0834 nan 0.1000 0.0097
## 6 1.0544 nan 0.1000 0.0109
## 7 1.0349 nan 0.1000 0.0052
## 8 1.0116 nan 0.1000 0.0070
## 9 0.9967 nan 0.1000 0.0046
## 10 0.9817 nan 0.1000 0.0055
## 20 0.8808 nan 0.1000 0.0007
## 40 0.7879 nan 0.1000 -0.0014
## 60 0.7295 nan 0.1000 -0.0043
## 80 0.6930 nan 0.1000 -0.0017
## 100 0.6522 nan 0.1000 -0.0007
## 120 0.6192 nan 0.1000 -0.0025
## 140 0.5852 nan 0.1000 0.0003
## 150 0.5697 nan 0.1000 -0.0020
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2558 nan 0.1000 0.0197
## 2 1.2213 nan 0.1000 0.0171
## 3 1.1902 nan 0.1000 0.0128
## 4 1.1659 nan 0.1000 0.0117
## 5 1.1489 nan 0.1000 0.0077
## 6 1.1301 nan 0.1000 0.0093
## 7 1.1159 nan 0.1000 0.0053
## 8 1.1024 nan 0.1000 0.0054
## 9 1.0876 nan 0.1000 0.0061
## 10 1.0771 nan 0.1000 0.0008
## 20 0.9718 nan 0.1000 0.0026
## 40 0.8803 nan 0.1000 0.0005
## 60 0.8410 nan 0.1000 -0.0007
## 80 0.8161 nan 0.1000 -0.0005
## 100 0.7962 nan 0.1000 -0.0001
## 120 0.7810 nan 0.1000 -0.0025
## 140 0.7699 nan 0.1000 -0.0015
## 150 0.7657 nan 0.1000 -0.0014
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2478 nan 0.1000 0.0220
## 2 1.2080 nan 0.1000 0.0181
## 3 1.1723 nan 0.1000 0.0164
## 4 1.1420 nan 0.1000 0.0149
## 5 1.1128 nan 0.1000 0.0129
## 6 1.0896 nan 0.1000 0.0105
## 7 1.0691 nan 0.1000 0.0088
## 8 1.0469 nan 0.1000 0.0079
## 9 1.0291 nan 0.1000 0.0081
## 10 1.0124 nan 0.1000 0.0068
## 20 0.9071 nan 0.1000 0.0021
## 40 0.8130 nan 0.1000 -0.0018
## 60 0.7734 nan 0.1000 -0.0013
## 80 0.7360 nan 0.1000 -0.0012
## 100 0.7127 nan 0.1000 -0.0029
## 120 0.6851 nan 0.1000 -0.0018
## 140 0.6577 nan 0.1000 -0.0002
## 150 0.6495 nan 0.1000 -0.0016
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2365 nan 0.1000 0.0259
## 2 1.1901 nan 0.1000 0.0203
## 3 1.1515 nan 0.1000 0.0185
## 4 1.1177 nan 0.1000 0.0143
## 5 1.0832 nan 0.1000 0.0123
## 6 1.0511 nan 0.1000 0.0138
## 7 1.0304 nan 0.1000 0.0054
## 8 1.0092 nan 0.1000 0.0088
## 9 0.9905 nan 0.1000 0.0045
## 10 0.9769 nan 0.1000 0.0043
## 20 0.8588 nan 0.1000 0.0003
## 40 0.7629 nan 0.1000 -0.0017
## 60 0.7090 nan 0.1000 -0.0008
## 80 0.6679 nan 0.1000 -0.0020
## 100 0.6330 nan 0.1000 -0.0015
## 120 0.5983 nan 0.1000 -0.0023
## 140 0.5685 nan 0.1000 -0.0016
## 150 0.5538 nan 0.1000 -0.0020
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2584 nan 0.1000 0.0182
## 2 1.2244 nan 0.1000 0.0167
## 3 1.1971 nan 0.1000 0.0117
## 4 1.1760 nan 0.1000 0.0071
## 5 1.1485 nan 0.1000 0.0073
## 6 1.1292 nan 0.1000 0.0085
## 7 1.1114 nan 0.1000 0.0075
## 8 1.0968 nan 0.1000 0.0065
## 9 1.0807 nan 0.1000 0.0064
## 10 1.0669 nan 0.1000 0.0032
## 20 0.9654 nan 0.1000 0.0025
## 40 0.8722 nan 0.1000 0.0000
## 60 0.8196 nan 0.1000 0.0009
## 80 0.7954 nan 0.1000 -0.0000
## 100 0.7793 nan 0.1000 -0.0012
## 120 0.7667 nan 0.1000 -0.0013
## 140 0.7576 nan 0.1000 -0.0011
## 150 0.7540 nan 0.1000 -0.0017
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2465 nan 0.1000 0.0196
## 2 1.2041 nan 0.1000 0.0174
## 3 1.1694 nan 0.1000 0.0121
## 4 1.1345 nan 0.1000 0.0136
## 5 1.1014 nan 0.1000 0.0126
## 6 1.0756 nan 0.1000 0.0119
## 7 1.0554 nan 0.1000 0.0048
## 8 1.0335 nan 0.1000 0.0044
## 9 1.0139 nan 0.1000 0.0087
## 10 0.9972 nan 0.1000 0.0035
## 20 0.8811 nan 0.1000 0.0019
## 40 0.7904 nan 0.1000 -0.0007
## 60 0.7506 nan 0.1000 -0.0010
## 80 0.7170 nan 0.1000 -0.0033
## 100 0.6899 nan 0.1000 -0.0010
## 120 0.6650 nan 0.1000 -0.0015
## 140 0.6467 nan 0.1000 -0.0026
## 150 0.6347 nan 0.1000 -0.0005
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2355 nan 0.1000 0.0229
## 2 1.1810 nan 0.1000 0.0240
## 3 1.1449 nan 0.1000 0.0140
## 4 1.1088 nan 0.1000 0.0148
## 5 1.0739 nan 0.1000 0.0166
## 6 1.0462 nan 0.1000 0.0121
## 7 1.0252 nan 0.1000 0.0068
## 8 1.0029 nan 0.1000 0.0069
## 9 0.9835 nan 0.1000 0.0082
## 10 0.9695 nan 0.1000 0.0041
## 20 0.8492 nan 0.1000 -0.0009
## 40 0.7548 nan 0.1000 -0.0009
## 60 0.7017 nan 0.1000 -0.0029
## 80 0.6579 nan 0.1000 -0.0027
## 100 0.6180 nan 0.1000 -0.0007
## 120 0.5804 nan 0.1000 -0.0022
## 140 0.5479 nan 0.1000 -0.0019
## 150 0.5319 nan 0.1000 -0.0013
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2629 nan 0.1000 0.0157
## 2 1.2343 nan 0.1000 0.0142
## 3 1.2093 nan 0.1000 0.0083
## 4 1.1850 nan 0.1000 0.0107
## 5 1.1662 nan 0.1000 0.0074
## 6 1.1493 nan 0.1000 0.0070
## 7 1.1332 nan 0.1000 0.0064
## 8 1.1177 nan 0.1000 0.0065
## 9 1.1032 nan 0.1000 0.0056
## 10 1.0914 nan 0.1000 0.0052
## 20 0.9937 nan 0.1000 0.0012
## 40 0.9102 nan 0.1000 0.0015
## 60 0.8666 nan 0.1000 -0.0013
## 80 0.8395 nan 0.1000 -0.0006
## 100 0.8266 nan 0.1000 -0.0011
## 120 0.8142 nan 0.1000 -0.0024
## 140 0.8018 nan 0.1000 -0.0005
## 150 0.7963 nan 0.1000 -0.0009
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2483 nan 0.1000 0.0170
## 2 1.2073 nan 0.1000 0.0175
## 3 1.1793 nan 0.1000 0.0120
## 4 1.1488 nan 0.1000 0.0117
## 5 1.1297 nan 0.1000 0.0044
## 6 1.1034 nan 0.1000 0.0070
## 7 1.0814 nan 0.1000 0.0081
## 8 1.0628 nan 0.1000 0.0057
## 9 1.0427 nan 0.1000 0.0077
## 10 1.0284 nan 0.1000 0.0035
## 20 0.9235 nan 0.1000 0.0015
## 40 0.8450 nan 0.1000 -0.0012
## 60 0.8004 nan 0.1000 -0.0017
## 80 0.7696 nan 0.1000 -0.0027
## 100 0.7372 nan 0.1000 -0.0031
## 120 0.7127 nan 0.1000 -0.0014
## 140 0.6914 nan 0.1000 -0.0013
## 150 0.6819 nan 0.1000 -0.0018
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2370 nan 0.1000 0.0274
## 2 1.1959 nan 0.1000 0.0171
## 3 1.1653 nan 0.1000 0.0132
## 4 1.1256 nan 0.1000 0.0164
## 5 1.1001 nan 0.1000 0.0092
## 6 1.0726 nan 0.1000 0.0093
## 7 1.0476 nan 0.1000 0.0094
## 8 1.0265 nan 0.1000 0.0084
## 9 1.0102 nan 0.1000 0.0058
## 10 0.9965 nan 0.1000 0.0045
## 20 0.8848 nan 0.1000 0.0014
## 40 0.7985 nan 0.1000 -0.0037
## 60 0.7376 nan 0.1000 -0.0014
## 80 0.6964 nan 0.1000 -0.0015
## 100 0.6534 nan 0.1000 -0.0022
## 120 0.6235 nan 0.1000 -0.0008
## 140 0.5941 nan 0.1000 -0.0017
## 150 0.5790 nan 0.1000 -0.0013
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2573 nan 0.1000 0.0193
## 2 1.2293 nan 0.1000 0.0145
## 3 1.1948 nan 0.1000 0.0147
## 4 1.1740 nan 0.1000 0.0084
## 5 1.1536 nan 0.1000 0.0096
## 6 1.1381 nan 0.1000 0.0068
## 7 1.1197 nan 0.1000 0.0075
## 8 1.1048 nan 0.1000 0.0044
## 9 1.0863 nan 0.1000 0.0057
## 10 1.0725 nan 0.1000 0.0062
## 20 0.9800 nan 0.1000 0.0020
## 40 0.8891 nan 0.1000 0.0004
## 60 0.8421 nan 0.1000 -0.0011
## 80 0.8193 nan 0.1000 -0.0002
## 100 0.7989 nan 0.1000 -0.0017
## 120 0.7858 nan 0.1000 -0.0017
## 140 0.7724 nan 0.1000 -0.0003
## 150 0.7668 nan 0.1000 -0.0009
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2445 nan 0.1000 0.0240
## 2 1.2048 nan 0.1000 0.0189
## 3 1.1617 nan 0.1000 0.0136
## 4 1.1311 nan 0.1000 0.0126
## 5 1.1052 nan 0.1000 0.0105
## 6 1.0841 nan 0.1000 0.0077
## 7 1.0650 nan 0.1000 0.0064
## 8 1.0439 nan 0.1000 0.0065
## 9 1.0204 nan 0.1000 0.0092
## 10 1.0029 nan 0.1000 0.0045
## 20 0.9038 nan 0.1000 -0.0008
## 40 0.8234 nan 0.1000 -0.0008
## 60 0.7748 nan 0.1000 -0.0001
## 80 0.7429 nan 0.1000 -0.0002
## 100 0.7169 nan 0.1000 -0.0014
## 120 0.6904 nan 0.1000 -0.0006
## 140 0.6681 nan 0.1000 -0.0017
## 150 0.6566 nan 0.1000 -0.0018
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2404 nan 0.1000 0.0244
## 2 1.1984 nan 0.1000 0.0207
## 3 1.1611 nan 0.1000 0.0142
## 4 1.1303 nan 0.1000 0.0129
## 5 1.0965 nan 0.1000 0.0114
## 6 1.0736 nan 0.1000 0.0094
## 7 1.0467 nan 0.1000 0.0091
## 8 1.0248 nan 0.1000 0.0089
## 9 1.0062 nan 0.1000 0.0059
## 10 0.9847 nan 0.1000 0.0062
## 20 0.8644 nan 0.1000 0.0004
## 40 0.7720 nan 0.1000 -0.0016
## 60 0.7039 nan 0.1000 -0.0023
## 80 0.6636 nan 0.1000 -0.0007
## 100 0.6227 nan 0.1000 -0.0010
## 120 0.5906 nan 0.1000 -0.0023
## 140 0.5549 nan 0.1000 -0.0018
## 150 0.5454 nan 0.1000 -0.0013
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2554 nan 0.1000 0.0156
## 2 1.2285 nan 0.1000 0.0129
## 3 1.2011 nan 0.1000 0.0120
## 4 1.1784 nan 0.1000 0.0090
## 5 1.1592 nan 0.1000 0.0077
## 6 1.1403 nan 0.1000 0.0065
## 7 1.1247 nan 0.1000 0.0063
## 8 1.1122 nan 0.1000 0.0053
## 9 1.0971 nan 0.1000 0.0068
## 10 1.0837 nan 0.1000 0.0056
## 20 0.9876 nan 0.1000 0.0012
## 40 0.9033 nan 0.1000 -0.0001
## 60 0.8632 nan 0.1000 -0.0008
## 80 0.8342 nan 0.1000 -0.0007
## 100 0.8201 nan 0.1000 -0.0013
## 120 0.8090 nan 0.1000 -0.0007
## 140 0.7960 nan 0.1000 -0.0013
## 150 0.7924 nan 0.1000 -0.0011
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2464 nan 0.1000 0.0184
## 2 1.2060 nan 0.1000 0.0164
## 3 1.1714 nan 0.1000 0.0138
## 4 1.1449 nan 0.1000 0.0080
## 5 1.1193 nan 0.1000 0.0098
## 6 1.0976 nan 0.1000 0.0079
## 7 1.0736 nan 0.1000 0.0108
## 8 1.0537 nan 0.1000 0.0066
## 9 1.0414 nan 0.1000 0.0040
## 10 1.0285 nan 0.1000 0.0042
## 20 0.9259 nan 0.1000 0.0026
## 40 0.8395 nan 0.1000 -0.0024
## 60 0.7969 nan 0.1000 -0.0012
## 80 0.7658 nan 0.1000 -0.0008
## 100 0.7410 nan 0.1000 -0.0014
## 120 0.7205 nan 0.1000 -0.0014
## 140 0.6959 nan 0.1000 -0.0021
## 150 0.6852 nan 0.1000 -0.0017
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2371 nan 0.1000 0.0254
## 2 1.1844 nan 0.1000 0.0217
## 3 1.1477 nan 0.1000 0.0140
## 4 1.1148 nan 0.1000 0.0149
## 5 1.0860 nan 0.1000 0.0126
## 6 1.0582 nan 0.1000 0.0096
## 7 1.0364 nan 0.1000 0.0085
## 8 1.0181 nan 0.1000 0.0067
## 9 1.0025 nan 0.1000 0.0060
## 10 0.9874 nan 0.1000 0.0036
## 20 0.8829 nan 0.1000 -0.0005
## 40 0.7840 nan 0.1000 -0.0002
## 60 0.7298 nan 0.1000 -0.0018
## 80 0.6875 nan 0.1000 -0.0021
## 100 0.6470 nan 0.1000 -0.0022
## 120 0.6157 nan 0.1000 -0.0003
## 140 0.5869 nan 0.1000 -0.0018
## 150 0.5714 nan 0.1000 -0.0018
##
## Iter TrainDeviance ValidDeviance StepSize Improve
## 1 1.2559 nan 0.1000 0.0181
## 2 1.2254 nan 0.1000 0.0146
## 3 1.1986 nan 0.1000 0.0117
## 4 1.1798 nan 0.1000 0.0068
## 5 1.1578 nan 0.1000 0.0096
## 6 1.1385 nan 0.1000 0.0073
## 7 1.1222 nan 0.1000 0.0070
## 8 1.1048 nan 0.1000 0.0072
## 9 1.0919 nan 0.1000 0.0044
## 10 1.0785 nan 0.1000 0.0048
## 20 0.9888 nan 0.1000 0.0007
## 40 0.9033 nan 0.1000 -0.0000
## 50 0.8780 nan 0.1000 -0.0002results <- resamples(models)
summary(results)##
## Call:
## summary.resamples(object = results)
##
## Models: rf, gbm, earth, knn, svmRadial
## Number of resamples: 30
##
## Accuracy
## Min. 1st Qu. Median Mean 3rd Qu. Max. NA's
## rf 0.6885246 0.7550899 0.7741935 0.7793143 0.8064516 0.8870968 0
## gbm 0.7049180 0.7419355 0.7723427 0.7750573 0.8032787 0.8709677 0
## earth 0.7096774 0.7580645 0.7741935 0.7794641 0.8056584 0.8360656 0
## knn 0.6290323 0.7125859 0.7258065 0.7355720 0.7704918 0.8360656 0
## svmRadial 0.6721311 0.7287811 0.7580645 0.7745373 0.8056584 0.8852459 0
##
## Kappa
## Min. 1st Qu. Median Mean 3rd Qu. Max. NA's
## rf 0.2687697 0.4380512 0.5055416 0.5017694 0.5637433 0.7559055 0
## gbm 0.2988506 0.3991629 0.4628713 0.4748325 0.5487934 0.6997579 0
## earth 0.3388626 0.4374468 0.4895308 0.4944476 0.5530608 0.6285018 0
## knn 0.1461078 0.3323591 0.3722698 0.3863427 0.4695976 0.6369048 0
## svmRadial 0.2015707 0.3831701 0.4604980 0.4840323 0.5515619 0.7305994 0- comparison by visualization Boxplots summarize performance across resampling repeats. This helps identify consistently strong models.
# Box plots to compare models
scales <- list(x=list(relation="free"), y=list(relation="free"))
bwplot(results, scales=scales)
- ensemble predictions on testdata Stacking trains a meta-model (here a GLM) on the predictions from base models. This is a practical and widely used ensembling strategy.
# Create the trainControl
set.seed(101)
stackControl <- trainControl(method="repeatedcv",
number=10,
repeats=3,
savePredictions=TRUE,
classProbs=TRUE)
# Ensemble the predictions of `models` to form a new combined prediction based on glm
stack.glm <- caretStack(models, method="glm", metric="Accuracy", trControl=stackControl)
(stack.glm)## The following models were ensembled: rf, gbm, earth, knn, svmRadial
##
## caret::train model:
## Generalized Linear Model
##
## 615 samples
## 5 predictor
## 2 classes: 'neg', 'pos'
##
## No pre-processing
## Resampling: Cross-Validated (10 fold, repeated 3 times)
## Summary of sample sizes: 553, 554, 553, 553, 554, 554, ...
## Resampling results:
##
## Accuracy Kappa
## 0.7751807 0.4829106
##
##
## Final model:
##
## Call: NULL
##
## Coefficients:
## (Intercept) rf gbm earth knn svmRadial
## -2.6702 1.2081 1.7151 3.5741 0.1624 -1.3166
##
## Degrees of Freedom: 614 Total (i.e. Null); 609 Residual
## Null Deviance: 796
## Residual Deviance: 571.8 AIC: 583.8- compute confusion matrix Finally, evaluate the stacked model on the test data. This provides a direct comparison against individual models.
# Predict on testData
stack_predicteds <- predict(stack.glm, newdata=testData4)
stack_predicteds$label <- ifelse(stack_predicteds$neg > stack_predicteds$pos, "neg", "pos")
confusionMatrix(reference = as.factor(testData$diabetes), data = as.factor(stack_predicteds$label ) )## Confusion Matrix and Statistics
##
## Reference
## Prediction neg pos
## neg 83 24
## pos 17 29
##
## Accuracy : 0.732
## 95% CI : (0.6545, 0.8003)
## No Information Rate : 0.6536
## P-Value [Acc > NIR] : 0.02367
##
## Kappa : 0.3893
##
## Mcnemar's Test P-Value : 0.34874
##
## Sensitivity : 0.8300
## Specificity : 0.5472
## Pos Pred Value : 0.7757
## Neg Pred Value : 0.6304
## Prevalence : 0.6536
## Detection Rate : 0.5425
## Detection Prevalence : 0.6993
## Balanced Accuracy : 0.6886
##
## 'Positive' Class : neg
## KNN Classifier
This section demonstrates KNN in a more direct way using the
class::knn() function. This is useful for learning and for
understanding how KNN works under the hood, separate from the caret
wrapper. A key reminder: KNN requires feature scaling because it relies
on distance calculations.
# Loading package
# library(e1071)
library(caTools)
library(class)Splitting data
We reload the dataset and split it again. In real projects, you would typically reuse the same split. Here the goal is to demonstrate a standalone KNN pipeline.
# load the Pima Indians dataset from the mlbench dataset
library(mlbench)
data(PimaIndiansDiabetes)
# rename dataset to have shorter name because lazy
diabetes <- PimaIndiansDiabetesWe create a train/test split using
caTools::sample.split(). This function is commonly used for
quick splits.
# Splitting data into train and test data
set.seed(100)
split <- sample.split(diabetes, SplitRatio = 0.8)
train_cl <- subset(diabetes, split == "TRUE")
test_cl <- subset(diabetes, split == "FALSE")Feature scaling is required for KNN. Here we scale predictors to have mean 0 and standard deviation 1. This is different from range scaling earlier; both are common. The key is to apply the same scaling logic consistently.
# Feature Scaling
train_scale <- scale(train_cl[, 1:8])
test_scale <- scale(test_cl[, 1:8])
# train_y <- scale(train_cl[, 5])
# test_y <- scale(test_cl[, 5])Creating KNN model
KNN predicts the class of each test observation based on the majority class among its K nearest neighbors in the training set. Smaller K tends to fit more locally (higher variance), while larger K smooths more (higher bias).
# Fitting KNN Model to training dataset
classifier_knn <- knn(train = train_scale,
cl = train_cl$diabetes,
test = test_scale,
k = 1)
classifier_knn## [1] pos neg neg neg pos neg neg neg neg neg neg neg pos neg neg pos neg pos
## [19] neg neg neg neg pos neg neg pos neg pos pos neg neg neg neg pos neg pos
## [37] neg neg neg pos neg pos pos neg neg neg pos pos neg pos neg neg neg neg
## [55] pos neg pos neg pos neg neg neg pos pos pos pos neg pos neg pos pos neg
## [73] pos neg neg pos neg neg neg pos pos neg neg pos neg pos pos neg neg neg
## [91] neg neg neg pos pos neg neg neg pos neg neg pos neg neg pos neg pos neg
## [109] neg neg neg neg pos pos pos pos pos pos neg pos pos pos neg neg neg neg
## [127] neg neg neg neg neg neg pos neg pos neg pos pos neg pos neg pos neg pos
## [145] neg neg neg pos neg neg neg pos pos pos neg pos neg pos neg neg neg neg
## [163] neg neg pos pos neg pos neg neg neg
## Levels: neg posModel Evaluation
- Creat confusion matrix A confusion matrix is the most basic evaluation tool for classifiers. It shows counts of correct/incorrect predictions by class.
# Confusion Matrix
cm <- table(test_cl$diabetes, classifier_knn)
cm## classifier_knn
## neg pos
## neg 79 32
## pos 27 33Calculate accuracy with different K
Accuracy is easy to compute, but do not rely on accuracy alone if classes are imbalanced. In applied medical problems, sensitivity/specificity (or ROC/AUC) often matter more.
# Model Evaluation - Choosing K =1
# Calculate out of Sample error
misClassError <- mean(classifier_knn != test_cl$diabetes)
print(paste('Accuracy =', 1-misClassError))## [1] "Accuracy = 0.654970760233918"This example uses a larger K. In practice, you tune K by evaluating a range of values and selecting the best under cross-validation or a validation set.
# K = 7
classifier_knn <- knn(train = train_scale,
test = test_scale,
cl = train_cl$diabetes,
k = 23)
misClassError <- mean(classifier_knn != test_cl$diabetes)
print(paste('Accuracy =', 1-misClassError))## [1] "Accuracy = 0.795321637426901"Optimization
- search better k parameter Here we try a sequence of K values and compute accuracy for each. This gives a quick sensitivity analysis of performance versus K. Note: In production, prefer cross-validation rather than a single split to reduce randomness.
i=1
k.optm=1
for (i in 1:39){
y_pred = knn(train = train_scale,
test = test_scale,
cl = train_cl$diabetes,
k = i )
k.optm[i] <- 1- mean(y_pred != test_cl$diabetes)
k=i
cat(k,'=',k.optm[i],'')
}## 1 = 0.6549708 2 = 0.6666667 3 = 0.7426901 4 = 0.6900585 5 = 0.7309942 6 = 0.748538 7 = 0.7368421 8 = 0.7309942 9 = 0.7368421 10 = 0.7251462 11 = 0.7602339 12 = 0.748538 13 = 0.7719298 14 = 0.748538 15 = 0.754386 16 = 0.754386 17 = 0.7602339 18 = 0.7192982 19 = 0.7719298 20 = 0.754386 21 = 0.7836257 22 = 0.7777778 23 = 0.7953216 24 = 0.7719298 25 = 0.7777778 26 = 0.7836257 27 = 0.7719298 28 = 0.7660819 29 = 0.7777778 30 = 0.7660819 31 = 0.7660819 32 = 0.7602339 33 = 0.7719298 34 = 0.754386 35 = 0.7602339 36 = 0.7719298 37 = 0.7777778 38 = 0.7836257 39 = 0.7836257- Accuracy plot
k=15Plotting performance across K helps identify a stable region rather than a single “lucky” optimum.
plot(k.optm, type="b", xlab="K- Value",ylab="RMSE level")
Visualization
This section is a placeholder for visualization of results. In practice, you might visualize: - decision boundaries (for 2D cases), - ROC curves, - or distributions of predicted probabilities.
# Visualising the Training set results
# Install ElemStatLearn if not presentKNN regression
KNN can also be used for regression problems, where the prediction is
a continuous value. Instead of majority vote, regression KNN typically
averages the outcomes of nearest neighbors. This section uses the Boston
housing dataset (MASS::Boston) and focuses on predicting
medv (median value of owner-occupied homes).
Data exploring
We first inspect missingness and correlation structure. For KNN regression, feature scaling is especially important because distance computations drive the model.
library("Amelia")The missingness map provides a quick visual summary of missing patterns.
data("Boston", package = "MASS")
missmap(Boston,col=c('yellow','black'),y.at=1,y.labels='',legend=TRUE)
Correlation plots help identify redundant predictors and strong relationships with the outcome.
library(corrplot)
corrplot(cor((Boston)))
Descriptive summaries help confirm variable ranges and distributions before selecting predictors.
library(Hmisc)
describe(Boston)## Boston
##
## 14 Variables 506 Observations
## --------------------------------------------------------------------------------
## crim
## n missing distinct Info Mean Gmd .05 .10
## 506 0 504 1 3.614 5.794 0.02791 0.03819
## .25 .50 .75 .90 .95
## 0.08204 0.25651 3.67708 10.75300 15.78915
##
## lowest : 0.00632 0.00906 0.01096 0.01301 0.01311
## highest: 45.7461 51.1358 67.9208 73.5341 88.9762
## --------------------------------------------------------------------------------
## zn
## n missing distinct Info Mean Gmd .05 .10
## 506 0 26 0.603 11.36 18.77 0.0 0.0
## .25 .50 .75 .90 .95
## 0.0 0.0 12.5 42.5 80.0
##
## lowest : 0 12.5 17.5 18 20 , highest: 82.5 85 90 95 100
## --------------------------------------------------------------------------------
## indus
## n missing distinct Info Mean Gmd .05 .10
## 506 0 76 0.982 11.14 7.705 2.18 2.91
## .25 .50 .75 .90 .95
## 5.19 9.69 18.10 19.58 21.89
##
## lowest : 0.46 0.74 1.21 1.22 1.25 , highest: 18.1 19.58 21.89 25.65 27.74
## --------------------------------------------------------------------------------
## chas
## n missing distinct Info Sum Mean Gmd
## 506 0 2 0.193 35 0.06917 0.129
##
## --------------------------------------------------------------------------------
## nox
## n missing distinct Info Mean Gmd .05 .10
## 506 0 81 1 0.5547 0.1295 0.4092 0.4270
## .25 .50 .75 .90 .95
## 0.4490 0.5380 0.6240 0.7130 0.7400
##
## lowest : 0.385 0.389 0.392 0.394 0.398, highest: 0.713 0.718 0.74 0.77 0.871
## --------------------------------------------------------------------------------
## rm
## n missing distinct Info Mean Gmd .05 .10
## 506 0 446 1 6.285 0.7515 5.314 5.594
## .25 .50 .75 .90 .95
## 5.886 6.208 6.623 7.152 7.588
##
## lowest : 3.561 3.863 4.138 4.368 4.519, highest: 8.375 8.398 8.704 8.725 8.78
## --------------------------------------------------------------------------------
## age
## n missing distinct Info Mean Gmd .05 .10
## 506 0 356 0.999 68.57 31.52 17.72 26.95
## .25 .50 .75 .90 .95
## 45.02 77.50 94.07 98.80 100.00
##
## lowest : 2.9 6 6.2 6.5 6.6 , highest: 98.8 98.9 99.1 99.3 100
## --------------------------------------------------------------------------------
## dis
## n missing distinct Info Mean Gmd .05 .10
## 506 0 412 1 3.795 2.298 1.462 1.628
## .25 .50 .75 .90 .95
## 2.100 3.207 5.188 6.817 7.828
##
## lowest : 1.1296 1.137 1.1691 1.1742 1.1781
## highest: 9.2203 9.2229 10.5857 10.7103 12.1265
## --------------------------------------------------------------------------------
## rad
## n missing distinct Info Mean Gmd
## 506 0 9 0.959 9.549 8.518
##
## Value 1 2 3 4 5 6 7 8 24
## Frequency 20 24 38 110 115 26 17 24 132
## Proportion 0.040 0.047 0.075 0.217 0.227 0.051 0.034 0.047 0.261
##
## For the frequency table, variable is rounded to the nearest 0
## --------------------------------------------------------------------------------
## tax
## n missing distinct Info Mean Gmd .05 .10
## 506 0 66 0.981 408.2 181.7 222 233
## .25 .50 .75 .90 .95
## 279 330 666 666 666
##
## lowest : 187 188 193 198 216, highest: 432 437 469 666 711
## --------------------------------------------------------------------------------
## ptratio
## n missing distinct Info Mean Gmd .05 .10
## 506 0 46 0.978 18.46 2.383 14.70 14.75
## .25 .50 .75 .90 .95
## 17.40 19.05 20.20 20.90 21.00
##
## lowest : 12.6 13 13.6 14.4 14.7, highest: 20.9 21 21.1 21.2 22
## --------------------------------------------------------------------------------
## black
## n missing distinct Info Mean Gmd .05 .10
## 506 0 357 0.986 356.7 65.5 84.59 290.27
## .25 .50 .75 .90 .95
## 375.38 391.44 396.23 396.90 396.90
##
## lowest : 0.32 2.52 2.6 3.5 3.65 , highest: 396.28 396.3 396.33 396.42 396.9
## --------------------------------------------------------------------------------
## lstat
## n missing distinct Info Mean Gmd .05 .10
## 506 0 455 1 12.65 7.881 3.708 4.680
## .25 .50 .75 .90 .95
## 6.950 11.360 16.955 23.035 26.808
##
## lowest : 1.73 1.92 1.98 2.47 2.87 , highest: 34.37 34.41 34.77 36.98 37.97
## --------------------------------------------------------------------------------
## medv
## n missing distinct Info Mean Gmd .05 .10
## 506 0 229 1 22.53 9.778 10.20 12.75
## .25 .50 .75 .90 .95
## 17.02 21.20 25.00 34.80 43.40
##
## lowest : 5 5.6 6.3 7 7.2 , highest: 46.7 48.3 48.5 48.8 50
## --------------------------------------------------------------------------------Prepareing data
We select a subset of variables for a simpler demonstration. In practice, variable selection may be based on domain knowledge, correlation screening, or model-based importance.
Boston <- dplyr::select (Boston ,medv , crim , rm , tax , lstat)We split the dataset into training and test sets. This is necessary to evaluate out-of-sample RMSE (or other regression metrics).
# Splitting the dataset into
# the Training set and Test set
# install.packages('caTools')
library(caTools)
set.seed(123)
split = sample.split(Boston$medv,
SplitRatio = 0.75)
training_set_origi = subset(Boston,
split == TRUE)
test_set_origi = subset(Boston,
split == FALSE)Feature scaling is applied to predictors. This is critical for distance-based methods like KNN.
# Feature Scaling
training_set = scale(training_set_origi[,-1] )
test_set = scale(test_set_origi [,-1])Creating model
This code fits a KNN model and predicts on the test set. Note: The
class::knn() function is primarily designed for
classification, but this code demonstrates a simplified approach for
regression-style prediction. In applied regression settings, you would
typically use a KNN regression implementation (e.g., from specialized
packages) to avoid forced categorical conversion.
# Fitting K-NN to the Training set
# and Predicting the Test set results
# library(class)
y_pred = knn(train = training_set[, -1],
test = test_set[, -1],
cl = training_set_origi[, 1],
k = 15 )
#Evaluation
We compute prediction error and a simple RMSE-like metric. In
standard regression, RMSE is computed as
sqrt(mean(error^2)). Here we follow the structure in the
code and inspect results.
# converting factor into character then into numeric
error <- test_set_origi[,1]-as.numeric (as.character(y_pred))
head(error)## [1] -3.5 -0.5 -1.4 -2.6 1.0 -8.8rmse <- sqrt(mean(error)^2)
rmse## [1] 0.8487179Plotting the error helps detect systematic bias or extreme outliers.
plot(error)
Viewing observed vs predicted values is a quick sanity check.
head(cbind(test_set_origi[,1], as.numeric (as.character(y_pred))))## [,1] [,2]
## [1,] 18.2 21.7
## [2,] 19.9 20.4
## [3,] 17.5 18.9
## [4,] 15.2 17.8
## [5,] 14.5 13.5
## [6,] 15.6 24.4Optimization
- search better k parameter As in classification, K controls the
bias-variance tradeoff. We search over a range of K values and compute
the metric stored in
k.optm.
i=1
k.optm=1
for (i in 1:29){
y_pred = knn(train = training_set[, -1],
test = test_set[, -1],
cl = training_set_origi[, 1],
k = i )
k.optm[i] <- sqrt(mean( test_set_origi[,1]-as.numeric (as.character(y_pred)) )^2)
k=i
cat(k,'=',k.optm[i],'')
}## 1 = 0.35 2 = 0.5371795 3 = 0.9705128 4 = 1.105128 5 = 1.373077 6 = 0.4512821 7 = 0.6230769 8 = 0.575641 9 = 1.325641 10 = 1.176923 11 = 0.6628205 12 = 0.15 13 = 0.04358974 14 = 0.724359 15 = 0.3551282 16 = 0.07820513 17 = 0.07820513 18 = 0.6346154 19 = 0.2628205 20 = 0.4769231 21 = 0.9294872 22 = 0.6423077 23 = 0.4333333 24 = 0.4320513 25 = 0.3807692 26 = 1.061538 27 = 0.924359 28 = 0.7230769 29 = 0.03461538- Accuracy plot
k=15Plotting the metric versus K helps identify the region where performance stabilizes and avoid choosing K based on a single noisy optimum.
plot(k.optm, type="b", xlab="K- Value",ylab="RMSE level")