Leydet_Lab09
David Leydet
2023-03-27
Pipelines
Introduction
In this lab, we’ll go an example of building a pipeline to feed data into a machine learning model. You will need the credit_data.csv dataset (attached).
## set working directory
setwd("~/Desktop/University of Utah PhD /Course Work/Spring 2023 Semester/GEOG6160 - Spatial Modeling/Labs/lab09/")
## Load required libraries
library(dplyr) ##adjusting data##
## Attaching package: 'dplyr'## The following objects are masked from 'package:stats':
##
## filter, lag## The following objects are masked from 'package:base':
##
## intersect, setdiff, setequal, unionlibrary(mlr3verse) ##all of the mlr3 packages## Loading required package: mlr3library(mlr3pipelines) #pipelines
library(paradox) ##tuning
library(mlr3filters) ##filter for feature (variable) selection
library(praznik) ##required package for the filterPipelines
Machine learning pipelines are commonly used to produce reproducible and consistent results from a machine learning model. The general goal is to link together a series of functions that undertake all (or most) of the data pre-processing steps, and link these directly into one or more algorithms. Pipelines have several advantages:
- They will process all data in the same way, making it easy to work with multiple datasets (including datasets used for predictions)
- They provide a more reproducible approach to data processing than a series of ad hoc steps and code
- They can easily be used in training, evaluating and tuning models
- They provide a description of the steps involved for later reference
We’ll work again with the credit_data.csv file: a dataset of credit rankings for over 4000 people (see appendix for a description of the fields). The goal will be to predict Status, a binary outcome with two levels: good and bad. We’ll start by loading the libraries we need:
## Read in the data
credit_data <- read.csv("../datafiles/credit_data.csv")
## Check
str(credit_data)## 'data.frame': 4454 obs. of 14 variables:
## $ Status : chr "good" "good" "bad" "good" ...
## $ Seniority: int 9 17 10 0 0 1 29 9 0 0 ...
## $ Home : chr "rent" "rent" "owner" "rent" ...
## $ Time : int 60 60 36 60 36 60 60 12 60 48 ...
## $ Age : int 30 58 46 24 26 36 44 27 32 41 ...
## $ Marital : chr "married" "widow" "married" "single" ...
## $ Records : chr "no" "no" "yes" "no" ...
## $ Job : chr "freelance" "fixed" "freelance" "fixed" ...
## $ Expenses : int 73 48 90 63 46 75 75 35 90 90 ...
## $ Income : int 129 131 200 182 107 214 125 80 107 80 ...
## $ Assets : int 0 0 3000 2500 0 3500 10000 0 15000 0 ...
## $ Debt : int 0 0 0 0 0 0 0 0 0 0 ...
## $ Amount : int 800 1000 2000 900 310 650 1600 200 1200 1200 ...
## $ Price : int 846 1658 2985 1325 910 1645 1800 1093 1957 1468 ...As we have several categorical variables, we need to make sure that R recognizes these as factors. The following line of code checks each column in the credit_data data frame, and if it contains character data, it then converts it to a factor. Note this is similar to the approach in previous labs, where we convert individual variables to factors:
## Use dplyr to convert from characters to factors
credit_data = credit_data %>%
mutate_if(is.character, as.factor)
## Check
str(credit_data)## 'data.frame': 4454 obs. of 14 variables:
## $ Status : Factor w/ 2 levels "bad","good": 2 2 1 2 2 2 2 2 2 1 ...
## $ Seniority: int 9 17 10 0 0 1 29 9 0 0 ...
## $ Home : Factor w/ 6 levels "ignore","other",..: 6 6 3 6 6 3 3 4 3 4 ...
## $ Time : int 60 60 36 60 36 60 60 12 60 48 ...
## $ Age : int 30 58 46 24 26 36 44 27 32 41 ...
## $ Marital : Factor w/ 5 levels "divorced","married",..: 2 5 2 4 4 2 2 4 2 2 ...
## $ Records : Factor w/ 2 levels "no","yes": 1 1 2 1 1 1 1 1 1 1 ...
## $ Job : Factor w/ 4 levels "fixed","freelance",..: 2 1 2 1 1 1 1 1 2 4 ...
## $ Expenses : int 73 48 90 63 46 75 75 35 90 90 ...
## $ Income : int 129 131 200 182 107 214 125 80 107 80 ...
## $ Assets : int 0 0 3000 2500 0 3500 10000 0 15000 0 ...
## $ Debt : int 0 0 0 0 0 0 0 0 0 0 ...
## $ Amount : int 800 1000 2000 900 310 650 1600 200 1200 1200 ...
## $ Price : int 846 1658 2985 1325 910 1645 1800 1093 1957 1468 ...If you run the summary() function, you should see that there are several missing values:
## Check
summary(credit_data)## Status Seniority Home Time Age
## bad :1254 Min. : 0.000 ignore : 20 Min. : 6.00 Min. :18.00
## good:3200 1st Qu.: 2.000 other : 319 1st Qu.:36.00 1st Qu.:28.00
## Median : 5.000 owner :2107 Median :48.00 Median :36.00
## Mean : 7.987 parents: 783 Mean :46.44 Mean :37.08
## 3rd Qu.:12.000 priv : 246 3rd Qu.:60.00 3rd Qu.:45.00
## Max. :48.000 rent : 973 Max. :72.00 Max. :68.00
## NA's : 6
## Marital Records Job Expenses Income
## divorced : 38 no :3681 fixed :2805 Min. : 35.00 Min. : 6.0
## married :3241 yes: 773 freelance:1024 1st Qu.: 35.00 1st Qu.: 90.0
## separated: 130 others : 171 Median : 51.00 Median :125.0
## single : 977 partime : 452 Mean : 55.57 Mean :141.7
## widow : 67 NA's : 2 3rd Qu.: 72.00 3rd Qu.:170.0
## NA's : 1 Max. :180.00 Max. :959.0
## NA's :381
## Assets Debt Amount Price
## Min. : 0 Min. : 0 Min. : 100 Min. : 105
## 1st Qu.: 0 1st Qu.: 0 1st Qu.: 700 1st Qu.: 1117
## Median : 3000 Median : 0 Median :1000 Median : 1400
## Mean : 5404 Mean : 343 Mean :1039 Mean : 1463
## 3rd Qu.: 6000 3rd Qu.: 0 3rd Qu.:1300 3rd Qu.: 1692
## Max. :300000 Max. :30000 Max. :5000 Max. :11140
## NA's :47 NA's :18As machine learning algorithms can’t use missing data to train, we need to decide what to do with these. Fortunately, mlr3 comes with a set of pre-processing tools in the mlr3pipelines library to help with these missing values. Let’s first set up a classification task with the credit dataset:
## Setup a new task
credit_task = TaskClassif$new(id = "credit",
backend = credit_data,
target = "Status")There are several steps that we might want to do to process these data:
- Impute any missing values
- Convert categorical/factor variables to numeric by one-hot encoding
- Scale the numerical variables to prevent biases while training our model
While it is possible to do this in an ad-hoc way (as we did in previous labs), we will set up a processing pipeline that contains all of these steps. This takes more time to set up, but has a number of advantages: we can use the pipeline directly in cross-validation or tuning, and we can use it to process any new data that we might to make predictions for, without having to remember the individual steps.
The main function is PipeOps or po() which creates a pipeline operator that will carry our a single data processing step. To see the set of options, simply type:
## Check the list of options
mlr_pipeops## <DictionaryPipeOp> with 64 stored values
## Keys: boxcox, branch, chunk, classbalancing, classifavg, classweights,
## colapply, collapsefactors, colroles, copy, datefeatures, encode,
## encodeimpact, encodelmer, featureunion, filter, fixfactors, histbin,
## ica, imputeconstant, imputehist, imputelearner, imputemean,
## imputemedian, imputemode, imputeoor, imputesample, kernelpca,
## learner, learner_cv, missind, modelmatrix, multiplicityexply,
## multiplicityimply, mutate, nmf, nop, ovrsplit, ovrunite, pca, proxy,
## quantilebin, randomprojection, randomresponse, regravg,
## removeconstants, renamecolumns, replicate, scale, scalemaxabs,
## scalerange, select, smote, spatialsign, subsample, targetinvert,
## targetmutate, targettrafoscalerange, textvectorizer, threshold,
## tunethreshold, unbranch, vtreat, yeojohnsonMore information on the individual operations (and some examples) can be found here.
It is possible to set up the full pipeline in one step, but we’ll work through this gradually so you can get a sense of what each operator is doing. First, we’ll design two operators to process the categorical data.
The first (impute_cat) will carry out a mode-based imputation of any missing values (i.e. fill in with the most common value). Note that we include an argument affect_columns to define which features should be processed (this will ignore any numerical features).
The second operator will one-hot encode the same set of features. The argument method = “treatment” will drop the first encode column to prevent redundancy and multicollinearity (and alternative is method = “onehot” which doesn’t drop the first column).
## Impute missing values using the mode (most common value)
impute_cat = po("imputemode",
affect_columns = selector_type("factor"))
## Encode the same factors
encode = po("encode",
method = "treatment",
affect_columns = selector_type("factor"))With these set up, we can now use a pipe function (%>>%) to combine these into a pipeline for the categorical features
cat = impute_cat %>>%
encodeThe pipeline we have just created has a train() method. This is does not train a model (as we haven’t linked one into the pipeline yet), but will run the set of operators on a task. We can use this to show what the resulting transformation is:
## Check the data before running it through the pipe
credit_task$data()## Status Age Amount Assets Debt Expenses Home Income Job Marital
## 1: good 30 800 0 0 73 rent 129 freelance married
## 2: good 58 1000 0 0 48 rent 131 fixed widow
## 3: bad 46 2000 3000 0 90 owner 200 freelance married
## 4: good 24 900 2500 0 63 rent 182 fixed single
## 5: good 26 310 0 0 46 rent 107 fixed single
## ---
## 4450: bad 39 900 0 0 69 rent 92 fixed married
## 4451: good 46 950 3000 600 60 owner 75 fixed married
## 4452: bad 37 500 3500 0 60 owner 90 partime married
## 4453: good 23 550 0 0 49 rent 140 freelance single
## 4454: good 32 1350 4000 1000 60 owner 140 freelance married
## Price Records Seniority Time
## 1: 846 no 9 60
## 2: 1658 no 17 60
## 3: 2985 yes 10 36
## 4: 1325 no 0 60
## 5: 910 no 0 36
## ---
## 4450: 1020 no 1 60
## 4451: 1263 no 22 60
## 4452: 963 no 0 24
## 4453: 550 no 0 48
## 4454: 1650 no 5 60## Train the task using the pipe
## What is the [[1]] for??
cat$train(credit_task)[[1]]$data()## Status Home.other Home.owner Home.parents Home.priv Home.rent
## 1: good 0 0 0 0 1
## 2: good 0 0 0 0 1
## 3: bad 0 1 0 0 0
## 4: good 0 0 0 0 1
## 5: good 0 0 0 0 1
## ---
## 4450: bad 0 0 0 0 1
## 4451: good 0 1 0 0 0
## 4452: bad 0 1 0 0 0
## 4453: good 0 0 0 0 1
## 4454: good 0 1 0 0 0
## Job.freelance Job.others Job.partime Marital.married Marital.separated
## 1: 1 0 0 1 0
## 2: 0 0 0 0 0
## 3: 1 0 0 1 0
## 4: 0 0 0 0 0
## 5: 0 0 0 0 0
## ---
## 4450: 0 0 0 1 0
## 4451: 0 0 0 1 0
## 4452: 0 0 1 1 0
## 4453: 1 0 0 0 0
## 4454: 1 0 0 1 0
## Marital.single Marital.widow Records.yes Age Amount Assets Debt Expenses
## 1: 0 0 0 30 800 0 0 73
## 2: 0 1 0 58 1000 0 0 48
## 3: 0 0 1 46 2000 3000 0 90
## 4: 1 0 0 24 900 2500 0 63
## 5: 1 0 0 26 310 0 0 46
## ---
## 4450: 0 0 0 39 900 0 0 69
## 4451: 0 0 0 46 950 3000 600 60
## 4452: 0 0 0 37 500 3500 0 60
## 4453: 1 0 0 23 550 0 0 49
## 4454: 0 0 0 32 1350 4000 1000 60
## Income Price Seniority Time
## 1: 129 846 9 60
## 2: 131 1658 17 60
## 3: 200 2985 10 36
## 4: 182 1325 0 60
## 5: 107 910 0 36
## ---
## 4450: 92 1020 1 60
## 4451: 75 1263 22 60
## 4452: 90 963 0 24
## 4453: 140 550 0 48
## 4454: 140 1650 5 60You should see here that each of the original factor variables (e.g. Home) has been one-hot encoded, and there are now m−1 new features, each representing one level in the original factor (e.g. Home.owner). Note that this automatically drops the original feature and has not affected the numerical variables. In fact, if you re-run the summary() function on this new dataset, you should see that there are still missing variables in the numerical features
## Check the summary for N/As
summary(cat$train(credit_task)[[1]]$data())## Status Home.other Home.owner Home.parents
## bad :1254 Min. :0.00000 Min. :0.0000 Min. :0.0000
## good:3200 1st Qu.:0.00000 1st Qu.:0.0000 1st Qu.:0.0000
## Median :0.00000 Median :0.0000 Median :0.0000
## Mean :0.07162 Mean :0.4744 Mean :0.1758
## 3rd Qu.:0.00000 3rd Qu.:1.0000 3rd Qu.:0.0000
## Max. :1.00000 Max. :1.0000 Max. :1.0000
##
## Home.priv Home.rent Job.freelance Job.others
## Min. :0.00000 Min. :0.0000 Min. :0.0000 Min. :0.00000
## 1st Qu.:0.00000 1st Qu.:0.0000 1st Qu.:0.0000 1st Qu.:0.00000
## Median :0.00000 Median :0.0000 Median :0.0000 Median :0.00000
## Mean :0.05523 Mean :0.2185 Mean :0.2299 Mean :0.03839
## 3rd Qu.:0.00000 3rd Qu.:0.0000 3rd Qu.:0.0000 3rd Qu.:0.00000
## Max. :1.00000 Max. :1.0000 Max. :1.0000 Max. :1.00000
##
## Job.partime Marital.married Marital.separated Marital.single
## Min. :0.0000 Min. :0.0000 Min. :0.00000 Min. :0.0000
## 1st Qu.:0.0000 1st Qu.:0.0000 1st Qu.:0.00000 1st Qu.:0.0000
## Median :0.0000 Median :1.0000 Median :0.00000 Median :0.0000
## Mean :0.1015 Mean :0.7279 Mean :0.02919 Mean :0.2194
## 3rd Qu.:0.0000 3rd Qu.:1.0000 3rd Qu.:0.00000 3rd Qu.:0.0000
## Max. :1.0000 Max. :1.0000 Max. :1.00000 Max. :1.0000
##
## Marital.widow Records.yes Age Amount
## Min. :0.00000 Min. :0.0000 Min. :18.00 Min. : 100
## 1st Qu.:0.00000 1st Qu.:0.0000 1st Qu.:28.00 1st Qu.: 700
## Median :0.00000 Median :0.0000 Median :36.00 Median :1000
## Mean :0.01504 Mean :0.1736 Mean :37.08 Mean :1039
## 3rd Qu.:0.00000 3rd Qu.:0.0000 3rd Qu.:45.00 3rd Qu.:1300
## Max. :1.00000 Max. :1.0000 Max. :68.00 Max. :5000
##
## Assets Debt Expenses Income
## Min. : 0 Min. : 0 Min. : 35.00 Min. : 6.0
## 1st Qu.: 0 1st Qu.: 0 1st Qu.: 35.00 1st Qu.: 90.0
## Median : 3000 Median : 0 Median : 51.00 Median :125.0
## Mean : 5404 Mean : 343 Mean : 55.57 Mean :141.7
## 3rd Qu.: 6000 3rd Qu.: 0 3rd Qu.: 72.00 3rd Qu.:170.0
## Max. :300000 Max. :30000 Max. :180.00 Max. :959.0
## NA's :47 NA's :18 NA's :381
## Price Seniority Time
## Min. : 105 Min. : 0.000 Min. : 6.00
## 1st Qu.: 1117 1st Qu.: 2.000 1st Qu.:36.00
## Median : 1400 Median : 5.000 Median :48.00
## Mean : 1463 Mean : 7.987 Mean :46.44
## 3rd Qu.: 1692 3rd Qu.:12.000 3rd Qu.:60.00
## Max. :11140 Max. :48.000 Max. :72.00
## Now we’ll set operators for the numerical variables: we’ll use median imputation (imputemedian) for missing values, and we’ll use a min-max scaling to a 0-1 range (scalerange):
## Impute missing numerical values
impute_num = po("imputemedian",
affect_columns = selector_type("integer"))
## min-max scaling to a 0-1 range
scale = po("scalerange",
param_vals = list(lower = 0, upper = 1),
affect_columns = selector_type("integer"))
## Set pipe
num = impute_num %>>%
scale## Train the task data
num$train(credit_task)[[1]]$data()## Status Age Amount Assets Debt Expenses Income
## 1: good 0.24 0.14285714 0.000000000 0.00000000 0.26206897 0.12906611
## 2: good 0.80 0.18367347 0.000000000 0.00000000 0.08965517 0.13116474
## 3: bad 0.56 0.38775510 0.010000000 0.00000000 0.37931034 0.20356768
## 4: good 0.12 0.16326531 0.008333333 0.00000000 0.19310345 0.18467996
## 5: good 0.16 0.04285714 0.000000000 0.00000000 0.07586207 0.10598111
## ---
## 4450: bad 0.42 0.16326531 0.000000000 0.00000000 0.23448276 0.09024134
## 4451: good 0.56 0.17346939 0.010000000 0.02000000 0.17241379 0.07240294
## 4452: bad 0.38 0.08163265 0.011666667 0.00000000 0.17241379 0.08814271
## 4453: good 0.10 0.09183673 0.000000000 0.00000000 0.09655172 0.14060860
## 4454: good 0.28 0.25510204 0.013333333 0.03333333 0.17241379 0.14060860
## Price Seniority Time Home Job Marital Records
## 1: 0.06714998 0.18750000 0.8181818 rent freelance married no
## 2: 0.14073403 0.35416667 0.8181818 rent fixed widow no
## 3: 0.26098777 0.20833333 0.4545455 owner freelance married yes
## 4: 0.11055732 0.00000000 0.8181818 rent fixed single no
## 5: 0.07294971 0.00000000 0.4545455 rent fixed single no
## ---
## 4450: 0.08291799 0.02083333 0.8181818 rent fixed married no
## 4451: 0.10493883 0.45833333 0.8181818 owner fixed married no
## 4452: 0.07775261 0.00000000 0.2727273 owner partime married no
## 4453: 0.04032623 0.00000000 0.6363636 rent freelance single no
## 4454: 0.14000906 0.10416667 0.8181818 owner freelance married noAs before, we can see the resulting transformation (note that this has not affected any of the factor variables):
We can now combine these two sets of operators into the full pipeline.
## Combine both pipes into the full pipeline
graph = cat %>>% numIf you now run this on the credit task, the resulting dataset has the full set of encoded variables and has now imputed values for anything that was missing.
## Process the data
graph$train(credit_task)[[1]]$data()## Status Age Amount Assets Debt Expenses Income
## 1: good 0.24 0.14285714 0.000000000 0.00000000 0.26206897 0.12906611
## 2: good 0.80 0.18367347 0.000000000 0.00000000 0.08965517 0.13116474
## 3: bad 0.56 0.38775510 0.010000000 0.00000000 0.37931034 0.20356768
## 4: good 0.12 0.16326531 0.008333333 0.00000000 0.19310345 0.18467996
## 5: good 0.16 0.04285714 0.000000000 0.00000000 0.07586207 0.10598111
## ---
## 4450: bad 0.42 0.16326531 0.000000000 0.00000000 0.23448276 0.09024134
## 4451: good 0.56 0.17346939 0.010000000 0.02000000 0.17241379 0.07240294
## 4452: bad 0.38 0.08163265 0.011666667 0.00000000 0.17241379 0.08814271
## 4453: good 0.10 0.09183673 0.000000000 0.00000000 0.09655172 0.14060860
## 4454: good 0.28 0.25510204 0.013333333 0.03333333 0.17241379 0.14060860
## Price Seniority Time Home.other Home.owner Home.parents
## 1: 0.06714998 0.18750000 0.8181818 0 0 0
## 2: 0.14073403 0.35416667 0.8181818 0 0 0
## 3: 0.26098777 0.20833333 0.4545455 0 1 0
## 4: 0.11055732 0.00000000 0.8181818 0 0 0
## 5: 0.07294971 0.00000000 0.4545455 0 0 0
## ---
## 4450: 0.08291799 0.02083333 0.8181818 0 0 0
## 4451: 0.10493883 0.45833333 0.8181818 0 1 0
## 4452: 0.07775261 0.00000000 0.2727273 0 1 0
## 4453: 0.04032623 0.00000000 0.6363636 0 0 0
## 4454: 0.14000906 0.10416667 0.8181818 0 1 0
## Home.priv Home.rent Job.freelance Job.others Job.partime Marital.married
## 1: 0 1 1 0 0 1
## 2: 0 1 0 0 0 0
## 3: 0 0 1 0 0 1
## 4: 0 1 0 0 0 0
## 5: 0 1 0 0 0 0
## ---
## 4450: 0 1 0 0 0 1
## 4451: 0 0 0 0 0 1
## 4452: 0 0 0 0 1 1
## 4453: 0 1 1 0 0 0
## 4454: 0 0 1 0 0 1
## Marital.separated Marital.single Marital.widow Records.yes
## 1: 0 0 0 0
## 2: 0 0 1 0
## 3: 0 0 0 1
## 4: 0 1 0 0
## 5: 0 1 0 0
## ---
## 4450: 0 0 0 0
## 4451: 0 0 0 0
## 4452: 0 0 0 0
## 4453: 0 1 0 0
## 4454: 0 0 0 0## Check the final data product
summary(graph$train(credit_task)[[1]]$data())## Status Age Amount Assets
## bad :1254 Min. :0.0000 Min. :0.0000 Min. :0.00000
## good:3200 1st Qu.:0.2000 1st Qu.:0.1224 1st Qu.:0.00000
## Median :0.3600 Median :0.1837 Median :0.01000
## Mean :0.3816 Mean :0.1916 Mean :0.01793
## 3rd Qu.:0.5400 3rd Qu.:0.2449 3rd Qu.:0.02000
## Max. :1.0000 Max. :1.0000 Max. :1.00000
## Debt Expenses Income Price
## Min. :0.00000 Min. :0.0000 Min. :0.00000 Min. :0.00000
## 1st Qu.:0.00000 1st Qu.:0.0000 1st Qu.:0.09129 1st Qu.:0.09173
## Median :0.00000 Median :0.1103 Median :0.12487 Median :0.11735
## Mean :0.01139 Mean :0.1419 Mean :0.14088 Mean :0.12304
## 3rd Qu.:0.00000 3rd Qu.:0.2552 3rd Qu.:0.16579 3rd Qu.:0.14377
## Max. :1.00000 Max. :1.0000 Max. :1.00000 Max. :1.00000
## Seniority Time Home.other Home.owner
## Min. :0.00000 Min. :0.0000 Min. :0.00000 Min. :0.0000
## 1st Qu.:0.04167 1st Qu.:0.4545 1st Qu.:0.00000 1st Qu.:0.0000
## Median :0.10417 Median :0.6364 Median :0.00000 Median :0.0000
## Mean :0.16639 Mean :0.6127 Mean :0.07162 Mean :0.4744
## 3rd Qu.:0.25000 3rd Qu.:0.8182 3rd Qu.:0.00000 3rd Qu.:1.0000
## Max. :1.00000 Max. :1.0000 Max. :1.00000 Max. :1.0000
## Home.parents Home.priv Home.rent Job.freelance
## Min. :0.0000 Min. :0.00000 Min. :0.0000 Min. :0.0000
## 1st Qu.:0.0000 1st Qu.:0.00000 1st Qu.:0.0000 1st Qu.:0.0000
## Median :0.0000 Median :0.00000 Median :0.0000 Median :0.0000
## Mean :0.1758 Mean :0.05523 Mean :0.2185 Mean :0.2299
## 3rd Qu.:0.0000 3rd Qu.:0.00000 3rd Qu.:0.0000 3rd Qu.:0.0000
## Max. :1.0000 Max. :1.00000 Max. :1.0000 Max. :1.0000
## Job.others Job.partime Marital.married Marital.separated
## Min. :0.00000 Min. :0.0000 Min. :0.0000 Min. :0.00000
## 1st Qu.:0.00000 1st Qu.:0.0000 1st Qu.:0.0000 1st Qu.:0.00000
## Median :0.00000 Median :0.0000 Median :1.0000 Median :0.00000
## Mean :0.03839 Mean :0.1015 Mean :0.7279 Mean :0.02919
## 3rd Qu.:0.00000 3rd Qu.:0.0000 3rd Qu.:1.0000 3rd Qu.:0.00000
## Max. :1.00000 Max. :1.0000 Max. :1.0000 Max. :1.00000
## Marital.single Marital.widow Records.yes
## Min. :0.0000 Min. :0.00000 Min. :0.0000
## 1st Qu.:0.0000 1st Qu.:0.00000 1st Qu.:0.0000
## Median :0.0000 Median :0.00000 Median :0.0000
## Mean :0.2194 Mean :0.01504 Mean :0.1736
## 3rd Qu.:0.0000 3rd Qu.:0.00000 3rd Qu.:0.0000
## Max. :1.0000 Max. :1.00000 Max. :1.0000Linking Learners
So far, we have a pipeline that transforms the dataset into a format ready for modeling. The next step is to link it to a learner, so that we can pass the transformed data directly to it. We’ll set up a classification based random forest, with mtry = 3 (number of variables per split) and num.trees = 500:
## Set up the learner
lrn_rf = lrn("classif.ranger",
mtry = 3,
num.trees = 500,
predict_type = "prob")And we can just add this to our existing pipeline
## Add the learner to the end of the pipeline
graph = cat %>>%
num %>>%
lrn_rfmlr3 uses a graph framework to connect all the pieces of a framework, which means that it can use R-based tools to visualize the pipeline:
## Pipeline Visualization
plot(graph)
Now if we call the train() method on the credit task, the data gets passed throught the pipeline, transformed, encoded and imputed and then used to train the random forest:
## Run the task through the new pipeline that includes the learner
graph$train(credit_task)## $classif.ranger.output
## NULLGraphLearners
In the previous section, we linked the processing steps through to the final algorithm and trained it. For more complex work, e.g. evaluation or tuning, it it is necessary to convert this into a GraphLearner. mlr3 recognizes this as a new learner (rather than a pipeline), with the same attributes as the standard learner objects we have used previously.
## Set up the graphlearner
glrn = GraphLearner$new(graph = graph)Cross-validation
Set up the resampling technique, in this case 5 fold cross-validation, and a performance metric:
## Resampling set up
resampling = rsmp("cv",
folds = 5)
## Instantiate
resampling$instantiate(credit_task)
## Performance Metric
msr_auc = msr("classif.auc")## Run the resampler
## Note: resample Runs a resampling (possibly in parallel): Repeatedly apply Learner learner on a training set of Task task to train a model, then use the trained model to predict observations of a test set. Training and test sets are defined by the Resampling resampling.
rr = resample(task = credit_task,
learner = glrn,
resampling = resampling,
store_models = TRUE)## INFO [19:07:17.269] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.classif.ranger' on task 'credit' (iter 1/5)
## INFO [19:07:18.279] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.classif.ranger' on task 'credit' (iter 2/5)
## INFO [19:07:19.257] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.classif.ranger' on task 'credit' (iter 3/5)
## INFO [19:07:20.087] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.classif.ranger' on task 'credit' (iter 4/5)
## INFO [19:07:21.056] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.classif.ranger' on task 'credit' (iter 5/5)## Individual Model Score
rr$score(msr_auc)## task_id learner_id
## 1: credit imputemode.encode.imputemedian.scalerange.classif.ranger
## 2: credit imputemode.encode.imputemedian.scalerange.classif.ranger
## 3: credit imputemode.encode.imputemedian.scalerange.classif.ranger
## 4: credit imputemode.encode.imputemedian.scalerange.classif.ranger
## 5: credit imputemode.encode.imputemedian.scalerange.classif.ranger
## resampling_id iteration classif.auc
## 1: cv 1 0.8509243
## 2: cv 2 0.8429924
## 3: cv 3 0.8164963
## 4: cv 4 0.8604532
## 5: cv 5 0.8082037
## Hidden columns: task, learner, resampling, prediction## Aggregate
rr$aggregate(msr_auc)## classif.auc
## 0.835814Interpretation: The model does fairly well.
Tuning
We can also use our combined pipeline and learner to tune hyperparameters of the learner. We’ll try tuning the number of trees used to between 100 and 500. To see the full set of available hyperparameters, just type glrn$param_set.
## Check the full set of parameters
glrn$param_set## <ParamSetCollection>
## id class lower upper nlevels
## 1: classif.ranger.alpha ParamDbl -Inf Inf Inf
## 2: classif.ranger.always.split.variables ParamUty NA NA Inf
## 3: classif.ranger.class.weights ParamUty NA NA Inf
## 4: classif.ranger.holdout ParamLgl NA NA 2
## 5: classif.ranger.importance ParamFct NA NA 4
## 6: classif.ranger.keep.inbag ParamLgl NA NA 2
## 7: classif.ranger.max.depth ParamInt 0 Inf Inf
## 8: classif.ranger.min.node.size ParamInt 1 Inf Inf
## 9: classif.ranger.min.prop ParamDbl -Inf Inf Inf
## 10: classif.ranger.minprop ParamDbl -Inf Inf Inf
## 11: classif.ranger.mtry ParamInt 1 Inf Inf
## 12: classif.ranger.mtry.ratio ParamDbl 0 1 Inf
## 13: classif.ranger.num.random.splits ParamInt 1 Inf Inf
## 14: classif.ranger.num.threads ParamInt 1 Inf Inf
## 15: classif.ranger.num.trees ParamInt 1 Inf Inf
## 16: classif.ranger.oob.error ParamLgl NA NA 2
## 17: classif.ranger.regularization.factor ParamUty NA NA Inf
## 18: classif.ranger.regularization.usedepth ParamLgl NA NA 2
## 19: classif.ranger.replace ParamLgl NA NA 2
## 20: classif.ranger.respect.unordered.factors ParamFct NA NA 3
## 21: classif.ranger.sample.fraction ParamDbl 0 1 Inf
## 22: classif.ranger.save.memory ParamLgl NA NA 2
## 23: classif.ranger.scale.permutation.importance ParamLgl NA NA 2
## 24: classif.ranger.se.method ParamFct NA NA 2
## 25: classif.ranger.seed ParamInt -Inf Inf Inf
## 26: classif.ranger.split.select.weights ParamUty NA NA Inf
## 27: classif.ranger.splitrule ParamFct NA NA 3
## 28: classif.ranger.verbose ParamLgl NA NA 2
## 29: classif.ranger.write.forest ParamLgl NA NA 2
## 30: encode.affect_columns ParamUty NA NA Inf
## 31: encode.method ParamFct NA NA 5
## 32: imputemedian.affect_columns ParamUty NA NA Inf
## 33: imputemode.affect_columns ParamUty NA NA Inf
## 34: scalerange.affect_columns ParamUty NA NA Inf
## 35: scalerange.lower ParamDbl -Inf Inf Inf
## 36: scalerange.upper ParamDbl -Inf Inf Inf
## id class lower upper nlevels
## default parents value
## 1: 0.5
## 2: <NoDefault[3]>
## 3:
## 4: FALSE
## 5: <NoDefault[3]>
## 6: FALSE
## 7:
## 8:
## 9: 0.1
## 10: 0.1
## 11: <NoDefault[3]> 3
## 12: <NoDefault[3]>
## 13: 1 classif.ranger.splitrule
## 14: 1 1
## 15: 500 500
## 16: TRUE
## 17: 1
## 18: FALSE
## 19: TRUE
## 20: ignore
## 21: <NoDefault[3]>
## 22: FALSE
## 23: FALSE classif.ranger.importance
## 24: infjack
## 25:
## 26:
## 27: gini
## 28: TRUE
## 29: TRUE
## 30: <Selector[1]> <Selector[1]>
## 31: <NoDefault[3]> treatment
## 32: <NoDefault[3]> <Selector[1]>
## 33: <NoDefault[3]> <Selector[1]>
## 34: <Selector[1]> <Selector[1]>
## 35: <NoDefault[3]> 0
## 36: <NoDefault[3]> 1
## default parents value## Adjust the number of trees
tune_ps = ParamSet$new(list(
ParamInt$new("classif.ranger.num.trees", lower = 100, upper =500)
))
## Check
tune_ps## <ParamSet>
## id class lower upper nlevels default value
## 1: classif.ranger.num.trees ParamInt 100 500 401 <NoDefault[3]>Note: If you are wondering where the name of the hyperparameter we are tuning (classif.ranger.num.trees) comes from, it’s a combination of the learner classif.ranger and the parameter in the learner (num.trees). To check the names of all the available hyperparameters, just type: glrn$param_set.
Now we set up a terminating condition (none = run all possible values), and a search strategy (simple grid choice with 10 steps):
## Set up termination criteria - none
evals = trm("none")
## Set up tuner
tuner = tnr("grid_search",
resolution = 10)And finally, we build an AutoTuner to carry out the tuning using a simple holdout strategy for the inner cross validation. Note that we use the GraphLearner, rather than the base random forest learner as the first argument:
## Set up the autotuner
at_rf = AutoTuner$new(learner = glrn,
resampling = rsmp("holdout", ratio = 0.8),
measure = msr_auc,
search_space = tune_ps,
terminator = evals,
tuner = tuner)Tune the model:
## Tune the model
at_rf$train(credit_task)## INFO [19:07:23.967] [bbotk] Starting to optimize 1 parameter(s) with '<TunerGridSearch>' and '<TerminatorNone>'
## INFO [19:07:23.969] [bbotk] Evaluating 1 configuration(s)
## INFO [19:07:23.989] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:07:23.991] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:07:24.301] [mlr3] Finished benchmark
## INFO [19:07:24.312] [bbotk] Result of batch 1:
## INFO [19:07:24.312] [bbotk] classif.ranger.num.trees classif.auc warnings errors runtime_learners
## INFO [19:07:24.312] [bbotk] 100 0.8232674 0 0 0.306
## INFO [19:07:24.312] [bbotk] uhash
## INFO [19:07:24.312] [bbotk] 6882ada1-7f39-41a8-9717-8b67aa175448
## INFO [19:07:24.313] [bbotk] Evaluating 1 configuration(s)
## INFO [19:07:24.329] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:07:24.331] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:07:25.131] [mlr3] Finished benchmark
## INFO [19:07:25.142] [bbotk] Result of batch 2:
## INFO [19:07:25.143] [bbotk] classif.ranger.num.trees classif.auc warnings errors runtime_learners
## INFO [19:07:25.143] [bbotk] 322 0.8270307 0 0 0.798
## INFO [19:07:25.143] [bbotk] uhash
## INFO [19:07:25.143] [bbotk] a075ee23-f873-43c6-8c7e-ce0e1fda98df
## INFO [19:07:25.144] [bbotk] Evaluating 1 configuration(s)
## INFO [19:07:25.159] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:07:25.162] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:07:25.685] [mlr3] Finished benchmark
## INFO [19:07:25.697] [bbotk] Result of batch 3:
## INFO [19:07:25.697] [bbotk] classif.ranger.num.trees classif.auc warnings errors runtime_learners
## INFO [19:07:25.697] [bbotk] 278 0.8242144 0 0 0.521
## INFO [19:07:25.697] [bbotk] uhash
## INFO [19:07:25.697] [bbotk] 5f9a0a56-eadb-4ed0-a5ac-ee1d2c001ffc
## INFO [19:07:25.698] [bbotk] Evaluating 1 configuration(s)
## INFO [19:07:25.715] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:07:25.718] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:07:26.075] [mlr3] Finished benchmark
## INFO [19:07:26.086] [bbotk] Result of batch 4:
## INFO [19:07:26.086] [bbotk] classif.ranger.num.trees classif.auc warnings errors runtime_learners
## INFO [19:07:26.086] [bbotk] 144 0.8255767 0 0 0.355
## INFO [19:07:26.086] [bbotk] uhash
## INFO [19:07:26.086] [bbotk] 9de59e97-9ecf-4aa4-89b0-b6de1d633521
## INFO [19:07:26.087] [bbotk] Evaluating 1 configuration(s)
## INFO [19:07:26.102] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:07:26.104] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:07:26.914] [mlr3] Finished benchmark
## INFO [19:07:26.925] [bbotk] Result of batch 5:
## INFO [19:07:26.925] [bbotk] classif.ranger.num.trees classif.auc warnings errors runtime_learners
## INFO [19:07:26.925] [bbotk] 500 0.8267985 0 0 0.806
## INFO [19:07:26.925] [bbotk] uhash
## INFO [19:07:26.925] [bbotk] 768bf4bf-bed2-40a2-9462-8a7cc234f0db
## INFO [19:07:26.926] [bbotk] Evaluating 1 configuration(s)
## INFO [19:07:26.942] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:07:26.944] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:07:27.644] [mlr3] Finished benchmark
## INFO [19:07:27.655] [bbotk] Result of batch 6:
## INFO [19:07:27.655] [bbotk] classif.ranger.num.trees classif.auc warnings errors runtime_learners
## INFO [19:07:27.655] [bbotk] 411 0.824868 0 0 0.697
## INFO [19:07:27.655] [bbotk] uhash
## INFO [19:07:27.655] [bbotk] ebb4dc7b-f302-491a-93ff-4b31225e771d
## INFO [19:07:27.656] [bbotk] Evaluating 1 configuration(s)
## INFO [19:07:27.671] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:07:27.674] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:07:28.072] [mlr3] Finished benchmark
## INFO [19:07:28.082] [bbotk] Result of batch 7:
## INFO [19:07:28.083] [bbotk] classif.ranger.num.trees classif.auc warnings errors runtime_learners
## INFO [19:07:28.083] [bbotk] 189 0.8245931 0 0 0.394
## INFO [19:07:28.083] [bbotk] uhash
## INFO [19:07:28.083] [bbotk] c78897f8-a988-40c6-8cc5-92b8be94d659
## INFO [19:07:28.084] [bbotk] Evaluating 1 configuration(s)
## INFO [19:07:28.099] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:07:28.102] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:07:28.868] [mlr3] Finished benchmark
## INFO [19:07:28.879] [bbotk] Result of batch 8:
## INFO [19:07:28.879] [bbotk] classif.ranger.num.trees classif.auc warnings errors runtime_learners
## INFO [19:07:28.879] [bbotk] 456 0.8261815 0 0 0.763
## INFO [19:07:28.879] [bbotk] uhash
## INFO [19:07:28.879] [bbotk] a76c5e1b-a7a3-402c-9d5a-9d556d6706fd
## INFO [19:07:28.880] [bbotk] Evaluating 1 configuration(s)
## INFO [19:07:28.896] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:07:28.898] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:07:29.355] [mlr3] Finished benchmark
## INFO [19:07:29.366] [bbotk] Result of batch 9:
## INFO [19:07:29.366] [bbotk] classif.ranger.num.trees classif.auc warnings errors runtime_learners
## INFO [19:07:29.366] [bbotk] 233 0.8188688 0 0 0.453
## INFO [19:07:29.366] [bbotk] uhash
## INFO [19:07:29.366] [bbotk] a7b4f8fc-4f87-491b-9d7a-f149c65879c6
## INFO [19:07:29.367] [bbotk] Evaluating 1 configuration(s)
## INFO [19:07:29.382] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:07:29.385] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:07:30.027] [mlr3] Finished benchmark
## INFO [19:07:30.038] [bbotk] Result of batch 10:
## INFO [19:07:30.038] [bbotk] classif.ranger.num.trees classif.auc warnings errors runtime_learners
## INFO [19:07:30.038] [bbotk] 367 0.8252835 0 0 0.639
## INFO [19:07:30.038] [bbotk] uhash
## INFO [19:07:30.038] [bbotk] 23a92a06-384e-4cff-b655-7a3722c77834
## INFO [19:07:30.041] [bbotk] Finished optimizing after 10 evaluation(s)
## INFO [19:07:30.042] [bbotk] Result:
## INFO [19:07:30.042] [bbotk] classif.ranger.num.trees learner_param_vals x_domain classif.auc
## INFO [19:07:30.042] [bbotk] 322 <list[10]> <list[1]> 0.8270307## Check
at_rf$learner## <GraphLearner:imputemode.encode.imputemedian.scalerange.classif.ranger>
## * Model: list
## * Parameters: imputemode.affect_columns=<Selector>,
## encode.method=treatment, encode.affect_columns=<Selector>,
## imputemedian.affect_columns=<Selector>, scalerange.lower=0,
## scalerange.upper=1, scalerange.affect_columns=<Selector>,
## classif.ranger.mtry=3, classif.ranger.num.threads=1,
## classif.ranger.num.trees=322
## * Packages: mlr3, mlr3pipelines, stats, mlr3learners, ranger
## * Predict Types: response, [prob]
## * Feature Types: logical, integer, numeric, character, factor, ordered,
## POSIXct
## * Properties: featureless, hotstart_backward, hotstart_forward,
## importance, loglik, missings, multiclass, oob_error,
## selected_features, twoclass, weights## Check the tuning result
at_rf$tuning_result## classif.ranger.num.trees learner_param_vals x_domain classif.auc
## 1: 322 <list[10]> <list[1]> 0.8270307Note: Slight improvement over our initial run
Pipeline Predictions
We can use the tuned learner to make predictions for a new data set. As the learner is built on the data transformation pipeline, we would need to carry out any transformations prior to prediction. Instead we can simply pass the new data to the learner, and leave it to do all that for us. We’ll create a single example and use the predict_new() method to get a prediction of credit risk (feel free to use different values to see the impact here):
## Build a new credit data frame for the predictions
new_credit = data.frame(Seniority = 8,
Home = "rent",
Time = 36,
Age = 26,
Marital = "single",
Records = "no",
Job = "fixed",
Expenses = 50,
Income = 100,
Assets = 0,
Debt = 10,
Amount = 100,
Price = 125)
## Carry out a prediction
at_rf$learner$predict_newdata(new_credit)## <PredictionClassif> for 1 observations:
## row_ids truth response prob.bad prob.good
## 1 <NA> good 0.4093246 0.5906754Feature Selection
When working with datasets with large numbers of features we often want to reduce the number of features used to build the model. We can further modify the pipeline to allow for feature selection using the mlr3filters package. The full set of filters can be found on the package website or by typing:
## List of filters
mlr_filters## <DictionaryFilter> with 21 stored values
## Keys: anova, auc, carscore, carsurvscore, cmim, correlation, disr,
## find_correlation, importance, information_gain, jmi, jmim,
## kruskal_test, mim, mrmr, njmim, performance, permutation, relief,
## selected_features, varianceWe’ll use the mim filter. For a given classification task this calculates an information based correlation between the outcome and each feature and then selects the subset with the highest values. We’ll use this to pick the top 3 features. As we want to use this in the pipeline, we again use the po() function to create a new operator. We specify the filter type (mim) and the number of features we want to retain:
## Create the filter pipe
filter_mim = po("filter",
flt("mim"),
filter.nfeat = 3)## Add the filter to the pipeline
graph = cat %>>%
num %>>%
filter_mimCheck to see which features are selected:
## Check
graph$train(credit_task)[[1]]$data()## Status Seniority Job.partime Records.yes
## 1: good 0.18750000 0 0
## 2: good 0.35416667 0 0
## 3: bad 0.20833333 0 1
## 4: good 0.00000000 0 0
## 5: good 0.00000000 0 0
## ---
## 4450: bad 0.02083333 0 0
## 4451: good 0.45833333 0 0
## 4452: bad 0.00000000 1 0
## 4453: good 0.00000000 0 0
## 4454: good 0.10416667 0 0Note that one impact of running this following the one-hot encoding is that the filter might select the encoding of individual levels of the original factor.
We don’t, however, know if 3 is the best subset of variables to include in the model. This is where the link between the pipeline, learner and tuning becomes very useful as we can tune this parameter (filter.nfeat) just as we would tune any hyperparameter. To do this, first create a new GraphLearner with the combination of data transformations, filter and learner:
## Step 1: Create a new graph learner that includes the filter and learner
graph = cat %>>%
num %>>%
filter_mim %>>%
lrn_rf
glrn = GraphLearner$new(graph)Now set up a new tuning grid that includes both the number of features and the number of trees in the random forest (as a reminder, type glrn$param_set to see the parameter names)
## Set up the parameter set
tune_ps = ParamSet$new(list(
ParamInt$new("mim.filter.nfeat", lower = 5, upper = 20), ##tuning the number of features selected
ParamInt$new("classif.ranger.num.trees", lower = 100, upper = 500)
))
# Check
tune_ps## <ParamSet>
## id class lower upper nlevels default value
## 1: mim.filter.nfeat ParamInt 5 20 16 <NoDefault[3]>
## 2: classif.ranger.num.trees ParamInt 100 500 401 <NoDefault[3]>Now we set up the tuning strategy (terminator, search). As there are 16 possible values for the number of features and 401 for num.trees, there are 6416 possible combinations to try. To save time in the lab, we’ll run a random search on 20 possible combinations:
## Set up new terminator
evals = trm("evals", n_evals = 20)
## Set up new tnr
tuner = tnr("random_search")## Set up new autotuner
at_rf = AutoTuner$new(learner = glrn,
resampling = rsmp("holdout"),
measure = msr_auc,
search_space = tune_ps,
terminator = evals,
tuner = tuner)And finally run. Note that this is not a very exhaustive search and you might want to increase n_evals if you have the time.
## Run the new autotuner
at_rf$train(credit_task)## INFO [19:07:31.014] [bbotk] Starting to optimize 2 parameter(s) with '<OptimizerRandomSearch>' and '<TerminatorEvals> [n_evals=20, k=0]'
## INFO [19:07:31.020] [bbotk] Evaluating 1 configuration(s)
## INFO [19:07:31.040] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:07:31.043] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.mim.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:07:31.686] [mlr3] Finished benchmark
## INFO [19:07:31.696] [bbotk] Result of batch 1:
## INFO [19:07:31.697] [bbotk] mim.filter.nfeat classif.ranger.num.trees classif.auc warnings errors
## INFO [19:07:31.697] [bbotk] 11 389 0.8236235 0 0
## INFO [19:07:31.697] [bbotk] runtime_learners uhash
## INFO [19:07:31.697] [bbotk] 0.641 cd452eb8-a533-415e-aa7d-4a6de21531ac
## INFO [19:07:31.698] [bbotk] Evaluating 1 configuration(s)
## INFO [19:07:31.718] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:07:31.721] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.mim.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:07:32.278] [mlr3] Finished benchmark
## INFO [19:07:32.289] [bbotk] Result of batch 2:
## INFO [19:07:32.290] [bbotk] mim.filter.nfeat classif.ranger.num.trees classif.auc warnings errors
## INFO [19:07:32.290] [bbotk] 16 313 0.8290275 0 0
## INFO [19:07:32.290] [bbotk] runtime_learners uhash
## INFO [19:07:32.290] [bbotk] 0.555 d5e5cb95-afb5-4d51-b720-9c2601d137a5
## INFO [19:07:32.291] [bbotk] Evaluating 1 configuration(s)
## INFO [19:07:32.311] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:07:32.314] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.mim.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:07:32.873] [mlr3] Finished benchmark
## INFO [19:07:32.884] [bbotk] Result of batch 3:
## INFO [19:07:32.884] [bbotk] mim.filter.nfeat classif.ranger.num.trees classif.auc warnings errors
## INFO [19:07:32.884] [bbotk] 18 310 0.82895 0 0
## INFO [19:07:32.884] [bbotk] runtime_learners uhash
## INFO [19:07:32.884] [bbotk] 0.555 4f076ac4-d0a1-4edb-a084-52e40d088c65
## INFO [19:07:32.886] [bbotk] Evaluating 1 configuration(s)
## INFO [19:07:32.905] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:07:32.908] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.mim.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:07:33.511] [mlr3] Finished benchmark
## INFO [19:07:33.522] [bbotk] Result of batch 4:
## INFO [19:07:33.522] [bbotk] mim.filter.nfeat classif.ranger.num.trees classif.auc warnings errors
## INFO [19:07:33.522] [bbotk] 18 343 0.8307301 0 0
## INFO [19:07:33.522] [bbotk] runtime_learners uhash
## INFO [19:07:33.522] [bbotk] 0.599 ba4f3efa-93e4-44d1-ae80-1ba6505f4f9e
## INFO [19:07:33.524] [bbotk] Evaluating 1 configuration(s)
## INFO [19:07:33.543] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:07:33.546] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.mim.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:07:34.203] [mlr3] Finished benchmark
## INFO [19:07:34.214] [bbotk] Result of batch 5:
## INFO [19:07:34.215] [bbotk] mim.filter.nfeat classif.ranger.num.trees classif.auc warnings errors
## INFO [19:07:34.215] [bbotk] 10 403 0.8283779 0 0
## INFO [19:07:34.215] [bbotk] runtime_learners uhash
## INFO [19:07:34.215] [bbotk] 0.654 00ac075a-54c3-49de-b406-e6143abadc13
## INFO [19:07:34.216] [bbotk] Evaluating 1 configuration(s)
## INFO [19:07:34.236] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:07:34.238] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.mim.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:07:34.795] [mlr3] Finished benchmark
## INFO [19:07:34.806] [bbotk] Result of batch 6:
## INFO [19:07:34.807] [bbotk] mim.filter.nfeat classif.ranger.num.trees classif.auc warnings errors
## INFO [19:07:34.807] [bbotk] 12 320 0.8257158 0 0
## INFO [19:07:34.807] [bbotk] runtime_learners uhash
## INFO [19:07:34.807] [bbotk] 0.546 eb64278d-f6ca-42d3-86df-b9fcc6d3b5a3
## INFO [19:07:34.808] [bbotk] Evaluating 1 configuration(s)
## INFO [19:07:34.828] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:07:34.830] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.mim.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:07:35.288] [mlr3] Finished benchmark
## INFO [19:07:35.299] [bbotk] Result of batch 7:
## INFO [19:07:35.299] [bbotk] mim.filter.nfeat classif.ranger.num.trees classif.auc warnings errors
## INFO [19:07:35.299] [bbotk] 6 250 0.8026364 0 0
## INFO [19:07:35.299] [bbotk] runtime_learners uhash
## INFO [19:07:35.299] [bbotk] 0.454 b0365515-f0d0-4796-96de-de33c17400b5
## INFO [19:07:35.301] [bbotk] Evaluating 1 configuration(s)
## INFO [19:07:35.320] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:07:35.323] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.mim.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:07:35.938] [mlr3] Finished benchmark
## INFO [19:07:35.950] [bbotk] Result of batch 8:
## INFO [19:07:35.950] [bbotk] mim.filter.nfeat classif.ranger.num.trees classif.auc warnings errors
## INFO [19:07:35.950] [bbotk] 13 315 0.8259597 0 0
## INFO [19:07:35.950] [bbotk] runtime_learners uhash
## INFO [19:07:35.950] [bbotk] 0.613 2687b8e6-ad7e-4eb7-922d-5e0812d3c388
## INFO [19:07:35.952] [bbotk] Evaluating 1 configuration(s)
## INFO [19:07:35.973] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:07:35.975] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.mim.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:07:36.646] [mlr3] Finished benchmark
## INFO [19:07:36.657] [bbotk] Result of batch 9:
## INFO [19:07:36.658] [bbotk] mim.filter.nfeat classif.ranger.num.trees classif.auc warnings errors
## INFO [19:07:36.658] [bbotk] 10 421 0.8277078 0 0
## INFO [19:07:36.658] [bbotk] runtime_learners uhash
## INFO [19:07:36.658] [bbotk] 0.667 7fd2973c-93ad-4bc0-b966-0d96f2557ca1
## INFO [19:07:36.659] [bbotk] Evaluating 1 configuration(s)
## INFO [19:07:36.679] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:07:36.681] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.mim.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:07:37.143] [mlr3] Finished benchmark
## INFO [19:07:37.154] [bbotk] Result of batch 10:
## INFO [19:07:37.154] [bbotk] mim.filter.nfeat classif.ranger.num.trees classif.auc warnings errors
## INFO [19:07:37.154] [bbotk] 14 239 0.8241522 0 0
## INFO [19:07:37.154] [bbotk] runtime_learners uhash
## INFO [19:07:37.154] [bbotk] 0.457 1654a5e6-f56a-40a4-b2ad-2948c46a46ba
## INFO [19:07:37.156] [bbotk] Evaluating 1 configuration(s)
## INFO [19:07:37.175] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:07:37.178] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.mim.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:07:37.900] [mlr3] Finished benchmark
## INFO [19:07:37.921] [bbotk] Result of batch 11:
## INFO [19:07:37.922] [bbotk] mim.filter.nfeat classif.ranger.num.trees classif.auc warnings errors
## INFO [19:07:37.922] [bbotk] 20 446 0.831619 0 0
## INFO [19:07:37.922] [bbotk] runtime_learners uhash
## INFO [19:07:37.922] [bbotk] 0.72 f5c37b2e-4790-49df-ac86-29f3ddd050d2
## INFO [19:07:37.924] [bbotk] Evaluating 1 configuration(s)
## INFO [19:07:37.949] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:07:37.951] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.mim.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:07:38.434] [mlr3] Finished benchmark
## INFO [19:07:38.445] [bbotk] Result of batch 12:
## INFO [19:07:38.446] [bbotk] mim.filter.nfeat classif.ranger.num.trees classif.auc warnings errors
## INFO [19:07:38.446] [bbotk] 16 257 0.8307027 0 0
## INFO [19:07:38.446] [bbotk] runtime_learners uhash
## INFO [19:07:38.446] [bbotk] 0.481 145a4ff4-f7dc-455c-b0cd-4fcc7b5bfb93
## INFO [19:07:38.447] [bbotk] Evaluating 1 configuration(s)
## INFO [19:07:38.467] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:07:38.470] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.mim.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:07:39.202] [mlr3] Finished benchmark
## INFO [19:07:39.213] [bbotk] Result of batch 13:
## INFO [19:07:39.214] [bbotk] mim.filter.nfeat classif.ranger.num.trees classif.auc warnings errors
## INFO [19:07:39.214] [bbotk] 8 444 0.8167778 0 0
## INFO [19:07:39.214] [bbotk] runtime_learners uhash
## INFO [19:07:39.214] [bbotk] 0.73 37f8b7b2-3859-43a2-bbba-1cc40af05933
## INFO [19:07:39.215] [bbotk] Evaluating 1 configuration(s)
## INFO [19:07:39.235] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:07:39.238] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.mim.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:07:39.972] [mlr3] Finished benchmark
## INFO [19:07:39.984] [bbotk] Result of batch 14:
## INFO [19:07:39.985] [bbotk] mim.filter.nfeat classif.ranger.num.trees classif.auc warnings errors
## INFO [19:07:39.985] [bbotk] 15 500 0.8193476 0 0
## INFO [19:07:39.985] [bbotk] runtime_learners uhash
## INFO [19:07:39.985] [bbotk] 0.731 951983e0-5199-426c-b8c8-8c756106e132
## INFO [19:07:39.986] [bbotk] Evaluating 1 configuration(s)
## INFO [19:07:40.008] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:07:40.010] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.mim.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:07:40.815] [mlr3] Finished benchmark
## INFO [19:07:40.826] [bbotk] Result of batch 15:
## INFO [19:07:40.827] [bbotk] mim.filter.nfeat classif.ranger.num.trees classif.auc warnings errors
## INFO [19:07:40.827] [bbotk] 5 450 0.778325 0 0
## INFO [19:07:40.827] [bbotk] runtime_learners uhash
## INFO [19:07:40.827] [bbotk] 0.802 c507aded-b525-4630-b068-5113e508ed59
## INFO [19:07:40.828] [bbotk] Evaluating 1 configuration(s)
## INFO [19:07:40.848] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:07:40.850] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.mim.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:07:41.241] [mlr3] Finished benchmark
## INFO [19:07:41.252] [bbotk] Result of batch 16:
## INFO [19:07:41.253] [bbotk] mim.filter.nfeat classif.ranger.num.trees classif.auc warnings errors
## INFO [19:07:41.253] [bbotk] 13 196 0.8263449 0 0
## INFO [19:07:41.253] [bbotk] runtime_learners uhash
## INFO [19:07:41.253] [bbotk] 0.388 20a93744-1e9f-4d41-a8d3-a551befa13b0
## INFO [19:07:41.254] [bbotk] Evaluating 1 configuration(s)
## INFO [19:07:41.275] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:07:41.278] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.mim.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:07:41.676] [mlr3] Finished benchmark
## INFO [19:07:41.687] [bbotk] Result of batch 17:
## INFO [19:07:41.688] [bbotk] mim.filter.nfeat classif.ranger.num.trees classif.auc warnings errors
## INFO [19:07:41.688] [bbotk] 15 154 0.8237237 0 0
## INFO [19:07:41.688] [bbotk] runtime_learners uhash
## INFO [19:07:41.688] [bbotk] 0.396 e818b277-45d0-4900-80c0-94ada1946a11
## INFO [19:07:41.689] [bbotk] Evaluating 1 configuration(s)
## INFO [19:07:41.710] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:07:41.713] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.mim.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:07:42.067] [mlr3] Finished benchmark
## INFO [19:07:42.079] [bbotk] Result of batch 18:
## INFO [19:07:42.080] [bbotk] mim.filter.nfeat classif.ranger.num.trees classif.auc warnings errors
## INFO [19:07:42.080] [bbotk] 13 148 0.8263836 0 0
## INFO [19:07:42.080] [bbotk] runtime_learners uhash
## INFO [19:07:42.080] [bbotk] 0.352 487073d8-9bbb-4ae1-ae7e-2bce20112f05
## INFO [19:07:42.082] [bbotk] Evaluating 1 configuration(s)
## INFO [19:07:42.104] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:07:42.106] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.mim.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:07:42.486] [mlr3] Finished benchmark
## INFO [19:07:42.497] [bbotk] Result of batch 19:
## INFO [19:07:42.497] [bbotk] mim.filter.nfeat classif.ranger.num.trees classif.auc warnings errors
## INFO [19:07:42.497] [bbotk] 14 173 0.8258799 0 0
## INFO [19:07:42.497] [bbotk] runtime_learners uhash
## INFO [19:07:42.497] [bbotk] 0.376 b930e62b-4acf-4890-99f2-416944b45059
## INFO [19:07:42.499] [bbotk] Evaluating 1 configuration(s)
## INFO [19:07:42.519] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:07:42.521] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.mim.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:07:43.195] [mlr3] Finished benchmark
## INFO [19:07:43.206] [bbotk] Result of batch 20:
## INFO [19:07:43.207] [bbotk] mim.filter.nfeat classif.ranger.num.trees classif.auc warnings errors
## INFO [19:07:43.207] [bbotk] 6 444 0.8037578 0 0
## INFO [19:07:43.207] [bbotk] runtime_learners uhash
## INFO [19:07:43.207] [bbotk] 0.671 53ad40dc-f326-4ca3-ad3a-1ec719b2b08c
## INFO [19:07:43.211] [bbotk] Finished optimizing after 20 evaluation(s)
## INFO [19:07:43.211] [bbotk] Result:
## INFO [19:07:43.212] [bbotk] mim.filter.nfeat classif.ranger.num.trees learner_param_vals x_domain
## INFO [19:07:43.212] [bbotk] 20 446 <list[12]> <list[2]>
## INFO [19:07:43.212] [bbotk] classif.auc
## INFO [19:07:43.212] [bbotk] 0.831619Let’s take a look at the output
## Check the output
at_rf$learner## <GraphLearner:imputemode.encode.imputemedian.scalerange.mim.classif.ranger>
## * Model: list
## * Parameters: imputemode.affect_columns=<Selector>,
## encode.method=treatment, encode.affect_columns=<Selector>,
## imputemedian.affect_columns=<Selector>, scalerange.lower=0,
## scalerange.upper=1, scalerange.affect_columns=<Selector>,
## mim.filter.nfeat=20, mim.threads=1, classif.ranger.mtry=3,
## classif.ranger.num.threads=1, classif.ranger.num.trees=446
## * Packages: mlr3, mlr3pipelines, stats, mlr3learners, ranger
## * Predict Types: response, [prob]
## * Feature Types: logical, integer, numeric, character, factor, ordered,
## POSIXct
## * Properties: featureless, hotstart_backward, hotstart_forward,
## importance, loglik, missings, multiclass, oob_error,
## selected_features, twoclass, weights## Check the result
at_rf$tuning_result## mim.filter.nfeat classif.ranger.num.trees learner_param_vals x_domain
## 1: 20 446 <list[12]> <list[2]>
## classif.auc
## 1: 0.831619PCA Transformation
For a final example, we’ll look at a different feature selection strategy. Rather than selecting out original features, we’ll use a PCA transformation to create new features. These are based on the original features, but a) are uncorrelated and b) try to maximize the amount of information contained in each one.
To do this, we’ll recreate our pipeline with a new operator that will carry out the PCA transformation (pca). We also add a new filter that selects the set of new features based on how much of the original variation in the dataset that they explain (we’ll start by choosing the set that explain >50%).
## Set up the PCA
pca = po("pca")
## Set up the filter
## 0.5 is 50% of the original data that is explained
filter <- po("filter", filter = flt("variance"), filter.frac = 0.5)
## Reset our graphlearner
graph <- cat %>>%
num %>>%
pca %>>%
filterCheck out the transformed data
## Check
graph$train(credit_task)[[1]]$data()## Status PC1 PC2 PC3 PC4 PC5
## 1: good 0.08389405 0.9325537 0.68336541 -0.45940509 0.091015364
## 2: good -0.47187127 0.6509299 -0.10514080 -0.43342929 -0.505434933
## 3: bad 0.79090489 -0.2069475 0.87236340 0.57590819 -0.273238044
## 4: good -1.10160699 0.3790336 -0.07681965 -0.52299916 -0.682559092
## 5: good -1.10895702 0.3560424 -0.04882622 -0.54446018 -0.677791543
## ---
## 4450: bad -0.02120892 0.9674283 -0.23883965 -0.27591297 -0.058936035
## 4451: good 0.64483970 -0.2837564 -0.33344168 -0.10041208 -0.005327027
## 4452: bad 0.47029790 -0.3014542 -0.50129179 -0.05002211 0.014579734
## 4453: good -0.99785998 0.3316809 0.86230522 -0.71513009 -0.535791580
## 4454: good 0.68975550 -0.3202094 0.57742967 -0.26860844 0.144579422
## PC6 PC7 PC8 PC9 PC10
## 1: 0.13473397 0.08509491 0.050020459 -0.28849057 -0.020567845
## 2: -0.03908715 -0.36378262 -0.102513180 0.22599631 -0.704552255
## 3: 0.11056538 0.10495689 -0.004749384 0.04459082 0.087251641
## 4: -0.02401566 -0.13541347 -0.042208806 -0.29829259 0.145612288
## 5: -0.03132757 -0.11459711 -0.029594829 0.03282090 0.305032394
## ---
## 4450: 0.12715108 -0.13219120 0.064380228 -0.17167998 -0.007770385
## 4451: 0.02496079 -0.15879510 0.023413827 -0.10121212 -0.095844480
## 4452: 0.23107589 0.88725458 0.022482381 0.33760362 0.056565550
## 4453: -0.02614890 0.09956490 -0.058768370 -0.21865450 0.178081381
## 4454: 0.05477999 0.15712194 0.013677640 -0.35503064 -0.046085943
## PC11
## 1: -0.004964123
## 2: 0.168685764
## 3: 0.046920651
## 4: 0.021609162
## 5: -0.069811420
## ---
## 4450: 0.018583438
## 4451: 0.136630579
## 4452: -0.077898577
## 4453: -0.010868937
## 4454: -0.058032799Which gives us 11 new features, called PC1, etc. Now we’ll connect this pipeline to our learner:
## Connect to the learner
graph = cat %>>%
num %>>%
pca %>>%
filter %>>%
lrn_rf
## Visualize
plot(graph)
Create new graphlearner
glrn = GraphLearner$new(graph)With all this in place, we can now tune our model. We’ll again tune the size of the hidden layer, but we’ll also use tune the number of new PC features that we use in the model by tuning the filter.frac parameter.
Set up parameter space:
## Set up
tune_ps = ParamSet$new(list(
ParamDbl$new("variance.filter.frac", lower = 0.25, upper = 0.95),
ParamInt$new("classif.ranger.num.trees", lower = 100, upper = 500)
))
## Check
tune_ps## <ParamSet>
## id class lower upper nlevels default value
## 1: variance.filter.frac ParamDbl 0.25 0.95 Inf <NoDefault[3]>
## 2: classif.ranger.num.trees ParamInt 100.00 500.00 401 <NoDefault[3]>## Same as the previous iteration
evals = trm("evals", n_evals = 20)
tuner = tnr("random_search")## Set up the autotuner
at_rf = AutoTuner$new(learner = glrn,
resampling = rsmp("holdout"),
measure = msr_auc,
search_space = tune_ps,
terminator = evals,
tuner = tuner)## Run the tuner
at_rf$train(credit_task)## INFO [19:07:44.447] [bbotk] Starting to optimize 2 parameter(s) with '<OptimizerRandomSearch>' and '<TerminatorEvals> [n_evals=20, k=0]'
## INFO [19:07:44.452] [bbotk] Evaluating 1 configuration(s)
## INFO [19:07:44.473] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:07:44.475] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:07:45.506] [mlr3] Finished benchmark
## INFO [19:07:45.517] [bbotk] Result of batch 1:
## INFO [19:07:45.517] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:07:45.517] [bbotk] 0.9048001 315 0.8121933 0 0
## INFO [19:07:45.517] [bbotk] runtime_learners uhash
## INFO [19:07:45.517] [bbotk] 1.028 e17112c1-739f-455a-81fc-5dbff8528cd4
## INFO [19:07:45.519] [bbotk] Evaluating 1 configuration(s)
## INFO [19:07:45.540] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:07:45.543] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:07:46.350] [mlr3] Finished benchmark
## INFO [19:07:46.361] [bbotk] Result of batch 2:
## INFO [19:07:46.362] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:07:46.362] [bbotk] 0.7031907 223 0.7775734 0 0
## INFO [19:07:46.362] [bbotk] runtime_learners uhash
## INFO [19:07:46.362] [bbotk] 0.805 f522a3eb-367d-4d6f-91a5-cd28a2b3eca2
## INFO [19:07:46.363] [bbotk] Evaluating 1 configuration(s)
## INFO [19:07:46.385] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:07:46.387] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:07:47.667] [mlr3] Finished benchmark
## INFO [19:07:47.678] [bbotk] Result of batch 3:
## INFO [19:07:47.679] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:07:47.679] [bbotk] 0.7135273 397 0.7810961 0 0
## INFO [19:07:47.679] [bbotk] runtime_learners uhash
## INFO [19:07:47.679] [bbotk] 1.278 3e0112e2-28aa-49ce-83ea-a168f5c0f7bb
## INFO [19:07:47.680] [bbotk] Evaluating 1 configuration(s)
## INFO [19:07:47.701] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:07:47.704] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:07:48.974] [mlr3] Finished benchmark
## INFO [19:07:48.985] [bbotk] Result of batch 4:
## INFO [19:07:48.986] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:07:48.986] [bbotk] 0.5574734 397 0.7661853 0 0
## INFO [19:07:48.986] [bbotk] runtime_learners uhash
## INFO [19:07:48.986] [bbotk] 1.267 6aeb8982-7f93-4fb6-8f26-4711d0be67c6
## INFO [19:07:48.987] [bbotk] Evaluating 1 configuration(s)
## INFO [19:07:49.009] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:07:49.011] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:07:50.146] [mlr3] Finished benchmark
## INFO [19:07:50.157] [bbotk] Result of batch 5:
## INFO [19:07:50.158] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:07:50.158] [bbotk] 0.3905746 342 0.7526431 0 0
## INFO [19:07:50.158] [bbotk] runtime_learners uhash
## INFO [19:07:50.158] [bbotk] 1.131 96e1dec5-b0f3-403e-a99d-e18aaf3f0830
## INFO [19:07:50.159] [bbotk] Evaluating 1 configuration(s)
## INFO [19:07:50.181] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:07:50.183] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:07:51.257] [mlr3] Finished benchmark
## INFO [19:07:51.268] [bbotk] Result of batch 6:
## INFO [19:07:51.269] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:07:51.269] [bbotk] 0.9412091 335 0.8201021 0 0
## INFO [19:07:51.269] [bbotk] runtime_learners uhash
## INFO [19:07:51.269] [bbotk] 1.07 e1badbf5-bdde-4ed8-bedb-49ce0b2c46ed
## INFO [19:07:51.270] [bbotk] Evaluating 1 configuration(s)
## INFO [19:07:51.291] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:07:51.294] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:07:52.521] [mlr3] Finished benchmark
## INFO [19:07:52.532] [bbotk] Result of batch 7:
## INFO [19:07:52.533] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:07:52.533] [bbotk] 0.7154053 378 0.7823022 0 0
## INFO [19:07:52.533] [bbotk] runtime_learners uhash
## INFO [19:07:52.533] [bbotk] 1.224 3e662ad6-1ff8-41d7-ace4-1e8073d7528c
## INFO [19:07:52.534] [bbotk] Evaluating 1 configuration(s)
## INFO [19:07:52.555] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:07:52.558] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:07:53.997] [mlr3] Finished benchmark
## INFO [19:07:54.008] [bbotk] Result of batch 8:
## INFO [19:07:54.009] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:07:54.009] [bbotk] 0.7591165 471 0.8068181 0 0
## INFO [19:07:54.009] [bbotk] runtime_learners uhash
## INFO [19:07:54.009] [bbotk] 1.435 5e5183a7-51b5-400a-a4b1-dab2ec303769
## INFO [19:07:54.010] [bbotk] Evaluating 1 configuration(s)
## INFO [19:07:54.032] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:07:54.034] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:07:54.727] [mlr3] Finished benchmark
## INFO [19:07:54.739] [bbotk] Result of batch 9:
## INFO [19:07:54.740] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:07:54.740] [bbotk] 0.8799517 159 0.8111066 0 0
## INFO [19:07:54.740] [bbotk] runtime_learners uhash
## INFO [19:07:54.740] [bbotk] 0.689 56b1776a-1bb7-415b-8171-0814322c94b2
## INFO [19:07:54.741] [bbotk] Evaluating 1 configuration(s)
## INFO [19:07:54.763] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:07:54.766] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:07:56.096] [mlr3] Finished benchmark
## INFO [19:07:56.108] [bbotk] Result of batch 10:
## INFO [19:07:56.109] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:07:56.109] [bbotk] 0.9130905 392 0.810278 0 0
## INFO [19:07:56.109] [bbotk] runtime_learners uhash
## INFO [19:07:56.109] [bbotk] 1.328 6dd02270-b18c-4005-a197-bc8a7e35a783
## INFO [19:07:56.110] [bbotk] Evaluating 1 configuration(s)
## INFO [19:07:56.132] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:07:56.135] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:07:56.643] [mlr3] Finished benchmark
## INFO [19:07:56.654] [bbotk] Result of batch 11:
## INFO [19:07:56.655] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:07:56.655] [bbotk] 0.9161208 113 0.8107584 0 0
## INFO [19:07:56.655] [bbotk] runtime_learners uhash
## INFO [19:07:56.655] [bbotk] 0.506 2aa57e1d-4cd4-4857-9bd1-564b34d7d390
## INFO [19:07:56.656] [bbotk] Evaluating 1 configuration(s)
## INFO [19:07:56.678] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:07:56.680] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:07:58.247] [mlr3] Finished benchmark
## INFO [19:07:58.258] [bbotk] Result of batch 12:
## INFO [19:07:58.259] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:07:58.259] [bbotk] 0.7100967 493 0.7827512 0 0
## INFO [19:07:58.259] [bbotk] runtime_learners uhash
## INFO [19:07:58.259] [bbotk] 1.563 2bd1cbdb-942e-4e6d-a933-4274f115b98a
## INFO [19:07:58.260] [bbotk] Evaluating 1 configuration(s)
## INFO [19:07:58.282] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:07:58.284] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:07:58.922] [mlr3] Finished benchmark
## INFO [19:07:58.933] [bbotk] Result of batch 13:
## INFO [19:07:58.934] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:07:58.934] [bbotk] 0.3489001 161 0.7491627 0 0
## INFO [19:07:58.934] [bbotk] runtime_learners uhash
## INFO [19:07:58.934] [bbotk] 0.634 2db3633c-69aa-4b82-854d-1ffe0ed6420c
## INFO [19:07:58.935] [bbotk] Evaluating 1 configuration(s)
## INFO [19:07:58.957] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:07:58.959] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:07:59.965] [mlr3] Finished benchmark
## INFO [19:07:59.977] [bbotk] Result of batch 14:
## INFO [19:07:59.977] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:07:59.977] [bbotk] 0.5234692 297 0.766066 0 0
## INFO [19:07:59.977] [bbotk] runtime_learners uhash
## INFO [19:07:59.977] [bbotk] 1.002 6d4ada0f-4b70-442c-a9ce-7b17f71f940e
## INFO [19:07:59.979] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:00.000] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:00.003] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:00.766] [mlr3] Finished benchmark
## INFO [19:08:00.777] [bbotk] Result of batch 15:
## INFO [19:08:00.778] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:00.778] [bbotk] 0.8988569 205 0.8082736 0 0
## INFO [19:08:00.778] [bbotk] runtime_learners uhash
## INFO [19:08:00.778] [bbotk] 0.761 45472c26-db49-4bb5-b99a-16df2fabee61
## INFO [19:08:00.779] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:00.801] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:00.803] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:01.633] [mlr3] Finished benchmark
## INFO [19:08:01.644] [bbotk] Result of batch 16:
## INFO [19:08:01.645] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:01.645] [bbotk] 0.6184057 228 0.7729617 0 0
## INFO [19:08:01.645] [bbotk] runtime_learners uhash
## INFO [19:08:01.645] [bbotk] 0.826 76483055-3e5b-45a4-80dd-27591745a3d9
## INFO [19:08:01.646] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:01.667] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:01.670] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:02.217] [mlr3] Finished benchmark
## INFO [19:08:02.228] [bbotk] Result of batch 17:
## INFO [19:08:02.228] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:02.228] [bbotk] 0.5088886 126 0.7641788 0 0
## INFO [19:08:02.228] [bbotk] runtime_learners uhash
## INFO [19:08:02.228] [bbotk] 0.543 26fda2bf-612e-4b1c-a46b-7c1674d423d8
## INFO [19:08:02.230] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:02.251] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:02.253] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:02.835] [mlr3] Finished benchmark
## INFO [19:08:02.846] [bbotk] Result of batch 18:
## INFO [19:08:02.847] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:02.847] [bbotk] 0.8746773 142 0.8065665 0 0
## INFO [19:08:02.847] [bbotk] runtime_learners uhash
## INFO [19:08:02.847] [bbotk] 0.579 17cf056b-cc3d-4c53-9b95-42454d5e2358
## INFO [19:08:02.848] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:02.870] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:02.872] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:04.364] [mlr3] Finished benchmark
## INFO [19:08:04.375] [bbotk] Result of batch 19:
## INFO [19:08:04.376] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:04.376] [bbotk] 0.2540936 482 0.7436465 0 0
## INFO [19:08:04.376] [bbotk] runtime_learners uhash
## INFO [19:08:04.376] [bbotk] 1.487 4969e355-fe9b-482f-9be4-e9cce4f2a704
## INFO [19:08:04.377] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:04.400] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:04.402] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:04.932] [mlr3] Finished benchmark
## INFO [19:08:04.943] [bbotk] Result of batch 20:
## INFO [19:08:04.944] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:04.944] [bbotk] 0.4110811 118 0.7517939 0 0
## INFO [19:08:04.944] [bbotk] runtime_learners uhash
## INFO [19:08:04.944] [bbotk] 0.526 dff75dc7-a03a-48ca-9502-f60c7eabbf08
## INFO [19:08:04.947] [bbotk] Finished optimizing after 20 evaluation(s)
## INFO [19:08:04.948] [bbotk] Result:
## INFO [19:08:04.948] [bbotk] variance.filter.frac classif.ranger.num.trees learner_param_vals x_domain
## INFO [19:08:04.948] [bbotk] 0.9412091 335 <list[11]> <list[2]>
## INFO [19:08:04.948] [bbotk] classif.auc
## INFO [19:08:04.948] [bbotk] 0.8201021When this is eventually done, inspect the output to see the final choices for the two hyperparameters, as well as the final AUC for the tuned model:
## Check
at_rf$learner## <GraphLearner:imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger>
## * Model: list
## * Parameters: imputemode.affect_columns=<Selector>,
## encode.method=treatment, encode.affect_columns=<Selector>,
## imputemedian.affect_columns=<Selector>, scalerange.lower=0,
## scalerange.upper=1, scalerange.affect_columns=<Selector>,
## variance.filter.frac=0.9412, classif.ranger.mtry=3,
## classif.ranger.num.threads=1, classif.ranger.num.trees=335
## * Packages: mlr3, mlr3pipelines, stats, mlr3learners, ranger
## * Predict Types: response, [prob]
## * Feature Types: logical, integer, numeric, character, factor, ordered,
## POSIXct
## * Properties: featureless, hotstart_backward, hotstart_forward,
## importance, loglik, missings, multiclass, oob_error,
## selected_features, twoclass, weights## Extract the tuning result
at_rf$tuning_result## variance.filter.frac classif.ranger.num.trees learner_param_vals x_domain
## 1: 0.9412091 335 <list[11]> <list[2]>
## classif.auc
## 1: 0.8201021## Note: This matches the parameter output during the at_rf$learner checkEvaluating Tuned Pipelines
While the rpevious code allows us to select the value of the hyperparameters, the performance score shown above is only based on the training set. It is calculated using only part of that training set, but is still considered to not be an independent test of predictive skill. To evaluate the model, we need to run the same nested cross-validation that we looked at in a previous lab. Fortunately, we have nearly all the elements we need to this (the pipeline / graph learner and the autotuner). We just need to do the following:
Define the inner and outer cross-validation strategies
## Set up the inner *used on the training data*
rsmp_inner <- rsmp("holdout", ratio = 0.8)
## Set up the outer *used to divide the overall data between test and training
rsmp_outer <- rsmp("cv", folds = 3)Update the autotuner
## Set up the autotuner
at_rf = AutoTuner$new(learner = glrn,
resampling = rsmp_inner,
measure = msr_auc,
search_space = tune_ps,
terminator = evals,
tuner = tuner)Run
## Re-run the learner
rr_rf <- resample(credit_task, at_rf, rsmp_outer,
store_models = TRUE)## INFO [19:08:06.526] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger.tuned' on task 'credit' (iter 1/3)
## INFO [19:08:06.581] [bbotk] Starting to optimize 2 parameter(s) with '<OptimizerRandomSearch>' and '<TerminatorEvals> [n_evals=20, k=0]'
## INFO [19:08:06.585] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:06.607] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:06.610] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:07.588] [mlr3] Finished benchmark
## INFO [19:08:07.598] [bbotk] Result of batch 1:
## INFO [19:08:07.599] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:07.599] [bbotk] 0.4308475 253 0.7319066 0 0
## INFO [19:08:07.599] [bbotk] runtime_learners uhash
## INFO [19:08:07.599] [bbotk] 0.976 a610d0bd-361c-4f39-a4d7-763310fc4263
## INFO [19:08:07.600] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:07.621] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:07.624] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:08.232] [mlr3] Finished benchmark
## INFO [19:08:08.244] [bbotk] Result of batch 2:
## INFO [19:08:08.244] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:08.244] [bbotk] 0.3947258 194 0.7302061 0 0
## INFO [19:08:08.244] [bbotk] runtime_learners uhash
## INFO [19:08:08.244] [bbotk] 0.606 bb7a33a3-c704-4cfa-91d6-2f32414dddb7
## INFO [19:08:08.246] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:08.268] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:08.270] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:09.182] [mlr3] Finished benchmark
## INFO [19:08:09.193] [bbotk] Result of batch 3:
## INFO [19:08:09.193] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:09.193] [bbotk] 0.6328068 359 0.7527546 0 0
## INFO [19:08:09.193] [bbotk] runtime_learners uhash
## INFO [19:08:09.193] [bbotk] 0.908 e53e4ad4-3d9d-4b27-8e64-923997bf8133
## INFO [19:08:09.195] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:09.216] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:09.218] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:09.802] [mlr3] Finished benchmark
## INFO [19:08:09.813] [bbotk] Result of batch 4:
## INFO [19:08:09.814] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:09.814] [bbotk] 0.7970826 201 0.7844967 0 0
## INFO [19:08:09.814] [bbotk] runtime_learners uhash
## INFO [19:08:09.814] [bbotk] 0.58 e0b717fa-c116-490d-be0b-a893705d2b58
## INFO [19:08:09.815] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:09.836] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:09.839] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:10.545] [mlr3] Finished benchmark
## INFO [19:08:10.556] [bbotk] Result of batch 5:
## INFO [19:08:10.557] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:10.557] [bbotk] 0.8056766 260 0.7833216 0 0
## INFO [19:08:10.557] [bbotk] runtime_learners uhash
## INFO [19:08:10.557] [bbotk] 0.703 323d1ac0-a7b2-4e5a-a73c-67d9bfc3df4d
## INFO [19:08:10.558] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:10.579] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:10.582] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:11.444] [mlr3] Finished benchmark
## INFO [19:08:11.455] [bbotk] Result of batch 6:
## INFO [19:08:11.456] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:11.456] [bbotk] 0.484498 344 0.7368698 0 0
## INFO [19:08:11.456] [bbotk] runtime_learners uhash
## INFO [19:08:11.456] [bbotk] 0.86 5efb9367-a3f2-4907-bf25-31f97005a0b4
## INFO [19:08:11.457] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:11.479] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:11.481] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:11.921] [mlr3] Finished benchmark
## INFO [19:08:11.932] [bbotk] Result of batch 7:
## INFO [19:08:11.933] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:11.933] [bbotk] 0.28061 119 0.7267637 0 0
## INFO [19:08:11.933] [bbotk] runtime_learners uhash
## INFO [19:08:11.933] [bbotk] 0.437 daa406cc-0e4c-4ef9-8176-126471cffb7e
## INFO [19:08:11.934] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:11.956] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:11.958] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:12.971] [mlr3] Finished benchmark
## INFO [19:08:12.982] [bbotk] Result of batch 8:
## INFO [19:08:12.983] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:12.983] [bbotk] 0.8288437 428 0.7827271 0 0
## INFO [19:08:12.983] [bbotk] runtime_learners uhash
## INFO [19:08:12.983] [bbotk] 1.01 e65bd98d-7395-4001-a825-b49e5bf6a132
## INFO [19:08:12.984] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:13.006] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:13.008] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:13.589] [mlr3] Finished benchmark
## INFO [19:08:13.600] [bbotk] Result of batch 9:
## INFO [19:08:13.601] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:13.601] [bbotk] 0.2909721 195 0.7230102 0 0
## INFO [19:08:13.601] [bbotk] runtime_learners uhash
## INFO [19:08:13.601] [bbotk] 0.578 9ebddaa2-85a3-4976-b305-f85f15877097
## INFO [19:08:13.602] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:13.623] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:13.626] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:14.081] [mlr3] Finished benchmark
## INFO [19:08:14.092] [bbotk] Result of batch 10:
## INFO [19:08:14.092] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:14.092] [bbotk] 0.2659589 128 0.7290241 0 0
## INFO [19:08:14.092] [bbotk] runtime_learners uhash
## INFO [19:08:14.092] [bbotk] 0.451 f8d0c985-bd96-47b6-84bf-957199fb8ad6
## INFO [19:08:14.094] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:14.114] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:14.117] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:14.806] [mlr3] Finished benchmark
## INFO [19:08:14.817] [bbotk] Result of batch 11:
## INFO [19:08:14.818] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:14.818] [bbotk] 0.6394162 251 0.7535426 0 0
## INFO [19:08:14.818] [bbotk] runtime_learners uhash
## INFO [19:08:14.818] [bbotk] 0.687 b7a17c89-24e3-443c-90e7-dd718aeb4f0b
## INFO [19:08:14.819] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:14.840] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:14.843] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:16.008] [mlr3] Finished benchmark
## INFO [19:08:16.019] [bbotk] Result of batch 12:
## INFO [19:08:16.020] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:16.020] [bbotk] 0.4384502 492 0.734326 0 0
## INFO [19:08:16.020] [bbotk] runtime_learners uhash
## INFO [19:08:16.020] [bbotk] 1.162 e0784ad7-d694-42d8-9454-52935538cc45
## INFO [19:08:16.021] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:16.042] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:16.045] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:17.016] [mlr3] Finished benchmark
## INFO [19:08:17.027] [bbotk] Result of batch 13:
## INFO [19:08:17.028] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:17.028] [bbotk] 0.4533942 399 0.732031 0 0
## INFO [19:08:17.028] [bbotk] runtime_learners uhash
## INFO [19:08:17.028] [bbotk] 0.969 978e7223-5ed6-4ede-bb83-b50db4340414
## INFO [19:08:17.029] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:17.050] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:17.053] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:18.150] [mlr3] Finished benchmark
## INFO [19:08:18.161] [bbotk] Result of batch 14:
## INFO [19:08:18.162] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:18.162] [bbotk] 0.8626887 472 0.7935659 0 0
## INFO [19:08:18.162] [bbotk] runtime_learners uhash
## INFO [19:08:18.162] [bbotk] 1.093 f3440d71-e23e-440c-a80a-0a0feddca9e7
## INFO [19:08:18.163] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:18.184] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:18.186] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:18.785] [mlr3] Finished benchmark
## INFO [19:08:18.796] [bbotk] Result of batch 15:
## INFO [19:08:18.797] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:18.797] [bbotk] 0.4631322 202 0.736773 0 0
## INFO [19:08:18.797] [bbotk] runtime_learners uhash
## INFO [19:08:18.797] [bbotk] 0.595 46516ce9-fca8-4243-80da-f8769b3a00ce
## INFO [19:08:18.798] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:18.819] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:18.821] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:19.460] [mlr3] Finished benchmark
## INFO [19:08:19.471] [bbotk] Result of batch 16:
## INFO [19:08:19.471] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:19.471] [bbotk] 0.916447 236 0.7974922 0 0
## INFO [19:08:19.471] [bbotk] runtime_learners uhash
## INFO [19:08:19.471] [bbotk] 0.634 92790eb2-8ce0-4f1f-a421-f51cc88e577b
## INFO [19:08:19.473] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:19.493] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:19.496] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:20.056] [mlr3] Finished benchmark
## INFO [19:08:20.067] [bbotk] Result of batch 17:
## INFO [19:08:20.068] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:20.068] [bbotk] 0.527746 128 0.7379758 0 0
## INFO [19:08:20.068] [bbotk] runtime_learners uhash
## INFO [19:08:20.068] [bbotk] 0.558 f9ad1c10-e587-4153-843a-1e8f7c51308e
## INFO [19:08:20.069] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:20.090] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:20.093] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:20.948] [mlr3] Finished benchmark
## INFO [19:08:20.960] [bbotk] Result of batch 18:
## INFO [19:08:20.961] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:20.961] [bbotk] 0.3165509 327 0.7291001 0 0
## INFO [19:08:20.961] [bbotk] runtime_learners uhash
## INFO [19:08:20.961] [bbotk] 0.853 4376859c-b769-4465-aaf1-27541a6b7f76
## INFO [19:08:20.962] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:20.986] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:20.989] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:22.020] [mlr3] Finished benchmark
## INFO [19:08:22.031] [bbotk] Result of batch 19:
## INFO [19:08:22.032] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:22.032] [bbotk] 0.9005537 433 0.7957779 0 0
## INFO [19:08:22.032] [bbotk] runtime_learners uhash
## INFO [19:08:22.032] [bbotk] 1.029 e30c9dc0-8ea3-4fa6-8d15-a4e3e5806704
## INFO [19:08:22.033] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:22.055] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:22.057] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:22.648] [mlr3] Finished benchmark
## INFO [19:08:22.659] [bbotk] Result of batch 20:
## INFO [19:08:22.659] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:22.659] [bbotk] 0.4078721 203 0.7261969 0 0
## INFO [19:08:22.659] [bbotk] runtime_learners uhash
## INFO [19:08:22.659] [bbotk] 0.586 bb3f694e-e576-47eb-b6d5-f26406ab6feb
## INFO [19:08:22.663] [bbotk] Finished optimizing after 20 evaluation(s)
## INFO [19:08:22.664] [bbotk] Result:
## INFO [19:08:22.664] [bbotk] variance.filter.frac classif.ranger.num.trees learner_param_vals x_domain
## INFO [19:08:22.664] [bbotk] 0.916447 236 <list[11]> <list[2]>
## INFO [19:08:22.664] [bbotk] classif.auc
## INFO [19:08:22.664] [bbotk] 0.7974922
## INFO [19:08:23.527] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger.tuned' on task 'credit' (iter 2/3)
## INFO [19:08:23.580] [bbotk] Starting to optimize 2 parameter(s) with '<OptimizerRandomSearch>' and '<TerminatorEvals> [n_evals=20, k=0]'
## INFO [19:08:23.585] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:23.606] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:23.608] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:24.530] [mlr3] Finished benchmark
## INFO [19:08:24.552] [bbotk] Result of batch 1:
## INFO [19:08:24.553] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:24.553] [bbotk] 0.6724712 384 0.7509711 0 0
## INFO [19:08:24.553] [bbotk] runtime_learners uhash
## INFO [19:08:24.553] [bbotk] 0.919 501de13f-cbdd-4cd1-846f-134a74512667
## INFO [19:08:24.556] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:24.578] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:24.580] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:25.467] [mlr3] Finished benchmark
## INFO [19:08:25.478] [bbotk] Result of batch 2:
## INFO [19:08:25.479] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:25.479] [bbotk] 0.5494267 366 0.7448391 0 0
## INFO [19:08:25.479] [bbotk] runtime_learners uhash
## INFO [19:08:25.479] [bbotk] 0.884 8b6a9d56-29ed-48a2-8d79-85a641b880e3
## INFO [19:08:25.480] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:25.501] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:25.504] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:26.305] [mlr3] Finished benchmark
## INFO [19:08:26.316] [bbotk] Result of batch 3:
## INFO [19:08:26.317] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:26.317] [bbotk] 0.3948031 314 0.7270394 0 0
## INFO [19:08:26.317] [bbotk] runtime_learners uhash
## INFO [19:08:26.317] [bbotk] 0.798 30e933e5-43b8-4964-a08a-0baedacb8534
## INFO [19:08:26.318] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:26.340] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:26.342] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:27.308] [mlr3] Finished benchmark
## INFO [19:08:27.320] [bbotk] Result of batch 4:
## INFO [19:08:27.320] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:27.320] [bbotk] 0.5983852 411 0.7437569 0 0
## INFO [19:08:27.320] [bbotk] runtime_learners uhash
## INFO [19:08:27.320] [bbotk] 0.962 42ee37aa-b2fd-4866-8273-6abdd68e76b0
## INFO [19:08:27.322] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:27.343] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:27.346] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:28.131] [mlr3] Finished benchmark
## INFO [19:08:28.143] [bbotk] Result of batch 5:
## INFO [19:08:28.143] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:28.143] [bbotk] 0.259048 312 0.7132214 0 0
## INFO [19:08:28.143] [bbotk] runtime_learners uhash
## INFO [19:08:28.143] [bbotk] 0.783 219b4ee0-163b-4592-abe0-173d0f5c7e75
## INFO [19:08:28.145] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:28.166] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:28.169] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:28.622] [mlr3] Finished benchmark
## INFO [19:08:28.633] [bbotk] Result of batch 6:
## INFO [19:08:28.634] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:28.634] [bbotk] 0.8957311 138 0.7925153 0 0
## INFO [19:08:28.634] [bbotk] runtime_learners uhash
## INFO [19:08:28.634] [bbotk] 0.45 cca2d835-ccc1-4bc5-8f61-128b79cb145a
## INFO [19:08:28.636] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:28.657] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:28.659] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:29.544] [mlr3] Finished benchmark
## INFO [19:08:29.555] [bbotk] Result of batch 7:
## INFO [19:08:29.555] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:29.555] [bbotk] 0.5308121 359 0.7426193 0 0
## INFO [19:08:29.555] [bbotk] runtime_learners uhash
## INFO [19:08:29.555] [bbotk] 0.881 879248ae-4a09-4fec-9315-d9b4b6d8f455
## INFO [19:08:29.557] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:29.579] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:29.581] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:30.077] [mlr3] Finished benchmark
## INFO [19:08:30.088] [bbotk] Result of batch 8:
## INFO [19:08:30.089] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:30.089] [bbotk] 0.5262019 161 0.7426748 0 0
## INFO [19:08:30.089] [bbotk] runtime_learners uhash
## INFO [19:08:30.089] [bbotk] 0.492 8251834c-b237-46bf-9031-1951a4245efb
## INFO [19:08:30.090] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:30.111] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:30.114] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:30.744] [mlr3] Finished benchmark
## INFO [19:08:30.755] [bbotk] Result of batch 9:
## INFO [19:08:30.756] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:30.756] [bbotk] 0.2505281 233 0.7164262 0 0
## INFO [19:08:30.756] [bbotk] runtime_learners uhash
## INFO [19:08:30.756] [bbotk] 0.628 b95951b9-56ae-44e4-93fd-3d870d74e41a
## INFO [19:08:30.757] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:30.779] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:30.781] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:31.281] [mlr3] Finished benchmark
## INFO [19:08:31.293] [bbotk] Result of batch 10:
## INFO [19:08:31.293] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:31.293] [bbotk] 0.6576528 156 0.7490566 0 0
## INFO [19:08:31.293] [bbotk] runtime_learners uhash
## INFO [19:08:31.293] [bbotk] 0.497 a4b7276c-79e6-4235-8c6a-bbf81090f525
## INFO [19:08:31.295] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:31.316] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:31.319] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:32.137] [mlr3] Finished benchmark
## INFO [19:08:32.148] [bbotk] Result of batch 11:
## INFO [19:08:32.148] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:32.148] [bbotk] 0.8787723 327 0.7812777 0 0
## INFO [19:08:32.148] [bbotk] runtime_learners uhash
## INFO [19:08:32.148] [bbotk] 0.814 90ca1028-d501-42ff-acdc-207be04a8f2b
## INFO [19:08:32.150] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:32.171] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:32.174] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:32.767] [mlr3] Finished benchmark
## INFO [19:08:32.778] [bbotk] Result of batch 12:
## INFO [19:08:32.779] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:32.779] [bbotk] 0.8492999 205 0.7745699 0 0
## INFO [19:08:32.779] [bbotk] runtime_learners uhash
## INFO [19:08:32.779] [bbotk] 0.591 00cbaf4d-d514-4bec-a83e-9357ab7f7765
## INFO [19:08:32.780] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:32.802] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:32.804] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:33.914] [mlr3] Finished benchmark
## INFO [19:08:33.925] [bbotk] Result of batch 13:
## INFO [19:08:33.926] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:33.926] [bbotk] 0.557352 485 0.7449917 0 0
## INFO [19:08:33.926] [bbotk] runtime_learners uhash
## INFO [19:08:33.926] [bbotk] 1.106 dd452b8b-ed97-4a89-a600-2326b114b86c
## INFO [19:08:33.927] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:33.961] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:33.964] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:34.602] [mlr3] Finished benchmark
## INFO [19:08:34.613] [bbotk] Result of batch 14:
## INFO [19:08:34.614] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:34.614] [bbotk] 0.5323667 237 0.7452691 0 0
## INFO [19:08:34.614] [bbotk] runtime_learners uhash
## INFO [19:08:34.614] [bbotk] 0.635 363206f3-11b7-48d5-a68b-74e9be09adb2
## INFO [19:08:34.615] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:34.636] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:34.639] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:35.201] [mlr3] Finished benchmark
## INFO [19:08:35.212] [bbotk] Result of batch 15:
## INFO [19:08:35.212] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:35.212] [bbotk] 0.6013207 189 0.7406493 0 0
## INFO [19:08:35.212] [bbotk] runtime_learners uhash
## INFO [19:08:35.212] [bbotk] 0.559 a2e99281-17e1-4605-9708-fd9384a9c340
## INFO [19:08:35.214] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:35.235] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:35.237] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:35.668] [mlr3] Finished benchmark
## INFO [19:08:35.679] [bbotk] Result of batch 16:
## INFO [19:08:35.680] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:35.680] [bbotk] 0.6570323 126 0.7492508 0 0
## INFO [19:08:35.680] [bbotk] runtime_learners uhash
## INFO [19:08:35.680] [bbotk] 0.428 ce6206ac-d1c7-4df6-a6d4-fea31286f448
## INFO [19:08:35.682] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:35.703] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:35.705] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:36.246] [mlr3] Finished benchmark
## INFO [19:08:36.258] [bbotk] Result of batch 17:
## INFO [19:08:36.258] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:36.258] [bbotk] 0.5319577 180 0.7459351 0 0
## INFO [19:08:36.258] [bbotk] runtime_learners uhash
## INFO [19:08:36.258] [bbotk] 0.538 aee6aaab-8505-4646-b0bc-80adaf552137
## INFO [19:08:36.260] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:36.281] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:36.284] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:36.852] [mlr3] Finished benchmark
## INFO [19:08:36.864] [bbotk] Result of batch 18:
## INFO [19:08:36.865] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:36.865] [bbotk] 0.8123645 143 0.7805216 0 0
## INFO [19:08:36.865] [bbotk] runtime_learners uhash
## INFO [19:08:36.865] [bbotk] 0.565 95666db2-7777-47fd-b9c4-ca0f61bf0edf
## INFO [19:08:36.866] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:36.888] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:36.890] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:38.003] [mlr3] Finished benchmark
## INFO [19:08:38.014] [bbotk] Result of batch 19:
## INFO [19:08:38.015] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:38.015] [bbotk] 0.4928661 482 0.7319784 0 0
## INFO [19:08:38.015] [bbotk] runtime_learners uhash
## INFO [19:08:38.015] [bbotk] 1.11 e490a9be-4270-43a1-85b7-6635e9226607
## INFO [19:08:38.016] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:38.038] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:38.041] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:38.944] [mlr3] Finished benchmark
## INFO [19:08:38.955] [bbotk] Result of batch 20:
## INFO [19:08:38.956] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:38.956] [bbotk] 0.4764227 356 0.729495 0 0
## INFO [19:08:38.956] [bbotk] runtime_learners uhash
## INFO [19:08:38.956] [bbotk] 0.901 a16a5622-8a72-4444-8563-0de72db05bc6
## INFO [19:08:38.960] [bbotk] Finished optimizing after 20 evaluation(s)
## INFO [19:08:38.960] [bbotk] Result:
## INFO [19:08:38.960] [bbotk] variance.filter.frac classif.ranger.num.trees learner_param_vals x_domain
## INFO [19:08:38.960] [bbotk] 0.8957311 138 <list[11]> <list[2]>
## INFO [19:08:38.960] [bbotk] classif.auc
## INFO [19:08:38.960] [bbotk] 0.7925153
## INFO [19:08:39.553] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger.tuned' on task 'credit' (iter 3/3)
## INFO [19:08:39.605] [bbotk] Starting to optimize 2 parameter(s) with '<OptimizerRandomSearch>' and '<TerminatorEvals> [n_evals=20, k=0]'
## INFO [19:08:39.610] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:39.632] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:39.634] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:40.188] [mlr3] Finished benchmark
## INFO [19:08:40.198] [bbotk] Result of batch 1:
## INFO [19:08:40.199] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:40.199] [bbotk] 0.256065 170 0.7349989 0 0
## INFO [19:08:40.199] [bbotk] runtime_learners uhash
## INFO [19:08:40.199] [bbotk] 0.55 479701fe-cdfe-48c8-97d2-4756af117d3a
## INFO [19:08:40.200] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:40.222] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:40.225] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:41.062] [mlr3] Finished benchmark
## INFO [19:08:41.085] [bbotk] Result of batch 2:
## INFO [19:08:41.086] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:41.086] [bbotk] 0.3005308 331 0.7394665 0 0
## INFO [19:08:41.086] [bbotk] runtime_learners uhash
## INFO [19:08:41.086] [bbotk] 0.835 2373f4d3-a943-41bf-8c47-ec0704fc3ad5
## INFO [19:08:41.088] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:41.127] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:41.129] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:41.926] [mlr3] Finished benchmark
## INFO [19:08:41.937] [bbotk] Result of batch 3:
## INFO [19:08:41.938] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:41.938] [bbotk] 0.2769361 313 0.7317951 0 0
## INFO [19:08:41.938] [bbotk] runtime_learners uhash
## INFO [19:08:41.938] [bbotk] 0.793 0c5695f4-f858-4141-938d-c0780290602f
## INFO [19:08:41.939] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:41.961] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:41.964] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:42.835] [mlr3] Finished benchmark
## INFO [19:08:42.846] [bbotk] Result of batch 4:
## INFO [19:08:42.847] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:42.847] [bbotk] 0.8770253 334 0.8021897 0 0
## INFO [19:08:42.847] [bbotk] runtime_learners uhash
## INFO [19:08:42.847] [bbotk] 0.867 d3884aff-2b5d-4473-bb6f-d8a457c4266c
## INFO [19:08:42.848] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:42.870] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:42.872] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:43.441] [mlr3] Finished benchmark
## INFO [19:08:43.452] [bbotk] Result of batch 5:
## INFO [19:08:43.453] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:43.453] [bbotk] 0.2587477 195 0.7365126 0 0
## INFO [19:08:43.453] [bbotk] runtime_learners uhash
## INFO [19:08:43.453] [bbotk] 0.565 9870efa1-0fb5-4532-bc31-d3e93293cc55
## INFO [19:08:43.454] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:43.476] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:43.478] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:44.502] [mlr3] Finished benchmark
## INFO [19:08:44.513] [bbotk] Result of batch 6:
## INFO [19:08:44.514] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:44.514] [bbotk] 0.6595594 411 0.7682563 0 0
## INFO [19:08:44.514] [bbotk] runtime_learners uhash
## INFO [19:08:44.514] [bbotk] 1.02 d8430d61-2205-4e72-8c0c-1f61b7081952
## INFO [19:08:44.515] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:44.537] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:44.539] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:45.591] [mlr3] Finished benchmark
## INFO [19:08:45.602] [bbotk] Result of batch 7:
## INFO [19:08:45.603] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:45.603] [bbotk] 0.7348571 424 0.7674774 0 0
## INFO [19:08:45.603] [bbotk] runtime_learners uhash
## INFO [19:08:45.603] [bbotk] 1.049 12db5ed0-9890-475d-a068-b3ab811a1dad
## INFO [19:08:45.604] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:45.626] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:45.629] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:46.112] [mlr3] Finished benchmark
## INFO [19:08:46.123] [bbotk] Result of batch 8:
## INFO [19:08:46.123] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:46.123] [bbotk] 0.9100892 150 0.8044823 0 0
## INFO [19:08:46.123] [bbotk] runtime_learners uhash
## INFO [19:08:46.123] [bbotk] 0.48 93c186f4-046c-4e80-879b-b647ef44af25
## INFO [19:08:46.125] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:46.146] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:46.149] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:46.862] [mlr3] Finished benchmark
## INFO [19:08:46.873] [bbotk] Result of batch 9:
## INFO [19:08:46.874] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:46.874] [bbotk] 0.4493627 249 0.7462415 0 0
## INFO [19:08:46.874] [bbotk] runtime_learners uhash
## INFO [19:08:46.874] [bbotk] 0.71 684a7481-5984-4b9e-872f-2854c8b24785
## INFO [19:08:46.875] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:46.897] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:46.900] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:48.074] [mlr3] Finished benchmark
## INFO [19:08:48.085] [bbotk] Result of batch 10:
## INFO [19:08:48.086] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:48.086] [bbotk] 0.7944389 496 0.8032772 0 0
## INFO [19:08:48.086] [bbotk] runtime_learners uhash
## INFO [19:08:48.086] [bbotk] 1.171 3ec8d195-4cea-44dc-8ddf-36556713ba23
## INFO [19:08:48.088] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:48.109] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:48.112] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:48.762] [mlr3] Finished benchmark
## INFO [19:08:48.773] [bbotk] Result of batch 11:
## INFO [19:08:48.774] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:48.774] [bbotk] 0.7582698 235 0.7980895 0 0
## INFO [19:08:48.774] [bbotk] runtime_learners uhash
## INFO [19:08:48.774] [bbotk] 0.646 6e5d2d4a-d524-4e3e-9be3-507650535b70
## INFO [19:08:48.775] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:48.796] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:48.799] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:49.555] [mlr3] Finished benchmark
## INFO [19:08:49.567] [bbotk] Result of batch 12:
## INFO [19:08:49.567] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:49.567] [bbotk] 0.3633404 274 0.7429642 0 0
## INFO [19:08:49.567] [bbotk] runtime_learners uhash
## INFO [19:08:49.567] [bbotk] 0.754 f05d4263-e3cf-4ed8-83f5-6765a56c0df8
## INFO [19:08:49.569] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:49.591] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:49.593] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:50.316] [mlr3] Finished benchmark
## INFO [19:08:50.327] [bbotk] Result of batch 13:
## INFO [19:08:50.328] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:50.328] [bbotk] 0.4357736 255 0.7400985 0 0
## INFO [19:08:50.328] [bbotk] runtime_learners uhash
## INFO [19:08:50.328] [bbotk] 0.72 2206a924-3ee4-47c6-a68b-a3a3dd0d88a4
## INFO [19:08:50.329] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:50.352] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:50.355] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:51.312] [mlr3] Finished benchmark
## INFO [19:08:51.323] [bbotk] Result of batch 14:
## INFO [19:08:51.324] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:51.324] [bbotk] 0.3232822 377 0.741377 0 0
## INFO [19:08:51.324] [bbotk] runtime_learners uhash
## INFO [19:08:51.324] [bbotk] 0.952 b4094778-4ae9-4d5a-9b32-447e6c356904
## INFO [19:08:51.325] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:51.347] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:51.349] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:51.947] [mlr3] Finished benchmark
## INFO [19:08:51.960] [bbotk] Result of batch 15:
## INFO [19:08:51.961] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:51.961] [bbotk] 0.6162626 194 0.7631126 0 0
## INFO [19:08:51.961] [bbotk] runtime_learners uhash
## INFO [19:08:51.961] [bbotk] 0.595 db21e050-9c6e-4c0d-858c-5ad05c736fd4
## INFO [19:08:51.962] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:51.985] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:51.987] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:52.763] [mlr3] Finished benchmark
## INFO [19:08:52.781] [bbotk] Result of batch 16:
## INFO [19:08:52.782] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:52.782] [bbotk] 0.7030181 295 0.765273 0 0
## INFO [19:08:52.782] [bbotk] runtime_learners uhash
## INFO [19:08:52.782] [bbotk] 0.773 33670396-5a49-4a19-8838-7ae0ee2d05f9
## INFO [19:08:52.785] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:52.816] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:52.819] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:53.755] [mlr3] Finished benchmark
## INFO [19:08:53.766] [bbotk] Result of batch 17:
## INFO [19:08:53.766] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:53.766] [bbotk] 0.7304223 374 0.7635094 0 0
## INFO [19:08:53.766] [bbotk] runtime_learners uhash
## INFO [19:08:53.766] [bbotk] 0.932 0cde4fbf-7808-46f3-8ff7-c64f1e2f811a
## INFO [19:08:53.768] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:53.789] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:53.792] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:54.534] [mlr3] Finished benchmark
## INFO [19:08:54.545] [bbotk] Result of batch 18:
## INFO [19:08:54.546] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:54.546] [bbotk] 0.9019294 276 0.8113454 0 0
## INFO [19:08:54.546] [bbotk] runtime_learners uhash
## INFO [19:08:54.546] [bbotk] 0.739 4390aba7-411b-488a-bef9-0df57b2e6535
## INFO [19:08:54.547] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:54.569] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:54.571] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:55.743] [mlr3] Finished benchmark
## INFO [19:08:55.754] [bbotk] Result of batch 19:
## INFO [19:08:55.755] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:55.755] [bbotk] 0.506491 487 0.7455213 0 0
## INFO [19:08:55.755] [bbotk] runtime_learners uhash
## INFO [19:08:55.755] [bbotk] 1.169 3ff74dbe-cf70-455b-94e8-d588f796890a
## INFO [19:08:55.756] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:55.778] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:55.781] [mlr3] Applying learner 'imputemode.encode.imputemedian.scalerange.pca.variance.classif.ranger' on task 'credit' (iter 1/1)
## INFO [19:08:56.388] [mlr3] Finished benchmark
## INFO [19:08:56.399] [bbotk] Result of batch 20:
## INFO [19:08:56.400] [bbotk] variance.filter.frac classif.ranger.num.trees classif.auc warnings errors
## INFO [19:08:56.400] [bbotk] 0.7702765 204 0.7953854 0 0
## INFO [19:08:56.400] [bbotk] runtime_learners uhash
## INFO [19:08:56.400] [bbotk] 0.604 81302df6-3397-4620-81c5-f859588ab459
## INFO [19:08:56.403] [bbotk] Finished optimizing after 20 evaluation(s)
## INFO [19:08:56.404] [bbotk] Result:
## INFO [19:08:56.404] [bbotk] variance.filter.frac classif.ranger.num.trees learner_param_vals x_domain
## INFO [19:08:56.404] [bbotk] 0.9019294 276 <list[11]> <list[2]>
## INFO [19:08:56.404] [bbotk] classif.auc
## INFO [19:08:56.404] [bbotk] 0.8113454Print the final set of results
## Extract the tuning results
extract_inner_tuning_results(rr_rf)[,
c("variance.filter.frac",
"classif.ranger.num.trees",
"classif.auc")]## variance.filter.frac classif.ranger.num.trees classif.auc
## 1: 0.9164470 236 0.7974922
## 2: 0.8957311 138 0.7925153
## 3: 0.9019294 276 0.8113454## Check the overall model results (outer tuning)
## Test Data AUC
rr_rf$score(msr_auc)[ , list(classif.auc)]## classif.auc
## 1: 0.8259100
## 2: 0.7985608
## 3: 0.8117100
Exercise
For the exercise we will once again use the data from the Sonar.csv file to model types of object (rocks ‘R’ or mines ‘M’) using the values of a set of frequency bands. The goal of the exercise is to build the best predictive model for predicting these data, and you are free to choose any of the algorithms/learners we have previously looked at. You should use the mlr3 framework to setup, train and test your model. You will need to choose a cross-validation strategy and calculate the AUC to assess the model.
As the data has a large number of features, you should build a pipeline to reduce the number of features using one of the two filter examples (mutual information or PCA) from the lab. Note that there are no categorical features so you can skip those steps. You should then tune both the filter and at least one hyperparameter of your model (if you are not sure about this, please ask!)
Your answer should consist of the following
- A description of your pipeline (this can include a figure showing the steps)
- The values you obtained for the number of features and the selected hyperparameter
- The cross-validated AUC
Data Read
## Read in the sonar data
sr = read.csv("../datafiles/sonar.csv")
## Check
str(sr)## 'data.frame': 208 obs. of 61 variables:
## $ V1 : num 0.02 0.0453 0.0262 0.01 0.0762 0.0286 0.0317 0.0519 0.0223 0.0164 ...
## $ V2 : num 0.0371 0.0523 0.0582 0.0171 0.0666 0.0453 0.0956 0.0548 0.0375 0.0173 ...
## $ V3 : num 0.0428 0.0843 0.1099 0.0623 0.0481 ...
## $ V4 : num 0.0207 0.0689 0.1083 0.0205 0.0394 ...
## $ V5 : num 0.0954 0.1183 0.0974 0.0205 0.059 ...
## $ V6 : num 0.0986 0.2583 0.228 0.0368 0.0649 ...
## $ V7 : num 0.154 0.216 0.243 0.11 0.121 ...
## $ V8 : num 0.16 0.348 0.377 0.128 0.247 ...
## $ V9 : num 0.3109 0.3337 0.5598 0.0598 0.3564 ...
## $ V10 : num 0.211 0.287 0.619 0.126 0.446 ...
## $ V11 : num 0.1609 0.4918 0.6333 0.0881 0.4152 ...
## $ V12 : num 0.158 0.655 0.706 0.199 0.395 ...
## $ V13 : num 0.2238 0.6919 0.5544 0.0184 0.4256 ...
## $ V14 : num 0.0645 0.7797 0.532 0.2261 0.4135 ...
## $ V15 : num 0.066 0.746 0.648 0.173 0.453 ...
## $ V16 : num 0.227 0.944 0.693 0.213 0.533 ...
## $ V17 : num 0.31 1 0.6759 0.0693 0.7306 ...
## $ V18 : num 0.3 0.887 0.755 0.228 0.619 ...
## $ V19 : num 0.508 0.802 0.893 0.406 0.203 ...
## $ V20 : num 0.48 0.782 0.862 0.397 0.464 ...
## $ V21 : num 0.578 0.521 0.797 0.274 0.415 ...
## $ V22 : num 0.507 0.405 0.674 0.369 0.429 ...
## $ V23 : num 0.433 0.396 0.429 0.556 0.573 ...
## $ V24 : num 0.555 0.391 0.365 0.485 0.54 ...
## $ V25 : num 0.671 0.325 0.533 0.314 0.316 ...
## $ V26 : num 0.641 0.32 0.241 0.533 0.229 ...
## $ V27 : num 0.71 0.327 0.507 0.526 0.7 ...
## $ V28 : num 0.808 0.277 0.853 0.252 1 ...
## $ V29 : num 0.679 0.442 0.604 0.209 0.726 ...
## $ V30 : num 0.386 0.203 0.851 0.356 0.472 ...
## $ V31 : num 0.131 0.379 0.851 0.626 0.51 ...
## $ V32 : num 0.26 0.295 0.504 0.734 0.546 ...
## $ V33 : num 0.512 0.198 0.186 0.612 0.288 ...
## $ V34 : num 0.7547 0.2341 0.2709 0.3497 0.0981 ...
## $ V35 : num 0.854 0.131 0.423 0.395 0.195 ...
## $ V36 : num 0.851 0.418 0.304 0.301 0.418 ...
## $ V37 : num 0.669 0.384 0.612 0.541 0.46 ...
## $ V38 : num 0.61 0.106 0.676 0.881 0.322 ...
## $ V39 : num 0.494 0.184 0.537 0.986 0.283 ...
## $ V40 : num 0.274 0.197 0.472 0.917 0.243 ...
## $ V41 : num 0.051 0.167 0.465 0.612 0.198 ...
## $ V42 : num 0.2834 0.0583 0.2587 0.5006 0.2444 ...
## $ V43 : num 0.282 0.14 0.213 0.321 0.185 ...
## $ V44 : num 0.4256 0.1628 0.2222 0.3202 0.0841 ...
## $ V45 : num 0.2641 0.0621 0.2111 0.4295 0.0692 ...
## $ V46 : num 0.1386 0.0203 0.0176 0.3654 0.0528 ...
## $ V47 : num 0.1051 0.053 0.1348 0.2655 0.0357 ...
## $ V48 : num 0.1343 0.0742 0.0744 0.1576 0.0085 ...
## $ V49 : num 0.0383 0.0409 0.013 0.0681 0.023 0.0264 0.0507 0.0285 0.0777 0.0092 ...
## $ V50 : num 0.0324 0.0061 0.0106 0.0294 0.0046 0.0081 0.0159 0.0178 0.0439 0.0198 ...
## $ V51 : num 0.0232 0.0125 0.0033 0.0241 0.0156 0.0104 0.0195 0.0052 0.0061 0.0118 ...
## $ V52 : num 0.0027 0.0084 0.0232 0.0121 0.0031 0.0045 0.0201 0.0081 0.0145 0.009 ...
## $ V53 : num 0.0065 0.0089 0.0166 0.0036 0.0054 0.0014 0.0248 0.012 0.0128 0.0223 ...
## $ V54 : num 0.0159 0.0048 0.0095 0.015 0.0105 0.0038 0.0131 0.0045 0.0145 0.0179 ...
## $ V55 : num 0.0072 0.0094 0.018 0.0085 0.011 0.0013 0.007 0.0121 0.0058 0.0084 ...
## $ V56 : num 0.0167 0.0191 0.0244 0.0073 0.0015 0.0089 0.0138 0.0097 0.0049 0.0068 ...
## $ V57 : num 0.018 0.014 0.0316 0.005 0.0072 0.0057 0.0092 0.0085 0.0065 0.0032 ...
## $ V58 : num 0.0084 0.0049 0.0164 0.0044 0.0048 0.0027 0.0143 0.0047 0.0093 0.0035 ...
## $ V59 : num 0.009 0.0052 0.0095 0.004 0.0107 0.0051 0.0036 0.0048 0.0059 0.0056 ...
## $ V60 : num 0.0032 0.0044 0.0078 0.0117 0.0094 0.0062 0.0103 0.0053 0.0022 0.004 ...
## $ Class: chr "R" "R" "R" "R" ...## Set class to a factor
sr = sr %>%
mutate_if(is.character, as.factor)
## Check
class(sr$Class)## [1] "factor"## Check for any missing values
summary(sr)## V1 V2 V3 V4
## Min. :0.00150 Min. :0.00060 Min. :0.00150 Min. :0.00580
## 1st Qu.:0.01335 1st Qu.:0.01645 1st Qu.:0.01895 1st Qu.:0.02438
## Median :0.02280 Median :0.03080 Median :0.03430 Median :0.04405
## Mean :0.02916 Mean :0.03844 Mean :0.04383 Mean :0.05389
## 3rd Qu.:0.03555 3rd Qu.:0.04795 3rd Qu.:0.05795 3rd Qu.:0.06450
## Max. :0.13710 Max. :0.23390 Max. :0.30590 Max. :0.42640
## V5 V6 V7 V8
## Min. :0.00670 Min. :0.01020 Min. :0.0033 Min. :0.00550
## 1st Qu.:0.03805 1st Qu.:0.06703 1st Qu.:0.0809 1st Qu.:0.08042
## Median :0.06250 Median :0.09215 Median :0.1070 Median :0.11210
## Mean :0.07520 Mean :0.10457 Mean :0.1217 Mean :0.13480
## 3rd Qu.:0.10028 3rd Qu.:0.13412 3rd Qu.:0.1540 3rd Qu.:0.16960
## Max. :0.40100 Max. :0.38230 Max. :0.3729 Max. :0.45900
## V9 V10 V11 V12
## Min. :0.00750 Min. :0.0113 Min. :0.0289 Min. :0.0236
## 1st Qu.:0.09703 1st Qu.:0.1113 1st Qu.:0.1293 1st Qu.:0.1335
## Median :0.15225 Median :0.1824 Median :0.2248 Median :0.2490
## Mean :0.17800 Mean :0.2083 Mean :0.2360 Mean :0.2502
## 3rd Qu.:0.23342 3rd Qu.:0.2687 3rd Qu.:0.3016 3rd Qu.:0.3312
## Max. :0.68280 Max. :0.7106 Max. :0.7342 Max. :0.7060
## V13 V14 V15 V16
## Min. :0.0184 Min. :0.0273 Min. :0.0031 Min. :0.0162
## 1st Qu.:0.1661 1st Qu.:0.1752 1st Qu.:0.1646 1st Qu.:0.1963
## Median :0.2640 Median :0.2811 Median :0.2817 Median :0.3047
## Mean :0.2733 Mean :0.2966 Mean :0.3202 Mean :0.3785
## 3rd Qu.:0.3513 3rd Qu.:0.3862 3rd Qu.:0.4529 3rd Qu.:0.5357
## Max. :0.7131 Max. :0.9970 Max. :1.0000 Max. :0.9988
## V17 V18 V19 V20
## Min. :0.0349 Min. :0.0375 Min. :0.0494 Min. :0.0656
## 1st Qu.:0.2059 1st Qu.:0.2421 1st Qu.:0.2991 1st Qu.:0.3506
## Median :0.3084 Median :0.3683 Median :0.4350 Median :0.5425
## Mean :0.4160 Mean :0.4523 Mean :0.5048 Mean :0.5630
## 3rd Qu.:0.6594 3rd Qu.:0.6791 3rd Qu.:0.7314 3rd Qu.:0.8093
## Max. :1.0000 Max. :1.0000 Max. :1.0000 Max. :1.0000
## V21 V22 V23 V24
## Min. :0.0512 Min. :0.0219 Min. :0.0563 Min. :0.0239
## 1st Qu.:0.3997 1st Qu.:0.4069 1st Qu.:0.4502 1st Qu.:0.5407
## Median :0.6177 Median :0.6649 Median :0.6997 Median :0.6985
## Mean :0.6091 Mean :0.6243 Mean :0.6470 Mean :0.6727
## 3rd Qu.:0.8170 3rd Qu.:0.8320 3rd Qu.:0.8486 3rd Qu.:0.8722
## Max. :1.0000 Max. :1.0000 Max. :1.0000 Max. :1.0000
## V25 V26 V27 V28
## Min. :0.0240 Min. :0.0921 Min. :0.0481 Min. :0.0284
## 1st Qu.:0.5258 1st Qu.:0.5442 1st Qu.:0.5319 1st Qu.:0.5348
## Median :0.7211 Median :0.7545 Median :0.7456 Median :0.7319
## Mean :0.6754 Mean :0.6999 Mean :0.7022 Mean :0.6940
## 3rd Qu.:0.8737 3rd Qu.:0.8938 3rd Qu.:0.9171 3rd Qu.:0.9003
## Max. :1.0000 Max. :1.0000 Max. :1.0000 Max. :1.0000
## V29 V30 V31 V32
## Min. :0.0144 Min. :0.0613 Min. :0.0482 Min. :0.0404
## 1st Qu.:0.4637 1st Qu.:0.4114 1st Qu.:0.3456 1st Qu.:0.2814
## Median :0.6808 Median :0.6071 Median :0.4904 Median :0.4296
## Mean :0.6421 Mean :0.5809 Mean :0.5045 Mean :0.4390
## 3rd Qu.:0.8521 3rd Qu.:0.7352 3rd Qu.:0.6420 3rd Qu.:0.5803
## Max. :1.0000 Max. :1.0000 Max. :0.9657 Max. :0.9306
## V33 V34 V35 V36
## Min. :0.0477 Min. :0.0212 Min. :0.0223 Min. :0.0080
## 1st Qu.:0.2579 1st Qu.:0.2176 1st Qu.:0.1794 1st Qu.:0.1543
## Median :0.3912 Median :0.3510 Median :0.3127 Median :0.3211
## Mean :0.4172 Mean :0.4032 Mean :0.3926 Mean :0.3848
## 3rd Qu.:0.5561 3rd Qu.:0.5961 3rd Qu.:0.5934 3rd Qu.:0.5565
## Max. :1.0000 Max. :0.9647 Max. :1.0000 Max. :1.0000
## V37 V38 V39 V40
## Min. :0.0351 Min. :0.0383 Min. :0.0371 Min. :0.0117
## 1st Qu.:0.1601 1st Qu.:0.1743 1st Qu.:0.1740 1st Qu.:0.1865
## Median :0.3063 Median :0.3127 Median :0.2835 Median :0.2781
## Mean :0.3638 Mean :0.3397 Mean :0.3258 Mean :0.3112
## 3rd Qu.:0.5189 3rd Qu.:0.4405 3rd Qu.:0.4349 3rd Qu.:0.4244
## Max. :0.9497 Max. :1.0000 Max. :0.9857 Max. :0.9297
## V41 V42 V43 V44
## Min. :0.0360 Min. :0.0056 Min. :0.0000 Min. :0.0000
## 1st Qu.:0.1631 1st Qu.:0.1589 1st Qu.:0.1552 1st Qu.:0.1269
## Median :0.2595 Median :0.2451 Median :0.2225 Median :0.1777
## Mean :0.2893 Mean :0.2783 Mean :0.2465 Mean :0.2141
## 3rd Qu.:0.3875 3rd Qu.:0.3842 3rd Qu.:0.3245 3rd Qu.:0.2717
## Max. :0.8995 Max. :0.8246 Max. :0.7733 Max. :0.7762
## V45 V46 V47 V48
## Min. :0.00000 Min. :0.00000 Min. :0.00000 Min. :0.00000
## 1st Qu.:0.09448 1st Qu.:0.06855 1st Qu.:0.06425 1st Qu.:0.04512
## Median :0.14800 Median :0.12135 Median :0.10165 Median :0.07810
## Mean :0.19723 Mean :0.16063 Mean :0.12245 Mean :0.09142
## 3rd Qu.:0.23155 3rd Qu.:0.20037 3rd Qu.:0.15443 3rd Qu.:0.12010
## Max. :0.70340 Max. :0.72920 Max. :0.55220 Max. :0.33390
## V49 V50 V51 V52
## Min. :0.00000 Min. :0.00000 Min. :0.000000 Min. :0.000800
## 1st Qu.:0.02635 1st Qu.:0.01155 1st Qu.:0.008425 1st Qu.:0.007275
## Median :0.04470 Median :0.01790 Median :0.013900 Median :0.011400
## Mean :0.05193 Mean :0.02042 Mean :0.016069 Mean :0.013420
## 3rd Qu.:0.06853 3rd Qu.:0.02527 3rd Qu.:0.020825 3rd Qu.:0.016725
## Max. :0.19810 Max. :0.08250 Max. :0.100400 Max. :0.070900
## V53 V54 V55 V56
## Min. :0.000500 Min. :0.001000 Min. :0.00060 Min. :0.000400
## 1st Qu.:0.005075 1st Qu.:0.005375 1st Qu.:0.00415 1st Qu.:0.004400
## Median :0.009550 Median :0.009300 Median :0.00750 Median :0.006850
## Mean :0.010709 Mean :0.010941 Mean :0.00929 Mean :0.008222
## 3rd Qu.:0.014900 3rd Qu.:0.014500 3rd Qu.:0.01210 3rd Qu.:0.010575
## Max. :0.039000 Max. :0.035200 Max. :0.04470 Max. :0.039400
## V57 V58 V59 V60
## Min. :0.00030 Min. :0.000300 Min. :0.000100 Min. :0.000600
## 1st Qu.:0.00370 1st Qu.:0.003600 1st Qu.:0.003675 1st Qu.:0.003100
## Median :0.00595 Median :0.005800 Median :0.006400 Median :0.005300
## Mean :0.00782 Mean :0.007949 Mean :0.007941 Mean :0.006507
## 3rd Qu.:0.01043 3rd Qu.:0.010350 3rd Qu.:0.010325 3rd Qu.:0.008525
## Max. :0.03550 Max. :0.044000 Max. :0.036400 Max. :0.043900
## Class
## M:111
## R: 97
##
##
##
## Pipeline Setup
Step 1: Define the Task
## Set up the task
## Outcome is the Class (classification task)
sr_task = TaskClassif$new(id = "sr",
backend = sr,
target = "Class")Step 2: Set up our initial pipeline to accomplish the following - Impute Missing Values; Convert factors to numeric by encoding; scale the numerical values to prevent biases
Note: Since there are no missing values or factors (other than the outcome) we will only scale our data.
## Pipeline setup
## Scaling using min/max
sr_scale = po("scalerange",
param_vals = list(lower = 0, upper = 1),
affect_columns = selector_type("numeric"))
## Impute median
sr_impute = po("imputemedian",
affect_columns = selector_type("numeric"))
sr_num = sr_impute %>>% sr_scale
## Train the data
sr_num$train(sr_task)[[1]]$data()## Class V1 V10 V11 V12 V13 V14
## 1: R 0.13643068 0.2857143 0.18715440 0.1972450 0.2956672 0.038362380
## 2: R 0.32300885 0.3945374 0.65631646 0.9255569 0.9694832 0.775910075
## 3: R 0.18215339 0.8695839 0.85694031 1.0000000 0.7715561 0.520470249
## 4: R 0.06268437 0.1645932 0.08393591 0.2573271 0.0000000 0.205011859
## 5: R 0.55088496 0.6214786 0.54771019 0.5445487 0.5861523 0.398267505
## ---
## 204: M 0.12684366 0.3676534 0.39968808 0.3952227 0.3009932 0.074352893
## 205: M 0.22713864 0.2918633 0.39642705 0.4673212 0.4039154 0.116427761
## 206: M 0.37389381 0.3454883 0.34410889 0.3133060 0.2438463 0.073218521
## 207: M 0.21238938 0.3204633 0.36991351 0.3774912 0.2006622 0.000000000
## 208: M 0.18067847 0.3204633 0.34467602 0.3232708 0.2131855 0.006496855
## V15 V16 V17 V18 V19 V2 V20
## 1: 0.06309560 0.2148382 0.28504818 0.2726234 0.4822218 0.15645092 0.44317209
## 2: 0.74561140 0.9446367 1.00000000 0.8830130 0.7921313 0.22160309 0.76648116
## 3: 0.64680510 0.6888866 0.66417988 0.7455584 0.8873343 0.24689241 0.85220462
## 4: 0.17032802 0.2003867 0.03564397 0.1980260 0.3751315 0.07072439 0.35498716
## 5: 0.45109841 0.5255445 0.72085794 0.6044675 0.1617926 0.28289756 0.42594178
## ---
## 204: 0.17755041 0.2074089 0.24691742 0.2583896 0.1620029 0.14573511 0.11601027
## 205: 0.12077440 0.1396296 0.16081235 0.2134026 0.1710499 0.04072010 0.18610873
## 206: 0.06540275 0.1364747 0.14236867 0.2040519 0.1544288 0.18474068 0.07908818
## 207: 0.06439964 0.1304702 0.17832349 0.2358442 0.2455291 0.14873553 0.38923373
## 208: 0.12749524 0.1573377 0.18941042 0.2909091 0.3387334 0.15302186 0.41695205
## V21 V22 V23 V24 V25 V26 V27
## 1: 0.5555438 0.4960638 0.3989615 0.5441041 0.6630123 0.6051327 0.6957664
## 2: 0.4953626 0.3918822 0.3596482 0.3764983 0.3084016 0.2510188 0.2930980
## 3: 0.7864671 0.6663940 0.3952527 0.3492470 0.5216189 0.1643353 0.4820885
## 4: 0.2349283 0.3548717 0.5290876 0.4719803 0.2971311 0.4860667 0.5016283
## 5: 0.3832209 0.4164196 0.5475257 0.5286344 0.2992828 0.1502368 0.6843156
## ---
## 204: 0.3813238 0.6809120 0.8007841 0.8415121 0.8892418 0.9228990 0.9986343
## 205: 0.2908938 0.5858297 0.8200699 0.9021617 0.9885246 0.9903073 0.9420107
## 206: 0.2795110 0.5795931 0.7489668 0.8435611 0.9409836 0.9984580 0.9683790
## 207: 0.5452150 0.6922605 0.7081700 0.7563774 0.9223361 1.0000000 0.9977939
## 208: 0.5467960 0.7338718 0.7946381 0.8808524 0.9418033 1.0000000 0.9343418
## V28 V29 V3 V30 V31 V32 V33
## 1: 0.8023878 0.6744115 0.13567674 0.3455843 0.08991826 0.2471355 0.4876615
## 2: 0.2555578 0.4341518 0.27201051 0.1507404 0.36032698 0.2856661 0.1582485
## 3: 0.8490119 0.5978084 0.35611038 0.8416960 0.87520436 0.5213435 0.1454374
## 4: 0.2301359 0.1974432 0.19973719 0.3138383 0.62975477 0.7791508 0.5925654
## 5: 1.0000000 0.7221997 0.15308804 0.4379461 0.50365123 0.5678499 0.2524415
## ---
## 204: 1.0000000 0.8076299 0.05026281 0.5950783 0.60588556 0.5777353 0.3867479
## 205: 1.0000000 0.9078734 0.09296978 0.7242996 0.78572207 0.7541002 0.5068781
## 206: 1.0000000 0.8609984 0.05420499 0.6781719 0.78855586 0.7752191 0.4956421
## 207: 0.8247221 0.6988636 0.15604468 0.6954298 0.69874659 0.5119074 0.2212538
## 208: 0.7532935 0.7080966 0.03975033 0.8250772 0.77820163 0.4676477 0.1300011
## V34 V35 V36 V37 V38 V39
## 1: 0.77742448 0.85036310 0.84949597 0.69330855 0.59415618 0.48197343
## 2: 0.22564918 0.11077017 0.41350806 0.38093155 0.07008423 0.15485979
## 3: 0.26465289 0.41004398 0.29868952 0.63033020 0.66268067 0.52751423
## 4: 0.34817170 0.38150762 0.29556452 0.55291931 0.87667672 1.00000000
## 5: 0.08150503 0.17674133 0.41340726 0.46501203 0.29468649 0.25901328
## ---
## 204: 0.13354531 0.06402782 0.05322581 0.06757052 0.10689404 0.13757116
## 205: 0.20339163 0.07210801 0.12268145 0.08834463 0.08765727 0.14726966
## 206: 0.16682565 0.08949576 0.13770161 0.07817625 0.10990954 0.16424204
## 207: 0.08023317 0.05656132 0.02862903 0.13350098 0.14973484 0.10309930
## 208: 0.14170641 0.14503426 0.18034274 0.25191340 0.13829677 0.06198608
## V4 V40 V41 V42 V43 V44
## 1: 0.03542558 0.28616558 0.01737116 0.33919414 0.36531747 0.5483123
## 2: 0.15002378 0.20185185 0.15217140 0.06434676 0.18117160 0.2097398
## 3: 0.24369948 0.50130719 0.49646786 0.30903541 0.27531359 0.2862664
## 4: 0.03495007 0.98583878 0.66716850 0.60439560 0.41510410 0.4125225
## 5: 0.07988588 0.25196078 0.18749276 0.29157509 0.23884650 0.1083484
## ---
## 204: 0.02829291 0.11797386 0.09739433 0.11965812 0.25565757 0.1725071
## 205: 0.12030433 0.02483660 0.13920093 0.09377289 0.14858399 0.2182427
## 206: 0.05563481 0.02265795 0.13086277 0.03321123 0.06944265 0.1686421
## 207: 0.13076557 0.07973856 0.01528662 0.16007326 0.20069831 0.1994331
## 208: 0.05087970 0.07091503 0.05639838 0.24664225 0.32393638 0.2398866
## V45 V46 V47 V48 V49 V5
## 1: 0.37546204 0.19007131 0.19032959 0.40221623 0.19333670 0.22495562
## 2: 0.08828547 0.02783873 0.09597972 0.22222222 0.20646138 0.28303322
## 3: 0.30011373 0.02413604 0.24411445 0.22282120 0.06562342 0.23002790
## 4: 0.61060563 0.50109709 0.48080406 0.47199760 0.34376577 0.03499873
## 5: 0.09837930 0.07240812 0.06465049 0.02545672 0.11610298 0.13264012
## ---
## 204: 0.12823429 0.14879320 0.27544368 0.40820605 0.43311459 0.08267816
## 205: 0.13562695 0.01097093 0.14306411 0.37586104 0.32660273 0.17575450
## 206: 0.12937162 0.10381240 0.19177834 0.30098832 0.27006562 0.07202638
## 207: 0.18751777 0.13507954 0.22781601 0.28571429 0.24684503 0.02536140
## 208: 0.20457777 0.20159078 0.17946396 0.01227913 0.07773852 0.03728126
## V50 V51 V52 V53 V54 V55
## 1: 0.39272727 0.23107570 0.02710414 0.15584416 0.4356725 0.14965986
## 2: 0.07393939 0.12450199 0.10841655 0.21818182 0.1111111 0.19954649
## 3: 0.12848485 0.03286853 0.31954351 0.41818182 0.2485380 0.39455782
## 4: 0.35636364 0.24003984 0.16119829 0.08051948 0.4093567 0.17913832
## 5: 0.05575758 0.15537849 0.03281027 0.12727273 0.2777778 0.23582766
## ---
## 204: 0.35151515 0.20219124 0.15406562 0.24155844 0.5526316 0.06122449
## 205: 0.21696970 0.05079681 0.07560628 0.22857143 0.3654971 0.12925170
## 206: 0.28484848 0.15438247 0.21683310 0.06233766 0.1198830 0.12698413
## 207: 0.29212121 0.04183267 0.11126961 0.10649351 0.3391813 0.06802721
## 208: 0.14060606 0.18027888 0.19686163 0.32207792 0.1081871 0.07482993
## V56 V57 V58 V59 V6 V60
## 1: 0.41794872 0.50284091 0.18535469 0.24517906 0.23757055 0.06004619
## 2: 0.47948718 0.38920455 0.10526316 0.14049587 0.66675625 0.08775982
## 3: 0.61538462 0.88920455 0.36842105 0.25895317 0.58532653 0.16628176
## 4: 0.17692308 0.13352273 0.09382151 0.10743802 0.07148616 0.25635104
## 5: 0.02820513 0.19602273 0.10297483 0.29201102 0.14700349 0.20323326
## ---
## 204: 0.24871795 0.17613636 0.25629291 0.52892562 0.41064230 0.34872979
## 205: 0.15128205 0.08806818 0.06636156 0.16804408 0.23004569 0.14087760
## 206: 0.21794872 0.38920455 0.30892449 0.20936639 0.28728836 0.05773672
## 207: 0.07948718 0.08806818 0.17391304 0.09641873 0.33646869 0.09699769
## 208: 0.14615385 0.10511364 0.07551487 0.16528926 0.06342381 0.25173210
## V7 V8 V9
## 1: 0.4074675 0.3409041 0.44928180
## 2: 0.5744048 0.7554576 0.48304457
## 3: 0.6488095 0.8194046 0.81785873
## 4: 0.2881494 0.2692393 0.07744706
## 5: 0.3181818 0.5318633 0.51665926
## ---
## 204: 0.5397727 0.3614112 0.33362950
## 205: 0.2589286 0.2123484 0.14141863
## 206: 0.3311688 0.2476295 0.17518140
## 207: 0.3874459 0.2355017 0.27691396
## 208: 0.1682900 0.2965821 0.26180957Step 3: Link the data processing pipe to a learner
## Define the learner
## Using a random forest
## mtry = the number of variables used per split
sr_rf_lrn = lrn("classif.ranger",
mtry = 3,
num.trees = 500,
predict_type = "prob") ## Add the learner to the pipeline
sr_graph = sr_num %>>%
sr_rf_lrnVisualize the Pipeline
## Visualize
plot(sr_graph)
Step 4: Run the sonar data through the entire pipeline: process the numeric data and train the random forest
## Run the full pipeline
sr_graph$train(sr_task)## $classif.ranger.output
## NULLStep 5: Convert this to a GraphLearner for model evaluation and tuning
## Convert the pipeline to a graphlearner
sr_glrn = GraphLearner$new(sr_graph)Step 6: Set up the cross-validation and performance metric
## Resampling strategy
sr_resamp = rsmp("cv",
folds = 5)
## Instantiate
sr_resamp$instantiate(sr_task)
## Performance metric
sr_msr_auc = msr("classif.auc")Step 7: Run the resampler
## First Run
sr_rf = resample(task = sr_task,
learner = sr_glrn,
resampling = sr_resamp,
store_models = TRUE)## INFO [19:08:57.815] [mlr3] Applying learner 'imputemedian.scalerange.classif.ranger' on task 'sr' (iter 1/5)
## INFO [19:08:57.943] [mlr3] Applying learner 'imputemedian.scalerange.classif.ranger' on task 'sr' (iter 2/5)
## INFO [19:08:58.071] [mlr3] Applying learner 'imputemedian.scalerange.classif.ranger' on task 'sr' (iter 3/5)
## INFO [19:08:58.218] [mlr3] Applying learner 'imputemedian.scalerange.classif.ranger' on task 'sr' (iter 4/5)
## INFO [19:08:58.346] [mlr3] Applying learner 'imputemedian.scalerange.classif.ranger' on task 'sr' (iter 5/5)Step 8: Check the performance of the model
## Individual scores
sr_rf$score(sr_msr_auc)## task_id learner_id resampling_id iteration
## 1: sr imputemedian.scalerange.classif.ranger cv 1
## 2: sr imputemedian.scalerange.classif.ranger cv 2
## 3: sr imputemedian.scalerange.classif.ranger cv 3
## 4: sr imputemedian.scalerange.classif.ranger cv 4
## 5: sr imputemedian.scalerange.classif.ranger cv 5
## classif.auc
## 1: 0.8840909
## 2: 0.9500000
## 3: 0.9336384
## 4: 0.8897436
## 5: 0.9377990
## Hidden columns: task, learner, resampling, predictionNote: These models perform very well
## Aggregate score
sr_rf$aggregate(sr_msr_auc)## classif.auc
## 0.9190544Hyperparameter Tuning
This dataset currently has 60 covariates. Here, I use a PCA filter to address any multicolinearity among the explanatory variables. Here, I tune the number of trees to see if it improves our models performance.
Step 0: Create the PCA feature selection
## PCA selection
sr_pca = po("pca")
## Filter the PCA - 0.5 means 50% of the variance in our data are explained by our new PCs
sr_filter = po("filter", filter = flt("variance"), filter.frac = 0.4143)
##Reconstruct our graph
sr_graph= sr_num %>>%
sr_pca %>>%
sr_filter
## Check
sr_graph$train(sr_task)[[1]]$data()## Class PC1 PC2 PC3 PC4 PC5
## 1: R -0.57677891 0.2846842 0.3657173 -0.36069301 0.20773620
## 2: R 1.12908360 1.1184984 0.1127545 0.92819347 0.58272792
## 3: R 0.41013437 1.3109394 -0.4981949 0.30912550 1.10874356
## 4: R -1.04114085 0.5618324 0.1864932 -0.15727435 -0.56060872
## 5: R 0.16551735 0.2563634 0.0553820 0.75555502 0.55275640
## ---
## 204: M -0.01511240 -0.7975700 -0.5899923 0.31310029 -0.07552906
## 205: M -0.15744835 -1.1368502 -0.3871416 0.27714909 0.09344938
## 206: M -0.24909642 -1.1089653 -0.3419851 0.34218302 0.04974453
## 207: M 0.10395654 -0.9907934 -0.3335576 0.23079675 0.04305514
## 208: M 0.09472768 -0.9887226 -0.1623886 0.04329377 0.02987063
## PC6 PC7 PC8 PC9 PC10 PC11
## 1: 0.528543582 -0.123782195 0.08819507 -0.227591065 -0.09312889 0.25185316
## 2: -0.274674870 -0.393741027 -0.31930830 0.305236856 0.10819148 -0.05294628
## 3: -0.183509654 -0.156100611 0.21407744 -0.003644596 -0.12533356 -0.20262155
## 4: 0.235282794 0.374238984 -0.72701091 0.180377975 0.16852617 0.13780457
## 5: 0.006754189 -0.307775505 -0.08087238 -0.019783755 0.08882964 -0.26198166
## ---
## 204: 0.148290754 -0.299050521 0.34511887 0.128833782 0.17976359 0.11392863
## 205: -0.102509044 -0.163151159 0.22037422 0.296157740 0.02401905 0.04827908
## 206: -0.040287078 -0.225205605 0.32838656 0.192316196 0.22224284 0.19236251
## 207: 0.221831234 0.033729097 0.14847931 0.044747243 0.18309057 0.05521199
## 208: 0.104329920 0.003497516 0.16703085 0.006421595 0.07265991 -0.02400257
## PC12 PC13 PC14 PC15 PC16 PC17
## 1: 0.20297078 0.17271630 -0.16553702 -0.46698018 -0.04867786 -0.07230858
## 2: 0.09256740 0.03070541 -0.06526108 -0.14208554 0.03108011 -0.09962736
## 3: -0.48922841 -0.04591209 0.06491690 0.23052191 -0.10144926 -0.58164019
## 4: -0.58093202 -0.10454618 0.32958242 0.23280050 0.23395977 0.30322567
## 5: -0.04712724 -0.47864639 -0.27872951 0.23101222 0.11769300 0.00371596
## ---
## 204: -0.36895664 -0.04496005 -0.15047693 -0.10026207 -0.10731608 0.15586908
## 205: -0.27500829 -0.29439682 -0.08507265 -0.06515491 -0.06339238 0.10280793
## 206: -0.28164701 -0.32953512 0.11860478 -0.15680489 0.02865568 -0.02142690
## 207: -0.10526312 -0.25207758 0.11399508 -0.02926640 -0.05806716 0.10901656
## 208: 0.09710126 -0.28193653 0.06642501 0.25969696 -0.02363676 -0.17598189
## PC18 PC19 PC20 PC21 PC22 PC23
## 1: -0.17171242 -0.10374916 0.13387524 -0.113503818 -0.04847561 0.06475228
## 2: -0.10842181 -0.11144770 -0.11390458 -0.420694898 0.01394495 -0.07960776
## 3: -0.11164127 -0.19806479 -0.12288581 -0.278612902 0.29464281 0.08725906
## 4: 0.04575541 -0.08843411 0.19814360 0.045463236 0.01211968 0.10193771
## 5: -0.19681689 0.11266728 0.03988766 0.180564272 -0.06109474 0.39525735
## ---
## 204: -0.30815054 -0.02136010 0.18181738 0.188614525 0.16349284 -0.07103692
## 205: 0.03130364 0.01241649 0.11582788 0.025772715 0.05303905 -0.12467401
## 206: -0.13419342 0.08812235 -0.07373809 0.009653788 0.11794073 0.11222203
## 207: -0.04841495 0.01883251 0.16673115 -0.041866811 0.36432836 -0.00481731
## 208: 0.02880947 -0.24713701 -0.04836768 0.033912284 0.29361943 0.04425913
## PC24 PC25
## 1: -0.20452439 0.23746685
## 2: -0.07893565 0.15487991
## 3: 0.26745913 0.26524062
## 4: 0.09754972 0.03575194
## 5: 0.03082182 0.09114421
## ---
## 204: -0.05223380 -0.19472026
## 205: -0.07732774 -0.03165790
## 206: -0.05878949 0.01592131
## 207: -0.01637808 -0.10863424
## 208: 0.09418055 -0.25643103Note: This gives us 25 principle components
Next, connect this to our learner
## Pipeline for our learner
sr_graph = sr_num %>>%
sr_pca %>>%
sr_filter %>>%
sr_rf_lrn
## Visualize
plot(sr_graph)
Create a new GraphLearner
## Create a new graph learner
sr_glrn = GraphLearner$new(sr_graph)Check the Parameter Set:
## Check
sr_glrn$param_set## <ParamSetCollection>
## id class lower upper nlevels
## 1: classif.ranger.alpha ParamDbl -Inf Inf Inf
## 2: classif.ranger.always.split.variables ParamUty NA NA Inf
## 3: classif.ranger.class.weights ParamUty NA NA Inf
## 4: classif.ranger.holdout ParamLgl NA NA 2
## 5: classif.ranger.importance ParamFct NA NA 4
## 6: classif.ranger.keep.inbag ParamLgl NA NA 2
## 7: classif.ranger.max.depth ParamInt 0 Inf Inf
## 8: classif.ranger.min.node.size ParamInt 1 Inf Inf
## 9: classif.ranger.min.prop ParamDbl -Inf Inf Inf
## 10: classif.ranger.minprop ParamDbl -Inf Inf Inf
## 11: classif.ranger.mtry ParamInt 1 Inf Inf
## 12: classif.ranger.mtry.ratio ParamDbl 0 1 Inf
## 13: classif.ranger.num.random.splits ParamInt 1 Inf Inf
## 14: classif.ranger.num.threads ParamInt 1 Inf Inf
## 15: classif.ranger.num.trees ParamInt 1 Inf Inf
## 16: classif.ranger.oob.error ParamLgl NA NA 2
## 17: classif.ranger.regularization.factor ParamUty NA NA Inf
## 18: classif.ranger.regularization.usedepth ParamLgl NA NA 2
## 19: classif.ranger.replace ParamLgl NA NA 2
## 20: classif.ranger.respect.unordered.factors ParamFct NA NA 3
## 21: classif.ranger.sample.fraction ParamDbl 0 1 Inf
## 22: classif.ranger.save.memory ParamLgl NA NA 2
## 23: classif.ranger.scale.permutation.importance ParamLgl NA NA 2
## 24: classif.ranger.se.method ParamFct NA NA 2
## 25: classif.ranger.seed ParamInt -Inf Inf Inf
## 26: classif.ranger.split.select.weights ParamUty NA NA Inf
## 27: classif.ranger.splitrule ParamFct NA NA 3
## 28: classif.ranger.verbose ParamLgl NA NA 2
## 29: classif.ranger.write.forest ParamLgl NA NA 2
## 30: imputemedian.affect_columns ParamUty NA NA Inf
## 31: pca.affect_columns ParamUty NA NA Inf
## 32: pca.center ParamLgl NA NA 2
## 33: pca.rank. ParamInt 1 Inf Inf
## 34: pca.scale. ParamLgl NA NA 2
## 35: scalerange.affect_columns ParamUty NA NA Inf
## 36: scalerange.lower ParamDbl -Inf Inf Inf
## 37: scalerange.upper ParamDbl -Inf Inf Inf
## 38: variance.affect_columns ParamUty NA NA Inf
## 39: variance.filter.cutoff ParamDbl -Inf Inf Inf
## 40: variance.filter.frac ParamDbl 0 1 Inf
## 41: variance.filter.nfeat ParamInt 0 Inf Inf
## 42: variance.filter.permuted ParamInt 1 Inf Inf
## 43: variance.na.rm ParamLgl NA NA 2
## id class lower upper nlevels
## default parents value
## 1: 0.5
## 2: <NoDefault[3]>
## 3:
## 4: FALSE
## 5: <NoDefault[3]>
## 6: FALSE
## 7:
## 8:
## 9: 0.1
## 10: 0.1
## 11: <NoDefault[3]> 3
## 12: <NoDefault[3]>
## 13: 1 classif.ranger.splitrule
## 14: 1 1
## 15: 500 500
## 16: TRUE
## 17: 1
## 18: FALSE
## 19: TRUE
## 20: ignore
## 21: <NoDefault[3]>
## 22: FALSE
## 23: FALSE classif.ranger.importance
## 24: infjack
## 25:
## 26:
## 27: gini
## 28: TRUE
## 29: TRUE
## 30: <NoDefault[3]> <Selector[1]>
## 31: <Selector[1]>
## 32: TRUE
## 33:
## 34: FALSE
## 35: <Selector[1]> <Selector[1]>
## 36: <NoDefault[3]> 0
## 37: <NoDefault[3]> 1
## 38: <Selector[1]>
## 39: <NoDefault[3]>
## 40: <NoDefault[3]> 0.4143
## 41: <NoDefault[3]>
## 42: <NoDefault[3]>
## 43: TRUE
## default parents valueStep 1: Create the tuning paramter set
## num.trees = number of trees in the random forest
sr_tuneps = ParamSet$new(list(
ParamInt$new("classif.ranger.num.trees", lower = 50, upper = 500),
ParamDbl$new("variance.filter.frac", lower = 0.40, upper = 0.95)
))
## Check
sr_tuneps## <ParamSet>
## id class lower upper nlevels default value
## 1: classif.ranger.num.trees ParamInt 50.0 500.00 451 <NoDefault[3]>
## 2: variance.filter.frac ParamDbl 0.4 0.95 Inf <NoDefault[3]>Step 2: Create the terminator and tuner
## Terminator - evals - set at 20 evaluations
sr_evals = trm("evals",
n_evals = 20)
## Tuner - random_search
sr_tuner = tnr("random_search")Step 3: Create the AutoTuner
## Create autotuner
at_sr = AutoTuner$new(learner = sr_glrn,
resampling = rsmp("holdout"),
measure = sr_msr_auc,
search_space = sr_tuneps,
terminator = sr_evals,
tuner = sr_tuner)Step 4: Tune
## Tune
at_sr$train(sr_task)## INFO [19:08:59.179] [bbotk] Starting to optimize 2 parameter(s) with '<OptimizerRandomSearch>' and '<TerminatorEvals> [n_evals=20, k=0]'
## INFO [19:08:59.184] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:59.205] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:59.207] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:08:59.347] [mlr3] Finished benchmark
## INFO [19:08:59.357] [bbotk] Result of batch 1:
## INFO [19:08:59.358] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:08:59.358] [bbotk] 184 0.505821 0.8459596 0 0
## INFO [19:08:59.358] [bbotk] runtime_learners uhash
## INFO [19:08:59.358] [bbotk] 0.137 986548f5-b93d-4209-aef5-079ecc7c2952
## INFO [19:08:59.359] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:59.378] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:59.381] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:08:59.584] [mlr3] Finished benchmark
## INFO [19:08:59.603] [bbotk] Result of batch 2:
## INFO [19:08:59.604] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:08:59.604] [bbotk] 393 0.9263799 0.8484848 0 0
## INFO [19:08:59.604] [bbotk] runtime_learners uhash
## INFO [19:08:59.604] [bbotk] 0.199 6951a1aa-a206-4cd4-b392-f84a49f7443a
## INFO [19:08:59.605] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:59.627] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:59.629] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:08:59.776] [mlr3] Finished benchmark
## INFO [19:08:59.788] [bbotk] Result of batch 3:
## INFO [19:08:59.788] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:08:59.788] [bbotk] 261 0.8780871 0.8535354 0 0
## INFO [19:08:59.788] [bbotk] runtime_learners uhash
## INFO [19:08:59.788] [bbotk] 0.144 cd22ddf4-72bf-4f9d-9c03-a2d8d5ce470d
## INFO [19:08:59.790] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:59.809] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:59.812] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:08:59.947] [mlr3] Finished benchmark
## INFO [19:08:59.958] [bbotk] Result of batch 4:
## INFO [19:08:59.958] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:08:59.958] [bbotk] 84 0.8830826 0.7525253 0 0
## INFO [19:08:59.958] [bbotk] runtime_learners uhash
## INFO [19:08:59.958] [bbotk] 0.132 7a215da0-aebc-479a-85e0-79b7f08eaa8b
## INFO [19:08:59.960] [bbotk] Evaluating 1 configuration(s)
## INFO [19:08:59.979] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:08:59.982] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:00.174] [mlr3] Finished benchmark
## INFO [19:09:00.185] [bbotk] Result of batch 5:
## INFO [19:09:00.186] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:00.186] [bbotk] 374 0.5633208 0.8678451 0 0
## INFO [19:09:00.186] [bbotk] runtime_learners uhash
## INFO [19:09:00.186] [bbotk] 0.189 d4892864-fd78-48e4-b7e7-d4039498efe5
## INFO [19:09:00.187] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:00.206] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:00.209] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:00.368] [mlr3] Finished benchmark
## INFO [19:09:00.379] [bbotk] Result of batch 6:
## INFO [19:09:00.380] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:00.380] [bbotk] 442 0.8721894 0.8358586 0 0
## INFO [19:09:00.380] [bbotk] runtime_learners uhash
## INFO [19:09:00.380] [bbotk] 0.157 3dedabf8-ddc5-4531-8a84-b99c70c66093
## INFO [19:09:00.381] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:00.401] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:00.403] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:00.536] [mlr3] Finished benchmark
## INFO [19:09:00.548] [bbotk] Result of batch 7:
## INFO [19:09:00.548] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:00.548] [bbotk] 92 0.4154764 0.8956229 0 0
## INFO [19:09:00.548] [bbotk] runtime_learners uhash
## INFO [19:09:00.548] [bbotk] 0.129 1acd15bb-5fd8-404f-891d-83957e418f84
## INFO [19:09:00.550] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:00.594] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:00.598] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:00.766] [mlr3] Finished benchmark
## INFO [19:09:00.778] [bbotk] Result of batch 8:
## INFO [19:09:00.778] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:00.778] [bbotk] 380 0.746174 0.8585859 0 0
## INFO [19:09:00.778] [bbotk] runtime_learners uhash
## INFO [19:09:00.778] [bbotk] 0.164 d0d11a60-c57b-4c42-8ece-90a25ce0e433
## INFO [19:09:00.780] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:00.799] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:00.802] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:00.948] [mlr3] Finished benchmark
## INFO [19:09:00.959] [bbotk] Result of batch 9:
## INFO [19:09:00.960] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:00.960] [bbotk] 280 0.4799861 0.8712121 0 0
## INFO [19:09:00.960] [bbotk] runtime_learners uhash
## INFO [19:09:00.960] [bbotk] 0.143 da9e4728-d7a5-4813-9eaf-59ab0ba0e8a8
## INFO [19:09:00.961] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:00.980] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:00.983] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:01.166] [mlr3] Finished benchmark
## INFO [19:09:01.179] [bbotk] Result of batch 10:
## INFO [19:09:01.179] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:01.179] [bbotk] 156 0.6627183 0.8644781 0 0
## INFO [19:09:01.179] [bbotk] runtime_learners uhash
## INFO [19:09:01.179] [bbotk] 0.179 bb0aba56-f569-425e-8597-4210b45ab805
## INFO [19:09:01.181] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:01.201] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:01.203] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:01.363] [mlr3] Finished benchmark
## INFO [19:09:01.374] [bbotk] Result of batch 11:
## INFO [19:09:01.375] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:01.375] [bbotk] 446 0.807932 0.8762626 0 0
## INFO [19:09:01.375] [bbotk] runtime_learners uhash
## INFO [19:09:01.375] [bbotk] 0.156 8a03b9de-f9b8-4a11-a36f-b19044ef2bba
## INFO [19:09:01.376] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:01.396] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:01.399] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:01.553] [mlr3] Finished benchmark
## INFO [19:09:01.564] [bbotk] Result of batch 12:
## INFO [19:09:01.565] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:01.565] [bbotk] 413 0.5761 0.8644781 0 0
## INFO [19:09:01.565] [bbotk] runtime_learners uhash
## INFO [19:09:01.565] [bbotk] 0.15 d5b792ca-d341-4dd3-97fd-7a599f3e8d79
## INFO [19:09:01.566] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:01.585] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:01.588] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:01.783] [mlr3] Finished benchmark
## INFO [19:09:01.794] [bbotk] Result of batch 13:
## INFO [19:09:01.795] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:01.795] [bbotk] 392 0.823472 0.8720539 0 0
## INFO [19:09:01.795] [bbotk] runtime_learners uhash
## INFO [19:09:01.795] [bbotk] 0.192 c5f1fb81-d9ae-4ccf-9976-2d6805eb0dc4
## INFO [19:09:01.796] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:01.816] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:01.819] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:01.979] [mlr3] Finished benchmark
## INFO [19:09:01.990] [bbotk] Result of batch 14:
## INFO [19:09:01.991] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:01.991] [bbotk] 367 0.7645228 0.8720539 0 0
## INFO [19:09:01.991] [bbotk] runtime_learners uhash
## INFO [19:09:01.991] [bbotk] 0.156 0c585c7e-8f9c-4668-b4f3-e2c459534a84
## INFO [19:09:01.992] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:02.011] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:02.014] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:02.185] [mlr3] Finished benchmark
## INFO [19:09:02.204] [bbotk] Result of batch 15:
## INFO [19:09:02.205] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:02.205] [bbotk] 324 0.7673088 0.8602694 0 0
## INFO [19:09:02.205] [bbotk] runtime_learners uhash
## INFO [19:09:02.205] [bbotk] 0.167 41076322-716f-4433-876b-b20b04c0b786
## INFO [19:09:02.207] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:02.238] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:02.241] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:02.400] [mlr3] Finished benchmark
## INFO [19:09:02.412] [bbotk] Result of batch 16:
## INFO [19:09:02.412] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:02.412] [bbotk] 459 0.5804672 0.8829966 0 0
## INFO [19:09:02.412] [bbotk] runtime_learners uhash
## INFO [19:09:02.412] [bbotk] 0.157 0cfafa47-188d-428f-b821-f23d92662ed5
## INFO [19:09:02.414] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:02.433] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:02.436] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:02.579] [mlr3] Finished benchmark
## INFO [19:09:02.590] [bbotk] Result of batch 17:
## INFO [19:09:02.591] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:02.591] [bbotk] 203 0.6683477 0.8324916 0 0
## INFO [19:09:02.591] [bbotk] runtime_learners uhash
## INFO [19:09:02.591] [bbotk] 0.14 ddfa3d13-9dd7-487e-b38e-9318b8e46240
## INFO [19:09:02.592] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:02.611] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:02.614] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:02.801] [mlr3] Finished benchmark
## INFO [19:09:02.812] [bbotk] Result of batch 18:
## INFO [19:09:02.812] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:02.812] [bbotk] 294 0.5149473 0.8872054 0 0
## INFO [19:09:02.812] [bbotk] runtime_learners uhash
## INFO [19:09:02.812] [bbotk] 0.184 77300007-cb8e-4c20-a07a-a3e51cd63cdd
## INFO [19:09:02.814] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:02.833] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:02.836] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:02.973] [mlr3] Finished benchmark
## INFO [19:09:02.984] [bbotk] Result of batch 19:
## INFO [19:09:02.985] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:02.985] [bbotk] 103 0.7700049 0.8535354 0 0
## INFO [19:09:02.985] [bbotk] runtime_learners uhash
## INFO [19:09:02.985] [bbotk] 0.135 b9792fdb-8c7f-4894-b6fd-600e55e71af6
## INFO [19:09:02.986] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:03.006] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:03.008] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:03.147] [mlr3] Finished benchmark
## INFO [19:09:03.172] [bbotk] Result of batch 20:
## INFO [19:09:03.173] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:03.173] [bbotk] 171 0.5890065 0.8644781 0 0
## INFO [19:09:03.173] [bbotk] runtime_learners uhash
## INFO [19:09:03.173] [bbotk] 0.135 c70f1606-0c47-4ab9-a304-4fc0f9692b16
## INFO [19:09:03.180] [bbotk] Finished optimizing after 20 evaluation(s)
## INFO [19:09:03.181] [bbotk] Result:
## INFO [19:09:03.182] [bbotk] classif.ranger.num.trees variance.filter.frac learner_param_vals x_domain
## INFO [19:09:03.182] [bbotk] 92 0.4154764 <list[8]> <list[2]>
## INFO [19:09:03.182] [bbotk] classif.auc
## INFO [19:09:03.182] [bbotk] 0.8956229Step 5: Check the Hyperparameter Choice and Performance
## Check the overall learner
at_sr$learner## <GraphLearner:imputemedian.scalerange.pca.variance.classif.ranger>
## * Model: list
## * Parameters: imputemedian.affect_columns=<Selector>,
## scalerange.lower=0, scalerange.upper=1,
## scalerange.affect_columns=<Selector>, variance.filter.frac=0.4155,
## classif.ranger.mtry=3, classif.ranger.num.threads=1,
## classif.ranger.num.trees=92
## * Packages: mlr3, mlr3pipelines, stats, mlr3learners, ranger
## * Predict Types: response, [prob]
## * Feature Types: logical, integer, numeric, character, factor, ordered,
## POSIXct
## * Properties: featureless, hotstart_backward, hotstart_forward,
## importance, loglik, missings, multiclass, oob_error,
## selected_features, twoclass, weights## Extract the Hyperparameters
at_sr$tuning_result## classif.ranger.num.trees variance.filter.frac learner_param_vals x_domain
## 1: 92 0.4154764 <list[8]> <list[2]>
## classif.auc
## 1: 0.8956229Tuning Result Hyperparameter Choices
- Number of Trees: 92
- Percentage of the Variance Explained: 0.4154764
Performance Measure (AUC)
Step 1: Define the inner and outer cross-validation strategies
## Inner
sr_rsmp_inner = rsmp("holdout",
ratio = 0.8)
## Outer
sr_rsmp_outer = rsmp("cv",
folds = 5)Step 2: Update the AutoTuner
## Update
at_sr = AutoTuner$new(learner = sr_glrn,
resampling = sr_rsmp_inner,
measure = sr_msr_auc,
search_space = sr_tuneps,
terminator = sr_evals,
tuner = sr_tuner)Step 3: Run the Model
## Run
sr_rf = resample(task = sr_task,
learner = at_sr,
resampling = sr_rsmp_outer,
store_models = TRUE)## INFO [19:09:03.367] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger.tuned' on task 'sr' (iter 1/5)
## INFO [19:09:03.417] [bbotk] Starting to optimize 2 parameter(s) with '<OptimizerRandomSearch>' and '<TerminatorEvals> [n_evals=20, k=0]'
## INFO [19:09:03.422] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:03.441] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:03.444] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:03.594] [mlr3] Finished benchmark
## INFO [19:09:03.604] [bbotk] Result of batch 1:
## INFO [19:09:03.604] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:03.604] [bbotk] 339 0.7676162 0.8851852 0 0
## INFO [19:09:03.604] [bbotk] runtime_learners uhash
## INFO [19:09:03.604] [bbotk] 0.147 15e9a4fd-64a4-4004-936f-9b8bbccf9325
## INFO [19:09:03.606] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:03.649] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:03.654] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:03.822] [mlr3] Finished benchmark
## INFO [19:09:03.833] [bbotk] Result of batch 2:
## INFO [19:09:03.834] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:03.834] [bbotk] 375 0.7331808 0.8925926 0 0
## INFO [19:09:03.834] [bbotk] runtime_learners uhash
## INFO [19:09:03.834] [bbotk] 0.163 3738fe07-6139-4441-a536-fa7c858710ed
## INFO [19:09:03.835] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:03.855] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:03.858] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:04.008] [mlr3] Finished benchmark
## INFO [19:09:04.019] [bbotk] Result of batch 3:
## INFO [19:09:04.020] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:04.020] [bbotk] 319 0.7940582 0.8666667 0 0
## INFO [19:09:04.020] [bbotk] runtime_learners uhash
## INFO [19:09:04.020] [bbotk] 0.148 83a303ff-96a1-420f-b1e3-fa91f28592ac
## INFO [19:09:04.021] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:04.040] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:04.043] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:04.241] [mlr3] Finished benchmark
## INFO [19:09:04.252] [bbotk] Result of batch 4:
## INFO [19:09:04.253] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:04.253] [bbotk] 421 0.4656809 0.937037 0 0
## INFO [19:09:04.253] [bbotk] runtime_learners uhash
## INFO [19:09:04.253] [bbotk] 0.194 f6a00470-e26c-4cd5-970a-8f47e5206d53
## INFO [19:09:04.254] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:04.274] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:04.276] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:04.435] [mlr3] Finished benchmark
## INFO [19:09:04.447] [bbotk] Result of batch 5:
## INFO [19:09:04.447] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:04.447] [bbotk] 278 0.6287491 0.9222222 0 0
## INFO [19:09:04.447] [bbotk] runtime_learners uhash
## INFO [19:09:04.447] [bbotk] 0.155 6db56796-57b1-4198-9e30-7fa341d346b0
## INFO [19:09:04.449] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:04.469] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:04.472] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:04.623] [mlr3] Finished benchmark
## INFO [19:09:04.634] [bbotk] Result of batch 6:
## INFO [19:09:04.635] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:04.635] [bbotk] 294 0.5154666 0.9 0 0
## INFO [19:09:04.635] [bbotk] runtime_learners uhash
## INFO [19:09:04.635] [bbotk] 0.147 551250da-4fa4-49c5-9594-b8871c1ae97e
## INFO [19:09:04.636] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:04.681] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:04.684] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:04.843] [mlr3] Finished benchmark
## INFO [19:09:04.854] [bbotk] Result of batch 7:
## INFO [19:09:04.855] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:04.855] [bbotk] 260 0.4159474 0.9 0 0
## INFO [19:09:04.855] [bbotk] runtime_learners uhash
## INFO [19:09:04.855] [bbotk] 0.155 81077917-2cae-470f-9648-209bf121777d
## INFO [19:09:04.856] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:04.876] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:04.878] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:05.015] [mlr3] Finished benchmark
## INFO [19:09:05.026] [bbotk] Result of batch 8:
## INFO [19:09:05.027] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:05.027] [bbotk] 97 0.7941585 0.8703704 0 0
## INFO [19:09:05.027] [bbotk] runtime_learners uhash
## INFO [19:09:05.027] [bbotk] 0.134 52b49f8c-e6e1-46bc-9154-a3a6dad4e10b
## INFO [19:09:05.028] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:05.048] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:05.051] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:05.250] [mlr3] Finished benchmark
## INFO [19:09:05.262] [bbotk] Result of batch 9:
## INFO [19:09:05.262] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:05.262] [bbotk] 496 0.4047996 0.9074074 0 0
## INFO [19:09:05.262] [bbotk] runtime_learners uhash
## INFO [19:09:05.262] [bbotk] 0.195 137d3766-aa1e-4268-acbb-146dfe98293d
## INFO [19:09:05.264] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:05.284] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:05.286] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:05.443] [mlr3] Finished benchmark
## INFO [19:09:05.454] [bbotk] Result of batch 10:
## INFO [19:09:05.454] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:05.454] [bbotk] 365 0.7147423 0.8592593 0 0
## INFO [19:09:05.454] [bbotk] runtime_learners uhash
## INFO [19:09:05.454] [bbotk] 0.153 b8a319d5-f7e5-45b8-83b7-817f93a62870
## INFO [19:09:05.456] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:05.475] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:05.478] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:05.623] [mlr3] Finished benchmark
## INFO [19:09:05.645] [bbotk] Result of batch 11:
## INFO [19:09:05.647] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:05.647] [bbotk] 283 0.4788635 0.9111111 0 0
## INFO [19:09:05.647] [bbotk] runtime_learners uhash
## INFO [19:09:05.647] [bbotk] 0.143 2a888e6d-5b7a-4294-af21-008ee6ab409d
## INFO [19:09:05.649] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:05.688] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:05.692] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:05.841] [mlr3] Finished benchmark
## INFO [19:09:05.853] [bbotk] Result of batch 12:
## INFO [19:09:05.853] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:05.853] [bbotk] 176 0.4659971 0.862963 0 0
## INFO [19:09:05.853] [bbotk] runtime_learners uhash
## INFO [19:09:05.853] [bbotk] 0.145 9ef7347f-e367-4893-b189-4bce2a92e710
## INFO [19:09:05.855] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:05.875] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:05.878] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:06.036] [mlr3] Finished benchmark
## INFO [19:09:06.047] [bbotk] Result of batch 13:
## INFO [19:09:06.048] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:06.048] [bbotk] 418 0.7172111 0.8925926 0 0
## INFO [19:09:06.048] [bbotk] runtime_learners uhash
## INFO [19:09:06.048] [bbotk] 0.155 cffed4ef-2fc6-4eb5-b773-a4d8fafc222e
## INFO [19:09:06.049] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:06.069] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:06.071] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:06.256] [mlr3] Finished benchmark
## INFO [19:09:06.267] [bbotk] Result of batch 14:
## INFO [19:09:06.268] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:06.268] [bbotk] 260 0.8716583 0.9074074 0 0
## INFO [19:09:06.268] [bbotk] runtime_learners uhash
## INFO [19:09:06.268] [bbotk] 0.181 9797e895-6cda-43cd-af7e-b6e7db5dab35
## INFO [19:09:06.270] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:06.289] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:06.292] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:06.438] [mlr3] Finished benchmark
## INFO [19:09:06.449] [bbotk] Result of batch 15:
## INFO [19:09:06.450] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:06.450] [bbotk] 230 0.7133536 0.8259259 0 0
## INFO [19:09:06.450] [bbotk] runtime_learners uhash
## INFO [19:09:06.450] [bbotk] 0.143 86dff05e-e616-4c39-a033-02ae68c46b44
## INFO [19:09:06.451] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:06.471] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:06.473] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:06.652] [mlr3] Finished benchmark
## INFO [19:09:06.671] [bbotk] Result of batch 16:
## INFO [19:09:06.672] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:06.672] [bbotk] 446 0.7354419 0.8740741 0 0
## INFO [19:09:06.672] [bbotk] runtime_learners uhash
## INFO [19:09:06.672] [bbotk] 0.175 221596b0-79c2-43f9-b877-20bd3b977e15
## INFO [19:09:06.673] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:06.705] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:06.708] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:06.863] [mlr3] Finished benchmark
## INFO [19:09:06.874] [bbotk] Result of batch 17:
## INFO [19:09:06.875] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:06.875] [bbotk] 370 0.7011967 0.8740741 0 0
## INFO [19:09:06.875] [bbotk] runtime_learners uhash
## INFO [19:09:06.875] [bbotk] 0.152 3236162f-dee9-4d54-b582-f36e16a3a73f
## INFO [19:09:06.876] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:06.896] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:06.898] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:07.054] [mlr3] Finished benchmark
## INFO [19:09:07.065] [bbotk] Result of batch 18:
## INFO [19:09:07.066] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:07.066] [bbotk] 439 0.6894157 0.8703704 0 0
## INFO [19:09:07.066] [bbotk] runtime_learners uhash
## INFO [19:09:07.066] [bbotk] 0.152 4c2f96ab-2289-4fcb-99f9-bb5cc2887957
## INFO [19:09:07.068] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:07.087] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:07.089] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:07.267] [mlr3] Finished benchmark
## INFO [19:09:07.278] [bbotk] Result of batch 19:
## INFO [19:09:07.279] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:07.279] [bbotk] 202 0.4628167 0.9074074 0 0
## INFO [19:09:07.279] [bbotk] runtime_learners uhash
## INFO [19:09:07.279] [bbotk] 0.174 eafeec0a-54c6-48d8-b8d5-13ce5b7cdb55
## INFO [19:09:07.280] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:07.300] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:07.303] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:07.447] [mlr3] Finished benchmark
## INFO [19:09:07.458] [bbotk] Result of batch 20:
## INFO [19:09:07.458] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:07.458] [bbotk] 221 0.5227663 0.9074074 0 0
## INFO [19:09:07.458] [bbotk] runtime_learners uhash
## INFO [19:09:07.458] [bbotk] 0.14 83b65525-b0b9-4bf1-b7c2-33fd627ed3d4
## INFO [19:09:07.462] [bbotk] Finished optimizing after 20 evaluation(s)
## INFO [19:09:07.462] [bbotk] Result:
## INFO [19:09:07.463] [bbotk] classif.ranger.num.trees variance.filter.frac learner_param_vals x_domain
## INFO [19:09:07.463] [bbotk] 421 0.4656809 <list[8]> <list[2]>
## INFO [19:09:07.463] [bbotk] classif.auc
## INFO [19:09:07.463] [bbotk] 0.937037
## INFO [19:09:07.680] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger.tuned' on task 'sr' (iter 2/5)
## INFO [19:09:07.730] [bbotk] Starting to optimize 2 parameter(s) with '<OptimizerRandomSearch>' and '<TerminatorEvals> [n_evals=20, k=0]'
## INFO [19:09:07.734] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:07.754] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:07.757] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:07.896] [mlr3] Finished benchmark
## INFO [19:09:07.906] [bbotk] Result of batch 1:
## INFO [19:09:07.906] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:07.906] [bbotk] 133 0.7939841 0.7807692 0 0
## INFO [19:09:07.906] [bbotk] runtime_learners uhash
## INFO [19:09:07.906] [bbotk] 0.135 af516c62-8d25-4583-8430-060ea65a287f
## INFO [19:09:07.908] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:07.927] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:07.929] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:08.104] [mlr3] Finished benchmark
## INFO [19:09:08.116] [bbotk] Result of batch 2:
## INFO [19:09:08.117] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:08.117] [bbotk] 134 0.7361545 0.8192308 0 0
## INFO [19:09:08.117] [bbotk] runtime_learners uhash
## INFO [19:09:08.117] [bbotk] 0.171 f6c7219d-12f6-4f30-b10d-f99a73e3e938
## INFO [19:09:08.118] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:08.138] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:08.141] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:08.294] [mlr3] Finished benchmark
## INFO [19:09:08.305] [bbotk] Result of batch 3:
## INFO [19:09:08.306] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:08.306] [bbotk] 303 0.8989201 0.8 0 0
## INFO [19:09:08.306] [bbotk] runtime_learners uhash
## INFO [19:09:08.306] [bbotk] 0.149 1e79ce22-b486-4d0e-bffb-e6a41facddd1
## INFO [19:09:08.308] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:08.327] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:08.330] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:08.471] [mlr3] Finished benchmark
## INFO [19:09:08.482] [bbotk] Result of batch 4:
## INFO [19:09:08.483] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:08.483] [bbotk] 154 0.9342429 0.7807692 0 0
## INFO [19:09:08.483] [bbotk] runtime_learners uhash
## INFO [19:09:08.483] [bbotk] 0.139 baa5bd9e-9377-4a07-9514-99153e1e6a6d
## INFO [19:09:08.496] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:08.535] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:08.540] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:08.682] [mlr3] Finished benchmark
## INFO [19:09:08.693] [bbotk] Result of batch 5:
## INFO [19:09:08.694] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:08.694] [bbotk] 64 0.7116054 0.7269231 0 0
## INFO [19:09:08.694] [bbotk] runtime_learners uhash
## INFO [19:09:08.694] [bbotk] 0.137 1f5d644a-b286-434f-be0e-2ea561988192
## INFO [19:09:08.695] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:08.715] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:08.717] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:08.873] [mlr3] Finished benchmark
## INFO [19:09:08.885] [bbotk] Result of batch 6:
## INFO [19:09:08.885] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:08.885] [bbotk] 401 0.7784476 0.7884615 0 0
## INFO [19:09:08.885] [bbotk] runtime_learners uhash
## INFO [19:09:08.885] [bbotk] 0.153 5137318e-78c7-4ae6-8256-7b70f3249793
## INFO [19:09:08.887] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:08.906] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:08.908] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:09.087] [mlr3] Finished benchmark
## INFO [19:09:09.098] [bbotk] Result of batch 7:
## INFO [19:09:09.099] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:09.099] [bbotk] 177 0.5656856 0.8076923 0 0
## INFO [19:09:09.099] [bbotk] runtime_learners uhash
## INFO [19:09:09.099] [bbotk] 0.176 1af1c770-128e-4524-83a5-6a8da78b3074
## INFO [19:09:09.100] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:09.120] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:09.123] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:09.277] [mlr3] Finished benchmark
## INFO [19:09:09.288] [bbotk] Result of batch 8:
## INFO [19:09:09.289] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:09.289] [bbotk] 362 0.7128245 0.7884615 0 0
## INFO [19:09:09.289] [bbotk] runtime_learners uhash
## INFO [19:09:09.289] [bbotk] 0.151 0ac05459-bae9-4ab5-9d62-d63674d8c504
## INFO [19:09:09.290] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:09.309] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:09.312] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:09.504] [mlr3] Finished benchmark
## INFO [19:09:09.516] [bbotk] Result of batch 9:
## INFO [19:09:09.517] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:09.517] [bbotk] 367 0.6570939 0.8153846 0 0
## INFO [19:09:09.517] [bbotk] runtime_learners uhash
## INFO [19:09:09.517] [bbotk] 0.188 0bc0a4c6-2c96-4d74-ac51-e8dd414c98bb
## INFO [19:09:09.518] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:09.538] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:09.541] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:09.690] [mlr3] Finished benchmark
## INFO [19:09:09.701] [bbotk] Result of batch 10:
## INFO [19:09:09.702] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:09.702] [bbotk] 315 0.8706045 0.8307692 0 0
## INFO [19:09:09.702] [bbotk] runtime_learners uhash
## INFO [19:09:09.702] [bbotk] 0.147 d0e7d19e-6414-4db4-ab4c-5999e2cd4621
## INFO [19:09:09.703] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:09.723] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:09.725] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:09.877] [mlr3] Finished benchmark
## INFO [19:09:10.036] [bbotk] Result of batch 11:
## INFO [19:09:10.037] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:10.037] [bbotk] 395 0.49024 0.8346154 0 0
## INFO [19:09:10.037] [bbotk] runtime_learners uhash
## INFO [19:09:10.037] [bbotk] 0.149 1b2c9101-6590-4b14-ae87-2d319fb3aab5
## INFO [19:09:10.040] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:10.083] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:10.087] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:10.257] [mlr3] Finished benchmark
## INFO [19:09:10.268] [bbotk] Result of batch 12:
## INFO [19:09:10.269] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:10.269] [bbotk] 331 0.5230113 0.8461538 0 0
## INFO [19:09:10.269] [bbotk] runtime_learners uhash
## INFO [19:09:10.269] [bbotk] 0.168 05eb5385-4e8d-4e2c-b0ec-d6605293f22a
## INFO [19:09:10.270] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:10.291] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:10.293] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:10.430] [mlr3] Finished benchmark
## INFO [19:09:10.442] [bbotk] Result of batch 13:
## INFO [19:09:10.442] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:10.442] [bbotk] 144 0.5167291 0.8884615 0 0
## INFO [19:09:10.442] [bbotk] runtime_learners uhash
## INFO [19:09:10.442] [bbotk] 0.134 45978d0f-2212-45b7-a8a3-95149918617f
## INFO [19:09:10.444] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:10.465] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:10.468] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:10.675] [mlr3] Finished benchmark
## INFO [19:09:10.692] [bbotk] Result of batch 14:
## INFO [19:09:10.693] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:10.693] [bbotk] 349 0.9113472 0.7884615 0 0
## INFO [19:09:10.693] [bbotk] runtime_learners uhash
## INFO [19:09:10.693] [bbotk] 0.202 7147b328-c385-49f3-ab24-3512af3239eb
## INFO [19:09:10.695] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:10.716] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:10.719] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:10.863] [mlr3] Finished benchmark
## INFO [19:09:10.874] [bbotk] Result of batch 15:
## INFO [19:09:10.875] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:10.875] [bbotk] 226 0.5822165 0.8653846 0 0
## INFO [19:09:10.875] [bbotk] runtime_learners uhash
## INFO [19:09:10.875] [bbotk] 0.142 993af717-227e-4b91-97c7-89091f24af2e
## INFO [19:09:10.876] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:10.896] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:10.899] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:11.030] [mlr3] Finished benchmark
## INFO [19:09:11.042] [bbotk] Result of batch 16:
## INFO [19:09:11.042] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:11.042] [bbotk] 53 0.5088048 0.8038462 0 0
## INFO [19:09:11.042] [bbotk] runtime_learners uhash
## INFO [19:09:11.042] [bbotk] 0.128 12df10b0-49d9-4d10-aee8-f049ff8fa68b
## INFO [19:09:11.044] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:11.063] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:11.066] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:11.282] [mlr3] Finished benchmark
## INFO [19:09:11.293] [bbotk] Result of batch 17:
## INFO [19:09:11.293] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:11.293] [bbotk] 320 0.7603932 0.7730769 0 0
## INFO [19:09:11.293] [bbotk] runtime_learners uhash
## INFO [19:09:11.293] [bbotk] 0.213 2d78bd7c-4fb0-47f0-a343-363fc97fa118
## INFO [19:09:11.295] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:11.315] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:11.317] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:11.464] [mlr3] Finished benchmark
## INFO [19:09:11.475] [bbotk] Result of batch 18:
## INFO [19:09:11.476] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:11.476] [bbotk] 220 0.939498 0.7807692 0 0
## INFO [19:09:11.476] [bbotk] runtime_learners uhash
## INFO [19:09:11.476] [bbotk] 0.143 7eb97379-ab4b-420b-8b5d-639c33888d46
## INFO [19:09:11.477] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:11.497] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:11.500] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:11.684] [mlr3] Finished benchmark
## INFO [19:09:11.703] [bbotk] Result of batch 19:
## INFO [19:09:11.705] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:11.705] [bbotk] 399 0.682273 0.8076923 0 0
## INFO [19:09:11.705] [bbotk] runtime_learners uhash
## INFO [19:09:11.705] [bbotk] 0.181 ac41989a-3e6b-46d8-beb8-d6cc4eb5b177
## INFO [19:09:11.707] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:11.745] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:11.748] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:11.913] [mlr3] Finished benchmark
## INFO [19:09:11.924] [bbotk] Result of batch 20:
## INFO [19:09:11.925] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:11.925] [bbotk] 466 0.6455269 0.8153846 0 0
## INFO [19:09:11.925] [bbotk] runtime_learners uhash
## INFO [19:09:11.925] [bbotk] 0.16 bf51ec24-e8c7-43f1-b426-dc81bc28eb09
## INFO [19:09:11.929] [bbotk] Finished optimizing after 20 evaluation(s)
## INFO [19:09:11.929] [bbotk] Result:
## INFO [19:09:11.930] [bbotk] classif.ranger.num.trees variance.filter.frac learner_param_vals x_domain
## INFO [19:09:11.930] [bbotk] 144 0.5167291 <list[8]> <list[2]>
## INFO [19:09:11.930] [bbotk] classif.auc
## INFO [19:09:11.930] [bbotk] 0.8884615
## INFO [19:09:12.096] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger.tuned' on task 'sr' (iter 3/5)
## INFO [19:09:12.144] [bbotk] Starting to optimize 2 parameter(s) with '<OptimizerRandomSearch>' and '<TerminatorEvals> [n_evals=20, k=0]'
## INFO [19:09:12.164] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:12.208] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:12.211] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:12.377] [mlr3] Finished benchmark
## INFO [19:09:12.388] [bbotk] Result of batch 1:
## INFO [19:09:12.389] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:12.389] [bbotk] 160 0.8989059 0.8823529 0 0
## INFO [19:09:12.389] [bbotk] runtime_learners uhash
## INFO [19:09:12.389] [bbotk] 0.163 dabe6b40-f909-422d-a9f7-c09347f9867f
## INFO [19:09:12.390] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:12.411] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:12.413] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:12.557] [mlr3] Finished benchmark
## INFO [19:09:12.569] [bbotk] Result of batch 2:
## INFO [19:09:12.569] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:12.569] [bbotk] 122 0.6263972 0.9338235 0 0
## INFO [19:09:12.569] [bbotk] runtime_learners uhash
## INFO [19:09:12.569] [bbotk] 0.14 40f9a155-3f8e-4071-a932-c266682ea7da
## INFO [19:09:12.571] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:12.591] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:12.594] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:12.805] [mlr3] Finished benchmark
## INFO [19:09:12.818] [bbotk] Result of batch 3:
## INFO [19:09:12.818] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:12.818] [bbotk] 271 0.462555 0.8970588 0 0
## INFO [19:09:12.818] [bbotk] runtime_learners uhash
## INFO [19:09:12.818] [bbotk] 0.208 6d871004-bc55-4d1b-8b25-c6981d36ec92
## INFO [19:09:12.820] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:12.841] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:12.844] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:12.999] [mlr3] Finished benchmark
## INFO [19:09:13.011] [bbotk] Result of batch 4:
## INFO [19:09:13.011] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:13.011] [bbotk] 325 0.5932042 0.9117647 0 0
## INFO [19:09:13.011] [bbotk] runtime_learners uhash
## INFO [19:09:13.011] [bbotk] 0.152 5a70448a-79c4-4cdf-9d88-71fb20f00ee9
## INFO [19:09:13.013] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:13.032] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:13.035] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:13.168] [mlr3] Finished benchmark
## INFO [19:09:13.179] [bbotk] Result of batch 5:
## INFO [19:09:13.180] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:13.180] [bbotk] 63 0.4859224 0.8897059 0 0
## INFO [19:09:13.180] [bbotk] runtime_learners uhash
## INFO [19:09:13.180] [bbotk] 0.131 5d8156b1-6de8-4005-bdd4-ca84dcaca744
## INFO [19:09:13.181] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:13.226] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:13.230] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:13.403] [mlr3] Finished benchmark
## INFO [19:09:13.415] [bbotk] Result of batch 6:
## INFO [19:09:13.415] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:13.415] [bbotk] 127 0.6826914 0.8823529 0 0
## INFO [19:09:13.415] [bbotk] runtime_learners uhash
## INFO [19:09:13.415] [bbotk] 0.169 ea22ce54-77ff-4385-970a-66c9b13a4fe5
## INFO [19:09:13.417] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:13.437] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:13.440] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:13.600] [mlr3] Finished benchmark
## INFO [19:09:13.612] [bbotk] Result of batch 7:
## INFO [19:09:13.613] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:13.613] [bbotk] 430 0.6012684 0.9375 0 0
## INFO [19:09:13.613] [bbotk] runtime_learners uhash
## INFO [19:09:13.613] [bbotk] 0.156 a7f1c608-8839-4fe5-b761-caf3451a8826
## INFO [19:09:13.614] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:13.634] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:13.636] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:13.836] [mlr3] Finished benchmark
## INFO [19:09:13.850] [bbotk] Result of batch 8:
## INFO [19:09:13.851] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:13.851] [bbotk] 214 0.7859189 0.9338235 0 0
## INFO [19:09:13.851] [bbotk] runtime_learners uhash
## INFO [19:09:13.851] [bbotk] 0.195 8149dad2-22e9-427d-bda9-8353c15c3897
## INFO [19:09:13.852] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:13.873] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:13.876] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:14.031] [mlr3] Finished benchmark
## INFO [19:09:14.042] [bbotk] Result of batch 9:
## INFO [19:09:14.043] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:14.043] [bbotk] 370 0.8972344 0.9044118 0 0
## INFO [19:09:14.043] [bbotk] runtime_learners uhash
## INFO [19:09:14.043] [bbotk] 0.153 40a2edca-0e6b-4897-9f3f-61c457205119
## INFO [19:09:14.045] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:14.064] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:14.067] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:14.206] [mlr3] Finished benchmark
## INFO [19:09:14.217] [bbotk] Result of batch 10:
## INFO [19:09:14.218] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:14.218] [bbotk] 143 0.866334 0.8602941 0 0
## INFO [19:09:14.218] [bbotk] runtime_learners uhash
## INFO [19:09:14.218] [bbotk] 0.135 f6d5c60c-cfba-4f38-9901-4183fff54662
## INFO [19:09:14.219] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:14.263] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:14.267] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:14.450] [mlr3] Finished benchmark
## INFO [19:09:14.461] [bbotk] Result of batch 11:
## INFO [19:09:14.462] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:14.462] [bbotk] 294 0.8126185 0.9264706 0 0
## INFO [19:09:14.462] [bbotk] runtime_learners uhash
## INFO [19:09:14.462] [bbotk] 0.18 74ca3686-3934-4668-a01a-282b9476a175
## INFO [19:09:14.463] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:14.483] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:14.486] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:14.629] [mlr3] Finished benchmark
## INFO [19:09:14.640] [bbotk] Result of batch 12:
## INFO [19:09:14.641] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:14.641] [bbotk] 177 0.8937534 0.8676471 0 0
## INFO [19:09:14.641] [bbotk] runtime_learners uhash
## INFO [19:09:14.641] [bbotk] 0.138 336e4b38-e0a9-412d-bdbd-078614308418
## INFO [19:09:14.642] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:14.662] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:14.664] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:14.867] [mlr3] Finished benchmark
## INFO [19:09:14.879] [bbotk] Result of batch 13:
## INFO [19:09:14.880] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:14.880] [bbotk] 254 0.7805551 0.9411765 0 0
## INFO [19:09:14.880] [bbotk] runtime_learners uhash
## INFO [19:09:14.880] [bbotk] 0.198 1a20b4f4-adf8-42b2-844a-bf12951581a8
## INFO [19:09:14.882] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:14.902] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:14.905] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:15.062] [mlr3] Finished benchmark
## INFO [19:09:15.073] [bbotk] Result of batch 14:
## INFO [19:09:15.074] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:15.074] [bbotk] 409 0.5903381 0.9411765 0 0
## INFO [19:09:15.074] [bbotk] runtime_learners uhash
## INFO [19:09:15.074] [bbotk] 0.154 5d5315b0-3b37-4627-9751-b4bf90c06358
## INFO [19:09:15.075] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:15.095] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:15.098] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:15.236] [mlr3] Finished benchmark
## INFO [19:09:15.247] [bbotk] Result of batch 15:
## INFO [19:09:15.248] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:15.248] [bbotk] 115 0.7548351 0.8161765 0 0
## INFO [19:09:15.248] [bbotk] runtime_learners uhash
## INFO [19:09:15.248] [bbotk] 0.135 9d134ddc-c83e-4833-b34c-ec0619b71f10
## INFO [19:09:15.249] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:15.301] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:15.306] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:15.473] [mlr3] Finished benchmark
## INFO [19:09:15.484] [bbotk] Result of batch 16:
## INFO [19:09:15.485] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:15.485] [bbotk] 86 0.9491571 0.8419118 0 0
## INFO [19:09:15.485] [bbotk] runtime_learners uhash
## INFO [19:09:15.485] [bbotk] 0.162 50762b50-6623-4401-b3ea-29ea3a81487c
## INFO [19:09:15.486] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:15.507] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:15.510] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:15.672] [mlr3] Finished benchmark
## INFO [19:09:15.684] [bbotk] Result of batch 17:
## INFO [19:09:15.685] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:15.685] [bbotk] 482 0.4663785 0.9338235 0 0
## INFO [19:09:15.685] [bbotk] runtime_learners uhash
## INFO [19:09:15.685] [bbotk] 0.16 27f86dac-64c2-43c2-8919-fde1dd3a5d01
## INFO [19:09:15.686] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:15.707] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:15.710] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:15.907] [mlr3] Finished benchmark
## INFO [19:09:15.920] [bbotk] Result of batch 18:
## INFO [19:09:15.921] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:15.921] [bbotk] 153 0.4245749 0.9227941 0 0
## INFO [19:09:15.921] [bbotk] runtime_learners uhash
## INFO [19:09:15.921] [bbotk] 0.194 f1dee2b5-887f-4710-9f45-de6e05faee07
## INFO [19:09:15.922] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:15.942] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:15.945] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:16.085] [mlr3] Finished benchmark
## INFO [19:09:16.096] [bbotk] Result of batch 19:
## INFO [19:09:16.097] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:16.097] [bbotk] 127 0.4929188 0.9080882 0 0
## INFO [19:09:16.097] [bbotk] runtime_learners uhash
## INFO [19:09:16.097] [bbotk] 0.137 8b5cd22e-eefb-4d72-8ed4-9ebdc31dca9d
## INFO [19:09:16.099] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:16.118] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:16.121] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:16.267] [mlr3] Finished benchmark
## INFO [19:09:16.287] [bbotk] Result of batch 20:
## INFO [19:09:16.288] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:16.288] [bbotk] 57 0.8641574 0.9411765 0 0
## INFO [19:09:16.288] [bbotk] runtime_learners uhash
## INFO [19:09:16.288] [bbotk] 0.142 ce27bd9b-712e-4fc4-9fee-10a8ad6c6511
## INFO [19:09:16.296] [bbotk] Finished optimizing after 20 evaluation(s)
## INFO [19:09:16.296] [bbotk] Result:
## INFO [19:09:16.297] [bbotk] classif.ranger.num.trees variance.filter.frac learner_param_vals x_domain
## INFO [19:09:16.297] [bbotk] 254 0.7805551 <list[8]> <list[2]>
## INFO [19:09:16.297] [bbotk] classif.auc
## INFO [19:09:16.297] [bbotk] 0.9411765
## INFO [19:09:16.504] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger.tuned' on task 'sr' (iter 4/5)
## INFO [19:09:16.554] [bbotk] Starting to optimize 2 parameter(s) with '<OptimizerRandomSearch>' and '<TerminatorEvals> [n_evals=20, k=0]'
## INFO [19:09:16.559] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:16.579] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:16.581] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:16.725] [mlr3] Finished benchmark
## INFO [19:09:16.750] [bbotk] Result of batch 1:
## INFO [19:09:16.751] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:16.751] [bbotk] 207 0.6256982 0.9038462 0 0
## INFO [19:09:16.751] [bbotk] runtime_learners uhash
## INFO [19:09:16.751] [bbotk] 0.139 0f42a2fe-aa4d-4fdf-aa13-61d0d40e7ed3
## INFO [19:09:16.754] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:16.795] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:16.799] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:16.967] [mlr3] Finished benchmark
## INFO [19:09:16.979] [bbotk] Result of batch 2:
## INFO [19:09:16.980] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:16.980] [bbotk] 334 0.4967597 0.9230769 0 0
## INFO [19:09:16.980] [bbotk] runtime_learners uhash
## INFO [19:09:16.980] [bbotk] 0.164 596b43b0-b800-453d-a5bc-e89da6adfe39
## INFO [19:09:16.981] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:17.001] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:17.003] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:17.159] [mlr3] Finished benchmark
## INFO [19:09:17.170] [bbotk] Result of batch 3:
## INFO [19:09:17.171] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:17.171] [bbotk] 372 0.8839681 0.9076923 0 0
## INFO [19:09:17.171] [bbotk] runtime_learners uhash
## INFO [19:09:17.171] [bbotk] 0.153 913a586a-76db-46ef-8fc9-f0bd401107a6
## INFO [19:09:17.173] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:17.192] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:17.195] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:17.404] [mlr3] Finished benchmark
## INFO [19:09:17.416] [bbotk] Result of batch 4:
## INFO [19:09:17.416] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:17.416] [bbotk] 375 0.4548788 0.9038462 0 0
## INFO [19:09:17.416] [bbotk] runtime_learners uhash
## INFO [19:09:17.416] [bbotk] 0.206 5170593c-086d-4746-99c8-615675e3e48d
## INFO [19:09:17.418] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:17.437] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:17.440] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:17.589] [mlr3] Finished benchmark
## INFO [19:09:17.600] [bbotk] Result of batch 5:
## INFO [19:09:17.601] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:17.601] [bbotk] 205 0.8766663 0.85 0 0
## INFO [19:09:17.601] [bbotk] runtime_learners uhash
## INFO [19:09:17.601] [bbotk] 0.146 bd7205df-dde3-488e-90b4-6bb0979f2b02
## INFO [19:09:17.602] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:17.622] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:17.625] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:17.817] [mlr3] Finished benchmark
## INFO [19:09:17.834] [bbotk] Result of batch 6:
## INFO [19:09:17.836] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:17.836] [bbotk] 427 0.4839079 0.9 0 0
## INFO [19:09:17.836] [bbotk] runtime_learners uhash
## INFO [19:09:17.836] [bbotk] 0.19 6e2db527-3940-4d6c-bb16-6d38ac7921d5
## INFO [19:09:17.838] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:17.869] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:17.872] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:18.024] [mlr3] Finished benchmark
## INFO [19:09:18.035] [bbotk] Result of batch 7:
## INFO [19:09:18.036] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:18.036] [bbotk] 311 0.557269 0.9 0 0
## INFO [19:09:18.036] [bbotk] runtime_learners uhash
## INFO [19:09:18.036] [bbotk] 0.148 8ea28c4a-3bf3-46de-85fc-098511456fe4
## INFO [19:09:18.037] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:18.057] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:18.060] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:18.206] [mlr3] Finished benchmark
## INFO [19:09:18.217] [bbotk] Result of batch 8:
## INFO [19:09:18.217] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:18.217] [bbotk] 261 0.66916 0.9 0 0
## INFO [19:09:18.217] [bbotk] runtime_learners uhash
## INFO [19:09:18.217] [bbotk] 0.142 1e5b5f42-822b-41a5-940f-e8918d25b5db
## INFO [19:09:18.219] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:18.259] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:18.263] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:18.448] [mlr3] Finished benchmark
## INFO [19:09:18.459] [bbotk] Result of batch 9:
## INFO [19:09:18.460] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:18.460] [bbotk] 305 0.798773 0.8615385 0 0
## INFO [19:09:18.460] [bbotk] runtime_learners uhash
## INFO [19:09:18.460] [bbotk] 0.179 73605aac-9edd-4868-9292-8e1cc741e967
## INFO [19:09:18.461] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:18.482] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:18.484] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:18.638] [mlr3] Finished benchmark
## INFO [19:09:18.649] [bbotk] Result of batch 10:
## INFO [19:09:18.650] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:18.650] [bbotk] 380 0.667818 0.9153846 0 0
## INFO [19:09:18.650] [bbotk] runtime_learners uhash
## INFO [19:09:18.650] [bbotk] 0.15 c2925d63-d343-481c-b3aa-b2b23a13f680
## INFO [19:09:18.651] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:18.670] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:18.673] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:18.875] [mlr3] Finished benchmark
## INFO [19:09:18.887] [bbotk] Result of batch 11:
## INFO [19:09:18.888] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:18.888] [bbotk] 211 0.9221721 0.7961538 0 0
## INFO [19:09:18.888] [bbotk] runtime_learners uhash
## INFO [19:09:18.888] [bbotk] 0.199 d2935ff9-0d6e-437f-a617-b4c23d929c1d
## INFO [19:09:18.889] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:18.909] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:18.911] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:19.076] [mlr3] Finished benchmark
## INFO [19:09:19.087] [bbotk] Result of batch 12:
## INFO [19:09:19.088] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:19.088] [bbotk] 476 0.4536144 0.8807692 0 0
## INFO [19:09:19.088] [bbotk] runtime_learners uhash
## INFO [19:09:19.088] [bbotk] 0.161 6a04fa2a-aa9d-407b-8563-207531c8cb9e
## INFO [19:09:19.089] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:19.109] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:19.111] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:19.282] [mlr3] Finished benchmark
## INFO [19:09:19.300] [bbotk] Result of batch 13:
## INFO [19:09:19.301] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:19.301] [bbotk] 94 0.80941 0.8038462 0 0
## INFO [19:09:19.301] [bbotk] runtime_learners uhash
## INFO [19:09:19.301] [bbotk] 0.167 a0bc50eb-9e78-4e09-8962-026d6c5380ca
## INFO [19:09:19.304] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:19.338] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:19.341] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:19.492] [mlr3] Finished benchmark
## INFO [19:09:19.504] [bbotk] Result of batch 14:
## INFO [19:09:19.504] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:19.504] [bbotk] 173 0.6282673 0.8730769 0 0
## INFO [19:09:19.504] [bbotk] runtime_learners uhash
## INFO [19:09:19.504] [bbotk] 0.147 ec97c54e-d3e1-42e1-924f-d5a91466b808
## INFO [19:09:19.506] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:19.526] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:19.528] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:19.666] [mlr3] Finished benchmark
## INFO [19:09:19.677] [bbotk] Result of batch 15:
## INFO [19:09:19.678] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:19.678] [bbotk] 129 0.6188582 0.9230769 0 0
## INFO [19:09:19.678] [bbotk] runtime_learners uhash
## INFO [19:09:19.678] [bbotk] 0.135 9659dca0-16df-4e6a-84e2-61879dbb1ef6
## INFO [19:09:19.680] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:19.721] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:19.726] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:19.896] [mlr3] Finished benchmark
## INFO [19:09:19.908] [bbotk] Result of batch 16:
## INFO [19:09:19.908] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:19.908] [bbotk] 68 0.6691631 0.9346154 0 0
## INFO [19:09:19.908] [bbotk] runtime_learners uhash
## INFO [19:09:19.908] [bbotk] 0.167 d6533f51-9438-42aa-aaa2-3fefd63f6175
## INFO [19:09:19.910] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:19.930] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:19.933] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:20.078] [mlr3] Finished benchmark
## INFO [19:09:20.089] [bbotk] Result of batch 17:
## INFO [19:09:20.090] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:20.090] [bbotk] 171 0.8121644 0.9192308 0 0
## INFO [19:09:20.090] [bbotk] runtime_learners uhash
## INFO [19:09:20.090] [bbotk] 0.143 a4f55ea4-8428-4de5-b753-ed4483c1db11
## INFO [19:09:20.092] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:20.111] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:20.114] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:20.309] [mlr3] Finished benchmark
## INFO [19:09:20.320] [bbotk] Result of batch 18:
## INFO [19:09:20.321] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:20.321] [bbotk] 105 0.4382572 0.8923077 0 0
## INFO [19:09:20.321] [bbotk] runtime_learners uhash
## INFO [19:09:20.321] [bbotk] 0.191 f65becde-4ef6-4c09-9cdb-39de4143e538
## INFO [19:09:20.323] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:20.343] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:20.345] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:20.504] [mlr3] Finished benchmark
## INFO [19:09:20.516] [bbotk] Result of batch 19:
## INFO [19:09:20.516] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:20.516] [bbotk] 366 0.9140019 0.8461538 0 0
## INFO [19:09:20.516] [bbotk] runtime_learners uhash
## INFO [19:09:20.516] [bbotk] 0.155 5f5d834c-c5af-4563-81a7-42860567d57b
## INFO [19:09:20.518] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:20.537] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:20.540] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:21.044] [mlr3] Finished benchmark
## INFO [19:09:21.054] [bbotk] Result of batch 20:
## INFO [19:09:21.055] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:21.055] [bbotk] 358 0.4445884 0.9038462 0 0
## INFO [19:09:21.055] [bbotk] runtime_learners uhash
## INFO [19:09:21.055] [bbotk] 0.501 f4e9db63-73c2-4e62-8b6d-de8eba5d4735
## INFO [19:09:21.059] [bbotk] Finished optimizing after 20 evaluation(s)
## INFO [19:09:21.059] [bbotk] Result:
## INFO [19:09:21.059] [bbotk] classif.ranger.num.trees variance.filter.frac learner_param_vals x_domain
## INFO [19:09:21.059] [bbotk] 68 0.6691631 <list[8]> <list[2]>
## INFO [19:09:21.059] [bbotk] classif.auc
## INFO [19:09:21.059] [bbotk] 0.9346154
## INFO [19:09:21.210] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger.tuned' on task 'sr' (iter 5/5)
## INFO [19:09:21.257] [bbotk] Starting to optimize 2 parameter(s) with '<OptimizerRandomSearch>' and '<TerminatorEvals> [n_evals=20, k=0]'
## INFO [19:09:21.262] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:21.281] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:21.283] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:21.431] [mlr3] Finished benchmark
## INFO [19:09:21.441] [bbotk] Result of batch 1:
## INFO [19:09:21.442] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:21.442] [bbotk] 304 0.7806328 0.9080882 0 0
## INFO [19:09:21.442] [bbotk] runtime_learners uhash
## INFO [19:09:21.442] [bbotk] 0.143 7c655bbb-832b-402a-83bf-5c3aee1d2319
## INFO [19:09:21.443] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:21.471] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:21.474] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:21.611] [mlr3] Finished benchmark
## INFO [19:09:21.622] [bbotk] Result of batch 2:
## INFO [19:09:21.623] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:21.623] [bbotk] 173 0.7418451 0.8602941 0 0
## INFO [19:09:21.623] [bbotk] runtime_learners uhash
## INFO [19:09:21.623] [bbotk] 0.133 1bbeeb56-ef8e-4c84-91ba-b68a2ec17e68
## INFO [19:09:21.624] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:21.643] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:21.646] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:21.777] [mlr3] Finished benchmark
## INFO [19:09:21.787] [bbotk] Result of batch 3:
## INFO [19:09:21.788] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:21.788] [bbotk] 68 0.8037831 0.8602941 0 0
## INFO [19:09:21.788] [bbotk] runtime_learners uhash
## INFO [19:09:21.788] [bbotk] 0.129 0fbb5117-8d3d-4975-8960-9e94199d0520
## INFO [19:09:21.789] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:21.808] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:21.811] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:21.965] [mlr3] Finished benchmark
## INFO [19:09:21.976] [bbotk] Result of batch 4:
## INFO [19:09:21.977] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:21.977] [bbotk] 260 0.8462173 0.9117647 0 0
## INFO [19:09:21.977] [bbotk] runtime_learners uhash
## INFO [19:09:21.977] [bbotk] 0.143 0ee55de0-4cf1-48d9-ad72-972c40d0953c
## INFO [19:09:21.979] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:21.998] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:22.000] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:22.136] [mlr3] Finished benchmark
## INFO [19:09:22.147] [bbotk] Result of batch 5:
## INFO [19:09:22.148] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:22.148] [bbotk] 136 0.7883893 0.8823529 0 0
## INFO [19:09:22.148] [bbotk] runtime_learners uhash
## INFO [19:09:22.148] [bbotk] 0.131 df3e1a65-e961-4116-a4a0-720684353ebc
## INFO [19:09:22.150] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:22.168] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:22.171] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:22.311] [mlr3] Finished benchmark
## INFO [19:09:22.323] [bbotk] Result of batch 6:
## INFO [19:09:22.323] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:22.323] [bbotk] 183 0.4962333 0.8786765 0 0
## INFO [19:09:22.323] [bbotk] runtime_learners uhash
## INFO [19:09:22.323] [bbotk] 0.138 8af2fc7d-dfca-426a-9ecb-7f6d71690f96
## INFO [19:09:22.325] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:22.344] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:22.347] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:22.495] [mlr3] Finished benchmark
## INFO [19:09:22.505] [bbotk] Result of batch 7:
## INFO [19:09:22.506] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:22.506] [bbotk] 186 0.5851934 0.8639706 0 0
## INFO [19:09:22.506] [bbotk] runtime_learners uhash
## INFO [19:09:22.506] [bbotk] 0.145 af6bfae8-f0db-4ad4-ac22-b69b98832c52
## INFO [19:09:22.507] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:22.526] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:22.529] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:22.671] [mlr3] Finished benchmark
## INFO [19:09:22.682] [bbotk] Result of batch 8:
## INFO [19:09:22.682] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:22.682] [bbotk] 209 0.8744202 0.8970588 0 0
## INFO [19:09:22.682] [bbotk] runtime_learners uhash
## INFO [19:09:22.682] [bbotk] 0.139 e570a1a9-bf88-4820-89c9-a564772d4067
## INFO [19:09:22.684] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:22.703] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:22.705] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:22.845] [mlr3] Finished benchmark
## INFO [19:09:22.857] [bbotk] Result of batch 9:
## INFO [19:09:22.857] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:22.857] [bbotk] 187 0.7766667 0.8566176 0 0
## INFO [19:09:22.857] [bbotk] runtime_learners uhash
## INFO [19:09:22.857] [bbotk] 0.137 d00882bc-20d4-45f4-90ff-4c1971744947
## INFO [19:09:22.859] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:22.887] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:22.889] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:23.026] [mlr3] Finished benchmark
## INFO [19:09:23.037] [bbotk] Result of batch 10:
## INFO [19:09:23.038] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:23.038] [bbotk] 175 0.7832197 0.8897059 0 0
## INFO [19:09:23.038] [bbotk] runtime_learners uhash
## INFO [19:09:23.038] [bbotk] 0.133 10c54cc0-c037-4c79-a169-2be59a9ada7f
## INFO [19:09:23.039] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:23.058] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:23.060] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:23.201] [mlr3] Finished benchmark
## INFO [19:09:23.212] [bbotk] Result of batch 11:
## INFO [19:09:23.212] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:23.212] [bbotk] 245 0.5867956 0.8566176 0 0
## INFO [19:09:23.212] [bbotk] runtime_learners uhash
## INFO [19:09:23.212] [bbotk] 0.137 92cb458e-e549-4183-9fec-916e777a9c3a
## INFO [19:09:23.214] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:23.232] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:23.235] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:23.388] [mlr3] Finished benchmark
## INFO [19:09:23.399] [bbotk] Result of batch 12:
## INFO [19:09:23.399] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:23.399] [bbotk] 247 0.7871251 0.8933824 0 0
## INFO [19:09:23.399] [bbotk] runtime_learners uhash
## INFO [19:09:23.399] [bbotk] 0.151 bf18c085-d4f6-4b8c-9fd6-99eb6bc561de
## INFO [19:09:23.401] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:23.420] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:23.422] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:23.571] [mlr3] Finished benchmark
## INFO [19:09:23.582] [bbotk] Result of batch 13:
## INFO [19:09:23.582] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:23.582] [bbotk] 410 0.4805939 0.8823529 0 0
## INFO [19:09:23.582] [bbotk] runtime_learners uhash
## INFO [19:09:23.582] [bbotk] 0.145 7c6034e7-7c9d-42b5-8d14-47aacfca4aff
## INFO [19:09:23.584] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:23.602] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:23.605] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:23.747] [mlr3] Finished benchmark
## INFO [19:09:23.758] [bbotk] Result of batch 14:
## INFO [19:09:23.758] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:23.758] [bbotk] 260 0.5287239 0.875 0 0
## INFO [19:09:23.758] [bbotk] runtime_learners uhash
## INFO [19:09:23.758] [bbotk] 0.14 b06431df-1fe1-4b8d-8412-81bd3ee421b5
## INFO [19:09:23.760] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:23.779] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:23.781] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:23.923] [mlr3] Finished benchmark
## INFO [19:09:23.934] [bbotk] Result of batch 15:
## INFO [19:09:23.935] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:23.935] [bbotk] 81 0.8776481 0.9080882 0 0
## INFO [19:09:23.935] [bbotk] runtime_learners uhash
## INFO [19:09:23.935] [bbotk] 0.139 7a092368-2e07-4a99-a15d-4b6a9fab9ae9
## INFO [19:09:23.936] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:23.955] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:23.957] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:24.093] [mlr3] Finished benchmark
## INFO [19:09:24.104] [bbotk] Result of batch 16:
## INFO [19:09:24.105] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:24.105] [bbotk] 163 0.4491448 0.8676471 0 0
## INFO [19:09:24.105] [bbotk] runtime_learners uhash
## INFO [19:09:24.105] [bbotk] 0.132 f581d38c-f489-4cbf-97b2-9942517f80c8
## INFO [19:09:24.106] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:24.125] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:24.128] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:24.265] [mlr3] Finished benchmark
## INFO [19:09:24.285] [bbotk] Result of batch 17:
## INFO [19:09:24.285] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:24.285] [bbotk] 155 0.4525341 0.8933824 0 0
## INFO [19:09:24.285] [bbotk] runtime_learners uhash
## INFO [19:09:24.285] [bbotk] 0.135 740e589c-ae7d-4820-8836-b8f0ec262c17
## INFO [19:09:24.287] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:24.305] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:24.308] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:24.451] [mlr3] Finished benchmark
## INFO [19:09:24.462] [bbotk] Result of batch 18:
## INFO [19:09:24.462] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:24.462] [bbotk] 299 0.4108343 0.8786765 0 0
## INFO [19:09:24.462] [bbotk] runtime_learners uhash
## INFO [19:09:24.462] [bbotk] 0.14 21e193e4-c0f5-48d0-be93-5bdbabcb9ac9
## INFO [19:09:24.464] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:24.483] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:24.486] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:24.642] [mlr3] Finished benchmark
## INFO [19:09:24.653] [bbotk] Result of batch 19:
## INFO [19:09:24.654] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:24.654] [bbotk] 468 0.530288 0.8786765 0 0
## INFO [19:09:24.654] [bbotk] runtime_learners uhash
## INFO [19:09:24.654] [bbotk] 0.154 6298ee60-41c4-4fc6-b844-69b724bf98b7
## INFO [19:09:24.656] [bbotk] Evaluating 1 configuration(s)
## INFO [19:09:24.675] [mlr3] Running benchmark with 1 resampling iterations
## INFO [19:09:24.677] [mlr3] Applying learner 'imputemedian.scalerange.pca.variance.classif.ranger' on task 'sr' (iter 1/1)
## INFO [19:09:24.829] [mlr3] Finished benchmark
## INFO [19:09:24.840] [bbotk] Result of batch 20:
## INFO [19:09:24.841] [bbotk] classif.ranger.num.trees variance.filter.frac classif.auc warnings errors
## INFO [19:09:24.841] [bbotk] 228 0.7041795 0.8933824 0 0
## INFO [19:09:24.841] [bbotk] runtime_learners uhash
## INFO [19:09:24.841] [bbotk] 0.149 3d4a6878-74ce-4519-af53-5de9fcc9b82d
## INFO [19:09:24.844] [bbotk] Finished optimizing after 20 evaluation(s)
## INFO [19:09:24.845] [bbotk] Result:
## INFO [19:09:24.845] [bbotk] classif.ranger.num.trees variance.filter.frac learner_param_vals x_domain
## INFO [19:09:24.845] [bbotk] 260 0.8462173 <list[8]> <list[2]>
## INFO [19:09:24.845] [bbotk] classif.auc
## INFO [19:09:24.845] [bbotk] 0.9117647Step 4: Extract Inner Tuning Results
## Inner Tuning Results
extract_inner_tuning_results(sr_rf)## iteration classif.ranger.num.trees variance.filter.frac classif.auc
## 1: 1 421 0.4656809 0.9370370
## 2: 2 144 0.5167291 0.8884615
## 3: 3 254 0.7805551 0.9411765
## 4: 4 68 0.6691631 0.9346154
## 5: 5 260 0.8462173 0.9117647
## learner_param_vals x_domain task_id
## 1: <list[8]> <list[2]> sr
## 2: <list[8]> <list[2]> sr
## 3: <list[8]> <list[2]> sr
## 4: <list[8]> <list[2]> sr
## 5: <list[8]> <list[2]> sr
## learner_id resampling_id
## 1: imputemedian.scalerange.pca.variance.classif.ranger.tuned cv
## 2: imputemedian.scalerange.pca.variance.classif.ranger.tuned cv
## 3: imputemedian.scalerange.pca.variance.classif.ranger.tuned cv
## 4: imputemedian.scalerange.pca.variance.classif.ranger.tuned cv
## 5: imputemedian.scalerange.pca.variance.classif.ranger.tuned cvStep 5: Extract outer tuning results
## Outer tuning results
## Test Data AUC
sr_rf$score(sr_msr_auc)[ , list(iteration, classif.auc)]## iteration classif.auc
## 1: 1 0.8832952
## 2: 2 0.9120370
## 3: 3 0.8796296
## 4: 4 0.8872549
## 5: 5 0.9076923Overall Interpretation:
- The pipeline used for processing this data is shown in the following diagram:
- The result of our model tune includes the following hyperparameter values:
- Number of Trees: see the tuning result hyperparameter section above
- Percentage of the Variance Explained: see the tuning result hyperparameter section above
- The percentage of the variance explained results in using 25 principle components for the model.
- The cross-validated AUC results include the following values:
- Inner CV results: see Performance Measure - Step 4
- Outer CV results: see Performance Measure - Step 5
It is important to note that a mismatch exists between our inner and outer AUC scores as the outer cross-validation considers the predictive ability of the model on the test data.