Intro to Machine Learning
Data Availabilty
R code has different ways to upload different types of data into the R environment. R also has different ways to manipulate the data. One way to du it is to use library(tidyverse). Within tidyverse there are different commands to manipulate data.
How to Start with the Statistical Package R
You will learn to work with R-Studio, how to organize data in a clean and tidy way, and about different data types.
- Install R and RStudio
- Work with R Studio
- R-Script window
- Command Window
- Environment window
- R-Script window
- Data Types (quantitative, text, categorical/factors)
- Variables in a braoder sense (objects)
- Upper Camel variable names
- Distinguish between Strings, Vectors, Factors and Data Frames/Matrices
- Tidy data
- Loading external date into R
Data Preprocessing
How Can R Manipulate Data?
You will learn how to filter, change, and create new data and variables using R’s tidtverse package.
- How to display very big and very small numbers in R
- The tidyverse Package
- Understand Tibbles/Data Frames
- Be able to Select from data (frames)
- Be able to Filter data (frames)
- Know how to use Mutate to change and create variables
- Understand and use the Pipe operator
- Apply the
ifelse()function to data analysis.
- Use an if else structure in an R-Sript.
# The #-sign starts a comment line
# Execute this line by line and read the comments
# Always load tidyverse
library(tidyverse)How to display very big and very small numbers in R
How to Display Very Big Numbers:
Big Number
BigNum=2700000
BigNum## [1] 2700000Bigger Number
BigNum=BigNum*10
print(BigNum)## [1] 2.7e+07How to Display Very Small Numbers: Small Number
SmallNum=0.00012
print(SmallNum)## [1] 0.00012Smaller Number
SmallNum=SmallNum/10
SmallNum## [1] 1.2e-051.2/100000## [1] 1.2e-05Be able to Select from data (frames)
Assign a vector to a variable
Note = and <- can be used interchangeably.
Number Vector
Inc=c(23000,20000,80000,50000,48000,37000)
Inc## [1] 23000 20000 80000 50000 48000 37000mean(Inc)## [1] 43000Character Vector
Sx=c("male","male","male","female","female","male")
mean(Sx) #generates error! Why? Because the vector is made of characters not number. ## Warning in mean.default(Sx): argument is not numeric or logical: returning NA## [1] NAUnderstand Tibbles/Data Frames
Create a tible/dataframe ‘MyData’ from vectors:
MyData=tibble(Sex=Sx,Income=Inc,Siblings=c(0,2,1,4,0,0)) #tibble is to create data frame.
datatable(MyData)Data Table number II
library(kableExtra)
MyData %>%
kable() %>%
kable_styling()| Sex | Income | Siblings |
|---|---|---|
| male | 23000 | 0 |
| male | 20000 | 2 |
| male | 80000 | 1 |
| female | 50000 | 4 |
| female | 48000 | 0 |
| male | 37000 | 0 |
Select variables from Dataframe/Tibble
MySelectedVariables=select(MyData,Sex,Siblings)
MySelectedVariables## # A tibble: 6 x 2
## Sex Siblings
## <chr> <dbl>
## 1 male 0
## 2 male 2
## 3 male 1
## 4 female 4
## 5 female 0
## 6 male 0Assign dataframe/tibble column to vector
MyVector=MyData$Income
MyVector## [1] 23000 20000 80000 50000 48000 37000Then calculate mean
mean(MyVector)## [1] 43000Subsetting Data Frames
Start with the original data table from before.
datatable(MyData)First Record / SecondVariable
Pay attention to the [ , ] brackets. The first diget incide the brackes before the comma referse to the first row. After the comma referse to the column. The example below would show the value of the first row and second column which is 23000.
MyData[1,2]## # A tibble: 1 x 1
## Income
## <dbl>
## 1 23000Third Record / All Variables. This will pring out the third row.
MyData[3,]## # A tibble: 1 x 3
## Sex Income Siblings
## <chr> <dbl> <dbl>
## 1 male 80000 1Record 3 to 6/ All Variables. 3 to 6 row. Use the : to call for 3 to 6 row.
MyData[3:6,]## # A tibble: 4 x 3
## Sex Income Siblings
## <chr> <dbl> <dbl>
## 1 male 80000 1
## 2 female 50000 4
## 3 female 48000 0
## 4 male 37000 0Record 2,4, and 6/ Variables 2 to 3. Create a vector object with c() and chose row 2, 4, and 6 because numbers skip. The culum side could olso be written as c(2,3).
MyData[c(2,4,6),2:3]## # A tibble: 3 x 2
## Income Siblings
## <dbl> <dbl>
## 1 20000 2
## 2 50000 4
## 3 37000 0variable 2 (income) for all records. 2nd column.
MyData[,2]## # A tibble: 6 x 1
## Income
## <dbl>
## 1 23000
## 2 20000
## 3 80000
## 4 50000
## 5 48000
## 6 37000Vector is not a vector all the times :)
Variable 2 (income) for all records
MyData[,2]## # A tibble: 6 x 1
## Income
## <dbl>
## 1 23000
## 2 20000
## 3 80000
## 4 50000
## 5 48000
## 6 37000Error!!! Does not produce the mean income
mean(MyData[,2])## Warning in mean.default(MyData[, 2]): argument is not numeric or logical:
## returning NA## [1] NAIt is a one column dataframe NOT a vector
class(MyData[,2])## [1] "tbl_df" "tbl" "data.frame"Does produce the mean income
class(MyData$Income)## [1] "numeric"mean(MyData$Income)## [1] 43000Tidyverse/dplyr
Selectedd variables (rows) from a Data Frame
SexAndIncome=select(MyData, Sex,Income) # Select the data. columns = select
datatable(SexAndIncome)Filter records for character/categorial varaible. E.g. to show only females.
MyFemaleData=filter(MyData,Sex=="female") # filter = rows. == to filter by.
datatable(MyFemaleData)Filter records for character/categorial varaible. E.g. to show only females.
MyLowIncomeData=filter(MyData,Income<47520)
MyLowIncomeData## # A tibble: 3 x 3
## Sex Income Siblings
## <chr> <dbl> <dbl>
## 1 male 23000 0
## 2 male 20000 2
## 3 male 37000 0Sort records ascending (default)
MyAscSortedRecords=arrange(MyData,Sex,Income)
MyAscSortedRecords## # A tibble: 6 x 3
## Sex Income Siblings
## <chr> <dbl> <dbl>
## 1 female 48000 0
## 2 female 50000 4
## 3 male 20000 2
## 4 male 23000 0
## 5 male 37000 0
## 6 male 80000 1Sort records descending (default)
MyAscSortedRecords=arrange(MyData,desc(Income))
MyAscSortedRecords## # A tibble: 6 x 3
## Sex Income Siblings
## <chr> <dbl> <dbl>
## 1 male 80000 1
## 2 female 50000 4
## 3 female 48000 0
## 4 male 37000 0
## 5 male 23000 0
## 6 male 20000 2Counting Records
length(MyData$Sex) #produces correct result (6 records)## [1] 6length(MyData) #wrong !!! produces incorrect results (the number of vectors in the dataframe)## [1] 3nrow(MyData) #gives the correct number of records in MyData## [1] 6count(MyData) #don't use as it returns a data frame## # A tibble: 1 x 1
## n
## <int>
## 1 6nrow(MyData) # use this instead## [1] 6The power of count()
count(MyData,Sex)## # A tibble: 2 x 2
## Sex n
## <chr> <int>
## 1 female 2
## 2 male 4count(MyData,Siblings)## # A tibble: 4 x 2
## Siblings n
## <dbl> <int>
## 1 0 3
## 2 1 1
## 3 2 1
## 4 4 1count(MyData,Sex, Siblings) #order matters## # A tibble: 5 x 3
## Sex Siblings n
## <chr> <dbl> <int>
## 1 female 0 1
## 2 female 4 1
## 3 male 0 2
## 4 male 1 1
## 5 male 2 1count(MyData,Siblings,Sex) #order ## # A tibble: 5 x 3
## Siblings Sex n
## <dbl> <chr> <int>
## 1 0 female 1
## 2 0 male 2
## 3 1 male 1
## 4 2 male 1
## 5 4 female 1Calculating new varaibles or changing existing variables
mutate(MyData, MonthlyIncome=Income/12)## # A tibble: 6 x 4
## Sex Income Siblings MonthlyIncome
## <chr> <dbl> <dbl> <dbl>
## 1 male 23000 0 1917.
## 2 male 20000 2 1667.
## 3 male 80000 1 6667.
## 4 female 50000 4 4167.
## 5 female 48000 0 4000
## 6 male 37000 0 3083.MyData## # A tibble: 6 x 3
## Sex Income Siblings
## <chr> <dbl> <dbl>
## 1 male 23000 0
## 2 male 20000 2
## 3 male 80000 1
## 4 female 50000 4
## 5 female 48000 0
## 6 male 37000 0MyData=mutate(MyData, MonthlyIncome=Income/12)
MyData## # A tibble: 6 x 4
## Sex Income Siblings MonthlyIncome
## <chr> <dbl> <dbl> <dbl>
## 1 male 23000 0 1917.
## 2 male 20000 2 1667.
## 3 male 80000 1 6667.
## 4 female 50000 4 4167.
## 5 female 48000 0 4000
## 6 male 37000 0 3083.# What if I want to change a Variable
#(e.g., I want monthly income in $1,000)
MyData=mutate(MyData, MonthlyIncome=Income/12/1000)
MyData## # A tibble: 6 x 4
## Sex Income Siblings MonthlyIncome
## <chr> <dbl> <dbl> <dbl>
## 1 male 23000 0 1.92
## 2 male 20000 2 1.67
## 3 male 80000 1 6.67
## 4 female 50000 4 4.17
## 5 female 48000 0 4
## 6 male 37000 0 3.08The Piping operator %>%
Generate a data frame for men only, without showing variable siblings and sorted by income.
Solution witout piping
TempMenOnly=filter(MyData, Sex=="male")
TempMenOnly## # A tibble: 4 x 4
## Sex Income Siblings MonthlyIncome
## <chr> <dbl> <dbl> <dbl>
## 1 male 23000 0 1.92
## 2 male 20000 2 1.67
## 3 male 80000 1 6.67
## 4 male 37000 0 3.08TempMenSexInc=select(TempMenOnly, -Siblings)
TempMenSexInc## # A tibble: 4 x 3
## Sex Income MonthlyIncome
## <chr> <dbl> <dbl>
## 1 male 23000 1.92
## 2 male 20000 1.67
## 3 male 80000 6.67
## 4 male 37000 3.08Result=arrange(TempMenSexInc,Income) # do not use sort!
# Solution with piping %>%:
Result= TempMenOnly %>% filter(Sex=="male") %>% select(-Siblings) %>% arrange(Income)
#You can break lines after(!) the %>% pipe
Result= TempMenOnly %>% filter(Sex=="male") %>%
select(-Siblings) %>% arrange(Income)
Result## # A tibble: 4 x 3
## Sex Income MonthlyIncome
## <chr> <dbl> <dbl>
## 1 male 20000 1.67
## 2 male 23000 1.92
## 3 male 37000 3.08
## 4 male 80000 6.67ifelse() in combination with mutate().
Add a variable SingleChild that is TRUE if the record presents a person growing up as single child to the data frame MyData otherwise add FALSE
MyData=mutate(MyData, SingleChild=ifelse(Siblings>0,TRUE,FALSE))
MyData=MyData %>% mutate(SingleChild=ifelse(Siblings>0,TRUE,FALSE))Calcualte mean of vector (make sure it is not a one column tibble)
mean(MyData$Income)## [1] 43000MyData[2] # this is still a dataframe! It just has only one column## # A tibble: 6 x 1
## Income
## <dbl>
## 1 23000
## 2 20000
## 3 80000
## 4 50000
## 5 48000
## 6 37000mean(MyData[2])# wrong!!! results in error. Try class(MyData[2]) to see why## Warning in mean.default(MyData[2]): argument is not numeric or logical:
## returning NA## [1] NAclass(MyData[2])## [1] "tbl_df" "tbl" "data.frame"# Calcualte (sample) standard deviation of vector
sd(MyData$Income)## [1] 21945.39Machine Learning Algorithms for Regression Problems
Regression Analysis for One and Multiple Variables.
Regression (supervised learning) e.g., based on data records for many firms such as number of employees, sales, material cost, etc. predict the firm size measured as balance sheet volume.
Regression analysis is a powerful technique to analyze and quantify the impact of one or more (explanatory) variables on a dependent variable. The basic idea of regression will be explained and a sample analysis with the Boston Housing dataset will be performed. The concept of pairwise correlation as well as multi-variable regression and the concept of dummy variable will be introduced. The concept will be extended to regressions where the dependent variable is a categorial variable. Logistic regression will be introduced.
library(tidyverse)
IAmInTheLab=TRUE # do not change unless instructed
if (!require(mlbench)) {
if (IAmInTheLab){
.libPaths(new=c("C:/Program Files/R/Libs",Sys.getenv("R_User")))
install.packages("mlbench", lib=Sys.getenv("R_User"))
} else {
install.packages("mlbench")
}
}## Loading required package: mlbenchlibrary(mlbench)
data("BostonHousing2")
DataCl=as_tibble(BostonHousing2[,c("cmedv","lstat")]) %>%
rename(MHP=cmedv, PLI=lstat)
ggplot(DataCl,aes(x=PLI,y=MHP))+
geom_point()
m=0
b=mean(DataCl$MHP) #22.52
OlsParameters=c(m,b)
ggplot(DataCl,aes(x=PLI,y=MHP))+
geom_point()+
geom_abline(slope=m, intercept = b, color="red", size=1.5)
FctMseErrorOls<- function(OlsParameters) {
m=OlsParameters[1]
b=OlsParameters[2]
SquaredErrors=(DataCl$MHP-(m*DataCl$PLI+b))^2
MSE=mean(SquaredErrors)
return(MSE)
}
FctMseErrorOls(OlsParameters)## [1] 84.14573ModelOlsOptimizer=optim(OlsParameters,FctMseErrorOls)# optim() is also in the Stats package
m=ModelOlsOptimizer$par[1]
b=ModelOlsOptimizer$par[2]
print(m)## [1] -0.9526324print(b)## [1] 34.58288FctMseErrorOls(c(m,b))## [1] 37.96336ggplot(DataCl,aes(x=PLI,y=MHP))+
geom_point()+
geom_abline(slope=m, intercept = b, color="red", size=1.5)
lm(data=DataCl, MHP~PLI)##
## Call:
## lm(formula = MHP ~ PLI, data = DataCl)
##
## Coefficients:
## (Intercept) PLI
## 34.5820 -0.9526ModelOls=lm(data=DataCl, MHP~PLI)
summary(ModelOls)##
## Call:
## lm(formula = MHP ~ PLI, data = DataCl)
##
## Residuals:
## Min 1Q Median 3Q Max
## -9.951 -3.997 -1.325 2.086 24.496
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 34.58200 0.55882 61.88 <2e-16 ***
## PLI -0.95259 0.03847 -24.76 <2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 6.174 on 504 degrees of freedom
## Multiple R-squared: 0.5488, Adjusted R-squared: 0.5479
## F-statistic: 613.1 on 1 and 504 DF, p-value: < 2.2e-16mean(ModelOls$residuals^2)#MSE## [1] 37.96336Error Functions and Optimizer Algorithm
You will learn about different measures to evalute prediction errors for regression and classification algorithms. You will also learn how to measure performance on data that were not used to train the ML algorithm. Various errors such as mean squared error, avg absolut error and the confusion matrix are introduced. In addition, validation in general and x-fold cross validation will be discussed.
library(tidyverse)
# The following packages need to be installed,
# before you use them the first time.
# Use Tools -> Instal Packages ...
library(gridExtra)
library(stats)
library(ggplot2)
#from https://www.dummies.com/programming/r/10-things-you-can-do-in-r-that-you-wouldve-done-in-microsoft-excel/
QdFct <- function(P) { 100 - 0.5 * P }
RevenueFct<- function(P) {P * QdFct(P)}
DataCl=tibble(Price=seq(50,150,5),
Qd=QdFct(Price),
Revenue=RevenueFct(Price))
Plot1=ggplot(DataCl, aes(x=Price, y=Qd))+
geom_line(size=2)
Plot2=ggplot(DataCl, aes(x=Price, y=Revenue))+
geom_line(size=2)
grid.arrange(Plot1, Plot2, nrow = 1)
ModelRevenue=optim(c(60), RevenueFct,
method = "Brent", lower = 1, upper = 300,
control=list(fnscale=-1) )
ModelRevenue$par## [1] 100Plot2=Plot2+geom_vline(aes(xintercept = ModelRevenue$par), color="red", size=2)
grid.arrange(Plot1, Plot2, nrow = 1)
Machine Learning Algorithms for Classifcation Problems
Classification, (supervised learning) e.g., using balance sheet ratios, and other variables to determine book keeping was fraudulent (=1) or not fraudulent (=0)
Logistic Regression
Links:
https://www.machinelearningplus.com/machine-learning/logistic-regression-tutorial-examples-r/
library(tidyverse)
IAmInTheLab=FALSE # do not change unless instructed
ExamplePlot=ggplot(tibble(x=c(-20:20),y=exp(0.5*x+3)/(1+exp(0.5*x+3))),aes(x=x,y=y))+
geom_line(col="red",size=2)
#con=gzcon(url("https://www.cpp.edu/~clange/RData/Datasets/#WineDataOrg.rds"))
WineDataOrg = read_rds("WineDataOrg.rds")
#WineDataOrg=readRDS(con)%>%na.omit()
#close(con)
if (!require(ggcorrplot)) {
if (IAmInTheLab){
.libPaths(new=c("C:/Program Files/R/Libs",Sys.getenv("R_User")))
install.packages(ggcorrplot, lib=Sys.getenv("R_User"))
} else {
install.packages(ggcorrplot)
}
}
library(ggcorrplot)
datatable(WineDataOrg)WineDataCl=WineDataOrg %>%
mutate(IsRedWine=ifelse(wineColor=="red",1,0)) %>%
select(IsRedWine,everything(),-wineColor, -RandomCl)
datatable(WineDataCl)if (version$major>5){
RNGkind(sample.kind = "Rounding")
}
set.seed(2019)
TestDataIndex=sample(1:3198,400,replace = FALSE)
TestData=WineDataCl[TestDataIndex,]
TrainData=WineDataCl[-TestDataIndex,]
rm(WineDataOrg,con,IAmInTheLab, WineDataCl)
ggcorrplot(cor(TrainData), hc.order = FALSE,
type = "lower",
lab = TRUE,
lab_size = 3,
method="circle",
colors = c("blue", "white", "red"),
ggtheme=theme_bw)
ModelLm1Var=lm(data=TrainData, IsRedWine~totalSulfurDioxide)
summary(ModelLm1Var)##
## Call:
## lm(formula = IsRedWine ~ totalSulfurDioxide, data = TrainData)
##
## Residuals:
## Min 1Q Median 3Q Max
## -0.98641 -0.23200 0.02671 0.19071 1.79138
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 1.1044928 0.0111792 98.80 <2e-16 ***
## totalSulfurDioxide -0.0065601 0.0001028 -63.81 <2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.3191 on 2796 degrees of freedom
## Multiple R-squared: 0.5929, Adjusted R-squared: 0.5927
## F-statistic: 4072 on 1 and 2796 DF, p-value: < 2.2e-16ggplot(TrainData,aes(x=totalSulfurDioxide,y=IsRedWine))+
geom_point()+
geom_smooth(method = "lm", se = FALSE)+
scale_y_continuous(limits=c(-0.5,1.5))+
scale_x_continuous(limits=c(-50,350))
TrueVsPredLmTestData=TestData %>%
mutate(PredValue=predict(ModelLm1Var, newdata = TestData),
IsPredRed=ifelse(PredValue>0.5,1,0)) %>%
select(PredValue,IsPredRed,IsRedWine)
TrueVsPredLmTestData## # A tibble: 400 x 3
## PredValue IsPredRed IsRedWine
## <dbl> <dbl> <dbl>
## 1 0.855 1 1
## 2 0.134 0 0
## 3 0.921 1 1
## 4 0.291 0 0
## 5 0.612 1 0
## 6 -0.286 0 0
## 7 0.960 1 1
## 8 0.901 1 1
## 9 0.986 1 1
## 10 0.737 1 1
## # ... with 390 more rowstable(TrueVsPredLmTestData$IsPredRed,TrueVsPredLmTestData$IsRedWine)##
## 0 1
## 0 182 19
## 1 29 170ggplot(TestData,aes(x=totalSulfurDioxide,y=IsRedWine))+
geom_point()+
geom_smooth(method = "lm", se = FALSE)+
geom_smooth(method = "glm", method.args = list(family = "binomial"),
se = FALSE, col="red")+
scale_y_continuous(limits=c(-0.5,1.5))+
scale_x_continuous(limits=c(-50,350))
# Derive Logistic Regression first in PowerPoint then
# run logistic regression with glm
ggcorrplot(cor(TrainData), hc.order = FALSE,
type = "lower",
lab = TRUE,
lab_size = 3,
method="circle",
colors = c("blue", "white", "red"),
ggtheme=theme_bw)
ModelLogistic1Var=glm(IsRedWine~totalSulfurDioxide+volatileAcidity,
family = "binomial",
data=TrainData)
ModelLogistic1Var##
## Call: glm(formula = IsRedWine ~ totalSulfurDioxide + volatileAcidity,
## family = "binomial", data = TrainData)
##
## Coefficients:
## (Intercept) totalSulfurDioxide volatileAcidity
## -0.16057 -0.06346 15.21217
##
## Degrees of Freedom: 2797 Total (i.e. Null); 2795 Residual
## Null Deviance: 3879
## Residual Deviance: 836.5 AIC: 842.5TrueVsPredGlmTestData=TestData %>%
mutate(PredProbRedWine =
predict(ModelLogistic1Var, newdata=TestData, type="response"),
PredProbRedWine=round(PredProbRedWine,4)) %>%
select(PredProbRedWine, everything())
TrueVsPredGlmTestData## # A tibble: 400 x 14
## PredProbRedWine IsRedWine acidity volatileAcidity citricAcid residualSugar
## <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 0.995 1 6.5 0.52 0.11 1.8
## 2 0.0106 0 6.5 0.33 0.38 2.5
## 3 1 1 9.1 0.785 0 2.6
## 4 0.0027 0 6.7 0.14 0.51 4.3
## 5 0.855 0 6.6 0.44 0.32 3
## 6 0.0253 0 7.3 0.655 0.2 10.2
## 7 0.999 1 6.9 0.580 0.2 1.75
## 8 1.00 1 7.2 0.65 0.02 2.3
## 9 1.00 1 6.5 0.615 0 1.9
## 10 1 1 7.3 0.91 0.1 1.8
## # ... with 390 more rows, and 8 more variables: Chlorides <dbl>,
## # freeSulfurDioxide <dbl>, totalSulfurDioxide <dbl>, Density <dbl>, pH <dbl>,
## # sulphates <dbl>, alcohol <dbl>, quality <dbl>TrueVsPredGlmTestData=TrueVsPredGlmTestData %>%
mutate(PredIsRedWine=ifelse(PredProbRedWine>0.5,1,0)) %>%
select(IsRedWine, PredIsRedWine, PredProbRedWine)
TrueVsPredGlmTestData## # A tibble: 400 x 3
## IsRedWine PredIsRedWine PredProbRedWine
## <dbl> <dbl> <dbl>
## 1 1 1 0.995
## 2 0 0 0.0106
## 3 1 1 1
## 4 0 0 0.0027
## 5 0 1 0.855
## 6 0 0 0.0253
## 7 1 1 0.999
## 8 1 1 1.00
## 9 1 1 1.00
## 10 1 1 1
## # ... with 390 more rowstable(TrueVsPredGlmTestData$PredIsRedWine,TrueVsPredGlmTestData$IsRedWine)##
## 0 1
## 0 192 9
## 1 19 180Machine Learning Algorithms for Clustering Problems
Clustering is consider unsupervised learning e.g., using 1 Mio. balance sheets to determine groups of similar balance sheets. After groups have been determined find what these groups have in common e.g. industry, size, region, etc.
k-Nearest Neighbors (kNN)
Set Up!!!
library(tidyverse)
library(readxl)
IAmInTheLab=FALSE # do not change unless instructed
if (!require(caret)) {
if (IAmInTheLab){
.libPaths(new=c("C:/Program Files/R/Libs",Sys.getenv("R_User")))
install.packages("caret", lib=Sys.getenv("R_User"))
} else {
install.packages("caret")
}
}
library(caret)
if (!require(e1071)) {
if (IAmInTheLab){
.libPaths(new=c("C:/Program Files/R/Libs",Sys.getenv("R_User")))
install.packages("e1071", lib=Sys.getenv("R_User"))
} else {
install.packages("e1071")
}
}
library(e1071)
con=gzcon(url("https://www.cpp.edu/~clange/RData/Datasets/WineDataOrg.rds"))
close(con)WineDataOrg=readRDS("WineDataOrg.rds")%>%na.omit()
WineDataOrg = WineDataOrg %>%
select(WineColor=wineColor, Chlorides,
TotalSulfurDioxide=totalSulfurDioxide,
Alcohol=alcohol) %>%
mutate(Id=c(1:nrow(WineDataOrg)))
WineData=WineDataOrg %>%
mutate(Chlorides=scale(Chlorides),
TotalSulfurDioxide=scale(TotalSulfurDioxide),
Alcohol=scale(Alcohol))
set.seed(123)
SampleTest=sample(1:nrow(WineData),round(nrow(WineData)*0.10))
WineDataTest=WineData[SampleTest,] %>% mutate(Prediction=NA)
WineDataTrain=WineData[-SampleTest,]Wine Plot
WinePlot=ggplot(NULL, aes(x=Chlorides,y=TotalSulfurDioxide,color=WineColor))+
geom_point(data=WineDataTrain,alpha=0.2)+
scale_color_manual(values=c("red", "gold2", "blue"))+
ylim(-1.5,4)+
xlim(-3,10)
WinePlot
for (i in 1:nrow(WineDataTest)) {
#i=1# number of test record
TestRecord=WineDataTest[i,1:3]
WinePlot+
geom_point(data=TestRecord, color="blue", size=3)
WineDataTrain=WineDataTrain %>%
mutate(EuclDistToTestRecordi=sqrt(
(Chlorides-as.numeric(WineDataTest[i,"Chlorides"]))^2+
(TotalSulfurDioxide-as.numeric(WineDataTest[i,"TotalSulfurDioxide"]))^2
)) %>% mutate_if(., is.numeric, as.numeric)
WineDataTrain=dplyr::arrange(WineDataTrain,EuclDistToTestRecordi)
WineDataTest[i,"Prediction"]=WineDataTrain[1,"WineColor"]
}# end i loop
table(WineDataTest$Prediction,WineDataTest$WineColor)Caret
library(caret)## Loading required package: lattice##
## Attaching package: 'caret'## The following object is masked from 'package:purrr':
##
## liftTrainKnn=WineDataTrain[,1:4]
TrainKnn=TrainKnn %>% mutate(WineColor=as.factor(WineColor))
set.seed(3333)
KnnModel <- train(WineColor ~., data = TrainKnn, method = "knn",
trControl=trainControl(method = "repeatedcv",
number = 10, repeats = 3),
#preProcess = c("center", "scale"),
tuneLength = 10)
KnnModel## k-Nearest Neighbors
##
## 2878 samples
## 3 predictor
## 2 classes: 'red', 'white'
##
## No pre-processing
## Resampling: Cross-Validated (10 fold, repeated 3 times)
## Summary of sample sizes: 2591, 2590, 2590, 2591, 2590, 2590, ...
## Resampling results across tuning parameters:
##
## k Accuracy Kappa
## 5 0.9740514 0.9481005
## 7 0.9698832 0.9397631
## 9 0.9704635 0.9409237
## 11 0.9703481 0.9406921
## 13 0.9708095 0.9416156
## 15 0.9698844 0.9397655
## 17 0.9689576 0.9379116
## 19 0.9684951 0.9369867
## 21 0.9680297 0.9360563
## 23 0.9674510 0.9348983
##
## Accuracy was used to select the optimal model using the largest value.
## The final value used for the model was k = 5.PlotDataCl=tibble(k=KnnModel$results$k, Accuracy=KnnModel$results$Accuracy ,
Kappa=KnnModel$results$Kappa)Plot the Values
ggplot()+
geom_line(data=PlotDataCl,aes(x=k,y=Accuracy),color="red")+
geom_line(data=PlotDataCl,aes(x=k,y=Kappa),color="blue")+
scale_x_continuous(breaks = seq(1,20,1))+
ylab("Kappa, Accuracy")
Test Predicted model
test_pred = predict(KnnModel, newdata = as.matrix(WineDataTest[,2:4]))
View(test_pred)
confusionMatrix(test_pred,as.factor(WineDataTest$WineColor))Support Vector Machines( SVM)
Set Up!!
# use Caret
# https://cran.r-project.org/web/packages/caret/vignettes/caret.html
#Data https://www.analyticsvidhya.com/blog/2016/12/practical-guide-to-implement-machine-learning-with-caret-package-in-r-with-practice-problem/
library(tidyverse)
IAmInTheLab=FALSE # do not change unless instructed
if (!require(caret)) {
if (IAmInTheLab){
.libPaths(new=c("C:/Program Files/R/Libs",Sys.getenv("R_User")))
install.packages("caret", lib=Sys.getenv("R_User"))
} else {
install.packages("caret")
}
}
library(caret)
if (!require(kernlab)) {
if (IAmInTheLab){
.libPaths(new=c("C:/Program Files/R/Libs",Sys.getenv("R_User")))
install.packages("kernlab", lib=Sys.getenv("R_User"))
} else {
install.packages("kernlab")
}
}
library(kernlab)
if (!require(RANN)) {
if (IAmInTheLab){
.libPaths(new=c("C:/Program Files/R/Libs",Sys.getenv("R_User")))
install.packages("RANN", lib=Sys.getenv("R_User"))
} else {
install.packages("RANN")
}
}
library(RANN)
if (version$major>5){
RNGkind(sample.kind = "Rounding")
}
set.seed(2019)
#Loading Mostly Unprocessed Data
#con=gzcon(url("https://www.cpp.edu/~clange/RData/Datasets/LoanDataSem#iAdj.rds"))
#DataOrgCl=readRDS(con) ### << Keep having problems with this line of #code.
#close(con)Gets the data and summarizes the data.
DataOrgCl = read_rds("LoanDataSemiAdj.rds")
str(DataOrgCl)## 'data.frame': 511 obs. of 12 variables:
## $ LoanStatusOk : chr "Y" "N" "Y" "Y" ...
## $ Gender : chr "Male" "Male" "Male" "Male" ...
## $ Married : chr "No" "Yes" "Yes" "Yes" ...
## $ Dependents : chr "0" "1" "0" "0" ...
## $ Education : chr "Graduate" "Graduate" "Graduate" "Not Graduate" ...
## $ Self_Employed : chr "No" "No" "Yes" "No" ...
## $ ApplicantIncome : int 5849 4583 3000 2583 6000 5417 2333 3036 4006 12841 ...
## $ CoapplicantIncome: num 0 1508 0 2358 0 ...
## $ LoanAmount : int NA 128 66 120 141 267 95 158 168 349 ...
## $ Loan_Amount_Term : int 360 360 360 360 360 360 360 360 360 360 ...
## $ Property_Area : chr "Urban" "Rural" "Urban" "Urban" ...
## $ CreditHistoryOk : chr "Y" "Y" "Y" "Y" ...sum(is.na(DataOrgCl))## [1] 31Creates the pre process mode.
ModelClPreProcess=preProcess(DataOrgCl,
method = c("knnImpute","center","scale"))
DataPreProcessed = predict(ModelClPreProcess, DataOrgCl)
sum(is.na(DataPreProcessed))## [1] 0DataPreProcessed[!complete.cases(DataPreProcessed),] ## [1] LoanStatusOk Gender Married Dependents
## [5] Education Self_Employed ApplicantIncome CoapplicantIncome
## [9] LoanAmount Loan_Amount_Term Property_Area CreditHistoryOk
## <0 rows> (or 0-length row.names)datatable(DataPreProcessed)Converting every categorical variable to numerical using dummy variables.
ModelClDummy <- dummyVars(" ~ .",
data = DataPreProcessed,
fullRank = T)
DataFinalCl =data.frame(predict(ModelClDummy,
newdata = DataPreProcessed)) %>%
rename(LoanStatusOk=LoanStatusOkY)Checking the structure of transformed train file
datatable(DataFinalCl)#Splitting data using caret
#Spliting training set into two parts based on outcome: 75% and 25%
if (version$major>5){
RNGkind(sample.kind = "Rounding")
}
set.seed(2019)
TrainIndex <- createDataPartition(DataFinalCl$LoanStatusOk, p=0.75, list=FALSE)
TrainDataSet = DataFinalCl[ TrainIndex,]
TestDataSet <- DataFinalCl[-TrainIndex,]
rm(DataPreProcessed,ModelClDummy,ModelClPreProcess,TrainIndex)
names(getModelInfo())## [1] "ada" "AdaBag" "AdaBoost.M1"
## [4] "adaboost" "amdai" "ANFIS"
## [7] "avNNet" "awnb" "awtan"
## [10] "bag" "bagEarth" "bagEarthGCV"
## [13] "bagFDA" "bagFDAGCV" "bam"
## [16] "bartMachine" "bayesglm" "binda"
## [19] "blackboost" "blasso" "blassoAveraged"
## [22] "bridge" "brnn" "BstLm"
## [25] "bstSm" "bstTree" "C5.0"
## [28] "C5.0Cost" "C5.0Rules" "C5.0Tree"
## [31] "cforest" "chaid" "CSimca"
## [34] "ctree" "ctree2" "cubist"
## [37] "dda" "deepboost" "DENFIS"
## [40] "dnn" "dwdLinear" "dwdPoly"
## [43] "dwdRadial" "earth" "elm"
## [46] "enet" "evtree" "extraTrees"
## [49] "fda" "FH.GBML" "FIR.DM"
## [52] "foba" "FRBCS.CHI" "FRBCS.W"
## [55] "FS.HGD" "gam" "gamboost"
## [58] "gamLoess" "gamSpline" "gaussprLinear"
## [61] "gaussprPoly" "gaussprRadial" "gbm_h2o"
## [64] "gbm" "gcvEarth" "GFS.FR.MOGUL"
## [67] "GFS.LT.RS" "GFS.THRIFT" "glm.nb"
## [70] "glm" "glmboost" "glmnet_h2o"
## [73] "glmnet" "glmStepAIC" "gpls"
## [76] "hda" "hdda" "hdrda"
## [79] "HYFIS" "icr" "J48"
## [82] "JRip" "kernelpls" "kknn"
## [85] "knn" "krlsPoly" "krlsRadial"
## [88] "lars" "lars2" "lasso"
## [91] "lda" "lda2" "leapBackward"
## [94] "leapForward" "leapSeq" "Linda"
## [97] "lm" "lmStepAIC" "LMT"
## [100] "loclda" "logicBag" "LogitBoost"
## [103] "logreg" "lssvmLinear" "lssvmPoly"
## [106] "lssvmRadial" "lvq" "M5"
## [109] "M5Rules" "manb" "mda"
## [112] "Mlda" "mlp" "mlpKerasDecay"
## [115] "mlpKerasDecayCost" "mlpKerasDropout" "mlpKerasDropoutCost"
## [118] "mlpML" "mlpSGD" "mlpWeightDecay"
## [121] "mlpWeightDecayML" "monmlp" "msaenet"
## [124] "multinom" "mxnet" "mxnetAdam"
## [127] "naive_bayes" "nb" "nbDiscrete"
## [130] "nbSearch" "neuralnet" "nnet"
## [133] "nnls" "nodeHarvest" "null"
## [136] "OneR" "ordinalNet" "ordinalRF"
## [139] "ORFlog" "ORFpls" "ORFridge"
## [142] "ORFsvm" "ownn" "pam"
## [145] "parRF" "PART" "partDSA"
## [148] "pcaNNet" "pcr" "pda"
## [151] "pda2" "penalized" "PenalizedLDA"
## [154] "plr" "pls" "plsRglm"
## [157] "polr" "ppr" "PRIM"
## [160] "protoclass" "qda" "QdaCov"
## [163] "qrf" "qrnn" "randomGLM"
## [166] "ranger" "rbf" "rbfDDA"
## [169] "Rborist" "rda" "regLogistic"
## [172] "relaxo" "rf" "rFerns"
## [175] "RFlda" "rfRules" "ridge"
## [178] "rlda" "rlm" "rmda"
## [181] "rocc" "rotationForest" "rotationForestCp"
## [184] "rpart" "rpart1SE" "rpart2"
## [187] "rpartCost" "rpartScore" "rqlasso"
## [190] "rqnc" "RRF" "RRFglobal"
## [193] "rrlda" "RSimca" "rvmLinear"
## [196] "rvmPoly" "rvmRadial" "SBC"
## [199] "sda" "sdwd" "simpls"
## [202] "SLAVE" "slda" "smda"
## [205] "snn" "sparseLDA" "spikeslab"
## [208] "spls" "stepLDA" "stepQDA"
## [211] "superpc" "svmBoundrangeString" "svmExpoString"
## [214] "svmLinear" "svmLinear2" "svmLinear3"
## [217] "svmLinearWeights" "svmLinearWeights2" "svmPoly"
## [220] "svmRadial" "svmRadialCost" "svmRadialSigma"
## [223] "svmRadialWeights" "svmSpectrumString" "tan"
## [226] "tanSearch" "treebag" "vbmpRadial"
## [229] "vglmAdjCat" "vglmContRatio" "vglmCumulative"
## [232] "widekernelpls" "WM" "wsrf"
## [235] "xgbDART" "xgbLinear" "xgbTree"
## [238] "xyf"# run models
ModelClLogistic=train(as.factor(LoanStatusOk)~.,
data=TrainDataSet, method = "glm",
family="binomial")
summary(ModelClLogistic)##
## Call:
## NULL
##
## Deviance Residuals:
## Min 1Q Median 3Q Max
## -2.3269 -0.2306 0.4636 0.6667 2.9161
##
## Coefficients:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) -4.70368 0.86492 -5.438 5.38e-08 ***
## GenderMale 0.43664 0.38979 1.120 0.26264
## MarriedYes 1.07981 0.35486 3.043 0.00234 **
## Dependents1 -0.33378 0.41712 -0.800 0.42359
## Dependents2 0.01554 0.45219 0.034 0.97259
## Dependents3. -0.33423 0.58758 -0.569 0.56948
## EducationNot.Graduate -0.24322 0.36797 -0.661 0.50863
## Self_EmployedYes 0.01393 0.40328 0.035 0.97245
## ApplicantIncome 0.22388 0.16756 1.336 0.18151
## CoapplicantIncome -0.11534 0.12494 -0.923 0.35592
## LoanAmount -0.34736 0.16084 -2.160 0.03080 *
## Loan_Amount_Term 0.08650 0.13536 0.639 0.52281
## Property_AreaSemiurban 0.87304 0.36749 2.376 0.01751 *
## Property_AreaUrban -0.08888 0.35950 -0.247 0.80473
## CreditHistoryOkY 4.91041 0.77240 6.357 2.05e-10 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## (Dispersion parameter for binomial family taken to be 1)
##
## Null deviance: 486.02 on 383 degrees of freedom
## Residual deviance: 318.59 on 369 degrees of freedom
## AIC: 348.59
##
## Number of Fisher Scoring iterations: 6confusionMatrix(as.factor(predict(ModelClLogistic,
newdata = TestDataSet)),
as.factor(TestDataSet$LoanStatusOk))## Confusion Matrix and Statistics
##
## Reference
## Prediction 0 1
## 0 16 8
## 1 22 81
##
## Accuracy : 0.7638
## 95% CI : (0.6803, 0.8346)
## No Information Rate : 0.7008
## P-Value [Acc > NIR] : 0.07077
##
## Kappa : 0.3702
##
## Mcnemar's Test P-Value : 0.01762
##
## Sensitivity : 0.4211
## Specificity : 0.9101
## Pos Pred Value : 0.6667
## Neg Pred Value : 0.7864
## Prevalence : 0.2992
## Detection Rate : 0.1260
## Detection Prevalence : 0.1890
## Balanced Accuracy : 0.6656
##
## 'Positive' Class : 0
## ModelClSvm=train(as.factor(LoanStatusOk)~.,
data=TrainDataSet, method = "svmPoly")
confusionMatrix(as.factor(predict(ModelClSvm, newdata = TestDataSet)),
as.factor(TestDataSet$LoanStatusOk))## Confusion Matrix and Statistics
##
## Reference
## Prediction 0 1
## 0 14 5
## 1 24 84
##
## Accuracy : 0.7717
## 95% CI : (0.6888, 0.8414)
## No Information Rate : 0.7008
## P-Value [Acc > NIR] : 0.0471389
##
## Kappa : 0.3645
##
## Mcnemar's Test P-Value : 0.0008302
##
## Sensitivity : 0.3684
## Specificity : 0.9438
## Pos Pred Value : 0.7368
## Neg Pred Value : 0.7778
## Prevalence : 0.2992
## Detection Rate : 0.1102
## Detection Prevalence : 0.1496
## Balanced Accuracy : 0.6561
##
## 'Positive' Class : 0
## modelLookup("svmPoly")## model parameter label forReg forClass probModel
## 1 svmPoly degree Polynomial Degree TRUE TRUE TRUE
## 2 svmPoly scale Scale TRUE TRUE TRUE
## 3 svmPoly C Cost TRUE TRUE TRUEgetModelInfo("svmPoly")## $lssvmPoly
## $lssvmPoly$label
## [1] "Least Squares Support Vector Machine with Polynomial Kernel"
##
## $lssvmPoly$library
## [1] "kernlab"
##
## $lssvmPoly$type
## [1] "Classification"
##
## $lssvmPoly$parameters
## parameter class label
## 1 degree numeric Polynomial Degree
## 2 scale numeric Scale
## 3 tau numeric Regularization Parameter
##
## $lssvmPoly$grid
## function(x, y, len = NULL, search = "grid") {
## if(search == "grid") {
## out <- expand.grid(degree = seq(1, min(len, 3)),
## scale = 10 ^((1:len) - 4),
## tau = 2 ^((1:len) - 5))
## } else {
## out <- data.frame(degree = sample(1:3, size = len, replace = TRUE),
## scale = 10^runif(len, min = -5, log10(2)),
## tau = 2^runif(len, min = -5, max = 10))
## }
## out
## }
##
## $lssvmPoly$loop
## NULL
##
## $lssvmPoly$fit
## function(x, y, wts, param, lev, last, classProbs, ...) {
## kernlab::lssvm(x = as.matrix(x), y = y,
## tau = param$tau,
## kernel = kernlab::polydot(degree = param$degree,
## scale = param$scale,
## offset = 1), ...)
## }
##
## $lssvmPoly$predict
## function(modelFit, newdata, submodels = NULL) {
## out <- kernlab::predict(modelFit, as.matrix(newdata))
## if(is.matrix(out)) out <- out[,1]
## out
## }
##
## $lssvmPoly$prob
## NULL
##
## $lssvmPoly$predictors
## function(x, ...) {
## if(hasTerms(x) & !is.null(x@terms))
## {
## out <- predictors.terms(x@terms)
## } else {
## out <- colnames(attr(x, "xmatrix"))
## }
## if(is.null(out)) out <- names(attr(x, "scaling")$xscale$`scaled:center`)
## if(is.null(out)) out <-NA
## out
## }
##
## $lssvmPoly$tags
## [1] "Kernel Method" "Support Vector Machines"
## [3] "Polynomial Model"
##
## $lssvmPoly$levels
## function(x) lev(x)
##
## $lssvmPoly$sort
## function(x) x
##
##
## $svmPoly
## $svmPoly$label
## [1] "Support Vector Machines with Polynomial Kernel"
##
## $svmPoly$library
## [1] "kernlab"
##
## $svmPoly$type
## [1] "Regression" "Classification"
##
## $svmPoly$parameters
## parameter class label
## 1 degree numeric Polynomial Degree
## 2 scale numeric Scale
## 3 C numeric Cost
##
## $svmPoly$grid
## function(x, y, len = NULL, search = "grid") {
## if(search == "grid") {
## out <- expand.grid(degree = seq(1, min(len, 3)),
## scale = 10 ^((1:len) - 4),
## C = 2 ^((1:len) - 3))
## } else {
## out <- data.frame(degree = sample(1:3, size = len, replace = TRUE),
## scale = 10^runif(len, min = -5, log10(2)),
## C = 2^runif(len, min = -5, max = 10))
## }
## out
## }
##
## $svmPoly$loop
## NULL
##
## $svmPoly$fit
## function(x, y, wts, param, lev, last, classProbs, ...) {
## if(any(names(list(...)) == "prob.model") | is.numeric(y)) {
## out <- kernlab::ksvm(x = as.matrix(x), y = y,
## kernel = kernlab::polydot(degree = param$degree,
## scale = param$scale,
## offset = 1),
## C = param$C, ...)
## } else {
## out <- kernlab::ksvm(x = as.matrix(x), y = y,
## kernel = kernlab::polydot(degree = param$degree,
## scale = param$scale,
## offset = 1),
## C = param$C,
## prob.model = classProbs,
## ...)
## }
## out
## }
##
## $svmPoly$predict
## function(modelFit, newdata, submodels = NULL) {
## svmPred <- function(obj, x) {
## hasPM <- !is.null(unlist(obj@prob.model))
## if(hasPM) {
## pred <- kernlab::lev(obj)[apply(kernlab::predict(obj, x, type = "probabilities"),
## 1, which.max)]
## } else pred <- kernlab::predict(obj, x)
## pred
## }
## out <- try(svmPred(modelFit, newdata), silent = TRUE)
## if(is.character(kernlab::lev(modelFit))) {
## if(class(out)[1] == "try-error") {
## warning("kernlab class prediction calculations failed; returning NAs")
## out <- rep("", nrow(newdata))
## out[seq(along = out)] <- NA
## }
## } else {
## if(class(out)[1] == "try-error") {
## warning("kernlab prediction calculations failed; returning NAs")
## out <- rep(NA, nrow(newdata))
## }
## }
## if(is.matrix(out)) out <- out[,1]
## out
## }
##
## $svmPoly$prob
## function(modelFit, newdata, submodels = NULL) {
## out <- try(kernlab::predict(modelFit, newdata, type="probabilities"),
## silent = TRUE)
## if(class(out)[1] != "try-error") {
## ## There are times when the SVM probability model will
## ## produce negative class probabilities, so we
## ## induce vlaues between 0 and 1
## if(any(out < 0)) {
## out[out < 0] <- 0
## out <- t(apply(out, 1, function(x) x/sum(x)))
## }
## out <- out[, kernlab::lev(modelFit), drop = FALSE]
## } else {
## warning("kernlab class probability calculations failed; returning NAs")
## out <- matrix(NA, nrow(newdata) * length(kernlab::lev(modelFit)), ncol = length(kernlab::lev(modelFit)))
## colnames(out) <- kernlab::lev(modelFit)
## }
## out
## }
##
## $svmPoly$predictors
## function(x, ...){
## if(hasTerms(x) & !is.null(x@terms)) {
## out <- predictors.terms(x@terms)
## } else {
## out <- colnames(attr(x, "xmatrix"))
## }
## if(is.null(out)) out <- names(attr(x, "scaling")$xscale$`scaled:center`)
## if(is.null(out)) out <-NA
## out
## }
##
## $svmPoly$tags
## [1] "Kernel Method" "Support Vector Machines"
## [3] "Polynomial Model" "Robust Methods"
##
## $svmPoly$levels
## function(x) kernlab::lev(x)
##
## $svmPoly$sort
## function(x) x[order(x$degree, x$C, x$scale),]#6.1 Using tune Grid
if (version$major>5){
RNGkind(sample.kind = "Rounding")
}
set.seed(2019)
ResamplingControlCl <- trainControl(
method = "repeatedcv",
number = 10,
repeats = 5, verboseIter = TRUE)Creating grid
ParameterGridCl<- expand.grid(C=c(0.01, 0.1, 0.5),scale=c(1), degree=c(1,2,3))Training the Model
ModelClBestSvm<-train(as.factor(LoanStatusOk)~.,
data=TrainDataSet,
method = "svmPoly",
trControl=ResamplingControlCl,tuneGrid=ParameterGridCl)## + Fold01.Rep1: C=0.01, scale=1, degree=1
## - Fold01.Rep1: C=0.01, scale=1, degree=1
## + Fold01.Rep1: C=0.10, scale=1, degree=1
## - Fold01.Rep1: C=0.10, scale=1, degree=1
## + Fold01.Rep1: C=0.50, scale=1, degree=1
## - Fold01.Rep1: C=0.50, scale=1, degree=1
## + Fold01.Rep1: C=0.01, scale=1, degree=2
## - Fold01.Rep1: C=0.01, scale=1, degree=2
## + Fold01.Rep1: C=0.10, scale=1, degree=2
## - Fold01.Rep1: C=0.10, scale=1, degree=2
## + Fold01.Rep1: C=0.50, scale=1, degree=2
## - Fold01.Rep1: C=0.50, scale=1, degree=2
## + Fold01.Rep1: C=0.01, scale=1, degree=3
## - Fold01.Rep1: C=0.01, scale=1, degree=3
## + Fold01.Rep1: C=0.10, scale=1, degree=3
## - Fold01.Rep1: C=0.10, scale=1, degree=3
## + Fold01.Rep1: C=0.50, scale=1, degree=3
## - Fold01.Rep1: C=0.50, scale=1, degree=3
## + Fold02.Rep1: C=0.01, scale=1, degree=1
## - Fold02.Rep1: C=0.01, scale=1, degree=1
## + Fold02.Rep1: C=0.10, scale=1, degree=1
## - Fold02.Rep1: C=0.10, scale=1, degree=1
## + Fold02.Rep1: C=0.50, scale=1, degree=1
## - Fold02.Rep1: C=0.50, scale=1, degree=1
## + Fold02.Rep1: C=0.01, scale=1, degree=2
## - Fold02.Rep1: C=0.01, scale=1, degree=2
## + Fold02.Rep1: C=0.10, scale=1, degree=2
## - Fold02.Rep1: C=0.10, scale=1, degree=2
## + Fold02.Rep1: C=0.50, scale=1, degree=2
## - Fold02.Rep1: C=0.50, scale=1, degree=2
## + Fold02.Rep1: C=0.01, scale=1, degree=3
## - Fold02.Rep1: C=0.01, scale=1, degree=3
## + Fold02.Rep1: C=0.10, scale=1, degree=3
## - Fold02.Rep1: C=0.10, scale=1, degree=3
## + Fold02.Rep1: C=0.50, scale=1, degree=3
## - Fold02.Rep1: C=0.50, scale=1, degree=3
## + Fold03.Rep1: C=0.01, scale=1, degree=1
## - Fold03.Rep1: C=0.01, scale=1, degree=1
## + Fold03.Rep1: C=0.10, scale=1, degree=1
## - Fold03.Rep1: C=0.10, scale=1, degree=1
## + Fold03.Rep1: C=0.50, scale=1, degree=1
## - Fold03.Rep1: C=0.50, scale=1, degree=1
## + Fold03.Rep1: C=0.01, scale=1, degree=2
## - Fold03.Rep1: C=0.01, scale=1, degree=2
## + Fold03.Rep1: C=0.10, scale=1, degree=2
## - Fold03.Rep1: C=0.10, scale=1, degree=2
## + Fold03.Rep1: C=0.50, scale=1, degree=2
## - Fold03.Rep1: C=0.50, scale=1, degree=2
## + Fold03.Rep1: C=0.01, scale=1, degree=3
## - Fold03.Rep1: C=0.01, scale=1, degree=3
## + Fold03.Rep1: C=0.10, scale=1, degree=3
## - Fold03.Rep1: C=0.10, scale=1, degree=3
## + Fold03.Rep1: C=0.50, scale=1, degree=3
## - Fold03.Rep1: C=0.50, scale=1, degree=3
## + Fold04.Rep1: C=0.01, scale=1, degree=1
## - Fold04.Rep1: C=0.01, scale=1, degree=1
## + Fold04.Rep1: C=0.10, scale=1, degree=1
## - Fold04.Rep1: C=0.10, scale=1, degree=1
## + Fold04.Rep1: C=0.50, scale=1, degree=1
## - Fold04.Rep1: C=0.50, scale=1, degree=1
## + Fold04.Rep1: C=0.01, scale=1, degree=2
## - Fold04.Rep1: C=0.01, scale=1, degree=2
## + Fold04.Rep1: C=0.10, scale=1, degree=2
## - Fold04.Rep1: C=0.10, scale=1, degree=2
## + Fold04.Rep1: C=0.50, scale=1, degree=2
## - Fold04.Rep1: C=0.50, scale=1, degree=2
## + Fold04.Rep1: C=0.01, scale=1, degree=3
## - Fold04.Rep1: C=0.01, scale=1, degree=3
## + Fold04.Rep1: C=0.10, scale=1, degree=3
## - Fold04.Rep1: C=0.10, scale=1, degree=3
## + Fold04.Rep1: C=0.50, scale=1, degree=3
## - Fold04.Rep1: C=0.50, scale=1, degree=3
## + Fold05.Rep1: C=0.01, scale=1, degree=1
## - Fold05.Rep1: C=0.01, scale=1, degree=1
## + Fold05.Rep1: C=0.10, scale=1, degree=1
## - Fold05.Rep1: C=0.10, scale=1, degree=1
## + Fold05.Rep1: C=0.50, scale=1, degree=1
## - Fold05.Rep1: C=0.50, scale=1, degree=1
## + Fold05.Rep1: C=0.01, scale=1, degree=2
## - Fold05.Rep1: C=0.01, scale=1, degree=2
## + Fold05.Rep1: C=0.10, scale=1, degree=2
## - Fold05.Rep1: C=0.10, scale=1, degree=2
## + Fold05.Rep1: C=0.50, scale=1, degree=2
## - Fold05.Rep1: C=0.50, scale=1, degree=2
## + Fold05.Rep1: C=0.01, scale=1, degree=3
## - Fold05.Rep1: C=0.01, scale=1, degree=3
## + Fold05.Rep1: C=0.10, scale=1, degree=3
## - Fold05.Rep1: C=0.10, scale=1, degree=3
## + Fold05.Rep1: C=0.50, scale=1, degree=3
## - Fold05.Rep1: C=0.50, scale=1, degree=3
## + Fold06.Rep1: C=0.01, scale=1, degree=1
## - Fold06.Rep1: C=0.01, scale=1, degree=1
## + Fold06.Rep1: C=0.10, scale=1, degree=1
## - Fold06.Rep1: C=0.10, scale=1, degree=1
## + Fold06.Rep1: C=0.50, scale=1, degree=1
## - Fold06.Rep1: C=0.50, scale=1, degree=1
## + Fold06.Rep1: C=0.01, scale=1, degree=2
## - Fold06.Rep1: C=0.01, scale=1, degree=2
## + Fold06.Rep1: C=0.10, scale=1, degree=2
## - Fold06.Rep1: C=0.10, scale=1, degree=2
## + Fold06.Rep1: C=0.50, scale=1, degree=2
## - Fold06.Rep1: C=0.50, scale=1, degree=2
## + Fold06.Rep1: C=0.01, scale=1, degree=3
## - Fold06.Rep1: C=0.01, scale=1, degree=3
## + Fold06.Rep1: C=0.10, scale=1, degree=3
## - Fold06.Rep1: C=0.10, scale=1, degree=3
## + Fold06.Rep1: C=0.50, scale=1, degree=3
## - Fold06.Rep1: C=0.50, scale=1, degree=3
## + Fold07.Rep1: C=0.01, scale=1, degree=1
## - Fold07.Rep1: C=0.01, scale=1, degree=1
## + Fold07.Rep1: C=0.10, scale=1, degree=1
## - Fold07.Rep1: C=0.10, scale=1, degree=1
## + Fold07.Rep1: C=0.50, scale=1, degree=1
## - Fold07.Rep1: C=0.50, scale=1, degree=1
## + Fold07.Rep1: C=0.01, scale=1, degree=2
## - Fold07.Rep1: C=0.01, scale=1, degree=2
## + Fold07.Rep1: C=0.10, scale=1, degree=2
## - Fold07.Rep1: C=0.10, scale=1, degree=2
## + Fold07.Rep1: C=0.50, scale=1, degree=2
## - Fold07.Rep1: C=0.50, scale=1, degree=2
## + Fold07.Rep1: C=0.01, scale=1, degree=3
## - Fold07.Rep1: C=0.01, scale=1, degree=3
## + Fold07.Rep1: C=0.10, scale=1, degree=3
## - Fold07.Rep1: C=0.10, scale=1, degree=3
## + Fold07.Rep1: C=0.50, scale=1, degree=3
## - Fold07.Rep1: C=0.50, scale=1, degree=3
## + Fold08.Rep1: C=0.01, scale=1, degree=1
## - Fold08.Rep1: C=0.01, scale=1, degree=1
## + Fold08.Rep1: C=0.10, scale=1, degree=1
## - Fold08.Rep1: C=0.10, scale=1, degree=1
## + Fold08.Rep1: C=0.50, scale=1, degree=1
## - Fold08.Rep1: C=0.50, scale=1, degree=1
## + Fold08.Rep1: C=0.01, scale=1, degree=2
## - Fold08.Rep1: C=0.01, scale=1, degree=2
## + Fold08.Rep1: C=0.10, scale=1, degree=2
## - Fold08.Rep1: C=0.10, scale=1, degree=2
## + Fold08.Rep1: C=0.50, scale=1, degree=2
## - Fold08.Rep1: C=0.50, scale=1, degree=2
## + Fold08.Rep1: C=0.01, scale=1, degree=3
## - Fold08.Rep1: C=0.01, scale=1, degree=3
## + Fold08.Rep1: C=0.10, scale=1, degree=3
## - Fold08.Rep1: C=0.10, scale=1, degree=3
## + Fold08.Rep1: C=0.50, scale=1, degree=3
## - Fold08.Rep1: C=0.50, scale=1, degree=3
## + Fold09.Rep1: C=0.01, scale=1, degree=1
## - Fold09.Rep1: C=0.01, scale=1, degree=1
## + Fold09.Rep1: C=0.10, scale=1, degree=1
## - Fold09.Rep1: C=0.10, scale=1, degree=1
## + Fold09.Rep1: C=0.50, scale=1, degree=1
## - Fold09.Rep1: C=0.50, scale=1, degree=1
## + Fold09.Rep1: C=0.01, scale=1, degree=2
## - Fold09.Rep1: C=0.01, scale=1, degree=2
## + Fold09.Rep1: C=0.10, scale=1, degree=2
## - Fold09.Rep1: C=0.10, scale=1, degree=2
## + Fold09.Rep1: C=0.50, scale=1, degree=2
## - Fold09.Rep1: C=0.50, scale=1, degree=2
## + Fold09.Rep1: C=0.01, scale=1, degree=3
## - Fold09.Rep1: C=0.01, scale=1, degree=3
## + Fold09.Rep1: C=0.10, scale=1, degree=3
## - Fold09.Rep1: C=0.10, scale=1, degree=3
## + Fold09.Rep1: C=0.50, scale=1, degree=3
## - Fold09.Rep1: C=0.50, scale=1, degree=3
## + Fold10.Rep1: C=0.01, scale=1, degree=1
## - Fold10.Rep1: C=0.01, scale=1, degree=1
## + Fold10.Rep1: C=0.10, scale=1, degree=1
## - Fold10.Rep1: C=0.10, scale=1, degree=1
## + Fold10.Rep1: C=0.50, scale=1, degree=1
## - Fold10.Rep1: C=0.50, scale=1, degree=1
## + Fold10.Rep1: C=0.01, scale=1, degree=2
## - Fold10.Rep1: C=0.01, scale=1, degree=2
## + Fold10.Rep1: C=0.10, scale=1, degree=2
## - Fold10.Rep1: C=0.10, scale=1, degree=2
## + Fold10.Rep1: C=0.50, scale=1, degree=2
## - Fold10.Rep1: C=0.50, scale=1, degree=2
## + Fold10.Rep1: C=0.01, scale=1, degree=3
## - Fold10.Rep1: C=0.01, scale=1, degree=3
## + Fold10.Rep1: C=0.10, scale=1, degree=3
## - Fold10.Rep1: C=0.10, scale=1, degree=3
## + Fold10.Rep1: C=0.50, scale=1, degree=3
## - Fold10.Rep1: C=0.50, scale=1, degree=3
## + Fold01.Rep2: C=0.01, scale=1, degree=1
## - Fold01.Rep2: C=0.01, scale=1, degree=1
## + Fold01.Rep2: C=0.10, scale=1, degree=1
## - Fold01.Rep2: C=0.10, scale=1, degree=1
## + Fold01.Rep2: C=0.50, scale=1, degree=1
## - Fold01.Rep2: C=0.50, scale=1, degree=1
## + Fold01.Rep2: C=0.01, scale=1, degree=2
## - Fold01.Rep2: C=0.01, scale=1, degree=2
## + Fold01.Rep2: C=0.10, scale=1, degree=2
## - Fold01.Rep2: C=0.10, scale=1, degree=2
## + Fold01.Rep2: C=0.50, scale=1, degree=2
## - Fold01.Rep2: C=0.50, scale=1, degree=2
## + Fold01.Rep2: C=0.01, scale=1, degree=3
## - Fold01.Rep2: C=0.01, scale=1, degree=3
## + Fold01.Rep2: C=0.10, scale=1, degree=3
## - Fold01.Rep2: C=0.10, scale=1, degree=3
## + Fold01.Rep2: C=0.50, scale=1, degree=3
## - Fold01.Rep2: C=0.50, scale=1, degree=3
## + Fold02.Rep2: C=0.01, scale=1, degree=1
## - Fold02.Rep2: C=0.01, scale=1, degree=1
## + Fold02.Rep2: C=0.10, scale=1, degree=1
## - Fold02.Rep2: C=0.10, scale=1, degree=1
## + Fold02.Rep2: C=0.50, scale=1, degree=1
## - Fold02.Rep2: C=0.50, scale=1, degree=1
## + Fold02.Rep2: C=0.01, scale=1, degree=2
## - Fold02.Rep2: C=0.01, scale=1, degree=2
## + Fold02.Rep2: C=0.10, scale=1, degree=2
## - Fold02.Rep2: C=0.10, scale=1, degree=2
## + Fold02.Rep2: C=0.50, scale=1, degree=2
## - Fold02.Rep2: C=0.50, scale=1, degree=2
## + Fold02.Rep2: C=0.01, scale=1, degree=3
## - Fold02.Rep2: C=0.01, scale=1, degree=3
## + Fold02.Rep2: C=0.10, scale=1, degree=3
## - Fold02.Rep2: C=0.10, scale=1, degree=3
## + Fold02.Rep2: C=0.50, scale=1, degree=3
## - Fold02.Rep2: C=0.50, scale=1, degree=3
## + Fold03.Rep2: C=0.01, scale=1, degree=1
## - Fold03.Rep2: C=0.01, scale=1, degree=1
## + Fold03.Rep2: C=0.10, scale=1, degree=1
## - Fold03.Rep2: C=0.10, scale=1, degree=1
## + Fold03.Rep2: C=0.50, scale=1, degree=1
## - Fold03.Rep2: C=0.50, scale=1, degree=1
## + Fold03.Rep2: C=0.01, scale=1, degree=2
## - Fold03.Rep2: C=0.01, scale=1, degree=2
## + Fold03.Rep2: C=0.10, scale=1, degree=2
## - Fold03.Rep2: C=0.10, scale=1, degree=2
## + Fold03.Rep2: C=0.50, scale=1, degree=2
## - Fold03.Rep2: C=0.50, scale=1, degree=2
## + Fold03.Rep2: C=0.01, scale=1, degree=3
## - Fold03.Rep2: C=0.01, scale=1, degree=3
## + Fold03.Rep2: C=0.10, scale=1, degree=3
## - Fold03.Rep2: C=0.10, scale=1, degree=3
## + Fold03.Rep2: C=0.50, scale=1, degree=3
## - Fold03.Rep2: C=0.50, scale=1, degree=3
## + Fold04.Rep2: C=0.01, scale=1, degree=1
## - Fold04.Rep2: C=0.01, scale=1, degree=1
## + Fold04.Rep2: C=0.10, scale=1, degree=1
## - Fold04.Rep2: C=0.10, scale=1, degree=1
## + Fold04.Rep2: C=0.50, scale=1, degree=1
## - Fold04.Rep2: C=0.50, scale=1, degree=1
## + Fold04.Rep2: C=0.01, scale=1, degree=2
## - Fold04.Rep2: C=0.01, scale=1, degree=2
## + Fold04.Rep2: C=0.10, scale=1, degree=2
## - Fold04.Rep2: C=0.10, scale=1, degree=2
## + Fold04.Rep2: C=0.50, scale=1, degree=2
## - Fold04.Rep2: C=0.50, scale=1, degree=2
## + Fold04.Rep2: C=0.01, scale=1, degree=3
## - Fold04.Rep2: C=0.01, scale=1, degree=3
## + Fold04.Rep2: C=0.10, scale=1, degree=3
## - Fold04.Rep2: C=0.10, scale=1, degree=3
## + Fold04.Rep2: C=0.50, scale=1, degree=3
## - Fold04.Rep2: C=0.50, scale=1, degree=3
## + Fold05.Rep2: C=0.01, scale=1, degree=1
## - Fold05.Rep2: C=0.01, scale=1, degree=1
## + Fold05.Rep2: C=0.10, scale=1, degree=1
## - Fold05.Rep2: C=0.10, scale=1, degree=1
## + Fold05.Rep2: C=0.50, scale=1, degree=1
## - Fold05.Rep2: C=0.50, scale=1, degree=1
## + Fold05.Rep2: C=0.01, scale=1, degree=2
## - Fold05.Rep2: C=0.01, scale=1, degree=2
## + Fold05.Rep2: C=0.10, scale=1, degree=2
## - Fold05.Rep2: C=0.10, scale=1, degree=2
## + Fold05.Rep2: C=0.50, scale=1, degree=2
## - Fold05.Rep2: C=0.50, scale=1, degree=2
## + Fold05.Rep2: C=0.01, scale=1, degree=3
## - Fold05.Rep2: C=0.01, scale=1, degree=3
## + Fold05.Rep2: C=0.10, scale=1, degree=3
## - Fold05.Rep2: C=0.10, scale=1, degree=3
## + Fold05.Rep2: C=0.50, scale=1, degree=3
## - Fold05.Rep2: C=0.50, scale=1, degree=3
## + Fold06.Rep2: C=0.01, scale=1, degree=1
## - Fold06.Rep2: C=0.01, scale=1, degree=1
## + Fold06.Rep2: C=0.10, scale=1, degree=1
## - Fold06.Rep2: C=0.10, scale=1, degree=1
## + Fold06.Rep2: C=0.50, scale=1, degree=1
## - Fold06.Rep2: C=0.50, scale=1, degree=1
## + Fold06.Rep2: C=0.01, scale=1, degree=2
## - Fold06.Rep2: C=0.01, scale=1, degree=2
## + Fold06.Rep2: C=0.10, scale=1, degree=2
## - Fold06.Rep2: C=0.10, scale=1, degree=2
## + Fold06.Rep2: C=0.50, scale=1, degree=2
## - Fold06.Rep2: C=0.50, scale=1, degree=2
## + Fold06.Rep2: C=0.01, scale=1, degree=3
## - Fold06.Rep2: C=0.01, scale=1, degree=3
## + Fold06.Rep2: C=0.10, scale=1, degree=3
## - Fold06.Rep2: C=0.10, scale=1, degree=3
## + Fold06.Rep2: C=0.50, scale=1, degree=3
## - Fold06.Rep2: C=0.50, scale=1, degree=3
## + Fold07.Rep2: C=0.01, scale=1, degree=1
## - Fold07.Rep2: C=0.01, scale=1, degree=1
## + Fold07.Rep2: C=0.10, scale=1, degree=1
## - Fold07.Rep2: C=0.10, scale=1, degree=1
## + Fold07.Rep2: C=0.50, scale=1, degree=1
## - Fold07.Rep2: C=0.50, scale=1, degree=1
## + Fold07.Rep2: C=0.01, scale=1, degree=2
## - Fold07.Rep2: C=0.01, scale=1, degree=2
## + Fold07.Rep2: C=0.10, scale=1, degree=2
## - Fold07.Rep2: C=0.10, scale=1, degree=2
## + Fold07.Rep2: C=0.50, scale=1, degree=2
## - Fold07.Rep2: C=0.50, scale=1, degree=2
## + Fold07.Rep2: C=0.01, scale=1, degree=3
## - Fold07.Rep2: C=0.01, scale=1, degree=3
## + Fold07.Rep2: C=0.10, scale=1, degree=3
## - Fold07.Rep2: C=0.10, scale=1, degree=3
## + Fold07.Rep2: C=0.50, scale=1, degree=3
## - Fold07.Rep2: C=0.50, scale=1, degree=3
## + Fold08.Rep2: C=0.01, scale=1, degree=1
## - Fold08.Rep2: C=0.01, scale=1, degree=1
## + Fold08.Rep2: C=0.10, scale=1, degree=1
## - Fold08.Rep2: C=0.10, scale=1, degree=1
## + Fold08.Rep2: C=0.50, scale=1, degree=1
## - Fold08.Rep2: C=0.50, scale=1, degree=1
## + Fold08.Rep2: C=0.01, scale=1, degree=2
## - Fold08.Rep2: C=0.01, scale=1, degree=2
## + Fold08.Rep2: C=0.10, scale=1, degree=2
## - Fold08.Rep2: C=0.10, scale=1, degree=2
## + Fold08.Rep2: C=0.50, scale=1, degree=2
## - Fold08.Rep2: C=0.50, scale=1, degree=2
## + Fold08.Rep2: C=0.01, scale=1, degree=3
## - Fold08.Rep2: C=0.01, scale=1, degree=3
## + Fold08.Rep2: C=0.10, scale=1, degree=3
## - Fold08.Rep2: C=0.10, scale=1, degree=3
## + Fold08.Rep2: C=0.50, scale=1, degree=3
## - Fold08.Rep2: C=0.50, scale=1, degree=3
## + Fold09.Rep2: C=0.01, scale=1, degree=1
## - Fold09.Rep2: C=0.01, scale=1, degree=1
## + Fold09.Rep2: C=0.10, scale=1, degree=1
## - Fold09.Rep2: C=0.10, scale=1, degree=1
## + Fold09.Rep2: C=0.50, scale=1, degree=1
## - Fold09.Rep2: C=0.50, scale=1, degree=1
## + Fold09.Rep2: C=0.01, scale=1, degree=2
## - Fold09.Rep2: C=0.01, scale=1, degree=2
## + Fold09.Rep2: C=0.10, scale=1, degree=2
## - Fold09.Rep2: C=0.10, scale=1, degree=2
## + Fold09.Rep2: C=0.50, scale=1, degree=2
## - Fold09.Rep2: C=0.50, scale=1, degree=2
## + Fold09.Rep2: C=0.01, scale=1, degree=3
## - Fold09.Rep2: C=0.01, scale=1, degree=3
## + Fold09.Rep2: C=0.10, scale=1, degree=3
## - Fold09.Rep2: C=0.10, scale=1, degree=3
## + Fold09.Rep2: C=0.50, scale=1, degree=3
## - Fold09.Rep2: C=0.50, scale=1, degree=3
## + Fold10.Rep2: C=0.01, scale=1, degree=1
## - Fold10.Rep2: C=0.01, scale=1, degree=1
## + Fold10.Rep2: C=0.10, scale=1, degree=1
## - Fold10.Rep2: C=0.10, scale=1, degree=1
## + Fold10.Rep2: C=0.50, scale=1, degree=1
## - Fold10.Rep2: C=0.50, scale=1, degree=1
## + Fold10.Rep2: C=0.01, scale=1, degree=2
## - Fold10.Rep2: C=0.01, scale=1, degree=2
## + Fold10.Rep2: C=0.10, scale=1, degree=2
## - Fold10.Rep2: C=0.10, scale=1, degree=2
## + Fold10.Rep2: C=0.50, scale=1, degree=2
## - Fold10.Rep2: C=0.50, scale=1, degree=2
## + Fold10.Rep2: C=0.01, scale=1, degree=3
## - Fold10.Rep2: C=0.01, scale=1, degree=3
## + Fold10.Rep2: C=0.10, scale=1, degree=3
## - Fold10.Rep2: C=0.10, scale=1, degree=3
## + Fold10.Rep2: C=0.50, scale=1, degree=3
## - Fold10.Rep2: C=0.50, scale=1, degree=3
## + Fold01.Rep3: C=0.01, scale=1, degree=1
## - Fold01.Rep3: C=0.01, scale=1, degree=1
## + Fold01.Rep3: C=0.10, scale=1, degree=1
## - Fold01.Rep3: C=0.10, scale=1, degree=1
## + Fold01.Rep3: C=0.50, scale=1, degree=1
## - Fold01.Rep3: C=0.50, scale=1, degree=1
## + Fold01.Rep3: C=0.01, scale=1, degree=2
## - Fold01.Rep3: C=0.01, scale=1, degree=2
## + Fold01.Rep3: C=0.10, scale=1, degree=2
## - Fold01.Rep3: C=0.10, scale=1, degree=2
## + Fold01.Rep3: C=0.50, scale=1, degree=2
## - Fold01.Rep3: C=0.50, scale=1, degree=2
## + Fold01.Rep3: C=0.01, scale=1, degree=3
## - Fold01.Rep3: C=0.01, scale=1, degree=3
## + Fold01.Rep3: C=0.10, scale=1, degree=3
## - Fold01.Rep3: C=0.10, scale=1, degree=3
## + Fold01.Rep3: C=0.50, scale=1, degree=3
## - Fold01.Rep3: C=0.50, scale=1, degree=3
## + Fold02.Rep3: C=0.01, scale=1, degree=1
## - Fold02.Rep3: C=0.01, scale=1, degree=1
## + Fold02.Rep3: C=0.10, scale=1, degree=1
## - Fold02.Rep3: C=0.10, scale=1, degree=1
## + Fold02.Rep3: C=0.50, scale=1, degree=1
## - Fold02.Rep3: C=0.50, scale=1, degree=1
## + Fold02.Rep3: C=0.01, scale=1, degree=2
## - Fold02.Rep3: C=0.01, scale=1, degree=2
## + Fold02.Rep3: C=0.10, scale=1, degree=2
## - Fold02.Rep3: C=0.10, scale=1, degree=2
## + Fold02.Rep3: C=0.50, scale=1, degree=2
## - Fold02.Rep3: C=0.50, scale=1, degree=2
## + Fold02.Rep3: C=0.01, scale=1, degree=3
## - Fold02.Rep3: C=0.01, scale=1, degree=3
## + Fold02.Rep3: C=0.10, scale=1, degree=3
## - Fold02.Rep3: C=0.10, scale=1, degree=3
## + Fold02.Rep3: C=0.50, scale=1, degree=3
## - Fold02.Rep3: C=0.50, scale=1, degree=3
## + Fold03.Rep3: C=0.01, scale=1, degree=1
## - Fold03.Rep3: C=0.01, scale=1, degree=1
## + Fold03.Rep3: C=0.10, scale=1, degree=1
## - Fold03.Rep3: C=0.10, scale=1, degree=1
## + Fold03.Rep3: C=0.50, scale=1, degree=1
## - Fold03.Rep3: C=0.50, scale=1, degree=1
## + Fold03.Rep3: C=0.01, scale=1, degree=2
## - Fold03.Rep3: C=0.01, scale=1, degree=2
## + Fold03.Rep3: C=0.10, scale=1, degree=2
## - Fold03.Rep3: C=0.10, scale=1, degree=2
## + Fold03.Rep3: C=0.50, scale=1, degree=2
## - Fold03.Rep3: C=0.50, scale=1, degree=2
## + Fold03.Rep3: C=0.01, scale=1, degree=3
## - Fold03.Rep3: C=0.01, scale=1, degree=3
## + Fold03.Rep3: C=0.10, scale=1, degree=3
## - Fold03.Rep3: C=0.10, scale=1, degree=3
## + Fold03.Rep3: C=0.50, scale=1, degree=3
## - Fold03.Rep3: C=0.50, scale=1, degree=3
## + Fold04.Rep3: C=0.01, scale=1, degree=1
## - Fold04.Rep3: C=0.01, scale=1, degree=1
## + Fold04.Rep3: C=0.10, scale=1, degree=1
## - Fold04.Rep3: C=0.10, scale=1, degree=1
## + Fold04.Rep3: C=0.50, scale=1, degree=1
## - Fold04.Rep3: C=0.50, scale=1, degree=1
## + Fold04.Rep3: C=0.01, scale=1, degree=2
## - Fold04.Rep3: C=0.01, scale=1, degree=2
## + Fold04.Rep3: C=0.10, scale=1, degree=2
## - Fold04.Rep3: C=0.10, scale=1, degree=2
## + Fold04.Rep3: C=0.50, scale=1, degree=2
## - Fold04.Rep3: C=0.50, scale=1, degree=2
## + Fold04.Rep3: C=0.01, scale=1, degree=3
## - Fold04.Rep3: C=0.01, scale=1, degree=3
## + Fold04.Rep3: C=0.10, scale=1, degree=3
## - Fold04.Rep3: C=0.10, scale=1, degree=3
## + Fold04.Rep3: C=0.50, scale=1, degree=3
## - Fold04.Rep3: C=0.50, scale=1, degree=3
## + Fold05.Rep3: C=0.01, scale=1, degree=1
## - Fold05.Rep3: C=0.01, scale=1, degree=1
## + Fold05.Rep3: C=0.10, scale=1, degree=1
## - Fold05.Rep3: C=0.10, scale=1, degree=1
## + Fold05.Rep3: C=0.50, scale=1, degree=1
## - Fold05.Rep3: C=0.50, scale=1, degree=1
## + Fold05.Rep3: C=0.01, scale=1, degree=2
## - Fold05.Rep3: C=0.01, scale=1, degree=2
## + Fold05.Rep3: C=0.10, scale=1, degree=2
## - Fold05.Rep3: C=0.10, scale=1, degree=2
## + Fold05.Rep3: C=0.50, scale=1, degree=2
## - Fold05.Rep3: C=0.50, scale=1, degree=2
## + Fold05.Rep3: C=0.01, scale=1, degree=3
## - Fold05.Rep3: C=0.01, scale=1, degree=3
## + Fold05.Rep3: C=0.10, scale=1, degree=3
## - Fold05.Rep3: C=0.10, scale=1, degree=3
## + Fold05.Rep3: C=0.50, scale=1, degree=3
## - Fold05.Rep3: C=0.50, scale=1, degree=3
## + Fold06.Rep3: C=0.01, scale=1, degree=1
## - Fold06.Rep3: C=0.01, scale=1, degree=1
## + Fold06.Rep3: C=0.10, scale=1, degree=1
## - Fold06.Rep3: C=0.10, scale=1, degree=1
## + Fold06.Rep3: C=0.50, scale=1, degree=1
## - Fold06.Rep3: C=0.50, scale=1, degree=1
## + Fold06.Rep3: C=0.01, scale=1, degree=2
## - Fold06.Rep3: C=0.01, scale=1, degree=2
## + Fold06.Rep3: C=0.10, scale=1, degree=2
## - Fold06.Rep3: C=0.10, scale=1, degree=2
## + Fold06.Rep3: C=0.50, scale=1, degree=2
## - Fold06.Rep3: C=0.50, scale=1, degree=2
## + Fold06.Rep3: C=0.01, scale=1, degree=3
## - Fold06.Rep3: C=0.01, scale=1, degree=3
## + Fold06.Rep3: C=0.10, scale=1, degree=3
## - Fold06.Rep3: C=0.10, scale=1, degree=3
## + Fold06.Rep3: C=0.50, scale=1, degree=3
## - Fold06.Rep3: C=0.50, scale=1, degree=3
## + Fold07.Rep3: C=0.01, scale=1, degree=1
## - Fold07.Rep3: C=0.01, scale=1, degree=1
## + Fold07.Rep3: C=0.10, scale=1, degree=1
## - Fold07.Rep3: C=0.10, scale=1, degree=1
## + Fold07.Rep3: C=0.50, scale=1, degree=1
## - Fold07.Rep3: C=0.50, scale=1, degree=1
## + Fold07.Rep3: C=0.01, scale=1, degree=2
## - Fold07.Rep3: C=0.01, scale=1, degree=2
## + Fold07.Rep3: C=0.10, scale=1, degree=2
## - Fold07.Rep3: C=0.10, scale=1, degree=2
## + Fold07.Rep3: C=0.50, scale=1, degree=2
## - Fold07.Rep3: C=0.50, scale=1, degree=2
## + Fold07.Rep3: C=0.01, scale=1, degree=3
## - Fold07.Rep3: C=0.01, scale=1, degree=3
## + Fold07.Rep3: C=0.10, scale=1, degree=3
## - Fold07.Rep3: C=0.10, scale=1, degree=3
## + Fold07.Rep3: C=0.50, scale=1, degree=3
## - Fold07.Rep3: C=0.50, scale=1, degree=3
## + Fold08.Rep3: C=0.01, scale=1, degree=1
## - Fold08.Rep3: C=0.01, scale=1, degree=1
## + Fold08.Rep3: C=0.10, scale=1, degree=1
## - Fold08.Rep3: C=0.10, scale=1, degree=1
## + Fold08.Rep3: C=0.50, scale=1, degree=1
## - Fold08.Rep3: C=0.50, scale=1, degree=1
## + Fold08.Rep3: C=0.01, scale=1, degree=2
## - Fold08.Rep3: C=0.01, scale=1, degree=2
## + Fold08.Rep3: C=0.10, scale=1, degree=2
## - Fold08.Rep3: C=0.10, scale=1, degree=2
## + Fold08.Rep3: C=0.50, scale=1, degree=2
## - Fold08.Rep3: C=0.50, scale=1, degree=2
## + Fold08.Rep3: C=0.01, scale=1, degree=3
## - Fold08.Rep3: C=0.01, scale=1, degree=3
## + Fold08.Rep3: C=0.10, scale=1, degree=3
## - Fold08.Rep3: C=0.10, scale=1, degree=3
## + Fold08.Rep3: C=0.50, scale=1, degree=3
## - Fold08.Rep3: C=0.50, scale=1, degree=3
## + Fold09.Rep3: C=0.01, scale=1, degree=1
## - Fold09.Rep3: C=0.01, scale=1, degree=1
## + Fold09.Rep3: C=0.10, scale=1, degree=1
## - Fold09.Rep3: C=0.10, scale=1, degree=1
## + Fold09.Rep3: C=0.50, scale=1, degree=1
## - Fold09.Rep3: C=0.50, scale=1, degree=1
## + Fold09.Rep3: C=0.01, scale=1, degree=2
## - Fold09.Rep3: C=0.01, scale=1, degree=2
## + Fold09.Rep3: C=0.10, scale=1, degree=2
## - Fold09.Rep3: C=0.10, scale=1, degree=2
## + Fold09.Rep3: C=0.50, scale=1, degree=2
## - Fold09.Rep3: C=0.50, scale=1, degree=2
## + Fold09.Rep3: C=0.01, scale=1, degree=3
## - Fold09.Rep3: C=0.01, scale=1, degree=3
## + Fold09.Rep3: C=0.10, scale=1, degree=3
## - Fold09.Rep3: C=0.10, scale=1, degree=3
## + Fold09.Rep3: C=0.50, scale=1, degree=3
## - Fold09.Rep3: C=0.50, scale=1, degree=3
## + Fold10.Rep3: C=0.01, scale=1, degree=1
## - Fold10.Rep3: C=0.01, scale=1, degree=1
## + Fold10.Rep3: C=0.10, scale=1, degree=1
## - Fold10.Rep3: C=0.10, scale=1, degree=1
## + Fold10.Rep3: C=0.50, scale=1, degree=1
## - Fold10.Rep3: C=0.50, scale=1, degree=1
## + Fold10.Rep3: C=0.01, scale=1, degree=2
## - Fold10.Rep3: C=0.01, scale=1, degree=2
## + Fold10.Rep3: C=0.10, scale=1, degree=2
## - Fold10.Rep3: C=0.10, scale=1, degree=2
## + Fold10.Rep3: C=0.50, scale=1, degree=2
## - Fold10.Rep3: C=0.50, scale=1, degree=2
## + Fold10.Rep3: C=0.01, scale=1, degree=3
## - Fold10.Rep3: C=0.01, scale=1, degree=3
## + Fold10.Rep3: C=0.10, scale=1, degree=3
## - Fold10.Rep3: C=0.10, scale=1, degree=3
## + Fold10.Rep3: C=0.50, scale=1, degree=3
## - Fold10.Rep3: C=0.50, scale=1, degree=3
## + Fold01.Rep4: C=0.01, scale=1, degree=1
## - Fold01.Rep4: C=0.01, scale=1, degree=1
## + Fold01.Rep4: C=0.10, scale=1, degree=1
## - Fold01.Rep4: C=0.10, scale=1, degree=1
## + Fold01.Rep4: C=0.50, scale=1, degree=1
## - Fold01.Rep4: C=0.50, scale=1, degree=1
## + Fold01.Rep4: C=0.01, scale=1, degree=2
## - Fold01.Rep4: C=0.01, scale=1, degree=2
## + Fold01.Rep4: C=0.10, scale=1, degree=2
## - Fold01.Rep4: C=0.10, scale=1, degree=2
## + Fold01.Rep4: C=0.50, scale=1, degree=2
## - Fold01.Rep4: C=0.50, scale=1, degree=2
## + Fold01.Rep4: C=0.01, scale=1, degree=3
## - Fold01.Rep4: C=0.01, scale=1, degree=3
## + Fold01.Rep4: C=0.10, scale=1, degree=3
## - Fold01.Rep4: C=0.10, scale=1, degree=3
## + Fold01.Rep4: C=0.50, scale=1, degree=3
## - Fold01.Rep4: C=0.50, scale=1, degree=3
## + Fold02.Rep4: C=0.01, scale=1, degree=1
## - Fold02.Rep4: C=0.01, scale=1, degree=1
## + Fold02.Rep4: C=0.10, scale=1, degree=1
## - Fold02.Rep4: C=0.10, scale=1, degree=1
## + Fold02.Rep4: C=0.50, scale=1, degree=1
## - Fold02.Rep4: C=0.50, scale=1, degree=1
## + Fold02.Rep4: C=0.01, scale=1, degree=2
## - Fold02.Rep4: C=0.01, scale=1, degree=2
## + Fold02.Rep4: C=0.10, scale=1, degree=2
## - Fold02.Rep4: C=0.10, scale=1, degree=2
## + Fold02.Rep4: C=0.50, scale=1, degree=2
## - Fold02.Rep4: C=0.50, scale=1, degree=2
## + Fold02.Rep4: C=0.01, scale=1, degree=3
## - Fold02.Rep4: C=0.01, scale=1, degree=3
## + Fold02.Rep4: C=0.10, scale=1, degree=3
## - Fold02.Rep4: C=0.10, scale=1, degree=3
## + Fold02.Rep4: C=0.50, scale=1, degree=3
## - Fold02.Rep4: C=0.50, scale=1, degree=3
## + Fold03.Rep4: C=0.01, scale=1, degree=1
## - Fold03.Rep4: C=0.01, scale=1, degree=1
## + Fold03.Rep4: C=0.10, scale=1, degree=1
## - Fold03.Rep4: C=0.10, scale=1, degree=1
## + Fold03.Rep4: C=0.50, scale=1, degree=1
## - Fold03.Rep4: C=0.50, scale=1, degree=1
## + Fold03.Rep4: C=0.01, scale=1, degree=2
## - Fold03.Rep4: C=0.01, scale=1, degree=2
## + Fold03.Rep4: C=0.10, scale=1, degree=2
## - Fold03.Rep4: C=0.10, scale=1, degree=2
## + Fold03.Rep4: C=0.50, scale=1, degree=2
## - Fold03.Rep4: C=0.50, scale=1, degree=2
## + Fold03.Rep4: C=0.01, scale=1, degree=3
## - Fold03.Rep4: C=0.01, scale=1, degree=3
## + Fold03.Rep4: C=0.10, scale=1, degree=3
## - Fold03.Rep4: C=0.10, scale=1, degree=3
## + Fold03.Rep4: C=0.50, scale=1, degree=3
## - Fold03.Rep4: C=0.50, scale=1, degree=3
## + Fold04.Rep4: C=0.01, scale=1, degree=1
## - Fold04.Rep4: C=0.01, scale=1, degree=1
## + Fold04.Rep4: C=0.10, scale=1, degree=1
## - Fold04.Rep4: C=0.10, scale=1, degree=1
## + Fold04.Rep4: C=0.50, scale=1, degree=1
## - Fold04.Rep4: C=0.50, scale=1, degree=1
## + Fold04.Rep4: C=0.01, scale=1, degree=2
## - Fold04.Rep4: C=0.01, scale=1, degree=2
## + Fold04.Rep4: C=0.10, scale=1, degree=2
## - Fold04.Rep4: C=0.10, scale=1, degree=2
## + Fold04.Rep4: C=0.50, scale=1, degree=2
## - Fold04.Rep4: C=0.50, scale=1, degree=2
## + Fold04.Rep4: C=0.01, scale=1, degree=3
## - Fold04.Rep4: C=0.01, scale=1, degree=3
## + Fold04.Rep4: C=0.10, scale=1, degree=3
## - Fold04.Rep4: C=0.10, scale=1, degree=3
## + Fold04.Rep4: C=0.50, scale=1, degree=3
## - Fold04.Rep4: C=0.50, scale=1, degree=3
## + Fold05.Rep4: C=0.01, scale=1, degree=1
## - Fold05.Rep4: C=0.01, scale=1, degree=1
## + Fold05.Rep4: C=0.10, scale=1, degree=1
## - Fold05.Rep4: C=0.10, scale=1, degree=1
## + Fold05.Rep4: C=0.50, scale=1, degree=1
## - Fold05.Rep4: C=0.50, scale=1, degree=1
## + Fold05.Rep4: C=0.01, scale=1, degree=2
## - Fold05.Rep4: C=0.01, scale=1, degree=2
## + Fold05.Rep4: C=0.10, scale=1, degree=2
## - Fold05.Rep4: C=0.10, scale=1, degree=2
## + Fold05.Rep4: C=0.50, scale=1, degree=2
## - Fold05.Rep4: C=0.50, scale=1, degree=2
## + Fold05.Rep4: C=0.01, scale=1, degree=3
## - Fold05.Rep4: C=0.01, scale=1, degree=3
## + Fold05.Rep4: C=0.10, scale=1, degree=3
## - Fold05.Rep4: C=0.10, scale=1, degree=3
## + Fold05.Rep4: C=0.50, scale=1, degree=3
## - Fold05.Rep4: C=0.50, scale=1, degree=3
## + Fold06.Rep4: C=0.01, scale=1, degree=1
## - Fold06.Rep4: C=0.01, scale=1, degree=1
## + Fold06.Rep4: C=0.10, scale=1, degree=1
## - Fold06.Rep4: C=0.10, scale=1, degree=1
## + Fold06.Rep4: C=0.50, scale=1, degree=1
## - Fold06.Rep4: C=0.50, scale=1, degree=1
## + Fold06.Rep4: C=0.01, scale=1, degree=2
## - Fold06.Rep4: C=0.01, scale=1, degree=2
## + Fold06.Rep4: C=0.10, scale=1, degree=2
## - Fold06.Rep4: C=0.10, scale=1, degree=2
## + Fold06.Rep4: C=0.50, scale=1, degree=2
## - Fold06.Rep4: C=0.50, scale=1, degree=2
## + Fold06.Rep4: C=0.01, scale=1, degree=3
## - Fold06.Rep4: C=0.01, scale=1, degree=3
## + Fold06.Rep4: C=0.10, scale=1, degree=3
## - Fold06.Rep4: C=0.10, scale=1, degree=3
## + Fold06.Rep4: C=0.50, scale=1, degree=3
## - Fold06.Rep4: C=0.50, scale=1, degree=3
## + Fold07.Rep4: C=0.01, scale=1, degree=1
## - Fold07.Rep4: C=0.01, scale=1, degree=1
## + Fold07.Rep4: C=0.10, scale=1, degree=1
## - Fold07.Rep4: C=0.10, scale=1, degree=1
## + Fold07.Rep4: C=0.50, scale=1, degree=1
## - Fold07.Rep4: C=0.50, scale=1, degree=1
## + Fold07.Rep4: C=0.01, scale=1, degree=2
## - Fold07.Rep4: C=0.01, scale=1, degree=2
## + Fold07.Rep4: C=0.10, scale=1, degree=2
## - Fold07.Rep4: C=0.10, scale=1, degree=2
## + Fold07.Rep4: C=0.50, scale=1, degree=2
## - Fold07.Rep4: C=0.50, scale=1, degree=2
## + Fold07.Rep4: C=0.01, scale=1, degree=3
## - Fold07.Rep4: C=0.01, scale=1, degree=3
## + Fold07.Rep4: C=0.10, scale=1, degree=3
## - Fold07.Rep4: C=0.10, scale=1, degree=3
## + Fold07.Rep4: C=0.50, scale=1, degree=3
## - Fold07.Rep4: C=0.50, scale=1, degree=3
## + Fold08.Rep4: C=0.01, scale=1, degree=1
## - Fold08.Rep4: C=0.01, scale=1, degree=1
## + Fold08.Rep4: C=0.10, scale=1, degree=1
## - Fold08.Rep4: C=0.10, scale=1, degree=1
## + Fold08.Rep4: C=0.50, scale=1, degree=1
## - Fold08.Rep4: C=0.50, scale=1, degree=1
## + Fold08.Rep4: C=0.01, scale=1, degree=2
## - Fold08.Rep4: C=0.01, scale=1, degree=2
## + Fold08.Rep4: C=0.10, scale=1, degree=2
## - Fold08.Rep4: C=0.10, scale=1, degree=2
## + Fold08.Rep4: C=0.50, scale=1, degree=2
## - Fold08.Rep4: C=0.50, scale=1, degree=2
## + Fold08.Rep4: C=0.01, scale=1, degree=3
## - Fold08.Rep4: C=0.01, scale=1, degree=3
## + Fold08.Rep4: C=0.10, scale=1, degree=3
## - Fold08.Rep4: C=0.10, scale=1, degree=3
## + Fold08.Rep4: C=0.50, scale=1, degree=3
## - Fold08.Rep4: C=0.50, scale=1, degree=3
## + Fold09.Rep4: C=0.01, scale=1, degree=1
## - Fold09.Rep4: C=0.01, scale=1, degree=1
## + Fold09.Rep4: C=0.10, scale=1, degree=1
## - Fold09.Rep4: C=0.10, scale=1, degree=1
## + Fold09.Rep4: C=0.50, scale=1, degree=1
## - Fold09.Rep4: C=0.50, scale=1, degree=1
## + Fold09.Rep4: C=0.01, scale=1, degree=2
## - Fold09.Rep4: C=0.01, scale=1, degree=2
## + Fold09.Rep4: C=0.10, scale=1, degree=2
## - Fold09.Rep4: C=0.10, scale=1, degree=2
## + Fold09.Rep4: C=0.50, scale=1, degree=2
## - Fold09.Rep4: C=0.50, scale=1, degree=2
## + Fold09.Rep4: C=0.01, scale=1, degree=3
## - Fold09.Rep4: C=0.01, scale=1, degree=3
## + Fold09.Rep4: C=0.10, scale=1, degree=3
## - Fold09.Rep4: C=0.10, scale=1, degree=3
## + Fold09.Rep4: C=0.50, scale=1, degree=3
## - Fold09.Rep4: C=0.50, scale=1, degree=3
## + Fold10.Rep4: C=0.01, scale=1, degree=1
## - Fold10.Rep4: C=0.01, scale=1, degree=1
## + Fold10.Rep4: C=0.10, scale=1, degree=1
## - Fold10.Rep4: C=0.10, scale=1, degree=1
## + Fold10.Rep4: C=0.50, scale=1, degree=1
## - Fold10.Rep4: C=0.50, scale=1, degree=1
## + Fold10.Rep4: C=0.01, scale=1, degree=2
## - Fold10.Rep4: C=0.01, scale=1, degree=2
## + Fold10.Rep4: C=0.10, scale=1, degree=2
## - Fold10.Rep4: C=0.10, scale=1, degree=2
## + Fold10.Rep4: C=0.50, scale=1, degree=2
## - Fold10.Rep4: C=0.50, scale=1, degree=2
## + Fold10.Rep4: C=0.01, scale=1, degree=3
## - Fold10.Rep4: C=0.01, scale=1, degree=3
## + Fold10.Rep4: C=0.10, scale=1, degree=3
## - Fold10.Rep4: C=0.10, scale=1, degree=3
## + Fold10.Rep4: C=0.50, scale=1, degree=3
## - Fold10.Rep4: C=0.50, scale=1, degree=3
## + Fold01.Rep5: C=0.01, scale=1, degree=1
## - Fold01.Rep5: C=0.01, scale=1, degree=1
## + Fold01.Rep5: C=0.10, scale=1, degree=1
## - Fold01.Rep5: C=0.10, scale=1, degree=1
## + Fold01.Rep5: C=0.50, scale=1, degree=1
## - Fold01.Rep5: C=0.50, scale=1, degree=1
## + Fold01.Rep5: C=0.01, scale=1, degree=2
## - Fold01.Rep5: C=0.01, scale=1, degree=2
## + Fold01.Rep5: C=0.10, scale=1, degree=2
## - Fold01.Rep5: C=0.10, scale=1, degree=2
## + Fold01.Rep5: C=0.50, scale=1, degree=2
## - Fold01.Rep5: C=0.50, scale=1, degree=2
## + Fold01.Rep5: C=0.01, scale=1, degree=3
## - Fold01.Rep5: C=0.01, scale=1, degree=3
## + Fold01.Rep5: C=0.10, scale=1, degree=3
## - Fold01.Rep5: C=0.10, scale=1, degree=3
## + Fold01.Rep5: C=0.50, scale=1, degree=3
## - Fold01.Rep5: C=0.50, scale=1, degree=3
## + Fold02.Rep5: C=0.01, scale=1, degree=1
## - Fold02.Rep5: C=0.01, scale=1, degree=1
## + Fold02.Rep5: C=0.10, scale=1, degree=1
## - Fold02.Rep5: C=0.10, scale=1, degree=1
## + Fold02.Rep5: C=0.50, scale=1, degree=1
## - Fold02.Rep5: C=0.50, scale=1, degree=1
## + Fold02.Rep5: C=0.01, scale=1, degree=2
## - Fold02.Rep5: C=0.01, scale=1, degree=2
## + Fold02.Rep5: C=0.10, scale=1, degree=2
## - Fold02.Rep5: C=0.10, scale=1, degree=2
## + Fold02.Rep5: C=0.50, scale=1, degree=2
## - Fold02.Rep5: C=0.50, scale=1, degree=2
## + Fold02.Rep5: C=0.01, scale=1, degree=3
## - Fold02.Rep5: C=0.01, scale=1, degree=3
## + Fold02.Rep5: C=0.10, scale=1, degree=3
## - Fold02.Rep5: C=0.10, scale=1, degree=3
## + Fold02.Rep5: C=0.50, scale=1, degree=3
## - Fold02.Rep5: C=0.50, scale=1, degree=3
## + Fold03.Rep5: C=0.01, scale=1, degree=1
## - Fold03.Rep5: C=0.01, scale=1, degree=1
## + Fold03.Rep5: C=0.10, scale=1, degree=1
## - Fold03.Rep5: C=0.10, scale=1, degree=1
## + Fold03.Rep5: C=0.50, scale=1, degree=1
## - Fold03.Rep5: C=0.50, scale=1, degree=1
## + Fold03.Rep5: C=0.01, scale=1, degree=2
## - Fold03.Rep5: C=0.01, scale=1, degree=2
## + Fold03.Rep5: C=0.10, scale=1, degree=2
## - Fold03.Rep5: C=0.10, scale=1, degree=2
## + Fold03.Rep5: C=0.50, scale=1, degree=2
## - Fold03.Rep5: C=0.50, scale=1, degree=2
## + Fold03.Rep5: C=0.01, scale=1, degree=3
## - Fold03.Rep5: C=0.01, scale=1, degree=3
## + Fold03.Rep5: C=0.10, scale=1, degree=3
## - Fold03.Rep5: C=0.10, scale=1, degree=3
## + Fold03.Rep5: C=0.50, scale=1, degree=3
## - Fold03.Rep5: C=0.50, scale=1, degree=3
## + Fold04.Rep5: C=0.01, scale=1, degree=1
## - Fold04.Rep5: C=0.01, scale=1, degree=1
## + Fold04.Rep5: C=0.10, scale=1, degree=1
## - Fold04.Rep5: C=0.10, scale=1, degree=1
## + Fold04.Rep5: C=0.50, scale=1, degree=1
## - Fold04.Rep5: C=0.50, scale=1, degree=1
## + Fold04.Rep5: C=0.01, scale=1, degree=2
## - Fold04.Rep5: C=0.01, scale=1, degree=2
## + Fold04.Rep5: C=0.10, scale=1, degree=2
## - Fold04.Rep5: C=0.10, scale=1, degree=2
## + Fold04.Rep5: C=0.50, scale=1, degree=2
## - Fold04.Rep5: C=0.50, scale=1, degree=2
## + Fold04.Rep5: C=0.01, scale=1, degree=3
## - Fold04.Rep5: C=0.01, scale=1, degree=3
## + Fold04.Rep5: C=0.10, scale=1, degree=3
## - Fold04.Rep5: C=0.10, scale=1, degree=3
## + Fold04.Rep5: C=0.50, scale=1, degree=3
## - Fold04.Rep5: C=0.50, scale=1, degree=3
## + Fold05.Rep5: C=0.01, scale=1, degree=1
## - Fold05.Rep5: C=0.01, scale=1, degree=1
## + Fold05.Rep5: C=0.10, scale=1, degree=1
## - Fold05.Rep5: C=0.10, scale=1, degree=1
## + Fold05.Rep5: C=0.50, scale=1, degree=1
## - Fold05.Rep5: C=0.50, scale=1, degree=1
## + Fold05.Rep5: C=0.01, scale=1, degree=2
## - Fold05.Rep5: C=0.01, scale=1, degree=2
## + Fold05.Rep5: C=0.10, scale=1, degree=2
## - Fold05.Rep5: C=0.10, scale=1, degree=2
## + Fold05.Rep5: C=0.50, scale=1, degree=2
## - Fold05.Rep5: C=0.50, scale=1, degree=2
## + Fold05.Rep5: C=0.01, scale=1, degree=3
## - Fold05.Rep5: C=0.01, scale=1, degree=3
## + Fold05.Rep5: C=0.10, scale=1, degree=3
## - Fold05.Rep5: C=0.10, scale=1, degree=3
## + Fold05.Rep5: C=0.50, scale=1, degree=3
## - Fold05.Rep5: C=0.50, scale=1, degree=3
## + Fold06.Rep5: C=0.01, scale=1, degree=1
## - Fold06.Rep5: C=0.01, scale=1, degree=1
## + Fold06.Rep5: C=0.10, scale=1, degree=1
## - Fold06.Rep5: C=0.10, scale=1, degree=1
## + Fold06.Rep5: C=0.50, scale=1, degree=1
## - Fold06.Rep5: C=0.50, scale=1, degree=1
## + Fold06.Rep5: C=0.01, scale=1, degree=2
## - Fold06.Rep5: C=0.01, scale=1, degree=2
## + Fold06.Rep5: C=0.10, scale=1, degree=2
## - Fold06.Rep5: C=0.10, scale=1, degree=2
## + Fold06.Rep5: C=0.50, scale=1, degree=2
## - Fold06.Rep5: C=0.50, scale=1, degree=2
## + Fold06.Rep5: C=0.01, scale=1, degree=3
## - Fold06.Rep5: C=0.01, scale=1, degree=3
## + Fold06.Rep5: C=0.10, scale=1, degree=3
## - Fold06.Rep5: C=0.10, scale=1, degree=3
## + Fold06.Rep5: C=0.50, scale=1, degree=3
## - Fold06.Rep5: C=0.50, scale=1, degree=3
## + Fold07.Rep5: C=0.01, scale=1, degree=1
## - Fold07.Rep5: C=0.01, scale=1, degree=1
## + Fold07.Rep5: C=0.10, scale=1, degree=1
## - Fold07.Rep5: C=0.10, scale=1, degree=1
## + Fold07.Rep5: C=0.50, scale=1, degree=1
## - Fold07.Rep5: C=0.50, scale=1, degree=1
## + Fold07.Rep5: C=0.01, scale=1, degree=2
## - Fold07.Rep5: C=0.01, scale=1, degree=2
## + Fold07.Rep5: C=0.10, scale=1, degree=2
## - Fold07.Rep5: C=0.10, scale=1, degree=2
## + Fold07.Rep5: C=0.50, scale=1, degree=2
## - Fold07.Rep5: C=0.50, scale=1, degree=2
## + Fold07.Rep5: C=0.01, scale=1, degree=3
## - Fold07.Rep5: C=0.01, scale=1, degree=3
## + Fold07.Rep5: C=0.10, scale=1, degree=3
## - Fold07.Rep5: C=0.10, scale=1, degree=3
## + Fold07.Rep5: C=0.50, scale=1, degree=3
## - Fold07.Rep5: C=0.50, scale=1, degree=3
## + Fold08.Rep5: C=0.01, scale=1, degree=1
## - Fold08.Rep5: C=0.01, scale=1, degree=1
## + Fold08.Rep5: C=0.10, scale=1, degree=1
## - Fold08.Rep5: C=0.10, scale=1, degree=1
## + Fold08.Rep5: C=0.50, scale=1, degree=1
## - Fold08.Rep5: C=0.50, scale=1, degree=1
## + Fold08.Rep5: C=0.01, scale=1, degree=2
## - Fold08.Rep5: C=0.01, scale=1, degree=2
## + Fold08.Rep5: C=0.10, scale=1, degree=2
## - Fold08.Rep5: C=0.10, scale=1, degree=2
## + Fold08.Rep5: C=0.50, scale=1, degree=2
## - Fold08.Rep5: C=0.50, scale=1, degree=2
## + Fold08.Rep5: C=0.01, scale=1, degree=3
## - Fold08.Rep5: C=0.01, scale=1, degree=3
## + Fold08.Rep5: C=0.10, scale=1, degree=3
## - Fold08.Rep5: C=0.10, scale=1, degree=3
## + Fold08.Rep5: C=0.50, scale=1, degree=3
## - Fold08.Rep5: C=0.50, scale=1, degree=3
## + Fold09.Rep5: C=0.01, scale=1, degree=1
## - Fold09.Rep5: C=0.01, scale=1, degree=1
## + Fold09.Rep5: C=0.10, scale=1, degree=1
## - Fold09.Rep5: C=0.10, scale=1, degree=1
## + Fold09.Rep5: C=0.50, scale=1, degree=1
## - Fold09.Rep5: C=0.50, scale=1, degree=1
## + Fold09.Rep5: C=0.01, scale=1, degree=2
## - Fold09.Rep5: C=0.01, scale=1, degree=2
## + Fold09.Rep5: C=0.10, scale=1, degree=2
## - Fold09.Rep5: C=0.10, scale=1, degree=2
## + Fold09.Rep5: C=0.50, scale=1, degree=2
## - Fold09.Rep5: C=0.50, scale=1, degree=2
## + Fold09.Rep5: C=0.01, scale=1, degree=3
## - Fold09.Rep5: C=0.01, scale=1, degree=3
## + Fold09.Rep5: C=0.10, scale=1, degree=3
## - Fold09.Rep5: C=0.10, scale=1, degree=3
## + Fold09.Rep5: C=0.50, scale=1, degree=3
## - Fold09.Rep5: C=0.50, scale=1, degree=3
## + Fold10.Rep5: C=0.01, scale=1, degree=1
## - Fold10.Rep5: C=0.01, scale=1, degree=1
## + Fold10.Rep5: C=0.10, scale=1, degree=1
## - Fold10.Rep5: C=0.10, scale=1, degree=1
## + Fold10.Rep5: C=0.50, scale=1, degree=1
## - Fold10.Rep5: C=0.50, scale=1, degree=1
## + Fold10.Rep5: C=0.01, scale=1, degree=2
## - Fold10.Rep5: C=0.01, scale=1, degree=2
## + Fold10.Rep5: C=0.10, scale=1, degree=2
## - Fold10.Rep5: C=0.10, scale=1, degree=2
## + Fold10.Rep5: C=0.50, scale=1, degree=2
## - Fold10.Rep5: C=0.50, scale=1, degree=2
## + Fold10.Rep5: C=0.01, scale=1, degree=3
## - Fold10.Rep5: C=0.01, scale=1, degree=3
## + Fold10.Rep5: C=0.10, scale=1, degree=3
## - Fold10.Rep5: C=0.10, scale=1, degree=3
## + Fold10.Rep5: C=0.50, scale=1, degree=3
## - Fold10.Rep5: C=0.50, scale=1, degree=3
## Aggregating results
## Selecting tuning parameters
## Fitting degree = 1, scale = 1, C = 0.01 on full training setSummarizing the Model
print(ModelClBestSvm)## Support Vector Machines with Polynomial Kernel
##
## 384 samples
## 14 predictor
## 2 classes: '0', '1'
##
## No pre-processing
## Resampling: Cross-Validated (10 fold, repeated 5 times)
## Summary of sample sizes: 345, 346, 346, 345, 345, 347, ...
## Resampling results across tuning parameters:
##
## C degree Accuracy Kappa
## 0.01 1 0.8204344 0.5263200
## 0.01 2 0.7975322 0.4811618
## 0.01 3 0.7630901 0.4286784
## 0.10 1 0.8204344 0.5263200
## 0.10 2 0.7740212 0.4561467
## 0.10 3 0.7194179 0.3421574
## 0.50 1 0.8204344 0.5263200
## 0.50 2 0.7557315 0.4257365
## 0.50 3 0.7016311 0.3143988
##
## Tuning parameter 'scale' was held constant at a value of 1
## Accuracy was used to select the optimal model using the largest value.
## The final values used for the model were degree = 1, scale = 1 and C = 0.01.plot(ModelClBestSvm)
confusionMatrix(as.factor(predict(ModelClBestSvm, newdata = TestDataSet)),
as.factor(TestDataSet$LoanStatusOk))## Confusion Matrix and Statistics
##
## Reference
## Prediction 0 1
## 0 14 5
## 1 24 84
##
## Accuracy : 0.7717
## 95% CI : (0.6888, 0.8414)
## No Information Rate : 0.7008
## P-Value [Acc > NIR] : 0.0471389
##
## Kappa : 0.3645
##
## Mcnemar's Test P-Value : 0.0008302
##
## Sensitivity : 0.3684
## Specificity : 0.9438
## Pos Pred Value : 0.7368
## Neg Pred Value : 0.7778
## Prevalence : 0.2992
## Detection Rate : 0.1102
## Detection Prevalence : 0.1496
## Balanced Accuracy : 0.6561
##
## 'Positive' Class : 0
## count(TrainDataSet,LoanStatusOk)## LoanStatusOk n
## 1 0 126
## 2 1 258Balancing the Dataset with UpSampling
DataUpSampledCl=upSample(x=DataFinalCl[,2:15],
y=as.factor(DataFinalCl[,1]), list = FALSE,
yname = "LoanStatusOk")Spliting training set into two parts based on outcome: 75% and 25%
TrainIndex <- createDataPartition(DataUpSampledCl$LoanStatusOk, p=0.75, list=FALSE)
TrainDataSet = DataUpSampledCl[ TrainIndex,]
TestDataSet <- DataUpSampledCl[-TrainIndex,]
count(TrainDataSet,LoanStatusOk)## LoanStatusOk n
## 1 0 261
## 2 1 261ModelClSvm=train(as.factor(LoanStatusOk)~.,
data=TrainDataSet, method = "svmPoly")
confusionMatrix(as.factor(predict(ModelClSvm, newdata = TestDataSet)),
as.factor(TestDataSet$LoanStatusOk))## Confusion Matrix and Statistics
##
## Reference
## Prediction 0 1
## 0 59 12
## 1 27 74
##
## Accuracy : 0.7733
## 95% CI : (0.7033, 0.8335)
## No Information Rate : 0.5
## P-Value [Acc > NIR] : 1.649e-13
##
## Kappa : 0.5465
##
## Mcnemar's Test P-Value : 0.02497
##
## Sensitivity : 0.6860
## Specificity : 0.8605
## Pos Pred Value : 0.8310
## Neg Pred Value : 0.7327
## Prevalence : 0.5000
## Detection Rate : 0.3430
## Detection Prevalence : 0.4128
## Balanced Accuracy : 0.7733
##
## 'Positive' Class : 0
## Decision Treeses
How Can Data Records be Routed Through a Tree-like Structure to Predict Outcomes? Decision Trees can be used for classification and regression problems. The goal of a decision tree is to guide data records based on splitting criteria through a tree-like structure towards different bins. In the case of classification each bin provides a predictions based on the majority of records from the training set ending up in that bin. In the case of regression (not covered in this course) each bin provides a predictions based on the mean of records from the training set ending up in that bin.
library(tidyverse)
library(readxl)
library(caret)
library(rattle)
HeartData <- read_excel("Data/heartData.xlsx")%>%
mutate(cp=ifelse(cp==3,0,1))%>%
select(-ca,-thal,-cp,-exang,-slope, HeartDisease=target) %>%
select(HeartDisease,everything())
sum(is.na(HeartData)) #check for missing numbers
if (version$major>5){
RNGkind(sample.kind = "Rounding")
}
set.seed(12345)
TrainRows=createDataPartition(HeartData$HeartDisease,p=0.85,list=FALSE)
HeartDataTrain=HeartData[TrainRows,]
HeartDataTest=HeartData[-TrainRows,]
(NoInformRate=mean(HeartDataTrain$HeartDisease))
FirstPlot=ggplot(HeartDataTrain,aes(x=age, y=sex, color=as.factor(HeartDisease))) +
geom_jitter(position=position_jitter(0.1,0.3))
FirstPlot+
geom_hline(yintercept = 0.5, size=1, color="red")
FirstPlot+
geom_vline(xintercept = 55, size=1, color="blue")
FirstPlot+
geom_vline(xintercept = 55, size=1, color="blue")+
geom_hline(yintercept = 0.5, size=1, color="red")
Male=filter( HeartDataTrain, sex==1)
Female=filter( HeartDataTrain, sex==0)
MaleN=nrow( Male)
MaleNSick=sum(Male$HeartDisease)
MaleNOk= MaleN- MaleNSick
MalePercSick= MaleNSick/ MaleN
MalePercOk= MaleNOk/ MaleN
MalePredict=ifelse( MalePercSick> MalePercOk,"Sick","Ok")
FemaleN=nrow( Female)
FemaleNSick=sum( Female$HeartDisease)
FemaleNOk= FemaleN- FemaleNSick
FemalePercSick= FemaleNSick/ FemaleN
FemalePercOk= FemaleNOk/ FemaleN
FemalePredict=ifelse( FemalePercSick> FemalePercOk,"Sick","Ok")
ResultsMaleFem=data.frame(Female=c(FemaleN,FemaleNSick,FemaleNOk,FemalePercSick,FemalePercOk,FemalePredict),Male=c(MaleN,MaleNSick,MaleNOk,MalePercSick,MalePercOk,MalePredict))
rownames(ResultsMaleFem)=c("N","NSick","NOk","PercSick","PercOk","Predict")
View(ResultsMaleFem)First Level: Node by Age
OldPatients=filter(HeartDataTrain, age>=55)
YoungPatients=filter(HeartDataTrain, age<55)
OldPatientsN=nrow(OldPatients)
OldPatientsNSick=sum(OldPatients$HeartDisease)
OldPatientsNOk=OldPatientsN-OldPatientsNSick
OldPatientsPercSick=OldPatientsNSick/OldPatientsN
OldPatientsPercOk=OldPatientsNOk/OldPatientsN
OldPatientsPredict=ifelse(OldPatientsPercSick>OldPatientsPercOk,"Sick","Ok")
YoungPatientsN=nrow(YoungPatients)
YoungPatientsNSick=sum(YoungPatients$HeartDisease)
YoungPatientsNOk=YoungPatientsN-YoungPatientsNSick
YoungPatientsPercSick=YoungPatientsNSick/YoungPatientsN
YoungPatientsPercOk=YoungPatientsNOk/YoungPatientsN
YoungPatientsPredict=ifelse(YoungPatientsPercSick>YoungPatientsPercOk,"Sick","Ok")
ResultsFirstNode=data.frame(OldPatients=c(OldPatientsN,OldPatientsNSick,OldPatientsNOk,OldPatientsPercSick,OldPatientsPercOk,OldPatientsPredict),YoungPatients=c(YoungPatientsN,YoungPatientsNSick,YoungPatientsNOk,YoungPatientsPercSick,YoungPatientsPercOk,YoungPatientsPredict))
rownames(ResultsFirstNode)=c("N","NSick","NOk","PercSick","PercOk","Predict")
View(ResultsFirstNode)Second Level Old and Young Patients by Sex. Male=1
Older patient
OldMale=filter(OldPatients, sex==1)
OldFemale=filter(OldPatients, sex==0)
OldMaleN=nrow(OldMale)
OldMaleNSick=sum(OldMale$HeartDisease)
OldMaleNOk=OldMaleN-OldMaleNSick
OldMalePercSick=OldMaleNSick/OldMaleN
OldMalePercOk=OldMaleNOk/OldMaleN
OldMalePredict=ifelse(OldMalePercSick>OldMalePercOk,"Sick","Ok")
OldFemaleN=nrow(OldFemale)
OldFemaleNSick=sum(OldFemale$HeartDisease)
OldFemaleNOk=OldFemaleN-OldFemaleNSick
OldFemalePercSick=OldFemaleNSick/OldFemaleN
OldFemalePercOk=OldFemaleNOk/OldFemaleN
OldFemalePredict=ifelse(OldFemalePercSick>OldFemalePercOk,"Sick","Ok")Younger patients
Results
Results=data.frame(
OldMale=c(OldMaleN,OldMaleNSick,OldMaleNOk,OldMalePercSick,OldMalePercOk,OldMalePredict),
OldFemale=c(OldFemaleN,OldFemaleNSick,OldFemaleNOk,OldFemalePercSick,OldFemalePercOk,OldFemalePredict),
YoungMale=c(YoungMaleN,YoungMaleNSick,YoungMaleNOk,YoungMalePercSick,YoungMalePercOk,YoungMalePredict),
YoungFemale=c(YoungFemaleN,YoungFemaleNSick,YoungFemaleNOk,YoungFemalePercSick,YoungFemalePercOk,YoungFemalePredict)
)
rownames(Results)=c("N","NSick","NOk","PercSick","PercOk","Predict")
View(Results)Random Forest
How can we Combine Several Decision Trees to a Random Forest to Increase Prediction Quality Random Forest generates multiple decision trees (hundrets) which are different in regards of: The splitting variables and values that are used The data used for training (often only a subset of the training data are used for optimizing a split based on boot strapping) When all trees are generated they can be used for prediction. In the case of classification the majority prediction of the trees determines the Random Forest prediction. In the case of regression the mean of the trees determines the prediction.
Theory of Neural Networks (Deep
Learning)
How can we predict using a Feed Forward Multi Layer Network( MLP)? A MLP is a network made up of neurons. Data can be fed into the MLP through the input neuron(s). Then the value(s) of each record are amplified or weakend (by multiplication) and led to the hidden neyrons. Inside the hidden neuron data are processed by an activation function The results from each hidden neuron are multiplied by a parameter. Then all results are added up and a parameter is added for the output.The output is the prediction of the MLP. Parameters are optimized to minimize the prediction error.
Application of Neural Networks (Deep
Learning)
How can we predict using a Feed Forward Multi Layer Network( MLP)? A MLP is a network made up of neurons. Data can be fed into the MLP through the input neuron(s). Then the value(s) of each record are amplified or weakend (by multiplication) and led to the hidden neyrons. Inside the hidden neuron data are processed by an activation function The results from each hidden neuron are multiplied by a parameter. Then all results are added up and a parameter is added for the output.The output is the prediction of the MLP. Parameters are optimized to minimize the prediction error.