1. CODE BLOCK FOR DATA PREPARATION

################################################################################
######## NAME: NIVEDITHA KM, AREZOU,STACEY     #################################
########     : SEJAL, AISHWARYA, SANA, SABRINA #################################
######## ASSIGNMENT 5 - DATA MINING#############################################
################################################################################

library(tidyverse)  # data manipulation
library(ggpubr)
#install.packages("tidymodels")
#install.packages("glmnet")
library(readr)
library(tidymodels)
library(glmnet)
library(caret)
library(class)
#install.packages("caret")
#install.packages("class")
################################################################################
#DATA SETS IMPORT
#DATA PRE-PROCESSING STARTS
################################################################################

#GET THE DATASETS IMPORTED

class_df <- read.csv("class.csv")
zoo_df <- read.csv("zoo.csv")
#gender_df <- read.csv("gender_submission.csv")


c_df <- class_df 
z_df <- zoo_df

#DATA PREPROCESSING/CLEAN UP BY REPLACING MEAN VALUES IN PLACE OF NA VALUES IN AGE FIELD

#c_df <- c_df %>% mutate(across(Age, ~replace_na(., mean(., na.rm=TRUE))))

#z_df <- z_df %>% mutate(across(Age, ~replace_na(., mean(., na.rm=TRUE))))

#FOLLOWING PIECE OF CODE IS TO REMOVE COMMAS FROM COLUMNS IF ANY(Not necessary for this though)

c_df <- data.frame(lapply(c_df, function(x) {
  gsub(",", "", x)
  }))

z_df <- data.frame(lapply(z_df, function(x) {
  gsub(",", "", x)
  }))

#################################################################################
#FOLLOWING PIECE OF CODE IS TO CHECK IF NUMERIC CONVERSION CAN HAPPEN ON COLUMN 
#THIS IS DONE ON BOTH TEST AND TRAINING DATASET
#USUALLY FULL DATASET IS BROKEN INTO TEST AND TRAINING. IN THIS CASE THESE WERE
#PROVIDED SEPARATELY WHICH WAS HELPFUL.
#################################################################################

is_all_numeric <- function(x) {
  !any(is.na(suppressWarnings(as.numeric(na.omit(x))))) & is.character(x)
}

c_df %>% 
  mutate_if(is_all_numeric,as.numeric) %>%
  str()
## 'data.frame':    7 obs. of  4 variables:
##  $ Class_Number                     : num  1 2 3 4 5 6 7
##  $ Number_Of_Animal_Species_In_Class: num  41 20 5 13 4 8 10
##  $ Class_Type                       : chr  "Mammal" "Bird" "Reptile" "Fish" ...
##  $ Animal_Names                     : chr  "aardvark antelope bear boar buffalo calf cavy cheetah deer dolphin elephant fruitbat giraffe girl goat gorilla "| __truncated__ "chicken crow dove duck flamingo gull hawk kiwi lark ostrich parakeet penguin pheasant rhea skimmer skua sparrow"| __truncated__ "pitviper seasnake slowworm tortoise tuatara" "bass carp catfish chub dogfish haddock herring pike piranha seahorse sole stingray tuna" ...
z_df %>% 
  mutate_if(is_all_numeric,as.numeric) %>%
  str()
## 'data.frame':    101 obs. of  18 variables:
##  $ animal_name: chr  "aardvark" "antelope" "bass" "bear" ...
##  $ hair       : num  1 1 0 1 1 1 1 0 0 1 ...
##  $ feathers   : num  0 0 0 0 0 0 0 0 0 0 ...
##  $ eggs       : num  0 0 1 0 0 0 0 1 1 0 ...
##  $ milk       : num  1 1 0 1 1 1 1 0 0 1 ...
##  $ airborne   : num  0 0 0 0 0 0 0 0 0 0 ...
##  $ aquatic    : num  0 0 1 0 0 0 0 1 1 0 ...
##  $ predator   : num  1 0 1 1 1 0 0 0 1 0 ...
##  $ toothed    : num  1 1 1 1 1 1 1 1 1 1 ...
##  $ backbone   : num  1 1 1 1 1 1 1 1 1 1 ...
##  $ breathes   : num  1 1 0 1 1 1 1 0 0 1 ...
##  $ venomous   : num  0 0 0 0 0 0 0 0 0 0 ...
##  $ fins       : num  0 0 1 0 0 0 0 1 1 0 ...
##  $ legs       : num  4 4 0 4 4 4 4 0 0 4 ...
##  $ tail       : num  0 1 1 0 1 1 1 1 1 0 ...
##  $ domestic   : num  0 0 0 0 0 0 1 1 0 1 ...
##  $ catsize    : num  1 1 0 1 1 1 1 0 0 0 ...
##  $ class_type : num  1 1 4 1 1 1 1 4 4 1 ...
# CONVERTS CHARACTER COLUMNS TO NUMERIC COLUMNS
c_df <- c_df %>% mutate_if(is_all_numeric,as.numeric)

z_df <- z_df %>% mutate_if(is_all_numeric,as.numeric)

###############################################################################
# COLUMNS WERE MODIFIED TO NUMBERS, SO THAT IT CAN BE USED IN LOGISTIC
#                                   REGRESSION
###############################################################################

#Survived Columns were made as factors so that it can used in Logistic Regression

z_df$class_type_f <- as.factor(z_df$class_type)

#new_tst_df <- subset(tst_df, select = c(Pclass, Gender,Age, SibSp,Parch,Cabinornot#,EmbarkedNum,Farebins))
################################################################################
#DATA PRE-PROCESSING ENDS
################################################################################

2. DATASETS CREATED FOR USE IN SVM(SUPPORT VECTOR MACHINE)

view(z_df)
view(c_df)

3. DATA ANALYSIS AND EXPLORATION OF DATASET

library(ggplot2) #data visualization
library(readr) #to read csv files
library(Hmisc) # descriptive statistics
## Warning: package 'Hmisc' was built under R version 4.3.3
## 
## Attaching package: 'Hmisc'
## The following object is masked from 'package:parsnip':
## 
##     translate
## The following objects are masked from 'package:dplyr':
## 
##     src, summarize
## The following objects are masked from 'package:base':
## 
##     format.pval, units
library(gridExtra) #to display plots in grids
## 
## Attaching package: 'gridExtra'
## The following object is masked from 'package:dplyr':
## 
##     combine
library(plyr) # data manipulation
## ------------------------------------------------------------------------------
## You have loaded plyr after dplyr - this is likely to cause problems.
## If you need functions from both plyr and dplyr, please load plyr first, then dplyr:
## library(plyr); library(dplyr)
## ------------------------------------------------------------------------------
## 
## Attaching package: 'plyr'
## The following objects are masked from 'package:Hmisc':
## 
##     is.discrete, summarize
## The following object is masked from 'package:ggpubr':
## 
##     mutate
## The following objects are masked from 'package:dplyr':
## 
##     arrange, count, desc, failwith, id, mutate, rename, summarise,
##     summarize
## The following object is masked from 'package:purrr':
## 
##     compact
#1. CLASS TYPE AND NUMBER OF ANIMALS PLOT
ggplot(c_df,aes(x=Class_Type,y=Number_Of_Animal_Species_In_Class),main="Number of Animals versus Types Of Animals")+geom_bar(stat = "identity",fill="steelblue3",width = 0.5)+geom_text(aes(label=c_df$Number_Of_Animal_Species_In_Class), vjust=1.6,color="white", size=3.5)
## Warning: Use of `c_df$Number_Of_Animal_Species_In_Class` is discouraged.
## ℹ Use `Number_Of_Animal_Species_In_Class` instead.

#2. CLASSIFICATION OF ANIMALS BASED ON CLASS TYPE - KNN

z_train <- z_df[1:70,2:17]
z_test <- z_df[71:101,2:17]

#view(z_train)
#view(z_test)

# Splitting last column whcih contains animal class types
z_train_labels <- z_df[1:70,18]
z_test_labels <- z_df[71:101,18]

# Applying KNN model on the training data keeping k value 9
zooClassification <- knn(train = z_train,test = z_test,cl=z_train_labels,k=9)

# Plotting result of KNN model
plot(zooClassification, col = rainbow(7),main="Classification of Animlas",xlab="Types of Animals")
legend(x=6.5,y=10,legend = c_df$Class_Type,cex=1,fill = rainbow(7),bty="n")

#DATA EXPLORATION

z_data <- z_df[,2:18]

class <- c_df
data <- z_df

class = rename(class, c("Class_Type" = "Types", "Class_Number" = "class_type"))
data = merge (data , class , by = c("class_type"))

library(caret)
library(corrplot)
## corrplot 0.92 loaded
zerovar = nearZeroVar(data, saveMetrics=TRUE)
library(stats)

#3. CORRELATION PLOT

corrplot.mixed(cor(data[,c("hair" , "feathers" , "eggs" , "milk" , "airborne" , "aquatic" , "predator" , "toothed" , "backbone" , "breathes" , "venomous" , "fins" ,  "tail" , "domestic" , "catsize" )]), lower="circle", upper="color", 
               tl.pos="lt", diag="n", order="hclust", hclust.method="complete")

# Create a pie chart of class type
type_counts <- table(data$Types)

#view(type_counts)

agg_count <- aggregate(cbind(hair,feathers,eggs,milk,airborne,aquatic,predator,
                toothed,backbone,breathes,venomous,
                fins,tail,domestic,catsize) ~ Types, data = data, FUN = sum, na.rm = TRUE)


data_long <- pivot_longer(agg_count, cols = -Types, names_to = "Attribute", values_to = "Count")


ggplot(data_long, aes(x = Types, y = Count, fill = Attribute)) +
  geom_bar(stat = "identity", position = "dodge") +
#  geom_text(aes(label = Count), position = position_dodge(width = 1), vjust = -0.5#, size = 3) +  # Add count labels above the bars
  geom_text(aes(label = Count), position = position_dodge(width = 1), vjust = -0.7,angle = 45, size = 2) +
  labs(title = "Counts of Attributes Based on Types",
       x = "Types", y = "Count") +
  theme_minimal() +
  theme(axis.text.x = element_text(angle = 45, hjust = 1))

data_long <- pivot_longer(agg_count, cols = -Types, names_to = "Attribute", values_to = "Count")


# Plotting a pie chart
ggplot(data_long, aes(x = "", y = Count, fill = Attribute)) +
  geom_bar(stat = "identity", width = 1) +
  geom_text(aes(label = Count), 
            position = position_stack(vjust = 0.5), angle = 45, size = 2) +
  coord_polar("y", start = 0) +
  labs(title = "Counts of Attributes Based on Types",
       fill = "Attribute") +
  theme_void() +
  theme(legend.position = "right")+
  theme(aspect.ratio = 1)  # Set aspect ratio to make the pie chart occupy the full space

4. SVM WITH e1071 PACKAGE WITH VARIOUS KERNELS AND ACCURACY

# Load the e1071 library
library(e1071)
## 
## Attaching package: 'e1071'
## The following object is masked from 'package:Hmisc':
## 
##     impute
## The following object is masked from 'package:tune':
## 
##     tune
## The following object is masked from 'package:rsample':
## 
##     permutations
## The following object is masked from 'package:parsnip':
## 
##     tune
# Assuming z_df is your dataset
# Splitting data into training and testing sets
z_trn <- z_df[1:70, 2:18]
z_tst <- z_df[71:101, 2:18]

# Converting class_type to factor
z_trn$class_type_f <- as.factor(z_df$class_type[1:70])
z_tst$class_type_f <- as.factor(z_df$class_type[71:101])

#view(z_trn)

# Train the SVM model with KERNEL as linear
svm_model <- svm(class_type_f ~ ., data = z_trn, kernel = "linear", cost = 10)

# Predict using the trained model
predictions <- predict(svm_model, newdata = z_tst)

# Confusion matrix
conf_matrix <- table(predictions, z_tst$class_type_f)
print(conf_matrix)
##            
## predictions 1 2 3 4 5 6 7
##           1 8 0 0 0 0 0 0
##           2 0 7 2 0 0 0 0
##           3 0 0 1 1 0 0 0
##           4 0 0 0 3 0 0 0
##           5 0 0 1 0 1 0 0
##           6 0 0 0 0 0 2 3
##           7 0 0 0 0 0 0 2
# Calculate accuracy
accuracy <- sum(diag(conf_matrix)) / sum(conf_matrix)
print(paste("Accuracy using linear kernel:", accuracy))
## [1] "Accuracy using linear kernel: 0.774193548387097"
#------------------------------------------------
# Train the SVM model with KERNEL as polynomial
svm_model_pol <- svm(class_type_f ~ ., data = z_trn, kernel = "polynomial", cost = 10)

# Predict using the trained model
predictions_pol <- predict(svm_model_pol, newdata = z_tst)

# Confusion matrix
conf_matrix_pol <- table(predictions_pol, z_tst$class_type_f)
print(conf_matrix_pol)
##                
## predictions_pol 1 2 3 4 5 6 7
##               1 8 0 3 0 0 0 1
##               2 0 7 0 0 0 0 0
##               3 0 0 1 0 0 0 0
##               4 0 0 0 4 0 0 0
##               5 0 0 0 0 1 0 0
##               6 0 0 0 0 0 2 2
##               7 0 0 0 0 0 0 2
# Calculate accuracy
accuracy_pol <- sum(diag(conf_matrix_pol)) / sum(conf_matrix_pol)
print(paste("Accuracy using polynomial Kernel:", accuracy_pol))
## [1] "Accuracy using polynomial Kernel: 0.806451612903226"
#------------------------------------------------
# Train the SVM model with KERNEL as radial
svm_model_rad <- svm(class_type_f ~ ., data = z_trn, kernel = "radial", cost = 10)

# Predict using the trained model
predictions_rad <- predict(svm_model_rad, newdata = z_tst)

# Confusion matrix
conf_matrix_rad <- table(predictions_rad, z_tst$class_type_f)
print(conf_matrix_rad)
##                
## predictions_rad 1 2 3 4 5 6 7
##               1 8 0 0 1 0 0 0
##               2 0 7 1 0 0 0 0
##               3 0 0 1 0 0 0 0
##               4 0 0 0 3 0 0 0
##               5 0 0 2 0 1 0 1
##               6 0 0 0 0 0 2 3
##               7 0 0 0 0 0 0 1
# Calculate accuracy
accuracy_rad <- sum(diag(conf_matrix_rad)) / sum(conf_matrix_rad)
print(paste("Accuracy using radial Kernel:", accuracy_rad))
## [1] "Accuracy using radial Kernel: 0.741935483870968"
#------------------------------------------------
# Train the SVM model with KERNEL as sigmoid
svm_model_sig <- svm(class_type_f ~ ., data = z_trn, kernel = "sigmoid", cost = 10)

# Predict using the trained model
predictions_sig <- predict(svm_model_sig, newdata = z_tst)

# Confusion matrix
conf_matrix_sig <- table(predictions_sig, z_tst$class_type_f)
print(conf_matrix_sig)
##                
## predictions_sig 1 2 3 4 5 6 7
##               1 8 0 0 0 0 0 0
##               2 0 7 1 0 0 0 0
##               3 0 0 2 1 0 0 0
##               4 0 0 0 3 0 0 0
##               5 0 0 1 0 1 0 0
##               6 0 0 0 0 0 2 3
##               7 0 0 0 0 0 0 2
# Calculate accuracy
accuracy_sig <- sum(diag(conf_matrix_sig)) / sum(conf_matrix_sig)
print(paste("Accuracy using sigmoid Kernel:", accuracy_sig))
## [1] "Accuracy using sigmoid Kernel: 0.806451612903226"

5. PCA DATA Visualization

#install.packages("factoextra")
library(factoextra) 
## Warning: package 'factoextra' was built under R version 4.3.3
## Welcome! Want to learn more? See two factoextra-related books at https://goo.gl/ve3WBa
library(caret)
# Convert zoo object to a matrix

# Assuming z_df is your dataset
# Splitting data into training and testing sets
z_trn <- z_df[1:70, 2:18]
z_tst <- z_df[71:101, 2:18]

#view(z_trn)
#view(z_tst)

pca_trn <- prcomp(z_trn[,-17],scale. = TRUE)
pca_tst <- prcomp(z_tst[,-17],scale. = TRUE)

#pc_data_trn <- as.data.frame(pca_trn$x)
#pc_data_tst <- as.data.frame(pca_tst$x)

#print(pca_trn)

fviz_eig(pca_trn, addlabels = TRUE)

#(pc_data_tst)

fviz_pca_var(pca_trn,
             col.var = "contrib")

fviz_pca_ind(pca_trn,
             geom.ind = "point",
             col.ind = as.factor(z_trn$class_type),
             addEllipses = TRUE)
## Too few points to calculate an ellipse
## Too few points to calculate an ellipse

fviz_pca_ind(pca_trn, geom.ind = c("point", "text"), 
             label = c("ind", "var"),
             col.ind = as.factor(z_trn$class_type),
             addEllipses = TRUE)
## Too few points to calculate an ellipse
## Too few points to calculate an ellipse

var = get_pca_var(pca_trn)

corrplot(var$cos2)

6. SVM WITH INPUTS AS PCA REDUCED DIMENSION TO 2 DIMENSIONS

library(caTools)
set.seed(123)
split = sample.split(z_df$class_type, SplitRatio = 0.8)
training_set = subset(z_df, split == TRUE)
test_set = subset(z_df, split == FALSE)

training_set <- training_set[,2:18]
test_set <- test_set[,2:18]

training_set[-17] = scale(training_set[-17])
test_set[-17] = scale(test_set[-17])

pca = preProcess(x = training_set[-17], method = 'pca', pcaComp = 2)

training_set = predict(pca, training_set)

training_set = training_set[c(2, 3, 1)]
test_set = predict(pca, test_set)
# we need to put the columns back in order we want
test_set = test_set[c(2, 3, 1)]

#head(training_set)

#---------------------Kernel is linear

classifier = svm(formula = class_type ~ .,
                 data = training_set,
                 type = 'C-classification',
                 kernel = 'linear')

y_pred = predict(classifier, newdata = test_set[-3])

#class(y_pred)

conf_max = confusionMatrix(y_pred, as.factor(test_set$class_type))

conf_max
## Confusion Matrix and Statistics
## 
##           Reference
## Prediction 1 2 3 4 5 6 7
##          1 8 0 0 0 0 0 0
##          2 0 4 0 0 1 0 2
##          3 0 0 1 0 0 0 0
##          4 0 0 0 3 0 0 0
##          5 0 0 0 0 0 0 0
##          6 0 0 0 0 0 2 0
##          7 0 0 0 0 0 0 0
## 
## Overall Statistics
##                                           
##                Accuracy : 0.8571          
##                  95% CI : (0.6366, 0.9695)
##     No Information Rate : 0.381           
##     P-Value [Acc > NIR] : 9.94e-06        
##                                           
##                   Kappa : 0.8119          
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: 1 Class: 2 Class: 3 Class: 4 Class: 5 Class: 6
## Sensitivity             1.000   1.0000  1.00000   1.0000  0.00000  1.00000
## Specificity             1.000   0.8235  1.00000   1.0000  1.00000  1.00000
## Pos Pred Value          1.000   0.5714  1.00000   1.0000      NaN  1.00000
## Neg Pred Value          1.000   1.0000  1.00000   1.0000  0.95238  1.00000
## Prevalence              0.381   0.1905  0.04762   0.1429  0.04762  0.09524
## Detection Rate          0.381   0.1905  0.04762   0.1429  0.00000  0.09524
## Detection Prevalence    0.381   0.3333  0.04762   0.1429  0.00000  0.09524
## Balanced Accuracy       1.000   0.9118  1.00000   1.0000  0.50000  1.00000
##                      Class: 7
## Sensitivity           0.00000
## Specificity           1.00000
## Pos Pred Value            NaN
## Neg Pred Value        0.90476
## Prevalence            0.09524
## Detection Rate        0.00000
## Detection Prevalence  0.00000
## Balanced Accuracy     0.50000
#cm = table(test_set[, 3], y_pred)
#cm

#conf_max = confusionMatrix(y_pred, test_set[, 3])
#conf_max

y_predTR = predict(classifier, newdata = training_set[-3])
#y_predTR

cmTR = table(training_set[, 3], y_predTR)
#cmTR

conf_max_trn = confusionMatrix(y_predTR, as.factor(training_set$class_type))
conf_max_trn
## Confusion Matrix and Statistics
## 
##           Reference
## Prediction  1  2  3  4  5  6  7
##          1 33  0  0  0  0  0  0
##          2  0 16  1  0  1  2  2
##          3  0  0  2  0  2  0  0
##          4  0  0  1 10  0  0  0
##          5  0  0  0  0  0  0  0
##          6  0  0  0  0  0  4  0
##          7  0  0  0  0  0  0  6
## 
## Overall Statistics
##                                           
##                Accuracy : 0.8875          
##                  95% CI : (0.7972, 0.9472)
##     No Information Rate : 0.4125          
##     P-Value [Acc > NIR] : < 2.2e-16       
##                                           
##                   Kappa : 0.8488          
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: 1 Class: 2 Class: 3 Class: 4 Class: 5 Class: 6
## Sensitivity            1.0000   1.0000   0.5000   1.0000   0.0000   0.6667
## Specificity            1.0000   0.9062   0.9737   0.9857   1.0000   1.0000
## Pos Pred Value         1.0000   0.7273   0.5000   0.9091      NaN   1.0000
## Neg Pred Value         1.0000   1.0000   0.9737   1.0000   0.9625   0.9737
## Prevalence             0.4125   0.2000   0.0500   0.1250   0.0375   0.0750
## Detection Rate         0.4125   0.2000   0.0250   0.1250   0.0000   0.0500
## Detection Prevalence   0.4125   0.2750   0.0500   0.1375   0.0000   0.0500
## Balanced Accuracy      1.0000   0.9531   0.7368   0.9929   0.5000   0.8333
##                      Class: 7
## Sensitivity             0.750
## Specificity             1.000
## Pos Pred Value          1.000
## Neg Pred Value          0.973
## Prevalence              0.100
## Detection Rate          0.075
## Detection Prevalence    0.075
## Balanced Accuracy       0.875
############# SVM PLOTS for KERNEL as LINEAR WITH TRAINING SET
#install.packages("Rfast")
library(Rfast)
## Warning: package 'Rfast' was built under R version 4.3.3
## Loading required package: Rcpp
## 
## Attaching package: 'Rcpp'
## The following object is masked from 'package:rsample':
## 
##     populate
## Loading required package: RcppZiggurat
## Warning: package 'RcppZiggurat' was built under R version 4.3.3
## Loading required package: RcppParallel
## Warning: package 'RcppParallel' was built under R version 4.3.3
## 
## Attaching package: 'RcppParallel'
## The following object is masked from 'package:Rcpp':
## 
##     LdFlags
## 
## Rfast: 2.1.0
##  ___ __ __ __ __    __ __ __ __ __ _             _               __ __ __ __ __     __ __ __ __ __ __   
## |  __ __ __ __  |  |  __ __ __ __ _/            / \             |  __ __ __ __ /   /__ __ _   _ __ __\  
## | |           | |  | |                         / _ \            | |                        / /          
## | |           | |  | |                        / / \ \           | |                       / /          
## | |           | |  | |                       / /   \ \          | |                      / /          
## | |__ __ __ __| |  | |__ __ __ __           / /     \ \         | |__ __ __ __ _        / /__/\          
## |    __ __ __ __|  |  __ __ __ __|         / /__ _ __\ \        |_ __ __ __ _   |      / ___  /           
## |   \              | |                    / _ _ _ _ _ _ \                     | |      \/  / /       
## | |\ \             | |                   / /           \ \                    | |         / /          
## | | \ \            | |                  / /             \ \                   | |        / /          
## | |  \ \           | |                 / /               \ \                  | |       / /          
## | |   \ \__ __ _   | |                / /                 \ \     _ __ __ __ _| |      / /          
## |_|    \__ __ __\  |_|               /_/                   \_\   /_ __ __ __ ___|      \/             team
## 
## Attaching package: 'Rfast'
## The following objects are masked from 'package:class':
## 
##     knn, knn.cv
## The following object is masked from 'package:dplyr':
## 
##     nth
## The following objects are masked from 'package:purrr':
## 
##     is_integer, transpose
set = training_set
X1 = seq(min(set[, 1]) - 1, max(set[, 1]) + 1, by = 0.01)
X2 = seq(min(set[, 2]) - 1, max(set[, 2]) + 1, by = 0.01)
grid_set = expand.grid(X1, X2)
colnames(grid_set) = c('PC1', 'PC2')
y_grid = predict(classifier, newdata = grid_set)

# Define colors for each class
class_colors <- c('blue3', 'green4', 'red3', 'purple', 'orange', 'yellow', 'cyan')

plot(set[, -3],
     main = 'Principal Component Analysis (PCA) (Training set)',
     xlab = 'PC1', ylab = 'PC2',
     xlim = range(X1), ylim = range(X2))

contour(X1, X2, matrix(as.numeric(y_grid), length(X1), length(X2)), add = TRUE)

# Assign colors based on predicted class labels
points(grid_set, pch = '.', col = class_colors[as.numeric(y_grid)])

# Color the original training data points based on their actual class labels
points(set, pch = 21, bg = class_colors[set[, 3]])

############# SVM PLOTS for KERNEL as LINEAR WITH TEST SET

set = test_set
X1 = seq(min(set[, 1]) - 1, max(set[, 1]) + 1, by = 0.01)
X2 = seq(min(set[, 2]) - 1, max(set[, 2]) + 1, by = 0.01)
grid_set = expand.grid(X1, X2)
colnames(grid_set) = c('PC1', 'PC2')
y_grid = predict(classifier, newdata = grid_set)

# Define colors for each class
class_colors <- c('blue3', 'green4', 'red3', 'purple', 'orange', 'yellow', 'cyan')

plot(set[, -3], main = 'Principal Component Analysis (PCA) (Test set)',
     xlab = 'PC1', ylab = 'PC2',
     xlim = range(X1), ylim = range(X2))

contour(X1, X2, matrix(as.numeric(y_grid), length(X1), length(X2)), add = TRUE)

# Assign colors based on predicted class labels
points(grid_set, pch = '.', col = class_colors[as.numeric(y_grid)])

# Color the original test data points based on their actual class labels
points(set, pch = 21, bg = class_colors[set[, 3]])

7. SVM WITH INPUTS AS PCA REDUCED DIMENSION TO 10 DIMENSIONS

z_trn <- z_df[1:70, 2:18]
z_tst <- z_df[71:101, 2:18]

#view(z_trn)
#view(z_tst)

#lc_z_trn <- z_trn[, ncol(z_trn)]
#lc_z_tst <- z_tst[, ncol(z_tst)]

#z_trn_ds <- z_trn[, -ncol(z_trn)]
#z_tst_ds <- z_tst[, -ncol(z_tst)]


#z_trn_pca <- prcomp(z_trn_ds, scale = T)
#z_tst_pca <- prcomp(z_tst_ds, scale = T)

z_trn[,-17] = scale(z_trn[,-17])
z_tst[,-17] = scale(z_tst[,-17])

#view(z_trn)
#view(z_tst)
pca = preProcess(x = z_trn[,-17], method = 'pca')
#z_tst_pca = preProcess(x = z_tst[-17], method = 'pca')

z_trn_pca = predict(pca, z_trn)
z_tst_pca = predict(pca, z_tst)

z_trn_pca = z_trn_pca[c(2, 3, 4,5,6,7,8,9,10,11,1)]
z_tst_pca = z_tst_pca[c(2, 3, 4,5,6,7,8,9,10,11,1)]
#test_set = test_set[c(2, 3, 4, 1)]

classifier = svm(formula = class_type ~ .,
                 data = z_trn_pca,
                 type = 'C-classification',
                 kernel = 'linear')

y_pred = predict(classifier, newdata = z_tst_pca[-11])

#class(y_pred)

conf_max = confusionMatrix(y_pred, as.factor(z_tst_pca$class_type))

print("Confusion Matrix with Linear Kernel")
## [1] "Confusion Matrix with Linear Kernel"
conf_max
## Confusion Matrix and Statistics
## 
##           Reference
## Prediction 1 2 3 4 5 6 7
##          1 8 0 2 0 0 0 1
##          2 0 7 1 0 0 0 0
##          3 0 0 0 0 0 0 0
##          4 0 0 1 4 0 0 0
##          5 0 0 0 0 1 0 0
##          6 0 0 0 0 0 2 2
##          7 0 0 0 0 0 0 2
## 
## Overall Statistics
##                                          
##                Accuracy : 0.7742         
##                  95% CI : (0.589, 0.9041)
##     No Information Rate : 0.2581         
##     P-Value [Acc > NIR] : 2.739e-09      
##                                          
##                   Kappa : 0.7211         
##                                          
##  Mcnemar's Test P-Value : NA             
## 
## Statistics by Class:
## 
##                      Class: 1 Class: 2 Class: 3 Class: 4 Class: 5 Class: 6
## Sensitivity            1.0000   1.0000    0.000   1.0000  1.00000  1.00000
## Specificity            0.8696   0.9583    1.000   0.9630  1.00000  0.93103
## Pos Pred Value         0.7273   0.8750      NaN   0.8000  1.00000  0.50000
## Neg Pred Value         1.0000   1.0000    0.871   1.0000  1.00000  1.00000
## Prevalence             0.2581   0.2258    0.129   0.1290  0.03226  0.06452
## Detection Rate         0.2581   0.2258    0.000   0.1290  0.03226  0.06452
## Detection Prevalence   0.3548   0.2581    0.000   0.1613  0.03226  0.12903
## Balanced Accuracy      0.9348   0.9792    0.500   0.9815  1.00000  0.96552
##                      Class: 7
## Sensitivity           0.40000
## Specificity           1.00000
## Pos Pred Value        1.00000
## Neg Pred Value        0.89655
## Prevalence            0.16129
## Detection Rate        0.06452
## Detection Prevalence  0.06452
## Balanced Accuracy     0.70000
classifier_poly = svm(formula = class_type ~ .,
                 data = z_trn_pca,
                 type = 'C-classification',
                 kernel = 'polynomial')

y_pred_poly = predict(classifier_poly, newdata = z_tst_pca[-11])

# Check unique levels in y_pred_poly and z_tst_pca$class_type
unique_levels <- unique(c(as.factor(y_pred_poly), as.factor(z_tst_pca$class_type)))

# Match levels and create confusion matrix
y_pred_poly_matched <- factor(y_pred_poly, levels = unique_levels)
z_tst_pca_matched <- factor(z_tst_pca$class_type, levels = unique_levels)

conf_max_poly = confusionMatrix(y_pred_poly_matched, z_tst_pca_matched)

print("Confusion Matrix with Polynomial Kernel")
## [1] "Confusion Matrix with Polynomial Kernel"
conf_max_poly
## Confusion Matrix and Statistics
## 
##           Reference
## Prediction 1 2 4 7 5 6 3
##          1 8 2 0 4 0 1 4
##          2 0 5 0 0 0 0 0
##          4 0 0 4 0 0 0 0
##          7 0 0 0 1 0 0 0
##          5 0 0 0 0 1 0 0
##          6 0 0 0 0 0 1 0
##          3 0 0 0 0 0 0 0
## 
## Overall Statistics
##                                           
##                Accuracy : 0.6452          
##                  95% CI : (0.4537, 0.8077)
##     No Information Rate : 0.2581          
##     P-Value [Acc > NIR] : 6.623e-06       
##                                           
##                   Kappa : 0.5453          
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: 1 Class: 2 Class: 4 Class: 7 Class: 5 Class: 6
## Sensitivity            1.0000   0.7143    1.000  0.20000  1.00000  0.50000
## Specificity            0.5217   1.0000    1.000  1.00000  1.00000  1.00000
## Pos Pred Value         0.4211   1.0000    1.000  1.00000  1.00000  1.00000
## Neg Pred Value         1.0000   0.9231    1.000  0.86667  1.00000  0.96667
## Prevalence             0.2581   0.2258    0.129  0.16129  0.03226  0.06452
## Detection Rate         0.2581   0.1613    0.129  0.03226  0.03226  0.03226
## Detection Prevalence   0.6129   0.1613    0.129  0.03226  0.03226  0.03226
## Balanced Accuracy      0.7609   0.8571    1.000  0.60000  1.00000  0.75000
##                      Class: 3
## Sensitivity             0.000
## Specificity             1.000
## Pos Pred Value            NaN
## Neg Pred Value          0.871
## Prevalence              0.129
## Detection Rate          0.000
## Detection Prevalence    0.000
## Balanced Accuracy       0.500
classifier_radial = svm(formula = class_type ~ .,
                 data = z_trn_pca,
                 type = 'C-classification',
                 kernel = 'radial')

y_pred_radial = predict(classifier_radial, newdata = z_tst_pca[-11])

#class(y_pred)

conf_max_radial = confusionMatrix(y_pred_radial, as.factor(z_tst_pca$class_type))

print("Confusion Matrix with radial Kernel")
## [1] "Confusion Matrix with radial Kernel"
conf_max_radial
## Confusion Matrix and Statistics
## 
##           Reference
## Prediction 1 2 3 4 5 6 7
##          1 8 0 3 0 0 0 3
##          2 0 7 1 0 0 0 0
##          3 0 0 0 0 0 0 0
##          4 0 0 0 4 0 0 0
##          5 0 0 0 0 1 0 0
##          6 0 0 0 0 0 2 0
##          7 0 0 0 0 0 0 2
## 
## Overall Statistics
##                                          
##                Accuracy : 0.7742         
##                  95% CI : (0.589, 0.9041)
##     No Information Rate : 0.2581         
##     P-Value [Acc > NIR] : 2.739e-09      
##                                          
##                   Kappa : 0.7152         
##                                          
##  Mcnemar's Test P-Value : NA             
## 
## Statistics by Class:
## 
##                      Class: 1 Class: 2 Class: 3 Class: 4 Class: 5 Class: 6
## Sensitivity            1.0000   1.0000    0.000    1.000  1.00000  1.00000
## Specificity            0.7391   0.9583    1.000    1.000  1.00000  1.00000
## Pos Pred Value         0.5714   0.8750      NaN    1.000  1.00000  1.00000
## Neg Pred Value         1.0000   1.0000    0.871    1.000  1.00000  1.00000
## Prevalence             0.2581   0.2258    0.129    0.129  0.03226  0.06452
## Detection Rate         0.2581   0.2258    0.000    0.129  0.03226  0.06452
## Detection Prevalence   0.4516   0.2581    0.000    0.129  0.03226  0.06452
## Balanced Accuracy      0.8696   0.9792    0.500    1.000  1.00000  1.00000
##                      Class: 7
## Sensitivity           0.40000
## Specificity           1.00000
## Pos Pred Value        1.00000
## Neg Pred Value        0.89655
## Prevalence            0.16129
## Detection Rate        0.06452
## Detection Prevalence  0.06452
## Balanced Accuracy     0.70000
classifier_sigmoid = svm(formula = class_type ~ .,
                 data = z_trn_pca,
                 type = 'C-classification',
                 kernel = 'sigmoid')

y_pred_sigmoid = predict(classifier_sigmoid, newdata = z_tst_pca[-11])

#class(y_pred)

conf_max_sigmoid = confusionMatrix(y_pred_sigmoid, as.factor(z_tst_pca$class_type))

print("Confusion Matrix with sigmoid Kernel")
## [1] "Confusion Matrix with sigmoid Kernel"
conf_max_sigmoid
## Confusion Matrix and Statistics
## 
##           Reference
## Prediction 1 2 3 4 5 6 7
##          1 8 0 3 0 0 0 0
##          2 0 7 0 0 0 0 0
##          3 0 0 0 0 0 0 0
##          4 0 0 0 4 0 0 0
##          5 0 0 1 0 1 0 0
##          6 0 0 0 0 0 1 0
##          7 0 0 0 0 0 1 5
## 
## Overall Statistics
##                                           
##                Accuracy : 0.8387          
##                  95% CI : (0.6627, 0.9455)
##     No Information Rate : 0.2581          
##     P-Value [Acc > NIR] : 2.068e-11       
##                                           
##                   Kappa : 0.7997          
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: 1 Class: 2 Class: 3 Class: 4 Class: 5 Class: 6
## Sensitivity            1.0000   1.0000    0.000    1.000  1.00000  0.50000
## Specificity            0.8696   1.0000    1.000    1.000  0.96667  1.00000
## Pos Pred Value         0.7273   1.0000      NaN    1.000  0.50000  1.00000
## Neg Pred Value         1.0000   1.0000    0.871    1.000  1.00000  0.96667
## Prevalence             0.2581   0.2258    0.129    0.129  0.03226  0.06452
## Detection Rate         0.2581   0.2258    0.000    0.129  0.03226  0.03226
## Detection Prevalence   0.3548   0.2258    0.000    0.129  0.06452  0.03226
## Balanced Accuracy      0.9348   1.0000    0.500    1.000  0.98333  0.75000
##                      Class: 7
## Sensitivity            1.0000
## Specificity            0.9615
## Pos Pred Value         0.8333
## Neg Pred Value         1.0000
## Prevalence             0.1613
## Detection Rate         0.1613
## Detection Prevalence   0.1935
## Balanced Accuracy      0.9808

8. LINEAR REGRESSION ON ZOO DATASET TO CHECK ON ACCURACY

set.seed(123)

z_df_kr <- z_df

z_df_kr <- z_df_kr[,2:18]

#view(z_df_kr)

train_ind= createDataPartition(y = z_df_kr$class_type,p = 0.80,list = FALSE)
train_kr = z_df_kr[train_ind,]
test_kr = z_df_kr[-train_ind,]

model = lm(class_type~hair+feathers
           +milk+airborne+aquatic+predator
           +toothed+backbone+breathes+venomous
           +fins+legs+tail+domestic+catsize, data=train_kr)
#summary(model)

par(mfrow=c(2,2))
plot(model)
## Warning: not plotting observations with leverage one:
##   75

#test_kr$pred <- predict(model, newdata = test_kr)

pred_class1_tst <- predict(model, newdata = test_kr,type = "response")

pred_class1_val <- ceiling(pred_class1_tst)

pred_class1_val <- as.factor(pred_class1_val)

#test_kr$pred_col <- round(test_kr$pred,digits=0)

#class(test_kr$class_type)

confusionMatrix(data = pred_class1_val, 
                reference = as.factor((test_kr$class_type))
                , mode = "prec_recall")
## Warning in levels(reference) != levels(data): longer object length is not a
## multiple of shorter object length
## Warning in confusionMatrix.default(data = pred_class1_val, reference =
## as.factor((test_kr$class_type)), : Levels are not in the same order for
## reference and data. Refactoring data to match.
## Confusion Matrix and Statistics
## 
##           Reference
## Prediction 1 2 3 4 5 6 7
##          1 8 0 0 0 0 0 0
##          2 0 1 0 0 0 0 0
##          3 0 3 0 0 0 0 0
##          4 0 0 1 2 0 0 0
##          5 0 0 0 0 1 0 0
##          6 0 0 0 0 0 0 0
##          7 0 0 0 0 0 2 1
## 
## Overall Statistics
##                                           
##                Accuracy : 0.6842          
##                  95% CI : (0.4345, 0.8742)
##     No Information Rate : 0.4211          
##     P-Value [Acc > NIR] : 0.01874         
##                                           
##                   Kappa : 0.5929          
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: 1 Class: 2 Class: 3 Class: 4 Class: 5 Class: 6
## Precision              1.0000  1.00000  0.00000   0.6667  1.00000       NA
## Recall                 1.0000  0.25000  0.00000   1.0000  1.00000   0.0000
## F1                     1.0000  0.40000      NaN   0.8000  1.00000       NA
## Prevalence             0.4211  0.21053  0.05263   0.1053  0.05263   0.1053
## Detection Rate         0.4211  0.05263  0.00000   0.1053  0.05263   0.0000
## Detection Prevalence   0.4211  0.05263  0.15789   0.1579  0.05263   0.0000
## Balanced Accuracy      1.0000  0.62500  0.41667   0.9706  1.00000   0.5000
##                      Class: 7
## Precision             0.33333
## Recall                1.00000
## F1                    0.50000
## Prevalence            0.05263
## Detection Rate        0.05263
## Detection Prevalence  0.15789
## Balanced Accuracy     0.94444

9. LOGISTIC REGRESSION ON ZOO DATASET(MULTINOMIAL)

library(nnet)

set.seed(123)

z_df_kr <- z_df

z_df_kr <- z_df_kr[,2:18]

#view(z_df_kr)

train_ind= createDataPartition(y = z_df_kr$class_type,p = 0.80,list = FALSE)
train_kr = z_df_kr[train_ind,]
test_kr = z_df_kr[-train_ind,]

multinom_model <- multinom(as.factor(class_type) ~ hair+feathers
           +milk+airborne+aquatic+predator
           +toothed+backbone+breathes+venomous
           +fins+legs+tail+domestic+catsize, data = train_kr)
## # weights:  119 (96 variable)
## initial  value 159.564632 
## iter  10 value 12.001030
## iter  20 value 0.077093
## iter  30 value 0.001126
## final  value 0.000078 
## converged
#summary(multinom_model)

test_kr$score_t1 <- predict(multinom_model, newdata = test_kr, type="class")

test_kr$prediction_f_t1 <- as.factor(test_kr$score_t1)

confusionMatrix(data = test_kr$prediction_f_t1, 
                reference = as.factor(test_kr$class_type), mode = "prec_recall")
## Confusion Matrix and Statistics
## 
##           Reference
## Prediction 1 2 3 4 5 6 7
##          1 7 0 0 0 0 0 0
##          2 0 4 0 0 0 0 0
##          3 0 0 1 0 0 0 0
##          4 0 0 0 2 0 0 0
##          5 0 0 0 0 1 0 0
##          6 0 0 0 0 0 2 0
##          7 1 0 0 0 0 0 1
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9474          
##                  95% CI : (0.7397, 0.9987)
##     No Information Rate : 0.4211          
##     P-Value [Acc > NIR] : 1.976e-06       
##                                           
##                   Kappa : 0.9314          
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: 1 Class: 2 Class: 3 Class: 4 Class: 5 Class: 6
## Precision              1.0000   1.0000  1.00000   1.0000  1.00000   1.0000
## Recall                 0.8750   1.0000  1.00000   1.0000  1.00000   1.0000
## F1                     0.9333   1.0000  1.00000   1.0000  1.00000   1.0000
## Prevalence             0.4211   0.2105  0.05263   0.1053  0.05263   0.1053
## Detection Rate         0.3684   0.2105  0.05263   0.1053  0.05263   0.1053
## Detection Prevalence   0.3684   0.2105  0.05263   0.1053  0.05263   0.1053
## Balanced Accuracy      0.9375   1.0000  1.00000   1.0000  1.00000   1.0000
##                      Class: 7
## Precision             0.50000
## Recall                1.00000
## F1                    0.66667
## Prevalence            0.05263
## Detection Rate        0.05263
## Detection Prevalence  0.10526
## Balanced Accuracy     0.97222

Inferences from the assessment of data set are

1. CODE BLOCK FOR DATA PREPARATION

  • In this code the data was cleaned, variables which were non-numeric was converted to numeric, NA values if any in field were converted.

2. DATASETS CREATED FOR USE IN LOGISTIC REGRESSION

  • The datasets created like z_df, c_df.

3. DATA ANALYSIS AND EXPLORATION OF DATASET.

  • Plot for number of animals Types of Animals was plotted.

  • KNN-K-NEAREST neighbour plot for various Animal class Type.

  • Correlation Plot between various dimensions.

  • Count of various class types over counts of its Attributes is plotted.

  • PIECHART of the class types over counts of its Attributes is plotted with various sections.

4. SVM WITH e1071 PACKAGE WITH VARIOUS KERNELS AND ACCURACY

  • There are various packages with SVM like e1071, CARET, kernlab etc.

  • I had used e1071 package with various kernels like linear, polynomial, radial, sigmoid

    • “Accuracy using linear kernel: 0.774193548387097”

    • Accuracy using polynomial Kernel: 0.806451612903226

    • Accuracy using radial Kernel: 0.741935483870968

    • Accuracy using sigmoid Kernel: 0.806451612903226

5. PCA DATA Visualization

  • PCA is principal component analysis. It is a method to reduce the dimensions in a dataset..

  • Extracting essential information from the different dimensions helps.

  • SCree plot shows which dimensions cause more impact.

  • The Variables plot shows which dimensions contribute more.

  • The Individuals PCA plot shows different classes as ellipses and points Correlation plot on the dimensions.

6. SVM WITH INPUTS AS PCA REDUCED DIMENSION TO 2 DIMENSIONS.

  • SVM is applied after creating reducing the dimensions using PCA to two components.

  • Finding the accuracy of the Model with input after appplying PCA reducing inputs to two components.

  • The accuracy of model was 0.8875 which is lot better than accuracy of models without the inputs from PCA applied.

  • Created SVM plots on Training and Test dataset.

7. SVM WITH INPUTS AS PCA REDUCED DIMENSION TO 10 DIMENSIONS.

  • SVM is applied after creating reducing the dimensions using PCA to 10 components.

  • SVM with principal components reduced to 10 dimensions with Linear Kernel has Accuracy : 0.7742

  • SVM with principal components reduced to 10 dimensions with Polynomial Kernel has Accuracy : 0.6452

  • SVM with principal components reduced to 10 dimensions with Radial Kernel has Accuracy : 0.7742

  • SVM with principal components reduced to 10 dimensions with Sigmoid Kernel has Accuracy : 0.8387

  • The confusion matrix of all these are available as well

8. LINEAR REGRESSION ON ZOO DATASET TO CHECK ON ACCURACY.

  • The Linear regression on ZOO dataset has accuracy of : 0.6842

  • Implemented it to collate older assignment as well for comparison of accuracy with SVM. The accuracy is less compared to SVM.

9. LOGISTIC REGRESSION ON ZOO DATASET(MULTINOMIAL).

  • The Multinomial Logistic regression on ZOO dataset has accuracy of : 0.9474

  • Implemented it to collate older assignment as well for comparison of accuracy with SVM. The accuracy is better than of SVM.

FINAL SUMMARY

The SVM model works better after applying PCA with accuracy of .8875(88%). The MULTINOMIAL logistic regression works slightly better than SVM

NOTE: Published the R-Mark Down in Rpubs

http://rpubs.com/Nivikm/Assignment5_DM