Model Architecture

A Convolutional Neural Network (ConvNet/CNN) is a deep learning method that can process an input image, assign importance to learnable weights and biases to be able to differentiate one image from the other [^4]. The CNN also can be performed to speech or audio signal inputs. The following are model architectures for model original.

• Convolutional layer is the first layer of CNN. This layer can extract features from the 2D image and identify the greater portions of the image.
• Max Pooling layer to downsample the image features
• Flattening layer to flatten data from the 2D array to 1D array
• Dense layer to capture more information
• Dense layer for output with softmax activation function.

We set the input shape, which consists of the target_size and the color channels ( RGB = 3 )

# input shape of the image
c(target_size, 3) 

## [1] 128 128   3

# Set Initial Random Weight
tensorflow::tf$random$set_seed(123)

model <- keras_model_sequential() %>% 
  
  # Convolution Layer 1
  layer_conv_2d(filters = 64,
                kernel_size = c(3,3),
                padding = "same",
                activation = "relu",
                input_shape = c(target_size, 3) 
                ) %>% 

  # Max Pooling Layer 1
  layer_max_pooling_2d(pool_size = c(2,2),
                       strides = c(2,2)) %>% 
  
  # Convolution Layer 2
  layer_conv_2d(filters = 64,
                kernel_size = c(3,3),
                padding = "same",
                activation = "relu"
                ) %>% 

  # Max Pooling Layer 2
  layer_max_pooling_2d(pool_size = c(2,2),
                       strides = c(2,2)) %>% 
  
  # Flattening Layer
  layer_flatten() %>% 
  
  # Dense Layer1
  layer_dense(units = 128,
              activation = "relu") %>%

  # Output Layer
  layer_dense(units = output_n,
              activation = "softmax",
              name = "Output")
  
model

## Model: "sequential"
## ________________________________________________________________________________
## Layer (type)                        Output Shape                    Param #     
## ================================================================================
## conv2d_1 (Conv2D)                   (None, 128, 128, 64)            1792        
## ________________________________________________________________________________
## max_pooling2d_1 (MaxPooling2D)      (None, 64, 64, 64)              0           
## ________________________________________________________________________________
## conv2d (Conv2D)                     (None, 64, 64, 64)              36928       
## ________________________________________________________________________________
## max_pooling2d (MaxPooling2D)        (None, 32, 32, 64)              0           
## ________________________________________________________________________________
## flatten (Flatten)                   (None, 65536)                   0           
## ________________________________________________________________________________
## dense (Dense)                       (None, 128)                     8388736     
## ________________________________________________________________________________
## Output (Dense)                      (None, 2)                       258         
## ================================================================================
## Total params: 8,427,714
## Trainable params: 8,427,714
## Non-trainable params: 0
## ________________________________________________________________________________

• The input image has 128 x 128 pixels from 64 filters.
• Set the padding = same to keep the dimension 128 x 128 pixels after being extracted.
• Downsample for each 2 x 2 pooling area, thus the data has 64 x 64 pixels from 32 filters.
• Set another convolutional layer and downsample, thus the data has 32 x 32 pixels from 64 filters.
  
• Flatten the 2D array into 1D with 32 x 32 x 64 = 65536 nodes.
• Extract information using a simple dense layer.
• Output layer will transform the softmax activation function to obtain the probability of each class as the output.

Model Fitting

Since we are dealing with binary classification dataset, we define the loss = binary_crossentropy and optimizer = optimizer_adam

# Compile Model
model %>% 
  compile(
    loss = "binary_crossentropy",
    optimizer = optimizer_adam(learning_rate = 0.001),
    metrics = "accuracy"
  )

The model will use validation dataset from the generator to evaluate the model. Also, we set the epoch = 5. We save the model as object called history.

# Fit data into model
history <- model %>% 
  fit(
  # training data
  train_image_array_gen,

  # training epochs
  steps_per_epoch = as.integer(train_samples / batch_size), 
  epochs = 5, 
  
  # validation data
  validation_data = val_image_array_gen,
  validation_steps = as.integer(valid_samples / batch_size)
) 

# Plot the history
plot(history)

Model Evaluation

We evaluate the model by using a confusion matrix using the validation data from the generator. We create an object called val_data containing the file name of the image. For this file name, we extract the categorical label as the actual value of the target variable.

val_data <- data.frame(file_name = paste0("santa/train/", val_image_array_gen$filenames)) %>% 
  mutate(class = str_extract(file_name, "not-a-santa|Santa"))

# Observe the val_data
tail(val_data, 10)

##                           file_name class
## 113 santa/train/santa/193.Santa.jpg Santa
## 114 santa/train/santa/194.Santa.jpg Santa
## 115 santa/train/santa/197.Santa.jpg Santa
## 116 santa/train/santa/198.Santa.jpg Santa
## 117   santa/train/santa/2.Santa.jpg Santa
## 118  santa/train/santa/20.Santa.jpg Santa
## 119 santa/train/santa/203.Santa.jpg Santa
## 120 santa/train/santa/204.Santa.jpg Santa
## 121 santa/train/santa/205.Santa.jpg Santa
## 122 santa/train/santa/206.Santa.jpg Santa

We convert the image into an array with a dimension of 128 x 128 pixels with 3 colour channels (RGB). We perform this to ensure the testing data is in the actual image.

# Function to convert image to array
image_prep <- function(x) {
  arrays <- lapply(x, function(path) {
    img <- image_load(path, target_size = target_size, 
                      grayscale = F # Set FALSE if image is RGB
                      )
    
    x <- image_to_array(img)
    x <- array_reshape(x, c(1, dim(x)))
    x <- x/255 # rescale image pixel
  })
  do.call(abind::abind, c(arrays, list(along = 1)))
}

test_x <- image_prep(val_data$file_name)

# Check dimension of testing data set
dim(test_x)

## [1] 122 128 128   3

The validation data consists of 122 images with dimensions of 128 x 128 pixels and 3 colour channels (RGB). Then, we perform prediction on the test_x data.

# Prediction
pred_val <- predict_classes(model, test_x) 

# Observe first 10 prediction
head(pred_val, 10)

##  [1] 0 0 1 0 0 0 1 0 0 0

We convert the encoding into proper class label by using function decode().

# Convert encoding to label
decode <- function(x){
  case_when(x == 0 ~ "not-a-santa",
            x == 1 ~ "Santa"
            )
}

# Apply the function to `pred_val`
pred_val <- sapply(pred_val, decode) 

# Observe `pred_val`
head(pred_val, 10)

##  [1] "not-a-santa" "not-a-santa" "Santa"       "not-a-santa" "not-a-santa"
##  [6] "not-a-santa" "Santa"       "not-a-santa" "not-a-santa" "not-a-santa"

confusionMatrix(as.factor(pred_val), as.factor(val_data$class), positive="Santa")

## Confusion Matrix and Statistics
## 
##              Reference
## Prediction    not-a-santa Santa
##   not-a-santa          48     3
##   Santa                13    58
##                                               
##                Accuracy : 0.8689              
##                  95% CI : (0.7958, 0.9231)    
##     No Information Rate : 0.5                 
##     P-Value [Acc > NIR] : < 0.0000000000000002
##                                               
##                   Kappa : 0.7377              
##                                               
##  Mcnemar's Test P-Value : 0.02445             
##                                               
##             Sensitivity : 0.9508              
##             Specificity : 0.7869              
##          Pos Pred Value : 0.8169              
##          Neg Pred Value : 0.9412              
##              Prevalence : 0.5000              
##          Detection Rate : 0.4754              
##    Detection Prevalence : 0.5820              
##       Balanced Accuracy : 0.8689              
##                                               
##        'Positive' Class : Santa               
## 

For this study, we want to have the highest accuracy for the model. As shown in the confusion matrix, the accuracy of the model is quite good but it could be improved. We can perform model tuning and observe if the tune model can increase the accuracy of the original model.

Model Architecture

The improvement model will have additional CNN layers, so that more information can be captured. The following is our improved model architecture:

• 1st Convolutional layer to extract features from 2D image with relu activation function
• 2nd Convolutional layer to extract features from 2D image with relu activation function
• Max pooling layer
• 3rd Convolutional layer to extract features from 2D image with relu activation function
• Max pooling layer
• 4th Convolutional layer to extract features from 2D image with relu activation function
• Max pooling layer
• 5th Convolutional layer to extract features from 2D image with relu activation function
• Max pooling layer
• Flattening layer from 2D array to 1D array
• Dense layer to capture more information
• Dense layer for output layer

Model Fitting

tensorflow::tf$random$set_seed(123)

model_big <- keras_model_sequential() %>% 
  
  # First convolutional layer
  layer_conv_2d(filters = 128,
                kernel_size = c(5,5), # 5 x 5 filters
                padding = "same",
                activation = "relu",
                input_shape = c(target_size, 3)
                ) %>% 
  
  # Second convolutional layer
  layer_conv_2d(filters = 128,
                kernel_size = c(3,3), # 3 x 3 filters
                padding = "same",
                activation = "relu"
                ) %>% 
  
  # Max pooling layer
  layer_max_pooling_2d(pool_size = c(2,2)) %>% 
  
  # Third convolutional layer
  layer_conv_2d(filters = 64,
                kernel_size = c(3,3),
                padding = "same",
                activation = "relu"
                ) %>% 

  # Max pooling layer
  layer_max_pooling_2d(pool_size = c(2,2)) %>% 
  
  # Fourth convolutional layer
  layer_conv_2d(filters = 128,
                kernel_size = c(3,3),
                padding = "same",
                activation = "relu"
                ) %>% 
  
  # Max pooling layer
  layer_max_pooling_2d(pool_size = c(2,2)) %>% 

  # Fifth convolutional layer
  layer_conv_2d(filters = 256,
                kernel_size = c(3,3),
                padding = "same",
                activation = "relu"
                ) %>% 
  
  # Max pooling layer
  layer_max_pooling_2d(pool_size = c(2,2)) %>% 
  
  # Flattening layer
  layer_flatten() %>% 
  
  # Dense layer
  layer_dense(units = 64,
              activation = "relu") %>% 
  
  # Output layer
  layer_dense(name = "Output",
              units = 2, 
              activation = "softmax")

model_big

## Model: "sequential_1"
## ________________________________________________________________________________
## Layer (type)                        Output Shape                    Param #     
## ================================================================================
## conv2d_6 (Conv2D)                   (None, 128, 128, 128)           9728        
## ________________________________________________________________________________
## conv2d_5 (Conv2D)                   (None, 128, 128, 128)           147584      
## ________________________________________________________________________________
## max_pooling2d_5 (MaxPooling2D)      (None, 64, 64, 128)             0           
## ________________________________________________________________________________
## conv2d_4 (Conv2D)                   (None, 64, 64, 64)              73792       
## ________________________________________________________________________________
## max_pooling2d_4 (MaxPooling2D)      (None, 32, 32, 64)              0           
## ________________________________________________________________________________
## conv2d_3 (Conv2D)                   (None, 32, 32, 128)             73856       
## ________________________________________________________________________________
## max_pooling2d_3 (MaxPooling2D)      (None, 16, 16, 128)             0           
## ________________________________________________________________________________
## conv2d_2 (Conv2D)                   (None, 16, 16, 256)             295168      
## ________________________________________________________________________________
## max_pooling2d_2 (MaxPooling2D)      (None, 8, 8, 256)               0           
## ________________________________________________________________________________
## flatten_1 (Flatten)                 (None, 16384)                   0           
## ________________________________________________________________________________
## dense_1 (Dense)                     (None, 64)                      1048640     
## ________________________________________________________________________________
## Output (Dense)                      (None, 2)                       130         
## ================================================================================
## Total params: 1,648,898
## Trainable params: 1,648,898
## Non-trainable params: 0
## ________________________________________________________________________________

We train the data with more epochs and save the model as history_tune.

model_big %>% 
  compile(
    loss = "binary_crossentropy",
    optimizer = optimizer_adam(lr = 0.001),
    metrics = "accuracy")

history_tune <- model_big %>% 
  fit_generator(
  # training data
  train_image_array_gen,
  
  # epochs
  steps_per_epoch = as.integer(train_samples / batch_size), 
  epochs = 30, 
  
  # validation data
  validation_data = val_image_array_gen,
  validation_steps = as.integer(valid_samples / batch_size),
  )

plot(history_tune)

## `geom_smooth()` using formula 'y ~ x'

Model Evaluation

We repeat the evaluate process the data to obtain the confusion matrix.

pred_tune <- predict_classes(model_big, test_x) 

head(pred_tune, 10)

##  [1] 0 0 0 0 0 0 1 0 0 0

# Apply the convert encoding
pred_tune <- sapply(pred_tune, decode) 

# Observe `pred_tune`
head(pred_tune, 10)

##  [1] "not-a-santa" "not-a-santa" "not-a-santa" "not-a-santa" "not-a-santa"
##  [6] "not-a-santa" "Santa"       "not-a-santa" "not-a-santa" "not-a-santa"

Now, let’s observe the confusion matrix.

confusionMatrix(as.factor(pred_tune), 
                as.factor(val_data$class), 
                positive="Santa"
                )

## Confusion Matrix and Statistics
## 
##              Reference
## Prediction    not-a-santa Santa
##   not-a-santa          54     2
##   Santa                 7    59
##                                              
##                Accuracy : 0.9262             
##                  95% CI : (0.8646, 0.9657)   
##     No Information Rate : 0.5                
##     P-Value [Acc > NIR] : <0.0000000000000002
##                                              
##                   Kappa : 0.8525             
##                                              
##  Mcnemar's Test P-Value : 0.1824             
##                                              
##             Sensitivity : 0.9672             
##             Specificity : 0.8852             
##          Pos Pred Value : 0.8939             
##          Neg Pred Value : 0.9643             
##              Prevalence : 0.5000             
##          Detection Rate : 0.4836             
##    Detection Prevalence : 0.5410             
##       Balanced Accuracy : 0.9262             
##                                              
##        'Positive' Class : Santa              
## 

The accuracy of the model improves with tuning the model. Thus, we can perform the prediction on our test dataset.