Check Image Dimension

To create a good deep learning model, we need to know the image dimension distribution from each image that we will used as the based for model creation. We can take a look the first image description.

# Full Image Description
img <- load.image(file_name[1])
img

## Image. Width: 512 pix Height: 512 pix Depth: 1 Colour channels: 3

From the chunk above, we can get the image’s width and height in pixel. Colour channels identify the image colour format, if the images in grayscale then the colour channels: 1 and if its in RGB / Red Green Blue then the colour channels: 3.

To collect the dimension only, we can use dim() function. This function will return the width, height, depth, and channel values.

# Image Dimension
dim(img)

## [1] 512 512   1   3

In below chunk, we will create a function to collect the image’s width and height then convert them into a dataframe.

get_dim <- function(x){
  img <- load.image(x) 
  
  df_img <- data.frame(height = height(img),
                       width = width(img),
                       filename = x
                       )
  
  return(df_img)
}

get_dim(file_name[1])

##   height width                              filename
## 1    512   512 data/training/glioma_tumor/gg (1).jpg

Using the above function, we can analyze the dimension of 1000 samples from the file_name object.

# Randomly get 1000 sample images
set.seed(123)
sample_file <- sample(file_name, 1000)

# Run the get_dim() function for each image
file_dim <- map_df(sample_file, get_dim)

head(file_dim, 10)

##    height width                                     filename
## 1     236   236         data/training/no_tumor/image(87).jpg
## 2     512   512     data/training/pituitary_tumor/p (13).jpg
## 3     275   220 data/training/no_tumor/image(238) - Copy.jpg
## 4     512   512      data/training/glioma_tumor/gg (572).jpg
## 5     512   512      data/training/glioma_tumor/gg (274).jpg
## 6     512   512    data/training/pituitary_tumor/p (558).jpg
## 7     236   236 data/training/no_tumor/image (52) - Copy.jpg
## 8     512   512    data/training/meningioma_tumor/m1(23).jpg
## 9     512   512  data/training/meningioma_tumor/m2 (123).jpg
## 10    512   512  data/training/meningioma_tumor/m2 (137).jpg

summary(file_dim)

##      height           width          filename        
##  Min.   : 198.0   Min.   : 200.0   Length:1000       
##  1st Qu.: 512.0   1st Qu.: 512.0   Class :character  
##  Median : 512.0   Median : 512.0   Mode  :character  
##  Mean   : 471.7   Mean   : 470.8                     
##  3rd Qu.: 512.0   3rd Qu.: 512.0                     
##  Max.   :1446.0   Max.   :1375.0

From 1000 samples in file_name object, we can see dimension variance from several images. Some image has height and width around 198 pixels up to 1446 pixels. Understanding the dimension variance, can be helpful when we are in pre-processing stage because we need to set the height and width dimensions. Each height and width dimension must be in the same size so we can train the model much better.

Data Augmentation

To make the height and width dimensions of each image into the same size, we can transform all images to a certain size, for example 64 x 64 pixels (height = 64 pixel and width = 64 pixel). What needs to be understood is that the larger the dimensions of height and width we choose, the more information we retain and the longer it will take to train our model. Of course the smaller the dimensions of height and width we choose, the more we will lose the required information from our data but the time required to train the model will be faster.

The dimensions of the height and width that will be used are 200 x 200 pixels. Do not forget to set the batch size as well so that after each batch of the model training, the model will be updated immediately. Keep in mind, the smaller the batch size, the longer the training process (because there are more model optimization processes) but it can prevent too large computations at one time. For this time, we are going to set the batch size into 100.

# Desired height and width of images
target_size <- c(200, 200)

# Batch size for training the model
batch_size <- 100

Since the number of images in the train dataset is not too many, we will create artificial data using the image augmentation method. Image augmentation can increase the size of the train dataset without the need for additional images. The purpose of using image augmentation is to teach the model not only to use the original image but also to study the modified images. Image augmentation modification results can be in the form of flipped image, rotated image, enlarged / reduced (zoom in / zoom out image), cropped image, and etc. By studying the original and modified data, our model will become better and improved.

We can do an image augmentation by using image data generator from keras library and we will apply following attributes to the image data generator:

• Scaling the pixel value by dividing the pixel value by 255
• Flip the image horizontally
• Flip the image vertically
• Rotate the image from 0 to 45 degrees
• Zoom in or zoom out by 25% (zoom 75% or 125%)
• Use 20% of the data as validation dataset (to test the model which trained using 80% of the data)

# Image Generator
train_data_gen <- image_data_generator(rescale = 1/255, # Scaling pixel value
                                       horizontal_flip = T, # Flip image horizontally
                                       vertical_flip = T, # Flip image vertically 
                                       rotation_range = 45, # Rotate image from 0 to 45 degrees
                                       zoom_range = 0.25, # Zoom in or zoom out range
                                       validation_split = 0.2 # 20% data as validation data
                                       )

After setting-up the image data generator in previous chunk, we can insert our image data into the generator using the flow_images_from_directory() function. Due to the save location for the data is located inside the training folder which saved inside the data folder, the directory must be set-up into: data/training/. We will use RGB color mode, target size and batch size are following the value in previous chunk, random seed = 123. Thus, from this process, we will get the augmented image both for training data and the validation data.

# Training Dataset
train_image_array_gen <- flow_images_from_directory(directory = "data/training/", # Folder of the data
                                                    target_size = target_size, # target of the image dimension (120 x 120)  
                                                    color_mode = "rgb", # use RGB color
                                                    batch_size = batch_size , 
                                                    seed = 123,  # set random seed
                                                    subset = "training", # declare that this is for training data
                                                    generator = train_data_gen
                                                    )

# Validation Dataset
val_image_array_gen <- flow_images_from_directory(directory = "data/training/",
                                                  target_size = target_size, 
                                                  color_mode = "rgb", 
                                                  batch_size = batch_size ,
                                                  seed = 123,
                                                  subset = "validation", # declare that this is the validation data
                                                  generator = train_data_gen
                                                  )

Then, using the chunk below we can get following information:

• Number of training samples, saved into train_samples object.
• Number of validation samples, saved into valid_samples object.
• Number of target classes/categories, saved into output_n object.
• Target variable class proportion.

# Number of training samples
train_samples <- train_image_array_gen$n

# Number of validation samples
valid_samples <- val_image_array_gen$n

# Number of target classes/categories
output_n <- n_distinct(train_image_array_gen$classes)

# Get the class proportion
table("\nFrequency" = factor(train_image_array_gen$classes)
      ) %>% 
  prop.table()

## 
## Frequency
##         0         1         2         3 
## 0.2500000 0.2488654 0.2507564 0.2503782

# glioma_tumor(0)
# meningioma_tumor(1)
# no_tumor(2)
# pituitary_tumor(3)

The above index represents target variable’s label or class which ordered alphabetically (0 = glioma_tumor, 1 = meningioma_tumor, 2 = no_tumor, 3 = pituitary_tumor).

Based on the proportion, each category’s proportion are quite balanced. Thus, accuracy metric in confusionMatrix() can be used to evaluate our model. Moreover, since we want the model to be able to precisely predict the result we will also use pos pred value as the second metric in confusionMatrix().

Model Architecture

If we look back at our first model architecture, actually we can extract more information while the data is still in an 2D image array. Only one CNN layer tasked to extract the general features of our image and then the features were downsampled using the max pooling layer. Even after pooling, we still have 100 x 100 array that has a lot of information to be extracted before flattening the data. Therefore, we can stacks more CNN layers into the model so there will be more information to be captured by the tuned model.

model

## Model
## Model: "simple_model"
## ________________________________________________________________________________
## Layer (type)                        Output Shape                    Param #     
## ================================================================================
## conv2d (Conv2D)                     (None, 200, 200, 16)            448         
## ________________________________________________________________________________
## max_pooling2d (MaxPooling2D)        (None, 100, 100, 16)            0           
## ________________________________________________________________________________
## flatten (Flatten)                   (None, 160000)                  0           
## ________________________________________________________________________________
## dense (Dense)                       (None, 16)                      2560016     
## ________________________________________________________________________________
## Output (Dense)                      (None, 4)                       68          
## ================================================================================
## Total params: 2,560,532
## Trainable params: 2,560,532
## Non-trainable params: 0
## ________________________________________________________________________________

The following is our improved model architecture (model_big):

• 1st Convolutional layer to extract features from 2D image with relu activation function, filter = 16, kernel size = 5 x 5.
• Max pooling layer with pool size 2 x 2.
• 2nd Convolutional layer to extract features from 2D image with relu activation function, filter = 32, kernel size = 5 x 5.
• Max pooling layer with pool size 2 x 2.
• 3rd Convolutional layer to extract features from 2D image with relu activation function, filter = 32, kernel size = 5 x 5.
• Max pooling layer with pool size 2 x 2.
• 4th Convolutional layer to extract features from 2D image with relu activation function, filter = 64, kernel size = 5 x 5.
• Max pooling layer with pool size 2 x 2.
• 5th Convolutional layer to extract features from 2D image with relu activation function, filter = 128, kernel size = 3 x 3.
• Max pooling layer with pool size 2 x 2.
• 6th Convolutional layer to extract features from 2D image with relu activation function, filter = 256, kernel size = 3 x 3.
• Max pooling layer with pool size 2 x 2.
• 1 Flattening layer to flatten data from 2D array to 1D array so it can be processed further by the dense layers then by the output layer.
• 1 Dense layer to capture more information from the flattening layer, with relu activation function, nodes = 64.
• 1 Dense layer to capture more information from the previous dense layer, with relu activation function, nodes = 128.
• 1 Dense layer to capture more information from the previous dense layer, with relu activation function, nodes = 256.
• 1 Dense layer for output with softmax activation function.

initializer <- initializer_random_normal(seed = 123)

model_big <- keras_model_sequential() %>%

  # First Convolution Layer
  layer_conv_2d(filters = 16,
                kernel_size = c(5,5),
                padding = "same",
                activation = "relu",
                kernel_initializer = initializer,
                bias_initializer = initializer,
                input_shape = c(target_size, 3)
                ) %>%

  # Max Pooling Layer
  layer_max_pooling_2d(pool_size = c(2,2)) %>%

  # Second convolutional layer
  layer_conv_2d(filters = 32,
                kernel_size = c(5,5),
                padding = "same",
                activation = "relu",
                kernel_initializer = initializer,
                bias_initializer = initializer
                ) %>%

  # Max pooling layer
  layer_max_pooling_2d(pool_size = c(2,2)) %>%

  # Third convolutional layer
  layer_conv_2d(filters = 32,
                kernel_size = c(5,5),
                padding = "same",
                activation = "relu",
                kernel_initializer = initializer,
                bias_initializer = initializer
                ) %>%

  # Max pooling layer
  layer_max_pooling_2d(pool_size = c(2,2)) %>%

  # Fourth convolutional layer
  layer_conv_2d(filters = 64,
                kernel_size = c(5,5),
                padding = "same",
                activation = "relu",
                kernel_initializer = initializer,
                bias_initializer = initializer
                ) %>%

  # Max pooling layer
  layer_max_pooling_2d(pool_size = c(2,2)) %>%

  # Fifth convolutional layer
  layer_conv_2d(filters = 128,
                kernel_size = c(3,3),
                padding = "same",
                activation = "relu",
                kernel_initializer = initializer,
                bias_initializer = initializer
                ) %>%

  # Max pooling layer
  layer_max_pooling_2d(pool_size = c(2,2)) %>%

  # Sixth convolutional layer
  layer_conv_2d(filters = 256,
                kernel_size = c(3,3),
                padding = "same",
                activation = "relu",
                kernel_initializer = initializer,
                bias_initializer = initializer
                ) %>%

  # Max pooling layer
  layer_max_pooling_2d(pool_size = c(2,2)) %>%

  # Flattening Layer
  layer_flatten() %>%

  # Dense Layer
  layer_dense(units = 64,
              activation = "relu",
              kernel_initializer = initializer,
              bias_initializer = initializer) %>%
  
  # Dense Layer
  layer_dense(units = 128,
              activation = "relu",
              kernel_initializer = initializer,
              bias_initializer = initializer) %>%
  
  # Dense Layer
  layer_dense(units = 256,
              activation = "relu",
              kernel_initializer = initializer,
              bias_initializer = initializer) %>%

  # Output Layer
  layer_dense(units = output_n,
              activation = "softmax",
              name = "Output",
              kernel_initializer = initializer,
              bias_initializer = initializer)

model_big 

## Model
## Model: "sequential"
## ________________________________________________________________________________
## Layer (type)                        Output Shape                    Param #     
## ================================================================================
## conv2d_6 (Conv2D)                   (None, 200, 200, 16)            1216        
## ________________________________________________________________________________
## max_pooling2d_6 (MaxPooling2D)      (None, 100, 100, 16)            0           
## ________________________________________________________________________________
## conv2d_5 (Conv2D)                   (None, 100, 100, 32)            12832       
## ________________________________________________________________________________
## max_pooling2d_5 (MaxPooling2D)      (None, 50, 50, 32)              0           
## ________________________________________________________________________________
## conv2d_4 (Conv2D)                   (None, 50, 50, 32)              25632       
## ________________________________________________________________________________
## max_pooling2d_4 (MaxPooling2D)      (None, 25, 25, 32)              0           
## ________________________________________________________________________________
## conv2d_3 (Conv2D)                   (None, 25, 25, 64)              51264       
## ________________________________________________________________________________
## max_pooling2d_3 (MaxPooling2D)      (None, 12, 12, 64)              0           
## ________________________________________________________________________________
## conv2d_2 (Conv2D)                   (None, 12, 12, 128)             73856       
## ________________________________________________________________________________
## max_pooling2d_2 (MaxPooling2D)      (None, 6, 6, 128)               0           
## ________________________________________________________________________________
## conv2d_1 (Conv2D)                   (None, 6, 6, 256)               295168      
## ________________________________________________________________________________
## max_pooling2d_1 (MaxPooling2D)      (None, 3, 3, 256)               0           
## ________________________________________________________________________________
## flatten_1 (Flatten)                 (None, 2304)                    0           
## ________________________________________________________________________________
## dense_3 (Dense)                     (None, 64)                      147520      
## ________________________________________________________________________________
## dense_2 (Dense)                     (None, 128)                     8320        
## ________________________________________________________________________________
## dense_1 (Dense)                     (None, 256)                     33024       
## ________________________________________________________________________________
## Output (Dense)                      (None, 4)                       1028        
## ================================================================================
## Total params: 649,860
## Trainable params: 649,860
## Non-trainable params: 0
## ________________________________________________________________________________

Model Fitting

After we set up the model architecture, we can compile the model by specifying the loss function and the optimizer before we fitting the data into the model. For multiclass classification, we will use categorical cross-entropy as the loss function. Then, we use adam optimizer with learning rate of 0.001 (before we set learning rate = 0.005). We also set epochs = 60 (before we set epochs = 20). The model will be also evaluated using the validation data from the generator.

model_big %>%
  compile(
    loss = "categorical_crossentropy",
    optimizer = optimizer_adam(lr = 0.001),
    metrics = "accuracy"
  )

history_big <- model_big %>%
  fit_generator(
  # training data
  train_image_array_gen,

  # epochs
  steps_per_epoch = as.integer(train_samples / batch_size),
  epochs = 60,

  # validation data
  validation_data = val_image_array_gen,
  validation_steps = as.integer(valid_samples / batch_size),

  # print progress but don't create graphic
  verbose = 1,
  view_metrics = 0
)

plot(history_big)

Model Evaluation

Now we will evaluate the data, the tuned model, and the confusion matrix for the validation data.

pred_test_big <- predict_classes(model_big, test_x)
pred_test_big <- sapply(pred_test_big, decode)
head(pred_test_big)

## [1] "no_tumor" "no_tumor" "no_tumor" "no_tumor" "no_tumor" "no_tumor"

confusionMatrix(as.factor(pred_test_big), 
                as.factor(val_data$class)
                )

history_big$metrics$accuracy[60]

Based on confusion matrix evaluation result, we found that the validation data accuracy amounting to 90% while the training data accuracy amounting to 97% which indicate the model is quite fit since the accuracy discrepancy is not too large. Further, We will also evaluate the model through the pos pred value (prediction) metric. On this metric, it’s obvious that the precision for all class already good enough, all passed 70%. Thus, considering the accuracy and prediction values, we can say that the tuned model (model_big) already improved a lot compare to the previous one (model).