Introduction

In the present example, we focus on classifying image using keras-TensorFlow in R. To this end, we selected 20 objects (10 airplanes and 10 cars) from google.com and we apply keras to image recognition.

Import the required packages.

#import packages
library(EBImage)
library(keras)

## 
## Attaching package: 'keras'

## The following object is masked from 'package:EBImage':
## 
##     normalize

Reading image

The images for this example were selected from google.com, airplanes and cars. It could be any other object.

# setwd('C:\Users\rzeze\Documents\Zezela\MyFiles_R\Image')

# test read image
image <- readImage('plane1.jpg')
display(image)

We selected differents 10 airplanes and 10 cars in order to classify image using keras - TensorFlow .

# list all images

pics <- c('plane1.jpg', 'plane2.jpg', 'plane3.jpg', 'plane4.jpg', 'plane5.jpg', 
          'plane6.jpg', 'plane7.jpg', 'plane8.jpg', 'plane9.jpg', 'plane10.jpg',
          'car1.jpg', 'car2.jpg', 'car3.jpg', 'car4.jpg', 'car5.jpg', 
          'car6.jpg', 'car7.jpg', 'car8.jpg', 'car9.jpg', 'car10.jpg')

mypics <- list()

for (i in 1:20){mypics[[i]] <- readImage(pics[i])}

To explore theses images let us use display.

#explore
print(mypics[[1]])

## Image 
##   colorMode    : Color 
##   storage.mode : double 
##   dim          : 275 183 3 
##   frames.total : 3 
##   frames.render: 1 
## 
## imageData(object)[1:5,1:6,1]
##           [,1]      [,2]      [,3]      [,4]      [,5]      [,6]
## [1,] 0.6549020 0.7019608 0.7529412 0.7803922 0.7607843 0.7254902
## [2,] 0.6431373 0.6941176 0.7490196 0.7764706 0.7568627 0.7215686
## [3,] 0.6274510 0.6745098 0.7372549 0.7686275 0.7529412 0.7098039
## [4,] 0.6000000 0.6549020 0.7176471 0.7607843 0.7490196 0.7058824
## [5,] 0.5725490 0.6274510 0.7019608 0.7450980 0.7411765 0.6941176

display(mypics[[5]])

summary(mypics[[5]])

##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##  0.0000  0.5647  0.6824  0.6628  0.7725  1.0000

hist(mypics[[5]])

str(mypics)

## List of 20
##  $ :Formal class 'Image' [package "EBImage"] with 2 slots
##   .. ..@ .Data    : num [1:275, 1:183, 1:3] 0.655 0.643 0.627 0.6 0.573 ...
##   .. ..@ colormode: int 2
##  $ :Formal class 'Image' [package "EBImage"] with 2 slots
##   .. ..@ .Data    : num [1:300, 1:168, 1:3] 0.0431 0.0471 0.0471 0.051 0.0549 ...
##   .. ..@ colormode: int 2
##  $ :Formal class 'Image' [package "EBImage"] with 2 slots
##   .. ..@ .Data    : num [1:300, 1:168, 1:3] 0.486 0.482 0.482 0.482 0.478 ...
##   .. ..@ colormode: int 2
##  $ :Formal class 'Image' [package "EBImage"] with 2 slots
##   .. ..@ .Data    : num [1:318, 1:159, 1:3] 0.98 0.98 0.984 0.992 0.992 ...
##   .. ..@ colormode: int 2
##  $ :Formal class 'Image' [package "EBImage"] with 2 slots
##   .. ..@ .Data    : num [1:275, 1:183, 1:3] 0.49 0.49 0.49 0.49 0.49 ...
##   .. ..@ colormode: int 2
##  $ :Formal class 'Image' [package "EBImage"] with 2 slots
##   .. ..@ .Data    : num [1:300, 1:168, 1:3] 0.561 0.561 0.561 0.561 0.565 ...
##   .. ..@ colormode: int 2
##  $ :Formal class 'Image' [package "EBImage"] with 2 slots
##   .. ..@ .Data    : num [1:300, 1:168, 1:3] 0.196 0.196 0.196 0.196 0.2 ...
##   .. ..@ colormode: int 2
##  $ :Formal class 'Image' [package "EBImage"] with 2 slots
##   .. ..@ .Data    : num [1:299, 1:168, 1:3] 0.357 0.357 0.361 0.361 0.361 ...
##   .. ..@ colormode: int 2
##  $ :Formal class 'Image' [package "EBImage"] with 2 slots
##   .. ..@ .Data    : num [1:300, 1:168, 1:3] 0.443 0.443 0.443 0.447 0.447 ...
##   .. ..@ colormode: int 2
##  $ :Formal class 'Image' [package "EBImage"] with 2 slots
##   .. ..@ .Data    : num [1:602, 1:339, 1:3] 0.698 0.698 0.694 0.69 0.686 ...
##   .. ..@ colormode: int 2
##  $ :Formal class 'Image' [package "EBImage"] with 2 slots
##   .. ..@ .Data    : num [1:259, 1:194, 1:3] 0.996 0.996 0.996 0.992 0.988 ...
##   .. ..@ colormode: int 2
##  $ :Formal class 'Image' [package "EBImage"] with 2 slots
##   .. ..@ .Data    : num [1:308, 1:164, 1:3] 1 1 1 1 1 1 1 1 1 1 ...
##   .. ..@ colormode: int 2
##  $ :Formal class 'Image' [package "EBImage"] with 2 slots
##   .. ..@ .Data    : num [1:259, 1:194, 1:3] 1 1 1 1 1 1 1 1 1 1 ...
##   .. ..@ colormode: int 2
##  $ :Formal class 'Image' [package "EBImage"] with 2 slots
##   .. ..@ .Data    : num [1:287, 1:175, 1:3] 1 1 1 1 1 1 1 1 1 1 ...
##   .. ..@ colormode: int 2
##  $ :Formal class 'Image' [package "EBImage"] with 2 slots
##   .. ..@ .Data    : num [1:276, 1:183, 1:3] 0.737 0.737 0.737 0.741 0.741 ...
##   .. ..@ colormode: int 2
##  $ :Formal class 'Image' [package "EBImage"] with 2 slots
##   .. ..@ .Data    : num [1:331, 1:152, 1:3] 1 1 1 1 1 1 1 1 1 1 ...
##   .. ..@ colormode: int 2
##  $ :Formal class 'Image' [package "EBImage"] with 2 slots
##   .. ..@ .Data    : num [1:275, 1:183, 1:3] 0.0863 0.0902 0.098 0.1098 0.1255 ...
##   .. ..@ colormode: int 2
##  $ :Formal class 'Image' [package "EBImage"] with 2 slots
##   .. ..@ .Data    : num [1:275, 1:183, 1:3] 1 1 1 1 1 1 1 1 1 1 ...
##   .. ..@ colormode: int 2
##  $ :Formal class 'Image' [package "EBImage"] with 2 slots
##   .. ..@ .Data    : num [1:300, 1:168, 1:3] 1 1 1 1 1 1 1 1 1 1 ...
##   .. ..@ colormode: int 2
##  $ :Formal class 'Image' [package "EBImage"] with 2 slots
##   .. ..@ .Data    : num [1:290, 1:174, 1:3] 1 1 1 1 1 1 1 1 1 1 ...
##   .. ..@ colormode: int 2

Notice that the dimensions of the objects are differents, therefore we are going to resize them in order to get the same size.

# Resize
for (i in 1:20){mypics[[i]] <- resize(mypics[[i]],28,28)}

str(mypics)

## List of 20
##  $ :Formal class 'Image' [package "EBImage"] with 2 slots
##   .. ..@ .Data    : num [1:28, 1:28, 1:3] 0.73 0.559 0.478 0.417 0.401 ...
##   .. ..@ colormode: int 2
##  $ :Formal class 'Image' [package "EBImage"] with 2 slots
##   .. ..@ .Data    : num [1:28, 1:28, 1:3] 0.0765 0.0944 0.0955 0.1118 0.1176 ...
##   .. ..@ colormode: int 2
##  $ :Formal class 'Image' [package "EBImage"] with 2 slots
##   .. ..@ .Data    : num [1:28, 1:28, 1:3] 0.478 0.462 0.429 0.4 0.393 ...
##   .. ..@ colormode: int 2
##  $ :Formal class 'Image' [package "EBImage"] with 2 slots
##   .. ..@ .Data    : num [1:28, 1:28, 1:3] 0.994 1 1 0.997 0.996 ...
##   .. ..@ colormode: int 2
##  $ :Formal class 'Image' [package "EBImage"] with 2 slots
##   .. ..@ .Data    : num [1:28, 1:28, 1:3] 0.49 0.49 0.49 0.502 0.502 ...
##   .. ..@ colormode: int 2
##  $ :Formal class 'Image' [package "EBImage"] with 2 slots
##   .. ..@ .Data    : num [1:28, 1:28, 1:3] 0.571 0.582 0.607 0.625 0.666 ...
##   .. ..@ colormode: int 2
##  $ :Formal class 'Image' [package "EBImage"] with 2 slots
##   .. ..@ .Data    : num [1:28, 1:28, 1:3] 0.204 0.217 0.23 0.247 0.264 ...
##   .. ..@ colormode: int 2
##  $ :Formal class 'Image' [package "EBImage"] with 2 slots
##   .. ..@ .Data    : num [1:28, 1:28, 1:3] 0.358 0.367 0.376 0.376 0.375 ...
##   .. ..@ colormode: int 2
##  $ :Formal class 'Image' [package "EBImage"] with 2 slots
##   .. ..@ .Data    : num [1:28, 1:28, 1:3] 0.453 0.457 0.473 0.478 0.512 ...
##   .. ..@ colormode: int 2
##  $ :Formal class 'Image' [package "EBImage"] with 2 slots
##   .. ..@ .Data    : num [1:28, 1:28, 1:3] 0.676 0.703 0.751 0.766 0.773 ...
##   .. ..@ colormode: int 2
##  $ :Formal class 'Image' [package "EBImage"] with 2 slots
##   .. ..@ .Data    : num [1:28, 1:28, 1:3] 0.93 0.901 0.902 0.952 0.953 ...
##   .. ..@ colormode: int 2
##  $ :Formal class 'Image' [package "EBImage"] with 2 slots
##   .. ..@ .Data    : num [1:28, 1:28, 1:3] 1 1 1 1 1 1 1 1 1 1 ...
##   .. ..@ colormode: int 2
##  $ :Formal class 'Image' [package "EBImage"] with 2 slots
##   .. ..@ .Data    : num [1:28, 1:28, 1:3] 1 1 1 1 1 1 1 1 1 1 ...
##   .. ..@ colormode: int 2
##  $ :Formal class 'Image' [package "EBImage"] with 2 slots
##   .. ..@ .Data    : num [1:28, 1:28, 1:3] 1 1 1 1 1 1 1 1 1 1 ...
##   .. ..@ colormode: int 2
##  $ :Formal class 'Image' [package "EBImage"] with 2 slots
##   .. ..@ .Data    : num [1:28, 1:28, 1:3] 0.752 0.764 0.771 0.832 0.789 ...
##   .. ..@ colormode: int 2
##  $ :Formal class 'Image' [package "EBImage"] with 2 slots
##   .. ..@ .Data    : num [1:28, 1:28, 1:3] 1 1 1 1 1 1 1 1 1 1 ...
##   .. ..@ colormode: int 2
##  $ :Formal class 'Image' [package "EBImage"] with 2 slots
##   .. ..@ .Data    : num [1:28, 1:28, 1:3] 0.13 0.239 0.303 0.261 0.184 ...
##   .. ..@ colormode: int 2
##  $ :Formal class 'Image' [package "EBImage"] with 2 slots
##   .. ..@ .Data    : num [1:28, 1:28, 1:3] 1 1 1 1 1 1 1 1 1 1 ...
##   .. ..@ colormode: int 2
##  $ :Formal class 'Image' [package "EBImage"] with 2 slots
##   .. ..@ .Data    : num [1:28, 1:28, 1:3] 1 1 1 1 1 1 1 1 1 1 ...
##   .. ..@ colormode: int 2
##  $ :Formal class 'Image' [package "EBImage"] with 2 slots
##   .. ..@ .Data    : num [1:28, 1:28, 1:3] 1 1 1 1 1 1 1 1 1 1 ...
##   .. ..@ colormode: int 2

Now we need to reshape the matrix.

# Reshape
for (i in 1:20) {mypics[[i]] <- array_reshape(mypics[[i]],c(28,28,3))}
str(mypics)

## List of 20
##  $ : num [1:28, 1:28, 1:3] 0.73 0.559 0.478 0.417 0.401 ...
##  $ : num [1:28, 1:28, 1:3] 0.0765 0.0944 0.0955 0.1118 0.1176 ...
##  $ : num [1:28, 1:28, 1:3] 0.478 0.462 0.429 0.4 0.393 ...
##  $ : num [1:28, 1:28, 1:3] 0.994 1 1 0.997 0.996 ...
##  $ : num [1:28, 1:28, 1:3] 0.49 0.49 0.49 0.502 0.502 ...
##  $ : num [1:28, 1:28, 1:3] 0.571 0.582 0.607 0.625 0.666 ...
##  $ : num [1:28, 1:28, 1:3] 0.204 0.217 0.23 0.247 0.264 ...
##  $ : num [1:28, 1:28, 1:3] 0.358 0.367 0.376 0.376 0.375 ...
##  $ : num [1:28, 1:28, 1:3] 0.453 0.457 0.473 0.478 0.512 ...
##  $ : num [1:28, 1:28, 1:3] 0.676 0.703 0.751 0.766 0.773 ...
##  $ : num [1:28, 1:28, 1:3] 0.93 0.901 0.902 0.952 0.953 ...
##  $ : num [1:28, 1:28, 1:3] 1 1 1 1 1 1 1 1 1 1 ...
##  $ : num [1:28, 1:28, 1:3] 1 1 1 1 1 1 1 1 1 1 ...
##  $ : num [1:28, 1:28, 1:3] 1 1 1 1 1 1 1 1 1 1 ...
##  $ : num [1:28, 1:28, 1:3] 0.752 0.764 0.771 0.832 0.789 ...
##  $ : num [1:28, 1:28, 1:3] 1 1 1 1 1 1 1 1 1 1 ...
##  $ : num [1:28, 1:28, 1:3] 0.13 0.239 0.303 0.261 0.184 ...
##  $ : num [1:28, 1:28, 1:3] 1 1 1 1 1 1 1 1 1 1 ...
##  $ : num [1:28, 1:28, 1:3] 1 1 1 1 1 1 1 1 1 1 ...
##  $ : num [1:28, 1:28, 1:3] 1 1 1 1 1 1 1 1 1 1 ...

We need to combine these 03 rows [1:28, 1:28, 1:3] in 01,

# Row Bind
X_train <- NULL
for (i in 1:8){X_train <- rbind(X_train, mypics[[i]])} # first 8 planes
str(X_train)

##  num [1:8, 1:2352] 0.7299 0.0765 0.4784 0.9937 0.4902 ...

for (i in 11:18){X_train <- rbind(X_train, mypics[[i]])} # first 8 cars
str(X_train)

##  num [1:16, 1:2352] 0.7299 0.0765 0.4784 0.9937 0.4902 ...

X_test <- rbind(mypics[[9]], mypics[[10]], mypics[[19]], mypics[[20]])

str(X_test)

##  num [1:4, 1:2352] 0.453 0.676 1 1 0.457 ...

The response variable are represented by letter 0 and 1 for airplane and car, respectively.

# response variable
y_train <- c(0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1) # 0 <- airplane; 1 <- car
y_test <- c(0,0,1,1)

Creating dummy variable

# One Hot Encoding
trainLabels <- to_categorical(y_train)
testLabels <- to_categorical(y_test)

testLabels

##      [,1] [,2]
## [1,]    1    0
## [2,]    1    0
## [3,]    0    1
## [4,]    0    1

Create a Model

# Model
model <- keras_model_sequential()

model %>%
  layer_dense(units = 256, activation = 'relu', input_shape = c(2352)) %>%
  layer_dense(units = 128, activation = 'relu') %>%
  layer_dense(units = 2, activation = 'softmax')

summary(model)

## ___________________________________________________________________________
## Layer (type)                     Output Shape                  Param #     
## ===========================================================================
## dense_1 (Dense)                  (None, 256)                   602368      
## ___________________________________________________________________________
## dense_2 (Dense)                  (None, 128)                   32896       
## ___________________________________________________________________________
## dense_3 (Dense)                  (None, 2)                     258         
## ===========================================================================
## Total params: 635,522
## Trainable params: 635,522
## Non-trainable params: 0
## ___________________________________________________________________________

Compile

# Compile
model %>%
  compile(loss = 'binary_crossentropy',
          optimizer = optimizer_rmsprop(),
          metrics = 'accuracy')

Fitting the Model

# Fit Model
fitModel <- model %>%
  fit(X_train,
      trainLabels,
      epochs = 30,
      batch_size = 32,
      validation_split = 0.2)

plot(fitModel)

Evaluation & Prediction: train data

# Evaluation & Prediction: train data
model %>% evaluate(X_train, trainLabels)

## $loss
## [1] 8.01512
## 
## $acc
## [1] 0.5

pred <- model %>% predict_classes(X_train)

table(Predicted = pred, Actual = y_train)

##          Actual
## Predicted 0 1
##         0 8 8

prob <- model %>% predict_proba(X_train)

cbind(prob, Predited = pred, Actual = y_train)

##                      Predited Actual
##  [1,] 1 6.821810e-21        0      0
##  [2,] 1 3.032461e-12        0      0
##  [3,] 1 8.329304e-18        0      0
##  [4,] 1 7.279287e-19        0      0
##  [5,] 1 1.178084e-18        0      0
##  [6,] 1 1.482327e-20        0      0
##  [7,] 1 3.028405e-15        0      0
##  [8,] 1 1.598068e-16        0      0
##  [9,] 1 1.206500e-10        0      1
## [10,] 1 2.552770e-16        0      1
## [11,] 1 4.754438e-22        0      1
## [12,] 1 5.840106e-20        0      1
## [13,] 1 4.744663e-11        0      1
## [14,] 1 7.043394e-17        0      1
## [15,] 1 2.520725e-09        0      1
## [16,] 1 8.223308e-21        0      1

display(mypics[[15]]) # misclassified

## Only the first frame of the image stack is displayed.
## To display all frames use 'all = TRUE'.

display(mypics[[17]]) # misclassified

## Only the first frame of the image stack is displayed.
## To display all frames use 'all = TRUE'.

Evaluation & Prediction: test data

# Evaluation & Prediction: test data
#model %>% evaluate(X_train, trainLabels)

pred <- model %>% predict_classes(X_test)

table(Predicted = pred, Actual = y_test)

##          Actual
## Predicted 0 1
##         0 2 2

display(mypics[[10]]) # classified corretly

## Only the first frame of the image stack is displayed.
## To display all frames use 'all = TRUE'.

display(mypics[[20]]) # classified corretly

## Only the first frame of the image stack is displayed.
## To display all frames use 'all = TRUE'.

Image recognition in R using keras - TensorFlow

Reinaldo Zezela; rzezela@gmail.com or r.zezela1@unimail.derby.ac.uk

05 November 2018