Background

Giant Companies like Youtube, Google, Facebook already use machine learning as one of their main tools to run their apps. One of the example is how Netflix or Spotify will give you recommendation based on the links you are clicking, the movies you are watching, what music you are playing and by using machine learning they will try to predict what you might want to watch or listen next.

Machine Learning itself is a process that take an input of the data, detect and find a pattern within it, and produce the output that can be used depends on the desirable goal of the users.

source: technologyreview

Nowadays, there are so many applications of machine learning such as Image Recognition, Virtual Assistant, Product Recommendation and event self driving cars which seems only seen in science fiction movies.

source : javapoint

Although some of these application only applied in several sector such as Health Industries and Tech Securities industries, Machine learning can also be applied even in the daily life basis. In this project, i will create a dashboard powered up by a model that able to classified data into several category labels and provide a recommendation based on the labels.

The model will try to predict the labels of the food image by using Convolutional Neural Network (CNN). The results will be used to provide a recommendation in the form of menu that can be created based on our labels.

Hopefully The output of this project will make use to prevent people to overspending on their food or groceries. These being a serious concern as Indonesia itself is among the country that create the highest food waste in the world. It also to help people to ease they daily life as some people do not have the same amount of knowledge in cooking. This project will also help people who have passion in food and beverages industry to enhance their skills in cooking and contribute to the growth of the industry itself.

For this project i will use a ECUSTFD dataset. This dataset consist of 20 different type of food class. Here are some examples of the data:

Exploratory Data Analysis

The ECUSTFD (ECUST Food Dataset) dataset is included with an annotation that contain information about the bounding box. This bounding box will form a rectangular form around our object and help our model to be more precise in predicting the labels.

Below are our data with their each bounding box information.

Each of the images contains side and top view and taken with smart phones camera. Each of the data consist of a picture with a coin and food in it. We use coin as a calibration object. This dataset consist of 2978 images of 19 different class of foods and ingredients:

Apple Banana Bread Bun Doughnut Egg Fired Dough Twist Grape Lemon Litchi Mango Mooncake Orange Peach Pear Plum Qiwi Sachima Tomato

Below is the summary of our labels, and the amount of data that fall into each category:

Before our data can be put into our model, we need to reprocess our data by creating bounding box around our object. The bounding box data can be accessed through our df_join dataframe. This data frame is a join dataframe between our meta_data list and df dataframe. these data frame will put our food class and the coin class in the same image (as it is in one image there are two different object, a food and a coin).

library(magick)

img <- image_read(df_join$path[1])

img <- image_draw(img)
for (i in 1:2) {
  rect(xleft = df_join$xmin[i],
       ybottom = df_join$ymin[i],
       xright = df_join$xmax[i],
       ytop = df_join$ymax[i],
       border = "white",
       lwd = 2)
  text(
    x = as.integer(df_join$xmax[i]),
    y = as.integer(df_join$ymax[i]),
    labels = df_join$name[i],
    offset = 1,
    pos = 2,
    cex = 3,
    col = "black"
  )
}

Below are the data that already processed and suitable to be put in our model. Our data is an image that contains bounding box surrounding our food object. This bounding box will help our model to predict the labels of our food.

print(img)
# A tibble: 1 x 7
  format width height colorspace matte filesize density
  <chr>  <int>  <int> <chr>      <lgl>    <int> <chr>  
1 JPEG     816    612 sRGB       TRUE         0 72x72

Modelling

One of our project objective is able to recognize the food based on an image. Therefore, we will need a model that suitable for image recognition. We will use a model with CNN architecture. below is a figure that shown how a model with CNN works:

These Convolutional Neural Network is consist of 50 layers deep. consist of 5 different stages each with a convolution and identity block. Each of this convolution and identity block has 3 convolution layers. To add output from previous layer to the next one, ResNet use skip connection which help the model mitigate any vanishing gradient problem. The skip connection also ensures that the next layer will perform at least as good as the low layer.

Below are the diagram that illustrate skip connection:

source : towardsdatascience

Recommendation

Through the models, we are expecting the results in the form of labels of our 19 different food class. We will connecting the labels generated from the model through a url template that will redirect our user to a allrecipes. a website consist of different recipes from foods, drinks, and many more.

Suppose from the models we able to generate a labels egg and lemon. we will save these labels in an object that will contain the prediction results.

# Save the prediction result in to object

result_label <- c("egg", "lemon")

We will create a replaceable string consist of our prediction result collapsed with %20. These string will be joined with our url template. Thus, every generated labels will be placed in our url template and formed a link that will redirect us to a page consist of menus that contain our ingredients.

# Create a replacable string

result_join <- paste0(result_label, collapse = "%20")
# Create a url template

url <- paste0("https://www.allrecipes.com/search/results/?wt=", result_join, "&sort=re&page=1")

With read_html function from rvest library, we will read url and store it in an object called webpage.

library(rvest)

# Read HTML document

webpage <- read_html(url)

Next, we will select nodes from webpage and extract the necessary attributes, text, and tag name from it. These will be resulted in a form of list. consist of our menu title.

# Select Nodes and Extracting attributes

menu_text <- html_nodes(webpage, css = ".fixed-recipe-card__title-link") %>%
  html_text()

We will tidy our list with str_squish() which will be used for trimming the whitespace from a string. Then, we will transform our list into a data frame.

# tidying up the list and transform it into data frame

menu <- menu_text %>%
  str_squish() %>%
  data.frame()
# Change data frame colom names

menu <- menu %>%
  rename(.data = menu, "Menu" = .) 

menu

Noticed on our menu data frame there are duplicate value for each of our menu. The reason are when we selecting nodes, we put the url and menu title into a different list for each different menu.

As we are also going to put the menu url for our data frame, we will try to eliminate the duplicate value by removing the rows that contain the menu title. Thus, left us with a data frame containing the menu and it page url.

First, we will create a new column named Link. This column contained the same information as we selecting nodes from our webpage object.

menu <- menu %>%
  mutate(Link = as.character(html_nodes(webpage, css = ".fixed-recipe-card__title-link")))

menu

From the data frame, we can see that the rows that contain the menu title always placed in the even rows. By simple subsetting we will remove the even rows and overwrite our Menu data frame.

delete <- seq(1, nrow(menu), 2)

# Remove the rows by selecting only the odd rows

menu <- menu[delete,]

# reindex the dataframe

rownames(menu) <- 1:nrow(menu)

menu

Finally, with str_extract() function from stringr library, we will extract the url that will direct us to the website’s menu page. We will simplify our recommendation by limit it into 5.

library(stringr)

url_pattern <- "http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\\(\\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+"

# Extracting URL

menu$Link <- str_extract(menu$Link, url_pattern)

# Limit the recommendation list

menu <- head(menu, 5)

menu

These data frame are the recommendation that we will deliver to the user that derived from the labels generated from the prediction results.