R Notebook

title: “GooglePlayStore-Analysis” author: “Khawla-BanyDOmi” date: “29/12/2020” output: pdf_document: default html_document: default —

#1.Loading Data

gg = read.csv("googleplaystore.csv")
review = read.csv("googleplaystore_user_reviews.csv")
library(e1071)
library(tidyverse)
review1 = review %>% select(App, Translated_Review)
head(review1)

knitr::kable(head(review1))

App	Translated_Review
10 Best Foods for You	I like eat delicious food. That’s I’m cooking food myself, case “10 Best Foods” helps lot, also “Best Before (Shelf Life)”
10 Best Foods for You	This help eating healthy exercise regular basis
10 Best Foods for You	nan
10 Best Foods for You	Works great especially going grocery store
10 Best Foods for You	Best idea us
10 Best Foods for You	Best way

head(review)

head(gg)

#2.Data Preprocessing

str(gg)

'data.frame':   10841 obs. of  13 variables:
 $ App           : Factor w/ 9660 levels "- Free Comics - Comic Apps",..: 7206 2551 8970 8089 7272 7103 8149 5568 4926 5806 ...
 $ Category      : Factor w/ 34 levels "1.9","ART_AND_DESIGN",..: 2 2 2 2 2 2 2 2 2 2 ...
 $ Rating        : num  4.1 3.9 4.7 4.5 4.3 4.4 3.8 4.1 4.4 4.7 ...
 $ Reviews       : Factor w/ 6002 levels "0","1","10","100",..: 1183 5924 5681 1947 5924 1310 1464 3385 816 485 ...
 $ Size          : Factor w/ 462 levels "1,000+","1.0M",..: 55 30 368 102 64 222 55 118 146 120 ...
 $ Installs      : Factor w/ 22 levels "0","0+","1,000,000,000+",..: 8 20 13 16 11 17 17 4 4 8 ...
 $ Type          : Factor w/ 4 levels "0","Free","NaN",..: 2 2 2 2 2 2 2 2 2 2 ...
 $ Price         : Factor w/ 93 levels "$0.99","$1.00",..: 92 92 92 92 92 92 92 92 92 92 ...
 $ Content.Rating: Factor w/ 7 levels "","Adults only 18+",..: 3 3 3 6 3 3 3 3 3 3 ...
 $ Genres        : Factor w/ 120 levels "Action","Action;Action & Adventure",..: 10 13 10 10 12 10 10 10 10 12 ...
 $ Last.Updated  : Factor w/ 1378 levels "1.0.19","April 1, 2016",..: 562 482 117 825 757 901 76 726 1317 670 ...
 $ Current.Ver   : Factor w/ 2834 levels "","0.0.0.2","0.0.1",..: 122 1020 468 2827 280 116 280 2393 1457 1431 ...
 $ Android.Ver   : Factor w/ 35 levels "","1.0 and up",..: 17 17 17 20 22 10 17 20 12 17 ...

There are a lot of factor variables which should actually be converted to numeric variables.

##2.1Converting variable types

library(lubridate)
library(tidyverse)
library(dplyr)
gg.new <- gg %>%
  mutate(
    # Eliminate "+" to transform Installs to numeric variable
   # Installs = gsub("\\+", "", as.character(Installs)),
   # Installs = as.numeric(gsub(",", "", Installs)),
    # Eliminate "M" to transform Size to numeric variable
    Size = gsub("M", "", Size),
    # For cells with k, divide it by 1024, since 1024kB = 1MB, the unit for size is MB
    Size = ifelse(grepl("k", Size),as.numeric(gsub("k", "", Size))/1024, as.numeric(Size)),
    # Transform reviews to numeric
    Reviews = as.numeric(Reviews),
    # Remove "$" from Price to transform it to numeric
    Price = as.numeric(gsub("\\$", "", as.character(Price))),
    # Convert Last Updated to date format
    Last.Updated = mdy(Last.Updated),
    # Replace "Varies with device" to NA since it is unknown
    Min.Android.Ver = gsub("Varies with device", NA, Android.Ver),
    # Keep only version number to 1 decimal as it's most representatice
    Min.Android.Ver = as.numeric(substr(Min.Android.Ver, start = 1, stop = 3)),
    # Drop old Android version column
    Android.Ver = NULL
  ) %>% 
  filter(
    # Two apps had type as 0 or NA, they will be removed 
    Type %in% c("Free", "Paid")
 )

Problem with `mutate()` input `Size`.
ℹ NAs introduced by coercion
ℹ Input `Size` is `ifelse(...)`.NAs introduced by coercionProblem with `mutate()` input `Size`.
ℹ NAs introduced by coercion
ℹ Input `Size` is `ifelse(...)`.NAs introduced by coercionProblem with `mutate()` input `Price`.
ℹ NAs introduced by coercion
ℹ Input `Price` is `as.numeric(gsub("\\$", "", as.character(Price)))`.NAs introduced by coercionProblem with `mutate()` input `Last.Updated`.
ℹ  1 failed to parse.
ℹ Input `Last.Updated` is `mdy(Last.Updated)`. 1 failed to parse.

str(gg.new)

'data.frame':   10839 obs. of  13 variables:
 $ App            : Factor w/ 9660 levels "- Free Comics - Comic Apps",..: 7206 2551 8970 8089 7272 7103 8149 5568 4926 5806 ...
 $ Category       : Factor w/ 34 levels "1.9","ART_AND_DESIGN",..: 2 2 2 2 2 2 2 2 2 2 ...
 $ Rating         : num  4.1 3.9 4.7 4.5 4.3 4.4 3.8 4.1 4.4 4.7 ...
 $ Reviews        : num  1183 5924 5681 1947 5924 ...
 $ Size           : num  19 14 8.7 25 2.8 5.6 19 29 33 3.1 ...
 $ Installs       : Factor w/ 22 levels "0","0+","1,000,000,000+",..: 8 20 13 16 11 17 17 4 4 8 ...
 $ Type           : Factor w/ 4 levels "0","Free","NaN",..: 2 2 2 2 2 2 2 2 2 2 ...
 $ Price          : num  0 0 0 0 0 0 0 0 0 0 ...
 $ Content.Rating : Factor w/ 7 levels "","Adults only 18+",..: 3 3 3 6 3 3 3 3 3 3 ...
 $ Genres         : Factor w/ 120 levels "Action","Action;Action & Adventure",..: 10 13 10 10 12 10 10 10 10 12 ...
 $ Last.Updated   : Date, format: "2018-01-07" "2018-01-15" ...
 $ Current.Ver    : Factor w/ 2834 levels "","0.0.0.2","0.0.1",..: 122 1020 468 2827 280 116 280 2393 1457 1431 ...
 $ Min.Android.Ver: num  4 4 4 4.2 4.4 2.3 4 4.2 3 4 ...

options(scipen=999)
table(gg.new$Installs)

             0             0+ 1,000,000,000+     1,000,000+         1,000+ 
             0             14             58           1579            907 
            1+    10,000,000+        10,000+            10+   100,000,000+ 
            67           1252           1054            386            409 
      100,000+           100+     5,000,000+         5,000+             5+ 
          1169            719            752            477             82 
   50,000,000+        50,000+            50+   500,000,000+       500,000+ 
           289            479            205             72            539 
          500+           Free 
           330              0 

gg.new$Installs%>%str()

 Factor w/ 22 levels "0","0+","1,000,000,000+",..: 8 20 13 16 11 17 17 4 4 8 ...

gg.new %>% filter(Installs == "500,000") %>% print

library(highcharter)
gg.new %>% select(-Min.Android.Ver) %>% 
    summarise_all(
        funs(sum(is.na(.)))
    ) %>%
  gather() %>%
  # Only show columns with NA
  filter(value> 1) %>%
  arrange(-value) %>%
    hchart('column', hcaes(x = 'key', y = 'value', color = 'key')) %>%
  hc_add_theme(hc_theme_elementary()) %>%
  hc_title(text = "Columns with Missing Value")

Most popular category

gg.new1 <- gg %>%
  mutate(
    # Eliminate "+" to transform Installs to numeric variable
    Installs = gsub("\\+", "", as.character(Installs)),
    Installs = as.numeric(gsub(",", "", Installs)),
    # Eliminate "M" to transform Size to numeric variable
    Size = gsub("M", "", Size),
    # For cells with k, divide it by 1024, since 1024kB = 1MB, the unit for size is MB
    Size = ifelse(grepl("k", Size),as.numeric(gsub("k", "", Size))/1024, as.numeric(Size)),
    # Transform reviews to numeric
    Reviews = as.numeric(Reviews),
    # Remove "$" from Price to transform it to numeric
    Price = as.numeric(gsub("\\$", "", as.character(Price))),
    # Convert Last Updated to date format
    Last.Updated = mdy(Last.Updated),
    # Replace "Varies with device" to NA since it is unknown
    Min.Android.Ver = gsub("Varies with device", NA, Android.Ver),
    # Keep only version number to 1 decimal as it's most representatice
    Min.Android.Ver = as.numeric(substr(Min.Android.Ver, start = 1, stop = 3)),
    # Drop old Android version column
    Android.Ver = NULL
  )

Problem with `mutate()` input `Installs`.
ℹ NAs introduced by coercion
ℹ Input `Installs` is `as.numeric(gsub(",", "", Installs))`.NAs introduced by coercionProblem with `mutate()` input `Size`.
ℹ NAs introduced by coercion
ℹ Input `Size` is `ifelse(...)`.NAs introduced by coercionProblem with `mutate()` input `Size`.
ℹ NAs introduced by coercion
ℹ Input `Size` is `ifelse(...)`.NAs introduced by coercionProblem with `mutate()` input `Price`.
ℹ NAs introduced by coercion
ℹ Input `Price` is `as.numeric(gsub("\\$", "", as.character(Price)))`.NAs introduced by coercionProblem with `mutate()` input `Last.Updated`.
ℹ  1 failed to parse.
ℹ Input `Last.Updated` is `mdy(Last.Updated)`. 1 failed to parse.

gg.new2 = gg.new1 %>% mutate(Interval = difftime(time1 = today(), time2 = Last.Updated)) %>% print

ggplot(gg.new2) + geom_line(aes(x = Interval, y = Installs)) + labs(x = "Days Since Last Update", y = "Installments")

gg.new1 %>% 
  group_by(Category) %>% filter(Category != 1.9) %>% 
  summarize(
    TotalInstalls = sum(as.numeric(Installs))
  ) %>%
  arrange(-TotalInstalls) %>%
  hchart('scatter', hcaes(x = "Category", y = "TotalInstalls", size = "TotalInstalls", color = "Category")) %>%
  hc_add_theme(hc_theme_538()) %>%
  hc_title(text = "Most popular categories")

`summarise()` ungrouping output (override with `.groups` argument)

###Correlation map

head(iris)

library(reshape2)
df_cor = iris[,2:3]
cormat <- round(cor(df_cor),2) 
melted_cormat <- melt(cormat)
ggplot(data = melted_cormat, aes(Var2, Var1, fill = value))+
 geom_tile(color = "white")+
 scale_fill_gradient2(low = "yellow", high = "purple", mid = "red",
   midpoint = 0, limit = c(-1,1), space = "Lab",
   name="Pearson\nCorrelation") +
  theme_minimal()+
 theme(axis.text.x = element_text(angle = 45, vjust = 1,
    size = 12, hjust = 1))+
 coord_fixed()

##2.2Divide Installs into 3 categories

library(tidyverse)
options(scipen=999)
# write function to convert installment
convert_install = function(data, installment) {
  #install.levels = factor(c("low", "medium", "high"))
  
  if (installment %in% c("0", "1", "50", "100", "500", "1,000", "5,000", "10,000", "50,000")) {
  Installs.cat = "low"
  }
  else if (installment %in% c ("100,000", "500,000", "1,000,000", "5,000,000")){
    Installs.cat = "medium"
  }
  else {
      Installs.cat = "high"
  }
}
#gg.new = gg.new %>% filter(!is.na(Installs)) %>% mutate(Installs.cat = factor(convert_install(gg.new, Installs), # levels = c("low", "medium", "high")))
sum((gg.new$Installs) %in% "10,000")

[1] 0

# gg.new = gg.new %>% mutate(Installs.cat = "1")
str(gg.new)

'data.frame':   10839 obs. of  13 variables:
 $ App            : Factor w/ 9660 levels "- Free Comics - Comic Apps",..: 7206 2551 8970 8089 7272 7103 8149 5568 4926 5806 ...
 $ Category       : Factor w/ 34 levels "1.9","ART_AND_DESIGN",..: 2 2 2 2 2 2 2 2 2 2 ...
 $ Rating         : num  4.1 3.9 4.7 4.5 4.3 4.4 3.8 4.1 4.4 4.7 ...
 $ Reviews        : num  1183 5924 5681 1947 5924 ...
 $ Size           : num  19 14 8.7 25 2.8 5.6 19 29 33 3.1 ...
 $ Installs       : Factor w/ 22 levels "0","0+","1,000,000,000+",..: 8 20 13 16 11 17 17 4 4 8 ...
 $ Type           : Factor w/ 4 levels "0","Free","NaN",..: 2 2 2 2 2 2 2 2 2 2 ...
 $ Price          : num  0 0 0 0 0 0 0 0 0 0 ...
 $ Content.Rating : Factor w/ 7 levels "","Adults only 18+",..: 3 3 3 6 3 3 3 3 3 3 ...
 $ Genres         : Factor w/ 120 levels "Action","Action;Action & Adventure",..: 10 13 10 10 12 10 10 10 10 12 ...
 $ Last.Updated   : Date, format: "2018-01-07" "2018-01-15" ...
 $ Current.Ver    : Factor w/ 2834 levels "","0.0.0.2","0.0.1",..: 122 1020 468 2827 280 116 280 2393 1457 1431 ...
 $ Min.Android.Ver: num  4 4 4 4.2 4.4 2.3 4 4.2 3 4 ...

table(gg.new$Installs)

             0             0+ 1,000,000,000+     1,000,000+         1,000+ 
             0             14             58           1579            907 
            1+    10,000,000+        10,000+            10+   100,000,000+ 
            67           1252           1054            386            409 
      100,000+           100+     5,000,000+         5,000+             5+ 
          1169            719            752            477             82 
   50,000,000+        50,000+            50+   500,000,000+       500,000+ 
           289            479            205             72            539 
          500+           Free 
           330              0 

table(gg.new$Installs.cat)

< table of extent 0 >

gg.new = gg.new %>% filter(Installs != "Free") %>% mutate(
  Installs.cat = fct_collapse(Installs, 
                              low = c("Free","0", "0+","1+", "5+", "10+","100+", "50+", "100+", "500+", "1,000+", "5,000+"), 
                              medium = c("10,000+", "50,000+", "100,000+", "500,000+"), 
                              high = c("1,000,000+", "5,000,000+", "1,000,000,000+", "10,000,000+", "100,000,000+", "50,000,000+", "500,000,000+")))
table(gg.new$Installs.cat)

   low   high medium 
  3187   4411   3241 

boxplot of different Installment categories

ggplot(data = gg.new) +
  geom_boxplot(aes(x = reorder(Installs.cat, -Rating), y = Rating)) + 
  labs(x = "Installment Categories",y = "Rating")

##2.3 Delete duplicated rows

# number of observations before deleting duplicated rows
(original_num_rows = nrow(gg.new))

[1] 10839

gg.new.uniq = gg.new %>% distinct
# number of rows after delete duplicated rows
(uniq_num_rows = nrow(gg.new.uniq))

[1] 10356

# number of duplicated rows
(dup_rows = original_num_rows - uniq_num_rows)

[1] 483

##2.4 Merge Category into 6

# gg.new.uniq %>% filter (!is.na(Category)) %>% print
levels(gg.new.uniq$Category)

 [1] "1.9"                 "ART_AND_DESIGN"      "AUTO_AND_VEHICLES"  
 [4] "BEAUTY"              "BOOKS_AND_REFERENCE" "BUSINESS"           
 [7] "COMICS"              "COMMUNICATION"       "DATING"             
[10] "EDUCATION"           "ENTERTAINMENT"       "EVENTS"             
[13] "FAMILY"              "FINANCE"             "FOOD_AND_DRINK"     
[16] "GAME"                "HEALTH_AND_FITNESS"  "HOUSE_AND_HOME"     
[19] "LIBRARIES_AND_DEMO"  "LIFESTYLE"           "MAPS_AND_NAVIGATION"
[22] "MEDICAL"             "NEWS_AND_MAGAZINES"  "PARENTING"          
[25] "PERSONALIZATION"     "PHOTOGRAPHY"         "PRODUCTIVITY"       
[28] "SHOPPING"            "SOCIAL"              "SPORTS"             
[31] "TOOLS"               "TRAVEL_AND_LOCAL"    "VIDEO_PLAYERS"      
[34] "WEATHER"            

mydata1 = gg.new.uniq %>% filter(Category != 1.9) %>% mutate(Cat.cat = fct_collapse(Category,
                                                        Education = c("EDUCATION", "BOOKS_AND_REFERENCE", "LIBRARIES_AND_DEMO", "ART_AND_DESIGN"),
                                                        Personalization = c("PERSONALIZATION", "BEAUTY", "SHOPPING", "DATING", "PHOTOGRAPHY"),
                                                        Lifestyle = c("HEALTH_AND_FITNESS", "MEDICAL", "LIFESTYLE", "SPORTS", "FOOD_AND_DRINK"),
                                                        Family = c("FAMILY", "PARENTING", "HOUSE_AND_HOME", "1.9"),
                                                        Entertainment = c("ENTERTAINMENT", "GAME", "COMICS", "VIDEO_PLAYERS"), 
                                                        Business = c("BUSINESS", "FINANCE", "PRODUCTIVITY", "TOOLS", "NEWS_AND_MAGAZINES", "EVENTS", "SOCIAL", "COMMUNICATION"),
                                                        Travel = c("MAPS_AND_NAVIGATION", "AUTO_AND_VEHICLES", "TRAVEL_AND_LOCAL", "WEATHER")))

mydata2 = mydata1 %>% mutate(Interval = difftime(time1 = today(), time2 = Last.Updated))
str(mydata2)

'data.frame':   10356 obs. of  16 variables:
 $ App            : Factor w/ 9660 levels "- Free Comics - Comic Apps",..: 7206 2551 8970 8089 7272 7103 8149 5568 4926 5806 ...
 $ Category       : Factor w/ 34 levels "1.9","ART_AND_DESIGN",..: 2 2 2 2 2 2 2 2 2 2 ...
 $ Rating         : num  4.1 3.9 4.7 4.5 4.3 4.4 3.8 4.1 4.4 4.7 ...
 $ Reviews        : num  1183 5924 5681 1947 5924 ...
 $ Size           : num  19 14 8.7 25 2.8 5.6 19 29 33 3.1 ...
 $ Installs       : Factor w/ 22 levels "0","0+","1,000,000,000+",..: 8 20 13 16 11 17 17 4 4 8 ...
 $ Type           : Factor w/ 4 levels "0","Free","NaN",..: 2 2 2 2 2 2 2 2 2 2 ...
 $ Price          : num  0 0 0 0 0 0 0 0 0 0 ...
 $ Content.Rating : Factor w/ 7 levels "","Adults only 18+",..: 3 3 3 6 3 3 3 3 3 3 ...
 $ Genres         : Factor w/ 120 levels "Action","Action;Action & Adventure",..: 10 13 10 10 12 10 10 10 10 12 ...
 $ Last.Updated   : Date, format: "2018-01-07" "2018-01-15" ...
 $ Current.Ver    : Factor w/ 2834 levels "","0.0.0.2","0.0.1",..: 122 1020 468 2827 280 116 280 2393 1457 1431 ...
 $ Min.Android.Ver: num  4 4 4 4.2 4.4 2.3 4 4.2 3 4 ...
 $ Installs.cat   : Factor w/ 3 levels "low","high","medium": 3 3 2 2 3 3 3 2 2 3 ...
 $ Cat.cat        : Factor w/ 7 levels "Family","Education",..: 2 2 2 2 2 2 2 2 2 2 ...
 $ Interval       : 'difftime' num  1089 1081 883 937 ...
  ..- attr(*, "units")= chr "days"

mydata2 %>% filter(Installs.cat == "low") %>% print

Impute missing values

#missForest
library(missForest)
#impute missing values, using all parameters as default values
gg.new.imp <- missForest(data.matrix(mydata2), maxiter = 5, ntree = 10)

  missForest iteration 1 in progress...done!
  missForest iteration 2 in progress...done!
  missForest iteration 3 in progress...done!
  missForest iteration 4 in progress...done!
  missForest iteration 5 in progress...done!

#check imputed values
# gg.new.imp$ximp
#check imputation error
gg.new.imp$OOBerror

      NRMSE 
0.001104393 

get the semantic score

# install.packages("stringr")
# install.packages("tidytext")
library(stringr)
library(tidytext)

# read in user reviews
user_review = read.csv("googleplaystore_user_reviews.csv")
str(user_review)

'data.frame':   64295 obs. of  5 variables:
 $ App                   : Factor w/ 1074 levels "10 Best Foods for You",..: 1 1 1 1 1 1 1 1 1 1 ...
 $ Translated_Review     : Factor w/ 27996 levels "","___ ___ ___ ___ ___ 0",..: 9279 23853 17229 27355 2076 2168 1032 17229 15968 13280 ...
 $ Sentiment             : Factor w/ 4 levels "nan","Negative",..: 4 4 1 4 4 4 4 1 3 3 ...
 $ Sentiment_Polarity    : num  1 0.25 NaN 0.4 1 1 0.6 NaN 0 0 ...
 $ Sentiment_Subjectivity: num  0.533 0.288 NaN 0.875 0.3 ...

user_review %>% print

head(user_review)

# get sentiment data frame
sents = get_sentiments("afinn") %>% print

range(sents$score)

Unknown or uninitialised column: `score`.no non-missing arguments to min; returning Infno non-missing arguments to max; returning -Inf

[1]  Inf -Inf

# left join the sentiment chart and the user reviews to get score
t1 = user_review %>% mutate(review = as.character(Translated_Review)) %>% unnest_tokens(word, review)
# t2 = user_review[1:500, ]
user_score = left_join(t1, sents) %>% group_by(App) %>% summarise(n = n(), score=sum(t1$score, na.rm=T)) %>% mutate(avg.score = score / n) %>% print

Joining, by = "word"
`summarise()` ungrouping output (override with `.groups` argument)

# range(user_score $ avg.score)

user_review %>% group_by(App) %>% count

t11 = user_score %>% inner_join(gg.new) %>% filter(Installs != 5000) %>% filter(Installs != 1000000000)

Joining, by = "App"

ggplot(t11) + geom_line(aes(x = Installs, y = avg.score))

ggplot(t11) + geom_boxplot(aes(x = reorder(as.factor(Installs), -avg.score), y = avg.score)) + labs(x = "Installments", y = "Average Score") + coord_flip()

# recover app name after data imputation
# add num_row to gg.new
mydata2 = mydata2 %>% mutate(r = row_number()) 
# split data into training and test data
# change the list to data frame 
gg.df = gg.new.imp[[1]] %>% unlist()
gg.data = data.frame(gg.df) %>% mutate(r = row_number()) 
t1 = left_join(gg.data, mydata2, by = "r") %>% 
  select(Rating.x, Reviews.y, Size.x, Installs.cat.y, Price.y, Content.Rating.y, Cat.cat.y, Interval.y) %>% print

# split data
(total_row = nrow(t1))

[1] 10356

ins.l= which(t1$Installs.cat.y == "low")
ins.m= which(t1$Installs.cat.y == "medium")
ins.h= which(t1$Installs.cat.y == "high")
train.id = c(sample(ins.l, size = trunc(0.8 *length(ins.l))),
             sample(ins.m, size = trunc(0.8 *length(ins.m))), 
             sample(ins.h, size = trunc(0.8 *length(ins.h))))
train.gg = t1[train.id, ]
test.gg = t1[-train.id, ]
levels(train.gg$`Installs`)

[1] "low"    "high"   "medium"

table(train.gg$`Installs`)

   low   high medium 
  2519   3243   2522 

# random forest
set.seed(415)
library(randomForest)
table(factor(train.gg$Installs.cat.y))

   low   high medium 
  2519   3243   2522 

bag.gg=randomForest(Installs.cat.y~., data=train.gg, mtry = ncol(train.gg) - 1,importance=TRUE)
bag.gg

Call:
 randomForest(formula = Installs.cat.y ~ ., data = train.gg, mtry = ncol(train.gg) -      1, importance = TRUE) 
               Type of random forest: classification
                     Number of trees: 500
No. of variables tried at each split: 7

        OOB estimate of  error rate: 35.01%
Confusion matrix:
        low high medium class.error
low    1623  282    614   0.3556967
high    118 2505    620   0.2275671
medium  409  857   1256   0.5019826

# plot
yhat.bag = predict(bag.gg, newdata=test.gg) 
# test error
(forest.test.err = mean(yhat.bag != test.gg$Installs.cat.y))

[1] 0.3397683

# get the importance
importance(bag.gg)

                       low      high   medium MeanDecreaseAccuracy MeanDecreaseGini
Rating.x          83.41493 145.52376 42.23615            157.29426         919.4855
Reviews.y        171.97873 124.97380 53.34318            191.67516        1621.0679
Size.x            41.47474 149.90416 15.38727            126.17961        1078.8905
Price.y           65.55221 126.03555 31.48082            125.50710         163.8186
Content.Rating.y  16.06423  19.74333 11.55403             27.17451         131.4630
Cat.cat.y         18.65104  78.45468 18.60626             75.61766         357.2444
Interval.y        38.97942 150.75211 15.99744            125.52877        1207.8295

varImpPlot(bag.gg)

# tree
set.seed(415)
library(tree)
#train.gg
#colnames(train.gg)[1] = "Rating"
#colnames(train.gg)[2] = "Reviews"
#colnames(train.gg)[3] = "Size"
#colnames(train.gg)[5] = "Price"
#colnames(train.gg)[6] = "Content Rating"
#colnames(train.gg)[7] = "Category"
#colnames(train.gg)[1] = "Time Since Last Update"
#train.gg
train.gg

tree.gg = tree(Installs.cat.y~., data = train.gg)

NAs introduced by coercion

summary(tree.gg)

Classification tree:
tree(formula = Installs.cat.y ~ ., data = train.gg)
Variables actually used in tree construction:
[1] "Reviews.y" "Size.x"    "Rating.x"  "Price.y"  
Number of terminal nodes:  8 
Residual mean deviance:  1.688 = 13970 / 8276 
Misclassification error rate: 0.4013 = 3324 / 8284 

plot(tree.gg)
text(tree.gg, pretty = 1, cex = 1)

yhat.tree = predict(tree.gg, newdata=test.gg) 

NAs introduced by coercion

# test error
(tree.test.err = mean(yhat.tree != test.gg$Installs.cat.y))

[1] 1

# prune the tree
cv.gg.tree=cv.tree(tree.gg,FUN=prune.misclass)

NAs introduced by coercionNAs introduced by coercionNAs introduced by coercionNAs introduced by coercionNAs introduced by coercionNAs introduced by coercionNAs introduced by coercionNAs introduced by coercionNAs introduced by coercionNAs introduced by coercionNAs introduced by coercionNAs introduced by coercionNAs introduced by coercionNAs introduced by coercionNAs introduced by coercionNAs introduced by coercionNAs introduced by coercionNAs introduced by coercionNAs introduced by coercionNAs introduced by coercion

cv.gg.tree

$size
[1] 8 7 6 5 4 3 2 1

$dev
[1] 3391 3456 3674 3728 3768 3850 4317 5041

$k
[1] -Inf   67   93  100  109  138  486  724

$method
[1] "misclass"

attr(,"class")
[1] "prune"         "tree.sequence"

# par(mfrow=c(1,2))
# plot(cv.gg.tree$size,cv.gg.tree$dev / length(train.gg),ylab="cv error", xlab="size",type="b")
# plot(cv.gg.tree$k, cv.gg.tree$dev / length(train.gg),ylab="cv error", xlab="k",type="b")
# predict using pruning tree
prune.tree=prune.misclass(tree.gg,best=8)
tree.pred=predict(prune.tree, test.gg,type="class")

NAs introduced by coercion

table(tree.pred, test.gg$Installs.cat.y)

         
tree.pred low high medium
   low    305   10     64
   high    68  545    176
   medium 257  256    391

(test.tree.err = mean(tree.pred != test.gg$Installs.cat.y)) 

[1] 0.4010618

# plot the tree
plot(prune.tree)
text(prune.tree, pretty = 0, cex = 1)

As we can see in both single tree and random forest, reviews is the most important predictor. When we dig into the reviews, we figure out that approxiamtely 1000 apps have more than 100 relevant text reviews / comments.

SVM on traning set

set.seed(415)
# get data frame ready to use
train.gg

table(factor(train.gg$Installs.cat.y))

   low   high medium 
  2519   3243   2522 

costVals = c(1, 5, 10, 50)
# linear kernel
# running too slow, be careful to change predictors
svm1 <- tune(svm, as.factor(Installs.cat.y) ~ ., data = train.gg,
             kernel = "linear",
             ranges = list("cost" = costVals)) 
summary(svm1)

Parameter tuning of ‘svm’:

- sampling method: 10-fold cross validation 

- best parameters:

- best performance: 0.4470082 

- Detailed performance results:

# find the best cost under linear kernel
best_mod_linear = svm1$best.model
summary(best_mod_linear)

Call:
best.tune(method = svm, train.x = as.factor(Installs.cat.y) ~ ., data = train.gg, 
    ranges = list(cost = costVals), kernel = "linear")


Parameters:
   SVM-Type:  C-classification 
 SVM-Kernel:  linear 
       cost:  10 

Number of Support Vectors:  6894

 ( 2211 2447 2236 )


Number of Classes:  3 

Levels: 
 low high medium

# thus the cost of the best model si 50.

# get the test error of the best model of the linear kernel
test.gg %>% str()

'data.frame':   2072 obs. of  8 variables:
 $ Rating.x        : num  4.1 4.3 4.7 4.7 4.7 4.2 4.4 3.8 4.7 4.2 ...
 $ Reviews.y       : num  1183 5924 485 443 3284 ...
 $ Size.x          : num  19 2.8 3.1 23 4.2 ...
 $ Installs.cat.y  : Factor w/ 3 levels "low","high","medium": 3 3 3 3 3 3 2 3 2 3 ...
 $ Price.y         : num  0 0 0 0 0 0 0 0 0 0 ...
 $ Content.Rating.y: Factor w/ 7 levels "","Adults only 18+",..: 3 3 3 3 6 3 3 3 3 3 ...
 $ Cat.cat.y       : Factor w/ 7 levels "Family","Education",..: 2 2 2 2 2 2 2 2 2 2 ...
 $ Interval.y      : 'difftime' num  1089 925 912 908 ...
  ..- attr(*, "units")= chr "days"

pred_test_linear = predict(best_mod_linear, newdata = test.gg)
table(predict = pred_test_linear, truth = test.gg$Installs.cat.y)

        truth
predict  low high medium
  low    351  105    225
  high   144  622    226
  medium 135   84    180

(test_err_linear = mean(pred_test_linear != test.gg$Installs.cat.y))

[1] 0.4435328

set.seed(415)
# kernel radial
gammaVals = c(1, 2, 3, 4)
svm_radial <-tune(svm, as.factor(Installs.cat.y) ~ ., data = train.gg, 
                  kernel = "radial",
                  cost = 100,
                               gamma =gammaVals)
summary(svm_radial)

Error estimation of ‘svm’ using 10-fold cross validation: 0.4412131

best_mod_radial = svm_radial$best.model
summary(best_mod_radial)

Call:
best.tune(method = svm, train.x = as.factor(Installs.cat.y) ~ ., data = train.gg, 
    kernel = "radial", cost = 100, gamma = gammaVals)


Parameters:
   SVM-Type:  C-classification 
 SVM-Kernel:  radial 
       cost:  100 

Number of Support Vectors:  6009

 ( 1772 2206 2031 )


Number of Classes:  3 

Levels: 
 low high medium

# get test error of kernel of the radial
pred_test_radial = predict(best_mod_radial, newdata = test.gg)
(test_err_radial = mean(pred_test_radial != test.gg$Installs.cat.y))

[1] 0.4377413

Is it true that people tends to give text review when they highly positively review the app?

# left join the user_score table and t3
mydata2 = mydata2 %>% mutate(r = row_number()) %>% print 

gg.df = gg.new.imp[[1]] %>% unlist()
gg.data = data.frame(gg.df) %>% mutate(r = row_number()) %>% print

t3 = left_join(gg.data, mydata2, by = "r") %>% 
  select(Rating.x, Reviews.y, App.y, Installs.cat.y) %>% print

colnames(t3)[3] = "App"
t2 = inner_join(user_score, t3, by = "App") %>% print

# raing and avg score
# add main title manually, which is "rating vs aaverage sentimental score"
ggplot(data = t2, aes(x = Rating.x, y = avg.score)) + geom_bar(stat = "identity") + labs(x = "Rating", y = "Average Sentimental Score", title = "Rating vs Average sentimental Score") 

ggplot(data = t2, aes(x = as.factor(Installs.cat.y), y = avg.score)) + geom_boxplot() + labs(x = "Installment Category", y = "Average Sentimental Score")

#boxplot(t2$Installs.cat.y ~ t2$avg.score)
# rating vs reviews
ggplot(data = t2, aes(x = Reviews.y, y = avg.score)) + geom_bar(stat = "identity") + labs(x = "Number of #Reviews", y = "Average Sentimental Score", title = "Number of Reviews vs Average sentimental Score") 

High avg score tends to concentrated at rating above and including 4.0

data frame that might not be used

final1 = left_join(gg.data, mydata2, by = "r") %>% select(App.y, Reviews.y, Rating.x, Interval.y, Size.x, Price.y, Cat.cat.y, Content.Rating.y) %>% print

colnames(final1)[1] = "App"
colnames(final1)[2] = "Reviews"
colnames(final1)[3] = "Rating"
colnames(final1)[4] = "Interval"
colnames(final1)[5] = "Size"
colnames(final1)[6] = "Price"
colnames(final1)[7] = "Category"
colnames(final1)[8] = "Content"
show((final1))

plot(final1)