Final project_analysis movie review data
Seonhyang Ji
2024.06.15
Executive summary
Most people leave a review after watching a movie. If they were satisfied with the film, they leave a positive review; if they were dissatisfied, they leave a negative one. People who read these reviews can decide whether to watch the movie or not. Additionally, filmmakers can learn which aspects of their movie were perceived positively or negatively through these reviews. For these reasons, I chose movie reviews as my topic.
My first research question is to understand the differences between positive and negative movie reviews. Specifically, I aim to identify the emotions expressed in positive reviews and those in negative reviews.
Secondly, I want to examine which aspects people primarily discuss when evaluating a movie. To achieve this, I will identify the words frequently used when evaluating a movie. Additionally, I will go beyond examining individual words by looking at words used in succession to understand their meanings more accurately and to grasp how the relationships between words are formed overall.
Consequently, I created bar graphs and word clouds using the Bing sentiment lexicon. Through this, I was able to identify the positive and negative emotions expressed in movie reviews. However, individual words alone did not provide enough context to understand how they were used. Therefore, I created a network graph using bigrams, which helped me understand which words were used in succession with the emotional words identified in the Bing graph. Overall, I was able to determine the positive and negative emotions present in movie reviews and aimed to identify which aspects people primarily discuss when writing reviews by examining frequently used words.
Data background
Explain where the data came from, what agency or company made it, how it is structured, what it shows, etc.
I used the IMDB dataset, which contains 50,000 movie reviews for natural language processing and text analytics. This dataset includes 25,000 highly polar movie reviews for training and 25,000 for testing, each categorized into positive or negative sentiment. I obtained this data from Lakshmipathi N, an expert in datasets on Kaggle.
1.Data loading, cleaning and preprocessing
1-1. Loading data
Using ‘read_csv’ function, I loaded the IMDB data from a CSV file. Additionally, I added a unique Id to each row, which is useful for referencing specific review later.
review_data <- read_csv("IMDB Dataset.csv") %>%
rowid_to_column(var = "Id")## Rows: 50000 Columns: 2
## ── Column specification ────────────────────────────────────────────────────────
## Delimiter: ","
## chr (2): review, sentiment
##
## ℹ Use `spec()` to retrieve the full column specification for this data.
## ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.review_data## # A tibble: 50,000 × 3
## Id review sentiment
## <int> <chr> <chr>
## 1 1 "One of the other reviewers has mentioned that after watchin… positive
## 2 2 "A wonderful little production. <br /><br />The filming tech… positive
## 3 3 "I thought this was a wonderful way to spend time on a too h… positive
## 4 4 "Basically there's a family where a little boy (Jake) thinks… negative
## 5 5 "Petter Mattei's \"Love in the Time of Money\" is a visually… positive
## 6 6 "Probably my all-time favorite movie, a story of selflessnes… positive
## 7 7 "I sure would like to see a resurrection of a up dated Seahu… positive
## 8 8 "This show was an amazing, fresh & innovative idea in the 70… negative
## 9 9 "Encouraged by the positive comments about this film on here… negative
## 10 10 "If you like original gut wrenching laughter you will like t… positive
## # ℹ 49,990 more rows1-2. Tokenization and removing stop words
To split each review into individual words, I used the ‘unnest_tokens()’. I also removed special characters like “br” and stop words for clearer analysis.
review_tidy <- review_data %>%
unnest_tokens(input = review, output = word) %>%
filter(!word %in% c("br")) %>%
anti_join(stop_words)## Joining with `by = join_by(word)`review_tidy## # A tibble: 4,399,678 × 3
## Id sentiment word
## <int> <chr> <chr>
## 1 1 positive reviewers
## 2 1 positive mentioned
## 3 1 positive watching
## 4 1 positive 1
## 5 1 positive oz
## 6 1 positive episode
## 7 1 positive hooked
## 8 1 positive happened
## 9 1 positive struck
## 10 1 positive oz
## # ℹ 4,399,668 more rows2.Text data analysis
2-1. Calculating word frequency in positive and negative review
Before creating bing sentiment graph, I wanted to check the frequent word both positive and negative reviews. Using the ‘count()’ function, I calculated the frequency of each word by sentiment. To see only the top 10 words in positive or negative reviews, I used the ‘filter()’ and ‘head()’ functions.
review_tidy %>%
count(word, sentiment, sort = TRUE) %>%
filter(sentiment == "positive") %>%
head(10)## # A tibble: 10 × 3
## word sentiment n
## <chr> <chr> <int>
## 1 film positive 40946
## 2 movie positive 37373
## 3 story positive 12857
## 4 time positive 12699
## 5 love positive 8670
## 6 people positive 8543
## 7 life positive 8042
## 8 films positive 7579
## 9 characters positive 7087
## 10 movies positive 6973review_tidy %>%
count(word, sentiment, sort = TRUE) %>%
filter(sentiment == "negative") %>%
head(10)## # A tibble: 10 × 3
## word sentiment n
## <chr> <chr> <int>
## 1 movie negative 49582
## 2 film negative 36662
## 3 bad negative 14707
## 4 time negative 12331
## 5 story negative 10134
## 6 people negative 9307
## 7 movies negative 8289
## 8 plot negative 8188
## 9 acting negative 8070
## 10 characters negative 73422-2. Using the bing sentiment lexicon
The bing sentiment lexicon classifies words as positive or negative. I initially checked frequently used words, but then used the bing sentiment lexicon to focus on the sentiment words specifically. This helps to understand the emotional content of the movie reviews and how reviewers felt about the movies.
First, I retrieved the Bing sentiment lexicon and merged the review data with it to add sentiment labels to each word. Then, using the count() function, I calculated the frequency of each word by sentiment.
Second, to select the top 10 most frequent positive or negative words according to the Bing lexicon, I used the filter() function.
#Top10 positive and negative words using bing lexicon
bing <- get_sentiments("bing")
bing_review_data <- review_tidy %>%
inner_join(get_sentiments("bing")) %>%
count(word, sentiment, sort = TRUE)## Joining with `by = join_by(sentiment, word)`bing_review_data## # A tibble: 5,451 × 3
## word sentiment n
## <chr> <chr> <int>
## 1 bad negative 14707
## 2 love positive 8670
## 3 plot negative 8188
## 4 worst negative 4884
## 5 funny negative 4636
## 6 excellent positive 3350
## 7 fun positive 3309
## 8 awful negative 3133
## 9 beautiful positive 3040
## 10 poor negative 3022
## # ℹ 5,441 more rowsbing_review_data %>%
filter(sentiment == "positive") %>%
head(10)## # A tibble: 10 × 3
## word sentiment n
## <chr> <chr> <int>
## 1 love positive 8670
## 2 excellent positive 3350
## 3 fun positive 3309
## 4 beautiful positive 3040
## 5 pretty positive 2991
## 6 wonderful positive 2660
## 7 worth positive 2490
## 8 perfect positive 2424
## 9 classic positive 2345
## 10 loved positive 2239bing_review_data %>%
filter(sentiment == "negative") %>%
head(10)## # A tibble: 10 × 3
## word sentiment n
## <chr> <chr> <int>
## 1 bad negative 14707
## 2 plot negative 8188
## 3 worst negative 4884
## 4 funny negative 4636
## 5 awful negative 3133
## 6 poor negative 3022
## 7 boring negative 2984
## 8 stupid negative 2956
## 9 terrible negative 2897
## 10 hard negative 27642-3. Creating Bigram
I obtained frequent sentiment words data from the Bing lexicon, but I was unable to understand the full context of the review text. By examining words used together rather than individually, I can better understand their context. This is important because even the same word can change or create new meanings depending on the words it is paired with. Therefore, I wanted to analyze words that frequently appear together using bigrams. This approach can provide insights and highlight important phrases within movie reviews.
First, I tokenized the text into bigrams using the ‘unnest_tokens()’ function, specifying n = 2 to indicate that I wanted bigrams. Next, I split each bigram into two separate columns, word1 and word2, based on the space between the words. Then, I removed stop words and the HTML tag “br” in either position to ensure that only meaningful bigrams were retained. Finally, I used the ‘count()’ function to count the frequency of each bigram pair and sort them in descending order. The ‘na.omit()’ function was used to remove any rows with missing values.
As a result, I could see pairs of words and their frequencies in movie reviews. In the Bing graph, the most frequent positive word was ‘love’, but I could not determine the exact meaning and context in which the word was used. Now, I can see the bigram ‘love story’, suggesting that people left reviews about love stories. Similarly, ‘bad’ was the most frequent negative word, and now I found that it is often used with ‘movie’ and ‘guy’. Additionally, I identified meaningful word pairs such as ‘low budget’, ‘character development’, ‘world war’, ‘martial arts’, and ‘science fiction’, which help to understand the context in which these words are used.
bigram_review <- review_data %>%
unnest_tokens(input = review,
output = bigram,
token = "ngrams",
n = 2)
bigram_review## # A tibble: 11,687,851 × 3
## Id sentiment bigram
## <int> <chr> <chr>
## 1 1 positive one of
## 2 1 positive of the
## 3 1 positive the other
## 4 1 positive other reviewers
## 5 1 positive reviewers has
## 6 1 positive has mentioned
## 7 1 positive mentioned that
## 8 1 positive that after
## 9 1 positive after watching
## 10 1 positive watching just
## # ℹ 11,687,841 more rowsbigram_seprated <- bigram_review %>%
separate(bigram, c("word1", "word2"), sep = " ")
bigram_seprated## # A tibble: 11,687,851 × 4
## Id sentiment word1 word2
## <int> <chr> <chr> <chr>
## 1 1 positive one of
## 2 1 positive of the
## 3 1 positive the other
## 4 1 positive other reviewers
## 5 1 positive reviewers has
## 6 1 positive has mentioned
## 7 1 positive mentioned that
## 8 1 positive that after
## 9 1 positive after watching
## 10 1 positive watching just
## # ℹ 11,687,841 more rowsbigram_seprated <- bigram_seprated %>%
filter(!word1 %in% c("br", stop_words$word),
!word2 %in% c("br", stop_words$word))
bigram_seprated## # A tibble: 1,521,479 × 4
## Id sentiment word1 word2
## <int> <chr> <chr> <chr>
## 1 1 positive 1 oz
## 2 1 positive oz episode
## 3 1 positive unflinching scenes
## 4 1 positive faint hearted
## 5 1 positive drugs sex
## 6 1 positive called oz
## 7 1 positive oswald maximum
## 8 1 positive maximum security
## 9 1 positive emerald city
## 10 1 positive experimental section
## # ℹ 1,521,469 more rowspair_bigram <- bigram_seprated %>%
count(word1, word2, sort = T) %>%
na.omit()
pair_bigram## # A tibble: 885,335 × 3
## word1 word2 n
## <chr> <chr> <int>
## 1 special effects 2240
## 2 low budget 1812
## 3 sci fi 1384
## 4 real life 1230
## 5 main character 1063
## 6 horror movie 954
## 7 horror film 860
## 8 worth watching 845
## 9 main characters 781
## 10 bad movie 780
## # ℹ 885,325 more rowsIndividual analysis and figures
Using the Bing sentiment lexicon data, I created top 20 bar graphs and word cloud to check frequent positive and negative words in movie reviews. The bar plot shows the top 20 words for each sentiment, providing a clear comparison. The word clouds display the most common positive and negative words, highlighting their prominence and giving a visual representation of the sentiment in the reviews.
To further clarify the meaning of the words generally used in movie reviews, I utilized bigrams. This not only allowed me to understand the context in which the emotional words identified in the Bing graph were used but also helped determine the meanings of the words in the reviews. By examining words used in succession, I gained a more accurate understanding of the aspects people focus on when evaluating a movie.
Anaysis and Figure 1 - Bar Plot of Top 20 Positive and Negative Words
Since I already pre-processed the data into the bing_review_data variable, I used it to create the bar plot to see top 20 frequent sentiment words. First, I grouped the data by sentiment and selected the top 20 words for each sentiment based on their frequency. After ungrouping the data, I used the ‘mutate()’ function to reorder the words by their frequency for better visualization.
To create the bar plot, I used the ‘ggplot()’, which makes a ggplot object with word frequencies on the x-axis and words on the y-axis, filled by sentiment. I hid the legend using ‘geom_col(show.legend = FALSE)’. To create separate plots for positive and negative words with independent y-scales, I used the ‘facet_wrap()’ function. Finally, I added labels to the x-axis and removed the y-axis label.
As a result, I got two separate plots for positive and negative words. The most frequent words in the negative plot were ‘bad’, ‘plot’, ‘worst’, ‘funny’, ‘awful’, ‘poor’, ‘boring’, ‘stupid’, etc. The most frequent words in the positive plot were ‘love’, ‘excellent’, ‘fun’, ‘beautiful’, ‘pretty’, ‘wonderful’, ‘worth’, etc. Notably, the most frequent words ‘bad’ and ‘love’ stood out as remarkably high. This allowed me to compare the emotions people felt when they viewed the movie positively versus negatively. I could also predict the positive or negative emotions that arose while watching the movie.
bing_review_data %>%
group_by(sentiment) %>%
slice_max(n, n = 20) %>%
ungroup() %>%
mutate(word = reorder(word, n)) %>%
ggplot(aes(n, word, fill = sentiment)) +
geom_col(show.legend = FALSE) +
facet_wrap(~sentiment, scales = "free_y") +
labs(x = "Top 20 positive and negative words",
y = NULL)
Anaysis and Figure 2 - Word Cloud for Positive and Negative Words
The bar plot was helpful for seeing the frequent positive and negative words, but it was difficult to compare many words at a glance. So, I decided to create a word cloud. Word clouds offer a compelling visual tool for quickly and effectively understanding the most prominent words in a dataset. They highlight key terms and sentiments, making it easier to communicate findings and identify patterns within the data.
I visualized the top 100 positive and negative words based on their frequency using the ‘wordcloud()’ function. In addition to the top 20 positive words obtained from the bar plot, I was able to identify many more positive emotion words such as ‘powerful’, ‘impressive’, ‘warm’, and ‘spectacular’. In the negative word cloud, I found other negative words like ‘mystery’, ‘disappointment’, and ‘lousy’. This provided a richer set of frequent words, helping to understand the positive or negative emotions after watching a movie.
However, by only looking at sentiment words that appear with high frequency, it is impossible to understand the context in which the words are used. In other words, there is a limitation that it is difficult to know why the word was used and in what context.
review_tidy %>%
inner_join(get_sentiments("bing") %>%
filter(sentiment == "positive")) %>%
count(word) %>%
with(wordcloud(word, n, max.words = 150))## Joining with `by = join_by(sentiment, word)`
review_tidy %>%
inner_join(get_sentiments("bing") %>%
filter(sentiment == "negative")) %>%
count(word) %>%
with(wordcloud(word, n, max.words = 150))## Joining with `by = join_by(sentiment, word)`
Anaysis and Figure 3 - Bigram network graph
Using the pair_bigram variable created earlier, I developed a bigram network graph to visualize the relationships between words more clearly.
First, I filtered words that were used together more than 200 times. Then, using the ‘as_tbl_graph()’ function, I converted the data into a graph object with undirected edges. I also calculated the degree centrality for each node and assigned a group based on the Infomap algorithm for community detection.
To create the bigram network graph, I set a seed for reproducibility of the graph layout. The ‘ggraph()’ function initializes a ggraph object using a Fruchterman-Reingold layout. I represented the connections between nodes with a gray color and 50% transparency. The nodes were sized by centrality and colored by group, with the legend hidden. To adjust the node sizes, I used the ‘scale_size()’ function, setting the range from 3 to 6. I added labels to the nodes using ‘geom_node_text()’, which repels labels to avoid overlap and sets the text size to 2.5.
Degree centrality indicates how closely a node is connected to other nodes. Adjusting the node size based on degree centrality makes it easier to determine which words to pay attention to. By distinguishing nodes by community and representing them in different colors, the network structure becomes easier to understand. Therefore, when examining the data through the visualized network graph, it becomes easier to see which words are used in succession. For example, with ‘motion’, you can see that it is followed by ‘picture’ and ‘slow’, and with ‘story’, it is followed by ‘love’, ‘true’, and ‘lines’. This makes it easy to understand how relationships between words are formed.
# Creating a bigram network graph data
review_graph_bigram <- pair_bigram %>%
filter(n >= 200) %>%
as_tbl_graph(directed = F) %>%
mutate(centrality = centrality_degree(),
group = as.factor(group_infomap()))
review_graph_bigram## # A tbl_graph: 165 nodes and 127 edges
## #
## # An undirected multigraph with 48 components
## #
## # Node Data: 165 × 3 (active)
## name centrality group
## <chr> <dbl> <fct>
## 1 special 1 12
## 2 low 1 22
## 3 sci 1 23
## 4 real 3 3
## 5 main 2 6
## 6 horror 5 4
## 7 worth 1 7
## 8 bad 7 8
## 9 love 1 11
## 10 worst 4 4
## # ℹ 155 more rows
## #
## # Edge Data: 127 × 3
## from to n
## <int> <int> <int>
## 1 1 92 2240
## 2 2 93 1812
## 3 3 94 1384
## # ℹ 124 more rows# Creating a bigram network graph
set.seed(1234)
ggraph(review_graph_bigram, layout = "fr") +
geom_edge_link(color = "gray50",
alpha = 0.5) +
geom_node_point(aes(size = centrality,
color = group),
show.legend = F) +
scale_size(range = c(3, 6)) +
geom_node_text(aes(label = name),
repel = T,
size = 2.5) +
theme_graph()
Thank you :)