Goals

In our last class, we saw how we can analyze text data from lyrics. Today, we are going to practice this code and process with our headlines dataset from Lab 1.

Data

We will continue working with the same data on \(n=2000\) article titles from our last lab. To load the data, copy and paste the code below into a chunk and press play.

headlines <- read.csv("https://www.dropbox.com/scl/fi/r9p76t3v8aluz2jfypy6u/headlines.csv?rlkey=pi5rpu21xkwjw8qm7bofkrrej&st=jhc4e0ad&dl=1")

Recall that the columns are:

• title: the title of the article
• clickbait: a human generated indicator for whether or not an article is clickbait; FALSE means the article is not clickbait while TRUE means the article is clickbait.
  
• ids: a numeric variable assigned to each article; think of this like an article identifier.

Once you have loaded the data, load the packages you will need for this lab:

library(tidytext)
library(tidyr)
library(dplyr)
library(ggplot2)
# New!! 
library(tm)
library(forcats)

As a reminder, anytime R tells you a package cannot be found means that it needs to be installed. You can do this from the Tools drop down menu at the top of your R screen. Let me know if you have any questions!

Clickbait Only: Data Cleaning

We are first going to do some EDA on the clickbait articles. In other words, we want to explore the titles that belong to clickbait articles to see if we can describe traits of these titles. This means that our first step today is to create a dataset that only contains clickbait titles, as headlines has both clickbait and non clickbait articles.

To create a dataset with only clickbait articles, we want to filter our dataset to contain only the rows in headlines that belong to clickbait titles. We will use the command filter to help us do this.

clickbait <- headlines |>
  filter(clickbait == "TRUE")

The filter command keeps only the rows in the dataset that meet the condition inside of the parentheses. In this case, we keep only the clickbait titles, which means only the ones with TRUE in the clickbait column.

We now have a dataset called clickbait which contains only clickbait titles. The first thing we typically do with this dataset is to tokenize the article titles.

Clickbait Only: Most Frequent Words

Now that we have tokenized the titles, let’s see if we can determine which words occur most frequently in the clickbait titles. Here is where the fact that tokenizing data in R also results in all words being converted to lowercase is a good thing. Without this, R would count “Today” and “today” as different words. Because we convert all words to lower case, this issue is avoided!

In order to find out which words appear most frequently, we need count the number of times each word appears. We can count the number of times each word appears in tidy_clickbaitonly using this code:

clickbait_count <- headlines |>
  filter(clickbait == "TRUE") |> 
  unnest_tokens(word, title, token = "regex") |>
  count(word, sort = TRUE)

This code will create a data frame with 2 columns. The first column (word) tells us one unique word that occurs in clickbait titles. The second column (n) is the number of times that word appears in our clickbait titles.

Stop Words

You will likely notice something about the words that have the highest counts in the clickbait titles. We see words like “the” and “to”. These are all very common words in the English language, but we were hoping to find some words that could distinguish clickbait titles. For this goal, words like “the” are not helpful to us.

We call words that are needed for grammatical reasons, but that do not add to the content, stop words. You will find that we use a lot of stop words in English, including “the”, “it”, “a”, “as”, “an”, and so on.

If you are interested in what other words are considered stop words, you can use the following code to explore:

# Load a list of stop words 
data("stop_words")

# Look at the first 5 words
stop_words[1:5,"word"]

Let’s take a look at the 13th title in the dataset:

headlines[13, "title"]

## [1] "Judge Guilty in Kickbacks Is Accused of Fixing Suit"

Since stop words do not actually add to the content, we often consider removing stop words before we conduct any analysis on text data. As we have already seen, if we do not remove stop words they often dominate the list of most frequent words, making it difficult to see which words might be useful features to differentiate clickbait and non-clickbait titles.

To (1) tokenize the 13th title and (2) remove the stop words, we use the following:

# Start with the 13th title
headlines[13,] |> 

  # Break the title into words 
  unnest_tokens(word,  title, token = "regex")|> # and then 

  # Remove all stop words
  anti_join(stop_words, by = "word")

clickbait_count <- headlines |>
  filter(clickbait == "TRUE") |> 
  unnest_tokens(word, title, token = "regex") |>
  count(word, sort = TRUE)

stop_words <- subset(stop_words, word != "you")

clickbait_count <- headlines |>
  filter(clickbait == "TRUE") |> 
  unnest_tokens(word, title, token = "regex") |>
  count(word, sort = TRUE)

Visualizing the Most Frequent Words

Before we got off on our tangent about stop words, we were interested in looking at the most frequent words in clickbait articles (excluding stop words!). We are going to visualize the most frequent words in clickbait articles using (1) bar plots and (2) word clouds.

ggplot( dataset, aes( x = count, y = what we are counting) ) + 
    geom_col( fill = "color you want") 

  # Choose only the top 15 most frequent words
  slice_max(n, n=15)

# Start with the headlines dataset 
tidy_clickbaitonly <- headlines |>
  # Choose only the clickbait titles
  filter(clickbait =="TRUE") |>
  # Tokenize the titles 
  unnest_tokens(word, title, token = "regex") |>
  # Remove the stop words
  anti_join(stop_words) |>
  # Remove the numbers 
  filter(!grepl('[0-9]', word)) 

# Load the library 
library(wordcloud)
set.seed(279)

# Make the plot 
tidy_clickbaitonly |>
  count(word,sort = TRUE) |>
  with(wordcloud(word, n, max.words = 10))

clickbait_count |>
  with(wordcloud(word, n, max.words = 10, random.order=FALSE, colors=brewer.pal(8, "Dark2")))