Data Science Capstone - Milestone Report
Vitalii Diakonov
2024-04-11
Instructions
The goal of this project is just to display that you’ve gotten used to working with the data and that you are on track to create your prediction algorithm. Please submit a report on R Pubs that explains your exploratory analysis and your goals for the eventual app and algorithm.
This document should be concise and explain only the major features of the data you have identified and briefly summarize your plans for creating the prediction algorithm and Shiny app in a way that would be understandable to a non-data scientist manager. You should make use of tables and plots to illustrate important summaries of the data set.
The motivation for this project is to:
1. Demonstrate that you’ve downloaded the data and have successfully loaded it in.
2. Create a basic report of summary statistics about the data sets.
3. Report any interesting findings that you amassed so far.
4. Get feedback on your plans for creating a prediction algorithm and Shiny app.
Load libraries
library(tm)
library(quanteda)
library(readtext)
library(data.table)
library(stringr)
library(wordcloud)Getting and Cleaning the Data
Load Data
I checked if the text files already exist in the specified directory. If not, the code downloaded and extracted the dataset. Then, I read the text files for the blogs, news, and Twitter datasets using the readtext function.
# Check if the files already exist
if (!file.exists('./final/en_US/en_US.blogs.txt')) {
# Download and extract the dataset if the files don't exist
download.file('https://d396qusza40orc.cloudfront.net/dsscapstone/dataset/Coursera-SwiftKey.zip',
destfile = 'Coursera-SwiftKey.zip',
method = 'auto', quiet = TRUE)
unzip('Coursera-SwiftKey.zip')
}
# Define path to unzipped file
data.files.path <- "./final/"
# Define path to English-language data
data.files.path.EN <- file.path(data.files.path, "en_US")
# Assign a variable name to each English dataset:
blog.file <- file.path(data.files.path.EN, "en_US.blogs.txt") # blogs
news.file <- file.path(data.files.path.EN, "en_US.news.txt") # news
twitter.file <- file.path(data.files.path.EN, "en_US.twitter.txt") # Twitter
# Read in blog data
blog_text <- readLines(blog.file, encoding = "UTF-8", skipNul = TRUE)
# Read in news data
news_text <- readLines(news.file, encoding = "UTF-8", skipNul = TRUE)
# Read in Twitter data
twitter_text <- readLines(twitter.file, encoding = "UTF-8", skipNul = TRUE)Basic Summary
Let’s proceed with counting the number of lines, words, and characters in each file, and then summarize this information in a table.
# Count the number of lines, words, and characters in each file
num_lines <- c(length(blog_text), length(news_text), length(twitter_text))
num_words <- c(sum(sapply(blog_text, function(x) length(unlist(strsplit(x, "\\s+"))))),
sum(sapply(news_text, function(x) length(unlist(strsplit(x, "\\s+"))))),
sum(sapply(twitter_text, function(x) length(unlist(strsplit(x, "\\s+"))))))
num_chars <- c(sum(nchar(blog_text)),
sum(nchar(news_text)),
sum(nchar(twitter_text)))
# Summarize the information in a table
files_summary <- data.frame(
filename = c("en_US.blogs.txt", "en_US.news.txt", "en_US.twitter.txt"),
lines = num_lines,
words = num_words,
characters = num_chars
)
# Print the table
print(files_summary)## filename lines words characters
## 1 en_US.blogs.txt 899288 37334131 206824505
## 2 en_US.news.txt 1010242 34372530 203223159
## 3 en_US.twitter.txt 2360148 30373583 162096241en_US.blogs.txt: This file contains approximately 37.33 million words and 206.82 million characters across 899,288 lines.
en_US.news.txt: This file contains around 34.37 million words and 203.22 million characters distributed over 1,010,242 lines.
en_US.twitter.txt: This file consists of roughly 30.37 million words and 162.10 million characters, spanning 2,360,148 lines.
Sampling and Merging the Data
To mitigate computational demands due to the large file sizes observed in the summary table, a subset of 10,000 lines is chosen from each file. These selected lines will undergo cleaning processes before being merged into a cohesive dataset.
# set the seed to ensure reproducible results
set.seed(123)
# set sample size
sample_size <- 10000
# sample data from each text file
sampled_blog <- sample(blog_text, sample_size)
sampled_news <- sample(news_text, sample_size)
sampled_twitter <- sample(twitter_text, sample_size)
# Combine the sampled data into one data file
combined_data <- rbind(sampled_blog, sampled_news, sampled_twitter)
# Write combined sample data to file
writeLines(combined_data, "combined_sample_data.txt")Now, I’ll unify the dataset into a corpus using the “quanteda” package.
# I'll convert the combined sampled data into a corpus
corpus <- corpus(combined_data)
# Verify the corpus content
print(summary(corpus))## Corpus consisting of 30000 documents, showing 100 documents:
##
## Text Types Tokens Sentences
## text1 30 37 2
## text2 32 37 1
## text3 16 16 2
## text4 13 15 1
## text5 30 38 2
## text6 10 13 2
## text7 57 74 2
## text8 21 24 2
## text9 24 25 2
## text10 3 3 1
## text11 10 10 1
## text12 6 6 2
## text13 48 66 5
## text14 24 25 1
## text15 7 10 1
## text16 14 15 1
## text17 22 23 1
## text18 26 40 2
## text19 18 22 1
## text20 39 45 2
## text21 6 6 1
## text22 53 69 6
## text23 24 28 1
## text24 20 22 1
## text25 169 254 11
## text26 60 91 5
## text27 29 32 2
## text28 39 54 3
## text29 39 49 1
## text30 24 29 1
## text31 65 92 8
## text32 40 50 5
## text33 12 15 3
## text34 13 14 2
## text35 40 46 1
## text36 10 11 1
## text37 5 5 1
## text38 31 38 4
## text39 19 21 1
## text40 27 40 2
## text41 45 60 2
## text42 5 5 1
## text43 3 3 1
## text44 45 48 2
## text45 22 23 1
## text46 16 17 1
## text47 26 31 1
## text48 6 7 1
## text49 71 97 4
## text50 31 39 3
## text51 14 16 3
## text52 30 57 1
## text53 39 49 2
## text54 14 15 1
## text55 3 3 1
## text56 15 20 2
## text57 18 18 2
## text58 16 18 2
## text59 21 23 1
## text60 3 3 1
## text61 5 7 1
## text62 6 8 2
## text63 22 23 2
## text64 48 58 2
## text65 11 14 2
## text66 27 30 1
## text67 31 36 1
## text68 3 3 1
## text69 26 33 3
## text70 50 63 5
## text71 73 103 5
## text72 26 38 3
## text73 55 83 5
## text74 33 38 1
## text75 23 28 2
## text76 4 4 1
## text77 48 66 5
## text78 20 20 1
## text79 18 19 1
## text80 34 39 2
## text81 6 7 1
## text82 23 27 2
## text83 54 88 5
## text84 3 4 1
## text85 36 41 1
## text86 25 25 1
## text87 20 21 1
## text88 49 59 3
## text89 33 39 1
## text90 11 17 3
## text91 17 24 1
## text92 50 64 3
## text93 3 3 1
## text94 33 39 2
## text95 43 62 3
## text96 8 8 1
## text97 2 2 1
## text98 39 49 4
## text99 25 29 2
## text100 33 36 3This corpus is now ready for analysis!
Data Cleaning
As I begin preparing the text data, my first task is to clean it up to ensure it’s consistent and free from unnecessary distractions. Here’s what I do:
I remove website URLs, Twitter handles, and email addresses.
I get rid of any characters that aren’t standard ASCII.
I take out numbers, as they aren’t relevant for our analysis.
I remove punctuation marks.
I clean up extra white spaces.
text_tokens <- tokens(corpus,
what="word1",
remove_numbers = TRUE,
remove_punct = TRUE,
remove_url =TRUE,
remove_separators = TRUE,
remove_symbols = TRUE,
verbose = quanteda_options("verbose"))
head(text_tokens)## Tokens consisting of 6 documents.
## text1 :
## [1] "The" "bruschetta" "however" "missed" "the"
## [6] "mark" "Instead" "of" "manageable" "two-bite"
## [11] "crostini" "these"
## [ ... and 20 more ]
##
## text2 :
## [1] "In" "general" "however" "the" "defense" "faces"
## [7] "an" "extremely" "high" "hurdle" "in" "getting"
## [ ... and 22 more ]
##
## text3 :
## [1] "You" "can't" "beat" "hours" "of" "power"
## [7] "Somebody" "get" "me" "a" "hour" "energy"
## [ ... and 1 more ]
##
## text4 :
## [1] "Walden" "Pond" "Mt" "Rainier" "Big"
## [6] "Sur" "Everglades" "and" "so" "forth"
##
## text5 :
## [1] "Tuominen" "attributes" "the" "school's" "success"
## [6] "to" "strong" "academic" "expectations" "and"
## [11] "solid" "citizenship"
## [ ... and 18 more ]
##
## text6 :
## [1] "Add" "me" "love" "me" "and" "love" "my" "stuff" "Ha"
## [10] "ha" "ha"I generate a version of the tokens without stopwords to conduct initial exploratory analysis. During the predictive model building, I’ll retain stopwords in the data to ensure more natural-sounding results. Stemming isn’t applied in either the exploratory analysis or the predictive model building for the same reason.
# Remove stopwords from the tokens
tokens_no_stopwords <- tokens_remove(text_tokens, stopwords("en"))
# Display the first few tokens after removing stopwords to ensure meaningful content remains
print(head(tokens_no_stopwords))## Tokens consisting of 6 documents.
## text1 :
## [1] "bruschetta" "however" "missed" "mark" "Instead"
## [6] "manageable" "two-bite" "crostini" "huge" "slices"
## [11] "grilled" "bread"
## [ ... and 9 more ]
##
## text2 :
## [1] "general" "however" "defense" "faces" "extremely" "high"
## [7] "hurdle" "getting" "new" "trial" "juror's" "actions"
## [ ... and 7 more ]
##
## text3 :
## [1] "beat" "hours" "power" "Somebody" "get" "hour" "energy"
## [8] "drink"
##
## text4 :
## [1] "Walden" "Pond" "Mt" "Rainier" "Big"
## [6] "Sur" "Everglades" "forth"
##
## text5 :
## [1] "Tuominen" "attributes" "school's" "success" "strong"
## [6] "academic" "expectations" "solid" "citizenship" "Students"
## [11] "asked" "follow"
## [ ... and 7 more ]
##
## text6 :
## [1] "Add" "love" "love" "stuff" "Ha" "ha" "ha"Now that I’ve seen the output after removing stopwords, I’m confident that my text tokens are prepared well for further analysis. The tokens retain meaningful content, which is essential for generating unigrams and creating a document-feature matrix (DFM) that truly captures the essential textual features without the clutter of common stopwords.
Removing Profanity
First, I need to ensure that I have a comprehensive list of profane words to filter out from my data. I’ve already downloaded a list of such words, and now I will apply this filter to my tokens.
# Define the name of the downloaded file
bad_words_file <- "bad-words.txt"
# Define the path to the file
data_file_path <- file.path(getwd(), bad_words_file)
# Define the URL where the text file containing a list of profane words/phrases can be downloaded
url <- "http://www.cs.cmu.edu/~biglou/resources/bad-words.txt"
# Check if the bad words file already exists, and download it if it doesn't
if (!file.exists(data_file_path)) {
message("Downloading bad words list...")
download.file(url, destfile = data_file_path, quiet = FALSE)
message("Download complete.")
} else {
message("Bad words list already exists.")
}
# Read the lines of the text file containing profanities
bad_words <- readLines(data_file_path, encoding = "UTF-8")
if (length(bad_words) == 0) {
warning("The bad words file is empty. Check if the download was successful and the file is not corrupted.")
}
# Remove profanity from the tokens
tokens_no_bad_words <- tokens_remove(text_tokens, phrase(bad_words))
# Display the first few tokens after removing profanity to ensure meaningful content remains
print(head(tokens_no_bad_words))## Tokens consisting of 6 documents.
## text1 :
## [1] "The" "bruschetta" "however" "missed" "the"
## [6] "mark" "Instead" "of" "manageable" "two-bite"
## [11] "crostini" "these"
## [ ... and 20 more ]
##
## text2 :
## [1] "In" "general" "however" "the" "defense" "faces"
## [7] "an" "extremely" "high" "hurdle" "in" "getting"
## [ ... and 21 more ]
##
## text3 :
## [1] "You" "can't" "beat" "hours" "of" "power"
## [7] "Somebody" "get" "me" "a" "hour" "energy"
## [ ... and 1 more ]
##
## text4 :
## [1] "Walden" "Pond" "Mt" "Rainier" "Big"
## [6] "Sur" "Everglades" "and" "so" "forth"
##
## text5 :
## [1] "Tuominen" "attributes" "the" "school's" "success"
## [6] "to" "strong" "academic" "expectations" "and"
## [11] "solid" "citizenship"
## [ ... and 18 more ]
##
## text6 :
## [1] "Add" "me" "love" "me" "and" "love" "my" "stuff" "Ha"
## [10] "ha" "ha"Exploring the Data and Enhancing Data Quality
During this phase, the objective is to gain deeper insights into the distribution and interrelations among words, tokens, and phrases within the dataset.
The analysis will address the following inquiries:
What are the distributions of word frequencies, considering that some words occur more frequently than others?
How prevalent are 2-grams and 3-grams in the dataset?
How many distinct words are required in a frequency-ordered dictionary to encompass 50% and 90% of all word instances in the language?
What methods can be employed to identify the proportion of words originating from foreign languages?
Is there a strategy to broaden coverage, such as identifying words absent in the corpora or employing a reduced number of words in the dictionary to encompass the same number of phrases?
Generating Unigrams
With the tokens now cleaned of URLs, punctuation, numbers, non-standard ASCII characters, extra whitespaces, stopwords, and profanity, I’m ready to generate unigrams. As mentioned, I will produce two different sets of unigrams for a comparative analysis.
# Generate Unigrams with Bad Words
unigrams_with_badwords <- tokens_ngrams(text_tokens, n = 1)
# Generate Unigrams without Bad Words
unigrams_no_badwords <- tokens_ngrams(tokens_no_bad_words, n = 1)
# Print the first few unigrams with and without bad words for inspection
print(head(unigrams_with_badwords))## Tokens consisting of 6 documents.
## text1 :
## [1] "The" "bruschetta" "however" "missed" "the"
## [6] "mark" "Instead" "of" "manageable" "two-bite"
## [11] "crostini" "these"
## [ ... and 20 more ]
##
## text2 :
## [1] "In" "general" "however" "the" "defense" "faces"
## [7] "an" "extremely" "high" "hurdle" "in" "getting"
## [ ... and 22 more ]
##
## text3 :
## [1] "You" "can't" "beat" "hours" "of" "power"
## [7] "Somebody" "get" "me" "a" "hour" "energy"
## [ ... and 1 more ]
##
## text4 :
## [1] "Walden" "Pond" "Mt" "Rainier" "Big"
## [6] "Sur" "Everglades" "and" "so" "forth"
##
## text5 :
## [1] "Tuominen" "attributes" "the" "school's" "success"
## [6] "to" "strong" "academic" "expectations" "and"
## [11] "solid" "citizenship"
## [ ... and 18 more ]
##
## text6 :
## [1] "Add" "me" "love" "me" "and" "love" "my" "stuff" "Ha"
## [10] "ha" "ha"print(head(unigrams_no_badwords))## Tokens consisting of 6 documents.
## text1 :
## [1] "The" "bruschetta" "however" "missed" "the"
## [6] "mark" "Instead" "of" "manageable" "two-bite"
## [11] "crostini" "these"
## [ ... and 20 more ]
##
## text2 :
## [1] "In" "general" "however" "the" "defense" "faces"
## [7] "an" "extremely" "high" "hurdle" "in" "getting"
## [ ... and 21 more ]
##
## text3 :
## [1] "You" "can't" "beat" "hours" "of" "power"
## [7] "Somebody" "get" "me" "a" "hour" "energy"
## [ ... and 1 more ]
##
## text4 :
## [1] "Walden" "Pond" "Mt" "Rainier" "Big"
## [6] "Sur" "Everglades" "and" "so" "forth"
##
## text5 :
## [1] "Tuominen" "attributes" "the" "school's" "success"
## [6] "to" "strong" "academic" "expectations" "and"
## [11] "solid" "citizenship"
## [ ... and 18 more ]
##
## text6 :
## [1] "Add" "me" "love" "me" "and" "love" "my" "stuff" "Ha"
## [10] "ha" "ha"Creating the Document-Feature Matrix
To analyze the textual data quantitatively, I will create document-feature matrices (DFMs) from both sets of unigrams. This will facilitate various analyses, including frequency analysis and potentially feeding the data into predictive models.
# Create DFM from unigrams with bad words
dfm_with_badwords <- dfm(unigrams_with_badwords)
# Create DFM from unigrams without bad words
dfm_no_badwords <- dfm(unigrams_no_badwords)
# Print summaries to compare the two DFMs
print(summary(dfm_with_badwords))## Length Class Mode
## 1629360000 dfm S4print(summary(dfm_no_badwords))## Length Class Mode
## 1616190000 dfm S4The lengths of the DFMs, while indicative of their robustness, also suggest that the removal of profanity did not significantly reduce the volume of data. This is good, as it means the core content remains largely intact, which is important for maintaining the context and meaningfulness of the analyses.
Frequency Analysis
To get a clearer picture of the top terms and their frequencies, I will analyze and visualize the most frequent words from both DFMs.
# Convert DFMs to frequency data frames
dfm_with_badwords <- dfm(unigrams_with_badwords)
dfm_no_badwords <- dfm(unigrams_no_badwords)
# Get top terms for each to use in the word cloud
top_features_with_badwords <- topfeatures(dfm_with_badwords, n = 100)
top_features_no_badwords <- topfeatures(dfm_no_badwords, n = 100)
# Create data frames for wordcloud
unigram.top.features.with.badwords.df <- data.frame(
unigram = names(top_features_with_badwords),
freq = unlist(top_features_with_badwords)
)
unigram.top.features.no.badwords.df <- data.frame(
unigram = names(top_features_no_badwords),
freq = unlist(top_features_no_badwords)
)
# Word cloud with bad words
wordcloud(
words = unigram.top.features.with.badwords.df$unigram,
freq = unigram.top.features.with.badwords.df$freq,
min.freq = 1,
max.words = 100,
random.order = FALSE,
rot.per = 0.35,
colors = brewer.pal(8, "Dark2"),
main = "With Profanity"
)
# Word cloud without bad words
wordcloud(
words = unigram.top.features.no.badwords.df$unigram,
freq = unigram.top.features.no.badwords.df$freq,
min.freq = 1,
max.words = 100,
random.order = FALSE,
rot.per = 0.35,
colors = brewer.pal(8, "Pastel1"),
main = "Without Profanity"
)
Analysis with Histograms
To deepen my understanding and provide a different perspective on the data, I plan to create histograms of the top 15 most common words from both document-feature matrices (DFMs). This will complement the word clouds by offering a precise quantification of word frequencies.
# Convert DFMs to frequency data frames for the top 15 words
top15_with_badwords <- topfeatures(dfm_with_badwords, n = 15)
top15_no_badwords <- topfeatures(dfm_no_badwords, n = 15)
# Create data frames for histogram plotting
top15_with_badwords_df <- data.frame(
Word = names(top15_with_badwords),
Frequency = unlist(top15_with_badwords),
Category = "With Profanity"
)
top15_no_badwords_df <- data.frame(
Word = names(top15_no_badwords),
Frequency = unlist(top15_no_badwords),
Category = "Without Profanity"
)
# Combine the data frames
top15_words_df <- rbind(top15_with_badwords_df, top15_no_badwords_df)
# Create the histogram using ggplot2
library(ggplot2)
ggplot(top15_words_df, aes(x = reorder(Word, -Frequency), y = Frequency, fill = Category)) +
geom_col(position = "dodge") +
labs(title = "Top 15 Words Frequency Comparison",
x = "Words",
y = "Frequency",
fill = "Category") +
theme_minimal() +
theme(axis.text.x = element_text(angle = 45, hjust = 1)) 
From this histogram, I can observe that the removal of profanity doesn’t significantly alter the frequency of the most common words in the corpus. This suggests that the profane words were not among the most frequent to begin with, or that their removal does not significantly impact the overall distribution of these top words.
The frequency of the articles ‘the’, ‘a’, and prepositions ‘of’, ‘in’, ‘on’ as top unigrams indicates a standard English text structure.
Top 15 Most Frequent Unigrams in text format for later analysis:
# Extract the top 15 words without bad words and print them
top15_unigrams_no_badwords <- topfeatures(dfm_no_badwords, n = 15)
print("Top 15 most frequent unigrams without bad words:")## [1] "Top 15 most frequent unigrams without bad words:"print(top15_unigrams_no_badwords)## the to and a of in i that for is it on you
## 44161 24170 22947 21085 18866 14918 13264 9623 9151 9014 7949 6856 6627
## with was
## 6549 5896Analyzing Bigrams and Trigrams
First, I’ll generate bigrams and trigrams to understand common phrases within the dataset. These n-grams can reveal more about the context and relationships between words than unigrams.
# Generate bigrams and trigrams
tokens_bigrams <- tokens_ngrams(tokens_no_bad_words, n = 2)
tokens_trigrams <- tokens_ngrams(tokens_no_bad_words, n = 3)
# Create DFMs from bigrams and trigrams
dfm_bigrams <- dfm(tokens_bigrams)
dfm_trigrams <- dfm(tokens_trigrams)
# Extract the top 15 bigrams and trigrams
top15_bigrams <- topfeatures(dfm_bigrams, n = 15)
top15_trigrams <- topfeatures(dfm_trigrams, n = 15)
# Print the results
print("Top 15 most frequent bigrams:")## [1] "Top 15 most frequent bigrams:"print(top15_bigrams)## of_the in_the to_the on_the for_the to_be and_the at_the
## 4122 3938 1989 1736 1703 1427 1269 1177
## in_a with_the is_a it_was from_the of_a with_a
## 1067 1015 944 881 849 801 776print("Top 15 most frequent trigrams:")## [1] "Top 15 most frequent trigrams:"print(top15_trigrams)## one_of_the a_lot_of the_end_of to_be_a some_of_the out_of_the
## 335 264 168 151 138 138
## going_to_be as_well_as it_was_a be_able_to i_want_to this_is_a
## 137 136 129 128 119 115
## a_couple_of part_of_the end_of_the
## 111 109 106It’s clear that common English words like articles and prepositions dominate the top unigrams, which is typical in most text corpora. The bigrams and trigrams seem to consist of common English phrases.
Bigrams and trigrams often contain these articles and prepositions, forming common phrases such as ‘of the’, ‘in the’, ‘to the’, ‘one of the’.
The commonality of phrases like ‘a lot of’ and ‘going to be’ suggests patterns in usage that could be relevant for natural language processing tasks.
Next Steps with N-gram Visualization
To further explore these findings, I’ll create visualizations for the bigrams and trigrams:
# Generate a bar plot for bigrams
bigrams_df <- data.frame(Bigram = names(top15_bigrams), Frequency = unname(top15_bigrams))
ggplot(bigrams_df, aes(x = reorder(Bigram, -Frequency), y = Frequency)) +
geom_bar(stat = "identity", fill = "steelblue") +
labs(title = "Top 15 Bigrams", x = "Bigrams", y = "Frequency") +
theme_minimal() +
theme(axis.text.x = element_text(angle = 45, hjust = 1))
# Generate a bar plot for trigrams
trigrams_df <- data.frame(Trigram = names(top15_trigrams), Frequency = unname(top15_trigrams))
ggplot(trigrams_df, aes(x = reorder(Trigram, -Frequency), y = Frequency)) +
geom_bar(stat = "identity", fill = "coral") +
labs(title = "Top 15 Trigrams", x = "Trigrams", y = "Frequency") +
theme_minimal() +
theme(axis.text.x = element_text(angle = 45, hjust = 1))
Distribution of Word Frequencies
The frequency distribution can indicate whether a few words dominate the corpus (a common characteristic in language called Zipf’s Law). I’ll use a log-log plot to visualize this distribution.
# Extract frequency data directly from the DFM
freq_vector <- rowSums(as.matrix(dfm_no_badwords))
# Create a data frame for the frequencies and ranks
freq_df <- data.frame(
word = names(freq_vector),
frequency = freq_vector
)
freq_df$rank <- rank(-freq_df$frequency) # Rank by frequency, high to low
# Create a log-log plot
ggplot(freq_df, aes(x = log(rank), y = log(frequency))) +
geom_point(alpha = 0.5, color = 'blue') +
geom_smooth(method = "lm", se = FALSE, color = "red") +
labs(title = "Log-Log Plot of Word Frequencies",
x = "Log(Rank)",
y = "Log(Frequency)") +
theme_minimal()
Looking at the log-log plot you’ve provided, it appears to illustrate a distribution that closely follows Zipf’s Law. The plot shows a clear linear trend on the log-log scale, which is typical of natural language distributions: a small number of words are extremely common (high frequency), and many words are rare (low frequency).
Prevalence of 2-grams and 3-grams
To evaluate the prevalence of 2-grams and 3-grams, I’ll need to compare their frequency distributions to the unigrams.
# Compare the top 15 n-grams frequencies in a combined bar plot
ngram_freqs <- rbind(
data.frame(ngram = names(top15_unigrams_no_badwords), frequency = unname(top15_unigrams_no_badwords), type = "Unigram"),
data.frame(ngram = names(top15_bigrams), frequency = unname(top15_bigrams), type = "Bigram"),
data.frame(ngram = names(top15_trigrams), frequency = unname(top15_trigrams), type = "Trigram")
)
ggplot(ngram_freqs, aes(x = reorder(ngram, frequency), y = frequency, fill = type)) +
geom_bar(stat = "identity", position = position_dodge()) +
labs(x = "N-gram", y = "Frequency", title = "Comparison of N-gram Frequencies") +
theme_minimal() +
theme(axis.text.x = element_text(angle = 90, hjust = 1))
Dictionary Coverage
To determine the dictionary coverage, I will find the point where 50% and 90% of all word instances are covered by a frequency-ordered list of distinct words.
# Calculate the cumulative frequency and the total frequency
cumulative_freq <- cumsum(freq_df$frequency)
total_freq <- sum(freq_df$frequency)
# Calculate the coverage
coverage_50 <- cumulative_freq[which(cumulative_freq >= (0.50 * total_freq))[1]]
coverage_90 <- cumulative_freq[which(cumulative_freq >= (0.90 * total_freq))[1]]
# Determine the number of words that make up 50% and 90% of all word instances
num_words_coverage_50 <- which(cumulative_freq >= (0.50 * total_freq))[1]
num_words_coverage_90 <- which(cumulative_freq >= (0.90 * total_freq))[1]
# Output the results
cat("Number of words covering 50% of all word instances:", num_words_coverage_50, "\n")## Number of words covering 50% of all word instances: 15010cat("Number of words covering 90% of all word instances:", num_words_coverage_90, "\n")## Number of words covering 90% of all word instances: 2703250% Coverage: The first figure shows that a relatively small subset of the vocabulary contains half of all word occurrences. This aligns with the principle of the Pareto distribution (or the 80/20 rule), although in this case, it’s more like a 50% rule. It suggests that the corpus is dominated by a limited ‘core’ vocabulary, which is common in natural language where some words are used much more frequently than others.
90% Coverage: The second figure illustrates that to account for 90% of the corpus, the vocabulary list needs to be significantly extended. This increase from 50% to 90% coverage includes many more words, indicating a long tail of less frequent words that add up to a substantial portion of the text.
In preparation for the next phase of building the predictive model, I’ll persist the relevant n-gram data to disk. This involves saving the unigram, bigram, and trigram datasets in a format that can be easily retrieved and utilized during model training.
# Save the unigrams to disk
saveRDS(unigrams_no_badwords, file = "unigrams.rds")
# Save the bigrams to disk
saveRDS(tokens_bigrams, file = "bigrams.rds")
# Save the trigrams to disk
saveRDS(tokens_bigrams, file = "trigrams.rds")
# Print a confirmation message
cat("The unigrams, bigrams, and trigrams have been successfully saved to disk.")## The unigrams, bigrams, and trigrams have been successfully saved to disk.For the development of the predictive model, I plan to utilize a blend of unigrams, bigrams, and trigrams to forecast the subsequent word based on input text. Here are my next steps:
Building N-gram Models:
Utilize the extracted unigrams, bigrams, and trigrams from the corpus to construct Markov chains.
Calculate the transition probabilities to another state for each N-gram (unigrams, bigrams, trigrams).
Store these transition probabilities to form the foundation of the predictive model.
Implementing Backoff Models:
Implement backoff models to handle cases where specific N-grams are missing or have limited data in the corpus.
Begin by considering the probability distribution of the next word given a specific trigram. If the trigram doesn’t exist, move to the bigram model, and then to the unigram model if necessary.
Dealing with Limited Data and Narrow Focus:
Address the issue of limited data for certain N-grams, which may lead to over-concentration of predictions on specific topics.
Apply relative weights to unigrams, bigrams, and trigrams in the predictive model to balance the influence of each N-gram category.
Explore the possibility of using the “Stupid Backoff” method, which assigns scores to each N-gram category based on their relative frequencies. This approach allows for a smoother transition between different levels of N-grams, avoiding overfitting on sparse data.
Model Evaluation and Fine-tuning:
Evaluate the performance of the predictive model using appropriate metrics such as perplexity, accuracy, and coverage.
Conduct fine-tuning of model parameters, including relative weights assigned to different N-gram categories, to optimize prediction accuracy and generalization ability.
Integration with Shiny App:
Integrate the trained N-gram predictive model with the Shiny web application to provide real-time word prediction functionality.
Ensure seamless interaction between the user interface and the predictive model to enhance usability.