Our goal for this project is to examine a set of English text data and clean and process it to use for building a predictive text application. The predictive text app will be able to provide a suggestion for the next word while the user is typing.
This experimental analysis aims to explore data sets in detail, look at the features of data sets, is working on a list of the most popular words, and is making a plan for the prediction algorithm.
The study used 3 datasets in the English language:
Twitter: short comments and chat messages.
Blogs: includes longer texts authored by various writers.
News: includes, news, articles, and formal written materials.
These datasets will give different writing styles and wording to help train a better model of predicting what will come up.
#Installing the necessary packages
install.packages(“quanteda”) install.packages(“stringr”) install.packages(“ggplot2”) install.packages(“knitr”)
# Load required packages
library(ggplot2)
library(quanteda)
library(knitr)
library(quanteda.textstats)
library(stringr)
# Load the three datasets
twitter <- readLines("en_US.twitter.txt",
encoding = "UTF-8",
skipNul = TRUE)
blogs <- readLines("en_US.blogs.txt",
encoding = "UTF-8",
skipNul = TRUE)
news <- readLines("en_US.news.txt",
encoding = "UTF-8",
skipNul = TRUE)
# 3. Exploratory Data Analysis
#The three datasets were imported and viewed in R.
#The analysis included:
#- All lines in each dataset.
#- total words,
#- Average and maximum line length.
#- Similar words that are often used.
#- Length of text distribution.
#Had some tables and graphs constructed to make comparisons between datasets.
# a. Calculating line counts in all three data sets
twitter_lines <- length(twitter)
blogs_lines <- length(blogs)
news_lines <- length(news)
line_summary <- data.frame(
Dataset = c("Twitter", "Blogs", "News"),
Total_Lines = c(
twitter_lines,
blogs_lines,
news_lines
)
)
kable(line_summary)
| Dataset | Total_Lines |
|---|---|
| 2360148 | |
| Blogs | 899288 |
| News | 1010206 |
# b. Calculating Word counts in all three data sets
twitter_tokens <- tokens(twitter)
blogs_tokens <- tokens(blogs)
news_tokens <- tokens(news)
twitter_words <- sum(ntoken(twitter_tokens))
blogs_words <- sum(ntoken(blogs_tokens))
news_words <- sum(ntoken(news_tokens))
word_summary <- data.frame(
Dataset = c("Twitter", "Blogs", "News"),
Total_Words = c(
twitter_words,
blogs_words,
news_words
)
)
kable(word_summary)
| Dataset | Total_Words |
|---|---|
| 36727367 | |
| Blogs | 42829003 |
| News | 39911565 |
# c. Calculating the average line length of every datasets
twitter_avg <- mean(nchar(twitter))
blogs_avg <- mean(nchar(blogs))
news_avg <- mean(nchar(news))
twitter_max <- max(nchar(twitter))
blogs_max <- max(nchar(blogs))
news_max <- max(nchar(news))
length_summary <- data.frame(
Dataset = c("Twitter", "Blogs", "News"),
Average_Line_Length = c(twitter_avg, blogs_avg, news_avg),
Maximum_Line_Length = c(twitter_max, blogs_max, news_max)
)
kable(length_summary)
| Dataset | Average_Line_Length | Maximum_Line_Length |
|---|---|---|
| 68.68054 | 140 | |
| Blogs | 229.98695 | 40833 |
| News | 201.16149 | 11384 |
# d. Finding the most frequently used words in every dataset
twitter_dfm <- dfm(
tokens(twitter, remove_punct = TRUE)
)
twitter_frequency <- textstat_frequency(
twitter_dfm
)
head(twitter_frequency, 10)
## feature frequency rank docfreq group
## 1 the 936629 1 728820 all
## 2 to 787290 2 646496 all
## 3 i 722392 3 567102 all
## 4 a 609432 4 516311 all
## 5 you 547755 5 443863 all
## 6 and 438154 6 392395 all
## 7 for 385062 7 352551 all
## 8 in 377957 8 343303 all
## 9 of 359149 9 321191 all
## 10 is 358716 10 328180 all
blogs_dfm <- dfm(
tokens(blogs, remove_punct = TRUE)
)
blogs_frequency <- textstat_frequency(
blogs_dfm
)
head(blogs_frequency, 10)
## feature frequency rank docfreq group
## 1 the 1857822 1 551888 all
## 2 and 1091885 2 472315 all
## 3 to 1066177 3 456762 all
## 4 a 897901 4 421543 all
## 5 of 875281 5 408593 all
## 6 i 773896 6 304974 all
## 7 in 594151 7 336699 all
## 8 that 460355 8 269170 all
## 9 is 432394 9 259014 all
## 10 it 403140 10 241049 all
news_dfm <- dfm(
tokens(news, remove_punct = TRUE)
)
news_frequency <- textstat_frequency(
news_dfm
)
head(news_frequency, 10)
## feature frequency rank docfreq group
## 1 the 1971472 1 748188 all
## 2 to 901044 2 528219 all
## 3 and 884857 3 530946 all
## 4 a 875061 4 513376 all
## 5 of 771030 5 475648 all
## 6 in 674012 6 442164 all
## 7 for 351152 7 275934 all
## 8 that 346941 8 265046 all
## 9 is 284100 9 221213 all
## 10 on 266785 10 220150 all
# e. Creating visualization graphs like histogram for more Analysis for each datasets
# 1. Twitter Text Length Histogram
twitter_length <- data.frame(
Line_Length = nchar(twitter)
)
ggplot(
twitter_length,
aes(x = Line_Length)
) +
geom_histogram(
bins = 50,
fill = "steelblue",
color = "white"
) +
labs(
title = "Distribution of Twitter Text Lengths",
x = "Number of Characters",
y = "Number of Lines"
) +
theme_minimal()
# 2. Blogs Text Length Histogram
blogs_length <- data.frame(
Line_Length = nchar(blogs)
)
ggplot(
blogs_length,
aes(x = Line_Length)
) +
geom_histogram(
bins = 50,
fill = "steelblue",
color = "white"
) +
labs(
title = "Distribution of Blogs Text Lengths",
x = "Number of Characters",
y = "Number of Lines"
) +
theme_minimal()
# 3. News Text Length Histogram
news_length <- data.frame(
Line_Length = nchar(news)
)
ggplot(
news_length,
aes(x = Line_Length)
) +
geom_histogram(
bins = 50,
fill = "steelblue",
color = "white"
) +
labs(
title = "Distribution of News Text Lengths",
x = "Number of Characters",
y = "Number of Lines"
) +
theme_minimal()
# 4. Top 15 Twitter Words
top_words_twitter <- head(twitter_frequency, 15)
ggplot(
top_words_twitter,
aes(
x = reorder(feature, frequency),
y = frequency
)
) +
geom_col(fill = "steelblue") +
coord_flip() +
labs(
title = "Top 15 Most Frequent Words in Twitter",
x = "Words",
y = "Frequency"
) +
theme_minimal()
# 5. Top 15 Blogs Words
top_words_blogs <- head(blogs_frequency, 15)
ggplot(
top_words_blogs,
aes(
x = reorder(feature, frequency),
y = frequency
)
) +
geom_col(fill = "steelblue") +
coord_flip() +
labs(
title = "Top 15 Most Frequent Words in Blogs",
x = "Words",
y = "Frequency"
) +
theme_minimal()
# 6. Top 15 News Words
top_words_news <- head(news_frequency, 15)
ggplot(
top_words_news,
aes(
x = reorder(feature, frequency),
y = frequency
)
) +
geom_col(fill = "steelblue") +
coord_flip() +
labs(
title = "Top 15 Most Frequent Words in News",
x = "Words",
y = "Frequency"
) +
theme_minimal()
# 7. Dataset Line Count Comparison
ggplot(
line_summary,
aes(
x = Dataset,
y = Total_Lines,
fill = Dataset
)
) +
geom_col() +
labs(
title = "Number of Lines in Each Dataset",
x = "Dataset",
y = "Number of Lines"
) +
theme_minimal() +
theme(legend.position = "none")
The exploratory analysis can help see the differences between the three sources of text.
Twitter has very short and casual messages, Blogs and News have longer and more formal messages. Most used words can give insights on the language pattern, which can be used to help predict these word.
The line-length histogram shows you the variation in lengths of the text. The word-frequency charts show the most popular vocabulary.
This will allow us to determine how to parse the text and, which combinations of words will be of use to us for prediction.
The subsequent step will be to load and process the text data, separate sentences into words and create unigram, bigram, and trigram models.
Create an algorithm for word prediction that will provide the next word based on the user previous words.
Backoff will be used to deal with word combinations not encountered in training. Prediction accuracy, speed, and memory usage will also be tested.
Lastly, a Shiny app will be created to input text and return a prediction of the subsequent word.
This exploratory study is the first step in our understanding of the three datasets of English texts as well as a platform for the predictive text mining project.
The subsequent steps will be implementing the prediction algorithm efficiently and creating an interactive Shiny app for next-word prediction.