1. Introduction

Our goal for this project is to examine a set of English text data and clean and process it to use for building a predictive text application. The predictive text app will be able to provide a suggestion for the next word while the user is typing.

This experimental analysis aims to explore data sets in detail, look at the features of data sets, is working on a list of the most popular words, and is making a plan for the prediction algorithm.

2. Dataset Description

The study used 3 datasets in the English language:

These datasets will give different writing styles and wording to help train a better model of predicting what will come up.

#Installing the necessary packages

install.packages(“quanteda”) install.packages(“stringr”) install.packages(“ggplot2”) install.packages(“knitr”)

# Load required packages
library(ggplot2)
library(quanteda)
library(knitr)
library(quanteda.textstats)
library(stringr)  

# Load the three datasets
twitter <- readLines("en_US.twitter.txt",
                     encoding = "UTF-8",
                     skipNul = TRUE)

blogs <- readLines("en_US.blogs.txt",
                   encoding = "UTF-8",
                   skipNul = TRUE)

news <- readLines("en_US.news.txt",
                  encoding = "UTF-8",
                  skipNul = TRUE)





# 3. Exploratory Data Analysis

#The three datasets were imported and viewed in R.

#The analysis included:

#- All lines in each dataset.

#- total words,

#- Average and maximum line length.

#- Similar words that are often used.

#- Length of text distribution.

#Had some tables and graphs constructed to make comparisons between datasets.



# a. Calculating line counts in all three data sets 

twitter_lines <- length(twitter)
blogs_lines <- length(blogs)
news_lines <- length(news)

line_summary <- data.frame(
  Dataset = c("Twitter", "Blogs", "News"),
  Total_Lines = c(
    twitter_lines,
    blogs_lines,
    news_lines
  )
)

kable(line_summary)
Dataset Total_Lines
Twitter 2360148
Blogs 899288
News 1010206
# b. Calculating Word counts in all three data sets 


twitter_tokens <- tokens(twitter)
blogs_tokens <- tokens(blogs)
news_tokens <- tokens(news)

twitter_words <- sum(ntoken(twitter_tokens))
blogs_words <- sum(ntoken(blogs_tokens))
news_words <- sum(ntoken(news_tokens))

word_summary <- data.frame(
  Dataset = c("Twitter", "Blogs", "News"),
  Total_Words = c(
    twitter_words,
    blogs_words,
    news_words
  )
)

kable(word_summary)
Dataset Total_Words
Twitter 36727367
Blogs 42829003
News 39911565
# c. Calculating the average line length of every datasets

twitter_avg <- mean(nchar(twitter))
blogs_avg <- mean(nchar(blogs))
news_avg <- mean(nchar(news))


twitter_max <- max(nchar(twitter))
blogs_max <- max(nchar(blogs))
news_max <- max(nchar(news))


length_summary <- data.frame(
  Dataset = c("Twitter", "Blogs", "News"),
  Average_Line_Length = c(twitter_avg, blogs_avg, news_avg),
  Maximum_Line_Length = c(twitter_max, blogs_max, news_max)
)


kable(length_summary)
Dataset Average_Line_Length Maximum_Line_Length
Twitter 68.68054 140
Blogs 229.98695 40833
News 201.16149 11384
# d. Finding the most frequently used words in every dataset

twitter_dfm <- dfm(
  tokens(twitter, remove_punct = TRUE)
)

twitter_frequency <- textstat_frequency(
  twitter_dfm
)

head(twitter_frequency, 10)
##    feature frequency rank docfreq group
## 1      the    936629    1  728820   all
## 2       to    787290    2  646496   all
## 3        i    722392    3  567102   all
## 4        a    609432    4  516311   all
## 5      you    547755    5  443863   all
## 6      and    438154    6  392395   all
## 7      for    385062    7  352551   all
## 8       in    377957    8  343303   all
## 9       of    359149    9  321191   all
## 10      is    358716   10  328180   all
blogs_dfm <- dfm(
  tokens(blogs, remove_punct = TRUE)
)

blogs_frequency <- textstat_frequency(
  blogs_dfm
)

head(blogs_frequency, 10)
##    feature frequency rank docfreq group
## 1      the   1857822    1  551888   all
## 2      and   1091885    2  472315   all
## 3       to   1066177    3  456762   all
## 4        a    897901    4  421543   all
## 5       of    875281    5  408593   all
## 6        i    773896    6  304974   all
## 7       in    594151    7  336699   all
## 8     that    460355    8  269170   all
## 9       is    432394    9  259014   all
## 10      it    403140   10  241049   all
news_dfm <- dfm(
  tokens(news, remove_punct = TRUE)
)

news_frequency <- textstat_frequency(
  news_dfm
)

head(news_frequency, 10)
##    feature frequency rank docfreq group
## 1      the   1971472    1  748188   all
## 2       to    901044    2  528219   all
## 3      and    884857    3  530946   all
## 4        a    875061    4  513376   all
## 5       of    771030    5  475648   all
## 6       in    674012    6  442164   all
## 7      for    351152    7  275934   all
## 8     that    346941    8  265046   all
## 9       is    284100    9  221213   all
## 10      on    266785   10  220150   all
# e. Creating visualization graphs like histogram for more Analysis for each datasets


# 1. Twitter Text Length Histogram

twitter_length <- data.frame(
  Line_Length = nchar(twitter)
)

ggplot(
  twitter_length,
  aes(x = Line_Length)
) +
  geom_histogram(
    bins = 50,
    fill = "steelblue",
    color = "white"
  ) +
  labs(
    title = "Distribution of Twitter Text Lengths",
    x = "Number of Characters",
    y = "Number of Lines"
  ) +
  theme_minimal()

# 2. Blogs Text Length Histogram


blogs_length <- data.frame(
  Line_Length = nchar(blogs)
)

ggplot(
  blogs_length,
  aes(x = Line_Length)
) +
  geom_histogram(
    bins = 50,
    fill = "steelblue",
    color = "white"
  ) +
  labs(
    title = "Distribution of Blogs Text Lengths",
    x = "Number of Characters",
    y = "Number of Lines"
  ) +
  theme_minimal()

# 3. News Text Length Histogram


news_length <- data.frame(
  Line_Length = nchar(news)
)

ggplot(
  news_length,
  aes(x = Line_Length)
) +
  geom_histogram(
    bins = 50,
    fill = "steelblue",
    color = "white"
  ) +
  labs(
    title = "Distribution of News Text Lengths",
    x = "Number of Characters",
    y = "Number of Lines"
  ) +
  theme_minimal()

# 4. Top 15 Twitter Words


top_words_twitter <- head(twitter_frequency, 15)

ggplot(
  top_words_twitter,
  aes(
    x = reorder(feature, frequency),
    y = frequency
  )
) +
  geom_col(fill = "steelblue") +
  coord_flip() +
  labs(
    title = "Top 15 Most Frequent Words in Twitter",
    x = "Words",
    y = "Frequency"
  ) +
  theme_minimal()

# 5. Top 15 Blogs Words


top_words_blogs <- head(blogs_frequency, 15)

ggplot(
  top_words_blogs,
  aes(
    x = reorder(feature, frequency),
    y = frequency
  )
) +
  geom_col(fill = "steelblue") +
  coord_flip() +
  labs(
    title = "Top 15 Most Frequent Words in Blogs",
    x = "Words",
    y = "Frequency"
  ) +
  theme_minimal()

# 6. Top 15 News Words


top_words_news <- head(news_frequency, 15)

ggplot(
  top_words_news,
  aes(
    x = reorder(feature, frequency),
    y = frequency
  )
) +
  geom_col(fill = "steelblue") +
  coord_flip() +
  labs(
    title = "Top 15 Most Frequent Words in News",
    x = "Words",
    y = "Frequency"
  ) +
  theme_minimal()

# 7. Dataset Line Count Comparison


ggplot(
  line_summary,
  aes(
    x = Dataset,
    y = Total_Lines,
    fill = Dataset
  )
) +
  geom_col() +
  labs(
    title = "Number of Lines in Each Dataset",
    x = "Dataset",
    y = "Number of Lines"
  ) +
  theme_minimal() +
  theme(legend.position = "none")

4. Key Findings

The exploratory analysis can help see the differences between the three sources of text.

Twitter has very short and casual messages, Blogs and News have longer and more formal messages. Most used words can give insights on the language pattern, which can be used to help predict these word.

The line-length histogram shows you the variation in lengths of the text. The word-frequency charts show the most popular vocabulary.

This will allow us to determine how to parse the text and, which combinations of words will be of use to us for prediction.

5. Future Project Plan

The subsequent step will be to load and process the text data, separate sentences into words and create unigram, bigram, and trigram models.

Create an algorithm for word prediction that will provide the next word based on the user previous words.

Backoff will be used to deal with word combinations not encountered in training. Prediction accuracy, speed, and memory usage will also be tested.

Lastly, a Shiny app will be created to input text and return a prediction of the subsequent word.

6. Conclusion

This exploratory study is the first step in our understanding of the three datasets of English texts as well as a platform for the predictive text mining project.

The subsequent steps will be implementing the prediction algorithm efficiently and creating an interactive Shiny app for next-word prediction.