Introduction

The goal of this project is to explore the data and develop a plan for building a predictive text algorithm.

The analysis focuses on three different sources of English text: blogs, news, and Twitter. The main purpose is to understand the size and characteristics of these datasets before developing the final prediction algorithm.

Data Sources

The data consists of three text files:

These datasets contain English language text from different sources and represent different writing styles.

Loading the Data

blogs <- readLines("en_US.blogs.txt",
                   encoding = "UTF-8",
                   skipNul = TRUE)

news <- readLines("en_US.news.txt",
                  encoding = "UTF-8",
                  skipNul = TRUE)

twitter <- readLines("en_US.twitter.txt",
                     encoding = "UTF-8",
                     skipNul = TRUE)

Number of Lines

The number of lines in each dataset provides a basic measure of the amount of text available for analysis.

line_counts <- data.frame(
  Dataset = c("Blogs", "News", "Twitter"),
  Lines = c(
    length(blogs),
    length(news),
    length(twitter)
  )
)

line_counts
##   Dataset   Lines
## 1   Blogs  899288
## 2    News 1010206
## 3 Twitter 2360148

Word Counts

The total number of words was calculated for each dataset.

word_counts <- data.frame(
  Dataset = c("Blogs", "News", "Twitter"),
  Words = c(
    sum(sapply(strsplit(blogs, "\\s+"), length)),
    sum(sapply(strsplit(news, "\\s+"), length)),
    sum(sapply(strsplit(twitter, "\\s+"), length))
  )
)

word_counts
##   Dataset    Words
## 1   Blogs 37334131
## 2    News 34371031
## 3 Twitter 30373583

Basic Data Summary

The following table combines the number of lines, total words, and average number of words per line for each data source.

data_summary <- data.frame(
  Dataset = c("Blogs", "News", "Twitter"),
  Lines = c(
    length(blogs),
    length(news),
    length(twitter)
  ),
  Words = c(
    sum(sapply(strsplit(blogs, "\\s+"), length)),
    sum(sapply(strsplit(news, "\\s+"), length)),
    sum(sapply(strsplit(twitter, "\\s+"), length))
  )
)

data_summary$Average_Words_Per_Line <- round(
  data_summary$Words / data_summary$Lines,
  2
)

data_summary
##   Dataset   Lines    Words Average_Words_Per_Line
## 1   Blogs  899288 37334131                  41.52
## 2    News 1010206 34371031                  34.02
## 3 Twitter 2360148 30373583                  12.87

Data Set Size Comparison

The following plot compares the number of lines in the three datasets.

barplot(
  data_summary$Lines,
  names.arg = data_summary$Dataset,
  main = "Number of Lines in Each Dataset",
  xlab = "Dataset",
  ylab = "Number of Lines"
)

Average Words Per Line

The following plot compares the average number of words per line.

barplot(
  data_summary$Average_Words_Per_Line,
  names.arg = data_summary$Dataset,
  main = "Average Words Per Line",
  xlab = "Dataset",
  ylab = "Average Words Per Line"
)

Interesting Findings

Exploratory analysis revealed significant differences between the three data sources.

Twitter has the most lines, with approximately 2.36 million lines. However, each Twitter line is comparatively short, with an average of about 12.87 words.

Blogs have fewer lines than Twitter, but they have the longest lines, with an average of about 41.52 words per line. News data is between the two sources, with an average of about 34.02 words per line.

These differences suggest that the three sources have different writing styles and sentence structures. Twitter contains shorter and more informal text, while blogs and news generally contain longer text.

Therefore, the prediction algorithm will need to handle different writing styles, sentence lengths, and word patterns.

Plans for the Prediction Algorithm

In the next stage of the project, a predictive text algorithm will be developed.

First, the text data will be cleaned and prepared. This will include handling punctuation, unnecessary characters, and differences between uppercase and lowercase letters.

Next, word combinations such as unigrams, bigrams, and trigrams will be created from the text data. These word patterns will be used to identify common sequences of words.

The prediction algorithm will use these patterns to predict the next most likely word based on the words entered by the user.

A separate validation sample will be used to evaluate the performance of the model. This will help determine which approach provides accurate predictions while maintaining reasonable speed.

Plans for the Shiny Application

The final prediction algorithm will be integrated into a Shiny application.

The application will contain a simple text input box where the user can enter a sentence or phrase. Based on the user’s input, the application will display the most likely next words.

The main goal will be to make the application simple, fast, and easy to use, even for users without technical knowledge.

Conclusion

The exploratory analysis successfully demonstrates that the three datasets have substantial differences in size and writing style.

The analysis provides a foundation for developing the predictive text algorithm. The next steps will focus on text cleaning, creation of n-grams, model development, validation, and integration with a Shiny application.