Introduction

This report presents the initial exploratory analysis of the datasets that will be used to create a text prediction algorithm and Shiny app. The goal of the project is to develop a predictive text model similar to the ones used in mobile phone keyboards. This document outlines the major features of the data, summarizes key findings, and describes the plan for developing the prediction algorithm and Shiny app.

Data Overview

The data consists of three text files sourced from blogs, news articles, and Twitter feeds. These files are large and contain a variety of text data, which will be used to train the predictive model. Here are the details of the datasets:

  1. Blogs
  2. News
  3. Twitter

Summary Statistics

library(stringr)
## Warning: package 'stringr' was built under R version 4.2.3
library(dplyr)
## Warning: package 'dplyr' was built under R version 4.2.3
## 
## Attaching package: 'dplyr'
## The following objects are masked from 'package:stats':
## 
##     filter, lag
## The following objects are masked from 'package:base':
## 
##     intersect, setdiff, setequal, union
blogs <- readLines("~/Desktop/Research/Dr. Raghavendra/Coursera/Data Science Capstone/final/en_US/en_US.blogs.txt", encoding = "UTF-8", warn = FALSE)
news <- readLines("~/Desktop/Research/Dr. Raghavendra/Coursera/Data Science Capstone/final/en_US/en_US.news.txt", encoding = "UTF-8", warn = FALSE)
twitter <- readLines("~/Desktop/Research/Dr. Raghavendra/Coursera/Data Science Capstone/final/en_US/en_US.twitter.txt", encoding = "UTF-8", warn = FALSE)

summary_stats <- data.frame(
  Dataset = c("Blogs", "News", "Twitter"),
  FileSize_MB = c(file.info("~/Desktop/Research/Dr. Raghavendra/Coursera/Data Science Capstone/final/en_US/en_US.blogs.txt")$size / 1e6,
                  file.info("~/Desktop/Research/Dr. Raghavendra/Coursera/Data Science Capstone/final/en_US/en_US.news.txt")$size / 1e6,
                  file.info("~/Desktop/Research/Dr. Raghavendra/Coursera/Data Science Capstone/final/en_US/en_US.twitter.txt")$size / 1e6),
  LineCount = c(length(blogs), length(news), length(twitter)),
  WordCount = c(sum(str_count(blogs, "\\S+")),
                sum(str_count(news, "\\S+")),
                sum(str_count(twitter, "\\S+"))),
  CharacterCount = c(sum(nchar(blogs)), sum(nchar(news)), sum(nchar(twitter)))
)

print(summary_stats)
##   Dataset FileSize_MB LineCount WordCount CharacterCount
## 1   Blogs    210.1600    899288  37334131      206824505
## 2    News    205.8119   1010242  34372530      203223159
## 3 Twitter    167.1053   2360148  30373543      162096031

These statistics highlight the substantial amount of text data available for analysis and model training.

Basic Data Tables

Below are basic data tables showing a snapshot of the first few lines from each dataset to illustrate the nature of the text data:

cat("**Blogs:**\n")
## **Blogs:**
cat(paste(head(blogs, 3), collapse="\n"))
## In the years thereafter, most of the Oil fields and platforms were named after pagan “gods”.
## We love you Mr. Brown.
## Chad has been awesome with the kids and holding down the fort while I work later than usual! The kids have been busy together playing Skylander on the XBox together, after Kyan cashed in his $$$ from his piggy bank. He wanted that game so bad and used his gift card from his birthday he has been saving and the money to get it (he never taps into that thing either, that is how we know he wanted it so bad). We made him count all of his money to make sure that he had enough! It was very cute to watch his reaction when he realized he did! He also does a very good job of letting Lola feel like she is playing too, by letting her switch out the characters! She loves it almost as much as him.
cat("\n\n**News:**\n")
## 
## 
## **News:**
cat(paste(head(news, 3), collapse="\n"))
## He wasn't home alone, apparently.
## The St. Louis plant had to close. It would die of old age. Workers had been making cars there since the onset of mass automotive production in the 1920s.
## WSU's plans quickly became a hot topic on local online sites. Though most people applauded plans for the new biomedical center, many deplored the potential loss of the building.
cat("\n\n**Twitter:**\n")
## 
## 
## **Twitter:**
cat(paste(head(twitter, 3), collapse="\n"))
## How are you? Btw thanks for the RT. You gonna be in DC anytime soon? Love to see you. Been way, way too long.
## When you meet someone special... you'll know. Your heart will beat more rapidly and you'll smile for no reason.
## they've decided its more fun if I don't.

Data Visualization

To gain insights into the distribution of word frequencies, I created histograms of word counts for each dataset. Below are the histograms for Blogs, News, and Twitter data.

Word Frequency Histograms:

get_word_counts <- function(text, n = 1000) {
  lines <- readLines(text, n = n, warn = FALSE)
  word_counts <- str_count(lines, "\\S+")
  return(word_counts)
}
blogs_word_counts <- get_word_counts("~/Desktop/Research/Dr. Raghavendra/Coursera/Data Science Capstone/final/en_US/en_US.blogs.txt")
news_word_counts <- get_word_counts("~/Desktop/Research/Dr. Raghavendra/Coursera/Data Science Capstone/final/en_US/en_US.news.txt")
twitter_word_counts <- get_word_counts("~/Desktop/Research/Dr. Raghavendra/Coursera/Data Science Capstone/final/en_US/en_US.twitter.txt")
par(mfrow = c(3, 1)) # Arrange plots in a 3x1 grid

  • Blogs:

    hist(blogs_word_counts, main = "Histogram of Word Counts in Blogs",
     xlab = "Number of Words per Line", col = "blue", breaks = 30)

  • News:

    hist(news_word_counts, main = "Histogram of Word Counts in News",
     xlab = "Number of Words per Line", col = "green", breaks = 30)

  • Twitter:

    hist(twitter_word_counts, main = "Histogram of Word Counts in Twitter",
     xlab = "Number of Words per Line", col = "red", breaks = 30)

Interesting Findings

  1. High Variability in Text Length: Twitter data has shorter text segments compared to blogs and news articles due to the 280-character limit, which significantly impacts the structure of the text.

  2. Common Words Across Datasets: Despite the different sources, there are common words that appear frequently in all datasets. This will be useful in building a generalized predictive model.

  3. Noise in Data: There are non-standard words, hashtags, and user mentions in the Twitter dataset that will need to be handled appropriately during pre-processing.

Plans for Prediction Algorithm and Shiny App

Prediction Algorithm

The predictive model will be built using Natural Language Processing (NLP) techniques. Here are the key steps:

  1. Data Cleaning: Remove special characters, stop words, and perform tokenization.

  2. N-Gram Model: Create n-grams (bi-grams, tri-grams) to understand word sequences.

  3. Frequency Analysis: Calculate the frequency of n-grams to determine the most likely next word.

  4. Model Training: Use machine learning algorithms such as Random Forests or Neural Networks to train the predictive model.

Shiny App

The Shiny app will provide a user-friendly interface for the text prediction model. Key features will include:

  1. Text Input: Allow users to input text and receive predictions for the next word.

  2. Real-Time Prediction: Display predictions as the user types.

  3. Customizable Settings: Options for users to adjust the sensitivity of predictions.

Conclusion

This exploratory analysis has provided valuable insights into the datasets and established a clear plan for developing the text prediction algorithm and Shiny app. The next steps involve detailed data cleaning, feature extraction, and model training. Feedback on this plan is welcomed to ensure the project is on the right track.