Data Science Capstone: Exploratory Data Analysis

1. Introduction

This report documents the exploratory analysis performed on the dataset for the Data Science Capstone project. It highlights the dataset’s main features, initial findings, and a proposed plan for developing a predictive text model and Shiny application. The content is tailored for a non-technical audience to provide clarity and insight into the progress made so far.

2. Data Overview

The dataset consists of text data derived from three primary sources:

1. Blogs

2. News articles

3. Twitter posts

The data was downloaded from the Capstone project resources and successfully loaded into R for analysis using libraries such as “tm” and “stringi”.

3. Exploratory Data Analysis

a. Summary Statistics

Key metrics for each dataset were calculated, including the number of lines, word counts, and total characters. A summary is presented in the table below:

*Load required libraries

library(ggplot2)
library(stringi)

*Load and format the datasets

blogs<-readLines("/Users/ritam/Downloads/final/en_US/en_US.blogs.txt",, encoding = "UTF-8", skipNul = TRUE, warn=FALSE)
news<-readLines("/Users/ritam/Downloads/final/en_US/en_US.news.txt", encoding = "UTF-8", skipNul = TRUE, warn=FALSE)
twitter<-readLines("/Users/ritam/Downloads/final/en_US/en_US.twitter.txt", encoding = "UTF-8", skipNul = TRUE, warn=FALSE)

*Summary statistics

blogs_summary <- c(lines = length(blogs), words = sum(stri_count_words(blogs)), characters = sum(nchar(blogs)))
news_summary <- c(lines = length(news), words = sum(stri_count_words(news)), characters = sum(nchar(news)))
twitter_summary <- c(lines = length(twitter), words = sum(stri_count_words(twitter)), characters = sum(nchar(twitter)))

*Create table

summary_table <- data.frame(
  Source = c("Blogs", "News", "Twitter"),
  Lines = c(blogs_summary["lines"], news_summary["lines"], twitter_summary["lines"]),
  Words = c(blogs_summary["words"], news_summary["words"], twitter_summary["words"]),
  Characters = c(blogs_summary["characters"], news_summary["characters"], twitter_summary["characters"])
)
summary_table

##    Source   Lines    Words Characters
## 1   Blogs  899288 37546250  206824505
## 2    News 1010242 34762395  203223159
## 3 Twitter 2360148 30093413  162096241

b. Visualization:

Word Count Distribution

The distribution of word counts across lines in each dataset was visualized to identify patterns and anomalies.

*Histograms for word counts

hist(stri_count_words(blogs), breaks = 50, main = "Word Count Distribution: Blogs", xlab = "Word Count", col = "blue")

hist(stri_count_words(news), breaks = 50, main = "Word Count Distribution: News", xlab = "Word Count", col = "green")

hist(stri_count_words(twitter), breaks = 50, main = "Word Count Distribution: Twitter", xlab = "Word Count", col = "red")

Line Length Comparison

Box plots were created to compare the line lengths across the three datasets.

*Boxplot for line lengths

boxplot(nchar(blogs), nchar(news), nchar(twitter), names = c("Blogs", "News", "Twitter"),
        main = "Line Length Comparison", ylab = "Number of Characters", col = c("blue", "green", "red"))

c. Preprocessing

Initial data cleaning included:

• Removing punctuation, numbers, and special characters.

• Converting text to lowercase.

• Tokenizing into unigrams, bigrams, and trigrams for further analysis.

4. Key Findings

• Blogs exhibit longer lines and richer vocabulary compared to the other datasets.

• Twitter’s character limits result in concise, repetitive word patterns.

• News articles have a balanced distribution of words and line lengths, reflecting a structured format.

5. Future Steps

a. Predictive Algorithm

• Approach: Implement an n-gram model to predict the next word based on preceding text.

• Techniques:

  • Generate frequency tables for unigrams, bigrams, and trigrams.

  • Apply smoothing algorithms (e.g., Katz Backoff) to manage unseen data.

  • Optimize for both performance and accuracy.

b. Shiny Application

• Features:

  • Real-time next-word predictions based on user input.

  • Display of the top three suggestions.

• Design:

  • Clean and intuitive interface with input and output panels.

6. Conclusion

The exploratory analysis provides a strong foundation for the project. The insights gained will guide the development of an efficient predictive model and user-friendly Shiny application.