library(stringi)
library(ggplot2)

blogs <- readLines("~/Desktop/project/final/en_US/en_US.blogs.txt",
                   encoding = "UTF-8", skipNul = TRUE)

news <- readLines("~/Desktop/project/final/en_US/en_US.news.txt",
                  encoding = "UTF-8", skipNul = TRUE)

twitter <- readLines("~/Desktop/project/final/en_US/en_US.twitter.txt",
                     encoding = "UTF-8", skipNul = TRUE)

Introduction

This report presents an exploratory analysis of the English language datasets used for the Coursera Data Science Capstone project. The datasets contain text collected from blogs, news articles, and Twitter. The purpose of this analysis is to understand the characteristics of the data before developing a predictive text algorithm.

Data Summary

summary_table <- data.frame(
  Dataset = c("Blogs","News","Twitter"),
  Lines = c(length(blogs), length(news), length(twitter)),
  Words = c(sum(stri_count_words(blogs)),
            sum(stri_count_words(news)),
            sum(stri_count_words(twitter))),
  Characters = c(sum(nchar(blogs)),
                 sum(nchar(news)),
                 sum(nchar(twitter)))
)

summary_table
##   Dataset   Lines    Words Characters
## 1   Blogs  899288 37546250  206824505
## 2    News 1010242 34762395  203223159
## 3 Twitter 2360148 30093413  162096241

Distribution of Line Lengths

Blogs

ggplot(data.frame(chars=nchar(blogs)), aes(chars)) +
  geom_histogram(bins=50) +
  labs(title="Blog Line Length Distribution",
       x="Characters per line",
       y="Frequency")

News

ggplot(data.frame(chars=nchar(news)), aes(chars)) +
  geom_histogram(bins=50) +
  labs(title="News Line Length Distribution",
       x="Characters per line",
       y="Frequency")

Twitter

ggplot(data.frame(chars=nchar(twitter)), aes(chars)) +
  geom_histogram(bins=50) +
  labs(title="Twitter Line Length Distribution",
       x="Characters per line",
       y="Frequency")

Findings

The three datasets differ in writing style and message length. Blog entries generally contain longer text, news articles are moderately long and more formal, while Twitter posts are much shorter. These differences should be considered when building a predictive text model because each source contributes different language patterns.

Future Plans

The next stage of the project will involve cleaning the text by removing punctuation, numbers, extra spaces, and profanity. N-gram models will then be created to predict the next word based on previously typed words. Finally, the prediction model will be integrated into a Shiny application that provides real-time word suggestions to users.