Exploratory Analysis of Blogs, News, and Twitter Data

Overview

This report presents a preliminary analysis of the SwiftKey text datasets (blogs, news, and Twitter) for the Data Science Capstone prediction app. The goal is to understand the size, structure, and key characteristics of the data before building the prediction model and Shiny app.

Data Loading & Summary

The three English datasets were successfully loaded into R. A summary of line counts and word counts is shown below.

library(stringi)

blogs   <- readLines(file.path(data_dir, "en_US.blogs.txt"),   warn = FALSE, skipNul = TRUE)
news    <- readLines(file.path(data_dir, "en_US.news.txt"),    warn = FALSE, skipNul = TRUE)
twitter <- readLines(file.path(data_dir, "en_US.twitter.txt"), warn = FALSE, skipNul = TRUE)

summary_table <- data.frame(
  File  = c("Blogs", "News", "Twitter"),
  Lines = c(length(blogs), length(news), length(twitter)),
  Words = c(sum(stri_count_words(blogs)),
            sum(stri_count_words(news)),
            sum(stri_count_words(twitter)))
)
summary_table

##      File   Lines    Words
## 1   Blogs  899288 37546806
## 2    News 1010206 34761151
## 3 Twitter 2360148 30096690

Basic Plots

The histograms below show the distribution of line lengths (characters per line). Twitter is heavily right-skewed due to its character limit, while blogs and news have longer, more variable text.

par(mfrow = c(1, 3))
hist(nchar(blogs),   breaks = 50, main = "Blogs",   xlab = "Characters per line", col = "steelblue")
hist(nchar(news),    breaks = 50, main = "News",    xlab = "Characters per line", col = "darkgreen")
hist(nchar(twitter), breaks = 50, main = "Twitter", xlab = "Characters per line", col = "tomato")

Key Findings

• Twitter has the most lines but the shortest messages, and contains informal language, abbreviations, and slang.
• Blogs contain the largest total word count and a mix of formal and informal writing.
• News uses longer sentences and more formal vocabulary.
• All three datasets require cleaning: removing punctuation, numbers, profanity, and non-ASCII characters.

Plan for the Prediction Algorithm and Shiny App

Algorithm: I will build an N-gram model (unigrams, bigrams, and trigrams) from a cleaned sample of the data. A back-off strategy will be used to predict the next word: if a trigram match is not found, the model falls back to bigrams, then to unigrams.

Shiny App: The user will type a phrase into a text box, and the app will display the top 3 most likely next words. The app will be simple, fast, and deployed on shinyapps.io.