Text Prediction App: Exploratory Data Analysis

Overview

This report is an early checkpoint for a text prediction app being built using the SwiftKey dataset (English blogs, news, and Twitter text). The purpose here is simple: show that the data has been downloaded and loaded successfully, summarize its basic structure, share a few interesting patterns, and outline the plan for building the prediction algorithm and Shiny app. This is not the final product — just a progress check.

1. Loading the Data

The dataset contains three English-language text files: blogs, news, and tweets. Below, each file is read in and some basic file-level statistics are calculated (file size, number of lines, number of words).

# Update these paths to match where you've unzipped the Coursera-SwiftKey data
blogs_path   <- "/Users/investguru/final/en_US/en_US.blogs.txt"
news_path    <- "/Users/investguru/final/en_US/en_US.news.txt"
twitter_path <- "/Users/investguru/final/en_US/en_US.twitter.txt"

blogs   <- readLines(blogs_path, encoding = "UTF-8", skipNul = TRUE, warn = FALSE)
news    <- readLines(news_path, encoding = "UTF-8", skipNul = TRUE, warn = FALSE)
twitter <- readLines(twitter_path, encoding = "UTF-8", skipNul = TRUE, warn = FALSE)

2. Basic Summary Statistics

Here’s a quick table comparing the three files: file size (MB), number of lines, total word count, and average words per line.

file_size_mb <- function(path) round(file.info(path)$size / 1024^2, 1)

word_counts <- c(
  sum(stri_count_words(blogs)),
  sum(stri_count_words(news)),
  sum(stri_count_words(twitter))
)

line_counts <- c(length(blogs), length(news), length(twitter))

summary_df <- data.frame(
  File           = c("Blogs", "News", "Twitter"),
  Size_MB        = c(file_size_mb(blogs_path), file_size_mb(news_path), file_size_mb(twitter_path)),
  Lines          = line_counts,
  Words          = word_counts,
  Avg_Words_Line = round(word_counts / line_counts, 1)
)

knitr::kable(summary_df, caption = "Basic summary statistics for the three source files")

Basic summary statistics for the three source files
File	Size_MB	Lines	Words	Avg_Words_Line
Blogs	200.4	899288	37546250	41.8
News	196.3	1010242	34762395	34.4
Twitter	159.4	2360148	30093413	12.8

Looking at the numbers above, Twitter has by far the most lines (2.36 million) but the shortest average line length at just 12.8 words — which makes sense given the character limits users are used to working within. Blogs and news entries run much longer per line, averaging 41.8 and 34.4 words respectively, suggesting more complete thoughts and sentences, likely because they’re written and edited rather than typed quickly on a phone. Combined, these three files add up to over 100 million words, so working from a random sample for the rest of this analysis makes sense for speed, without losing the overall patterns in the data.

3. Sampling the Data

Because the full files are large, a random sample is drawn from each for the exploratory work below. This keeps the analysis fast without losing the overall patterns in the data.

set.seed(1234)
sample_pct <- 0.01  # 1% sample; adjust as needed

sample_lines <- function(x, pct) {
  x[rbinom(length(x), 1, pct) == 1]
}

blogs_sample   <- sample_lines(blogs, sample_pct)
news_sample    <- sample_lines(news, sample_pct)
twitter_sample <- sample_lines(twitter, sample_pct)

combined_sample <- c(blogs_sample, news_sample, twitter_sample)

4. Distribution of Line Lengths

A histogram of word counts per line gives a sense of how long a typical entry is in each source.

line_lengths <- data.frame(
  words  = c(stri_count_words(blogs_sample), stri_count_words(news_sample), stri_count_words(twitter_sample)),
  source = c(rep("Blogs", length(blogs_sample)),
             rep("News", length(news_sample)),
             rep("Twitter", length(twitter_sample)))
)

ggplot(line_lengths, aes(x = words, fill = source)) +
  geom_histogram(binwidth = 5, alpha = 0.6, position = "identity") +
  xlim(0, 100) +
  labs(title = "Distribution of Words per Line by Source",
       x = "Words per Line", y = "Count") +
  theme_minimal()

5. Most Common Words

To understand what the prediction model will be working with, it helps to look at the most frequently used single words across the sample.

library(tidytext)

words_df <- data.frame(text = combined_sample, stringsAsFactors = FALSE) %>%
  unnest_tokens(word, text) %>%
  count(word, sort = TRUE)

top20 <- head(words_df, 20)

ggplot(top20, aes(x = reorder(word, n), y = n)) +
  geom_col(fill = "steelblue") +
  coord_flip() +
  labs(title = "Top 20 Most Frequent Words", x = "", y = "Frequency") +
  theme_minimal()

As expected, the most frequent words are short, common function words like “the,” “and,” and “to” rather than anything topic-specific. This confirms that if the final model weights predictions purely on frequency, it’ll lean heavily toward these common connector words — something to keep in mind when designing how the app suggests the next word, since always suggesting “the” isn’t very useful in practice.

6. Word Coverage

A useful question for building the predictive model: how many unique words are needed to cover a given percentage of all word instances in the data? This helps decide how large the model’s vocabulary needs to be.

words_df <- words_df %>%
  mutate(cum_freq = cumsum(n) / sum(n))

words_for_50pct <- which(words_df$cum_freq >= 0.5)[1]
words_for_90pct <- which(words_df$cum_freq >= 0.9)[1]

coverage_df <- data.frame(
  Coverage = c("50%", "90%"),
  Unique_Words_Needed = c(words_for_50pct, words_for_90pct)
)

knitr::kable(coverage_df, caption = "Number of unique words needed to cover a given percentage of all word instances")

Number of unique words needed to cover a given percentage of all word instances
Coverage	Unique_Words_Needed
50%	155
90%	7588

These numbers show that just 155 unique words already cover half of everything people write, and only about 7,600 unique words are needed to cover 90% of all word instances. This is a good sign for the prediction model: it means the app doesn’t need to memorize the entire English language to be useful most of the time, since a fairly compact set of common words and phrases handles the vast majority of real-world typing.

7. Plans for the Prediction Algorithm and Shiny App

In plain terms, here’s the plan going forward:

Build an n-gram model. Rather than looking at single words, the next step is looking at pairs and triples of words (bigrams, trigrams) to capture how words tend to follow each other. This is the foundation of predicting “what comes next.”
Handle unseen word combinations. Not every possible word combination will show up in the training data, so a technique called “smoothing” (e.g., Katz backoff or similar) will be used to make reasonable predictions even for word sequences the model hasn’t seen before.
Balance size versus speed. The final app needs to run quickly and can’t carry the entire dataset with it, so the model will be pruned down to the most useful and common word patterns, trading a small amount of accuracy for a much faster, lighter app.
Build a Shiny app. The final deliverable is a simple web app where a user types a partial phrase and the app suggests the most likely next word, updating as they type.
Test and refine. Once a first version works, it’ll be tested against sample phrases to check accuracy and adjusted as needed before final submission.

Summary

This report shows the three data files have been successfully downloaded and loaded, provides basic statistics and visualizations describing their structure, and outlines a clear next-step plan for building an n-gram based prediction model and accompanying Shiny app.