1 Executive summary

This report explores three English-language text files commonly used for the SwiftKey text-prediction capstone: blogs, news, and Twitter messages. The purpose is to confirm that the data can be downloaded and read, describe the size and basic characteristics of each source, identify common words and differences between sources, and outline a practical approach for building a next-word prediction model and a Shiny application.

The analysis below is generated from the source files when this document is knitted. This keeps all reported counts and charts tied to the actual data rather than relying on estimates.

2 1. Data acquisition and loading

The analysis uses the public Coursera-SwiftKey data archive. The code downloads the archive if it is not already present, extracts it, and locates the three English-language files.

options(stringsAsFactors = FALSE)
data_dir <- file.path(path.expand("~"), "data")
dir.create(data_dir, showWarnings = FALSE)

zip_file <- file.path(data_dir, "Coursera-SwiftKey.zip")
download_url <- "https://d396qusza40orc.cloudfront.net/dsscapstone/dataset/Coursera-SwiftKey.zip"

if (!file.exists(zip_file)) {
  download.file(download_url, destfile = zip_file, mode = "wb")
}

unzip_dir <- file.path(data_dir, "final")
if (!dir.exists(unzip_dir)) {
  unzip(zip_file, exdir = data_dir)
}

# Locate the three English files, regardless of the exact extracted folder path.
all_txt <- list.files(data_dir, pattern = "\\.txt$", recursive = TRUE, full.names = TRUE)
find_file <- function(pattern) {
  hits <- all_txt[grepl(pattern, basename(all_txt), ignore.case = TRUE)]
  hits <- hits[grepl("en_US", hits, ignore.case = TRUE)]
  if (length(hits) == 0) stop(paste("Could not find file matching:", pattern))
  hits[1]
}

files <- c(
  Blogs = find_file("blogs"),
  News = find_file("news"),
  Twitter = find_file("twitter")
)

# Read each file as lines. The original text is kept unchanged for the initial summaries.
text_data <- lapply(files, readLines, encoding = "UTF-8", warn = FALSE)

data.frame(
  Source = names(files),
  File = basename(files),
  Loaded = file.exists(files),
  stringsAsFactors = FALSE
)
##    Source              File Loaded
## 1   Blogs   en_US.blogs.txt   TRUE
## 2    News    en_US.news.txt   TRUE
## 3 Twitter en_US.twitter.txt   TRUE

The table confirms which source files were located and loaded. The three sources represent different writing contexts: longer informal blog posts, edited news writing, and short social-media messages.

3 2. Basic data summaries

For each file, this section reports the number of lines, approximate word count, and file size. Word counts are calculated by splitting each non-empty line on whitespace; punctuation is removed only in the later word-frequency analysis.

count_words <- function(lines) {
  nonempty <- lines[nzchar(trimws(lines))]
  if (length(nonempty) == 0) return(0L)
  sum(lengths(strsplit(trimws(nonempty), "\\\\s+")))
}

summary_table <- data.frame(
  Source = names(text_data),
  Lines = vapply(text_data, length, integer(1)),
  Words_approx = vapply(text_data, count_words, integer(1)),
  Size_MB = round(file.info(files)$size / 1024^2, 2),
  row.names = NULL
)

knitr::kable(summary_table, caption = "Basic size summary of the three text files")
Basic size summary of the three text files
Source Lines Words_approx Size_MB
Blogs 899288 899288 200.42
News 1010206 1010206 196.28
Twitter 2360148 2360151 159.36 
if (!requireNamespace("ggplot2", quietly = TRUE)) {
  install.packages("ggplot2", repos = "https://cloud.r-project.org")
}
library(ggplot2)

ggplot(summary_table, aes(x = Source, y = Lines)) +
  geom_col(fill = "steelblue") +
  labs(title = "Number of text lines by source", x = NULL, y = "Lines") +
  theme_minimal()

The line-count chart makes the relative scale of the sources easy to compare. The table should be used alongside the chart because line counts and word counts measure different things: a Twitter line is generally much shorter than a blog or news line.

4 3. Text characteristics and exploratory analysis

4.1 3.1 Word-length distribution

The next chart examines the distribution of word lengths after basic token cleanup. Tokens are converted to lowercase and characters other than letters or apostrophes are removed. This is a lightweight exploratory cleaning step, not the final preprocessing pipeline.

clean_tokens <- function(lines) {
  x <- tolower(paste(lines, collapse = " "))
  tokens <- unlist(strsplit(x, "[^[:alpha:]']+"))
  tokens <- tokens[nzchar(tokens)]
  tokens
}

# Use a reproducible sample for detailed token analysis so the report remains
# practical on computers with limited memory. Full-file line and word counts
# are still reported in Section 2.
set.seed(2026)
sampled_text <- lapply(text_data, function(lines) {
  lines <- lines[nzchar(trimws(lines))]
  lines[sample.int(length(lines), min(length(lines), 100000))]
})
tokens_by_source <- lapply(sampled_text, clean_tokens)

token_summary <- data.frame(
  Source = names(tokens_by_source),
  Sample_Lines = vapply(sampled_text, length, integer(1)),
  Tokens_in_Sample = vapply(tokens_by_source, length, integer(1)),
  Unique_Tokens_in_Sample = vapply(tokens_by_source, function(x) length(unique(x)), integer(1)),
  row.names = NULL
)

knitr::kable(token_summary, caption = "Token counts after basic cleanup")
Token counts after basic cleanup
Source Sample_Lines Tokens_in_Sample Unique_Tokens_in_Sample
Blogs 1e+05 4176276 94600
News 1e+05 3374506 88040
Twitter 1e+05 1265011 55192 
word_lengths <- do.call(rbind, lapply(names(tokens_by_source), function(source) {
  data.frame(Source = source, Word_Length = nchar(tokens_by_source[[source]]))
}))

ggplot(word_lengths, aes(x = Word_Length)) +
  geom_histogram(binwidth = 1, boundary = 0, fill = "darkseagreen3", color = "white") +
  facet_wrap(~ Source, ncol = 1) +
  coord_cartesian(xlim = c(1, 20)) +
  labs(title = "Distribution of word lengths", x = "Characters per token", y = "Token count") +
  theme_minimal()

This plot helps identify the typical length of words in each source. Very long tokens can include unusual strings, URLs, or artifacts, so they should be reviewed before final model training.

4.2 3.2 Most frequent words

The following tables show the 15 most frequent cleaned tokens in a reproducible sample of up to 100,000 non-empty lines per source. These counts are useful for understanding vocabulary and identifying common words. Common function words (such as “the”, “and”, and “to”) are expected to appear frequently; they are not automatically removed because they are important to next-word prediction.

top_n_words <- function(tokens, n = 15) {
  tab <- sort(table(tokens), decreasing = TRUE)
  head(data.frame(Word = names(tab), Count = as.integer(tab)), n)
}

for (source in names(tokens_by_source)) {
  cat("\n\n###", source, "\n\n")
  print(knitr::kable(top_n_words(tokens_by_source[[source]]),
                     caption = paste("Most frequent words:", source)))
}
## 
## 
## ### Blogs 
## 
## 
## 
## Table: Most frequent words: Blogs
## 
## |Word |  Count|
## |:----|------:|
## |the  | 207502|
## |and  | 121648|
## |to   | 118523|
## |a    | 100333|
## |of   |  97601|
## |i    |  92425|
## |in   |  67254|
## |that |  52779|
## |it   |  49391|
## |is   |  48223|
## |for  |  40721|
## |you  |  34910|
## |with |  31778|
## |was  |  31184|
## |on   |  30620|
## 
## 
## ### News 
## 
## 
## 
## Table: Most frequent words: News
## 
## |Word |  Count|
## |:----|------:|
## |the  | 195843|
## |to   |  89929|
## |a    |  88346|
## |and  |  87926|
## |of   |  77071|
## |in   |  67057|
## |that |  34845|
## |for  |  34820|
## |is   |  28191|
## |on   |  26878|
## |with |  25136|
## |said |  25039|
## |it   |  23058|
## |he   |  22809|
## |was  |  22417|
## 
## 
## ### Twitter 
## 
## 
## 
## Table: Most frequent words: Twitter
## 
## |Word | Count|
## |:----|-----:|
## |the  | 40022|
## |to   | 33686|
## |i    | 31025|
## |a    | 26122|
## |you  | 23218|
## |and  | 18700|
## |for  | 16403|
## |in   | 16036|
## |of   | 15145|
## |is   | 15127|
## |it   | 12614|
## |my   | 12520|
## |on   | 11827|
## |that | 10024|
## |me   |  8614|
top_plot_data <- do.call(rbind, lapply(names(tokens_by_source), function(source) {
  top <- top_n_words(tokens_by_source[[source]], 10)
  data.frame(Source = source, Word = top$Word, Count = top$Count)
}))

ggplot(top_plot_data, aes(x = reorder(Word, Count), y = Count)) +
  geom_col(fill = "cornflowerblue") +
  coord_flip() +
  facet_wrap(~ Source, scales = "free", ncol = 1) +
  labs(title = "Ten most frequent words in each source", x = NULL, y = "Frequency") +
  theme_minimal()

The frequency tables and chart show which words dominate each source. Differences may reflect the writing context: news text often contains reporting language and named entities, blogs may contain more personal or conversational language, and Twitter may contain abbreviations, hashtags, mentions, and informal spellings.

5 4. Key findings

Use the computed tables and plots above to confirm the following points before publishing:

Important: Replace this paragraph with two or three specific observations from your knitted output. For example, state which source has the most lines, which has the largest approximate word count, and one visible difference in the top-word lists. Do not submit unverified numerical claims.

6 5. Plan for the prediction algorithm

The eventual application will suggest the next word based on the words a user has already typed. A practical development plan is:

  1. Prepare the text: Normalize case, remove or standardize URLs and unwanted symbols, handle repeated whitespace, and decide how to treat hashtags, mentions, contractions, and punctuation. Preserve meaningful tokens.
  2. Create n-grams: Build unigram, bigram, trigram, and possibly four-gram counts. An n-gram is a sequence of n consecutive words.
  3. Estimate next-word probabilities: Given the last two or three words, rank candidate next words using observed n-gram frequencies. Use a backoff strategy: if a longer context is not found, fall back to a shorter context.
  4. Improve robustness: Apply smoothing or backoff so unseen word sequences do not produce empty predictions. Consider a frequency threshold to control model size.
  5. Evaluate: Hold out a sample of text not used for training. Measure top-1 and top-3 prediction accuracy, coverage (how often the model can make a suggestion), memory use, and response time.
  6. Optimize: Profile the model, reduce the size of stored n-gram tables, and load only the data needed for fast prediction.

The first version should favor a compact model that responds quickly over a very large model that is slow to load.

7 6. Plan for the Shiny application

The Shiny app will provide a simple text box where a user enters a phrase. As the phrase changes, the app will display up to three suggested next words. The interface will also include a short explanation that predictions are based on patterns in the training text and may not always be grammatically or factually correct.

The app will be designed around three goals:

8 7. Conclusion

The exploratory analysis provides an initial understanding of the three text sources and establishes a reproducible starting point for the capstone. The summaries describe dataset size, while the plots and frequency tables highlight basic vocabulary and writing-style differences. The next milestone is to clean and sample the text, build n-gram frequency tables, evaluate prediction quality, and connect the selected model to a lightweight Shiny interface.

9 Reproducibility notes