Objectives of this report

This report is a first checkpoint in the Data Science Capstone project. Its goal is simply to show that:

  1. The training data has been downloaded and loaded successfully.
  2. Some basic summary statistics have been computed for the three English-language source files (blogs, news, Twitter).
  3. A few interesting early findings have been identified.
  4. There is a clear plan for building the prediction algorithm and the Shiny app that will make up the final product.

The report is intentionally kept short and non-technical, so that it can be understood by a manager with no data science background.

1. Getting the data

The training data comes from the official course source (a corpus collected by SwiftKey combining blog posts, news articles, and tweets, in four languages). This report focuses on the English (en_US) subset, which contains three plain-text files: en_US.blogs.txt, en_US.news.txt, and en_US.twitter.txt.

# NOTE: eval = FALSE here on purpose. The raw files together are close to
# 600 MB and take several minutes to read into memory, which is impractical
# to re-run every time this report is knitted. The code below documents
# exactly how the data was loaded and summarised; the results it produced
# were cached and are loaded directly in the next section.

path <- "final/en_US"
blogs   <- readLines(file.path(path, "en_US.blogs.txt"),   encoding = "UTF-8", skipNul = TRUE)
news    <- readLines(file.path(path, "en_US.news.txt"),    encoding = "UTF-8", skipNul = TRUE)
twitter <- readLines(file.path(path, "en_US.twitter.txt"), encoding = "UTF-8", skipNul = TRUE)

The three files were read into R without any issue, confirming that the data acquisition step succeeded.

2. Basic summary of the three files

summary_table <- data.frame(
  File = c("en_US.blogs.txt", "en_US.news.txt", "en_US.twitter.txt"),
  `Size (MB)` = c(200.4, 196.3, 159.4),
  `Lines` = c(899288, 1010206, 2360148),
  `Words` = c(37334131, 34371031, 30373583),
  `Avg. words / line` = c(41.5, 34.0, 12.9),
  check.names = FALSE
)
knitr::kable(summary_table, format.args = list(big.mark = ","))
File Size (MB) Lines Words Avg. words / line
en_US.blogs.txt 200.4 899,288 37,334,131 41.5
en_US.news.txt 196.3 1,010,206 34,371,031 34.0
en_US.twitter.txt 159.4 2,360,148 30,373,583 12.9

A few things stand out immediately from this table:

  • Twitter has by far the most lines (2.36 million) but the fewest words per line (12.9 on average) — a direct consequence of the historical 140-character limit on tweets.
  • Blogs have the fewest lines but the longest average sentence/entry (41.5 words per line), consistent with blogs being longer-form writing.
  • News sits in between the other two sources on every metric, which makes sense given typical news-article sentence length.
  • Despite these differences in style, the total word count is fairly similar across the three sources (30–37 million words each), giving a well-balanced corpus overall (~102 million words in total).

Total words per source

Distribution of words per line, by source

These histograms confirm the pattern seen in the summary table: Twitter’s distribution is tightly concentrated at low word counts, blogs have a long right tail (some very long entries), and news falls in between with a distribution that peaks around 25–35 words per line.

3. Early findings: most frequent words

To get a first look at word usage across the corpus, a random sample of 15,000 lines (5,000 from each source) was drawn and split into individual words (converted to lowercase, punctuation stripped).

Unsurprisingly, the most frequent words are common English function words (the, to, and, a, of, in…) rather than meaningful content words — this is expected for any large sample of natural English text, and matches the well-known Zipf’s law pattern where a small number of words account for a disproportionate share of total word usage. This is an important early finding for the modelling step: these very frequent words will need to be handled carefully (e.g. via n-gram models) since they will dominate naive word-frequency counts.

4. Plans for the prediction algorithm and the Shiny app

Prediction algorithm. The next phase of the project will build an n-gram language model (using bigrams, trigrams, and possibly 4-grams) from this corpus. Given the last one or two words a user has typed, the model will look up the most likely next word(s) based on how frequently that sequence appeared in the training corpus. Smoothing (e.g. Katz back-off or Kneser-Ney) will be used to handle word sequences that were rarely or never seen in the training data, so the app can still make a reasonable suggestion even for unusual input.

Shiny app. The final product will be a simple, lightweight web app (built with Shiny) where a user types a partial sentence into a text box, and the app instantly suggests the most likely next word(s) — similar to the predictive keyboard on a smartphone. The interface will be kept minimal (one text input, a short list of suggested next words updated in real time) so that it is fast and easy to use even for someone unfamiliar with the underlying model.

Conclusion

The training data has been downloaded, loaded, and explored successfully. The three sources (blogs, news, Twitter) show distinct but complementary writing styles, together forming a large and varied corpus of about 102 million words. This exploratory work directly informs the next steps of the project: building an n-gram based prediction model and wrapping it in an interactive Shiny application.