Executive summary

The project uses English-language text from blogs, news articles, and Twitter to build a next-word prediction application. The files have been downloaded, located, and loaded successfully. Together they provide a large and varied sample of formal and informal English.

The main finding is that the sources behave differently: tweets are short and conversational, while blogs and news contain much longer passages. The final model will therefore need to learn common word sequences without allowing one source or a few unusually long lines to dominate the results.

Data successfully loaded

The files are processed in chunks rather than all being held in memory at the same time. This makes the analysis reliable on an ordinary computer.

summarize_file <- function(path, sample_n = 50000, chunk_n = 100000) {
  con <- file(path, open = "rb")
  on.exit(close(con), add = TRUE)

  line_count <- 0L
  longest <- 0L
  working_sample <- character()

  repeat {
    chunk <- readLines(
      con,
      n = chunk_n,
      encoding = "UTF-8",
      skipNul = TRUE,
      warn = FALSE
    )
    if (!length(chunk)) break

    line_count <- line_count + length(chunk)
    longest <- max(longest, nchar(chunk), na.rm = TRUE)

    if (length(working_sample) < sample_n) {
      needed <- sample_n - length(working_sample)
      working_sample <- c(working_sample, head(chunk, needed))
    }
  }

  list(lines = line_count, longest = longest, sample = working_sample)
}

file_summaries <- lapply(files, summarize_file)

The following table confirms the scale and basic structure of the three files. File sizes use binary megabytes (1024^2 bytes).

summary_table <- data.frame(
  Source = names(files),
  `File size (MB)` = round(file.info(files)$size / 1024^2, 1),
  Lines = vapply(file_summaries, function(x) x$lines, integer(1)),
  `Longest line (characters)` =
    vapply(file_summaries, function(x) x$longest, integer(1)),
  check.names = FALSE
)

kable(summary_table, format = "html", digits = 1,
      caption = "Summary of the English SwiftKey text files")
Summary of the English SwiftKey text files
Source File size (MB) Lines Longest line (characters)
Blogs Blogs 200.4 899288 40833
News News 196.3 1010242 11384
Twitter Twitter 159.4 2360148 140

What the text looks like

To keep the exploration fast and reproducible, the next chart uses a fixed working sample of up to 50,000 lines from each source.

sampled <- do.call(rbind, lapply(names(file_summaries), function(source) {
  x <- file_summaries[[source]]$sample
  words_per_line <- lengths(regmatches(
    x,
    gregexpr("\\b[[:alpha:]']+\\b", x, perl = TRUE)
  ))

  data.frame(
    Source = source,
    Words = words_per_line
  )
}))

Interesting findings

  • The Twitter file has more than two million lines, but each line is short.
  • Blogs contain the longest individual lines, including an extreme line of more than 40,000 characters.
  • The three sources mix conversational language, headlines, narrative writing, punctuation, contractions, URLs, and other web-specific text.
  • File size alone is not a reliable measure of linguistic variety. The model should balance sources and evaluate prediction quality on held-out text.
  • Cleaning must be conservative. Apostrophes and sentence boundaries provide useful predictive information and should not be removed automatically.

Prediction algorithm plan

The first model will be an n-gram backoff model. In plain language, it will look at the last few words typed and find the words that most often followed that phrase in the training data.

  1. Clean and normalize the text while preserving useful sentence boundaries and contractions.
  2. Split the data into training and validation samples.
  3. Count common two-, three-, and four-word sequences.
  4. Given a user’s phrase, try the most specific match first. If it has not been seen often enough, fall back to a shorter phrase.
  5. Rank likely next words using frequency, source balance, and simple smoothing so rare phrases do not produce unstable predictions.
  6. Measure top-1 and top-3 prediction accuracy, speed, and model size on text excluded from training.

This approach is intentionally practical: it is explainable, fast enough for an interactive application, and can be improved incrementally.

Shiny application plan

The Shiny app will provide one text box. As the user types a phrase, the app will display the three most likely next words. Selecting a suggestion will add it to the text, allowing the user to continue composing a sentence.

The interface will also include:

  • a brief explanation for first-time users;
  • a clear indication when the model has limited evidence;
  • fast responses suitable for ordinary typing;
  • optional examples that demonstrate formal and conversational text; and
  • a small information panel describing the data and model limitations.

Next steps and feedback requested

The next milestone is to complete cleaning, build the n-gram tables, and compare model accuracy with model size and response speed. Feedback would be most useful on two product decisions: whether three suggestions are sufficient, and whether the app should favor general language or adapt its rankings to the selected source style.