Introduction

The objective of this milestone report is to explore the English-language SwiftKey corpus and identify the main characteristics that will support the development of a next-word prediction application. The corpus contains text from three sources: blogs, news, and Twitter.

The analysis examines the size and structure of the datasets, differences in text length, frequently occurring words, and common word sequences. These findings will guide the development of an efficient n-gram prediction algorithm and Shiny application.

Data Loading

The original SwiftKey corpus is stored in a compressed ZIP file. The analysis reads the selected language directly from the archive so that the workflow remains reproducible and does not require additional extracted copies of the source files.

sources <- tibble(
  source = c("Blogs", "News", "Twitter"),
  type = c("blogs", "news", "twitter")
)

files <- sources %>%
  mutate(
    file = paste0(
      "final/",
      language,
      "/",
      language,
      ".",
      type,
      ".txt"
    )
  )

files
## # A tibble: 3 × 3
##   source  type    file                         
##   <chr>   <chr>   <chr>                        
## 1 Blogs   blogs   final/en_US/en_US.blogs.txt  
## 2 News    news    final/en_US/en_US.news.txt   
## 3 Twitter twitter final/en_US/en_US.twitter.txt

A reusable function reads each source and converts it to a tidy data frame.

read_swiftkey_file <- function(zip_file, file_path, source_name, language_name) {

  text <- readLines(
    unz(zip_file, file_path),
    encoding = "UTF-8",
    skipNul = TRUE
  )

  tibble(
    language = language_name,
    source = source_name,
    text = text
  )
}

The three datasets are then combined into one corpus, with one row per line of text.

corpus <- map2_dfr(
  files$file,
  files$source,
  ~ read_swiftkey_file(
    zip_file = zip_file,
    file_path = .x,
    source_name = .y,
    language_name = language
  )
)

glimpse(corpus)
## Rows: 4,269,678
## Columns: 3
## $ language <chr> "en_US", "en_US", "en_US", "en_US", "en_US", "en_US", "en_US"…
## $ source   <chr> "Blogs", "Blogs", "Blogs", "Blogs", "Blogs", "Blogs", "Blogs"…
## $ text     <chr> "In the years thereafter, most of the Oil fields and platform…

Dataset Summary

Basic descriptive statistics were calculated to compare the size and structure of the three sources. The summary includes file size, number of lines, word and character counts, and measures of text length.

# File metadata inside the ZIP archive
zip_contents <- unzip(
  zip_file,
  list = TRUE
)

file_sizes <- files %>%
  left_join(
    zip_contents %>%
      transmute(
        file = Name,
        file_size_mb = Length / 1024^2
      ),
    by = "file"
  ) %>%
  select(
    source,
    file_size_mb
  )

# Text-level descriptive statistics
corpus_summary <- corpus %>%
  mutate(
    words = str_count(str_squish(text), "\\S+"),
    characters = nchar(text)
  ) %>%
  group_by(source) %>%
  summarise(
    lines = n(),
    total_words = sum(words),
    total_characters = sum(characters),
    average_words_per_line = mean(words),
    median_words_per_line = median(words),
    min_words_per_line = min(words),
    max_words_per_line = max(words),
    average_characters_per_line = mean(characters),
    median_characters_per_line = median(characters),
    min_characters_per_line = min(characters),
    max_characters_per_line = max(characters),
    .groups = "drop"
  ) %>%
  left_join(
    file_sizes,
    by = "source"
  )

corpus_summary
## # A tibble: 3 × 13
##   source    lines total_words total_characters average_words_per_line
##   <chr>     <int>       <int>            <int>                  <dbl>
## 1 Blogs    899288    37334116        206824505                   41.5
## 2 News    1010242    34372529        203223159                   34.0
## 3 Twitter 2360148    30373583        162096241                   12.9
## # ℹ 8 more variables: median_words_per_line <dbl>, min_words_per_line <int>,
## #   max_words_per_line <int>, average_characters_per_line <dbl>,
## #   median_characters_per_line <dbl>, min_characters_per_line <int>,
## #   max_characters_per_line <int>, file_size_mb <dbl>
Summary Statistics of the English SwiftKey Corpus
Source File Size (MB) Lines Total Words Total Characters Avg. Words/Line Median Words/Line Max Words/Line Avg. Characters/Line
Blogs 200.4 899,288 37,334,116 206,824,505 41.52 28 6,630 229.99
News 196.3 1,010,242 34,372,529 203,223,159 34.02 31 1,792 201.16
Twitter 159.4 2,360,148 30,373,583 162,096,241 12.87 12 47 68.68

For the English corpus, Twitter contains the largest number of observations, with approximately 2.36 million lines, but its messages are substantially shorter, averaging about 12.9 words per line. Blogs contain fewer observations but have the longest entries, averaging about 41.5 words per line, while news averages about 34.0 words per line. The maximum line length is particularly large for blogs, indicating the presence of substantial outliers. Overall, the three sources provide a mixture of short conversational text and longer-form writing that is useful for training a next-word prediction model.

Exploratory Visualizations

Distribution of Words per Line

The distribution of words per line illustrates differences in writing style among the three sources. Because a small number of unusually long lines can dominate the scale, the visualization focuses on lines containing 100 words or fewer.

line_lengths <- corpus %>%
  mutate(
    words_per_line = str_count(str_squish(text), "\\S+"),
    characters_per_line = nchar(text)
  )

Twitter is concentrated at much shorter line lengths, whereas blogs and news contain longer text sequences. This difference is consistent with the summary statistics and highlights the variety of contexts that the prediction algorithm will need to handle.

Distribution of Characters per Line

A second view of text length uses the number of characters per line. The plot is limited to 500 characters so that the main distribution is visible without being dominated by the longest blog and news entries.

The character distributions reinforce the same pattern: Twitter is dominated by shorter messages, while blogs and news provide longer text sequences.

Total Words by Source

The total word count provides a concise comparison of how much text each source contributes to the corpus.

Although Twitter has far more lines than the other two sources, blogs contain the largest total number of words because individual blog lines are considerably longer.

Sampling for Text Analysis

The complete corpus is large. For the exploratory word-frequency and n-gram analysis, a reproducible random sample of 50,000 lines from each source is used. Equal sampling prevents the source with the largest number of lines from dominating this stage of the exploratory analysis while reducing processing time and memory requirements.

set.seed(123)

sample_size <- 50000

corpus_sample <- corpus %>%
  group_by(source) %>%
  slice_sample(n = sample_size) %>%
  ungroup()

corpus_sample %>%
  count(source) %>%
  knitr::kable(
    col.names = c("Source", "Sampled Lines"),
    caption = "Exploratory Analysis Sample"
  )
Exploratory Analysis Sample
Source Sampled Lines
Blogs 50000
News 50000
Twitter 50000

Text Preprocessing

The sampled text is tokenized with quanteda. Punctuation, numbers, symbols, and URLs are removed, and tokens are converted to lowercase. Common words such as the, to, of, and you are intentionally retained because they contain important information for next-word prediction.

tokens_sample <- corpus_sample$text %>%
  tokens(
    remove_punct = TRUE,
    remove_numbers = TRUE,
    remove_symbols = TRUE,
    remove_url = TRUE
  ) %>%
  tokens_tolower()

Most Frequent Words

Word frequencies provide an initial view of the vocabulary structure of the corpus.

word_dfm <- dfm(tokens_sample)

top_words_vector <- topfeatures(
  word_dfm,
  n = 20
)

top_words <- tibble(
  feature = names(top_words_vector),
  frequency = as.numeric(top_words_vector)
)
Twenty Most Frequent Words
Word Frequency
the 220074
to 119921
and 113141
a 105533
of 93931
in 74311
i 65706
that 47537
is 45436
for 45415
it 39233
on 34148
you 33025
with 32010
was 28846
at 23990
this 23844
be 23352
as 23142
my 23122

The frequency distribution is concentrated among a relatively small number of commonly used words. This is relevant to predictive text because high-frequency words and phrases are likely to account for a substantial share of useful next-word suggestions.

Common Word Sequences

Next-word prediction depends on combinations of words rather than only individual word frequencies. Bigrams and trigrams were therefore examined to identify frequently recurring two-word and three-word sequences.

Bigrams

bigram_dfm <- tokens_sample %>%
  tokens_ngrams(
    n = 2,
    concatenator = " "
  ) %>%
  dfm()

top_bigrams_vector <- topfeatures(
  bigram_dfm,
  n = 15
)

top_bigrams <- tibble(
  feature = names(top_bigrams_vector),
  frequency = as.numeric(top_bigrams_vector)
)
Fifteen Most Frequent Bigrams
Bigram Frequency
of the 20875
in the 18979
to the 9709
on the 8746
for the 8244
to be 7137
at the 6221
and the 6094
in a 5607
with the 4953
is a 4451
it was 4323
from the 4207
for a 3896
with a 3864

Trigrams

trigram_dfm <- tokens_sample %>%
  tokens_ngrams(
    n = 3,
    concatenator = " "
  ) %>%
  dfm()

top_trigrams_vector <- topfeatures(
  trigram_dfm,
  n = 15
)

top_trigrams <- tibble(
  feature = names(top_trigrams_vector),
  frequency = as.numeric(top_trigrams_vector)
)
Fifteen Most Frequent Trigrams
Trigram Frequency
one of the 1652
a lot of 1318
to be a 795
going to be 739
as well as 693
out of the 693
some of the 675
the end of 666
it was a 640
be able to 600
part of the 588
i want to 575
thanks for the 501
the rest of 494
a couple of 492

Repeated bigrams and trigrams demonstrate that preceding words contain useful information about what word is likely to occur next. These patterns provide the foundation for an n-gram prediction model.

Key Findings

The exploratory analysis identifies three main characteristics that are important for the prediction problem. First, the corpus is large and contains distinct styles of written communication. Blogs and news provide longer text sequences, while Twitter contributes a much larger number of shorter, conversational observations.

Second, word usage is concentrated among frequently occurring terms. This suggests that the final model may be made smaller by removing extremely rare word combinations while retaining much of the useful predictive information.

Third, recurring bigrams and trigrams show that recent word context contains meaningful information about the next word. This supports the use of an n-gram model as the initial prediction approach.

Prediction Algorithm and Shiny Application Plan

The final prediction model will estimate the next word based on the most recent words entered by the user. The initial model will use n-grams of up to four words. When three preceding words are available, the algorithm will first search the four-gram model for the most likely continuation. If the combination has not been observed in the training corpus, the model will progressively back off to trigram, bigram, and eventually unigram information.

Because the model will ultimately run inside a Shiny application, both memory usage and response time will be important. Rare n-grams can be removed, and only the most useful prediction candidates can be retained. The final lookup tables can also be indexed to provide fast predictions.

The Shiny application will provide a simple text-entry interface. As the user enters a phrase, the application will identify the relevant preceding words and display a small set of likely next-word suggestions. Model performance will be evaluated using prediction accuracy on held-out text, memory requirements, and prediction response time.

Conclusion

The exploratory analysis confirms that the SwiftKey corpus provides a large and diverse source of language data for predictive text modeling. Differences in source length and style, together with the presence of frequently recurring words and word sequences, support the planned use of an n-gram model with a backoff strategy. The next stage of the project will focus on constructing and evaluating that prediction algorithm before integrating it into a Shiny application.