Introduction

The purpose of this project is to explore a large collection of English text data and understand its structure before developing a predictive text application.

The final objective is to create a model that can suggest the next word a user is likely to type, similar to the predictive text features available in modern messaging applications. Before building this model, it is important to understand the characteristics of the available text data, including word frequencies, common word combinations, and differences between data sources.

This report presents the initial exploration of the SwiftKey text corpus, including summary statistics, visualizations, and the planned approach for developing the prediction algorithm and Shiny application.

Data Sources

The dataset contains text collected from three different sources:

Blogs: informal long-form writing from online blogs.
News: more structured and formal written content.
Twitter: short messages representing everyday language use.

These different sources provide a broad representation of how people write in different contexts.

Loading and Exploring the Data

The three datasets were loaded into R and basic statistics were calculated to understand their size and characteristics.

library(tidyverse)
library(tidytext)
library(dplyr)
library(stringr)

blogs <- readLines(
  "en_US.blogs.txt",
  encoding="UTF-8",
  warn=FALSE
)

news <- readLines(
  "en_US.news.txt",
  encoding="UTF-8",
  warn=FALSE
)

twitter <- readLines(
  "en_US.twitter.txt",
  encoding="UTF-8",
  warn=FALSE
)

Overview of the Dataset

The following table summarizes the size of each dataset.

summary_table <- tibble(
  Dataset=c("Blogs","News","Twitter"),
  
  Lines=c(
    length(blogs),
    length(news),
    length(twitter)
  ),
  
  Words=c(
    sum(str_count(blogs,"\\S+")),
    sum(str_count(news,"\\S+")),
    sum(str_count(twitter,"\\S+"))
  )
)

summary_table

## # A tibble: 3 × 3
##   Dataset   Lines    Words
##   <chr>     <int>    <int>
## 1 Blogs    899288 37334131
## 2 News    1010206 34371031
## 3 Twitter 2360148 30373543

The datasets contain millions of words, which provides enough information to identify common language patterns. Because the full datasets are large, a random sample was used for the exploratory analysis to make the computations more efficient.

Creating a Sample for Analysis

set.seed(50)

sample_text <- c(
  sample(blogs,200),
  sample(news,200),
  sample(twitter,200)
)

length(sample_text)

## [1] 600

Text Processing

The text was transformed into individual words using tokenization. This step converts raw text into a format that can be analyzed statistically.

tokens <- tibble(text=sample_text) %>%
  unnest_tokens(word,text)

head(tokens)

## # A tibble: 6 × 1
##   word 
##   <chr>
## 1 but  
## 2 all  
## 3 of   
## 4 this 
## 5 pales
## 6 into

For the frequency analysis, common stop words (such as “the”, “and”, and “is”) were removed because they appear frequently but usually provide limited information about language patterns.

data(stop_words)

clean_words <- tokens %>%
  anti_join(stop_words, by="word")


word_frequency <- clean_words %>%
  count(word, sort=TRUE)

head(word_frequency)

## # A tibble: 6 × 2
##   word       n
##   <chr>  <int>
## 1 time      32
## 2 people    28
## 3 world     25
## 4 day       21
## 5 4         20
## 6 life      17

Most Common Words

The following chart shows the 20 most frequent words after removing common stop words.

word_frequency %>%
  slice_max(n,n=20) %>%
  ggplot(
    aes(
      reorder(word,n),
      n
    )
  ) +
  geom_col(fill="steelblue") +
  coord_flip() +
  labs(
    title="Most Frequent Words",
    x="Word",
    y="Frequency"
  )

The results show that a relatively small number of words appear very often, while many words occur only a few times. This pattern is common in natural language data.

Exploring Common Word Combinations

Individual words provide useful information, but combinations of words can reveal more about language structure. For this reason, bigrams (pairs of consecutive words) were also analyzed.

bigrams <- tibble(text=sample_text) %>%
  unnest_tokens(
    bigram,
    text,
    token="ngrams",
    n=2
  )

bigram_frequency <- bigrams %>%
  count(bigram, sort=TRUE)

head(bigram_frequency)

## # A tibble: 6 × 2
##   bigram     n
##   <chr>  <int>
## 1 in the    94
## 2 of the    89
## 3 to the    40
## 4 to be     33
## 5 on the    31
## 6 at the    29

bigram_frequency %>%
  slice_max(n,n=20) %>%
  ggplot(
    aes(
      reorder(bigram,n),
      n
    )
  ) +
  geom_col(fill="darkgreen") +
  coord_flip() +
  labs(
    title="Most Frequent Word Pairs",
    x="Bigram",
    y="Frequency"
  )

Common word pairs provide valuable information for predicting what word is likely to appear next. For example, if a user types the beginning of a common phrase, the model can use previous examples from the dataset to suggest a continuation.

Word Frequency Distribution

word_frequency %>%
  ggplot(aes(n)) +
  geom_histogram(
    binwidth=1,
    fill="orange",
    color="white"
  ) +
  coord_cartesian(xlim=c(1,10)) +
  labs(
    title="Distribution of Word Frequencies",
    x="Number of Occurrences",
    y="Number of Words"
  )

The distribution shows that most words appear relatively rarely, while a small number of words dominate the dataset. This is an important consideration when designing the prediction algorithm because the model must handle both common and uncommon language patterns.

Future Prediction Algorithm

The next stage of the project will focus on developing a predictive text model using n-grams.

The model will learn from sequences of words in the training data:

Bigrams: predict the next word based on the previous word.
Trigrams: predict the next word based on the previous two words.
Four-grams: predict the next word based on the previous three words.

To improve predictions, the model will use a backoff strategy. It will first look for the longest matching sequence and gradually move to shorter sequences if the exact combination has not been observed before.

Shiny Application Plan

The final product will be an interactive Shiny application where users can enter text and receive a suggested next word.

The application will:

Accept user input.
Analyze the previous words entered.
Use the prediction model to generate suggestions.
Display the most likely next word.

The goal is to create a simple and intuitive interface that demonstrates how language models can support predictive typing.

Conclusion

This exploratory analysis provided an understanding of the structure and characteristics of the SwiftKey text datasets. The analysis identified important language patterns through word frequencies and common word combinations.

These findings will guide the development of the prediction algorithm and provide the foundation for the final Shiny application.

Exploratory Analysis

Mssm

2026-07-15