The objective of this project is to develop a predictive text model capable of suggesting the next word in a sentence as a user types. The final product will be an interactive Shiny application that predicts the most probable next word using Natural Language Processing (NLP) techniques.
This report summarizes the exploratory analysis performed on the English language corpus provided by SwiftKey, highlights the major characteristics of the data, and outlines the proposed methodology for building the prediction algorithm.
Predictive text systems are widely used in smartphones, search engines, and messaging applications. These systems improve typing speed and accuracy by predicting the next word based on previous words entered by the user.
The dataset supplied for this project consists of three large English-language text files:
Blogs News articles Twitter posts
Together these files provide a diverse collection of everyday English language suitable for training a language model.
The corpus consists of approximately four million lines of text.
Dataset File Size (MB) Number of Lines Number of Words Blogs ~200 899,288 ~37 million News ~196 1,010,242 ~34 million Twitter ~159 2,360,148 ~30 million
The Twitter dataset contains the largest number of documents, whereas the Blogs dataset contains the longest individual documents.
blogs <- readLines("~/final/en_US/en_US.blogs.txt",
encoding = "UTF-8",
skipNul = TRUE)
news <- readLines("~/final/en_US/en_US.news.txt",
encoding = "UTF-8",
skipNul = TRUE)
twitter <- readLines("~/final/en_US/en_US.twitter.txt",
encoding = "UTF-8",
skipNul = TRUE)Because the original corpus is very large, processing the entire dataset repeatedly would require considerable computing resources.
A random sample (approximately 1–5% of each dataset) will therefore be used during development while preserving the distribution of language across the three data sources.
The sampled datasets are combined into one corpus for analysis.
Prior to analysis the following preprocessing steps were performed:
Converted text to lowercase Removed punctuation Removed numbers Removed URLs Removed extra white spaces Removed profanity Converted text into plain English tokens
The corpus contains millions of words. Unsurprisingly, common English words dominate the dataset.
Examples include:
the and to of a in
After removing common stop words, more informative terms become visible.
Twitter contains many short informal sentences and abbreviations. Blogs contain longer and more conversational writing. News articles use more formal vocabulary. A relatively small number of words account for a large proportion of the corpus, consistent with Zipf’s Law.
Exploratory analysis demonstrates that the SwiftKey corpus provides a rich source of English language suitable for predictive text modelling. The datasets have been successfully loaded, sampled, cleaned, and analysed.
The next stage of the project is to develop an efficient N-gram prediction model incorporating backoff techniques and deploy the model as an interactive Shiny application.