Project Overview

The project goal was to build a predictive model of English text. The user must enter a word in a textbox and the algorithm predicts the next word. The skills needed to complete this task include natural language processing and text mining. The app was created using the Shiny Application in RStudio.

The source data for this project can be found at: https://d396qusza40orc.cloudfront.net/dsscapstone/dataset/Coursera-SwiftKey.zip

My Shiny App can be found at: https://starscream.shinyapps.io/Final_Project/

Working with the Data

In order to build a prediction algorithm, data was scraped from blogs, twitter and the news. There are several processes that need to be completed before the model can be built.

• The data was read into R
• Data cleaning was performed: stripping out of numbers and punctuation, changing all text to lowercase and removing the whitespace.
• N-grams were created, these are a sequence of items collected from a corpus. The N refers to the number of items within the sequence. For this project, bigrams, trigrams and quadgrams were used.
• The N-grams were sorted and the metadata saved as an .rds file.

The Prediction Model

The model for the next word prediction was based on the Katz Back-off algorithm. This project’s algorithm contains:

• Efficient text cleaning
• Optimized efficiency of loading files
• Optimized memory usage
• Backoff algorithm first applied to first four reduced N-grams

Features

• Contains progress bar showing loading of required n-grams
• Predict button to predict text and clear button to clear text area
• Option to select news data or twitter data