Text Prediction Prediction

Web Application

Sara Rafi

Introduction

The Text Prediction Application is a web application which predicts the next possible word.

When a single word or a phrase areventered , the app gives a list of possible next words along with their probabilities.

The model development was based on a corpus containing data from three sources. These were:

  1. en_US.blogs.txt
  2. en_US.news.txt
  3. en_US.twitter.txt

Data Cleaning and Text Mining

The first task was to prepare the words as 'tokens' for analysis and language model development.

A few of the important steps were:

  • Convert to lower case
  • Remove punctutaution
  • Remove all numbers
  • Remove stop words
  • Remove special characters
  • Remove extra whitespace

Prediction Model

The prediction model was developed on an n-grams model.

Trigrams and bigrams were computed. The model looks for the next word the trigrams first, incases where two or more words are entered. If there are no matches in the trigram, the next word is looked up in the bigrams data.

This is known as Katz Back off model and was implemented and addition to Good Turing estimates of probabilites in order to address the issue of unseen words.