Capestone Coursera Data Science Final Project

Language Modelling - Text Predictor

Avinash Singh Pundhir
Analyst

Introduction

  1. The application is the capstone project for the Coursera Data Science specialization from Johns Hopkins University in collaboration with SwiftKey.

  2. Main goal of this project is to build an app that is able to predict the next word that should follow any specific sentense.

  3. The app will request an english sentense as input and will predict the most probable words that can follow.

  4. Natural language modelling concepts such as tokenization, N-grams, language modelling are evaluated and applied in the application logic.

Methodology

  1. The application uses a data corpus provided specifically for the project purposes.

  2. The corpus has data gathered from news, blogs and twitter feeds.

  3. Inout data has been preprocessed and cleaned and using N Grams tokenization unigrams, bigrams, trigrams and Ngrams are generated.

  4. These N Grams are used as an input to the text prediction methology implemented based on Katz backoff and good turing methods.

  5. A combination of the above two approaches is used to generate relevance score. A higher relevance score signifies better match.

Shiny Application Walkthrough

width

  • App will request an english sentense.
  • User can select the number of predictions desired.
  • After hitting predict button predictions will be shown on right side.

Proposed Enhancements

  1. The app is relatively fast, and returns a default list of 3 suggested matches in just a few seconds.

  2. The app does not "learn" from user input. Future improvements could include building n-gram tables based upon a user's frequent input

  3. Also we can enhance the app to suggest a word pair rathe than just suggesting a single word match.

  4. We can increase accuracy by adding some more context sensitive logic based on part of speeches and opposite words.