9/7/2020

Introduction

NLP can be defined as human language automatic (or semiautomatic) processing (Eisenstein 2019). Many computer scientists consider NLP as computational linguistics. It is rather true in terms of computer science, NLP can be considered as a kind of computer modeling or computerization of linguistics (Lee 2020).

NLP is a field of AI in which computers analyze, understand, and derive meaning from human language in a smart and useful way. NLP is frequently used for text mining, machine translation, and automated question-answering.

Objective of the project:

  • Identify a Next Word Prediction Model, as a basis for an application.

Text Prediction Model Methodology

Key model steps:

  1. Input: raw text files for model training.

  2. Clean training data; separate into 2 word, 3 word, and 4 word n grams, save as tibbles.

  3. Sort n grams tibbles by frequency, save as repos

  4. N grams function: uses a “back-off algorithm” type prediction model: a). User supplies an input phrase, b). Model uses last 3, 2, or 1 words to predict the best 4th, 3rd, or 2nd match in the repos)

  5. Output: next word prediction

The statistical analysis was carried out on a free platform (R packages).

Next Word Prediction App

Text Prediction Model app provides a simple user interface to the next word prediction.

Key Features:

  1. Text box for user typing
  2. Predicted next blue color word output below user typing
  3. Tabs with plots of most frequent n grams and wordcloud in the data-set

Key Benefits:

  1. Fast response
  2. Method allows for large training sets leading to better next word predictions

Shiny App Link

Literature Consulted