Executive Summary

N-gram model building for predicting the next word based on the previous 1, 2, or 3 words
N-gram model is executed on the 3 files from twitters, blogs and news forming the dataset
To achieve the goal from data analysis, the focus is to first read, sample and create a clean data set from 3 files, then create the predictive model and finally a verification script to validate the output
A pre-computation step is designed to build a fast lookup dictionary which is used to fetch the next word based on the user input
The system looks at the pre-calculated target column next to search phrase and immediately shows the user the highest-frequency predictions
The solution is then packaged into a shiny app where the next word is predicted based on the user input
The model will read the last two words from the text and predict word based on the 3-grams
The model will read the last word from text and predict word based on 2-grams

#Data Exploration - Zipf’s law says that a small number of words appear very often. Most words appear only a few times.The frequency of the top 20 most frequent words is plotted in a bar chart.

Including Plots

The plot analysis in the report is documented for en_US.twitter.txt. The summary plots for blogs and news datasets is in the end of report as supporting plots.

## Warning: package 'tidytext' was built under R version 4.6.1

## Warning: package 'dplyr' was built under R version 4.6.1

## 
## Attaching package: 'dplyr'

## The following objects are masked from 'package:stats':
## 
##     filter, lag

## The following objects are masked from 'package:base':
## 
##     intersect, setdiff, setequal, union

## Warning: package 'stringr' was built under R version 4.6.1

## en_US.twitter.txt: lines 2360148

## # A tibble: 10 × 2
##    word         n
##    <chr>    <int>
##  1 love    106732
##  2 day      91748
##  3 rt       89601
##  4 time     76803
##  5 lol      70162
##  6 people   52047
##  7 happy    49009
##  8 follow   48108
##  9 tonight  44706
## 10 night    41180

## Warning: package 'ggplot2' was built under R version 4.6.1

## Warning: Using `size` aesthetic for lines was deprecated in ggplot2 3.4.0.
## ℹ Please use `linewidth` instead.
## This warning is displayed once per session.
## Call `lifecycle::last_lifecycle_warnings()` to see where this warning was
## generated.

Preparing the data

Read the 3 files and randomly extract 50% of lines to reduce computational load.
The data sets are en_US.blogs.txt, en_US.news.txt, en_US.twitter.txt
Breaks the sampled text blocks down into individual sentences
The stringi data normalization patterns - Normalize the text, example convert all text to lowercase.
Remove the noise - remove the Twitter hashtags, punctuation, non-alphabetic characters, and excess whitespace
Add sentence boundary with tags in start and end markers
Generate the word tokens - Splits the tagged sentences into a single, continuous flat vector of words

Creation of N-Gram Tables

Creation of sentence-aware tokenization pipelines
Coding the flat vector of words into structured frequency tables (Unigrams to Quadgrams) using the data.table library to serve as the core lookup database for the prediction model
Generate the N-Gram combination by creating the overlapping sequence of 1,2,3,4 words
Separate the context and target into separate columns in the dataset
When a user types “look forward”, the model doesn’t scan the whole database; it immediately jumps straight to the “look forward” key in the Trigram table and pulls all the matching target words (like “to”, “hearing”, “visualizing”) sorted by their frequency

Prediction Model

Designed and executed multiple models, to reduce error and increase model prediction
The statistical n-gram backoff model searches hierarchically - 4-gram else if 3-gram else if 2-gram else if unigram
Built a 4-gram predictive language model using a modified Jelinek-Mercer interpolation algorithm that blends quadgram, trigram, and bigram statistics simultaneously

Shiny App

Shiny app with a text input box that is running on shinyapps.io
The app loads to the point where it can accept a user input
The app gives prediction for every phrase entered
A 50% accuracy score on entirely unobserved, open-ended sentences is a strong result for a purely statistical n-gram model
The quantified uncertainly in the sample is given by Zipf’s law which says that rare phrases occur so infrequently that even a large 50% text sample won’t capture them.
Statistical n-gram models are limited because they require seeing exact text matches to make predictions.
Memory Management: Successfully scaled the pipeline up to 50% of the entire multi-gigabyte corpus and serialized it into a compact .rds file, making it light enough to run within a free 1 GB RAM web hosting tier

Supporting Plots - Blogs data set

## en_US.twitter.txt: lines 899288

## # A tibble: 10 × 2
##    word       n
##    <chr>  <int>
##  1 time   90920
##  2 people 59575
##  3 day    52373
##  4 love   45230
##  5 life   41254
##  6 it’s   38660
##  7 world  29306
##  8 i’m    29192
##  9 don’t  28389
## 10 book   28153

Supporting Plots - News data set

## en_US.twitter.txt: lines 1010206

## # A tibble: 10 × 2
##    word        n
##    <chr>   <int>
##  1 time    57061
##  2 people  47664
##  3 city    37950
##  4 school  35498
##  5 game    34947
##  6 percent 34690
##  7 day     31901
##  8 million 30913
##  9 home    30602
## 10 county  30172

Capstone- dataproduct