Milestone Report - NLP Capstone

Summary

This report explains my exploratory analysis and your goals for the eventual app and algorithm.

Motivation

The motivation for this project is to: 1. Demonstrate that you’ve downloaded the data and have successfully loaded it in.2. Create a basic report of summary statistics about the data sets.3. Report any interesting findings that you amassed so far.4. Get feedback on your plans for creating a prediction algorithm and Shiny app.

Overview

Sample 20% of the data to significantly speed up processing and use less compute. Ensure that the sample represents the full set.
Basic statistics: Get and display num lines, word counts form sample
Tables and plots of word and n-gram distribution frequencies
Word coverage calculations
Feature earnings
Summarization for creating prediction algoritm and Shiny app

Scope

US-English training data files only (3 files) - en_US.blogs.txt - en_US.news.txt - en_US.twitter.txt

## Loading required package: NLP

## Loading required package: readtext

## Warning: package 'readtext' was built under R version 4.2.3

## Loading required package: quanteda

## Warning: package 'quanteda' was built under R version 4.2.3

## Warning in .recacheSubclasses(def@className, def, env): undefined subclass
## "ndiMatrix" of class "replValueSp"; definition not updated

## Package version: 4.0.1
## Unicode version: 13.0
## ICU version: 69.1

## Parallel computing: 20 of 20 threads used.

## See https://quanteda.io for tutorials and examples.

## 
## Attaching package: 'quanteda'

## The following object is masked from 'package:readtext':
## 
##     texts

## The following object is masked from 'package:tm':
## 
##     stopwords

## The following objects are masked from 'package:NLP':
## 
##     meta, meta<-

## Loading required package: quanteda.textstats

## Warning: package 'quanteda.textstats' was built under R version 4.2.3

## Warning: undefined subclass "ndiMatrix" of class "replValueSp"; definition not
## updated

## Loading required package: quanteda.textplots

## Warning: package 'quanteda.textplots' was built under R version 4.2.3

## Loading required package: quanteda.corpora

## Warning in library(package, lib.loc = lib.loc, character.only = TRUE,
## logical.return = TRUE, : there is no package called 'quanteda.corpora'

## Installing package into 'C:/Users/RussS/AppData/Local/R/win-library/4.2'
## (as 'lib' is unspecified)

## Warning: package 'quanteda.corpora' is not available for this version of R
## 
## A version of this package for your version of R might be available elsewhere,
## see the ideas at
## https://cran.r-project.org/doc/manuals/r-patched/R-admin.html#Installing-packages

## Warning: package 'ggplot2' was built under R version 4.2.3

## 
## Attaching package: 'ggplot2'

## The following object is masked from 'package:NLP':
## 
##     annotate

Exploratory analysis

Perform a thorough exploratory analysis of the data, understanding the distribution of words and relationship between the words in the corpora.
Understand frequencies of words and word pairs - build figures and tables to understand variation in the frequencies of words and word pairs in the data.

Number of lines in each file (sampled at 20%)

## Number of lines in  en_US.blogs.txt :  184000 
## Number of lines in  en_US.twitter.txt :  471800 
## Number of lines in  en_US.news.txt :  206000

Word Counts (all word instances - including dups) for all sampled files

Files have been sampled 20%

## [1] "Number of words instances per file"

##   en_US.blogs.txt    en_US.news.txt en_US.twitter.txt 
##           7593393           2635572           5993176

## [1] "Total number of words in the corpus"

## [1] 16222141

Some words are more frequent than others - what are the distributions of word frequencies?

Create dfm (document-frequency matrix)
Calculate and rank word frequency (top 50)
Plot distribution of word freq.
Plot a word cloud to help enphasis which words (features) are most prevalent and by how much.

## Document-feature matrix of: 3 documents, 306,153 features (53.38% sparse) and 0 docvars.
##                    features
## docs                       in    the years thereafter most     of  oil fields
##   en_US.blogs.txt   15 120852 377995  6749         48 8068 178311 1168    244
##   en_US.news.txt     0  51452 151489  3655         14 2726  59040  488     89
##   en_US.twitter.txt  0  75519 187941  2614          8 3743  71687  278     78
##                    features
## docs                   and
##   en_US.blogs.txt   222284
##   en_US.news.txt     68255
##   en_US.twitter.txt  87851
## [ reached max_nfeat ... 306,143 more features ]

## Top 50 most frequent words - EN-US Docs (Sampled):

##    feature frequency rank docfreq group
## 1      the    717425    1       3   all
## 2       to    443697    2       3   all
## 3      and    378390    3       3   all
## 4        a    371975    4       3   all
## 5        i    315102    5       3   all
## 6       of    309038    6       3   all
## 7       in    247823    7       3   all
## 8       is    182513    8       3   all
## 9      for    178054    9       3   all
## 10     you    177409   10       3   all
## 11    that    167005   11       3   all
## 12      it    158305   12       3   all
## 13      on    132100   13       3   all
## 14      my    116266   14       3   all
## 15    with    112699   15       3   all
## 16     was     98158   16       3   all
## 17    this     95046   17       3   all
## 18      be     91582   18       3   all
## 19    have     89276   19       3   all
## 20      at     88697   20       3   all
## 21     are     81690   21       3   all
## 22     but     79054   22       3   all
## 23      as     74141   23       3   all
## 24      we     72305   24       3   all
## 25      me     71289   25       3   all
## 26      so     70645   26       3   all
## 27     not     68804   27       3   all
## 28      he     58795   28       3   all
## 29     all     58755   29       3   all
## 30    from     58288   30       3   all
## 31    your     56858   31       3   all
## 32    just     54587   32       3   all
## 33    they     50811   33       3   all
## 34     out     50772   34       3   all
## 35      or     50730   35       3   all
## 36      up     50017   36       3   all
## 37    will     49984   37       3   all
## 38    what     49365   38       3   all
## 39   about     49051   39       3   all
## 40      if     48354   40       3   all
## 41    like     48293   41       3   all
## 42     one     48270   42       3   all
## 43      by     47112   43       3   all
## 44      an     44847   44       3   all
## 45    when     43600   45       3   all
## 46      do     43486   46       3   all
## 47     can     42727   47       3   all
## 48     his     41150   48       3   all
## 49     get     40131   49       3   all
## 50    more     38216   50       3   all

What are the frequencies of 2-grams and 3-grams in the dataset?

Generate bigrams and trigrams in while loop.
Generate word cloud (100 word) to show which bigrams and trigrams are most prevent and by what magnitude.

## Top 20  2-gram - EN-US Docs (Sampled):

## Top 20  3-gram - EN-US Docs (Sampled):

## Warning in wordcloud(x, min_size, max_size, min_count, max_words, color, :
## what_do_you could not be fit on page. It will not be plotted.

## Warning in wordcloud(x, min_size, max_size, min_count, max_words, color, :
## do_you_think could not be fit on page. It will not be plotted.

## Warning in wordcloud(x, min_size, max_size, min_count, max_words, color, :
## the_united_states could not be fit on page. It will not be plotted.

## Warning in wordcloud(x, min_size, max_size, min_count, max_words, color, :
## the_same_time could not be fit on page. It will not be plotted.

## Warning in wordcloud(x, min_size, max_size, min_count, max_words, color, :
## in_the_morning could not be fit on page. It will not be plotted.

How many unique words do you need in a frequency sorted dictionary to cover 50% of all word instances in the language? 90%?

## Unique words do you need in a frequency sorted dictionary to cover 50 % of all word instances in the language:  141 
## Unique words do you need in a frequency sorted dictionary to cover 90 % of all word instances in the language:  7973

Sample Versus Full data set

Full set (not sampled)
- unique words do you need in a frequency sorted dictionary to cover 50 % of all word instances in the language: 132
- unique words do you need in a frequency sorted dictionary to cover 90 % of all word instances in the language: 7409
Sample set
- Unique words do you need in a frequency sorted dictionary to cover 50 % of all word instances in the language: 141
- Unique words do you need in a frequency sorted dictionary to cover 90 % of all word instances in the language: 7973

Interesting findings that you amassed so far.

20% Sample represents full data set very well
we only need (7,973/742,067)*100 or about 1 percent of all of the words to cover 90% of all word instances.
If the same holds true for 2-gram and 3-gram frequency data which I suspect is true, we won’t need a lot of data to quickly provide a prediction for the 2nd and 3rd words (small table) so that there is no delay. Any
significant delay to predicting the next word would defeat the purpose (i.e., user could type it faster).
most frequent words, 2 gram words are very small (1-3 character), same for 3 gram except they start to have some
slightly larger words more often (typically 2 small words and one mid-sized word (4-7 chars).

Plan summarization for creating prediction algoritm and Shiny app

use sample data to ensure that training the model will be fast
make shiny app configurable to allow the user to choose their preference of performance versus accuracy
- for example, ‘fast’ will only attempt to predict 50% of all word instances, but ‘slower’ will attempt to predict ‘90%’ of all instances.
remove profane words
- profanity set identified https://github.com/surge-ai/profanity
algorithm will try to predict 2nd word given the 1st, and 3rd word given the second.
- this will be based on the most frequent combinations of the 1st word with the second word for bigram
  and most frequent combinations of the 1st and second word with the third word for trigram.
  For example, if the users first 2 words were ‘I have’, we would predict that the next word is ‘a’ because it is higher ranked than ‘to’ (refTop 20 3-gram - EN-US Docs (Sampled) above).