Word prediction application

Data Science Capstone project

This presentation will briefly introduce my Data Science Capstone application for predicting the next word.

Application is written in R language with the help of additional R packages. Namely:

• shinythemes, shiny, markdown
• data.table, dplyr, dtplyr
• quanteda

Prediction uses HC Corpora dataset which is quite large (at least for my coputer), so the model was build on sample of 30% of all data.

After creating a data sample from the HC Corpora data, following transformations (cleaning) were done on the dataset:

• conversion to lowercase,
• removing punctuation, links, white space, numbers and special characters.
• removing of profanity words (coarse language) - to prevent offending somebody

This data sample was then converted into various n-grams (bi, tri, quad and penta gram)

Prediction is based on lookup mechanism which goes from longest n-grams to shortest and tries to find most frequent combination, which is then used as the next word. Search is based on user input

Not everything went well during creation of prediction model. I had severe performance problems and had to try many different R packages to get to the finish line.

For bad performance I discarder (in order):

• tm package with RWeka tokenizer (took 2 hours to make bi- to tri-grams)
• ngram package (difficult to oparate)

I also had to use data.table package as regular data.frame was to slow for operations with 150MB dataset.