• The main goal of this capstone project is to build a shiny application that is able to predict the next word.

• This exercise was divided into seven sub tasks including data cleansing, exploratory analysis, the creation of a predictive model using Good Tuning and Katz BackOff Model Implementation, and User UI using Shiny application.

• All text mining and natural language processing was done with the usage of a variety of well-known R packages such as Corpos TM and Weka, and wordnet for word cloud, and the data.table package.

Data cleansing steps :

• Sampling the data by 1% of three documents(3 files).
• Remove The non-ASCII characters, punctuation,numbers,stop words and stemming.

Using N-gram Model

• Ref : https://en.wikipedia.org/wiki/N-gram
• Those aggregated uni, bi- and tri-dgram term frequency matrices have been transferred into probilitiy table in csv format.

While entering the text (1), the field with the predicted next word (2) refreshes instantaneously and also the whole text input (3) gets displayed.

Application : https://jackskmok.shinyapps.io/Data_Science_Capstone/

• https://en.wikipedia.org/wiki/N-gram
• https://en.wikipedia.org/wiki/Katz's_back-off_model
• http://nlp.stanford.edu/~wcmac/papers/20050421-smoothing-tutorial.pdf
• https://rstudio-pubs-static.s3.amazonaws.com/52230_5ae0d25125b544caab32f75f0360e775.html
• https://cran.r-project.org/web/packages/data.table/data.table.pdf