Data Science Capstone - Predictive Text Data Product

The following slide deck relates to the Capstone Project for the Data Science Capstone from the Johns Hopkins University on Cousera.

The problem this activity and app tries to solve is that of predicting text based on input, for instance when someone types “I went to the” the application should predict options for what the next word might be.

In order to solve this problem I opted for building an n-gram model, that is building a model that given one or two words the model will predict what the most likely next word.

The technology I used to solve this problem was as follows

• R Studio as an Integrated Development Environment for coding
• Microsoft Open R + Intel MKL libraries (http://bit.ly/2bUIg5D) which allows for more optimal parallel processing on Windows environments.
• Microsoft Azure cloud to scale up to large amounts of CPU and RAM. The VM instance used for model training had 112GB RAM and 32 CPU Cores.

The steps for processing the data were as follows

1. Ingest the data into an R Data frame and cleanse the data removing non ASCII characters, white space and set all to lower case, implemented using the tm package and gsub.
2. Tokenise the data using RWeka and output a document term matrix, tokenisation allowed for config of the size of the input n-gram (1,2,3,etc. words).
3. Using this document term matrix create and R Data Frame that included search term, follow on term and number of occurrences in the corpus of text.

The output was a set of data frames for the data product.

Using the data frames a data product was put together (http://bit.ly/2rwxU1S) which takes input text and then does the following:

1. Cleanse the input text removing non ASCII alpha characters, white space and set all characters to lower case, this was done using the tm package and gsub.
2. Displays the most likely word for the search term based on a lookup in the data frame.
3. Displays a word cloud (using wordcloud pkg) of up the most 10 likely follow on terms based on the data frame lookup.
4. Displays an emotion classification graph based on the input text from the user using syuzhet.

The data product could be described as a minimal viable product and does have some issues namely

1. Even with the 32 core 112GB ram VM training the n-gram model beyond 2 input words was not possible.
2. The data product itself makes use of a cut down one word input data frame due to memory limitations.

I was to approach this problem again from scratch the data cleansing method would likely remain the same however I may use a predictive model such as a neural net which may give better performance. It should be noted that the method used above gave good results with the test quizzes on the course giving high degree of matches.