Data science capstone project: NLP

NLP (Natural Language Processing) is a subfield of the Artificial Intelligence that focuses on the interaction between the machine and the interpretation, understanding and processing human languages.

The goal for this project was to create an app that presents the options for what the next word might be given the phrase input by the user.

For that purpose, many steps had to be taken: data extraction and exploration, modeling, programming the algorithm, predictive creating the data product…etc.

Here I present some of the most relevant concepts utilized in this solution:

• NGram https://en.wikipedia.org/wiki/N-gram
• Smoothing https://courses.engr.illinois.edu/cs447/fa2018/Slides/Lecture04.pdf
• Good-Turing https://en.wikipedia.org/wiki/Good%E2%80%93Turing_frequency_estimation

The dataset was collected from publicly available sources by a web crawler. Text from blogs, news and twitters were used.

10 % of each source was sampled and merged into a full corpus to pro process everything together.

For more details of the exploration go to : https://rpubs.com/dvarelat/760261

• Numbers and special characters were removed
• Profanity words were removed using this list of bad words:https://www.cs.cmu.edu/~biglou/resources/bad-words.txt

After random sampling, cleaning and pre-processing the data, the algorithm was implemented to do the following tasks:

1. Create a df with the NGrams and their frequencies separating the last word as the prediction. FUN= create_ngram_freqDF()
2. Calculate the probability for each frequency using Good Turing. FUN=good_tunning_smoothing()
3. For each unique NGram, create a dataframe with all possible predictions and their probability. Merge all dfs in a dictionary. FUN = create_full_ngram()

The functions mentioned above are in the functions.R script, and the actual code for running them and create the full dictionaries is in modelling.R. Prediction process is in pred.R