Week 2: Milestone Report

Data Science Capstone Project

1 Executive summary

This report is part of the Coursera data science capstone project. Provided with a large corpus of text documents in the form of blog entries, tweets or news articles, the goal of this week’s task is to perform first exploratory analyses and describe the further approach to build a next word prediction model.

Given the considerable size of the whole corpus, I analyse a subsample that covers 10% of the total English texts. As expected, I find that the distribution of words in the data set is heavily skewed. However, since the goal of this project is a next word prediction model, I decided to preprocess the corpus without stopword removal. Moreover, I analysed the probability distribution of bigrams and trigrams within the corpus. Not surprisingly, since the bigram model results in higher probabilities than the trigram model, I am planning to use the former for the word prediction model.

2 Exploratory data analysis

In my analysis, I focused on the English texts of the Swiftkey dataset which were provided as three single text documents containing lines of single contributions in the form of blog entries, news articles and tweets.

In total,there are 2545 words that occur in at least 0.1% of the documents. The following graph visualizes the number of occurences of terms that occur more than 15000 times in the corpus.

In the following is a table of the 10 least frequently occurring terms in the corpus:

lesson	raising	pointed	rick	supporting	numerous	determine	declined	regardless	upcoming
286	285	285	285	284	284	283	283	282	280

In the following a word cloud of the most common words is displayed:

3 Analysis of N-grams

For a first evaluation of n-grams, I used the #rstats-package ngram due to its good performance for large data sets. Not surprisingly, I find that the probabilities of an n-gram occuring are higher for the bigram as for the trigram. For this reason, I plan to use the bigram for my prediction algorithm.

Bigrams

Bigram	Frequency	Probability
of the	37138	0.0046946
in the	34211	0.0043246
to the	18060	0.0022830
on the	15893	0.0020090
for the	14849	0.0018771
to be	13005	0.0016440
at the	11412	0.0014426
and the	11127	0.0014066
in a	10014	0.0012659
with the	8895	0.0011244

Trigrams

Trigram	Frequency	Probability
one of the	2909	0.0003677
a lot of	2548	0.0003221
to be a	1360	0.0001719
the end of	1302	0.0001646
going to be	1273	0.0001609
as well as	1237	0.0001564
out of the	1222	0.0001545
some of the	1187	0.0001500
it was a	1176	0.0001487
be able to	1158	0.0001464

4 Next steps

In the next week I will focus on the prediction model. In order to be able to train the model, first an evaluation routine has to be defined.

In order to be able to measure the quality of my prediction model, I’m going to apply a train/test split. Hence, only the train subsample is going to be used to fit my actual prediction model to the data. Then, I measure the performance on the test set. In general, I assume that the higher the assigned probability to the test set is, the better my model performs. As a metric, I am going to use perplexity since minimizing perplexity is equivalent to maximizing probability.

Going forward, I plan to use a prediction model based on the ngrams that were generated by the data.