Data Science Capstone - Milestone Report - EDA

John Hopkins University Data Science Specialisation

Author

Alex Hall

1 Introduction

I’ll acknowledge that this report could be more brief and concise, but hopefully is written in a way that a non-technical person could understand. I also think it would lose important context on the exploration to date if it was shorter. Hopefully reviewers will not grade this element too harshly.

1.1 Background

This milestone report is for the week 2 assignment of the capstone project, as part of the John Hopkins University Data Science Specialisation.

The aim of the capstone project is to build a predictive text model that can predict the next word in a sentence. This model will be used for an R Shiny dashboard product and accompanying presentation.

This report summarises tasks 0 (understanding the problem), 1 (getting and cleaning the data), 2 (exploratory data analysis) and 3 (modelling), with particular focus on the latter two. Specific questions are outlined below.

1.2 Task 2 aims

1.2.1 Tasks to accomplish

  • Exploratory analysis - perform a thorough exploratory analysis of the data, understanding the distribution of words and relationship between the words in the corpora.
  • Understand frequencies of words and word pairs - build figures and tables to understand variation in the frequencies of words and word pairs in the data.

1.2.2 Questions to consider

  • Some words are more frequent than others - what are the distributions of word frequencies?
  • What are the frequencies of 2-grams and 3-grams in the dataset?
  • How many unique words do you need in a frequency sorted dictionary to cover 50% of all word instances in the language? 90%?
  • How do you evaluate how many of the words come from foreign languages?
  • Can you think of a way to increase the coverage – identifying words that may not be in the corpora or using a smaller number of words in the dictionary to cover the same number of phrases?

1.3 Task 3 aims

1.3.1 Tasks to accomplish

  • Build basic n-gram model: Using the exploratory analysis you performed, build a basic n-gram model for predicting the next word based on the previous 1, 2, or 3 words.
  • Build a model to handle unseen n-grams: In some cases people will want to type a combination of words that does not appear in the corpora. Build a model to handle cases where a particular n-gram isn’t observed.

1.3.2 Questions to consider

  • How can you efficiently store an n-gram model (think Markov Chains)?
  • How can you use the knowledge about word frequencies to make your model smaller and more efficient?
  • How many parameters do you need (i.e. how big is n in your n-gram model)?
  • Can you think of simple ways to “smooth” the probabilities (think about giving all n-grams a non-zero probability even if they aren’t observed in the data) ?
  • How do you evaluate whether your model is any good?
  • How can you use backoff models to estimate the probability of unobserved n-grams?

2 Data

2.1 Data sources

The raw data for this project was text in four languages: English, German, Finnish and Russian. For each language, there were three text files for text sourced from Twitter, News and Blogs. As an English speaker, I chose to only use the English language text files.

2.2 Data source summary

The following table shows a summary of the three source files.

Source Size (MB) Number of lines Number of words Mean words per line SD words per line
Twitter 334.48 2,360,148 30,373,543 12.87 6.90
Blogs 267.76 899,288 37,334,131 41.52 46.27
News 269.84 1,010,242 34,372,530 34.02 22.59

2.3 Sampling

Given each source was a large dataset, initial exploration and modelling was conducted on a sample. If the amount of data turned out to be a problem, then more data could be sampled in the future.

In order to not bias the dataset, an equal number of lines were randomly sampled from each source, even though each had a very different number of lines. 10,000 was chosen as an arbitrary round number that made up nearly 10% of the Blogs dataset.

Data was loaded using readr::read_lines() and then sampled used base::sample(). This report is one step in a ‘Targets’ pipeline so does not include the full code, but it can be found here.

3 Transformations

3.1 Remove profanity

Course instructions in the first week suggested removing profanity from the data so that offensive words would not be modelled and therefore suggested as part of the prediction.

This was done on the raw text using the sentimentr::profanity() function to detect instances of profanity in a document and remove that document from the dataset. In practice, this removed <5% of documents so was deemed acceptable. The alternative would have been to remove single words which would have produced unnatural n-grams.

This was done before further pre-processing for two reasons. Firstly so that further steps would run more quickly with a slightly reduced dataset. Also, the profanity detection function depended on sentences, which could not be determined if punctuation was removed, which is what happens in later pre-processing steps.

3.2 Further preprocessing

After documents containing profanity were removed, further text reprocessing was conducted to manipulate the data into a form suitable for text predictive models. This was done using the tidytext::unnest_tokens() function and included removing digits, punctuation and stop words.

4 Single word (unigram) exploration

4.1 Overall frequency

Across all sources, it appears that the most frequently used words were ‘time’, ‘day’ and ‘people’.

## Count by word
count_word_df_head <- 
    word_df %>% count(word, sort = TRUE) %>%
    head(25) %>%
    mutate(word = reorder(word, n))

## plot
count_word_df_head %>%
    ggplot(aes(y = word, x = n)) +
    geom_col()

4.2 Frequency by source

By source, there are some more differences. ‘Time’ and ‘people’ are still top for Blogs and News sources, but ‘love’, ‘day’ and ‘rt’ (retweet) are higher for the Twitter source.

## Count by word - grouped
count_word_df_grp_head <-
    word_df %>%
    dplyr::group_by(text_source) %>%
    count(word, sort = TRUE) %>%
    slice_head(n = 25)

## plot
count_word_df_grp_head %>%
    mutate(word = reorder_within(word, n, text_source)) %>%
    ggplot(aes(y = word, x = n)) +
    geom_col() +
    facet_wrap(vars(text_source), scales="free") + 
    scale_y_reordered()

4.3 Overall distribution

Looking at the word frequencies in a boxplot shows a very skewed distribution with a large proportion of words only appearing once, or very few times.

## Count by word
count_word_df <- 
    word_df %>% count(word, sort = TRUE) %>%
    mutate(word = reorder(word, n))

## plot
count_word_df %>%
    ggplot(aes(x = n)) +
    # geom_histogram(binwidth = 50)
    geom_boxplot()

4.4 Distribution by source

The same overall pattern is observed for each source, but for Twitter, it seems it has a slightly higher proportion of words appearing only once, 61.03% for Twitter as opposed to 50.90% for Blogs. Overall there are 23529 words that only appear once in the corpus (50.27%).

## Table calculated above
## plot
count_word_df_grp %>%
    ggplot(aes(x = n, group = text_source, colour = text_source)) +
    # geom_histogram(binwidth = 50)
    geom_boxplot() +
    scale_x_continuous(limits = c(0, 100))

Source Words appearing once (n) % of total
Blogs 14550 50.90%
News 14330 50.87%
Twitter 8413 61.03%

5 Bigram exploration

5.1 Overall frequency

Interestingly, place names are the most frequent when looking at bigrams. Given that Twitter, News and Blogs would be heavily influenced by current affairs at the time, it is possible that coincidentally something had happened in St Louis, Los Angeles and San Fransisco at the time this data was collected.

## Count by bigram
count_bigram_df <- 
    bigram_df %>% count(bigram, sort = TRUE) %>%
    head(25) %>%
    mutate(bigram = reorder(bigram, n))

## plot
count_bigram_df %>%
    ggplot(aes(y = bigram, x = n)) +
    geom_col()

5.2 Frequency by source

By source the above observations hold true for the News source, but not Blogs or Twitter. Mentions of special occasions (Mother’s Day and Birthday) are frequent in the Twitter source, particularly ‘Happy Birthday’ appearing twice as many times as the next bigram. For Blogs, the bigram frequencies are more evenly dispersed numerically but also in topics.

## Count by word - grouped
count_bigram_df_grp <-
    bigram_df %>%
    dplyr::group_by(text_source) %>%
    count(bigram, sort = TRUE) %>%
    slice_head(n = 25)


## plot
count_bigram_df_grp %>%
    mutate(bigram = reorder_within(bigram, n, text_source)) %>%
    ggplot(aes(y = bigram, x = n)) +
    geom_col() +
    facet_wrap(vars(text_source), scales="free") +
    scale_y_reordered()

6 Trigram exploration

6.1 Overall frequency

Gov Chris Christie was the most frequently observed trigram. Other names (people, places, titles) appeared towards the top of this list.

## Count by trigram
count_trigram_df <- 
    trigram_df %>% count(trigram, sort = TRUE) %>%
    head(25) %>%
    mutate(trigram = reorder(trigram, n))

## plot
count_trigram_df %>%
    ggplot(aes(y = trigram, x = n)) +
    geom_col()

6.2 Frequency by source

As with bigrams, trigrams from the News source appeared frequently. Other than that, in this limited sample, occurrences were generally low so are difficult to infer meaning from.

## Count by word - grouped
count_trigram_df_grp <-
    trigram_df %>%
    dplyr::group_by(text_source) %>%
    count(trigram, sort = TRUE) %>%
    slice_head(n = 25)


## plot
count_trigram_df_grp %>%
    mutate(trigram = reorder_within(trigram, n, text_source)) %>%
    ggplot(aes(y = trigram, x = n)) +
    geom_col() +
    facet_wrap(vars(text_source), scales="free") +
    scale_y_reordered()

7 Unique words required for coverage

One of the quested posed in the assignment was to consider how many unique words would be required to cover 50% of all words in the data. The results of this are below.

## Count by word, with cumulative proportion and row number
count_word_df <- 
    word_df %>% count(word, sort = TRUE) %>%
    mutate(word = reorder(word, n)) %>%
    mutate(
        cum_prop = cumsum(n) / sum(n),
        no_words = row_number())

## work out min number of words
cov_words_50 <-
    count_word_df %>%
    filter(cum_prop >= 0.50) %>%
    select(no_words) %>%
    summarise(min_value = min(no_words)) %>%
    pull()


## work out min number of words
cov_words_90 <-
    count_word_df %>%
    filter(cum_prop >= 0.90) %>%
    select(no_words) %>%
    summarise(min_value = min(no_words)) %>%
    pull()

7.1 50% coverage

In this sample, 1693 unique words are required to cover 50% of word occurrences.

7.2 90% coverage

In this sample, 17641 unique words are required to cover 90% of word occurrences. Interestingly, this shows that not many more words are required to achieve much greater coverage of the data.

8 Identify foreign languages

Another question for consideration was how non-English language words could be identified.

## Append to dataframe with language detection and spell check
word_df_spellcheck <-
    word_df %>%
    mutate(
        language = cld2::detect_language(word),
        spell_check = hunspell::hunspell_check(word) ## Default english US language
    )

## words failed spell check - count
word_df_spellcheck_fail <- sum(word_df_spellcheck$spell_check)

## total words
word_df_spellcheck_total <- nrow(word_df_spellcheck)
    
## prop
word_df_spellcheck_prop <- scales::percent(word_df_spellcheck_fail / word_df_spellcheck_total, accuracy = 0.01)

The cld2 package was unable to detect language in most cases and in cases when it did, it seemed to classify some English words as non-English, e.g. “funky”.

The hunspell::hunspell_check() function classified a lot of words (297668 out of 347934, 85.55%) as not in the dictionary, suggesting they are non-English. It is not feasible to remove this many words from the corpus and indeed it suggests that natural language is not always spelt correctly. Therefore, we will not remove words highlighted by these approaches.

9 Plans for prediction algorithm and Shiny app

I will first answer the questions posed for consideration in the assignment, then I will give a summary of my plans for the prediction algorithm and Shiny app.

9.1 Task 3 Questions to consider

9.1.1 How can you efficiently store an n-gram model (think Markov Chains)?

My current plan is to use ngram models which, to my understanding, are a form of Markov Chain. The assumption is that the previous word (bigram) or word before the previous word (trigram) can be used to predict the next word, and probabilities can be calculated for transitioning from one state to another.

My only reservation with this approach so far is that my models do not include frequencies for tokens (unigrams, bigrams, trigrams) that were not in the sampled data. This does dramatically reduce the size of the models, but does not seem to allow smoothing, at least not for the purposes of making unseen words possible, probability-wise. Perhaps unseen tokens could be generated and smoothed probabilities assigned, which could aid the final prediction. This will be explored in later steps of this project.

I am currently planning to use 5-gram and 6-gram models, but these may be too sparse and not generalisable.

9.1.2 How can you use the knowledge about word frequencies to make your model smaller and more efficient?

For predicted words with low probabilities (based on low frequency in corpus), I plan to replace them with a dummy ‘unknown’ token, which will reduce the overall number of predicted words and therefore the model size.

9.1.3 How many parameters do you need (i.e. how big is n in your n-gram model)?

I am going to attempt a combination of six models (unigram to six-gram), so that up to the last five words of a user input can be used in the prediction. As above, this may be revised if it means the model is too large and/or this does not significantly improve the accuracy.

9.1.4 Can you think of simple ways to “smooth” the probabilities (think about giving all n-grams a non-zero probability even if they aren’t observed in the data)?

There are various methods available for smoothing from simple Laplace smoothing up to Kneser-Ney smoothing. As it stands, my models only include tokens observed in the sampled data. The plan is that a ‘backoff’ approach can be used until a token is ‘seen’, rather than generating unseen tokens and calculating smoothed probabilities. This approach may change depending on results.

9.1.5 How do you evaluate whether your model is any good?

I will split the sampled data into train, development and test subsets, mostly likely in a 60/30/10 ratio. The development subset will be used to tune hyper-parameters in the models built using the training data and the test dataset will be used to test final accuracy. Accuracy evaluation metrics will include the proportion of time the actual word is within the top three or top five predicted words. Perplexity will also be used to comparatively assess different models, aiming for the lowest perplexity.

9.1.6 How can you use backoff models to estimate the probability of unobserved n-grams?

I plan to use a combination of backoff models and interpolation to combine predictions from multiple ngram models. For example, the last five words entered by the user will be checked in the six-gram model and if it doesn’t appear, the last 4 words will be checked in the five-gram model and so on until the unigram model, which will simply suggest the most frequently occurring words in the corpus. In occasions where multiple ngrams can be found in the different ngram models, probabilities will be calculated across all of them to provide final predictions. These probabilities will be weighted via interpolation using values informed by the hyperparamter tuning.

9.2 Prediction algorithm

The detail of my plan to create a prediction algorithm is outlined above. In addition to this, there are few deviations I will make from the way the data has been processed so far, and a few changes I will consider making depending on initial prediction results.

I have maintained this report as a point in time summary of progress and so it won’t completely describe the data that I end up using in the prediction algorithm.

9.2.1 Stop words

Based on my understanding of common natural language processing (NLP) key pre-processing steps, I removed stop words from my sample but I can now see that this was the wrong decision. Stop words (“the”, “I”, “a”) actually are an important part of natural language and so should be included if the goal is to predict the next word in a sequence. They are less relevant in other NLP tasks like classification.

9.2.2 Sample size

I initially took 10,000 items from each source (~10%), but I think I will probably need to include more to enable more accurate predictions.

9.2.3 Train/test split for model evaluation

A larger sample will be particularly useful considering that I will split the sample into training, development and testing subsets.

9.2.4 Other changes to consider

My planned approach does not incorporate any smoothing and my models do not include 0-frequency words. Hopefully what this enables in terms of reduced model size compensates for the lack of potential benefits from smoothing.

I expect this to be a trade-off throughout the remainder of the project: Including enough data in the model to maximise it’s exposure to word sequence combinations, whilst not making it so large that it exceeds Shiny app limitations or becomes inefficient and leads to a poor user experience.

9.3 Shiny App

For the R Shiny application for this project, I plan to build a relatively simple app whereby users can enter text into a text box and ultimately the app will suggest the most likely next five words. It will do this by looking at the last n words in the text and looking for these in various n-gram models, as described above. Potentially I will make these suggested words into buttons so that the user can click on them and that word will be added to the entered text so that the possible next word is suggested, and so on.

I plan to include project background information in the app and details on how to use it and interpret the results.

I will consider including other features to enhance the user experience like a timer to show computation time for each prediction.