Purpose

This report presents the findings of an exploratory analysis of text data from news, blogs, and Twitter. The corpus contains text in four languages: English, Finnish, German, and Russian. Ultimately, this data will be used to produce an n-gram text prediction algorithm, which predicts the next word based on user-entered text. This algorithm will be used to create a Shiny application. This exploratory analysis uses sample text from each file to identify the basic features of the corpus, including the frequency distribution of one-, two-, and three-word sequences and the number of unique words required to account for 50% and 90% of observed words in the data.

Approach

We first select a random sample of 5,000 lines from each file. Each file is relatively large, so using a smaller sample will give us an adequate summary of the data without requiring us to download and parse each entire file. We combine the sample text from each file into a corpus containing the language (English, Finnish, German, or Russian), source file name, and sampled text. We clean the sample data by removing malformed characters, white space, puncutation, and blank lines. We also convert all text to lowercase. These steps make it easier to identify individual words and word sequences (i.e., tokens).

Summary statistics

Before tokenizing the data, we first examine the number of lines and words for each language after sampling 5,000 lines for each of the three files per language. As expected, the resulting table shows there are 15,000 lines per language:

## # A tibble: 4 × 3
##   language lines  words
##   <chr>    <int>  <int>
## 1 english  15000 447337
## 2 german   15000 500478
## 3 finnish  15000 308837
## 4 russian  15000 427729

Next, we tokenize the data, producing a dataset containing one row per token. Tokens include unigrams (individual words), bigrams (sequences of two words) and trigrams (sequences of three words). These tokens will be the basis for the model that predicts the next word based on user input. Therefore, we examine the distribution of the frequencies of the most common tokens.

Distribution of Word Frequency

We use a rank-frequency plot to identify whether each language follows Zipf’s Law. That is, that a small number of words occurs more frequently than larger numbers of words in each language. The plot indicates that each language follows Zipf’s Law.

We also examined the presence of “cross-script” tokens (i.e., Cyrillic alphabet characters in the non-Russian language files and Latin alphabet characters in the Russian language files). The proportion of tokens that appear to be a foreign language is extremely low for each language, albeit slightly higher in Russian compared to other languages:

## # A tibble: 4 × 2
##   language foreign_percent
##   <chr>              <dbl>
## 1 english            0    
## 2 finnish            0.002
## 3 german             0    
## 4 russian            0.71

Coverage Analysis

Finally, we evaluated how many unique words are required to account for 50% of observed word occurrences and, separately, how many words are required to account for 90% of observed occurrences. We find that coverage varies by language, but generally, the data has the best coverage for English: approximately 21% of words are needed to account for 90% of occurrences.

## # A tibble: 4 × 6
##   language vocab_50 vocab_90 total_vocab pct_vocab_50 pct_vocab_90
##   <chr>       <int>    <int>       <int>        <dbl>        <dbl>
## 1 english       156     7368       35216          0.4         20.9
## 2 finnish      1813    55495       86581          2.1         64.1
## 3 german        223    19938       65544          0.3         30.4
## 4 russian       975    35912       78503          1.2         45.7

Plan/Next Steps

Based on this preliminary examination, we will construct a backoff n-gram probability model that calculates the conditional probabilities for each bigram and trigram sequence (the likelihood of a word occurring given the preceding word(s)). The backoff model will start by searching for matching trigrams. If the model does not identify matching trigrams, it will “back off” to search for matching bigrams, and then search for the most frequent unigrams if no bigrams are identified. Given the evaluation of coverage, we expect that the model will have the strongest predictive power for English-language predictions.