Introduction

This milestone report describes the process-to-date of obtaining, preparing and exploring the three en_US data sets in preparation for building a predictive text model application.

Overview and Data Preparation

Three distinct English-language corpora (text data sets) were supplied to use for building and testing a predictive text model: Twitter posts, blog entries, and news clips. In each of these three data sets, a row of data contains a line of distinct text, such as a phrase, sentence, or combination of sentences. The following steps were taken to prepare the data prior to computing summary statistics and conducting basic exploratory data analysis:

  1. All lines of text containing profanity were eliminated from the corpora using a custom filter.
  2. The three corpora were unioned and all non-apostrophe, non-alphanumeric characters were removed. For the Twitter data, this was an especially useful cleaning step as it eliminated instances of emojis, which appear in text data as complex sequences of symbols and would not be useful for a text predictive application.
  3. Only lines of text with four or more words were retained, as the predictive application will be built to predict the fourth word based on a three-word entry.
  4. Data were split into training (30%), testing (10%), and validation (10%) data sets. The remaining data were discarded so that model building could be done with a reasonably sized dataset. All following analysis refers to the data in the training set.

Style and Length

The style and length of the lines of text in the three corpora depends on the corpus. For example, Twitter text lines tend to be short, unstructured, include unique combinations of punctuation (emojis), and follow loose rules of grammar and punctuation:

## [1] "how are you? btw thanks for the rt. you gonna be in dc anytime soon? love to see you. been way, way too long."  
## [2] "when you meet someone special... you'll know. your heart will beat more rapidly and you'll smile for no reason."
## [3] "they've decided its more fun if i don't."                                                                       
## [4] "so tired d; played lazer tag & ran a lot d; ughh going to sleep like in 5 minutes ;)"                           
## [5] "words from a complete stranger! made my birthday even better :)"

News clips, while still succinct, show much more formal application of language:

## [1] "he wasn't home alone, apparently."                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
## [2] "the st. louis plant had to close. it would die of old age. workers had been making cars there since the onset of mass automotive production in the 1920s."                                                                                                                                                                                                                                                                                                                                                         
## [3] "wsu's plans quickly became a hot topic on local online sites. though most people applauded plans for the new biomedical center, many deplored the potential loss of the building."                                                                                                                                                                                                                                                                                                                                 
## [4] "the alaimo group of mount holly was up for a contract last fall to evaluate and suggest improvements to trenton water works. but campaign finance records released this week show the two employees donated a total of $4,500 to the political action committee (pac) partners for progress in early june. partners for progress reported it gave more than $10,000 in both direct and in-kind contributions to mayor tony mack in the two weeks leading up to his victory in the mayoral runoff election june 15."
## [5] "and when it's often difficult to predict a law's impact, legislators should think twice before carrying any bill. is it absolutely necessary? is it an issue serious enough to merit their attention? will it definitely not make the situation worse?"

The blog posts are a mix. The language and grammar is less formal than a news clip, but more formal than a blog post. Some blog posts are quite lengthy:

## [1] "chad has been awesome with the kids and holding down the fort while i work later than usual! the kids have been busy together playing skylander on the xbox together, after kyan cashed in his $$$ from his piggy bank. he wanted that game so bad and used his gift card from his birthday he has been saving and the money to get it (he never taps into that thing either, that is how we know he wanted it so bad). we made him count all of his money to make sure that he had enough! it was very cute to watch his reaction when he realized he did! he also does a very good job of letting lola feel like she is playing too, by letting her switch out the characters! she loves it almost as much as him."

Indeed, the blog posts contain the longest text entries, with an average character length of 213.6 characters per entry with the longest single entry containing 10785 characters. In contrast, the average Twitter entry contains 68.2 characters with no entry exceeding 144 characters. News clips, occupying the middle ground, have an average character length of 197.1 characters, with no entry exceeding 2000 characters. As if to make up for its abbreviated characters-per-entry, the Twitter corpus, with 2224675 text entries, is vastly larger than the other two corpora. The blogs corpus contains 835571 rows of text and the news corpus contains only 72006 rows of text. As a result, the predictive model will be trained on proportionately more Twitter data than the other two sources and this will have an impact on the kinds of words it predicts.

N-Gram Frequency Distribution

N-grams were extracted from the corpus. N-grams are phrases that contain N words. For example, the phrase “I love you” is a three-gram because it is composed of three words. For this work, one, two, three, and four-grams were extracted and their frequencies compiled. The figure below contrasts the frequency distributions of the various N-grams. Analysis follows.

One-Gram The one-gram (green), containing only single words, has the highest frequency entries; the most frequent single word appears nearly one million times in the corpus. Frequencies drop rather quickly from here. The 50,000th-ranked single word by frequency occurs between 5 and 10 times in the entire corpus. The green line comes to an abrupt end near the 350,000th rank. This is because there are no further unique words to rank.

Two-Gram In contrast to the one-gram, the two-gram’s (purple) top-ranked phrase occurs nearly 70,000 times, well off the almost one million-frequency mark achieved by the most common one-gram. Another clear point of contrast is the much more shallow curve that follows from the 1,000-frequency mark. Compared to the one-gram, which hits the 10-frequency mark around the 45,000th rank, there is no 10-frequency two-gram until just beyond the 150,000th rank. These results tell us something about the structure of the English language as communicated in text. A small number of single words are used at very high frequencies, and then the supply of single words in regular use dissipates quite quickly. The act of combining two words together still results in a small number of high-frequency occurrences, but even the most frequent two-gram phrase occurs far less frequently than the highest one-gram. However, there are far more unique two-grams being put to use. The supply of two-grams does not dissipate; even the 500,000th ranked two-gram appears ~ 3 times in the corpus.

Three and Four Grams The three-gram (aqua) mirrors the two-gram’s pattern closely, except that it is shifted downward on the vertical axis. This means that, like the two-gram, the three-gram has a small number of high-frequency phrases and frequency tapers off slowly, where even low-ranked three-grams still have periodic use. The downward shifting means that, given the same frequency rank, counts are lower for three-grams compared to two-grams.

The four-gram’s (red) curve appears to share certain features of all the other curves. Like the one-gram, the supply of four-grams in regular use diminishes rapidly. In fact, the four-gram’s curve has the steepest drop in frequency through the 10-frequency mark. After this point, the four-gram curve resembles that of the two and three-gram curves; the supply of four-grams in use is robust and diminishes slowly. Even the lowest-ranked four-gram appears twice in the corpus.

N-Gram Top 20 Words and Phrases

The figure below displays the most frequently used words or phases in the four N-grams compiled for this work.

Summary The common phrases reflect the nature of bias towards Twitter-sourced text in the training data. The most popular four-gram is “thanks for the follow”, a phrase that is unlikely to be all that popular in non-Twitter corpora. The most popular three-gram is similar to the four-gram. In both charts, thanking phrases appear at least twice. Interestingly, thanking phrases are absent from the top twenty two-grams, where more generic two-word phrases dominate the distribution. The two-gram appears less biased toward any one corpus, as is the case with the one-gram. One final observation - it is much more plain to see here that the top 20 one-grams appear far more frequently than the top 20 from the other N-grams. The top 20 frequencies taper off according to the N in the N-grams, as might be predicted from the earlier line graph.

Next Steps

The next step is to build a 4th-word predictive model from the N-grams. The model will output three suggestions of a fourth word (like a cell phone text application does), given user input of three words. For this, a Katz backoff model with unseen-word discounting will be utilized. The Katz backoff model works by utilizing the frequency distributions of the various N-grams, modified to allow for words and phrases that don’t appear anywhere in the training data to nonetheless produce a response from the model.

For example, if the user inputs three words, the model will first search the four-gram for instances where the three words appear as the first three words and will suggest the top three fourth words using a modified probability calculation. If no there are no instances where the user three-word input matches a four-gram in the training data, the model will move to the three-gram and search for instances where the second and third word of the user’s input appear together at the start of the three-word phrase. If there is a match, the model will again return three suggestions based on a modified probability. The model proceeds all the way down to the one-gram, where, if there are still no matching phrases, the model will output the three most frequent one-grams by their adjusted probabilities: “the”, “to”, and “and”.

Once the model is built, the speed and accuracy of the model will be evaluated. Can a model be built from less data and still retain accuracy based on how well it predicts the test data?

Finally, the application will be built and published to shinyapps.io.