Introduction

The goal of this project is to develop a predictive text application that can suggest the next word while a user is typing. Before building the prediction model, I explored the English training data to understand its size, structure, and common patterns.

The data comes from three sources: news, blogs, and Twitter. These sources have different writing styles, which makes them useful for understanding how word usage changes across different types of text.

Data Summary

The three datasets contain a large amount of text.

Source Lines Words
News 1,010,206 34,371,031
Blogs 899,288 37,334,131
Twitter 2,360,148 30,373,543

Blogs contains the largest number of words, while Twitter contains the largest number of individual lines. This reflects the fact that Twitter entries are generally much shorter than news or blog entries.

Most Frequent Words

I examined the most frequently occurring words in each source. The word the was the most frequent word in all three datasets.

In the News data, common words included the, to, a, and, and of. In Blogs, words such as i, you, and my were more prominent. Twitter also showed high frequencies for i, you, my, and me.

These differences give some indication of the different writing styles. Blogs and Twitter appear to contain more personal and conversational language than News.

News

Blogs

Twitter

Word Frequency Distribution

The frequency distribution is highly uneven. A relatively small number of words occur very frequently, while a much larger number of words occur less often.

For the News data, there were 212,224 unique words in the frequency-sorted dictionary. The 168 most frequent words covered approximately 50% of all word instances, while 7,140 words covered approximately 90%.

This is useful for the prediction model because it shows that a relatively small vocabulary can cover a large proportion of normal word usage.

Two-Word Combinations

I also examined common two-word combinations, or bigrams, using a 50,000-line sample of the News data.

The most frequent bigrams included of the, in the, to the, on the, and for the.

Most of the common bigrams are grammatical combinations. However, combinations such as he said also provide useful contextual information.

Three-Word Combinations

I also examined trigrams, or three-word combinations. Some of the most frequent examples were one of the, a lot of, the u s, it s a, and i don t.

These repeated phrases are useful for a predictive text model because they provide more context than individual words.

Data Cleaning Observations

One issue noticed during cleaning was the treatment of contractions. For example, it's was converted to it s, and don't was converted to don t.

This explains the relatively high frequency of tokens such as s and t. This is an important issue to consider before developing the final prediction model.

The corpus may also contain unusual or foreign words. I do not plan to remove every unusual word automatically because some may be legitimate parts of the original text.

Plans for the Prediction Algorithm

The next stage will be to build an n-gram based prediction model. The model will use the previous one, two, or three words to estimate the most likely next word.

If a particular combination is not present in the training data, I plan to use a backoff approach. The model can first look for a longer n-gram and then use a shorter n-gram when the longer combination is unavailable.

The model will also need to balance prediction accuracy with memory usage and prediction speed. A model that is slightly simpler but much faster may provide a better user experience.

Plans for the Shiny Application

The final goal is to create a simple Shiny application where a user can enter text and receive a suggested next word.

The application should be easy to use and provide a prediction quickly. The prediction algorithm will run behind the interface and return the most likely next word based on the text entered by the user.

Conclusion

The exploratory analysis provided a better understanding of the English training data and the differences between News, Blogs, and Twitter.

The data contains millions of words, but the frequency analysis shows that a relatively small number of common words account for a large proportion of normal word usage. The bigram and trigram analysis also shows that the relationship between neighboring words is important.

These findings will be used to develop the n-gram prediction model, with particular attention to unseen word combinations, accuracy, memory usage, and prediction speed.