Exploratory Data Analysis On Natural Language Prediction
Okwumuo Martin
2026-08-25
1. Project Overview
The objective of this project is to develop a next-word prediction system for natural language that can suggest the most likely word a user may type based on the words that have already been entered.
The project uses three large English-language text datasets containing material from news, blogs and Twitter. Exploratory Data Analysis (EDA) was first performed to understand the size, structure, vocabulary, word usage and challenges contained in the data before developing the predictive model.
The eventual system will be implemented as an interactive Shiny application, allowing a user to enter text and receive a ranked list of likely next words.
2. Overview of the Three Datasets
The three datasets contain more than 4.4 million lines of text and approximately 572 million characters.
| Dataset | Lines | Characters | Mean Line Length | Median Line Length |
|---|---|---|---|---|
| News | 1,010,206 | 203,214,543 | 201.2 | 185 |
| Blogs | 899,288 | 206,824,505 | 230.0 | 156 |
| 2,360,148 | 162,096,031 | 68.7 | 64 |
Key observations
- Twitter contains the largest number of lines, reflecting its large volume of short social-media messages.
- Blogs have the longest average line length, reflecting their longer-form writing style.
- Twitter has substantially shorter messages, making it particularly different from the news and blog sources.
- Together, the three datasets provide a mixture of formal, informal and conversational English.
3. Distribution of Text Length
The distribution of line lengths was examined to understand how much text users are likely to encounter in each source.
The analysis confirms that the datasets have substantially different text-length characteristics. Twitter messages are generally short, while blogs and news contain considerably longer passages.
This difference is important because the prediction system must be capable of handling both short conversational input and longer-form language.
4. Word Counts and Vocabulary
A development sample was used for detailed word-level analysis.
The sample contained:
- 879,617 tokens
- 96,053 unique tokens
- Type–Token Ratio (TTR): 10.92%
The source-specific analysis produced the following results:
| Dataset | Tokens | Unique Tokens | TTR |
|---|---|---|---|
| News | 340,221 | 50,759 | 14.92% |
| Blogs | 410,349 | 53,056 | 12.93% |
| 129,047 | 24,982 | 19.36% |
The Type-Token Ratio measures vocabulary diversity. A higher value indicates that a larger proportion of the words are unique.
Twitter recorded the highest TTR in the analyzed sample. This suggests that, relative to its token count, Twitter contains considerable variation in the words being used.
5. Most Frequently Used Words
The most frequent words were dominated by common English words:
| Rank | Word | Frequency |
|---|---|---|
| 1 | the | 43,512 |
| 2 | to | 23,783 |
| 3 | and | 22,154 |
| 4 | a | 20,890 |
| 5 | of | 18,758 |
| 6 | in | 14,967 |
| 7 | i | 12,578 |
| 8 | for | 9,128 |
| 9 | that | 9,309 |
| 10 | is | 8,875 |
These results are expected because words such as the, and, to, of and in perform important grammatical functions in English.
They also demonstrate an important characteristic of language data; a relatively small number of common words occur very frequently, while many other words occur only rarely.
6. Word Frequency Distribution
The frequency distribution showed a strong imbalance between common and rare words.
In the analyzed vocabulary:
| Frequency Threshold | Vocabulary | Percentage |
|---|---|---|
| Frequency = 1 | 61,170 | 63.68% |
| Frequency ≤ 2 | 73,123 | 76.13% |
| Frequency ≤ 5 | 84,011 | 87.46% |
Therefore, almost 64% of the vocabulary appeared only once.
This is an important finding for the predictive model. A model cannot rely entirely on memorizing individual word combinations because many combinations will occur too rarely to provide reliable predictions.
7. Data Quality and Text Characteristics
Several characteristics were identified during pre-processing and tokenization.
The datasets contain:
- URLs such as
http://... - Twitter user mentions such as
@username - Hashtags such as
#football - Numbers
- Punctuation
- Emoticons such as
:),:(,;)and:/ - Contractions such as
don't,it's,I'm,you'reandcan't - Words attached to punctuation such as
said.andtime,
These characteristics mean that text pre-processing and tokenization are critical components of the project.
The pre-processing stage therefore standardizes elements such as URLs, users and numbers while retaining useful linguistic information.
8. Bi-gram and Tri-gram Analysis
To determine whether previous words can help predict the next word, bi-grams and tri-grams were constructed.
A bi-gram contains two consecutive words, while a tri-gram contains three.
Examples from the analysis include:
| Context | Most Likely Next Word | Probability |
|---|---|---|
i want |
to |
58.99% |
the first |
time |
19.68% |
you can |
see |
6.95% |
These results demonstrate that context provides valuable information for predicting the next word.
For example, when the system sees:
“I want …”
the word “to” is substantially more likely than many alternatives.
9. The Main Modeling Challenge: Data Sparsity
The tri-gram analysis identified an important challenge.
There were:
- 804,726 unique trigrams
- 735,373 occurred only once
- 91.38% were singletons
- Only 8.62% occurred more than once
- Only 3.57% occurred at least three times
- Only 1.39% occurred at least five times
This phenomenon is known as data sparsity.
In practical terms, the model will frequently encounter word combinations that were not seen during training.
Therefore, a model based exclusively on exact tri-grams would have limited coverage.
10. Implication for the Prediction Algorithm
The EDA findings directly influence the design of the eventual algorithm.
The planned prediction strategy is:
Tri-gram - Bi-gram - Uni-gram
The system will first look for the most specific available context.
For example:
I want to
If a reliable tri-gram pattern is available, it will be used.
If sufficient tri-gram information is unavailable, the model can fall back to the bi-gram:
want - ?
If that is also unavailable, the uni-gram frequency distribution can provide a general prediction.
This back-off strategy should make the application more robust when users enter uncommon or previously unseen combinations.
11. Model Development Strategy
A separate development sample was divided into three groups:
| Dataset | Records | Percentage |
|---|---|---|
| Training | 24,000 | 80% |
| Validation | 3,000 | 10% |
| Testing | 3,000 | 10% |
The three groups contained no overlapping text records.
The purpose of the split is:
- Training data: build the language model.
- Validation data: tune and compare modeling approaches.
- Testing data: provide an independent measure of final performance.
12. Goal of the Final Shiny Application
The final objective is to develop a simple and useful next-word prediction application using Shiny.
A user might enter:
“I want”
The application could respond with:
- to
- a
- you
- it
- the
The application should provide predictions quickly and present several alternatives rather than only one word.
The planned workflow is:
User Input - Text Preprocessing - Context Detection - Language Model - Ranked Predictions - Shiny Interface
13. Planned Model Evaluation
The model will eventually be evaluated using measures appropriate for a next-word prediction system, including:
- Top-1 Accuracy; whether the correct word is the first prediction.
- Top-3 Accuracy; whether the correct word appears among the first three predictions.
- Top-5 Accuracy; whether the correct word appears among the first five predictions.
- Coverage; how often the system can produce a meaningful prediction.
- Prediction speed: how quickly predictions can be generated.
These measures will help determine whether the model is not only statistically accurate but also useful in an interactive application.
14. Key Findings from the EDA
The exploratory analysis produced five major findings:
- The datasets are large and diverse, containing more than 4.4 million lines of text.
- The sources have different writing characteristics, ranging from long-form blogs and news to short Twitter messages.
- A small number of words occur very frequently, while the majority of vocabulary is relatively rare.
- Context strongly influences the next word, as demonstrated by the bi-gram and tri-gram probabilities.
- Tri-gram sparsity is a major challenge, making a back-off strategy necessary.
Overall, the EDA provides evidence that the datasets are suitable for developing a next-word prediction system, while also identifying the pre-processing and modeling challenges that must be addressed.
15. Conclusion
The exploratory stage has established a clear foundation for the predictive modeling phase.
The data contain sufficient linguistic information to learn meaningful word relationships, but their high vocabulary diversity, punctuation, social-media conventions and substantial n-gram sparsity require careful pre-processing and model design.
The next stage will therefore focus on building and optimizing the uni-gram, bi-gram and tri-gram language models, followed by smoothing, back-off and predictive performance evaluation.
The ultimate goal is to convert these statistical patterns into a fast, accurate and user-friendly Shiny application capable of suggesting the most likely next word as a user types.
Reproducibility Note
All results in this report were generated from the English-language text datasets used in the project. The analysis was performed in R, with visualization and reporting implemented using R Markdown, supporting reproducibility and transparent analysis.