Data Science Capstone Project: Milestone Report
Agnieszka Chojecka
2026-08-07
Introduction
The goal of this project is to develop a predictive text application that suggests the next word based on the words entered by the user.
This project is part of the Data Science Specialization by Johns Hopkins University. The final product will be an interactive Shiny application using natural language processing techniques.
Data Description
The training data consists of three text sources provided by the course:
- Blogs
- News articles
- Twitter messages
The data contains text samples collected from the United States in English.
Loading the Data
The datasets were loaded into R and basic statistics were calculated, including the number of lines and total number of words.
Dataset Summary
| Dataset | Lines | Words |
|---|---|---|
| Blogs | 899288 | 37334131 |
| News | 1010206 | 34371031 |
| 2360148 | 30373543 |
Exploratory Data Analysis
Word Frequencies
Most Frequent Bigrams
Findings
The datasets differ considerably in size. Twitter contains the largest number of text lines, with more than 2.3 million lines, while the blogs and news datasets contain approximately 0.9 million and 1.0 million lines, respectively.
The three datasets contain approximately 37.3 million words in blogs, 34.4 million words in news, and 30.4 million words in Twitter messages.
The word frequency analysis shows that common English words dominate the corpus. The n-gram analysis also revealed frequent word combinations that can provide useful context for predicting the next word.
Prediction Algorithm Plan
The final prediction algorithm will use an n-gram language model.
The planned approach:
- Use trigram probabilities to predict the next word.
- If no trigram match is available, use bigram probabilities.
- Return the most frequent word as a fallback.
Shiny Application Plan
The final application will allow users to enter a phrase and receive a suggested next word.
The application will contain:
- a text input field,
- a prediction button,
- a displayed prediction result.
The goal is to provide a simple and intuitive interface that allows users to interact with the prediction model without needing any technical knowledge.