The objective of this project is to explore English text datasets provided for the SwiftKey Data Science Capstone project.
The analysis focuses on three datasets: blogs, news and Twitter. The ultimate goal is to develop a next-word prediction algorithm and an interactive Shiny application.
The project uses three English text datasets:
These datasets provide a large collection of real-world English text for understanding language patterns.
The following table presents the basic statistics calculated from the three datasets.
| Dataset | Lines | Words | Size (MB) |
|---|---|---|---|
| Blogs | 899,288 | 37,334,131 | 200.42 |
| News | 1,010,206 | 34,371,031 | 196.28 |
| 2,360,148 | 30,373,543 | 159.36 | |
| Total | 4,269,642 | 102,078,705 | 556.06 |
Twitter contains the highest number of lines, while blogs contain the highest number of words.
Together, the three datasets contain more than 102 million words.
A random sample of 10,000 lines was selected from each dataset, giving a combined sample of 30,000 lines.
The text was converted to lowercase, and punctuation and special characters were removed. Common English stop words were removed to identify meaningful words.
The following table shows the 20 most frequent meaningful words in the sample.
| Word | Frequency |
|---|---|
| said | 3048 |
| will | 2767 |
| one | 2680 |
| can | 2551 |
| just | 2328 |
| like | 2089 |
| time | 1882 |
| get | 1717 |
| new | 1628 |
| now | 1440 |
| people | 1389 |
| day | 1322 |
| also | 1319 |
| good | 1311 |
| year | 1298 |
| first | 1294 |
| know | 1232 |
| back | 1226 |
| don | 1203 |
| two | 1173 |
The analysis identifies words such as “said”, “will”, “one”, “can” and “just” among the most frequently used words.
The following table summarizes the number of words per text line.
| Statistic | Words |
|---|---|
| Minimum | 1.00 |
| First Quartile | 9.00 |
| Median | 20.00 |
| Mean | 29.45 |
| Third Quartile | 39.00 |
| Maximum | 808.00 |
The average number of words per line is approximately 29.45, while the median is 20. The maximum observed line contains 808 words.
This indicates considerable variation in text length.
The exploratory analysis identified the following findings:
The next stage will focus on developing a next-word prediction algorithm.
The planned approach includes:
An interactive Shiny application will be developed to demonstrate the prediction algorithm.
The application will allow users to:
The exploratory analysis successfully examined the three English text datasets.
Basic statistics, word frequency analysis and visualizations provided useful insights into the structure and characteristics of the data.
The next phase will focus on developing a next-word prediction algorithm and integrating it into an interactive Shiny application.