1. Introduction

The objective of this project is to explore English text datasets provided for the SwiftKey Data Science Capstone project.

The analysis focuses on three datasets: blogs, news and Twitter. The ultimate goal is to develop a next-word prediction algorithm and an interactive Shiny application.

2. Data Description

The project uses three English text datasets:

These datasets provide a large collection of real-world English text for understanding language patterns.

3. Basic Data Summary

The following table presents the basic statistics calculated from the three datasets.

Dataset Lines Words Size (MB)
Blogs 899,288 37,334,131 200.42
News 1,010,206 34,371,031 196.28
Twitter 2,360,148 30,373,543 159.36
Total 4,269,642 102,078,705 556.06

Twitter contains the highest number of lines, while blogs contain the highest number of words.

Together, the three datasets contain more than 102 million words.

4. Exploratory Data Analysis

A random sample of 10,000 lines was selected from each dataset, giving a combined sample of 30,000 lines.

The text was converted to lowercase, and punctuation and special characters were removed. Common English stop words were removed to identify meaningful words.

4.1 Most Frequent Words

The following table shows the 20 most frequent meaningful words in the sample.

Top 20 Most Frequent Words
Word Frequency
said 3048
will 2767
one 2680
can 2551
just 2328
like 2089
time 1882
get 1717
new 1628
now 1440
people 1389
day 1322
also 1319
good 1311
year 1298
first 1294
know 1232
back 1226
don 1203
two 1173

4.2 Word Frequency Chart

The analysis identifies words such as “said”, “will”, “one”, “can” and “just” among the most frequently used words.

4.3 Text Length Analysis

The following table summarizes the number of words per text line.

Words per Text Line
Statistic Words
Minimum 1.00
First Quartile 9.00
Median 20.00
Mean 29.45
Third Quartile 39.00
Maximum 808.00

4.4 Text Length Distribution

The average number of words per line is approximately 29.45, while the median is 20. The maximum observed line contains 808 words.

This indicates considerable variation in text length.

5. Key Findings

The exploratory analysis identified the following findings:

  1. The three datasets contain more than 102 million words.
  2. Twitter has the highest number of text lines.
  3. Blogs contain the highest total number of words.
  4. Word frequency analysis helps identify commonly used words.
  5. Text lines vary considerably in length.

6. Future Prediction Algorithm

The next stage will focus on developing a next-word prediction algorithm.

The planned approach includes:

7. Shiny Application Plan

An interactive Shiny application will be developed to demonstrate the prediction algorithm.

The application will allow users to:

8. Conclusion

The exploratory analysis successfully examined the three English text datasets.

Basic statistics, word frequency analysis and visualizations provided useful insights into the structure and characteristics of the data.

The next phase will focus on developing a next-word prediction algorithm and integrating it into an interactive Shiny application.