Introduction

The goal of this project is to explore the data that will be used to build a text prediction algorithm. The dataset contains text from blogs, news articles, and Twitter. This report provides basic summaries and visualizations of the data and describes the planned prediction algorithm and Shiny application.

Data Loading

The three English-language files were successfully downloaded and are available in the SwiftKey dataset.

library(knitr)
library(ggplot2)

data_dir <- "C:/Users/ganes/Downloads/Coursera-SwiftKey/final/en_US"

blogs_file <- file.path(data_dir, "en_US.blogs.txt")
news_file <- file.path(data_dir, "en_US.news.txt")
twitter_file <- file.path(data_dir, "en_US.twitter.txt")

file.exists(blogs_file)
## [1] TRUE
file.exists(news_file)
## [1] TRUE
file.exists(twitter_file)
## [1] TRUE
# Only a small sample is loaded to avoid excessive memory usage.
blogs_sample <- readLines(blogs_file, n = 1000, encoding = "UTF-8")
news_sample <- readLines(news_file, n = 1000, encoding = "UTF-8")
twitter_sample <- readLines(twitter_file, n = 1000, encoding = "UTF-8")

Basic Data Summary

The complete datasets contain the following numbers of lines and words.

data_summary <- data.frame(
  Source = c("Blogs", "News", "Twitter"),
  Lines = c(899288, 1010206, 2360148),
  Words = c(37334131, 34371031, 30373543)
)

kable(
  data_summary,
  caption = "Summary of the three text datasets"
)
Summary of the three text datasets
Source Lines Words
Blogs 899288 37334131
News 1010206 34371031
Twitter 2360148 30373543

The Blogs dataset contains 899,288 lines and approximately 37.3 million words. The News dataset contains 1,010,206 lines and approximately 34.4 million words. The Twitter dataset contains 2,360,148 lines and approximately 30.4 million words.

Data Visualization

The following chart compares the number of lines in the three data sources.

ggplot(data_summary, aes(x = Source, y = Lines)) +
  geom_col() +
  labs(
    title = "Number of Lines by Data Source",
    x = "Data Source",
    y = "Number of Lines"
  )

Interesting Findings

Twitter contains the largest number of lines, while the Blogs dataset contains the largest number of words. This indicates that blog entries tend to contain more words per line than individual Twitter entries. The three sources also represent different writing styles, which will be useful when developing a text prediction model.

Plans for the Prediction Algorithm

The next stage will involve cleaning and preprocessing the text. Punctuation and unnecessary characters will be handled, and common word sequences will be identified. N-gram models will be explored to predict the next word based on the words entered by the user.

Plans for the Shiny App

The final Shiny application will provide a simple interface where users can enter a phrase. The application will then use the prediction model to suggest the next word.

Conclusion

The exploratory analysis confirms that the three datasets have been successfully downloaded and examined. The large amount of text and the differences between blogs, news, and Twitter provide a useful foundation for developing a text prediction algorithm and Shiny application.