Exploratory Analysis

Introduction

The objective of this report is to perform an exploratory analysis of the English text datasets provided for the Capstone Project. The datasets consist of texts collected from blogs, news articles, and Twitter posts. The purpose of this analysis is to obtain an understanding of the data and form a plan to develop a predictive text model and an interactive Shiny application.

The Shiny application will allow users to enter a phrase and receive predictions for the most likely word that follows. When a user submits a phrase, the app will analyze the last few words entered and return the most probable next word, with some additional suggestions.

Loading Data

library(stringi)
library(dplyr)

## Warning: package 'dplyr' was built under R version 4.5.3

## 
## Attaching package: 'dplyr'

## The following objects are masked from 'package:stats':
## 
##     filter, lag

## The following objects are masked from 'package:base':
## 
##     intersect, setdiff, setequal, union

library(ggplot2)

## Warning: package 'ggplot2' was built under R version 4.5.3

blogs <- readLines("C:\\Coursera-SwiftKey\\final\\en_US\\en_US.blogs.txt",encoding = "UTF-8", skipNul = TRUE)

news <- readLines("C:\\Coursera-SwiftKey\\final\\en_US\\en_US.news.txt",
                  encoding = "UTF-8", skipNul = TRUE)

twitter <- readLines("C:\\Coursera-SwiftKey\\final\\en_US\\en_US.twitter.txt",
                     encoding = "UTF-8", skipNul = TRUE)

Summary Statistics

The table below summarizes the basic characteristics of the datasets.

summary_table <- data.frame(
  Dataset = c("Blogs", "News", "Twitter"),
  Lines = c(length(blogs),
            length(news),
            length(twitter)),
  Words = c(sum(stri_count_words(blogs)),
            sum(stri_count_words(news)),
            sum(stri_count_words(twitter))),
  Characters = c(sum(nchar(blogs)),
                 sum(nchar(news)),
                 sum(nchar(twitter)))
)

summary_table

##   Dataset   Lines    Words Characters
## 1   Blogs  899288 37546806  206824505
## 2    News 1010206 34761151  203214543
## 3 Twitter 2360148 30096690  162096241

Exploratory Analysis

The three datasets differ in size and composition. The Twitter dataset has the largest number of lines, followed by the news and the blogs datasets.

#Plot

avg_words <- data.frame(
  Dataset = c("Blogs", "News", "Twitter"),
  AvgWords = c(
    mean(stri_count_words(blogs)),
    mean(stri_count_words(news)),
    mean(stri_count_words(twitter))
  )
)
ggplot(avg_words, aes(x = Dataset, y = AvgWords)) +
  geom_col() +
  labs(
    title = "Average Number of Words per Line",
    x = "Dataset",
    y = "Average Words"
  )

Observations:

• The datasets vary in average line length.
• Blogs contain longer and more conversational text than Twitter posts.
• Twitter texts contains many short messages, abbreviations, and informal language.
• Common words occur very frequently.

Planning ahead

The next stage of the project will involve cleaning and preprocessing the text data, including removing unnecessary symbols and preparing the text for analysis.

The completed model will be incorporated into a Shiny application that allows users to enter a phrase and receive predictions for the most likely next word.

Conclusion

This exploratory analysis provides an overview of the training datasets and highlights several characteristics that will influence the design of the predictive text model. The results presented here will serve as the foundation for developing the prediction algorithm and the accompanying Shiny application.