R Markdown

This is an R Markdown document. Markdown is a simple formatting syntax for authoring HTML, PDF, and MS Word documents. For more details on using R Markdown see http://rmarkdown.rstudio.com.

When you click the Knit button a document will be generated that includes both content as well as the output of any embedded R code chunks within the document. You can embed an R code chunk like this:

setwd("C:/Users/sumim/Desktop/Coursera datascience/Capstone/Capstone dataset/en_US")

blogs <- readLines("en_US.blogs.txt", encoding = "UTF-8")
news <- readLines("en_US.news.txt", encoding = "UTF-8")
twitter <- readLines("en_US.twitter.txt", encoding = "UTF-8")
## Warning in readLines("en_US.twitter.txt", encoding = "UTF-8"): line 167155
## appears to contain an embedded nul
## Warning in readLines("en_US.twitter.txt", encoding = "UTF-8"): line 268547
## appears to contain an embedded nul
## Warning in readLines("en_US.twitter.txt", encoding = "UTF-8"): line 1274086
## appears to contain an embedded nul
## Warning in readLines("en_US.twitter.txt", encoding = "UTF-8"): line 1759032
## appears to contain an embedded nul
data_summary <- data.frame(
  Dataset = c("Blogs", "News", "Twitter"),
  Size_MB = c(
    file.info("en_US.blogs.txt")$size,
    file.info("en_US.news.txt")$size,
    file.info("en_US.twitter.txt")$size
  ) / 1024^2,
  Lines = c(length(blogs), length(news), length(twitter)),
  Tokens = c(
    sum(sapply(strsplit(blogs, "\\s+"), length)),
    sum(sapply(strsplit(news, "\\s+"), length)),
    sum(sapply(strsplit(twitter, "\\s+"), length))
  ),
  Longest_Line = c(
    max(nchar(blogs)),
    max(nchar(news)),
    max(nchar(twitter))
  )
)

data_summary
##   Dataset  Size_MB   Lines   Tokens Longest_Line
## 1   Blogs 200.4242  899288 37334131        40833
## 2    News 196.2775 1010206 34371031        11384
## 3 Twitter 159.3641 2360148 30373543          140
library(ggplot2)

ggplot(data_summary, aes(x = Dataset, y = Tokens)) +
  geom_col() +
  labs(
    title = " Figure 1. Number of Tokens by Data Source",
    x = "Data Source",
    y = "Number of Tokens"
  )

blog_length <- nchar(blogs)
news_length <- nchar(news)
twitter_length <- nchar(twitter)

line_lengths <- data.frame(
  Length = c(blog_length, news_length, twitter_length),
  Dataset = c(
    rep("Blogs", length(blogs)),
    rep("News", length(news)),
    rep("Twitter", length(twitter))
  )
)

ggplot(line_lengths, aes(x = Length)) +
  geom_histogram(bins = 50) +
  facet_wrap(~ Dataset, scales = "fixed") +
  scale_x_continuous(
    limits = c(0, 1000),
    breaks = seq(0, 1000, 200)
  ) +
  scale_y_continuous(
    limits = c(0, 500000),
    breaks = seq(0, 500000, 100000)
  ) +
  labs(
    title = "Figure 2. Distribution of Line Lengths",
    x = "Characters per Line",
    y = "Frequency"
  )
## Warning: Removed 14419 rows containing non-finite outside the scale range
## (`stat_bin()`).
## Warning: Removed 6 rows containing missing values or values outside the scale range
## (`geom_bar()`).

1. Project Overview

The goal of this project is to develop a simple application that predicts the next word a user is likely to type based on a phrase they have entered. The initial exploratory analysis focuses on understanding the available text data and identifying patterns that can be used to develop the prediction algorithm.

2. Data Acquisition and Loading

Three English-language datasets were downloaded and loaded into R: Blogs, News, Twitter The datasets contain a large amount of text, and the three sources have different writing styles: blogs contain longer text and Twitter contains shorter text.

3. Exploratory Findings

Figure 1 provides a simple comparison of the amount of text available from each source. Figure 2 provides a distribution of line lengths highlighting the different characteristics of the three sources.

4. Plan for the Prediction Algorithm

The next stage will focus on identifying common words and word combinations in the datasets. The planned approach is to: Clean and prepare the text. Identify frequently occurring individual words. Identify frequently occurring two- and three-word combinations. Estimate the most likely next word using these patterns. Evaluate the prediction performance using data that were not used to build the model.

5. Plan for the Shiny Application

The final Shiny application will provide a simple interface for users. A user will enter a phrase, and the application will analyze the words already entered and suggest one or more likely next words. For example: User: “I would like to”, Application: “know”, “see”, “have”

6. Next Steps and Feedback

The exploratory analysis confirms that the available data are suitable for developing a next-word prediction system. The next major step is to analyze word and word-combination frequencies and use these patterns to develop and evaluate the prediction algorithm. Feedback would be useful regarding: whether the proposed prediction approach is appropriate; how many word suggestions should be displayed; how prediction accuracy should be evaluated;