#Introduction The purpose of this milestone report is to explore the text data and understand its main characteristics before developing the final prediction algorithm and shiny application. The project uses three english text files: blogs, news, and Twitter data. The analysis focuses on the size of the datasets, the number of lines and words, common words, and the distribution of next length. #Data Download The data was downloaded from the coursera Swiftkey dataset and contains three text files.

options(stringsAsFactors = FALSE)
options(stringsAsFactors = FALSE)
options(timeout = 3600)
data_url <-"https://d396qusza40orc.cloudfront.net/dsscapstone/dataset/Coursera-SwiftKey.zip"
zip_file <- "C:/Users/Aashi/OneDerive/Documents/Coursera-SwiftKey.zip"

 download.file(
   data_url,
   zip_file,
   mode = "wb",
   method = "libcurl"
 )
 unzip(zip_file)

#Loading the Data The three English files were loaded for exploratory analysis.

blogs_file <- "final/en_US/en_US.blogs.txt"
news_file <- "final/en_US/en_US.news.txt"
twitter_file <- "final/en_US/en_US.twitter.txt"
blogs <- readLines(blogs_file, encoding = "UTF - 8", warn = FALSE)
news <- readLines(news_file, encoding = "UTF - 8", warn = FALSE)
twitter <- readLines(twitter_file, encoding = "UTF - 8", warn = FALSE)

#Basic summary of the data The following table shows the number of lines and words in each dataset.

count_words <- function(x) {
  sum(lengths(strsplit(x, "\\s+")))
}
summary_table <- data.frame(Dataset = c("Blogs", "News", "Twitter"),Lines = c(length(blogs),length(news), length(twitter)),Words = c(count_words(blogs), count_words(news), count_words(twitter)
                                                                                                                                     )
                            )
summary.table
## function (object, ...) 
## {
##     if (!inherits(object, "table")) 
##         stop(gettextf("'object' must inherit from class %s", 
##             dQuote("table")), domain = NA)
##     n.cases <- sum(object)
##     n.vars <- length(dim(object))
##     y <- list(n.vars = n.vars, n.cases = n.cases)
##     if (n.vars > 1) {
##         m <- vector("list", length = n.vars)
##         relFreqs <- object/n.cases
##         for (k in 1L:n.vars) m[[k]] <- apply(relFreqs, k, sum)
##         expected <- apply(do.call("expand.grid", m), 1L, prod) * 
##             n.cases
##         statistic <- sum((c(object) - expected)^2/expected)
##         lm <- lengths(m)
##         parameter <- prod(lm) - 1L - sum(lm - 1L)
##         y <- c(y, list(statistic = statistic, parameter = parameter, 
##             approx.ok = all(expected >= 5), p.value = stats::pchisq(statistic, 
##                 parameter, lower.tail = FALSE), call = attr(object, 
##                 "call")))
##     }
##     class(y) <- "summary.table"
##     y
## }
## <bytecode: 0x0000023c4f64feb0>
## <environment: namespace:base>

#Exploratory Analysis The three datasets have different sizes and characteristics. Blogs and news contain longer piece of text, while Twitter messages are generally shorter because of the nature of the platform. ##Distribution of text Length To examine the distribution of text length, a sample of lines was selecteed from each dataset.

set.seed(123)
blog_sample <- sample(blogs, min(5000, length(blogs)))
blog_words <- sapply(strsplit(blog_sample, "\\s+"), length)
hist(
  blog_words,
  breaks = 30,
  main = "Distribution of Words per Blog Line",
  xlab = "Number of Words",
  ylab = "Frequency"
)

##Most Frequent Words A sample of the data was used to examine commonly occuring words.

sample_text <- sample(c(blogs, news, twitter), min(20000, length(c(blogs, news, twitter))))
words <- unlist(strsplit(tolower(sample_text), "\\w+"))
words <- words[nchar(words) > 2]
word_table <- sort(table(words), decreasing = TRUE)
head(word_table, 20)
## words
##  ,"   ...   -    &   . " ...    –   --    :   ),    —   , "   :)  ,”   !!   ).  
##  625  623  561  483  378  291  244  240  224  202  199  160  149  149  147  147 
##  ."  ....  !!!  .,  
##  142  137  135  127

##Bar Plot of frequency Words

top_words <- head(word_table, 10)
barplot(
  top_words,
  main = "Most frequent Words",
  xlab = "Words",
  ylab = "Frequency",
  las = 2
)

#Interesting Findings The exploratory analysis shows that the three datasets have different characteristics. Blogs and news contain longer pieces of text, while Twitter contains shorter messages. Some words occur much more frequently than others. These common words and word cobinations may be useful for predicting the next word. The analysis also shows that the length of next varies considerably. This will be considered when designing the prediction algorithm. #Plan For The Predictor Algorithm The next stage of the project will focus on developing a predicion algorithm. The algorithm will use word combinations observed in the training data to predict the next word entered by a user. N-gram models such as bigrams and trigrams will be considered. The model will give higher priority to word combinations that occur frequently in the training data. The algorithm will also need to handle words or combinations that were not observed previously. #Plan for the Shiny Application The final Shiny application will provide a simple interface where users can enter a phrase. The application will then suggest possible next words. The main goal is to make the Application simple, fast, and easy to understand for users without a data science background. #Conclusion The exploratory analysis successfully examined the three text datasets and identified their basic characteristics. The summary statistics and plots provide a useful starting point for developing the prediction algorithm. The next steps are to improve the prediction model, evaluate its performance, and integrate it into a shiny application.