Introduction

This report presents an exploratory analysis of the English language corpus supplied for the Coursera Data Science Capstone. The objective is to understand the characteristics of the Blogs, News and Twitter datasets before developing a next-word prediction model and a Shiny application.

Summary Statistics

summary_tbl <- data.frame(
  Dataset=names(files),
  File_Size_MB=sapply(file.path(data_path,files),file_size_mb),
  Lines=sapply(file.path(data_path,files),line_count),
  Words=sapply(file.path(data_path,files),word_count),
  Longest_Line=sapply(file.path(data_path,files),longest_line)
)

kable(summary_tbl)

	Dataset	File_Size_MB	Lines	Words	Longest_Line
D:/New Folder/Trainings/Data Science John Hopkins/Final Project/Coursera-SwiftKey/final/en_US/en_US.blogs.txt	Blogs	200.42	899288	37546806	40833
D:/New Folder/Trainings/Data Science John Hopkins/Final Project/Coursera-SwiftKey/final/en_US/en_US.news.txt	News	196.28	1010206	34761151	11384
D:/New Folder/Trainings/Data Science John Hopkins/Final Project/Coursera-SwiftKey/final/en_US/en_US.twitter.txt	Twitter	159.36	2360148	30096649	140

Plot 1: Number of Lines

ggplot(summary_tbl,aes(Dataset,Lines))+
  geom_col()+
  labs(title="Lines in Each Dataset")

Plot 2: Words per Line (Sample)

set.seed(123)

blogs <- readLines(file.path(data_path,files["Blogs"]),warn=FALSE)
blogs <- sample(blogs,min(5000,length(blogs)))
wc <- stri_count_words(blogs)

ggplot(data.frame(Words=wc),aes(Words))+
  geom_histogram(bins=30)+
  labs(title="Distribution of Words per Line (Blogs Sample)")

Plot 3: Most Frequent Words

sample_text <- c(
  sample(readLines(file.path(data_path,files["Blogs"]),warn=FALSE),3000),
  sample(readLines(file.path(data_path,files["News"]),warn=FALSE),3000),
  sample(readLines(file.path(data_path,files["Twitter"]),warn=FALSE),3000)
)

sample_text <- tolower(sample_text)
sample_text <- gsub("[^a-z ]"," ",sample_text)

words <- unlist(strsplit(sample_text,"\\s+"))
words <- words[words!=""]

freq <- sort(table(words),decreasing=TRUE)
top20 <- head(freq,20)

df <- data.frame(
  Word=factor(names(top20),levels=rev(names(top20))),
  Frequency=as.numeric(top20)
)

ggplot(df,aes(Word,Frequency))+
  geom_col()+
  coord_flip()+
  labs(title="Top 20 Most Frequent Words")

Findings

Twitter contains many short messages.
Blogs generally contain the longest text entries.
News articles have a more consistent writing style.
A relatively small number of words dominate the corpus.

Future Work

The next stage will clean the text further, generate unigram, bigram and trigram models, estimate next-word probabilities, and deploy the final prediction model through a Shiny application.

Conclusion

This exploratory analysis confirms that the datasets have been successfully loaded and summarized. The findings provide a foundation for building an efficient predictive text model.

SwiftKey Capstone Milestone Report

Umesh Rijal

2026-08-12