Introduction

The purpose of this project is to demonstrate successful loading and exploration of the HC Corpora dataset that will be used to build a next-word prediction algorithm and an interactive Shiny application.

The final application will predict the next word based on user-entered text using Natural Language Processing (NLP) techniques.

Data Loading

blogsFile <- "final/en_US/en_US.blogs.txt"
newsFile <- "final/en_US/en_US.news.txt"
twitterFile <- "final/en_US/en_US.twitter.txt"

blogs <- readLines(blogsFile,
                   encoding = "UTF-8",
                   skipNul = TRUE)

news <- readLines(newsFile,
                  encoding = "UTF-8",
                  skipNul = TRUE)

twitter <- readLines(twitterFile,
                     encoding = "UTF-8",
                     skipNul = TRUE)

Summary Statistics

summaryTable <- data.frame(
  Dataset = c("Blogs","News","Twitter"),
  FileSizeMB = round(c(
    file.info(blogsFile)$size,
    file.info(newsFile)$size,
    file.info(twitterFile)$size
  )/1024^2,2),
  Lines = c(
    length(blogs),
    length(news),
    length(twitter)
  ),
  Words = c(
    sum(stri_count_words(blogs)),
    sum(stri_count_words(news)),
    sum(stri_count_words(twitter))
  ),
  Characters = c(
    sum(nchar(blogs)),
    sum(nchar(news)),
    sum(nchar(twitter))
  )
)

kable(summaryTable,
      caption = "Summary Statistics of the HC Corpora Dataset")

Summary Statistics of the HC Corpora Dataset
Dataset	FileSizeMB	Lines	Words	Characters
Blogs	200.42	899288	37546806	206824505
News	196.28	1010206	34761151	203214543
Twitter	159.36	2360148	30096690	162096241

Sampling the Data

Only 1% of each dataset is sampled to reduce computation time.

set.seed(12345)

sampleBlogs <- sample(blogs,
                      size = length(blogs)*0.01)

sampleNews <- sample(news,
                     size = length(news)*0.01)

sampleTwitter <- sample(twitter,
                        size = length(twitter)*0.01)

sampleData <- c(sampleBlogs,
                sampleNews,
                sampleTwitter)

Text Cleaning

corpus <- VCorpus(VectorSource(sampleData))

corpus <- tm_map(corpus,
                 content_transformer(tolower))

corpus <- tm_map(corpus,
                 removeNumbers)

corpus <- tm_map(corpus,
                 removePunctuation)

corpus <- tm_map(corpus,
                 removeWords,
                 stopwords("english"))

corpus <- tm_map(corpus,
                 stripWhitespace)

Create Document Term Matrix

library(slam)

dtm <- DocumentTermMatrix(corpus)

wordFreq <- sort(col_sums(dtm), decreasing = TRUE)

freq <- data.frame(
  Word = names(wordFreq),
  Frequency = as.numeric(wordFreq)
)


head(freq)

##   Word Frequency
## 1 will      3180
## 2 just      3006
## 3 said      2966
## 4  one      2782
## 5 like      2690
## 6  can      2422

Top 20 Most Frequent Words

top20 <- freq %>%
  slice_max(Frequency,
            n = 20)

ggplot(top20,
       aes(reorder(Word,
                   Frequency),
           Frequency)) +
  geom_col(fill = "steelblue") +
  coord_flip() +
  labs(title = "Top 20 Most Frequent Words",
       x = "Word",
       y = "Frequency") +
  theme_minimal()

Word Cloud

wordcloud(
  words = freq$Word,
  freq = freq$Frequency,
  max.words = 100,
  random.order = FALSE,
  colors = brewer.pal(8,"Dark2")
)

Distribution of Line Lengths

lineLength <- nchar(sampleData)

ggplot(data.frame(lineLength),
       aes(lineLength)) +
  geom_histogram(fill = "orange",
                 color = "black",
                 bins = 40) +
  labs(title = "Distribution of Line Lengths",
       x = "Characters per Line",
       y = "Count") +
  theme_minimal()

Most Frequent Words

kable(head(freq,20),
      caption = "Top 20 Most Frequent Words")

Top 20 Most Frequent Words
Word	Frequency
will	3180
just	3006
said	2966
one	2782
like	2690
can	2422
get	2314
time	2133
new	1945
good	1779
now	1732
know	1651
day	1616
people	1611
love	1510
dont	1499
back	1498
first	1387
also	1373
see	1341

Exploratory Findings

The exploratory analysis revealed several important characteristics of the corpus.

The Twitter dataset contains the largest number of text records.
Blog entries contain substantially longer documents than Twitter messages.
The combined dataset contains more than 100 million words.
A 1% random sample provides sufficient data for exploratory analysis while reducing processing time.
Common English stop words dominate the corpus before preprocessing.
Word frequencies follow Zipf’s Law, where a small number of words occur very frequently while most words appear rarely.

Plans for the Prediction Algorithm

The prediction algorithm will be developed using N-gram language models.

The workflow will include:

Tokenize the cleaned text.
Build unigram, bigram, trigram and four-gram frequency tables.
Predict the next word using the highest-order matching N-gram.
Apply back-off methods when no exact match exists.
Evaluate prediction accuracy using validation data.

Plans for the Shiny App

The Shiny application will allow users to type a phrase and receive the predicted next word instantly.

The application will include:

Text input box.
Predicted next word.
Top alternative predictions.
Fast response using precomputed N-gram tables.
Simple interface suitable for non-technical users.

Conclusion

This exploratory analysis demonstrates that the HC Corpora dataset has been successfully loaded and summarized. The data are suitable for building an efficient language model for next-word prediction. The next stage of the project will involve constructing N-gram models, evaluating prediction accuracy, and deploying the final model as an interactive Shiny application.

Exploratory Analysis Using Corpora Data

FL

2026-08-07