1. Introduction

This milestone report presents an exploratory text analysis for the Johns Hopkins University Data Science Capstone. The overall objective of the capstone is to develop a next-word prediction system. A typical solution learns patterns from a large text corpus and uses word and n-gram frequencies to estimate likely next words.

To make this report completely reproducible and portable, the analysis below uses a small text corpus embedded directly in this R Markdown file. Therefore, the report does not require Java, RWeka, external datasets, internet downloads, or additional R packages.

2. Dataset

2.1 Description

The demonstration corpus contains short English-language sentences representing conversational, blog-like, and news-like writing. It is intentionally small so that the complete analysis can run on any standard R installation.

blogs <- c(
  "Data science is an exciting field with many practical applications.",
  "I love learning new skills and working on interesting projects.",
  "Machine learning can help people discover useful patterns in data.",
  "Today I worked on my data science capstone project.",
  "A good predictive model should be simple fast and useful.",
  "The project is a great opportunity to practice data analysis.",
  "Learning R has helped me understand statistics and visualization.",
  "Text mining is useful when working with large collections of documents."
)

news <- c(
  "The research team released a new data analysis report today.",
  "Technology companies continue to invest in artificial intelligence.",
  "Researchers are developing better tools for language prediction.",
  "The new software can process text data quickly and efficiently.",
  "Students are using data science methods to solve real world problems.",
  "The study shows that predictive models improve with better training data.",
  "New applications of machine learning are appearing across many industries.",
  "Scientists use statistical models to understand complex information."
)

twitter <- c(
  "I am learning data science today and it is going well",
  "This course is really interesting and I learned a lot",
  "Working on the capstone project tonight",
  "R makes data analysis fun and reproducible",
  "Just finished building a simple prediction model",
  "Text analysis is easier when the data is clean",
  "Machine learning is a useful skill to learn",
  "Great day for coding statistics and data science"
)

text_data <- data.frame(
  source = rep(c("Blogs", "News", "Twitter"), each = 8),
  text = c(blogs, news, twitter),
  stringsAsFactors = FALSE
)

head(text_data)
##   source                                                                text
## 1  Blogs Data science is an exciting field with many practical applications.
## 2  Blogs     I love learning new skills and working on interesting projects.
## 3  Blogs  Machine learning can help people discover useful patterns in data.
## 4  Blogs                 Today I worked on my data science capstone project.
## 5  Blogs           A good predictive model should be simple fast and useful.
## 6  Blogs       The project is a great opportunity to practice data analysis.

2.2 Dataset Summary

summary_table <- data.frame(
  Source = c("Blogs", "News", "Twitter", "Total"),
  Documents = c(length(blogs), length(news), length(twitter), nrow(text_data)),
  Characters = c(
    sum(nchar(blogs)),
    sum(nchar(news)),
    sum(nchar(twitter)),
    sum(nchar(text_data$text))
  ),
  stringsAsFactors = FALSE
)

knitr::kable(summary_table, caption = "Summary of the embedded text corpus")
Summary of the embedded text corpus
Source Documents Characters
Blogs 8 501
News 8 538
Twitter 8 372
Total 24 1411

3. Data Cleaning

The text is converted to lowercase, punctuation and numbers are removed, and repeated whitespace is collapsed. This produces a simple normalized corpus suitable for exploratory n-gram analysis.

clean_text <- function(x) {
  x <- tolower(x)
  x <- gsub("https?://[^ ]+", " ", x)
  x <- gsub("[^a-z[:space:]]", " ", x)
  x <- gsub("[[:space:]]+", " ", x)
  trimws(x)
}

text_data$clean_text <- clean_text(text_data$text)

knitr::kable(
  head(text_data[, c("source", "clean_text")], 6),
  caption = "Sample of cleaned text"
)
Sample of cleaned text
source clean_text
Blogs data science is an exciting field with many practical applications
Blogs i love learning new skills and working on interesting projects
Blogs machine learning can help people discover useful patterns in data
Blogs today i worked on my data science capstone project
Blogs a good predictive model should be simple fast and useful
Blogs the project is a great opportunity to practice data analysis

4. Exploratory Data Analysis

4.1 Tokenization

tokenize_words <- function(x) {
  words <- unlist(strsplit(x, " +"), use.names = FALSE)
  words[nzchar(words)]
}

all_words <- tokenize_words(paste(text_data$clean_text, collapse = " "))
word_freq <- sort(table(all_words), decreasing = TRUE)

cat("Total word instances:", length(all_words), "\n")
## Total word instances: 221
cat("Unique words:", length(word_freq), "\n")
## Unique words: 126

4.2 Most Frequent Words

top_words <- head(word_freq, 15)
unigram_df <- data.frame(
  Word = names(top_words),
  Frequency = as.integer(top_words),
  row.names = NULL
)
knitr::kable(unigram_df, caption = "15 most frequent words")
15 most frequent words
Word Frequency
data 12
and 8
is 8
a 6
learning 6
the 6
science 5
to 5
analysis 4
i 4
new 4
useful 4
are 3
machine 3
on 3 
barplot(
  rev(top_words),
  horiz = TRUE,
  las = 1,
  main = "Most Frequent Words",
  xlab = "Frequency"
)

4.3 N-Gram Analysis

An n-gram is a sequence of n consecutive words. Unigrams contain one word, bigrams contain two words, and trigrams contain three words. N-grams are important for next-word prediction because they capture local word-order patterns.

make_ngrams <- function(text_vector, n = 2) {
  result <- character(0)
  for (txt in text_vector) {
    w <- tokenize_words(txt)
    if (length(w) >= n) {
      starts <- seq_len(length(w) - n + 1)
      grams <- vapply(
        starts,
        function(i) paste(w[i:(i + n - 1)], collapse = " "),
        character(1)
      )
      result <- c(result, grams)
    }
  }
  result
}

bigrams <- make_ngrams(text_data$clean_text, 2)
trigrams <- make_ngrams(text_data$clean_text, 3)

bigram_freq <- sort(table(bigrams), decreasing = TRUE)
trigram_freq <- sort(table(trigrams), decreasing = TRUE)

Most Frequent Bigrams

top_bigrams <- head(bigram_freq, 10)
bigram_df <- data.frame(
  Bigram = names(top_bigrams),
  Frequency = as.integer(top_bigrams),
  row.names = NULL
)
knitr::kable(bigram_df, caption = "10 most frequent bigrams")
10 most frequent bigrams
Bigram Frequency
data science 5
data analysis 3
machine learning 3
capstone project 2
is a 2
statistics and 2
working on 2
a good 1
a great 1
a lot 1 
barplot(
  rev(top_bigrams),
  horiz = TRUE,
  las = 1,
  main = "Most Frequent Bigrams",
  xlab = "Frequency",
  cex.names = 0.8
)

Most Frequent Trigrams

top_trigrams <- head(trigram_freq, 10)
trigram_df <- data.frame(
  Trigram = names(top_trigrams),
  Frequency = as.integer(top_trigrams),
  row.names = NULL
)
knitr::kable(trigram_df, caption = "10 most frequent trigrams")
10 most frequent trigrams
Trigram Frequency
a good predictive 1
a great opportunity 1
a new data 1
a simple prediction 1
a useful skill 1
across many industries 1
am learning data 1
an exciting field 1
analysis fun and 1
analysis is easier 1 
barplot(
  rev(top_trigrams),
  horiz = TRUE,
  las = 1,
  main = "Most Frequent Trigrams",
  xlab = "Frequency",
  cex.names = 0.7
)

5. Vocabulary Coverage

One important question is how many unique words are needed to cover a given percentage of all word instances. A compact vocabulary that covers a large percentage of observed tokens is useful when designing an efficient prediction model.

coverage_size <- function(freq, target) {
  cumulative <- cumsum(as.numeric(freq)) / sum(freq)
  which(cumulative >= target)[1]
}

coverage_50 <- coverage_size(word_freq, 0.50)
coverage_90 <- coverage_size(word_freq, 0.90)

coverage_table <- data.frame(
  Coverage = c("50%", "90%"),
  Unique_Words_Required = c(coverage_50, coverage_90)
)

knitr::kable(coverage_table, caption = "Vocabulary required for corpus coverage")
Vocabulary required for corpus coverage
Coverage Unique_Words_Required
50% 28
90% 104

For this demonstration corpus, 28 unique words cover at least 50% of all word instances, while 104 unique words cover at least 90%.

cum_coverage <- cumsum(as.numeric(word_freq)) / sum(word_freq)
plot(
  seq_along(cum_coverage),
  cum_coverage * 100,
  type = "l",
  xlab = "Number of Unique Words",
  ylab = "Cumulative Coverage (%)",
  main = "Vocabulary Coverage"
)
abline(h = c(50, 90), lty = 2)

6. Simple Next-Word Prediction Demonstration

A basic prediction approach can use bigram counts. Given a word, the model finds bigrams beginning with that word and ranks the possible following words by frequency.

predict_next_word <- function(input_word, bigram_table, top_n = 5) {
  input_word <- clean_text(input_word)
  grams <- names(bigram_table)
  counts <- as.numeric(bigram_table)
  first_word <- sub(" .*", "", grams)
  matches <- which(first_word == input_word)

  if (length(matches) == 0) {
    return(data.frame(Prediction = character(0), Frequency = integer(0)))
  }

  candidates <- data.frame(
    Prediction = sub("^[^ ]+ ", "", grams[matches]),
    Frequency = counts[matches],
    stringsAsFactors = FALSE
  )
  candidates <- candidates[order(-candidates$Frequency), ]
  head(candidates, top_n)
}

prediction_example <- predict_next_word("data", bigram_freq)
knitr::kable(prediction_example, caption = "Example predictions after the word 'data'")
Example predictions after the word ‘data’
Prediction Frequency
science 5
analysis 3
is 1
quickly 1

This simple model demonstrates the central idea of the capstone project: use previously observed word sequences to estimate likely next words. A production model would use a much larger training corpus, smoothing, and a backoff strategy that combines higher- and lower-order n-grams.

7. Findings

The exploratory analysis shows several useful characteristics of text data:

  1. Word frequencies are uneven: a relatively small group of words accounts for a large fraction of the corpus.
  2. Bigrams and trigrams capture recurring phrases that single-word frequencies cannot represent.
  3. Vocabulary coverage can help determine how much of the dictionary should be retained for an efficient model.
  4. A frequency-based n-gram model provides a straightforward foundation for next-word prediction.

Because this report uses a compact embedded corpus, its numerical results are illustrative rather than representative of the full English language. The same workflow can be scaled to a larger corpus for the final capstone application.

8. Next Steps

The next stage of the project is to develop a more complete prediction algorithm. The planned model will use trigram and bigram matches when enough context is available and back off to unigram frequencies when no higher-order match is found. The final model can then be connected to a Shiny interface where a user enters a phrase and receives predicted next words.

9. Conclusion

This milestone report establishes the exploratory foundation for a next-word prediction application. The analysis includes text cleaning, tokenization, frequency analysis, n-gram analysis, vocabulary coverage, visualization, and a small next-word prediction demonstration. The results show how simple statistical properties of language can be transformed into features for a predictive text model.

Appendix: Reproducibility

This document intentionally uses only functions included with base R plus knitr, which is already used by R Markdown when knitting the report. It does not require RWeka, rJava, external text files, web downloads, or an external corpus.

sessionInfo()
## R version 4.5.2 (2025-10-31 ucrt)
## Platform: x86_64-w64-mingw32/x64
## Running under: Windows 11 x64 (build 26300)
## 
## Matrix products: default
##   LAPACK version 3.12.1
## 
## locale:
## [1] LC_COLLATE=English_India.utf8  LC_CTYPE=English_India.utf8   
## [3] LC_MONETARY=English_India.utf8 LC_NUMERIC=C                  
## [5] LC_TIME=English_India.utf8    
## 
## time zone: Asia/Calcutta
## tzcode source: internal
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## loaded via a namespace (and not attached):
##  [1] digest_0.6.39     R6_2.6.1          fastmap_1.2.0     xfun_0.56        
##  [5] cachem_1.1.0      knitr_1.51        htmltools_0.5.9   rmarkdown_2.32   
##  [9] lifecycle_1.0.5   cli_3.6.5         sass_0.4.10       jquerylib_0.1.4  
## [13] compiler_4.5.2    rstudioapi_0.18.0 tools_4.5.2       evaluate_1.0.5   
## [17] bslib_0.10.0      yaml_2.3.12       otel_0.2.0        jsonlite_2.0.0   
## [21] rlang_1.1.7