Milestone Report - Data Science Capstone

Milestone Report

Introduction

The objective of this report is to perform an exploratory analysis of the three English text datasets provided for the Data Science Capstone project: blogs, news and Twitter.

The final goal of the project is to build a predictive text algorithm capable of suggesting the next word in a sentence and to implement it in a Shiny application.

library(stringi)
library(ggplot2)
library(dplyr)

Loading the data

The three English datasets are loaded from the downloaded corpus.

blogs_file <- "C:/Users/jjesu/OneDrive/Documentos/final/en_US/en_US.blogs.txt"
news_file <- "C:/Users/jjesu/OneDrive/Documentos/final/en_US/en_US.news.txt"
twitter_file <- "C:/Users/jjesu/OneDrive/Documentos/final/en_US/en_US.twitter.txt"

blogs <- readLines(blogs_file, encoding = "UTF-8", skipNul = TRUE)
news <- readLines(news_file, encoding = "UTF-8", skipNul = TRUE)

## Warning in readLines(news_file, encoding = "UTF-8", skipNul = TRUE): incomplete
## final line found on
## 'C:/Users/jjesu/OneDrive/Documentos/final/en_US/en_US.news.txt'

twitter <- readLines(twitter_file, encoding = "UTF-8", skipNul = TRUE)

Basic summary

First, the size, number of lines and approximate number of words in each dataset are calculated.

file_sizes <- c(
  file.info(blogs_file)$size / 1024^2,
  file.info(news_file)$size / 1024^2,
  file.info(twitter_file)$size / 1024^2
)

line_counts <- c(
  length(blogs),
  length(news),
  length(twitter)
)

word_counts <- c(
  sum(stri_count_words(blogs)),
  sum(stri_count_words(news)),
  sum(stri_count_words(twitter))
)

summary_data <- data.frame(
  Dataset = c("Blogs", "News", "Twitter"),
  Size_MB = round(file_sizes, 2),
  Lines = line_counts,
  Words = word_counts
)

summary_data

##   Dataset Size_MB   Lines    Words
## 1   Blogs  200.42  899288 37546806
## 2    News  196.28   77259  2674561
## 3 Twitter  159.36 2360148 30096690

The datasets contain a large amount of text from different sources. Twitter contains a particularly large number of individual text entries, while blog entries tend to be considerably longer.

Text length

The length of each line was calculated in characters to better understand the structure of the three datasets.

blog_length <- nchar(blogs)
news_length <- nchar(news)
twitter_length <- nchar(twitter)

length_summary <- data.frame(
  Dataset = c("Blogs", "News", "Twitter"),
  Mean = round(c(
    mean(blog_length),
    mean(news_length),
    mean(twitter_length)
  ), 2),
  Median = c(
    median(blog_length),
    median(news_length),
    median(twitter_length)
  ),
  Maximum = c(
    max(blog_length),
    max(news_length),
    max(twitter_length)
  )
)

length_summary

##   Dataset   Mean Median Maximum
## 1   Blogs 229.99    156   40833
## 2    News 202.43    186    5760
## 3 Twitter  68.68     64     140

This analysis shows clear differences between the sources. Tweets are generally short because of the nature of the platform, while blog and news entries can contain much longer pieces of text.

Distribution of text length

Because the complete datasets are very large, a random sample of 10,000 observations from each dataset is used for visualization.

set.seed(123)

blogs_sample <- sample(blogs, min(10000, length(blogs)))
news_sample <- sample(news, min(10000, length(news)))
twitter_sample <- sample(twitter, min(10000, length(twitter)))

sample_data <- data.frame(
  text = c(
    blogs_sample,
    news_sample,
    twitter_sample
  ),
  Dataset = c(
    rep("Blogs", length(blogs_sample)),
    rep("News", length(news_sample)),
    rep("Twitter", length(twitter_sample))
  )
)

sample_data$characters <- nchar(sample_data$text)

ggplot(sample_data, aes(x = characters)) +
  geom_histogram(bins = 50) +
  facet_wrap(~Dataset, scales = "free_y") +
  coord_cartesian(xlim = c(0,1000)) +
  labs(
    title = "Distribution of text length",
    x = "Number of characters",
    y = "Frequency"
  ) +
  theme_minimal()

The histogram illustrates that Twitter messages are concentrated at shorter text lengths, whereas blogs and news contain a wider distribution of text lengths.

Word frequency

A basic word-frequency analysis was also performed using the random sample.

all_sample <- paste(
  blogs_sample,
  news_sample,
  twitter_sample,
  collapse = " "
)

words <- unlist(
  strsplit(
    tolower(all_sample),
    "\\s+"
  )
)

words <- gsub(
  "[^a-z']",
  "",
  words
)

words <- words[words != ""]

word_table <- sort(
  table(words),
  decreasing = TRUE
)

head(word_table, 20)

## words
##   the    to   and     a    of    in     i  that   for    is    it    on  with 
## 43987 24248 23089 21164 18750 14806 13256  9521  9233  8996  7777  6896  6531 
##   you   was    at  this    be    my    as 
##  6423  5870  4881  4732  4717  4644  4610

The most frequent words can also be represented graphically.

top_words <- data.frame(
  word = names(word_table[1:20]),
  frequency = as.numeric(word_table[1:20])
)

ggplot(
  top_words,
  aes(
    x = reorder(word, frequency),
    y = frequency
  )
) +
  geom_col() +
  coord_flip() +
  labs(
    title = "20 most frequent words",
    x = "Word",
    y = "Frequency"
  ) +
  theme_minimal()

Main findings

The exploratory analysis reveals several important characteristics of the corpus.

The Twitter dataset contains a very large number of short text entries. In contrast, blogs contain longer pieces of text and show much greater variation in text length. News articles also contain relatively long and structured text.

The frequency analysis shows that common English words dominate the corpus. Before developing the prediction model, additional text cleaning will therefore be required.

Another important aspect is the large size of the datasets. Processing the complete corpus for every operation would require considerable memory and computational resources. Sampling and efficient text-processing techniques will therefore be useful during model development.

Prediction strategy

The next step of the project will consist of preparing the text for the predictive algorithm.

The text will first be converted to lowercase and unnecessary punctuation, numbers and special characters will be removed. The cleaned text will then be tokenized into individual words. After preprocessing, sequences of words called n-grams will be created.

For example:

• unigram: “data”
• bigram: “data science”
• trigram: “data science project”

The frequencies of these sequences will be calculated from the corpus. The prediction algorithm will use these frequencies to estimate the most likely word following a sequence entered by the user. For example, if the user enters:

“thank you for”

the model will search the n-gram frequency tables and return the most probable next word. Higher-order n-grams will be used whenever enough information is available. If a sequence cannot be found, the algorithm will fall back to shorter n-grams. This approach should provide a balance between prediction accuracy, computational efficiency and application speed.

Shiny application

The final prediction algorithm will be implemented in an interactive Shiny application.

The user will enter a sentence or group of words into a text box. The application will process the input and use the n-gram prediction model to determine the most likely next word. The predicted word, or potentially several candidate words, will then be displayed to the user.

The application will therefore provide a simple demonstration of predictive text technology similar to the word prediction systems used in mobile keyboards.

Conclusion

This exploratory analysis provides an overview of the three English text datasets used in the Data Science Capstone project.

The datasets contain millions of words from blogs, news articles and Twitter messages and represent different forms of written communication.

The analysis of file sizes, number of lines, number of words and text lengths demonstrates important differences between the three sources.

These results provide the foundation for the next stage of the project: cleaning the text, constructing n-gram frequency models and developing a predictive text algorithm that will ultimately be deployed through a Shiny application.