Exploratory Analysis of Text Data
Prashasna
1. Load the Data
blogs <- readLines("en_US.blogs.txt", encoding = "UTF-8")
news <- readLines("en_US.news.txt", encoding = "UTF-8")
twitter <- readLines("en_US.twitter.txt", encoding = "UTF-8")2. Basic Dataset Information
data_summary <- data.frame(
Dataset = c("Blogs", "News", "Twitter"),
Lines = c(
length(blogs),
length(news),
length(twitter)
),
Total_Characters = c(
sum(nchar(blogs)),
sum(nchar(news)),
sum(nchar(twitter))
)
)
data_summary## Dataset Lines Total_Characters
## 1 Blogs 899288 206824505
## 2 News 1010206 203214543
## 3 Twitter 2360148 1620960313. Number of Words
count_words <- function(x) {
sum(lengths(strsplit(x, "\\s+")))
}
word_summary <- data.frame(
Dataset = c("Blogs", "News", "Twitter"),
Words = c(
count_words(blogs),
count_words(news),
count_words(twitter)
)
)
word_summary## Dataset Words
## 1 Blogs 37334131
## 2 News 34371031
## 3 Twitter 303735434. Basic Statistics of Line Length
line_statistics <- data.frame(
Dataset = c("Blogs", "News", "Twitter"),
Mean_Line_Length = c(
mean(nchar(blogs)),
mean(nchar(news)),
mean(nchar(twitter))
),
Median_Line_Length = c(
median(nchar(blogs)),
median(nchar(news)),
median(nchar(twitter))
),
Maximum_Line_Length = c(
max(nchar(blogs)),
max(nchar(news)),
max(nchar(twitter))
)
)
line_statistics## Dataset Mean_Line_Length Median_Line_Length Maximum_Line_Length
## 1 Blogs 229.98695 156 40833
## 2 News 201.16149 185 11384
## 3 Twitter 68.68045 64 1405. Blog Line-Length Plot
blog_lengths <- nchar(blogs)
hist(
blog_lengths,
breaks = 50,
main = "Distribution of Blog Line Length",
xlab = "Number of Characters"
)
6. News Line-Length Plot
news_lengths <- nchar(news)
hist(
news_lengths,
breaks = 50,
main = "Distribution of News Line Length",
xlab = "Number of Characters"
)
7. Twitter Line-Length Plot
twitter_lengths <- nchar(twitter)
hist(
twitter_lengths,
breaks = 50,
main = "Distribution of Twitter Line Length",
xlab = "Number of Characters"
)
8. Sample of the Data
head(blogs, 3)## [1] "In the years thereafter, most of the Oil fields and platforms were named after pagan “gods”."
## [2] "We love you Mr. Brown."
## [3] "Chad has been awesome with the kids and holding down the fort while I work later than usual! The kids have been busy together playing Skylander on the XBox together, after Kyan cashed in his $$$ from his piggy bank. He wanted that game so bad and used his gift card from his birthday he has been saving and the money to get it (he never taps into that thing either, that is how we know he wanted it so bad). We made him count all of his money to make sure that he had enough! It was very cute to watch his reaction when he realized he did! He also does a very good job of letting Lola feel like she is playing too, by letting her switch out the characters! She loves it almost as much as him."head(news, 3)## [1] "He wasn't home alone, apparently."
## [2] "The St. Louis plant had to close. It would die of old age. Workers had been making cars there since the onset of mass automotive production in the 1920s."
## [3] "WSU's plans quickly became a hot topic on local online sites. Though most people applauded plans for the new biomedical center, many deplored the potential loss of the building."head(twitter, 3)## [1] "How are you? Btw thanks for the RT. You gonna be in DC anytime soon? Love to see you. Been way, way too long."
## [2] "When you meet someone special... you'll know. Your heart will beat more rapidly and you'll smile for no reason."
## [3] "they've decided its more fun if I don't."9. Initial Findings
The exploratory analysis shows that the three datasets have different characteristics. The Twitter dataset contains the largest number of lines, with 2,360,148 lines, followed by the News dataset with 1,010,206 lines and the Blogs dataset with 899,288 lines. The Blogs dataset has the highest average line length at approximately 230 characters, while News has an average of approximately 201 characters and Twitter has an average of approximately 69 characters. Twitter lines are therefore considerably shorter than the Blogs and News lines, which is consistent with the short-message nature of the Twitter data. The maximum line length also differs substantially across the datasets, with Blogs having the largest maximum line length.
10. Interesting Findings
One important finding is the difference in the size and structure of the three datasets. Although Twitter has the largest number of lines, its total character count is lower than the Blogs dataset. The word counts are also substantial across all three datasets, providing a large amount of text for developing a prediction model. The line-length distributions show that Twitter has a much narrower range of line lengths than Blogs and News. These differences will be considered when preparing the text data for the prediction algorithm.
11. Plan for the Prediction Algorithm
The next stage of the project will focus on identifying frequently occurring words and combinations of words in the Blogs, News, and Twitter datasets. The text will be prepared so that common patterns can be identified and used to predict a likely next word. Different word combinations will be explored to determine which patterns are useful for prediction. The prediction approach will be evaluated based on how well it can identify an appropriate next word. The goal is to develop a prediction method that is useful while also being efficient enough for an interactive application.
12. Plan for the Shiny Application
The final Shiny application will allow a user to enter a short phrase and receive a predicted next word. The application will use the prediction model developed from the text datasets to generate the suggested word. The interface will be kept simple so that users without a data science background can easily understand and use it. The application will provide an interactive way to demonstrate the results of the text prediction analysis.
13. Conclusion
This exploratory analysis successfully loaded and examined the Blogs, News, and Twitter datasets. Basic line counts, word counts, character counts, line-length statistics, and visualizations were used to understand the major characteristics of the data. The analysis shows clear differences between the three sources, particularly in the number of lines and typical line lengths. These findings provide a foundation for the next stage of the project, which will focus on developing a prediction algorithm and Shiny application.