The objective of this project is to explore and analyse the SwiftKey text datasets as part of the Data Science Capstone.
The project uses three English-language datasets:
The purpose of this milestone is to understand the datasets, examine their structure, calculate basic statistics, explore word frequencies, and prepare for the development of a predictive text application.
Load the required datasets.
files <- c(
Blogs = "../data/en_US.blogs.txt",
News = "../data/en_US.news.txt",
Twitter = "../data/en_US.twitter.txt"
)
file.exists(files)
## [1] TRUE TRUE TRUE
Check that all three datasets are available.
if (!all(file.exists(files))) {
stop("One or more SwiftKey dataset files could not be found. Check the data folder and file paths.")
}
cat("All three SwiftKey datasets were successfully located.")
## All three SwiftKey datasets were successfully located.
The project uses the following datasets:
| Dataset | Description |
|---|---|
| Blogs | Text collected from online blogs |
| News | Text collected from news articles |
| Short messages collected from Twitter |
The datasets contain different writing styles and sentence structures. They will be used to understand language patterns and develop a predictive text model.
The following function analyses each dataset without loading the entire file into memory.
It calculates:
count_words <- function(text) {
matches <- gregexpr(
"[[:alpha:]]+(?:'[[:alpha:]]+)?",
text,
perl = TRUE
)
lengths(regmatches(text, matches))
}
analyse_file <- function(path, sample_limit = 20000) {
con <- file(
path,
open = "r",
encoding = "UTF-8"
)
on.exit(close(con))
total_lines <- 0
total_words <- 0
sample_lines <- character()
repeat {
lines <- readLines(
con,
n = 5000,
warn = FALSE
)
if (length(lines) == 0) {
break
}
total_lines <- total_lines + length(lines)
word_counts <- count_words(lines)
total_words <- total_words + sum(word_counts)
remaining <- sample_limit - length(sample_lines)
if (remaining > 0) {
selected <- head(lines, remaining)
sample_lines <- c(
sample_lines,
selected
)
}
}
sample_word_counts <- count_words(sample_lines)
tokens <- unlist(
regmatches(
sample_lines,
gregexpr(
"[[:alpha:]]+(?:'[[:alpha:]]+)?",
sample_lines,
perl = TRUE
)
),
use.names = FALSE
)
tokens <- tolower(tokens)
word_frequency <- sort(
table(tokens),
decreasing = TRUE
)
list(
lines = total_lines,
words = total_words,
size_MB = round(
file.info(path)$size / 1024^2,
2
),
sample_lines = sample_lines,
sample_word_counts = sample_word_counts,
word_frequency = word_frequency
)
}
The analysis function is applied to all three datasets.
results <- lapply(
files,
analyse_file
)
Create a summary table containing the size, total lines, total words and sample size for each dataset.
summary_table <- data.frame(
Dataset = names(files),
Size_MB = sapply(
results,
function(x) x$size_MB
),
Total_Lines = sapply(
results,
function(x) x$lines
),
Total_Words = sapply(
results,
function(x) x$words
),
Sample_Lines = sapply(
results,
function(x) length(x$sample_lines)
)
)
knitr::kable(
summary_table,
caption = "Summary Statistics of SwiftKey Training Datasets"
)
| Dataset | Size_MB | Total_Lines | Total_Words | Sample_Lines | |
|---|---|---|---|---|---|
| Blogs | Blogs | 200.42 | 899288 | 37533649 | 20000 |
| News | News | 196.28 | 1010206 | 34059076 | 20000 |
| 159.36 | 2360148 | 29749933 | 20000 |
Display a few sample lines from each dataset.
for (dataset in names(results)) {
cat("\n\nDATASET:", dataset, "\n\n")
print(
head(
results[[dataset]]$sample_lines,
5
)
)
}
##
##
## DATASET: Blogs
##
## [1] "In the years thereafter, most of the Oil fields and platforms were named after pagan “gods”."
## [2] "We love you Mr. Brown."
## [3] "Chad has been awesome with the kids and holding down the fort while I work later than usual! The kids have been busy together playing Skylander on the XBox together, after Kyan cashed in his $$$ from his piggy bank. He wanted that game so bad and used his gift card from his birthday he has been saving and the money to get it (he never taps into that thing either, that is how we know he wanted it so bad). We made him count all of his money to make sure that he had enough! It was very cute to watch his reaction when he realized he did! He also does a very good job of letting Lola feel like she is playing too, by letting her switch out the characters! She loves it almost as much as him."
## [4] "so anyways, i am going to share some home decor inspiration that i have been storing in my folder on the puter. i have all these amazing images stored away ready to come to life when we get our home."
## [5] "With graduation season right around the corner, Nancy has whipped up a fun set to help you out with not only your graduation cards and gifts, but any occasion that brings on a change in one's life. I stamped the images in Memento Tuxedo Black and cut them out with circle Nestabilities. I embossed the kraft and red cardstock with TE's new Stars Impressions Plate, which is double sided and gives you 2 fantastic patterns. You can see how to use the Impressions Plates in this tutorial Taylor created. Just one pass through your die cut machine using the Embossing Pad Kit is all you need to do - super easy!"
##
##
## DATASET: News
##
## [1] "He wasn't home alone, apparently."
## [2] "The St. Louis plant had to close. It would die of old age. Workers had been making cars there since the onset of mass automotive production in the 1920s."
## [3] "WSU's plans quickly became a hot topic on local online sites. Though most people applauded plans for the new biomedical center, many deplored the potential loss of the building."
## [4] "The Alaimo Group of Mount Holly was up for a contract last fall to evaluate and suggest improvements to Trenton Water Works. But campaign finance records released this week show the two employees donated a total of $4,500 to the political action committee (PAC) Partners for Progress in early June. Partners for Progress reported it gave more than $10,000 in both direct and in-kind contributions to Mayor Tony Mack in the two weeks leading up to his victory in the mayoral runoff election June 15."
## [5] "And when it's often difficult to predict a law's impact, legislators should think twice before carrying any bill. Is it absolutely necessary? Is it an issue serious enough to merit their attention? Will it definitely not make the situation worse?"
##
##
## DATASET: Twitter
##
## [1] "How are you? Btw thanks for the RT. You gonna be in DC anytime soon? Love to see you. Been way, way too long."
## [2] "When you meet someone special... you'll know. Your heart will beat more rapidly and you'll smile for no reason."
## [3] "they've decided its more fun if I don't."
## [4] "So Tired D; Played Lazer Tag & Ran A LOT D; Ughh Going To Sleep Like In 5 Minutes ;)"
## [5] "Words from a complete stranger! Made my birthday even better :)"
The following histograms show the distribution of the number of words per line in the first 20,000 lines of each dataset.
par(mfrow = c(1, 3))
for (dataset in names(results)) {
hist(
results[[dataset]]$sample_word_counts,
breaks = 30,
main = paste(dataset, "Word Distribution"),
xlab = "Words per Line",
col = "lightblue",
border = "white"
)
}
par(mfrow = c(1, 1))
Identify the 15 most frequently occurring words in each dataset sample.
par(mfrow = c(1, 3))
for (dataset in names(results)) {
top_words <- head(
results[[dataset]]$word_frequency,
15
)
barplot(
top_words,
main = paste(dataset, "Top 15 Words"),
las = 2,
col = "steelblue",
cex.names = 0.7,
ylab = "Frequency"
)
}
par(mfrow = c(1, 1))
The exploratory analysis provides an initial understanding of the three SwiftKey datasets.
The following observations will be examined:
The summary statistics and visualisations generated above provide a foundation for further natural language processing.
The next stage of the project will focus on developing a predictive text algorithm.
The proposed approach includes:
The model will aim to predict the most likely next word based on the user’s input.
A Shiny application will be developed to demonstrate the predictive text algorithm.
The proposed application will include:
The application will demonstrate how natural language processing can be used to improve text entry and communication.
This milestone establishes the foundation for the SwiftKey predictive text application.
The three datasets have been identified and analysed using a memory-efficient approach.
Basic statistical summaries, sample text, word distribution histograms and word frequency analysis provide an initial understanding of the data.
The next phase will involve text preprocessing, n-gram modelling, word prediction and development of the interactive Shiny application.