title: “Exploratory Analysis of Text Data” author: “Jay Prakash Kumar” date: “2026-06-05” output: html_document ———————

Executive Summary

The purpose of this report is to perform exploratory analysis on the English language datasets provided for the Data Science Capstone Project. The analysis focuses on understanding the structure, size, and characteristics of the datasets before building a next-word prediction application.

The datasets contain text collected from blogs, news articles, and Twitter feeds. Understanding these datasets is important because preprocessing and exploratory analysis directly impact the performance of predictive language models.

This report summarizes dataset statistics, preprocessing steps, exploratory findings, and visualizations.

Introduction

Natural Language Processing (NLP) is widely used in modern applications such as mobile keyboards, search engines, chatbots, and recommendation systems. One of the key NLP tasks is predicting the next word in a sentence.

The objective of the Capstone Project is to build a predictive text model capable of suggesting the next word based on a user’s input phrase.

Before building predictive models, it is necessary to perform exploratory analysis on the training datasets. This helps identify:

The datasets analyzed in this report are written in English and were provided as part of the SwiftKey and Coursera Data Science Capstone Project.

Dataset Description

The project uses three separate English language datasets:

  1. Blogs Dataset
  2. News Dataset
  3. Twitter Dataset

Each dataset contains different writing styles and structures.

Blogs Dataset

The blogs dataset contains long-form informal text written by bloggers. The text often contains conversational language and detailed paragraphs.

News Dataset

The news dataset contains professionally written news articles. The language is generally formal and grammatically correct.

Twitter Dataset

The Twitter dataset contains short social media posts limited by tweet character restrictions. The language is highly informal and includes abbreviations, hashtags, and slang.

File Statistics

The following table summarizes the approximate size and structure of the datasets.

Dataset File Size Number of Lines Approximate Word Count
Blogs 200 MB 899,288 37 Million
News 196 MB 1,010,242 34 Million
Twitter 159 MB 2,360,148 30 Million

Observations

  • The Twitter dataset contains the largest number of lines because tweets are generally very short.
  • The Blogs dataset contains the highest number of total words.
  • The News dataset provides more formal sentence structures.
  • Combining all three datasets provides a balanced source of natural language text.

Data Cleaning and Preprocessing

Raw text data cannot be directly used for predictive modeling. Therefore, several preprocessing steps are necessary.

The following preprocessing techniques were considered:

Converting Text to Lowercase

Converting all text to lowercase ensures consistency during word matching.

Example:

  • “Hello” and “hello” become identical.

Removing Punctuation

Punctuation marks such as commas, periods, and quotation marks were removed to simplify text analysis.

Removing Numbers

Numbers were removed because they generally do not contribute significantly to next-word prediction.

Removing Extra White Spaces

Extra spaces and formatting inconsistencies were cleaned to standardize the datasets.

Removing Special Characters

Special symbols, emojis, and non-English characters were removed where necessary.

These preprocessing steps improve model efficiency and reduce noise in the datasets.

Exploratory Analysis

Exploratory analysis was conducted to understand the structure and composition of the datasets.

The analysis focused on:

Understanding these characteristics helps in selecting suitable approaches for predictive text modeling.

Word Count Visualization

The following plot compares approximate word counts across the three datasets.

word_counts <- c(37,34,30)

barplot(word_counts,
        names.arg = c("Blogs","News","Twitter"),
        main = "Approximate Word Counts by Dataset",
        col = c("steelblue","darkgreen","orange"),
        ylab = "Millions of Words")

Interpretation

The Blogs dataset contains the highest number of words. This suggests that blog posts contain longer and more descriptive text compared to tweets and news articles.

Line Count Visualization

The following plot compares the number of lines across datasets.

line_counts <- c(899288,1010242,2360148)

barplot(line_counts,
        names.arg = c("Blogs","News","Twitter"),
        main = "Number of Lines by Dataset",
        col = c("purple","red","gold"),
        ylab = "Line Count")

Interpretation

The Twitter dataset contains the highest number of lines because tweets are short and separated into individual entries.

Importance of Exploratory Analysis

Exploratory analysis is a critical step in data science projects because it allows data scientists to:

For text prediction systems, understanding language structure and word frequency is essential for building accurate models.

Challenges in Text Prediction

Several challenges exist when working with natural language datasets:

Addressing these challenges requires effective preprocessing and model optimization techniques.

Future Work

The next stages of the project will include:

These steps will ultimately lead to a fully functional next-word prediction system.

Conclusion

This exploratory analysis provided valuable insights into the English text datasets used for the Data Science Capstone Project.

The datasets are large, diverse, and suitable for natural language processing tasks. Preprocessing and exploratory analysis revealed important characteristics related to word counts, line counts, and dataset composition.

The findings from this report will support the development of predictive text models and help improve the performance of the final next-word prediction application.