Data Science Capstone - Exploratory Data Analysis

1. Introduction

The Data Science Capstone project focuses on analysing a large collection of text data obtained from blogs, news articles, and Twitter. The main objective of this exploratory analysis is to understand the structure and characteristics of the data before developing a predictive text application.

This report presents basic summaries, exploratory statistics, visualizations, key findings, and a plan for developing a future prediction algorithm and Shiny application.

2. Loading and Understanding the Data

The dataset contains three English-language text files:

• Blogs
• News
• Twitter

The files were downloaded from the Coursera SwiftKey dataset and stored in the final/en_US folder.

3. Dataset Summary

Number of Lines

The following table shows the total number of lines present in each dataset.

Number of Lines in Each Dataset

	Dataset	Lines
Blogs	Blogs	899288
News	News	1010206
Twitter	Twitter	2360148

Number of Words

The following table shows the estimated total number of words in each dataset.

Number of Words in Each Dataset

	Dataset	Words
Blogs	Blogs	37546806
News	News	34761151
Twitter	Twitter	30096690

File Size

The following table shows the size of each dataset file.

File Size of Each Dataset in MB

Dataset	Size_MB
Blogs	200.42
News	196.28
Twitter	159.36

Exploratory Analysis

The following analysis is based on a sample of 50,000 lines from each dataset.

Summary of the Exploratory Samples

Dataset	Sample_Lines	Average_Line_Length
Blogs	50000	228.66
News	50000	202.28
Twitter	50000	68.52

4. Exploratory Visualizations

Average Line Length

The distribution of line lengths provides an initial understanding of the structure of the text data.

Key Findings

The exploratory analysis provides several useful observations about the datasets.

• The Blogs, News, and Twitter datasets contain a large number of text entries.
• The datasets differ considerably in their number of lines and words.
• Twitter data generally contains shorter text entries compared with Blogs and News.
• Blogs and News contain longer text entries and therefore provide richer textual information.
• The variation in line lengths shows that different text sources have different writing patterns.

These observations will be useful when developing a predictive text model in the later stages of the project.

5. Prediction Algorithm Plan

The final stage of the project will focus on developing a predictive text algorithm.

The model will analyse sequences of words and use them to predict the next likely word entered by a user. The cleaned text data will be used to create a suitable training dataset.

The planned steps include:

1. Cleaning and preprocessing the text.
2. Tokenizing the text into words.
3. Creating n-gram sequences.
4. Calculating word frequencies and probabilities.
5. Building a predictive model.
6. Evaluating the prediction performance.

The model will be designed to balance prediction accuracy with computational efficiency.

6. Shiny Application Plan

A Shiny application will be developed to provide an interactive interface for the predictive text model.

Users will enter a phrase into the application, and the model will return possible next-word predictions.

The application will focus on simplicity, usability, and fast response time.

7. Conclusion

The exploratory analysis provided an initial understanding of the Blogs, News, and Twitter datasets. The data was summarized using line counts, word counts, file sizes, and sample-based statistics.

The visualizations also showed differences in text length across the three sources. These findings provide a useful foundation for further text preprocessing, predictive modeling, and development of the Shiny application.

The next stage will focus on building and evaluating the predictive text algorithm using the insights gained from this exploratory analysis.