Introduction to Text Analysis

Initial Steps

Downloading

import requests
from collections import Counter

url = "https://www.gutenberg.org/files/1342/1342-0.txt"

document = requests.get(url).text

Tokenizing

words = document.split()

print(words)

Activity 1

Try downloading and tokenizing a copy of another book from Project Gutenberg.

Save the results as my_document and my_words

Word Frequencies

counts = Counter(words)

counts.most_common(25)

Cleaning Data

# Use Regular Expressions to get rid of non-word characters
import re

cleaned = []

for word in words:
  clean_word = re.sub(r'[^a-zA-Z]', '', word)
  clean_word = clean_word.lower()
  cleaned.append(clean_word)

print(cleaned)

frequencies = Counter(cleaned)
frequencies.most_common(25)

Activity 2

Take a few minutes to find the top 10 most frequent words from your book (my_document).

What are they? Put a few of them in the chat.

Removing Stopwords

# Deal with stopwords
import nltk
from nltk.corpus import stopwords
nltk.download('stopwords')
# 'punkt' is for sentence tokenizing later
nltk.download('punkt')

stop_words = stopwords.words('english')

print(stop_words)

cleaned = []

for word in words:
  clean_word = re.sub(r'[^a-zA-Z]', '', word)
  clean_word = clean_word.lower()
  if clean_word not in stop_words:
    cleaned.append(clean_word)

frequencies = Counter(cleaned)
frequencies.most_common(25)

Adding our own stopwords

stop_words = stopwords.words('english')

my_stop_words = ['mr', 'could', 'would', 'may', 'might']

for word in my_stop_words:
  stop_words.append(word)

cleaned = []

for word in words:
  clean_word = re.sub(r'[^a-zA-Z]', '', word)
  clean_word = clean_word.lower()
  if clean_word not in stop_words:
    cleaned.append(clean_word)

frequencies = Counter(cleaned)
frequencies.most_common(25)

Activity 3

Add some custom stopwords to your own list.

What are they? What new words made it into your top 25 after this?

Saving Results

import pandas as pd
import seaborn as sns

# Writing to a file in colab requires mounting GDrive
from google.colab import drive
drive.mount('/content/gdrive')

word_col = []
count_col = []

for word, count in frequencies.most_common(25):
  word_col.append(word)
  count_col.append(count)

most_frequent = pd.DataFrame(list(zip(word_col, count_col)), columns = ['word', 'count'])
most_frequent

most_frequent.to_csv('/content/gdrive/MyDrive/most_frequent.csv')

Activity 4

Save your results to your own Google Drive.

# Making a basic visualization
sns.catplot(
    data = most_frequent,
    x = "word",
    y = "count",
    kind = "bar",
    aspect = 3
)

Sentiment Analysis

# install vaderSentiment to do sentiment analysis
# VADER (Valence Aware Dictionary and sEntiment Reasoner)
# -4 to 4 for each word
!pip install vaderSentiment

from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer

sa = SentimentIntensityAnalyzer()

# Preview the lexicon
sa.lexicon

# overall polarity scores are -1 to 1
my_negative_sentence = "I absolutely hate Victorian novels. They are the worst."

sa.polarity_scores(my_negative_sentence)

my_positive_sentence = "Victorian novels are fantastic. I completely adore them."

sa.polarity_scores(my_positive_sentence)

Activity 5

Write a very positive sentence and a very negative sentence.

Analyze their polarity scores with VADER.

compound = sa.polarity_scores(my_positive_sentence)['compound']
print(compound)

from nltk.tokenize import sent_tokenize

sentences = sent_tokenize(document)

sentences[0:10]

score = []
text = []


for sentence in sentences:
  text.append(sentence)
  score.append(sa.polarity_scores(sentence)['compound'])

scores = pd.DataFrame(list(zip(score, text)), columns = ['score', 'text'])
scores

scores.to_csv('/content/gdrive/MyDrive/scores.csv')

sns.relplot(
    data = scores,
    x = scores.index,
    y = 'score',
    kind = 'line',
    aspect = 4
)

Activity 6

Plot the polarity scores (sentence-by-sentence) of your book.