Word2Vec

Load the libraries + functions

Load all the libraries or functions that you will use to for the rest of the assignment. It is helpful to define your libraries and functions at the top of a report, so that others can know what they need for the report to compile correctly.

library(reticulate)

Load the Python libraries or functions that you will use for that section.

import gensim
import re
import nltk
import pandas as pd
import numpy as np
from bs4 import BeautifulSoup
from nltk.corpus import stopwords
from nltk.corpus import abc
from sklearn.metrics import classification_report
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from gensim.models.fasttext import FastText
from gensim.models import Doc2Vec
from gensim.models.doc2vec import TaggedDocument
from sklearn import utils

The Data

The dataset is a set of Youtube comments that have been coded as: - 1: spam youtube messages - 0: good youtube messages - This data is stored in the CLASS column

Import the data using either R or Python. I put a Python chunk here because you will need one to import the data, but if you want to first import into R, that’s fine.

df = pd.read_csv (r'youtube_spam.csv')
print (df)

##                                        COMMENT_ID  ... CLASS
## 0     LZQPQhLyRh80UYxNuaDWhIGQYNQ96IuCg-AYWqNPjpU  ...     1
## 1     LZQPQhLyRh_C2cTtd9MvFRJedxydaVW-2sNg5Diuo4A  ...     1
## 2     LZQPQhLyRh9MSZYnf8djyk0gEF9BHDPYrrK-qCczIY8  ...     1
## 3             z13jhp0bxqncu512g22wvzkasxmvvzjaz04  ...     1
## 4             z13fwbwp1oujthgqj04chlngpvzmtt3r3dw  ...     1
## ...                                           ...  ...   ...
## 1951          z12sjp3zgtqnvlysj23zuxxaolrvd1oj504  ...     0
## 1952        z132enrpoy35yxpoe04cjr4zur3jvbyq3xo0k  ...     0
## 1953            z132jbmxfqm4fjysg23nwjfb2mv2vxnua  ...     1
## 1954          z12cdlswetvnejcri04cex0jfwy2u3tzj54  ...     0
## 1955        z120e5uautvcuper304ccf4bjrjugdpbwrc0k  ...     0
## 
## [1956 rows x 5 columns]

Clean up the data

Use one of our clean text functions to clean up the CONTENT column in the dataset.

REPLACE_BY_SPACE_RE = re.compile('[/(){}\[\]\|@,;]')
BAD_SYMBOLS_RE = re.compile('[^0-9a-z #+_]')
STOPWORDS = set(stopwords.words('english'))

def clean_text(text):
    text = BeautifulSoup(text, "html.parser").text
    text = text.lower()
    text = REPLACE_BY_SPACE_RE.sub(' ', text)
    text = BAD_SYMBOLS_RE.sub('', text)
    text = ' '.join(word for word in text.split() if word not in STOPWORDS)
    return text

df['CONTENT'] = df['CONTENT'].apply(clean_text)

df.head()

##                                     COMMENT_ID  ... CLASS
## 0  LZQPQhLyRh80UYxNuaDWhIGQYNQ96IuCg-AYWqNPjpU  ...     1
## 1  LZQPQhLyRh_C2cTtd9MvFRJedxydaVW-2sNg5Diuo4A  ...     1
## 2  LZQPQhLyRh9MSZYnf8djyk0gEF9BHDPYrrK-qCczIY8  ...     1
## 3          z13jhp0bxqncu512g22wvzkasxmvvzjaz04  ...     1
## 4          z13fwbwp1oujthgqj04chlngpvzmtt3r3dw  ...     1
## 
## [5 rows x 5 columns]

Split the data

Split the data into testing and training data.

X=df['CONTENT']
y=df['CLASS']
X_train, X_test, y_train, y_test = train_test_split(X,y, test_size=0.20, random_state=42)

Process the data

For FastText OR word2vec, create the tokenized vectors of the text.

tokenized_train = [nltk.tokenize.word_tokenize(text)
                   for text in X_train.to_list()]
tokenized_test = [nltk.tokenize.word_tokenize(text)
                   for text in X_test.to_list()]

Build Model

Build either a word2vec or FastText model.

w2v_model = gensim.models.Word2Vec(tokenized_train, 
                                   size=100, window=6,
                                   min_count=2, iter=5, workers=4)

Convert the model

Convert the model into a set of features to use in our classifier.

def document_vectorizer(corpus, model, num_features):
    vocabulary = set(model.wv.index2word)
    
    def average_word_vectors(words, model, vocabulary, num_features):
        feature_vector = np.zeros((num_features,), dtype="float64")
        nwords = 0.
        
        for word in words:
            if word in vocabulary: 
                nwords = nwords + 1.
                feature_vector = np.add(feature_vector, model.wv[word])
        if nwords:
            feature_vector = np.divide(feature_vector, nwords)

        return feature_vector

    features = [average_word_vectors(tokenized_sentence, model, vocabulary, num_features)
                    for tokenized_sentence in corpus]
    return np.array(features)
  
  
avg_wv_train_features = document_vectorizer(corpus=tokenized_train,
                                                    model=w2v_model,
                                                     num_features=100)
avg_wv_test_features = document_vectorizer(corpus=tokenized_test, 
                                                    model=w2v_model,
                                                    num_features=100)

Build a classifier model

In class, we used logistic regression to classify the data. You can use any machine learning algorithm you want here, and build a classification model.

my_tags = ["not spam", "spam"]
logreg = LogisticRegression(solver='lbfgs', multi_class='ovr', max_iter=10000)
logreg = logreg.fit(avg_wv_train_features, y_train)

Examine the results

Print out the accuracy, recall, and precision of your model.

y_pred = logreg.predict(avg_wv_test_features)
print('accuracy %s' % accuracy_score(y_pred, y_test))

## accuracy 0.6785714285714286

print(classification_report(y_test, y_pred,target_names=my_tags))

##               precision    recall  f1-score   support
## 
##     not spam       0.73      0.59      0.65       200
##         spam       0.64      0.77      0.70       192
## 
##     accuracy                           0.68       392
##    macro avg       0.69      0.68      0.68       392
## weighted avg       0.69      0.68      0.68       392