Ridge and Lasso binary logistic regression in NLP using R

Vilnius University

• What is natural language processing
• NLP definitions
• Logistic regression in NLP tasks
• Quora example

Natural language processing (NLP) is a sub field of computer science concerned with the interactions between computers and human (natural) languages, in particular how to program computers to process and analyze large amounts of natural language data.

• Classifying text
• Chat bots
• Word embedding

Corpus

A text corpus is a collection of texts of written (or spoken) language presented in electronic form.

A document is the row of a matrix

A token is single term from the corpus

Tokens

Eligijus is presenting now.

[Eligijus], [is], [presenting], [now]

[Eligijus is], [presenting now]

N - grams

An n-gram is a contiguous sequence of n items from a given sequence of text.

2 - gram (bigram):

[Eligijus is], [is presenting], [presenting now]

3 - gram:

[Eligijus is presenting], [is presenting now]

A document is about cars:

The document term matrix would then look like:

Let us say we have n documents and k unique tokens. Each document is assigned a binary label - {0, 1}.

Then the general model is:

\[ log \dfrac{P(Y = 1)}{P(Y = 0)} = \sum_{i = 0}^{k}\beta_{i} X_{i} \, (1) \]

We start indexing from zero because of the intercept.

We get the coefficients \( \widehat{\beta} \) by maximizing:

\[ \sum_{i = 1}^{n}\left(y_{i}(\beta_{0} + \beta^T x_{i}) - log(1 + e^{\beta_{0} + \beta^T x_{i}})) \right) \]

in respect to \( \beta_{0} \) and \( \beta \)

Ridge (more uniform distribution of coefficient values):

\[ \dfrac{1}{N} \sum_{i = 1}^{n}\left(y_{i}(\beta_{0} + \beta^T x_{i}) - log(1 + e^{\beta_{0} + \beta^T x_{i}})) \right) - \alpha \sum_{j = 1}^{k} \beta_{j} ^ 2 \]

Lasso (reduces number of features):

\[ \dfrac{1}{N} \sum_{i = 1}^{n}\left(y_{i}(\beta_{0} + \beta^T x_{i}) - log(1 + e^{\beta_{0} + \beta^T x_{i}})) \right) - \alpha \sum_{j = 1}^{k} |\beta_{j}| \]

Competition:

https://www.kaggle.com/c/quora-insincere-questions-classification

Data:

https://www.kaggle.com/c/quora-insincere-questions-classification/data

Y = 1 if the question is not sincere

Y = 0 if the question is sincere

dim(dtm)

[1] 161620  10956

model <- glmnet(x = dtm, y = train$target,
                  family = 'binomial',
                  alpha = 0,
                  lambda = 0.01)

dim(coef_table)

[1] 10957     2

model <- glmnet(x = dtm, y = train$target,
                  family = 'binomial',
                  alpha = 1,
                  lambda = 0.01)

dim(coef_table)

[1] 266   2