Naïve Bayes Classifier (NBC)

How can we improve it?
We will improve ’s the NBC for discrete predictors. Let us recall the NBC formula

\[Pr(Y = k | X = x) = \frac{\pi_{k}\times f_{k1}(x_{1}) \times f_{k2}(x_{2}) \times\ \times f_{kp}(x_{p}) }{\sum\limits_{l=1}^{K}\pi_{l}\times f_{l1}(x_{1}) \times f_{l2}(x_{2}) \times … \times f_{lp}(x_{p})}\]

\[k: \text{Some class}\]

\[\pi_{k} = Pr(Y = k)\]

\[f_{li} = Pr(X = x | Y = k)\]

Naïve Bayes Classifier (NBC)

How can we improve it?
Collapsing the NBC, we have

\[Pr(Y = k | X = x) = \alpha \cdot\ \pi_{k}\prod\limits_{j=1}^{p}{f_{kj}(x_{j})}\]

\[\alpha = \sum\limits_{l=1}^{K}\pi_{l}\times f_{l1}(x) \times f_{l2}(x_{2}) \times … \times f_{lp}(x_{p})\]

Naïve Bayes Classifier (NBC)

How can we improve it?
We can rewrite the values of \(\pi_{k}\) and \(\prod\limits_{j=1}^{p}{f_{kj}(x_{j})}\) as follows

\[\pi_{k} = \frac{N(Y=k)}{N(D)}\]

\[N(Y = k): \text{ The number of observations in a particular class}\]

\[N(D):\text{ The number of observations in the data set}\]

\[\prod\limits_{j=1}^{p}{f_{kj}(x_{j})} = \prod\limits_{j=1}^{p} \frac{N(C=k, f_{kj}(x_{j}))}{N(Y = k)}\]

\[N(C=k, f_{kj}(x_{j})): \text{ The number of observations that share the same value for a predictor in some class}\]

\[N(Y = k): \text{ The number of values in a class}\]

Naïve Bayes Classifier (NBC)

How can we improve it?
It’s typically believed that the more times an observation takes on a particular value of an predictor, the more importance it holds within the overall population, therefore we define weights for each predictor. However there might be an instance where we do not observe a value of a particular predictor, so we create weights utilizing a Laplacian transformation

\[w_{j} = \frac{N(A_{j} = x_{j}) + 1}{N(D) + q_{j}} \quad\quad q_{j}\text{ is the number of unique values a predictor can take on}\]

\[N(A_{j} = x_{j}) \text{ The number of observations that share the same value for a predictor}\]

Naïve Bayes Classifier (NBC)

How can we improve it?
Now rewriting NBC with the weights, we have

\[Pr(Y = k | X = x) = \alpha \cdot\ \pi_{k}\prod\limits_{j=1}^{p}{f_{kj}(x_{j})}\cdot w_{j} = \alpha \cdot \frac{N(Y=k)}{N(D)} \cdot \prod\limits_{j=1}^{p}{\frac{N(C=k, f_{kj}(x_{j}))}{N(Y = k)}\cdot \frac{N(A_{j} = x_{j}) + 1}{N(D)+q_{j}}}\]

Naïve Bayes Classifier (NBC)

How can we improve it?
Here are the results of this updated NBC based on simulated data

Model accuracy for increasing sample sizes

Naïve Bayes Classifier (NBC)

How can we improve it?
Here are the results of this updated NBC

Model accuracy for increasing number of predictors

Naïve Bayes Classifier (NBC)

How can we improve it?
Here are the results of this updated NBC

Model accuracy for increasing number of categories

# Naïve Bayes Classifier (NBC)

How can we improve it?
Here are the results of this improved NBC Empirical research found that this improved NBC improved accuracy from 49.5% to 92% in comparison to the original NBC.