Working with categorical predictors

Let’s keep using the Credit data set part of the ISLR package. Load the package:

library(ISLR)
str (Credit)
## 'data.frame':    400 obs. of  12 variables:
##  $ ID       : int  1 2 3 4 5 6 7 8 9 10 ...
##  $ Income   : num  14.9 106 104.6 148.9 55.9 ...
##  $ Limit    : int  3606 6645 7075 9504 4897 8047 3388 7114 3300 6819 ...
##  $ Rating   : int  283 483 514 681 357 569 259 512 266 491 ...
##  $ Cards    : int  2 3 4 3 2 4 2 2 5 3 ...
##  $ Age      : int  34 82 71 36 68 77 37 87 66 41 ...
##  $ Education: int  11 15 11 11 16 10 12 9 13 19 ...
##  $ Gender   : Factor w/ 2 levels " Male","Female": 1 2 1 2 1 1 2 1 2 2 ...
##  $ Student  : Factor w/ 2 levels "No","Yes": 1 2 1 1 1 1 1 1 1 2 ...
##  $ Married  : Factor w/ 2 levels "No","Yes": 2 2 1 1 2 1 1 1 1 2 ...
##  $ Ethnicity: Factor w/ 3 levels "African American",..: 3 2 2 2 3 3 1 2 3 1 ...
##  $ Balance  : int  333 903 580 964 331 1151 203 872 279 1350 ...

The categorical predictors in the Credit data frame are Gender, Student (student status), Married (marital status, and Ethnicity).

How to find the reference level of a categorical predictor?

Two ways: Using levels() or contrasts()

# For example, for Gender

levels(Credit$Gender)
## [1] " Male"  "Female"
contrasts(Credit$Gender)
##        Female
##  Male       0
## Female      1
# For example, for Ethnicity

levels(Credit$Ethnicity)
## [1] "African American" "Asian"            "Caucasian"
contrasts(Credit$Ethnicity)
##                  Asian Caucasian
## African American     0         0
## Asian                1         0
## Caucasian            0         1

Optional: How to change the reference level?

If you want to change the reference level, you can use the relevel() function:

relevel(x, ref, …)

x: an unordered factor. ref: the reference level, typically a string.

For example, let’s find out the reference level of marital status and then change it:

levels(Credit$Married)
## [1] "No"  "Yes"
contrasts(Credit$Married)
##     Yes
## No    0
## Yes   1

The reference level is “No” (Not married). Change the reference level to “Yes” (Married).

To do it in a SAFER way, let’s create a new variable called MarriedNew with the new reference level.

Credit$MarriedNew = relevel(Credit$Married, "Yes")
str(Credit)
## 'data.frame':    400 obs. of  13 variables:
##  $ ID        : int  1 2 3 4 5 6 7 8 9 10 ...
##  $ Income    : num  14.9 106 104.6 148.9 55.9 ...
##  $ Limit     : int  3606 6645 7075 9504 4897 8047 3388 7114 3300 6819 ...
##  $ Rating    : int  283 483 514 681 357 569 259 512 266 491 ...
##  $ Cards     : int  2 3 4 3 2 4 2 2 5 3 ...
##  $ Age       : int  34 82 71 36 68 77 37 87 66 41 ...
##  $ Education : int  11 15 11 11 16 10 12 9 13 19 ...
##  $ Gender    : Factor w/ 2 levels " Male","Female": 1 2 1 2 1 1 2 1 2 2 ...
##  $ Student   : Factor w/ 2 levels "No","Yes": 1 2 1 1 1 1 1 1 1 2 ...
##  $ Married   : Factor w/ 2 levels "No","Yes": 2 2 1 1 2 1 1 1 1 2 ...
##  $ Ethnicity : Factor w/ 3 levels "African American",..: 3 2 2 2 3 3 1 2 3 1 ...
##  $ Balance   : int  333 903 580 964 331 1151 203 872 279 1350 ...
##  $ MarriedNew: Factor w/ 2 levels "Yes","No": 1 1 2 2 1 2 2 2 2 1 ...

Checked that the changed took place:

levels(Credit$MarriedNew)
## [1] "Yes" "No"
contrasts(Credit$MarriedNew)
##     No
## Yes  0
## No   1

Now remove the MarriedNew variable because we actually do NOT need it. I just created it to illustrate how to change the reference level of a categorical predictor.

Credit = Credit[ , colnames(Credit)!="MarriedNew"]
str(Credit)
## 'data.frame':    400 obs. of  12 variables:
##  $ ID       : int  1 2 3 4 5 6 7 8 9 10 ...
##  $ Income   : num  14.9 106 104.6 148.9 55.9 ...
##  $ Limit    : int  3606 6645 7075 9504 4897 8047 3388 7114 3300 6819 ...
##  $ Rating   : int  283 483 514 681 357 569 259 512 266 491 ...
##  $ Cards    : int  2 3 4 3 2 4 2 2 5 3 ...
##  $ Age      : int  34 82 71 36 68 77 37 87 66 41 ...
##  $ Education: int  11 15 11 11 16 10 12 9 13 19 ...
##  $ Gender   : Factor w/ 2 levels " Male","Female": 1 2 1 2 1 1 2 1 2 2 ...
##  $ Student  : Factor w/ 2 levels "No","Yes": 1 2 1 1 1 1 1 1 1 2 ...
##  $ Married  : Factor w/ 2 levels "No","Yes": 2 2 1 1 2 1 1 1 1 2 ...
##  $ Ethnicity: Factor w/ 3 levels "African American",..: 3 2 2 2 3 3 1 2 3 1 ...
##  $ Balance  : int  333 903 580 964 331 1151 203 872 279 1350 ...

End of Optional

Let’s practice how to write and use a regression equation with categorical predictors

To keep it simple, let’s obtain the equation of Balance versus two predictors: Ethnicity (categorical) and Income (quantitative)

out_Balance_Ethn_Inc = lm(Balance ~ Income + Ethnicity, data= Credit)

summary(out_Balance_Ethn_Inc)
## 
## Call:
## lm(formula = Balance ~ Income + Ethnicity, data = Credit)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -806.45 -346.85  -52.38  332.57 1097.57 
## 
## Coefficients:
##                    Estimate Std. Error t value Pr(>|t|)    
## (Intercept)        242.4882    49.5674   4.892 1.45e-06 ***
## Income               6.0507     0.5813  10.410  < 2e-16 ***
## EthnicityAsian       2.4566    57.7230   0.043    0.966    
## EthnicityCaucasian   6.6188    50.3215   0.132    0.895    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 408.9 on 396 degrees of freedom
## Multiple R-squared:  0.215,  Adjusted R-squared:  0.2091 
## F-statistic: 36.16 on 3 and 396 DF,  p-value: < 2.2e-16

None of the two dummies created from Ethnicity are significant. There is NO statistically significant difference in average balance between Caucasian and African American and between Asian and African American.

Therefore, the equation that we would use in this case is (notice that the dummies for Ethnicity are NOT included!):

Predicted Balance = 242.48 + 6.0507*Income

Example: Use the equation to estimate the Balance for a person (of any ethnicity) with an Income of $70 000.

# Manually plug in the value of Income in the equation

242.4882  + 6.0507*70
## [1] 666.0372

Using code to make the same prediction:

predict(out_Balance_Ethn_Inc, data.frame(Income= 70, Ethnicity='African American'))
##        1 
## 666.0397
# If the categorical variable is NOT significant, we have to use the reference category when using predict()

Let’s MAKE UP a hypothetical situation where we obtain a significant result for the Asian dummy but not for the Caucasian dummy. In this made up (invented) situation, the PV for EthnicityAsian was less than alpha but the PV for EthnicityCaucasian was greater than alpha.

In this INVENTED SCENARIO, the equation that we use would be:

Predicted Balance = 242.48 + 6.0507 * Income + 2.4566 * EthnicityAsian

Now, let’s use this equation to estimate the Balance for an Asian person with an Income of $70 000.

242.4882  + 6.0507*70 + 2.4566*1
## [1] 668.4938

Using code:

predict(out_Balance_Ethn_Inc, data.frame(Income= 70, Ethnicity='Asian'))
##        1 
## 668.4964

Now, let’s use this equation to estimate the Balance for an African American (or a Caucasian) person with an Income of $70 000. For both African American and Caucasian, we do it the same way.

 242.4882  + 6.0507*70 + 2.4566*0
## [1] 666.0372
predict(out_Balance_Ethn_Inc, data.frame(Income= 70, Ethnicity= 'African American'))
##        1 
## 666.0397
# Use the reference category for any non-significant dummy

Let’s MAKE UP a second hypothetical situation and IMAGINE that we got a significant result for both Asian and Caucasian.

Predicted Balance = 242.48 + 6.0507 * Income + 2.4566 * EthnicityAsian + 6.6188 * EthnicityCaucasian

Now, let’s use this equation to estimate the Balance for an Asian person with an Income of $70 000.

 242.4882  + 6.0507*70 + 2.4566*1 + 6.6188*0
## [1] 668.4938

Estimate the Balance for a Caucasian person with an Income of $70 000.

242.4882  + 6.0507*70 + 2.4566*0 + 6.6188*1
## [1] 672.656

Estimate the Balance for an African American person with an Income of $70 000.

242.4882 + 6.0507*70 + 2.4566*0 + 6.6188*0
## [1] 666.0372