library(tidyverse)## -- Attaching packages --------------------------------------- tidyverse 1.3.1 --## v ggplot2 3.3.5 v purrr 0.3.4
## v tibble 3.1.2 v dplyr 1.0.6
## v tidyr 1.1.3 v stringr 1.4.0
## v readr 1.4.0 v forcats 0.5.1## -- Conflicts ------------------------------------------ tidyverse_conflicts() --
## x dplyr::filter() masks stats::filter()
## x dplyr::lag() masks stats::lag()library(openintro)## Loading required package: airports## Loading required package: cherryblossom## Loading required package: usdataThis workbook is an in class assignment. After completion you will have a good practical knowledge of Multiple Regression and Logistic Regression.
There are two types of exercises in this Tutorial.
Response questions These questions are thought exercises and require a written response.
Response Question Example:
What is the Capital of North Dakota?
Response: Bismarck
Coding questions These questions require entering R code in the provided chunck.
Coding question example
Write the R code to calculate the average of the whole numbers from 1 to 10, enter your code here:
mean(1:10)## [1] 5.5We will consider data about loans from the peer-to-peer lender, Lending Club. The data is contained in the loan.csv file contained in the project directory.
loanloan <- read_csv("loan.csv")##
## -- Column specification --------------------------------------------------------
## cols(
## interest_rate = col_double(),
## income_ver = col_character(),
## debt_to_income = col_double(),
## credit_util = col_double(),
## bankruptcy = col_double(),
## term = col_double(),
## issued = col_character(),
## credit_checks = col_double()
## )You may find the following variable dictionary useful.
loan.| variable | description |
|---|---|
interest_rate |
Interest rate for the loan |
income_ver |
Categorical variable describing whether the borrower’s income source and amount have been verified, levels verified,source_only, not |
dept_to_income |
Debt-to-income ratio, which is the percent of total debt of the borrower divided by their total income. |
credit_util |
Of all the Credit available to the borrower, what fraction are they using. For example the credit utilization on the credit card would be the card’s balance divided by the card’s credit limit |
bankruptcy |
An indicator variable for whether the borrower has a past bankruptcy in her record. This variable takes a value of 1 if the answer is “yes” and 0 if the answer is “no”. |
term |
The length of the loan, in months. |
issued |
The month and year the loan was issued. |
credit_checks |
Number of credit checks in the last 12 months. For example, when filing an application for a credit cards, it is common for the company receiving the application to run a credit check. |
Recall the single variable models we have been studying. The prediction looks like this. \[ \hat{Y} = \hat{\beta_0} + \hat{\beta_1}X\] Where the variables denoted by the \(\hat{\,}\) over them are the estimates obtained by lm.
In every case we have, more or less implicitly, assumed that X is a numeric variable. Would this make any sense if \(X\) is a categorical variable?
Response Question What is a categorical variable? Variable which represents types of data which may be divided into groups
Coding Question
Consider the variable income_ver, what are the possible values of income_ver and what is the proportion of all records in each category of income_ver? Use code to find this, do not use the above dictionary, it might be wrong or out of date.
group_by(loan, income_ver) %>% summarise(proportion = n()/nrow(loan))So if we are interested in determining the variation of interest_rate as a function of income_ver. What does lm mean for problems like this? ‘lm’ (linear model) function carries out regression, single stratum analysis of varience and analysis of covariance.
To understand how to handle categorical variables, we will start with special type of categorical variable called an indicator variable. We have such a variable in the loan data set, it’s called bankruptcy and it takes the value 0 if the applicant has had no previous bankruptcies and 1 if the applicant has had at least one previous bankruptcy.
group_by(loan, bankruptcy) %>% summarise(proportion = n()/nrow(loan))income_ver and bankruptcy? Bankruptcy is an indicator variable while income_ver is a categorical variable. Income_ver describes whether the borrower’s income source and amount has been verified while bankruptcy describes whether the borrower had a past bankruptcy in the records.Given a value \(x\) of bankruptcy does it makes sense to multiply \(x\) by a number, e.g., \(2.3*x\)? Hint: consider the possible values of \(x\) When regressing, it does not make sense to multiply x by a number in the case of ‘bankruptcy’ because we have defined 0 as the situation in which the borrower has no bankruptcies in the records. Because of this, (and the way indicator variables work), any number other than 0 would be considered as ‘equivalent’ to 1 in terms of the result: there exists at least one bankruptcy on the records.
Summarize the results of regressing interest_rate on bankruptcy using lm
summary(lm(interest_rate ~ bankruptcy, data=loan))##
## Call:
## lm(formula = interest_rate ~ bankruptcy, data = loan)
##
## Residuals:
## Min 1Q Median 3Q Max
## -7.7648 -3.6448 -0.4548 2.7120 18.6020
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 12.3380 0.0533 231.490 < 2e-16 ***
## bankruptcy 0.7368 0.1529 4.819 1.47e-06 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 4.996 on 9998 degrees of freedom
## Multiple R-squared: 0.002317, Adjusted R-squared: 0.002217
## F-statistic: 23.22 on 1 and 9998 DF, p-value: 1.467e-06Interpret the results of this regression. Regressing interest rate on bankruptcy gave us an estimate of 12.33 for the intercept and 0.7368 for bankruptcy. The median of the residuals is also close to 0 while the magnitude of the max is greater than that of the min.
library(dplyr)x <- filter(loan, bankruptcy == "0")
summarize(x, Average = mean(interest_rate, na.rm = T))How can you explain this value? The values are the same because bankruptcy is an indicator variable, and the intercept represents the value when bankruptcy = 0.
What is the average interest rate for people who have had at least one bankruptcy?
y <- filter(loan, bankruptcy == "1")
summarize(y, Average = mean(interest_rate, na.rm = T))Could you have determined this value from the regression summary? If so, how? Yes, we add the estimates for intercept and bankruptcy: 12.338+0.7368=13.0748
How do you interpret the meaning of slope in this context? Slope is how much you can expect Y to change as X increases. In this context, an increase of 1 in ‘bankruptcy’ increases interest_rate by 0.7368.
Which is more important the estimated slope or \(R^2\)? The R^2 is more important because it is a measure of the correlation. The R^2 indicates the percentage of the variance in the dependent variable that the independent variables explain collectively. In cases where the data isn’t highly correlated, estimated slope is highly unreliable.
interest_rate on income_versummary(lm(interest_rate ~ income_ver, data=loan))##
## Call:
## lm(formula = interest_rate ~ income_ver, data = loan)
##
## Residuals:
## Min 1Q Median 3Q Max
## -9.0437 -3.7495 -0.6795 2.5345 19.6905
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 11.09946 0.08091 137.18 <2e-16 ***
## income_versource_only 1.41602 0.11074 12.79 <2e-16 ***
## income_ververified 3.25429 0.12970 25.09 <2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 4.851 on 9997 degrees of freedom
## Multiple R-squared: 0.05945, Adjusted R-squared: 0.05926
## F-statistic: 315.9 on 2 and 9997 DF, p-value: < 2.2e-16Response Question How many variables does the regression say we have? Three: intercept, income_versource_only, and income_ververified
Coding Question Calculate mean(interest_rate) for each level of the categorical variable income_ver
group_by(loan, income_ver) %>% summarise(average_int_rate = mean(interest_rate))How do your values correspond to the coefficient values in the regression? The not variable is equivalent to the intercept and source_only is the intercept + income_versource_only. Vice versa for verified.
Given what you know now why do you think you have two new variables related to income_ver? We now have two variables related to income_ver because income_ver is a categorical variable, under income_ver, data points are labeled as either ‘not’, ‘source_only’, or ‘verified’.
Do you get an extra variable in case of single variable regression? If so what is it? It is possible if you have a categorical variable with two or more definitions.
How would you write a regression model for the regression of interest_rate on income_verified? \[ \hat{Y} = \hat{\beta_0} + \hat{\beta_1}X_1 + \hat{\beta_2}X_2\] Where Y is the dependent variable (interest_rate), B0, B1 and B2 are the estimates of the intercept, and the two variables source_only and verified respectively.
Multiple regression means that we are regressing on a sum of variables. In fact when we regress on a categorical variable we are doing multiple regression! Why? Because when categorical variables with more than two levels are regressed on, the variable with n levels will be transformed into n-1 variables each.
Since the loan data set gives us a lot of variables let’s try regression on all of them. (run the code below)
summary(lm(interest_rate ~ income_ver + debt_to_income + credit_util + bankruptcy + term + issued + credit_checks, data = loan))##
## Call:
## lm(formula = interest_rate ~ income_ver + debt_to_income + credit_util +
## bankruptcy + term + issued + credit_checks, data = loan)
##
## Residuals:
## Min 1Q Median 3Q Max
## -19.9070 -3.4362 -0.7239 2.5397 18.0874
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 3.969e+00 2.087e-01 19.016 < 2e-16 ***
## income_versource_only 1.083e+00 1.036e-01 10.456 < 2e-16 ***
## income_ververified 2.482e+00 1.223e-01 20.293 < 2e-16 ***
## debt_to_income 3.787e-02 3.121e-03 12.137 < 2e-16 ***
## credit_util -4.323e-06 8.733e-07 -4.950 7.54e-07 ***
## bankruptcy 5.043e-01 1.383e-01 3.645 0.000269 ***
## term 1.495e-01 4.123e-03 36.257 < 2e-16 ***
## issuedJan2018 -2.061e-02 1.128e-01 -0.183 0.854969
## issuedMar2018 -9.280e-02 1.112e-01 -0.834 0.404037
## credit_checks 2.233e-01 1.917e-02 11.647 < 2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 4.489 on 9966 degrees of freedom
## (24 observations deleted due to missingness)
## Multiple R-squared: 0.1942, Adjusted R-squared: 0.1934
## F-statistic: 266.8 on 9 and 9966 DF, p-value: < 2.2e-16Comment on the \(p\)-values for the variables, noting anything interesting. The p-value (2.2e-16) is very close to 0 which shows that the result is of major practical importance. Since the p-value is less than 0.05, we can reject the null hypothesis.
Comment on the difference between \(R^2\) and adj-\(R^2\). Which do you think is more variable. Do you think the value of either \(R^2\) is too low to make this model useful? R^2 shows how well data points fit a curve or line, while adjusted R^2 also indicates how well data points fit a curve or line, but adjusts for the number of terms in a model. I think R^2 is more variable since adjusted R^2 is typically positive and is always lower than R^2.
How would you suggest making the model better? For multiple regression, adding more terms to the regression improves the fit. Additional terms will always improve the model whether it has significant value or not.
The multi-variate model we have just constructed is not necessarily the best model.
Why might this be the case? In the model, some of the standard errors are quite high (>0.05)
How might we try to improve things? Use a larger sample of data.
Let’s look at the coefficient estimates for the full model (the one with all the variables)
coef(summary(lm(interest_rate ~ income_ver + debt_to_income + credit_util + bankruptcy + term + issued + credit_checks, data = loan)))## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 3.969028e+00 2.087232e-01 19.0157518 3.163114e-79
## income_versource_only 1.082767e+00 1.035565e-01 10.4558002 1.866232e-25
## income_ververified 2.482072e+00 1.223104e-01 20.2932296 9.462920e-90
## debt_to_income 3.787448e-02 3.120564e-03 12.1370626 1.160422e-33
## credit_util -4.323176e-06 8.733479e-07 -4.9501194 7.538276e-07
## bankruptcy 5.042764e-01 1.383482e-01 3.6449793 2.687739e-04
## term 1.495000e-01 4.123314e-03 36.2572457 1.696856e-270
## issuedJan2018 -2.060967e-02 1.127529e-01 -0.1827863 8.549694e-01
## issuedMar2018 -9.279620e-02 1.112041e-01 -0.8344677 4.040375e-01
## credit_checks 2.232706e-01 1.916941e-02 11.6472314 3.770754e-31Response Question Look at the \(p\)-values. What are the largest what are smallest. Are any of the variable estimates not significant at the 95% level? The variable with the largest p-value is ‘issuedJan2018’. The variable with the smallest p-value is ‘term’. The variable estimates for ‘issuedJan2018’ and ‘issuedMar2018’ are not significant at the 95% level.
Coding Question Rerun the regression leaving out the non-significant variables.
coef(summary(lm(interest_rate ~ income_ver + debt_to_income + credit_util + bankruptcy + term + credit_checks, data = loan)))## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 3.928358e+00 1.960052e-01 20.042113 1.241835e-87
## income_versource_only 1.080223e+00 1.035067e-01 10.436255 2.287914e-25
## income_ververified 2.479061e+00 1.222547e-01 20.277831 1.277232e-89
## debt_to_income 3.790429e-02 3.120197e-03 12.148042 1.016557e-33
## credit_util -4.319428e-06 8.732526e-07 -4.946367 7.684602e-07
## bankruptcy 5.054572e-01 1.383325e-01 3.653931 2.595819e-04
## term 1.495189e-01 4.122211e-03 36.271519 1.065862e-270
## credit_checks 2.233441e-01 1.916791e-02 11.651979 3.568771e-31Compare this more parsimonious model with the full model. What are your observations? The p-values in this regression model are slightly smaller than the previous model.
If you were to delete a variable from this model, which one would you delete? The variable ‘bankruptcy’.
summary(lm(interest_rate ~ income_ver + debt_to_income + credit_util + term + credit_checks, data = loan))##
## Call:
## lm(formula = interest_rate ~ income_ver + debt_to_income + credit_util +
## term + credit_checks, data = loan)
##
## Residuals:
## Min 1Q Median 3Q Max
## -20.0667 -3.4379 -0.6904 2.5357 18.0371
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 3.980e+00 1.956e-01 20.344 < 2e-16 ***
## income_versource_only 1.084e+00 1.036e-01 10.465 < 2e-16 ***
## income_ververified 2.481e+00 1.223e-01 20.279 < 2e-16 ***
## debt_to_income 3.818e-02 3.121e-03 12.234 < 2e-16 ***
## credit_util -4.554e-06 8.714e-07 -5.226 1.76e-07 ***
## term 1.496e-01 4.125e-03 36.265 < 2e-16 ***
## credit_checks 2.295e-01 1.911e-02 12.010 < 2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 4.491 on 9969 degrees of freedom
## (24 observations deleted due to missingness)
## Multiple R-squared: 0.193, Adjusted R-squared: 0.1925
## F-statistic: 397.4 on 6 and 9969 DF, p-value: < 2.2e-16What are your observations? The median of the residuals is close to 0 while the magnitudes of the min and max are quite close to each other. Most of the p-values are <2e-16 (0 via significance codes). The standard errors are also slightly smaller than in the previous model.