Midterm: Chicago Housing Case Study
Emma Ross
April 1, 2024
Instruction
The midterm will be conducted from 7:00 PM to 9:00 PM. The submission will be closed sharp at 9:15 PM.
Instruction: This midterm requires you to use R. It
is completely open book/notes/internet. However any versions of ChatGPT
or alike are strictly prohibitive. Write your answers using this .rmd
file and knitr it into the html file. Show your codes, plots or R-output
when needed. You always need to show your code with
echo = TRUE which is the default setup for this file. If
you have trouble formatting the plots, don’t worry about it. We are not
looking for pretty solutions, rather to see if you are able to make
sense out of the data. Make sure the compiled html (and/or pdf) file
shows your answers completely and that they are not cut-off. Throughout
the exam, you do not need to use any LaTeX or mathematical equations.
Whenever we ask for test at some significant level, assume all
the model assumptions are satisfied.
All the answers should be clearly supported by relevant R code or based on the R output. There are many ways to provide answers.
There are 4 questions with various parts:
- Question 1: 5 parts
- Question 2: 4 parts
- Question 3: 2 parts
- Question 4: 2 parts
DO NOT spend too much time on a single question. Come back to where you stuck after you have tried all the questions.
Files needed for the midterm:
any_folder/midterm.Rmdany_folder/midterm_24.csv
Electronic Submission: Two files needed: your
.rmd file and a compiled html file. If you have trouble
submitting the files to Canvas, email them to lzhao@wharton.upenn.edu, dongwooo@wharton.upenn.edu and neil.fasching@asc.upenn.edu.
Label them with your full name. In the
Assignments section, go to the Midterm
assignment and upload your completed files.
The submission folder will be closed sharp at 9:15PM.
On Site Help: We will answer any clarification questions. We may also help out with some minor code issues. We will, however, not provide any answers as to what functions to use for example.
Raise your hand if you want to talk to one of us.
In case of emergency, here is Linda’s cell: 6106590187 (text or call her)
Background
The Dataset: For the midterm, we will continue using
the Chicago Housing dataset. Hailing from the Cook
County Assessor, where they have compiled and integrated collection
of information, meticulously gathered from a variety of distinct sources
to offer an extensive array of individual housing attributes. Beyond the
mere physical aspects, this dataset uniquely incorporates social
dimensions through spatial information or proximity variables as well as
demographic variables by census tract, amongst other fascinating
properties. Notice we have also included a categorical variable
over_500K: 1 indicates a house was sold at a
price higher than 500K and otherwise it is a
0. To be specific:
over_500K = 1if thelog_priceis greater than or equal tolog(5e+5).over_500K = 0if thelog_priceis smaller thanlog(5e+5).
| Feature | Description |
|---|---|
log_price |
Log sale price |
neighborhood |
neighborhood: A few areas in the Cook County |
beds |
Bedrooms: Number of bedrooms in the building |
bldg |
Building square feet |
fbath |
Full Baths: Number of full bathrooms, defined as having a bath or shower. If this value is missing, the default value is set to 1. |
hbath |
Half baths: Number of half baths, defined as bathrooms without a shower or bathtub. |
land |
Lot/land square feet |
rooms |
Rooms: Number of total rooms in the building (excluding baths). Not to be confused with bedrooms. |
lake_michigan_dist_ft |
Distance in feet to the Lake Michigan coastline. |
nearest_cta_stop_dist_ft |
Distance in feet to the nearest CTA (subway) stop. |
percent_employment_unemployed |
Percent of neighborhood unemployed |
percent_age_children |
Percent of neighborhood children age |
percent_education_high_school |
Percent of neighborhood in high school |
percent_income_below_poverty_level |
Percent of neighborhood below poverty line |
over_500K |
If a house is over $500k, 1: yes, 0: no |
There are two main goals for this case study:
Develop a regression model to predict the logarithmic value of house prices in a few neighborhoods of Chicago area. This model aims to capture the pricing trends and provide an accurate estimation of housing values in these areas. (We focus on this part of the analyses.) Keep housing prices in log scale throughout the exam.
Create a binary classification model to distinguish between houses valued at over $500K dollars and those valued at under $500K dollars. The goal is to effectively categorize properties into these two distinct price brackets.
To make our case study here simple and manageable in a timely
fashion, we have assembled a subset of data called:
midterm_24.csv. It includes all the variables listed above
plus a few additional variables. The name of each variable should be
self-explanatory.
Question 1. EDA
Question 1-1.
We first take a look at the data:
i. How many houses and how many variables are included in the data set?
## [1] 1291 29ii. Which areas in the Cook County are included in the data?
## [1] IRVING PARK JEFFERSON PARK LOGAN SQUARE
## Levels: IRVING PARK JEFFERSON PARK LOGAN SQUARE## [1] 3Please write your answer starting from here:
There are 1,291 houses and 29 variables included in this dataset.
We are told that the neighborhoods included are all from the Cook County above, and the data set includes information about 3 in total (Irving Park, Jefferson Park, Logan Square).
Question 1-2.
Report the minimum, median, mean and maximum of log price
(log_price), year built (yrblt), number of
bedrooms (beds) and number of full bathrooms
(fbath), respectively.
## log_price neighborhood yrblt beds
## Min. :11.7 IRVING PARK :475 Min. :1863 Min. : 1.00
## 1st Qu.:12.7 JEFFERSON PARK:282 1st Qu.:1903 1st Qu.: 3.00
## Median :13.0 LOGAN SQUARE :534 Median :1918 Median : 4.00
## Mean :13.1 Mean :1934 Mean : 3.96
## 3rd Qu.:13.4 3rd Qu.:1958 3rd Qu.: 4.00
## Max. :14.5 Max. :2018 Max. :18.00
## rooms fbath hbath tax_bill_rate
## Min. : 2.0 Min. :1.00 Min. :0.00 Min. :6.89
## 1st Qu.: 6.0 1st Qu.:1.00 1st Qu.:0.00 1st Qu.:6.91
## Median : 7.0 Median :2.00 Median :0.00 Median :6.92
## Mean : 8.1 Mean :2.08 Mean :0.38 Mean :6.91
## 3rd Qu.: 9.0 3rd Qu.:3.00 3rd Qu.:1.00 3rd Qu.:6.92
## Max. :36.0 Max. :7.00 Max. :6.00 Max. :7.27
## tax_bill_amount_total bldg lake_michigan_dist_ft
## Min. : 0 Min. : 441 Min. :10987
## 1st Qu.: 5639 1st Qu.: 1374 1st Qu.:17799
## Median : 7366 Median : 1889 Median :22481
## Mean : 8491 Mean : 2028 Mean :22885
## 3rd Qu.:10183 3rd Qu.: 2410 3rd Qu.:26402
## Max. :39346 Max. :10437 Max. :37024
## nearest_cta_stop_dist_ft percent_age_children percent_age_senior
## Min. : 184 Min. :0.0723 Min. :0.0217
## 1st Qu.:1965 1st Qu.:0.1713 1st Qu.:0.0738
## Median :2923 Median :0.2120 Median :0.0979
## Mean :3211 Mean :0.2118 Mean :0.1068
## 3rd Qu.:4323 3rd Qu.:0.2454 3rd Qu.:0.1540
## Max. :8886 Max. :0.2964 Max. :0.1806
## percent_mobility_no_move percent_mobility_moved_in_county
## Min. :0.726 Min. :0.0412
## 1st Qu.:0.820 1st Qu.:0.0792
## Median :0.862 Median :0.1142
## Mean :0.856 Mean :0.1152
## 3rd Qu.:0.908 3rd Qu.:0.1473
## Max. :0.956 Max. :0.2142
## percent_mobility_moved_from_state percent_household_family_married
## Min. :0.0000 Min. :0.227
## 1st Qu.:0.0064 1st Qu.:0.376
## Median :0.0126 Median :0.410
## Mean :0.0163 Mean :0.420
## 3rd Qu.:0.0207 3rd Qu.:0.471
## Max. :0.0663 Max. :0.629
## percent_household_nonfamily_alone percent_education_high_school
## Min. :0.131 Min. :0.030
## 1st Qu.:0.251 1st Qu.:0.147
## Median :0.285 Median :0.219
## Mean :0.293 Mean :0.207
## 3rd Qu.:0.333 3rd Qu.:0.258
## Max. :0.542 Max. :0.356
## percent_education_bachelor percent_education_graduate
## Min. :0.103 Min. :0.063
## 1st Qu.:0.232 1st Qu.:0.101
## Median :0.254 Median :0.155
## Mean :0.301 Mean :0.173
## 3rd Qu.:0.370 3rd Qu.:0.231
## Max. :0.521 Max. :0.385
## percent_income_below_poverty_level
## Min. :0.0078
## 1st Qu.:0.0704
## Median :0.0940
## Mean :0.1022
## 3rd Qu.:0.1292
## Max. :0.2531
## percent_income_household_received_snap_past_year percent_employment_unemployed
## Min. :0.0000 Min. :0.0074
## 1st Qu.:0.0418 1st Qu.:0.0258
## Median :0.0823 Median :0.0346
## Mean :0.0884 Mean :0.0373
## 3rd Qu.:0.1302 3rd Qu.:0.0505
## Max. :0.2712 Max. :0.0835
## percent_household_owner_occupied percent_household_total_occupied_w_sel_cond
## Min. :0.195 Min. :0.210
## 1st Qu.:0.410 1st Qu.:0.289
## Median :0.476 Median :0.323
## Mean :0.508 Mean :0.339
## 3rd Qu.:0.629 3rd Qu.:0.406
## Max. :0.818 Max. :0.538
## land over_500K
## Min. : 0.65 Min. :0.000
## 1st Qu.: 3.12 1st Qu.:0.000
## Median : 3.62 Median :0.000
## Mean : 3.69 Mean :0.442
## 3rd Qu.: 4.17 3rd Qu.:1.000
## Max. :10.25 Max. :1.000## [1] 11.7## [1] 13## [1] 13.1## [1] 14.5## [1] 1863## [1] 1918## [1] 1934## [1] 2018## [1] 1## [1] 4## [1] 3.96## [1] 18## [1] 1## [1] 2## [1] 2.08## [1] 7i. What is the most expensive housing price in this data set?
ii. Which year at which the oldest house was built?
Please write your answer starting from here: Respectively: log price has a mean of 13.1, median of 13, max of 14.5, min of 11.7; year built has a mean of 1934, median of 1918, max of 2018, min of 1863; number of bedrooms has a mean of 3.96, median of 4, max of 18, min of 1; number of full bathrooms has a mean of 2.08, median of 2, max of 7, min of 1.
The most expensive housing price is a log price of 14.5.
The oldest house in the data set was built in 1863.
Question 1-3.
Create back-to-back boxplots of log_price versus
neighborhood. Use no more than three sentences to describe
the comparison of the log_price by
neighborhood.
sales %>%
ggplot(aes(x = neighborhood,y = log_price,group=neighborhood, fill=neighborhood))+
geom_boxplot()+
ggtitle("Within Neighborhood Variability of Log house price")+
ylab("Log price of house")
Please write your answer starting from here: We see that, overall, Logan Square and Irving Park have similar levels of variability and ranges of housing log prices, with Logan Square sitting slightly more expensive and with a larger range than Irving Park. Jefferson Park has less variability and a lower price range than the other two neighborhoods. Jefferson Park and Logan Square have more outliers than Irving Park.
Question 1-4.
Report the ratio of houses over $500K
(i.e. over_500K = 1) in percentage for each
neighborhood.
Please write your answer starting from here:
Question 1-5.
To investigate how lake_michigan_dist_ft affects
over_500K, please draw histogram of
lake_michigan_dist_ft (distance in feet to the Lake
Michigan coastline) for each over_500k. Overlay two
histograms with different colors in one plot. Use no more than three
sentences to describe the comparison of the
lake_michigan_dist_ft by over_500K.
Hint: make sure that over_500K is a factor.
If you can’t get the desired histograms, proceed with your own plot
to answer the question of how lake_michigan_dist_ft relates
over_500K.
sales %<>%
mutate(over_500K = as.factor(over_500K))
p1 <- ggplot(sales) +
geom_histogram(aes(x = lake_michigan_dist_ft), bins = 10, fill = "blue") +
labs( title = "Histogram of Lake Michigan Distance in Feet & if price is over 500k", x = "lake mich / if price over 500k" , y = "Frequency")
plot(p1)Please write your answer starting from here:
Closer to lake michigan we see higher levels of houses over 500 K prices.
Question 2. fit1: A Linear Model for
log_price
In the following analyses, we focus on the effects of
beds, fbath, bldg,
lake_michigan_dist_ft and neighborhood on
log_price. Run a regression of log_price
vs. beds, fbath, bldg,
lake_michigan_dist_ft and neighborhood without
interactions as fit1. Report the summary of
fit1. Answering the following questions based on
fit1.
Question 2-1.
fit1 <- lm(log_price ~ beds + fbath + bldg + lake_michigan_dist_ft+ neighborhood, sales)
summary(fit1)##
## Call:
## lm(formula = log_price ~ beds + fbath + bldg + lake_michigan_dist_ft +
## neighborhood, data = sales)
##
## Residuals:
## Min 1Q Median 3Q Max
## -2.515 -0.222 0.024 0.243 1.046
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 13.32579141 0.06784266 196.42 < 0.0000000000000002
## beds -0.04300204 0.00866470 -4.96 0.0000007877253316
## fbath 0.12233390 0.01560709 7.84 0.0000000000000095
## bldg 0.00015973 0.00001611 9.92 < 0.0000000000000002
## lake_michigan_dist_ft -0.00003293 0.00000275 -11.96 < 0.0000000000000002
## neighborhoodJEFFERSON PARK 0.14706770 0.03860886 3.81 0.00015
## neighborhoodLOGAN SQUARE 0.15869622 0.02353039 6.74 0.0000000000231869
##
## (Intercept) ***
## beds ***
## fbath ***
## bldg ***
## lake_michigan_dist_ft ***
## neighborhoodJEFFERSON PARK ***
## neighborhoodLOGAN SQUARE ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.346 on 1284 degrees of freedom
## Multiple R-squared: 0.499, Adjusted R-squared: 0.497
## F-statistic: 213 on 6 and 1284 DF, p-value: <0.0000000000000002## Analysis of Variance Table
##
## Response: log_price
## Df Sum Sq Mean Sq F value Pr(>F)
## beds 1 36.0 36.0 300.3 < 0.0000000000000002 ***
## fbath 1 51.9 51.9 433.0 < 0.0000000000000002 ***
## bldg 1 12.4 12.4 103.8 < 0.0000000000000002 ***
## lake_michigan_dist_ft 1 46.1 46.1 384.7 < 0.0000000000000002 ***
## neighborhood 2 7.0 3.5 29.4 0.00000000000034 ***
## Residuals 1284 153.9 0.1
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1i. In this model is
lake_michigan_dist_ft a useful variable at .01 level?
ii. Use no more than three sentences to interpret
the lake_michigan_dist_ft effect on the
log_price. iii. What is the effect of
beds on log_price?
iv. On average, which neighborhood has the lowest
log_price controlling for all other variables?
Please write your answer starting from here: i. Yes, ’lake_michigan_dist_ft` is useful since it is significant at the .01 level because its p value is less than 0.0000000000000002 (way less than .01).
On average, for a 1% increase in the distance to lake michigan, there is a decrease in the log house price by 0.00003293. Being closer to Lake Michigan adds to the property value of a home, on average.
On average, for a 1% increase in the number of bedrooms in the building, there is a decrease in the log house price by 0.04300204.
Irving Park has the lowest
log_pricewhen we are controlling for all other variables in this model (beds, fbath, building, distance to lake michigan).
Question 2-2.
Are the housing prices same on average across from the three neighborhood in this model? Show your test at .01 significant level.
## Analysis of Variance Table
##
## Response: log_price
## Df Sum Sq Mean Sq F value Pr(>F)
## beds 1 36.0 36.0 300.3 < 0.0000000000000002 ***
## fbath 1 51.9 51.9 433.0 < 0.0000000000000002 ***
## bldg 1 12.4 12.4 103.8 < 0.0000000000000002 ***
## lake_michigan_dist_ft 1 46.1 46.1 384.7 < 0.0000000000000002 ***
## neighborhood 2 7.0 3.5 29.4 0.00000000000034 ***
## Residuals 1284 153.9 0.1
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1Please write your answer starting from here: No, we actually see that, on average, the housing prices across the three neighborhoods in this model are different because the neighborhood variable is significant at the .01 level with a p value of 0.00000000000034.
Question 2-3.
Using the fit1 model, predict the log_price
of a house with 4 bedrooms (beds = 4), 3 full bathrooms
(fbath = 3), 1,500 square feet building size
(bldg = 1500), 20,000 feet away from the Lake Michigan
coastline (lake_michigan_dist_ft = 20000), and located in
the neighborhood of JEFFERSON PARK. In
addition to the prediction, provide the 95% prediction interval.
You may use the following chunk of codes:
house <- tibble(beds = 4, fbath = 3, bldg = 1500, lake_michigan_dist_ft = 20000, neighborhood = "JEFFERSON PARK")
predict(fit1, house, interval = "predict", se.fit = TRUE, level=.95)## $fit
## fit lwr upr
## 1 13.2 12.6 13.9
##
## $se.fit
## [1] 0.0432
##
## $df
## [1] 1284
##
## $residual.scale
## [1] 0.346Please write your answer starting from here: The predicted log price of the house described is given by the fit value, which is 13.2. The 95% prediction interval for the log price of this home ranges from 12.6 to 13.9.
Question 2-4.
To check the model assumptions are met, provide relevant plots . Do you think the linear model assumptions are met? Why or why not?
Please write your answer starting from here:
From our residual plot in the model diagnostics above, we see that the residuals follow a reasonably symmetric pattern with respect to the line residual = 0 and they are relatively evenly distributed within a band and do not fan out – so we can assume that linearity and homoscedasticity are satisfied. From our Normal Q-Q plot, we can see that the observations are also reasonably symmetric about the diagonal line, but the ends of the qq plot might indicate some skewness and we might want to check on those outliers; but overall we can assume that normality is met.
Question 3. fit2: Relaxed LASSO
Use LASSO to pick up a few variables among the entire data. We
require that neighborhood is forced in all the LASSO models
as we know that LOCATION is an important factor affecting the housing
prices.
Question 3-1
Get the variables from LASSO. Here are the specifications to get the LASSO variables
fit_lasso_cv: Run LASSO with the following settings to
get the same results.
- Use
set.seed(1)to control the cross-validation - Use 15-fold cross validations
- Force
neighborhoodin all the LASSO models - Pick up the variables using
s = exp(-3.5) - Collect the LASSO variables
i. Explain why we need to take
over_500K out of the dataset in this analysis? One sentence
is enough.
ii. Show the LASSO plot
iii. How many variables are selected in addition to
neighborhood?
iv. Gather variable names selected by LASSO and
neighborhood as lasso_selected <- c(“neighborhood”,…
)
To help out we provide the following setting up chunk to get
fit_lasso_cv
sales_temp <- sales %>% select(-over_500K)
# names(sales_temp)
Y <- as.matrix(sales_temp[, 'log_price']) # extract Y
X <- model.matrix(log_price ~ ., data = sales_temp)[, -1]
force_in_indicator <- c(rep(0, 2), rep(1, ncol(X)-2))
set.seed(1)
fit.lasso <- cv.glmnet(X, Y, alpha=1, nfolds=15, intercept=T, penalty.factor = force_in_indicator)
coef_1se <- coef(fit.lasso, s=exp(-3.5))
coef_1se <- coef_1se[which(coef_1se != 0), ]Please write your answer starting from here:
i. The housing log price is already what we are predicting, it doesn’t make sense to have a version of it as an explanatory variable.
ii.
iii. We select 5 variables in addition to neighborhood, including :“bldg”, “percent_education_high_school”, “percent_education_graduate”, percent_employment_unemployed”, and “land”.
iv.
## [1] "neighborhood" "bldg"
## [3] "percent_education_high_school" "percent_education_graduate"
## [5] "percent_employment_unemployed" "land"Question 3-2
For consistency, we assume that the following variables are selected by LASSO regardless your final answers in Question 3-1.
lasso_selected <- c("neighborhood", "yrblt", "fbath", "tax_bill_amount_total", "bldg", "land", "percent_employment_unemployed")Get a relaxed LASSO fit2 using the variables specified
in lasso_selected above. Report the summary of
fit2.
dat_fit2 <- select(sales, all_of(c("log_price", lasso_selected)))
fit2 <- lm(log_price ~. , dat_fit2)
summary(fit2)##
## Call:
## lm(formula = log_price ~ ., data = dat_fit2)
##
## Residuals:
## Min 1Q Median 3Q Max
## -2.5865 -0.1621 0.0223 0.1868 1.1164
##
## Coefficients:
## Estimate Std. Error t value
## (Intercept) 9.34480689 0.45738968 20.43
## neighborhoodJEFFERSON PARK -0.14552085 0.02341860 -6.21
## neighborhoodLOGAN SQUARE 0.18430123 0.02178502 8.46
## yrblt 0.00157323 0.00023417 6.72
## fbath 0.07290875 0.01328829 5.49
## tax_bill_amount_total 0.00004262 0.00000265 16.11
## bldg 0.00004985 0.00001322 3.77
## land 0.04464215 0.00774777 5.76
## percent_employment_unemployed -3.53947417 0.53295249 -6.64
## Pr(>|t|)
## (Intercept) < 0.0000000000000002 ***
## neighborhoodJEFFERSON PARK 0.000000000697 ***
## neighborhoodLOGAN SQUARE < 0.0000000000000002 ***
## yrblt 0.000000000028 ***
## fbath 0.000000049298 ***
## tax_bill_amount_total < 0.0000000000000002 ***
## bldg 0.00017 ***
## land 0.000000010402 ***
## percent_employment_unemployed 0.000000000046 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.3 on 1282 degrees of freedom
## Multiple R-squared: 0.626, Adjusted R-squared: 0.623
## F-statistic: 268 on 8 and 1282 DF, p-value: <0.0000000000000002i. Is neighborhood a significant
variable in fit2 at .01 level?
## Analysis of Variance Table
##
## Response: log_price
## Df Sum Sq Mean Sq F value Pr(>F)
## neighborhood 2 71.4 35.7 397.6 < 0.0000000000000002
## yrblt 1 22.5 22.5 250.4 < 0.0000000000000002
## fbath 1 43.9 43.9 488.3 < 0.0000000000000002
## tax_bill_amount_total 1 46.2 46.2 514.5 < 0.0000000000000002
## bldg 1 2.1 2.1 23.6 0.000001361663
## land 1 2.3 2.3 25.3 0.000000556042
## percent_employment_unemployed 1 4.0 4.0 44.1 0.000000000046
## Residuals 1282 115.1 0.1
##
## neighborhood ***
## yrblt ***
## fbath ***
## tax_bill_amount_total ***
## bldg ***
## land ***
## percent_employment_unemployed ***
## Residuals
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1ii. Controlling for all other variables which neighborhood has the lowest housing price on average?
iii. Briefly comment on how the features are selected and how each variable is related to housing price and in what way.
Please write your answer starting from here: i. Neighborhood is a significant variable in fit2 at .01 level, it has a p value of < 0.000000000000000.
Controlling for all other variables, Jefferson Park has the lowest housing price on average.
These features are selected through the LASSO (Least Absolute Shrinkage and Selection Operator) method and by choosing to include neighborhood through force in, which gives us a sparse model. On average: a 1% increase in the year the house was built indicates a .00157 increase in the log price of the home (historic homes are worth more on average); a 1% increase in the number of full bathrooms in the home indicates a .0729 increase in the log price of the home (homes with more bathrooms are worth more on average); a 1% increase in the amount of taxes billed indicates a .00004262 increase in the log price of the home (more taxes might mean higher property values); a 1% increase in the Building square feet indicates a .0000499 increase in the log price of the home (larger homes are worth more on average); a 1% increase in the land sq ft indicates a .0446 increase in the log price of the home (homes on more land are worth more on average); a 1% increase in percentage unemployed indicates a 3.539 decrease in the log price of the home (homes in areas with lots of unemployed people are worth less, maybe due to homelessness issues, poverty, and worse schools). On average, a home in Logan Square is most expensive, then Irvine Park, then Jefferson Park is cheapest.
Question 4. fit3: Classification Model for
over_500K
We will focus on the classification problem for the binary response
variable over_500K.
Question 4-1.
Run a logistic regression of over_500K
vs. beds, fbath, bldg,
lake_michigan_dist_ft and neighborhood with no
interactions. Save the result as fit3. Report
summary(fit3).
fit3 <- glm(over_500K ~ beds+fbath+bldg+lake_michigan_dist_ft + neighborhood, sales, family = binomial(logit))
summary(fit3)##
## Call:
## glm(formula = over_500K ~ beds + fbath + bldg + lake_michigan_dist_ft +
## neighborhood, family = binomial(logit), data = sales)
##
## Coefficients:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) -0.9601078 0.5007765 -1.92 0.055 .
## beds -0.3647033 0.0746292 -4.89 0.00000102451698 ***
## fbath 0.7210565 0.1295363 5.57 0.00000002599889 ***
## bldg 0.0013454 0.0001813 7.42 0.00000000000012 ***
## lake_michigan_dist_ft -0.0001066 0.0000201 -5.29 0.00000012019186 ***
## neighborhoodJEFFERSON PARK -0.7798707 0.3462906 -2.25 0.024 *
## neighborhoodLOGAN SQUARE 0.9596572 0.1617572 5.93 0.00000000297987 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## (Dispersion parameter for binomial family taken to be 1)
##
## Null deviance: 1772.0 on 1290 degrees of freedom
## Residual deviance: 1145.5 on 1284 degrees of freedom
## AIC: 1159
##
## Number of Fisher Scoring iterations: 5i. Is lake_michigan_dist_ft a
significant variable in this model at .01 level?
ii. How to interpret the coefficient of
lake_michigan_dist_ft in this model?
Please write your answer starting from here:
Yes,
lake_michigan_dist_fta significant variable in this model at .01 level with a p value of 0.00000012019186.Log odds of a house being over 500K decreases -0.0001066 when the distance the home is from Lake Michigan is increased by 1.
Question 4-2.
Consider the house presented earlier with 4 bedrooms
(beds = 4), 3 full bathrooms (fbath = 3),
1,500 square feet building size (bldg = 1500), 20,000 feet
away from the Lake Michigan coastline
(lake_michigan_dist_ft = 20000), and located in the
neighborhood of JEFFERSON PARK. Based on the
model fit3:
i. What is the estimated probability of the house being sold over $500K?
ii. Would this house predicted to be over $500K?
Suppose that the cost of misclassifying 1 to 0
is 3 times of that of misclassifying 0 to 1
and no loss with correct predictions. We use the optimal threshold which
minimizes WMCE under this assumption.
You may use the object house defined in Question
2-3 for simplicity.
Please write your answer starting from here:
## [1] 0.686- the estimated probability the house being sold over 500K is .686
- The weighted misclassification error (may not be a number between 0 and 1) is then
\(a_{1,0}=L(Y=1, \hat Y=0)\) = 3, the loss (cost) of making an “1” to a “0” (false negative)
\(a_{0,1}=L(Y=0, \hat Y=1) = 1\), the loss of making a “0” to an “1” (false positive)
\(a_{0, 0} = a_{1, 1}=0\) (correct classification)
\[MCE=\frac{a_{1,0} \sum 1 \{\hat Y = 0 | Y=1\} + a_{0,1} \sum 1 \{\hat Y = 1 | Y=0\}}{n}\]