HW 3
NAME SURNAME
DANG AU
1. Import the dataset Apartments.xlsx
library(readxl)
ApartmentsData <- read_xlsx("./Apartments.xlsx")Description:
- Age: Age of an apartment in years
- Distance: The distance from city center in km
- Price: Price per m2
- Parking: 0-No, 1-Yes
- Balcony: 0-No, 1-Yes
2. Change categorical variables into factors. (0.5 p)
ApartmentsData$ParkingFactor <- factor(ApartmentsData$Parking,
levels = c(1, 0),
labels = c("Yes", "No"))
ApartmentsData$BalconyFactor <- factor(ApartmentsData$Balcony,
levels = c(1, 0),
labels = c("Yes", "No"))3. Test the hypothesis H0: Mu_Price = 1900 eur. What can you conclude? (1 p)
shapiro.test(ApartmentsData$Price)##
## Shapiro-Wilk normality test
##
## data: ApartmentsData$Price
## W = 0.94017, p-value = 0.0006513wilcox.test(ApartmentsData$Price, mu = 1900, correct = FALSE)##
## Wilcoxon signed rank test
##
## data: ApartmentsData$Price
## V = 2328, p-value = 0.02828
## alternative hypothesis: true location is not equal to 1900- We reject the null hypothesis as the p-value is smaller than 0.05 which suggests the true location of the population mean price is not equal to 1900 EUR.
4. Estimate the simple regression function: Price = f(Age). Save results in object fit1 and explain the estimate of regression coefficient, coefficient of correlation and coefficient of determination. (2 p)
fit1 <- lm(Price ~ Age, data = ApartmentsData)
summary(fit1)##
## Call:
## lm(formula = Price ~ Age, data = ApartmentsData)
##
## Residuals:
## Min 1Q Median 3Q Max
## -623.9 -278.0 -69.8 243.5 776.1
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 2185.455 87.043 25.108 <2e-16 ***
## Age -8.975 4.164 -2.156 0.034 *
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 369.9 on 83 degrees of freedom
## Multiple R-squared: 0.05302, Adjusted R-squared: 0.04161
## F-statistic: 4.647 on 1 and 83 DF, p-value: 0.03401cor_coef <- sqrt(summary(fit1)$r.squared)
cor_coef ## [1] 0.230255Explaining the regression coefficients:
For the intercept the coefficient is 2185.455 which means when all other variables are 0 (ceteris paribus) the price of an apartment is 2185.45 per m2.
For age the coefficient is -8.975 meaning every 1 year increase of age an apartment has its price per m2 decreases by 8.975 keeping all other variables constant (ceteris paribus).
Explaining the correlation coefficient:
- A correlation coefficient of 0.230255 suggests a weak positive correlation between price and age.
Explaining the coefficient of determination:
- This is explained through R squared which is 0.05302 which tells us the model explains 5.3% of the variation in price.
5. Show the scateerplot matrix between Price, Age and Distance. Based on the matrix determine if there is potential problem with multicolinearity. (0.5 p)
library(car)## Loading required package: carDatascatterplotMatrix(ApartmentsData, smooth = FALSE)
scatterplotMatrix(~ Price + Age + Distance, data = ApartmentsData, smooth = FALSE)
- Based on the matrix age and distance do not seem to have any strong linear relationships or patterns with each other suggesting that multicollinearity will not be an issue.
6. Estimate the multiple regression function: Price = f(Age, Distance). Save it in object named fit2.
fit2 <- lm(Price ~ Age + Distance, data = ApartmentsData)
summary(fit2)##
## Call:
## lm(formula = Price ~ Age + Distance, data = ApartmentsData)
##
## Residuals:
## Min 1Q Median 3Q Max
## -603.23 -219.94 -85.68 211.31 689.58
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 2460.101 76.632 32.10 < 2e-16 ***
## Age -7.934 3.225 -2.46 0.016 *
## Distance -20.667 2.748 -7.52 6.18e-11 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 286.3 on 82 degrees of freedom
## Multiple R-squared: 0.4396, Adjusted R-squared: 0.4259
## F-statistic: 32.16 on 2 and 82 DF, p-value: 4.896e-117. Chech the multicolinearity with VIF statistics. Explain the findings. (1 p)
vif(fit2)## Age Distance
## 1.001845 1.001845- The vif statistics suggest there is no significant multicollinearity as it close to 1, VIF values that are greater than 5-10 would be considered an indication of multicollinearity.
8. Calculate standardized residuals and Cooks Distances for model fit2. Remove any potentially problematic units (outliers or units with high influence). (2 p)
ApartmentsData$StdResid <- round(rstandard(fit2), 3)
ApartmentsData$CooksD <- round(cooks.distance(fit2), 3)
head(ApartmentsData[order(ApartmentsData$StdResid),], 3) ## # A tibble: 3 × 9
## Age Distance Price Parking Balcony ParkingFactor BalconyFactor StdResid CooksD
## <dbl> <dbl> <dbl> <dbl> <dbl> <fct> <fct> <dbl> <dbl>
## 1 7 2 1760 0 1 No Yes -2.15 0.066
## 2 12 14 1650 0 1 No Yes -1.50 0.013
## 3 12 14 1650 0 0 No No -1.50 0.013head(ApartmentsData[order(-ApartmentsData$CooksD),], 6) ## # A tibble: 6 × 9
## Age Distance Price Parking Balcony ParkingFactor BalconyFactor StdResid CooksD
## <dbl> <dbl> <dbl> <dbl> <dbl> <fct> <fct> <dbl> <dbl>
## 1 5 45 2180 1 1 Yes Yes 2.58 0.32
## 2 43 37 1740 0 0 No No 1.44 0.104
## 3 2 11 2790 1 0 Yes No 2.05 0.069
## 4 7 2 1760 0 1 No Yes -2.15 0.066
## 5 37 3 2540 1 1 Yes Yes 1.58 0.061
## 6 40 2 2400 0 1 No Yes 1.09 0.038library(dplyr)##
## Attaching package: 'dplyr'## The following object is masked from 'package:car':
##
## recode## The following objects are masked from 'package:stats':
##
## filter, lag## The following objects are masked from 'package:base':
##
## intersect, setdiff, setequal, unionApartmentsData <- ApartmentsData %>%
filter(!CooksD == "0.32")
ApartmentsData <- ApartmentsData %>%
filter(!CooksD == "0.104")
#Refitting model with excluded units
fit2 <- lm(Price ~ Age + Distance, data = ApartmentsData)
summary(fit2)##
## Call:
## lm(formula = Price ~ Age + Distance, data = ApartmentsData)
##
## Residuals:
## Min 1Q Median 3Q Max
## -627.27 -212.96 -46.23 205.05 578.98
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 2490.112 76.189 32.684 < 2e-16 ***
## Age -7.850 3.244 -2.420 0.0178 *
## Distance -23.945 2.826 -8.473 9.53e-13 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 273.5 on 80 degrees of freedom
## Multiple R-squared: 0.4968, Adjusted R-squared: 0.4842
## F-statistic: 39.49 on 2 and 80 DF, p-value: 1.173e-129. Check for potential heteroskedasticity with scatterplot between standarized residuals and standrdized fitted values. Explain the findings. (1 p)
ApartmentsData$StdFitted <- scale(fit2$fitted.values)
library(car)
scatterplot(y = ApartmentsData$StdResid, x = ApartmentsData$StdFitted,
ylab = "Standardized residuals",
xlab = "Standardized fitted values",
boxplots = FALSE,
regLine = FALSE,
smooth = FALSE)
- The scatterplot shows a constant increase in variance which could be an indicator of significant heteroskedasticity
10. Are standardized residuals ditributed normally? Show the graph and formally test it. Explain the findings. (1 p)
hist(ApartmentsData$StdResid,
xlab = "Standardized residuals",
ylab = "Frequency",
main = "Histogram of standardized residuals")
shapiro.test(ApartmentsData$StdResid)##
## Shapiro-Wilk normality test
##
## data: ApartmentsData$StdResid
## W = 0.94963, p-value = 0.002636Based on the graph alone the residuals do not seem to be distributed normally.
The shapiro test is our formal test and it gave a p-value of 0.002636 which is smaller than 0.05 and therefore we can reject the null hypothesis and conclude there is a significant indication that the standard residuals are not distributed normally.
11. Estimate the fit2 again without potentially excluded units and show the summary of the model. Explain all coefficients. (2 p)
#Have to reimport data and refactor to estimate the dataset without the exlcuded units
ApartmentsData <- read_xlsx("./Apartments.xlsx")
ApartmentsData$ParkingFactor <- factor(ApartmentsData$Parking,
levels = c(1, 0),
labels = c("Yes", "No"))
ApartmentsData$BalconyFactor <- factor(ApartmentsData$Balcony,
levels = c(1, 0),
labels = c("Yes", "No"))
fit2 <- lm(Price ~ Age + Distance, data = ApartmentsData)
summary(fit2)##
## Call:
## lm(formula = Price ~ Age + Distance, data = ApartmentsData)
##
## Residuals:
## Min 1Q Median 3Q Max
## -603.23 -219.94 -85.68 211.31 689.58
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 2460.101 76.632 32.10 < 2e-16 ***
## Age -7.934 3.225 -2.46 0.016 *
## Distance -20.667 2.748 -7.52 6.18e-11 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 286.3 on 82 degrees of freedom
## Multiple R-squared: 0.4396, Adjusted R-squared: 0.4259
## F-statistic: 32.16 on 2 and 82 DF, p-value: 4.896e-11For the intercept the coefficient is 2460.101 which means when all other variables are 0 (ceteris paribus) the price per m2 of an apartment is 2460.101 per m2.
For age the coefficient is -7.934 meaning every 1 year increase of age an apartment has, its price per m2 decreases by 7.934 ceteris paribus.
-The coefficient for distance is -20.667 meaning that for every 1 km increase in distance from the city centre the price per m2 decreases by 20.667
12. Estimate the linear regression function Price = f(Age, Distance, Parking and Balcony). Be careful to correctly include categorical variables. Save the object named fit3.
fit3 <- lm(Price ~ Age + Distance + ParkingFactor + BalconyFactor , data = ApartmentsData)13. With function anova check if model fit3 fits data better than model fit2. (1 p)
anova(fit2, fit3)## Analysis of Variance Table
##
## Model 1: Price ~ Age + Distance
## Model 2: Price ~ Age + Distance + ParkingFactor + BalconyFactor
## Res.Df RSS Df Sum of Sq F Pr(>F)
## 1 82 6720983
## 2 80 5991088 2 729894 4.8732 0.01007 *
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1- As the p-value for the ANOVA test is 0.01007 which is smaller than 0.05 we reject the null hypothesis and therefore conclude that adding more variables to our model does significantly improve the fit of the model to our data.
14. Show the results of fit3 and explain regression coefficient for both categorical variables. Can you write down the hypothesis which is being tested with F-statistics, shown at the bottom of the output? (2 p)
summary(fit3)##
## Call:
## lm(formula = Price ~ Age + Distance + ParkingFactor + BalconyFactor,
## data = ApartmentsData)
##
## Residuals:
## Min 1Q Median 3Q Max
## -459.92 -200.66 -57.48 260.08 594.37
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 2499.770 80.645 30.997 < 2e-16 ***
## Age -6.799 3.110 -2.186 0.03172 *
## Distance -18.045 2.758 -6.543 5.28e-09 ***
## ParkingFactorNo -196.168 62.868 -3.120 0.00251 **
## BalconyFactorNo -1.935 60.014 -0.032 0.97436
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 273.7 on 80 degrees of freedom
## Multiple R-squared: 0.5004, Adjusted R-squared: 0.4754
## F-statistic: 20.03 on 4 and 80 DF, p-value: 1.849e-11For ParkingFactorNo the coefficient is -196.168 this means that in comparison to apartments that have a parking space apartments that do not have a parking space decreases price per m2 by 196.168. This variable is significant at the 0.01 level.
For Balcony FactorNo the coefficient is -1.935 this means that in comparison to apartments that have a balcony apartments that do not have a balcony decrease price per m2 by 1.935. This variable is not significant and cannot be concluded to have a significant effect on price per m2.
For the F-statistic - H0: The coefficients of ALL the independent variables in the model are equal to zero and together have no effect on the dependent variable.𝛽1 = 𝛽2 = ⋯ = 𝛽k = 0
- H1: Atleast one of the coefficients of ALL the independent variables in the model is not equal to zero and together do have an effect on the dependent variable. At least one 𝛽 is different from 0
15. Save fitted values and calculate the residual for apartment ID2. (1 p)
ApartmentsData$Fitted <- fitted.values(fit3)
ApartmentsData$Residuals <- residuals(fit3)
head(ApartmentsData[colnames(ApartmentsData) %in% c("Age", "Distance", "Fitted", "Residuals")]) ## # A tibble: 6 × 4
## Age Distance Fitted Residuals
## <dbl> <dbl> <dbl> <dbl>
## 1 7 28 1751. -111.
## 2 18 1 2357. 443.
## 3 7 28 1749. -88.8
## 4 28 29 1590. 260.
## 5 18 18 2053. -413.
## 6 28 12 1897. -127.- The residuals for apartment with the ID2 is 442.58893 ~ 443.