Linear Regression on House Price Prediction

Intro

In this report, we will create linear regression model to predict house price using House Price dataset. We want to know the corelation among the variables, especially between house price with other variables. You can download the data here.

Data Preparation

Load the dataset

library(dplyr)

## 
## Attaching package: 'dplyr'

## The following objects are masked from 'package:stats':
## 
##     filter, lag

## The following objects are masked from 'package:base':
## 
##     intersect, setdiff, setequal, union

options(scipen = 100, max.print = 1e+06)

house_data <- read.csv("House.csv")

rmarkdown::paged_table(house_data)

glimpse(house_data)

## Rows: 545
## Columns: 13
## $ price            <int> 13300000, 12250000, 12250000, 12215000, 11410000, 108…
## $ area             <int> 7420, 8960, 9960, 7500, 7420, 7500, 8580, 16200, 8100…
## $ bedrooms         <int> 4, 4, 3, 4, 4, 3, 4, 5, 4, 3, 3, 4, 4, 4, 3, 4, 4, 3,…
## $ bathrooms        <int> 2, 4, 2, 2, 1, 3, 3, 3, 1, 2, 1, 3, 2, 2, 2, 1, 2, 2,…
## $ stories          <int> 3, 4, 2, 2, 2, 1, 4, 2, 2, 4, 2, 2, 2, 2, 2, 2, 2, 4,…
## $ mainroad         <chr> "yes", "yes", "yes", "yes", "yes", "yes", "yes", "yes…
## $ guestroom        <chr> "no", "no", "no", "no", "yes", "no", "no", "no", "yes…
## $ basement         <chr> "no", "no", "yes", "yes", "yes", "yes", "no", "no", "…
## $ hotwaterheating  <chr> "no", "no", "no", "no", "no", "no", "no", "no", "no",…
## $ airconditioning  <chr> "yes", "yes", "no", "yes", "yes", "yes", "yes", "no",…
## $ parking          <int> 2, 3, 2, 3, 2, 2, 2, 0, 2, 1, 2, 2, 1, 2, 0, 2, 1, 2,…
## $ prefarea         <chr> "yes", "no", "yes", "yes", "no", "yes", "yes", "no", …
## $ furnishingstatus <chr> "furnished", "furnished", "semi-furnished", "furnishe…

We can see the data has 545 rows and 13 columns. We will use price as our target variable, which is the house price and use other variables as our predictors.

Before we go further, we can see that some categorical variable not in corect data type. Lets change it first. After that check if the data is clean, no missing values.

house_data <- house_data %>% 
  mutate_at(vars(mainroad, guestroom, basement, hotwaterheating, airconditioning, prefarea, furnishingstatus), as.factor)

colSums(is.na(house_data))

##            price             area         bedrooms        bathrooms 
##                0                0                0                0 
##          stories         mainroad        guestroom         basement 
##                0                0                0                0 
##  hotwaterheating  airconditioning          parking         prefarea 
##                0                0                0                0 
## furnishingstatus 
##                0

Great, our data is clean!!

Exploratory Data Analysis

Before create the model, lets check the correlation among the variables, especially between our target variable, price

library(GGally)

## Loading required package: ggplot2

## Registered S3 method overwritten by 'GGally':
##   method from   
##   +.gg   ggplot2

ggcorr(house_data,label = T)

## Warning in ggcorr(house_data, label = T): data in column(s) 'mainroad',
## 'guestroom', 'basement', 'hotwaterheating', 'airconditioning', 'prefarea',
## 'furnishingstatus' are not numeric and were ignored

From above, we see that the all numeric variables have positif correlation on price, which area and bathrooms are higher than others.

Modeling

First, we will create model using the numeric variables because they have positive correlation on price.

m1 <- lm(price~area+bedrooms+bathrooms+stories+parking, house_data)
summary(m1)

## 
## Call:
## lm(formula = price ~ area + bedrooms + bathrooms + stories + 
##     parking, data = house_data)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -3396744  -731825   -64056   601486  5651126 
## 
## Coefficients:
##              Estimate Std. Error t value             Pr(>|t|)    
## (Intercept) -145734.5   246634.5  -0.591               0.5548    
## area            331.1       26.6  12.448 < 0.0000000000000002 ***
## bedrooms     167809.8    82932.7   2.023               0.0435 *  
## bathrooms   1133740.2   118828.3   9.541 < 0.0000000000000002 ***
## stories      547939.8    68894.5   7.953   0.0000000000000107 ***
## parking      377596.3    66804.1   5.652   0.0000000256872725 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 1244000 on 539 degrees of freedom
## Multiple R-squared:  0.5616, Adjusted R-squared:  0.5575 
## F-statistic: 138.1 on 5 and 539 DF,  p-value: < 0.00000000000000022

From the model m1 summary, we see that all predictor have significant correlation with our target, which means the p-value < 0.05 (alpha). But, the goodness of fit (Adjusted R-squared) still low, 0.5575.

Next, lets try to create model with all variable.

m2 <- lm(price~., house_data)
summary(m2)

## 
## Call:
## lm(formula = price ~ ., data = house_data)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -2619718  -657322   -68409   507176  5166695 
## 
## Coefficients:
##                                  Estimate Std. Error t value
## (Intercept)                      42771.69  264313.31   0.162
## area                               244.14      24.29  10.052
## bedrooms                        114787.56   72598.66   1.581
## bathrooms                       987668.11  103361.98   9.555
## stories                         450848.00   64168.93   7.026
## mainroadyes                     421272.59  142224.13   2.962
## guestroomyes                    300525.86  131710.22   2.282
## basementyes                     350106.90  110284.06   3.175
## hotwaterheatingyes              855447.15  223152.69   3.833
## airconditioningyes              864958.31  108354.51   7.983
## parking                         277107.10   58525.89   4.735
## prefareayes                     651543.80  115682.34   5.632
## furnishingstatussemi-furnished  -46344.62  116574.09  -0.398
## furnishingstatusunfurnished    -411234.39  126210.56  -3.258
##                                            Pr(>|t|)    
## (Intercept)                                0.871508    
## area                           < 0.0000000000000002 ***
## bedrooms                                   0.114445    
## bathrooms                      < 0.0000000000000002 ***
## stories                         0.00000000000654763 ***
## mainroadyes                                0.003193 ** 
## guestroomyes                               0.022901 *  
## basementyes                                0.001587 ** 
## hotwaterheatingyes                         0.000141 ***
## airconditioningyes              0.00000000000000891 ***
## parking                         0.00000281718245051 ***
## prefareayes                     0.00000002888201475 ***
## furnishingstatussemi-furnished             0.691118    
## furnishingstatusunfurnished                0.001192 ** 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 1068000 on 531 degrees of freedom
## Multiple R-squared:  0.6818, Adjusted R-squared:  0.674 
## F-statistic: 87.52 on 13 and 531 DF,  p-value: < 0.00000000000000022

Using all variable as predictors, we got 0.674 Adjusted R-square.

Next, we will try to create model using step-wise regression and backward elimination method.

m3 <- step(m2,
           direction = "backward",
           trace = T)

## Start:  AIC=15144.37
## price ~ area + bedrooms + bathrooms + stories + mainroad + guestroom + 
##     basement + hotwaterheating + airconditioning + parking + 
##     prefarea + furnishingstatus
## 
##                    Df       Sum of Sq             RSS   AIC
## <none>                                605597308645000 15144
## - bedrooms          1   2851161358493 608448470003493 15145
## - guestroom         1   5937644133984 611534952778984 15148
## - mainroad          1  10006201718832 615603510363832 15151
## - basement          1  11493844019590 617091152664590 15153
## - furnishingstatus  2  16378973821075 621976282466075 15155
## - hotwaterheating   1  16759904007445 622357212652445 15157
## - parking           1  25567525473517 631164834118518 15165
## - prefarea          1  36177832317486 641775140962487 15174
## - stories           1  56298925801686 661896234446686 15191
## - airconditioning   1  72675122408683 678272431053684 15204
## - bathrooms         1 104133352147696 709730660792696 15229
## - area              1 115226555744690 720823864389690 15237

summary(m3)

## 
## Call:
## lm(formula = price ~ area + bedrooms + bathrooms + stories + 
##     mainroad + guestroom + basement + hotwaterheating + airconditioning + 
##     parking + prefarea + furnishingstatus, data = house_data)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -2619718  -657322   -68409   507176  5166695 
## 
## Coefficients:
##                                  Estimate Std. Error t value
## (Intercept)                      42771.69  264313.31   0.162
## area                               244.14      24.29  10.052
## bedrooms                        114787.56   72598.66   1.581
## bathrooms                       987668.11  103361.98   9.555
## stories                         450848.00   64168.93   7.026
## mainroadyes                     421272.59  142224.13   2.962
## guestroomyes                    300525.86  131710.22   2.282
## basementyes                     350106.90  110284.06   3.175
## hotwaterheatingyes              855447.15  223152.69   3.833
## airconditioningyes              864958.31  108354.51   7.983
## parking                         277107.10   58525.89   4.735
## prefareayes                     651543.80  115682.34   5.632
## furnishingstatussemi-furnished  -46344.62  116574.09  -0.398
## furnishingstatusunfurnished    -411234.39  126210.56  -3.258
##                                            Pr(>|t|)    
## (Intercept)                                0.871508    
## area                           < 0.0000000000000002 ***
## bedrooms                                   0.114445    
## bathrooms                      < 0.0000000000000002 ***
## stories                         0.00000000000654763 ***
## mainroadyes                                0.003193 ** 
## guestroomyes                               0.022901 *  
## basementyes                                0.001587 ** 
## hotwaterheatingyes                         0.000141 ***
## airconditioningyes              0.00000000000000891 ***
## parking                         0.00000281718245051 ***
## prefareayes                     0.00000002888201475 ***
## furnishingstatussemi-furnished             0.691118    
## furnishingstatusunfurnished                0.001192 ** 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 1068000 on 531 degrees of freedom
## Multiple R-squared:  0.6818, Adjusted R-squared:  0.674 
## F-statistic: 87.52 on 13 and 531 DF,  p-value: < 0.00000000000000022

This step-wise regression method will create model using optimum formula by evaluating the Akaike Information Criterion (AIC) number. More lowwer the AIC, the information loss will lowwer too. We get from m3 that the optimum formula is using all variable as predictors. This is same as our m2 model and the Adjusted R-squared is 0.674.

Prediction

We will try to predict price value using our m1 and m2 models. Then we will count the RMSE using the preducted values. RMSE is used to check the performance of our models.

library(MLmetrics)

## 
## Attaching package: 'MLmetrics'

## The following object is masked from 'package:base':
## 
##     Recall

m1_predict <- predict(m1, newdata = house_data)
m2_predict <- predict(m2, newdata = house_data)
RMSE(y_pred = m1_predict, y_true = house_data$price)

## [1] 1237339

RMSE(y_pred = m2_predict, y_true = house_data$price)

## [1] 1054129

Based on RMSE value, m2 model have 1054129 RMSE value, lowwer than m1 with 1237339. This means, m2 model have better performance on predicting price values.

Evaluation

hist(m2$residuals, breaks = 10)

shapiro.test(m2$residuals)

## 
##  Shapiro-Wilk normality test
## 
## data:  m2$residuals
## W = 0.95399, p-value = 0.00000000000531

plot(x = m2$fitted.values, y = m2$residuals)
abline(h = 0, col = "red")

library(lmtest)

## Warning: package 'lmtest' was built under R version 4.2.2

## Loading required package: zoo

## Warning: package 'zoo' was built under R version 4.2.2

## 
## Attaching package: 'zoo'

## The following objects are masked from 'package:base':
## 
##     as.Date, as.Date.numeric

bptest(m2)

## 
##  studentized Breusch-Pagan test
## 
## data:  m2
## BP = 68.416, df = 13, p-value = 0.000000001569

library(car)

## Loading required package: carData

## 
## Attaching package: 'car'

## The following object is masked from 'package:dplyr':
## 
##     recode

vif(m2)

##                      GVIF Df GVIF^(1/(2*Df))
## area             1.325250  1        1.151195
## bedrooms         1.369477  1        1.170246
## bathrooms        1.286621  1        1.134293
## stories          1.478055  1        1.215753
## mainroad         1.172728  1        1.082926
## guestroom        1.212838  1        1.101289
## basement         1.323050  1        1.150239
## hotwaterheating  1.041506  1        1.020542
## airconditioning  1.211840  1        1.100836
## parking          1.212837  1        1.101289
## prefarea         1.149196  1        1.072006
## furnishingstatus 1.109639  2        1.026350

Model Improvement

We have seen that our model doesnt pass normality and homoscedasticity. To fix homoscedasticity, we can try to delete the outliers.

boxplot(house_data$price)

house_data_clean <- house_data[house_data$price <= 9000000,]

boxplot(house_data_clean$area)

house_data_clean <- house_data_clean[house_data_clean$area <=10000,]

boxplot(house_data_clean$bedrooms)

house_data_clean <- house_data_clean[house_data_clean$bedrooms<=4,]

boxplot(house_data_clean$stories)

house_data_clean <- house_data_clean[house_data_clean$stories<=3,]

m_in <- lm(price~., house_data_clean)
m4 <- step(object = m_in,
           direction = "backward",
           trace = T)

## Start:  AIC=12709.64
## price ~ area + bedrooms + bathrooms + stories + mainroad + guestroom + 
##     basement + hotwaterheating + airconditioning + parking + 
##     prefarea + furnishingstatus
## 
##                    Df      Sum of Sq             RSS   AIC
## - bedrooms          1  1097043970355 364990343917111 12709
## <none>                               363893299946756 12710
## - parking           1  5129600257424 369022900204180 12714
## - basement          1  6065708694352 369959008641108 12715
## - guestroom         1  7135483002196 371028782948952 12717
## - hotwaterheating   1  8384030030852 372277329977608 12718
## - mainroad          1  9048743898739 372942043845495 12719
## - stories           1 11998423155173 375891723101929 12723
## - furnishingstatus  2 19165392347118 383058692293874 12729
## - prefarea          1 20161021519383 384054321466139 12733
## - bathrooms         1 44402687650479 408295987597235 12761
## - airconditioning   1 51026086103683 414919386050439 12768
## - area              1 59709970939889 423603270886645 12778
## 
## Step:  AIC=12709.04
## price ~ area + bathrooms + stories + mainroad + guestroom + basement + 
##     hotwaterheating + airconditioning + parking + prefarea + 
##     furnishingstatus
## 
##                    Df      Sum of Sq             RSS   AIC
## <none>                               364990343917111 12709
## - parking           1  5659704157590 370650048074701 12714
## - basement          1  6839948734190 371830292651301 12716
## - guestroom         1  6879625326173 371869969243284 12716
## - hotwaterheating   1  8187109705829 373177453622940 12717
## - mainroad          1  8414887307587 373405231224698 12718
## - furnishingstatus  2 19680926538883 384671270455994 12729
## - stories           1 19612236176766 384602580093877 12731
## - prefarea          1 20892799300138 385883143217249 12733
## - bathrooms         1 49935346112994 414925690030106 12766
## - airconditioning   1 51368670737708 416359014654819 12768
## - area              1 62524563504330 427514907421441 12780

summary(m4)

## 
## Call:
## lm(formula = price ~ area + bathrooms + stories + mainroad + 
##     guestroom + basement + hotwaterheating + airconditioning + 
##     parking + prefarea + furnishingstatus, data = house_data_clean)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -2247733  -545787   -34330   428742  3699288 
## 
## Coefficients:
##                                  Estimate Std. Error t value
## (Intercept)                     816930.07  232354.53   3.516
## area                               247.99      28.24   8.780
## bathrooms                       797173.49  101597.55   7.846
## stories                         352340.10   71652.76   4.917
## mainroadyes                     396267.50  123026.52   3.221
## guestroomyes                    366699.21  125910.52   2.912
## basementyes                     288887.93   99480.37   2.904
## hotwaterheatingyes              666956.42  209926.17   3.177
## airconditioningyes              796764.61  100118.72   7.958
## parking                         143866.21   54462.35   2.642
## prefareayes                     551189.88  108601.88   5.075
## furnishingstatussemi-furnished   20778.49  109132.79   0.190
## furnishingstatusunfurnished    -438289.65  117100.09  -3.743
##                                            Pr(>|t|)    
## (Intercept)                                0.000483 ***
## area                           < 0.0000000000000002 ***
## bathrooms                        0.0000000000000315 ***
## stories                          0.0000012318595984 ***
## mainroadyes                                0.001370 ** 
## guestroomyes                               0.003765 ** 
## basementyes                                0.003866 ** 
## hotwaterheatingyes                         0.001590 ** 
## airconditioningyes               0.0000000000000143 ***
## parking                                    0.008540 ** 
## prefareayes                      0.0000005671010761 ***
## furnishingstatussemi-furnished             0.849084    
## furnishingstatusunfurnished                0.000206 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 900600 on 450 degrees of freedom
## Multiple R-squared:  0.6135, Adjusted R-squared:  0.6032 
## F-statistic: 59.53 on 12 and 450 DF,  p-value: < 0.00000000000000022

m4_predict <- predict(object = m4,
                      newdata = house_data_clean)

RMSE(y_pred = m4_predict, y_true = house_data_clean$price)

## [1] 887871.7

hist(m4$residuals, breaks = 10)

shapiro.test(m4$residuals)

## 
##  Shapiro-Wilk normality test
## 
## data:  m4$residuals
## W = 0.96829, p-value = 0.00000001852

plot(x = m4$fitted.values, y = m4$residuals)
abline(h = 0, col = "red")

library(lmtest)
bptest(m4)

## 
##  studentized Breusch-Pagan test
## 
## data:  m4
## BP = 58.416, df = 12, p-value = 0.00000004382

Conclusion

After doing some tuning on our dataset, our final m4 model still doesnt meet all the linear regression assumptions. The Adjusted R still low on 0.6032, lowwer than m2 on 0.674. But the RMSE of m4 model is better than m2, which is 887871.7.