Zillow’s Home Value Prediction (Zestimate)

Predicting error to improve the Zestimate

Introduction

The purpose of this project is to predict the log error of Zillow’s Zestimate at six potential sale times in 2016 and 2017 using time series forecasting models. Zillow made all 2016 home sales data from three counties available on Kaggle including all fields in Table 1. Predicting error rates over time will help Zillow to improve their model over time and give consumers better information as they buy and sell the largest assets they’ll likely ever own.

R code for all calculations and visuals can be found here

Column Name	Description
'airconditioningtypeid'	Type of cooling system present in the home (if any)
'architecturalstyletypeid'	Architectural style of the home (i.e. ranch, colonial, split-level, etc )
'basementsqft'	Finished living area below or partially below ground level
'bathroomcnt'	Number of bathrooms in home including fractional bathrooms
'bedroomcnt'	Number of bedrooms in home
'buildingqualitytypeid'	Overall assessment of condition of the building from best (lowest) to worst (highest)
'buildingclasstypeid'	The building framing type (steel frame, wood frame, concrete/brick)
'calculatedbathnbr'	Number of bathrooms in home including fractional bathroom
'decktypeid'	Type of deck (if any) present on parcel
'threequarterbathnbr'	Number of 3/4 bathrooms in house (shower + sink + toilet)
'finishedfloor1squarefeet'	Size of the finished living area on the first (entry) floor of the home
'calculatedfinishedsquarefeet'	Calculated total finished living area of the home
'finishedsquarefeet6'	Base unfinished and finished area
'finishedsquarefeet12'	Finished living area
'finishedsquarefeet13'	Perimeter living area
'finishedsquarefeet15'	Total area
'finishedsquarefeet50'	Size of the finished living area on the first (entry) floor of the home
'fips'	Federal Information Processing Standard code - see https://en.wikipedia.org/wiki/FIPS_county_code for more details
'fireplacecnt'	Number of fireplaces in a home (if any)
'fireplaceflag'	Is a fireplace present in this home
'fullbathcnt'	Number of full bathrooms (sink, shower + bathtub, and toilet) present in home
'garagecarcnt'	Total number of garages on the lot including an attached garage
'garagetotalsqft'	Total number of square feet of all garages on lot including an attached garage
'hashottuborspa'	Does the home have a hot tub or spa
'heatingorsystemtypeid'	Type of home heating system
'latitude'	Latitude of the middle of the parcel multiplied by 10e6
'longitude'	Longitude of the middle of the parcel multiplied by 10e6
'lotsizesquarefeet'	Area of the lot in square feet
'numberofstories'	Number of stories or levels the home has
'parcelid'	Unique identifier for parcels (lots)
'poolcnt'	Number of pools on the lot (if any)
'poolsizesum'	Total square footage of all pools on property
'pooltypeid10'	Spa or Hot Tub
'pooltypeid2'	Pool with Spa/Hot Tub
'pooltypeid7'	Pool without hot tub
'propertycountylandusecode'	County land use code i.e. it's zoning at the county level
'propertylandusetypeid'	Type of land use the property is zoned for
'propertyzoningdesc'	Description of the allowed land uses (zoning) for that property
'rawcensustractandblock'	Census tract and block ID combined - also contains blockgroup assignment by extension
'censustractandblock'	Census tract and block ID combined - also contains blockgroup assignment by extension
'regionidcounty'	County in which the property is located
'regionidcity'	City in which the property is located (if any)
'regionidzip'	Zip code in which the property is located
'regionidneighborhood'	Neighborhood in which the property is located
'roomcnt'	Total number of rooms in the principal residence
'storytypeid'	Type of floors in a multi-story house (i.e. basement and main level, split-level, attic, etc.). See tab for details.
'typeconstructiontypeid'	What type of construction material was used to construct the home
'unitcnt'	Number of units the structure is built into (i.e. 2 = duplex, 3 = triplex, etc...)
'yardbuildingsqft17'	Patio in yard
'yardbuildingsqft26'	Storage shed/building in yard
'yearbuilt'	The Year the principal residence was built
'taxvaluedollarcnt'	The total tax assessed value of the parcel
'structuretaxvaluedollarcnt'	The assessed value of the built structure on the parcel
'landtaxvaluedollarcnt'	The assessed value of the land area of the parcel
'taxamount'	The total property tax assessed for that assessment year
'assessmentyear'	The year of the property tax assessment
'taxdelinquencyflag'	Property taxes for this parcel are past due as of 2015
'taxdelinquencyyear'	Year for which the unpaid propert taxes were due

Exploratory Data Analysis

As a first exploratory step, the number of null values for each variable are distplayed in Figure 0.1. Many of the variables in the data set have significant numbers of missing values. This could be due to lack of self-reporting or differing standards for reporting from various government data sources employed by Zillow. A decision rule of 75% completion is used for retention of a variable. Missing values in the remaining columns are imputed using boosted regression and classification trees.

Joining, by = "parcelid"

From here I create dummy variables for all categorical values and then summarise all variables so we have average daily values rather than multiple values per day. Some variables (like city id) are removed in this step because we have more granular data (like zip codes) that measure the same thing.

 iter imp variable
  1   1  calculatedbathnbr  calculatedfinishedsquarefeet  lotsizesquarefeet  yearbuilt  structuretaxvaluedollarcnt  taxvaluedollarcnt  landtaxvaluedollarcnt  taxamount
  1   2  calculatedbathnbr  calculatedfinishedsquarefeet  lotsizesquarefeet  yearbuilt  structuretaxvaluedollarcnt  taxvaluedollarcnt  landtaxvaluedollarcnt  taxamount
  1   3  calculatedbathnbr  calculatedfinishedsquarefeet  lotsizesquarefeet  yearbuilt  structuretaxvaluedollarcnt  taxvaluedollarcnt  landtaxvaluedollarcnt  taxamount
  1   4  calculatedbathnbr  calculatedfinishedsquarefeet  lotsizesquarefeet  yearbuilt  structuretaxvaluedollarcnt  taxvaluedollarcnt  landtaxvaluedollarcnt  taxamount
  1   5  calculatedbathnbr  calculatedfinishedsquarefeet  lotsizesquarefeet  yearbuilt  structuretaxvaluedollarcnt  taxvaluedollarcnt  landtaxvaluedollarcnt  taxamount

The logerror rates almost represent a perfect white noise time series over the course of 2016.

Model Development and Testing

Three models are attempted for this project. The first model is a multiple linear regression model using variables selected with a backwards variable selection method. The second model is a simple exponential smoothing model. The final model uses the ARIMA method. Models are evaluated by MAE, ME, and RMSE as well as the average error in the six out of sample test points tests as part of Zillow’s Kaggle competition.

Multiple Linear Regression Model

A multiple linear regression model uses linear relationships between predictors and a target value to predict future unseen values. The model takes the form \(y_t=\beta_0 + \beta_1 x_{1,t}+ \beta_2 x_{2,t}+ \beta_3 x_{3,t}+ \beta_i x_{i,t}+\varepsilon_t.\) for i predictor variables.

The backwards variable selection method employed here starts with all possible predictors in the data set and whittles them away iteratively after testing to see whether or not the inclusion of a variable minimizes the Akaike Information Criterion (AIC). The algorithm’s output is the best possible model for each potential number of predictor variables to be included as well as various model health metrics for each potential model. Figure X plots the first 60 or so potential models against residual sum of squares (RSS) and adjusted r-squared (Adj. R-squared), with each variable indexed against it’s maximum value vor purposes of comparing them on the same plot [^2]. The 3-variable model is the best possible linear combination of predictor variables according to this method since that value maximizes Adj. R-squared and is near the point of diminishing marginal returns for RSS.

In implementing the model chosen through backwards variable selection, we see fitted values that are more conservative than the actual LogError values observed in the data. The model only explains 22% of the variance in the data and several predictor variables with correlation coefficients indistinguishable frmo zero, although for predictive purposes that is less important than optimizing AIC or out of sample error rates [^3].

[1] "calculatedfinishedsquarefeet" "structuretaxvaluedollarcnt"   "taxvaluedollarcnt"           

Where the model residuals depart from the normal distribution most notably is a heavy negative tail. In general, the linear model is more conservative than the data - hewing closer to the overall average value than.

Simple Exponential Smoothing

Exponential smoothing methods produce weighted averages of past values controlled by a smoothing parameter \(\alpha\) in order to optimize the level \(\ell_t\). A smaller value for \(\alpha\) means more weight is applied to older observations while a larger value applies more weight to more recent values[^3].

The optimal simple exponential smoothing function for the zillow data uses \(\ell_{t} = .0133 y_{t} + (1 -.0133)\ell_{t-1}\) and an initial \(\ell\) value of .0101.

The smoothed predicted values are mostly clustered alightly above zero (around .01). The residuals are not exactly normally distributed because they cluster too closely around 0. Additionally, there are a few outlying values (moreso than the MLR model).

ARIMA Model

The AutoRegressive Integrated Moving Average (ARIMA) model is a regression model acounting for lagged error terms and lagged predicted values of the outcome of interest. The model is presented in the form ARIMA(p,d,q) where p represents the autoregressive formula, d indicates the amount of differencing applied to the time series, and q describes the moving average component[^3].

The auto.arima() R function, which uses a step-wise variation of the Hyndman-Khandakar model selection method [^4] to optimize for AIC_c, returned a first-order autoregressive model as the optimal model for the data. Essentially, the method uses a single period lag as a regressor to make predictions about future outcomes.

The fitted values of the ARIMA model follow a similar pattern to the linear model. Fitted values follow a similar, if more conservative, pattern as the observed values and, while there are some outlying residuals, the bulk of the values are clustered around zero.

Model Evaluation

As stated earlier, models are evaluated by MAE, ME, and RMSE as well as the average error in the six out of sample test points tests as part of Zillow’s Kaggle competition.

The average error among all models was fairly small. Differences in RMSE and MAE were also fairly small. Simple Exponential Smoothing performed best on MAE which makes sense because the pattern of the fitted values makes it difficult to imagine that model overfitting the data. ARIMA and MLR perform similarly across the board.

Finally, the big test is how well the predictions compared to actual values on six prescribed months that Zillow is interested in checking: October 2016 (201610), November 2016 (201611), December 2016 (201612), October 2017 (201710), November 2017 (201711), and December 2017 (201712). The predicted values for the first day of each 2016 month are below:

Scale for 'x' is already present. Adding another scale for 'x', which will replace the existing scale.

Conclusion

The goal of this project was to predict Zillow Zestimate LogError values. Three models were trained and tested for their predictive accuracy - multiple linear regression, simple exponential smoothing, and ARIMA. The ARIMA and multiple regression models appeared to perform best and teh ARIMA model was used to make predictions about out of sample data points.

Citations

[^1:] Time Series Analysis and Its Application with R examples [^2:] James, G., Witten, D., Hastie, T., & Tibshirani, R. (2017). An Introduction to Statistical Learning: with Applications in R. New York: Springer. [^3:] Hyndman (http://otexts.org/fpp2/Regr-LSprinciple.html) [^4:] Hyndman, Rob J, and Yeasmin Khandakar. 2008. “Automatic Time Series Forecasting: The Forecast Package for R.” Journal of Statistical Software 27 (1): 1-22.