We used housing market data sourced from Zillow, the leading real estate marketplace, to predict future housing prices, investigate housing hotspots and analyze volatility in certain regions. We are using housing data calculated by zip code with the following variables:

• Region ID: Unique id for each zip
  
• SizeRank: Rank based on region size in descending order
  
• RegionName: The zip code of each region
  
• StateName: Abbreviated state name
  
• State: Abbreviated state name (yes there are 2 of these)
  
• City: City name
  
• Metro: The metropolitan area the zip is a part of
  
• CountyName: The county the zip Is a part of

The final variables are the date columns. There are 290 date columns with the Zillow Home Value Index (ZHVI) of each zip code, from 1/31/2000 to 2/29/2024 with a ZHVI calculated each month.

The ZHVI we are using is a measure of the typical home value, including condominium and co-op values, smoothed and seasonally adjusted, representing the typical home value as a weighted average of the middle third percentile of homes, or more simply. This data is used by Zillow for most consumer facing presentations of a region’s average home value.

The data comes from Zillow in a csv which needs to be imported and cleaned.

We will not be using the regionID, SizeRank, RegionType or StateName variables, so we must filter those out.

ZHVI_filtered = ZHVI_full %>% select(!c(RegionID, SizeRank, RegionType, StateName))
head(ZHVI_filtered[c(1:7)]) # stop it from spewing all dates

##   RegionName State     City                                 Metro
## 1      77494    TX     Katy  Houston-The Woodlands-Sugar Land, TX
## 2       8701    NJ Lakewood New York-Newark-Jersey City, NY-NJ-PA
## 3      77449    TX     Katy  Houston-The Woodlands-Sugar Land, TX
## 4      11368    NY New York New York-Newark-Jersey City, NY-NJ-PA
## 5      77084    TX  Houston  Houston-The Woodlands-Sugar Land, TX
## 6      79936    TX  El Paso                           El Paso, TX
##         CountyName X2000.01.31 X2000.02.29
## 1 Fort Bend County   210077.00   210259.33
## 2     Ocean County   137227.27   137793.64
## 3    Harris County   103358.70   103372.98
## 4    Queens County   146558.05   148014.69
## 5    Harris County   102016.82   101974.04
## 6   El Paso County    89584.73    89569.51

The data is currently in wide format, making it difficult to use, we must change it into long format

## # A tibble: 6 × 7
##   RegionName State City  Metro                           CountyName Date    ZHVI
##        <int> <chr> <chr> <chr>                           <chr>      <chr>  <dbl>
## 1      77494 TX    Katy  Houston-The Woodlands-Sugar La… Fort Bend… X200… 2.10e5
## 2      77494 TX    Katy  Houston-The Woodlands-Sugar La… Fort Bend… X200… 2.10e5
## 3      77494 TX    Katy  Houston-The Woodlands-Sugar La… Fort Bend… X200… 2.11e5
## 4      77494 TX    Katy  Houston-The Woodlands-Sugar La… Fort Bend… X200… 2.12e5
## 5      77494 TX    Katy  Houston-The Woodlands-Sugar La… Fort Bend… X200… 2.12e5
## 6      77494 TX    Katy  Houston-The Woodlands-Sugar La… Fort Bend… X200… 2.12e5

The names of the collection dates aren’t formatted nicely and start with an X, so we must remove that and turn it into a standard format (dd/mm/yy)

ZHVI_filtered_M <- ZHVI_filtered_long %>%
  mutate(Date = sub("X", "", Date)) %>%
  mutate(Date = as.Date(Date,"%Y.%m.%d"))
head(ZHVI_filtered_M)

## # A tibble: 6 × 7
##   RegionName State City  Metro                      CountyName Date         ZHVI
##        <int> <chr> <chr> <chr>                      <chr>      <date>      <dbl>
## 1      77494 TX    Katy  Houston-The Woodlands-Sug… Fort Bend… 2000-01-31 2.10e5
## 2      77494 TX    Katy  Houston-The Woodlands-Sug… Fort Bend… 2000-02-29 2.10e5
## 3      77494 TX    Katy  Houston-The Woodlands-Sug… Fort Bend… 2000-03-31 2.11e5
## 4      77494 TX    Katy  Houston-The Woodlands-Sug… Fort Bend… 2000-04-30 2.12e5
## 5      77494 TX    Katy  Houston-The Woodlands-Sug… Fort Bend… 2000-05-31 2.12e5
## 6      77494 TX    Katy  Houston-The Woodlands-Sug… Fort Bend… 2000-06-30 2.12e5

We’re almost done cleaning the data, but there are a few area codes which did not exist as far back as the data goes, like 77573, so we’ll have to remove those.

The data has now been cleaned and reorganized, making it ready for analysis with 7 variables and ~5.6M observations.

head(ZHVI_filtered_M[ZHVI_filtered_M$RegionName == 77573, ])

## # A tibble: 6 × 7
##   RegionName State City        Metro                 CountyName Date        ZHVI
##        <int> <chr> <chr>       <chr>                 <chr>      <date>     <dbl>
## 1      77573 TX    League City Houston-The Woodland… Galveston… 2000-01-31    NA
## 2      77573 TX    League City Houston-The Woodland… Galveston… 2000-02-29    NA
## 3      77573 TX    League City Houston-The Woodland… Galveston… 2000-03-31    NA
## 4      77573 TX    League City Houston-The Woodland… Galveston… 2000-04-30    NA
## 5      77573 TX    League City Houston-The Woodland… Galveston… 2000-05-31    NA
## 6      77573 TX    League City Houston-The Woodland… Galveston… 2000-06-30    NA

ZHVI_filtered_M <- ZHVI_filtered_M %>%
  filter(ZHVI != is.na(ZHVI))
head(ZHVI_filtered_M[ZHVI_filtered_M$RegionName == 77573, ])

## # A tibble: 6 × 7
##   RegionName State City        Metro                CountyName Date         ZHVI
##        <int> <chr> <chr>       <chr>                <chr>      <date>      <dbl>
## 1      77573 TX    League City Houston-The Woodlan… Galveston… 2001-01-31 1.53e5
## 2      77573 TX    League City Houston-The Woodlan… Galveston… 2001-02-28 1.53e5
## 3      77573 TX    League City Houston-The Woodlan… Galveston… 2001-03-31 1.53e5
## 4      77573 TX    League City Houston-The Woodlan… Galveston… 2001-04-30 1.53e5
## 5      77573 TX    League City Houston-The Woodlan… Galveston… 2001-05-31 1.53e5
## 6      77573 TX    League City Houston-The Woodlan… Galveston… 2001-06-30 1.53e5

ZHVI_max = ZHVI_filtered_M %>%
  filter(Date >= '2024-02-01') %>%
  arrange(desc(ZHVI))
head(ZHVI_max)

## # A tibble: 6 × 7
##   RegionName State City          Metro              CountyName Date         ZHVI
##        <int> <chr> <chr>         <chr>              <chr>      <date>      <dbl>
## 1      94027 CA    Atherton      San Francisco-Oak… San Mateo… 2024-02-29 7.48e6
## 2      33109 FL    Miami Beach   Miami-Fort Lauder… Miami-Dad… 2024-02-29 5.82e6
## 3      90210 CA    Beverly Hills Los Angeles-Long … Los Angel… 2024-02-29 5.38e6
## 4      11962 NY    Sagaponack    New York-Newark-J… Suffolk C… 2024-02-29 5.28e6
## 5      94022 CA    Los Altos     San Jose-Sunnyval… Santa Cla… 2024-02-29 4.76e6
## 6      92657 CA    Newport Beach Los Angeles-Long … Orange Co… 2024-02-29 4.60e6

The 85253 zip code in Paradise valley holds the crown as having the most expensive average home value at $2.7M, with Scottsdale holding second and third place at over a million dollars cheaper. All of these Zips being in the Pheonix Metropolitain Area.

ZHVI_max_az = ZHVI_filtered_M %>%
  filter(Date >= '2024-02-01' & State == "AZ") %>%
  arrange(desc(ZHVI))
head(ZHVI_max_az)

## # A tibble: 6 × 7
##   RegionName State City            Metro            CountyName Date         ZHVI
##        <int> <chr> <chr>           <chr>            <chr>      <date>      <dbl>
## 1      85253 AZ    Paradise Valley Phoenix-Mesa-Ch… Maricopa … 2024-02-29 2.79e6
## 2      85262 AZ    Scottsdale      Phoenix-Mesa-Ch… Maricopa … 2024-02-29 1.46e6
## 3      85266 AZ    Scottsdale      Phoenix-Mesa-Ch… Maricopa … 2024-02-29 1.26e6
## 4      85264 AZ    Fort McDowell   Phoenix-Mesa-Ch… Maricopa … 2024-02-29 1.21e6
## 5      85255 AZ    Scottsdale      Phoenix-Mesa-Ch… Maricopa … 2024-02-29 1.21e6
## 6      85377 AZ    Carefree        Phoenix-Mesa-Ch… Maricopa … 2024-02-29 1.18e6

We are taking the mean value of homes over the past 24 years and grouping them by month to find trends in monthly home value in Arizona.

Although the data is seasonally adjusted, there is still a trend in the data showing house prices rising and falling depending on the month. The year starts off strong, rising until a steep decline in march, with home values gradually increasing throughout the year.

Our Zillow data set can allow us to get in depth information on the housing market in Maricopa County

Let’s start by filtering the data to target Maricopa County.

AZ_Maricopa_data <- ZHVI_filtered_M %>% 
  filter(State == "AZ" & CountyName == "Maricopa County")

Let’s start with the basics!

What has been the median housing price for each city in maricopa county last month…

# Calculate median housing prices for each city
AZ_Maricopa_data_median_Feb_2024 <- AZ_Maricopa_data %>%
  filter(Date == as.Date("2024-02-29")) %>%
  group_by(City) %>%
  summarise(Median_ZHVI = median(ZHVI, na.rm = TRUE))

This snippet of code dives into the columns of our maricopa subset and adds up the prices of various zip codes to find the median price.

Let’s investigate further….

What are the top and bottom 5 cities in Maricopa County for median prices?

# Identify the top 5 highest and lowest median prices
top_cities <- head(AZ_Maricopa_data_median_Feb_2024$City[order(-AZ_Maricopa_data_median_Feb_2024$Median_ZHVI)], 5)
bottom_cities <- head(AZ_Maricopa_data_median_Feb_2024$City[order(AZ_Maricopa_data_median_Feb_2024$Median_ZHVI)], 5)

This is a compact way of grabbing the top and bottom 5 cities in maricopa county. The “-” before our data frame allows us to access the dataframe in descending order.

AZ_Maricopa_data_median_Feb_2024$Color <- ifelse(AZ_Maricopa_data_median_Feb_2024$City %in% top_cities, 'Top 5', ifelse(AZ_Maricopa_data_median_Feb_2024$City %in% bottom_cities, 'Bottom 5', 'Other'))

This code snippet is going to add a column to designate whether a city is top or bottom 5 for presentation purposes.

Let’s investigate how zipcodes are related to prices! Let’s once again use data from February 2024.

AZ_Maricopa_Zipcode_Price_cor <-AZ_Maricopa_data %>%
  filter(Date == as.Date("2024-02-29"))

# Calculate the correlation coefficient
correlation <- cor(AZ_Maricopa_Zipcode_Price_cor$RegionName, AZ_Maricopa_Zipcode_Price_cor$ZHVI)

# Print the correlation coefficient
print(paste("The correlation between RegionName and ZHVI is:", correlation))

## [1] "The correlation between RegionName and ZHVI is: 0.0984620031120423"

Let’s now rewind time!

What has been the trend in housing prices in Maricopa County since 2000?

AZ_Maricopa_Cities_median <- AZ_Maricopa_data %>%
  group_by(City, Date) %>%
  summarise(Median_ZHVI = median(ZHVI, na.rm = TRUE))

## `summarise()` has grouped output by 'City'. You can override using the
## `.groups` argument.

The Median Zillow Home Value Index (ZHVI) of cities in Maricopa County has seen a significant increase from 2000 to 2024. A majority of cities hover around $500,000. Some outliers make our graph a little harder to understand.

In order to simplify we need to generalize.

Here is our line graph for the entirety of Maricopa County over the years! We can see that the median price right now sits around $500,000 which is what we saw with a majority of cities in the previous breakdown.

We can also see the historic drop from the 2007-2008 Housing bubble and the road to recovery aftwerwards.

We can also see a rapid increase during the COVID pandemic.

Let’s take a closer look at these increases and decreases over the years.

We will need to calculate our percent changes and add them to a new container.

# Calculate the percentage change in median housing prices
AZ_Maricopa_Cities_Change_median <- AZ_Maricopa_data %>% 
  group_by(City, Date) %>% 
  summarise(Median_ZHVI = median(ZHVI, na.rm = TRUE)) %>% 
  arrange(City, Date) %>% 
  mutate(Percent_Change = (Median_ZHVI/lag(Median_ZHVI) - 1) * 100)

## `summarise()` has grouped output by 'City'. You can override using the
## `.groups` argument.

# Remove NA values 
AZ_Maricopa_Cities_Change_median <- AZ_Maricopa_Cities_Change_median %>% 
  filter(!is.na(Percent_Change))

Let’s first calculate the percentage change for only Maricopa County This will require multiple filtering steps.

“Lag” will allow us to calculate the monthly change at the lost of some NA values, we can use built in functions to remove them.

Much better!

Maricopa County has experienced significant fluctuations from 2000 to 2024. This indicates variability in the monthly percent change of median ZHVI.

There have been periods of both positive and negative growth, with some instances of rapid increases or decreases. This suggests that the housing market in Maricopa County has been dynamic during this period.

Let’s take a look at these rapid increases!

## # A tibble: 6 × 4
##   Date       Median_ZHVI Percent_Change Location       
##   <date>           <dbl>          <dbl> <chr>          
## 1 2005-07-31     238450.           4.02 Maricopa County
## 2 2005-08-31     247375.           3.74 Maricopa County
## 3 2005-06-30     229240.           3.71 Maricopa County
## 4 2005-09-30     256530.           3.70 Maricopa County
## 5 2005-10-31     264651.           3.17 Maricopa County
## 6 2001-01-31     145325.           3.12 Maricopa County

Now let’s take a look at those rapid decreases!

## # A tibble: 6 × 4
##   Date       Median_ZHVI Percent_Change Location       
##   <date>           <dbl>          <dbl> <chr>          
## 1 2008-12-31     190650.          -3.76 Maricopa County
## 2 2009-01-31     184152.          -3.41 Maricopa County
## 3 2008-11-30     198107.          -3.38 Maricopa County
## 4 2008-04-30     238901.          -3.02 Maricopa County
## 5 2008-09-30     210968.          -2.92 Maricopa County
## 6 2008-10-31     205042.          -2.81 Maricopa County

Let’s move on to our linear regression model for ZHVI prices over the next 10 years.

We will use the R linear regression model to quickly predict and plot our data into a forecast.

On average the model’s predictions are about $53,182.83 away from the actual values. THe closer the MAE’s value is to 0 the better the model is performing.

The large MSE can be due to the nature of the dataset and also outliers within the dataset.

# Assuming ZHVI_full is already loaded and filtered for Arizona as ZHVI_az


ZHVI_az <- ZHVI_filtered_M %>% 
  filter(State == "AZ" )


# Calculate the percentage change in ZHVI
ZHVI_changes <- ZHVI_az %>%
  group_by(RegionName) %>%
  summarize(
    StartZHVI = first(ZHVI),
    EndZHVI = last(ZHVI),
    PercentChange = ((EndZHVI - StartZHVI) / StartZHVI) * 100
  ) %>%
  ungroup() %>%
  arrange(desc(PercentChange))

# Get the top 5 increases and decreases
top_increases <- ZHVI_changes %>% top_n(5, PercentChange)
top_decreases <- ZHVI_changes %>% top_n(-5, PercentChange)

# Visualizations
# Plot the top 5 increases
ggplot(top_increases, aes(x = reorder(RegionName, PercentChange), y = PercentChange)) +
  geom_col() +
  coord_flip() +
  labs(title = "Top 5 Increases in ZHVI in Arizona", x = "Region", y = "Percent Change")

# Plot the top 5 decreases
ggplot(top_decreases, aes(x = reorder(RegionName, -PercentChange), y = PercentChange)) +
  geom_col() +
  coord_flip() +
  labs(title = "Top 5 Decreases in ZHVI in Arizona", x = "Region", y = "Percent Change")

Increasing : This bar chart illustrates the top five regions in Arizona that have experienced the highest percentage increase in ZHVI, a measure of home value, over a specified period. The regions are ranked from the highest to the lowest increase, indicating a substantial growth in property values in these areas. This trend suggests a potentially higher demand for housing or an increase in market value, making these regions particularly noteworthy for investors or stakeholders in the real estate market

Decreasing : The depicted bar chart reveals the five regions in Arizona with the most significant percentage decrease in ZHVI, reflecting a decline in home values. The regions are arranged from the greatest to the smallest decrease, indicating areas where the market might be cooling or facing challenges. This information is critical for understanding potential risks or for identifying areas that may offer more affordable housing options.

# Assuming ZHVI_changes is already created from the previous question
# Calculate IQR to identify outliers
ZHVI_stats <- ZHVI_changes %>%
  summarize(
    Q1 = quantile(PercentChange, 0.25),
    Q3 = quantile(PercentChange, 0.75)
  ) %>%
  mutate(IQR = Q3 - Q1, lower_bound = Q1 - 1.5 * IQR, upper_bound = Q3 + 1.5 * IQR)

# Add a new column to flag the outliers
ZHVI_changes <- ZHVI_changes %>%
  mutate(Outlier = PercentChange < ZHVI_stats$lower_bound | PercentChange > ZHVI_stats$upper_bound)

# Plot the outliers
ggplot(ZHVI_changes, aes(x = RegionName, y = PercentChange, color = Outlier)) +
  geom_point() +
  theme_minimal() +
  labs(
    title = "Outliers in ZHVI Percent Change in Arizona",
    x = "Region",
    y = "Percent Change"
  ) +
  coord_flip() + # Flip coordinates for a horizontal plot
  scale_color_manual(values = c("FALSE" = "blue", "TRUE" = "red")) # Blue for non-outliers, red for outliers

This scatter plot illustrates the outliers in the percentage change of ZHVI across different regions in Arizona. Each point represents a region, with the x-axis showing the percentage change in ZHVI and the y-axis representing the specific regions. The vast majority of data points are colored in blue, indicating typical behavior within the expected range of ZHVI changes. The red points signify outliers which are regions with ZHVI changes that stand out significantly from the rest, either due to exceptionally high increases or decreases. These outliers may warrant further investigation to understand the underlying factors contributing to their anomalous behavior. Identifying such outliers can help in recognizing unique market trends, potential data errors, or special economic conditions affecting those regions.