• This project utilizes the House Sales in King County, USA dataset, which is found on Kaggle.

• The House Sales in King County, USA dataset contains data on house sales in King County, Washington.

• King County is home to the city of Seattle and is one of the most populous regions in the state of Washington.

The price of houses in our dataset have a right-skew distribution, meaning that the median price of houses would be a better indicator of central tendancy than the mean.

There is generally a positive correlation between the price and sqft_living features, indicating that larger houses tend to be more expensive. However, this relationship is not perfectly linear, suggesting the presence of some outliers. Additionally, other factors such as location, number of bedrooms and bathrooms, and view quality are also likely influencing house prices.

When plotting price against various square footage variables, it is observed that sqft_basement and sqft_lot do not show a distinctive relationship with the price variable. The presence of numerous outliers further suggests that these variables may be less predictive of house prices and could potentially be excluded from model training.

It is essential to identify rows with zero values in certain features to prevent them from becoming outliers and hindering our model’s performance.

We observe that bathrooms and bedrooms have very few zeros, allowing us to simply exclude rows where these features are zero. The yr_renovated feature has almost all zeros, making it meaningless for model training. Conversely, yr_built has no zeros, indicating it is worth exploring its effect on the price.

Since the price of houses is roughly increasing with the bedrooms count, the number of bedrooms can be considered one of the strong indicators of price .

Since the price of houses is roughly increasing with the bathrooms count, the number of bathrooms can be considered one of the strong indicators of price .

The price of houses seems to be roughly increasing with an increase in view, grade, waterfront, floors, and condition values which was totally expected. These features would also serve as good indicators for our target variable price.

The plot of yr_built v/s price does not give out any significant information.

A correlation plot helps visualize the relationships between different features in our dataset and helps us identify patterns and dependencies among them.

We can see that variables such as waterfront and view are highly correlated and variables such as yr_renovated and sqft_lot are not correlated. This gives us a basic picture of interdependencies between features which might be helpful for further model development.

# Removing 0's from bedrooms and bathrooms
kc_data_clean <- kc_data %>%
  filter(bedrooms != 0 & bathrooms != 0)

# Remove columns sqft_lot, sqft_lot15, sqft_living15, sqft_basement
kc_data_clean <- kc_data_clean %>%
  select(-sqft_lot, -sqft_lot15, -sqft_living15, -sqft_basement)

# Removing column yr_renovated, date and id
kc_data_clean <- kc_data_clean %>%
  select(-yr_renovated, -date, -id)

We discarded rows with zero bedrooms and bathrooms since these instances were few and had minimal impact on the dataset size.

Columns such as sqft_lot, sqft_lot15, sqft_living15, and sqft_basement were removed due to numerous outliers and a lack of significant correlation with the target variable, price. sqft_living15 was also highly correlated with sqft_living, making it redundant.

The yr_renovated column primarily contained ‘0’ values, indicating minimal renovation activity, thus irrelevant for predictive modeling. The date and id features were also removed as id is just a unique identifier, and date in character form is not useful for predicting price.

While further data cleaning steps could be undertaken, only these actions were executed for simplicity.

set.seed(142)

# Perform train-test split (80% train, 20% test)
train_index <- createDataPartition(kc_data$price, p = 0.8, list = FALSE)
train_data <- kc_data_clean[train_index, ]
test_data <- kc_data_clean[-train_index, ]


predictors <- setdiff(names(train_data), "price")
target <- "price"

# Create x_train, y_train, x_test, y_test
x_train <- train_data[, predictors]   # training features
y_train <- train_data[, target]       # training target (price)
x_test <- test_data[, predictors]     # testing features
y_test <- test_data[, target]         # testing target (price)

# Remove NA values (if any)
complete_rows <- complete.cases(x_train, y_train)
x_train <- x_train[complete_rows, ]
y_train <- y_train[complete_rows]

We split our dataset into training and testing sets to ensure fair evaluation. The model will be trained on the training data and tested on the unseen test data. Additionally, the features (x_train, x_test) are separated from the target variable price (y_train, y_test).

# Train the linear regression model
lm_model <- lm(y_train ~ ., data = x_train)

# Make predictions on the test set
predictions <- predict(lm_model, newdata = x_test)

# Calculate evaluation metrics
rmse_lr <- sqrt(mean((predictions - y_test)^2))
r_squared_lr <- cor(predictions, y_test)^2

results_lr <- data.frame(Actual = y_test, Predicted = predictions)

# Plot using ggplot
ggplot(results_lr, aes(x = Actual, y = Predicted)) +
  geom_point(alpha = 0.6, color = "green") +
  geom_abline(intercept = 0, slope = 1, color = "red", linetype = "dashed") +
  ggtitle("Actual vs. Predicted Prices (Linear Regression: Test Data)") +
  xlab("Actual Prices (in $)") +
  ylab("Predicted Prices (in $)") +
  theme_minimal() +
  theme(
    plot.title = element_text(hjust = 0.5),
    axis.title.x = element_text(face = "bold"),
    axis.title.y = element_text(face = "bold")
  )

Our initial Linear Regression model exhibits less promising results on the Test Data since the slope of the dotted red line is noticeably greater than 1. To delve deeper, we’ll explore residual plots and training plots to understand the underlying dynamics.

From the training plot, it’s apparent that our model isn’t adequately capturing the underlying pattern of our training data. This discrepancy could stem from the fact that our data doesn’t conform to a linear relationship.

Analyzing the scatter plot for residuals reveals a distinct pattern: there’s a noticeable curve where residuals tend to be positive for lower predicted values and negative for higher predicted values. This suggests that our model might be underestimating the prices of lower-valued houses while overestimating those of higher-valued ones.

# Train the Random Forest Regression model
rf_model <- randomForest(y_train ~ ., data = x_train)

# Make predictions on the test set
predictions <- predict(rf_model, newdata = x_test)

# Calculate evaluation metrics
rmse_rf <- sqrt(mean((predictions - y_test)^2))
r_squared_rf <- cor(predictions, y_test)^2

results_rf <- data.frame(Actual = y_test, Predicted = predictions)

# Plot using ggplot
ggplot(results_rf, aes(x = Actual, y = Predicted)) +
  geom_point(alpha = 0.6, color = "lightblue") +
  geom_abline(intercept = 0, slope = 1, color = "red", linetype = "dashed") +
  ggtitle("Actual vs. Predicted Prices (Random Forest)") +
  xlab("Actual Prices") +
  ylab("Predicted Prices") +
  theme_minimal() +
  theme(
    plot.title = element_text(hjust = 0.5),
    axis.title.x = element_text(face = "bold"),
    axis.title.y = element_text(face = "bold")
  )

The Random Forest Model exhibits superior performance compared to the Simple Linear Regression Model, as indicated by the closer alignment of the scatter plot points to the diagonal red dashed line, which represents perfect prediction (slope = 1). This proximity suggests that our Random Forest model’s predictions are more accurate, particularly for houses with lower to moderate prices.

However, as the price of houses increases beyond $2,000,000, the model’s accuracy diminishes, evident from the widening spread of data points away from the ideal line. This discrepancy indicates that our model struggles to accurately predict the prices of higher-value houses, potentially due to the complexity of factors influencing their pricing beyond what our model captures.

The Random Forest Feature Importance Plot displays the relative significance of different features in predicting house prices. Features with higher importance contribute more to the model’s predictive accuracy, while those with lower importance have less impact. This visualization helps identify which features are most influential in determining house prices.

The Random Forest Feature Importance Plot reveals that sqft_living and grade are the two most influential features in predicting house price. Interestingly, location-related features such as lat, long, and zipcode also rank among the top 10 most important features, underscoring the significant role of a property’s location in determining its value.

# Convert training and testing data to DMatrix format
dtrain <- xgb.DMatrix(data = as.matrix(x_train), label = y_train)
dtest <- xgb.DMatrix(data = as.matrix(x_test), label = y_test)

params <- list(
  objective = "reg:squarederror",
  eval_metric = "rmse",
  max_depth = 6,
  eta = 0.3,
  subsample = 0.7,
  colsample_bytree = 0.7
)

# Train the xgboost model
xgb_model <- xgb.train(params = params, data = dtrain, nrounds = 100)

# Make predictions on the test set
predictions_xgb <- predict(xgb_model, newdata = dtest)

# Calculate evaluation metrics
rmse_xgb <- sqrt(mean((predictions_xgb - y_test)^2))
r_squared_xgb <- cor(predictions_xgb, y_test)^2

# Create a data frame for actual vs predicted values
results_xgb <- data.frame(Actual = y_test, Predicted = predictions_xgb)

# Plot using ggplot
ggplot(results_xgb, aes(x = Actual, y = Predicted)) +
  geom_point(alpha = 0.6, color = "purple") +
  geom_abline(intercept = 0, slope = 1, color = "red", linetype = "dashed") +
  ggtitle("Actual vs. Predicted Prices (Gradient Boosting)") +
  xlab("Actual Prices (in $)") +
  ylab("Predicted Prices (in $)") +
  theme_minimal() +
  theme(
    plot.title = element_text(hjust = 0.5),
    axis.title.x = element_text(face = "bold"),
    axis.title.y = element_text(face = "bold")
  )

The Gradient Boosting model demonstrates superior performance compared to the Linear Regression model. The data points cluster closely around the ideal y=x line, suggesting that our model’s predictions closely align with actual house prices. However, challenges still persist in accurately predicting the values of higher-priced houses, as indicated by the scattered data points deviating from the line.

The RMSE (Root Mean Squared Error) is a metric used to measure the difference between the predicted values from a model and the actual values. A lower RMSE indicates a better fit for the model on the data used to create the graph. The Linear regression model has the highest RMSE, while the random forest model and XGBoost model have similar RMSEs (Random Forest model having a sightly lower RMSE).

Linear Regression has the lowest R-squared: This indicates that the linear regression model explains the least amount of variance in the target variable compared to the other two models. In other words, a significant portion of the variation in the data is not explained by the linear model.

Random Forest likely has the best fit: This is because Random Forest has the highest R-squared value among the three models. A higher R-squared signifies a better fit for the model, meaning it explains a larger proportion of the variance in the target variable.

• The major concern we notice is that our models struggle to predict the prices of high-value houses accurately, possibly due to outliers influencing predictions. Handling outliers is crucial to improve model accuracy.

• Additionally, the Linear Regression Model performs notably worse than the other two models. Linear regression models are sensitive to feature scales, so normalizing or standardizing features is advisable to ensure equal contribution to predictions.

• Feature Engineering, such as combining location-related features like latitude, longitude, and zipcode, could enhance model performance by capturing location-specific patterns. Also removing unnecessary features could improve model results

• Hyperparameter Tuning for the XGBoost model is essential. Experimenting with different parameter sets and selecting the one yielding the best results can significantly improve model performance.

• More Advanced Algorithms, such as gradient boosting machines (GBM), support vector machines (SVM), or neural networks, can be done to leverage their potential for capturing complex patterns in the data.