Prediction of Auto Insurance Claims

By Kajal Nehra

Introduction

In the insurance industry, accurate pricing of policies is critical for achieving a delicate balance between competitive rates and profitability. Generalized Linear Models (GLMs) provide a robust framework for this purpose by allowing insurers to account for non-normal distributions in claim amounts, ensuring better predictions while maintaining pricing accuracy. By leveraging historical data, GLMs enable businesses to make data-driven decisions that benefit both the company and policyholders.

This article explores the application of GLMs in predicting auto insurance claim amounts using historical claims data. Specifically, we fit a Gamma GLM with a log link function to model the skewed nature of claim amounts.

Our Data

The data set used for this analysis is the Auto Insurance Claims data set.Click Here

Key steps in data preparation included:

  • Removing Irrelevant Columns: Unnecessary columns such as policy_number, insured_zip, and incident_location were dropped to focus on variables relevant to pricing.

  • Handling Missing Values: Missing entries in columns like collision_type and property_damage were replaced with their respective modes.

  • Converting Data Types: Character variables were transformed into factors to ensure compatibility with statistical modeling.

  • Train-Test Split: The data set was divided into 80% training and 20% testing subsets for model development and evaluation.

library(tidyverse)
insurance_data <- read.csv("insurance_claims.csv")
str(insurance_data)
summary(insurance_data)
# Removing columns using base R
columns_to_remove <- c("X_c39", "policy_number", "insured_zip", "policy_bind_date", 
                       "incident_date", "incident_location")
insurance_data <- insurance_data[, setdiff(names(insurance_data), columns_to_remove)]

colnames(insurance_data)
# Replacing '?' with NA
insurance_data[insurance_data == "?"] <- NA
# Imputing missing values with the mode
for (col in c("collision_type", "property_damage", "police_report_available")) {
  mode_val <- as.character(na.omit(insurance_data[[col]])) %>% table() %>% which.max()
  insurance_data[[col]][is.na(insurance_data[[col]])] <- mode_val
}
# Converting character variables to factors
insurance_data <- insurance_data %>%
  mutate(across(where(is.character), as.factor))
library(caTools)
set.seed(42)
# Split data: 80% training, 20% testing
split <- sample.split(insurance_data$total_claim_amount, SplitRatio = 0.8)
training_data <- subset(insurance_data, split == TRUE)
testing_data <- subset(insurance_data, split == FALSE)

 Data Dictionary

Below is a summary of the main variables used in the analysis:

Variable Description
total_claim_amount Total amount claimed in USD
age Age of the policyholder
policy_annual_premium Annual premium amount in USD
policy_deductable Deductible amount in USD
incident_severity Severity of the incident
collision_type Type of collision
number_of_vehicles_involved Number of vehicles involved in the incident
insured_sex Gender of the insured individual
insured_education_level Education level of the insured
insured_relationship Relationship status of the insured individual

Exploratory Data Analysis

Fitting Different Distribution on the claims amount to see which one fits best.

library(ggplot2)
# Identify numeric columns
numeric_cols <- names(insurance_data)[sapply(insurance_data, is.numeric)]
# Loop through numeric columns and plot density
for (col in numeric_cols) {
  p <- ggplot(insurance_data, aes(x = .data[[col]])) +
    geom_density(fill = "blue", alpha = 0.5) +
    labs(title = paste("Density Plot for", col), x = col, y = "Density") +
    theme_minimal()
  print(p)
}

# Extract the total_claim_amount column
claim_data <- insurance_data$total_claim_amount
# Fit distributions
library(MASS)
claim_data_rescaled <- claim_data / 1000
library(fitdistrplus)
gamma_fit <- fitdist(claim_data_rescaled, "gamma")
exp_fit <- fitdistr(claim_data_rescaled, "exponential")
normal_fit <- fitdistr(claim_data_rescaled, "normal")
# Plot with superimposed distributions
ggplot(data.frame(claim_data_rescaled), aes(x = claim_data_rescaled)) +
  geom_density(color = "blue", fill = "lightblue", alpha = 0.4) +
  stat_function(fun = dgamma, args = list(shape = gamma_fit$estimate[1], rate = gamma_fit$estimate[2]),
                aes(color = "Gamma"), size = 1) +
  stat_function(fun = dexp, args = list(rate = 1/exp_fit$estimate),
                aes(color = "Exponential"), size = 1) +
  stat_function(fun = dnorm, args = list(mean = normal_fit$estimate[1], sd = normal_fit$estimate[2]),
                aes(color = "Normal"), size = 1) +
  labs(title = "Density Plot with Fitted Distributions", x = "Total Claim Amount", color = "Distribution") +
  theme_minimal()

Exponential Distribution (Red)

  • Peaks sharply near the origin and fails to capture the shape of the data for larger claim amounts.

  • Underestimates the variability and spread, making it unsuitable for the dataset.

Normal Distribution (Blue)

  • Symmetric, unlike the positively skewed claim data.

  • Overestimates mid-range and high values while failing to match the peak near lower values.

  • Unsuitable for modeling due to its inability to capture data skewness.

Gamma Distribution (Green)

  • Captures the positive skewness and follows the overall trend better than Exponential and Normal distributions.

  • Aligns reasonably well with the tail of the data, making it more appropriate for modeling insurance claims with extreme values.

To determine the best GLM model for predicting total_claim_amount using various distributions, we will fit models with different error distributions and compare them using the Akaike Information Criterion (AIC). Lower AIC values indicate better model performance.

# Gaussian GLM
glm_gaussian <- glm(total_claim_amount ~ age + policy_annual_premium + 
                      policy_deductable + incident_severity + 
                      number_of_vehicles_involved + collision_type,
                    family = gaussian(link = "identity"), 
                    data = insurance_data)
# Gamma GLM
glm_gamma <- glm(total_claim_amount ~ age + policy_annual_premium + 
                   policy_deductable + incident_severity + 
                   number_of_vehicles_involved + collision_type,
                 family = Gamma(link = "log"), 
                 data = insurance_data)

# Poisson GLM
glm_poisson <- glm(total_claim_amount ~ age + policy_annual_premium + 
                     policy_deductable + incident_severity + 
                     number_of_vehicles_involved + collision_type,
                   family = poisson(link = "log"), 
                   data = insurance_data)

# Compare AIC values
aic_values <- AIC(glm_gaussian, glm_gamma, glm_poisson)

From the output

glm_gaussian has AIC 22009.34

glm_gamma has AIC 21497.02

glm_poisson has AIC 3439839.04

The Gamma GLM is the preferred model for predicting the Total Claim Amount due to its superior fit, as evidenced by the lower AIC value. This supports the earlier conclusion drawn from the density plots that the Gamma distribution better captures the skewed nature of the data.

# Random Forest Comparison
library(randomForest)
rf_model <- randomForest(total_claim_amount ~ age + policy_annual_premium + policy_deductable +
                           incident_severity + number_of_vehicles_involved + collision_type,
                         data = insurance_data, ntree = 500)

The output provides key performance metrics for the Random Forest Regression Model.

  1. Model Type:

    • The model is a regression Random Forest because the response variable (total_claim_amount) is continuous.

  2. Number of Trees:

    • The model was built using 500 trees (ntree = 500). Increasing the number of trees typically enhances performance and stability, though at the cost of computational time.

  3. Number of Variables Tried at Each Split:

    • At each node split, 2 variables were randomly selected from the total predictors to find the best split. This is a standard practice to reduce overfitting and improve generalization.

  4. Mean of Squared Residuals (MSE):

    • The Mean Squared Residuals is reported as 224,346,960. This value represents the average squared difference between the observed and predicted claim amounts.

    • While this value is large, it must be interpreted in the context of the scale of the total_claim_amount variable.

  5. Percentage of Variance Explained:

    • The model explains approximately 67.78% of the variance in the target variable (total_claim_amount).

    • This indicates that the model captures a significant portion of the variability in the claim amounts, suggesting it is a reasonably good fit for the data.

The Random Forest model strikes a balance between predictive performance and flexibility, making it suitable for predicting claim amounts. However, its lack of transparency highlights the need to weigh accuracy against interpretability, especially in contexts requiring explainable and accountable modeling. Complementary analyses using interpretable models (e.g., GLMs) may help address these concerns while maintaining predictive effectiveness.

GLM Modeling

A Generalized Linear Model (GLM) is a flexible statistical framework that extends linear regression by allowing the response variable to follow distributions from the exponential family (e.g., normal, Poisson, gamma). It connects the predictors to the response variable through a link function, enabling the model to handle non-normal data, accommodate varying variance structures, and model complex relationships effectively.

For this analysis, a Gamma distribution with a log link function was chosen to model the positively skewed total_claim_amount variable. The Gamma distribution effectively captures the skewed, positive-only nature of claim amounts and accommodates the heavy tails often observed in insurance data. The log link function ensures all predicted values are positive and transforms the relationship between predictors and the response variable into a multiplicative scale, making predictions proportional and interpertable. This approach is particularly useful for understanding the impact of predictors on claim amounts while addressing the skewness and variability inherent in the data.

# Predict on test data
test_predictions <- predict(glm_gamma, newdata = testing_data, type = "response")

# Calculate RMSE and MAE
library(Metrics)
rmse_val <- rmse(testing_data$total_claim_amount, test_predictions)
mae_val <- mae(testing_data$total_claim_amount, test_predictions)

  • Root Mean Squared Error (RMSE): 15,437.95

    • Indicates the standard deviation of prediction errors.
    • Suggests the model’s predictions are off by ~15,438 on average in squared unit

  • Mean Absolute Error (MAE): 11,563.65

    • Indicates the average absolute difference between actual and predicted values.
    • Suggests that, on average, the model is off by ~11,564 in absolute terms.
# Diagnostic plots
par(mfrow = c(2, 2))
plot(glm_gamma)

# Actual vs Predicted
ggplot(data.frame(Actual = testing_data$total_claim_amount, Predicted = test_predictions),
       aes(x = Actual, y = Predicted)) +
  geom_point() +
  geom_abline(slope = 1, intercept = 0, color = "red") +
  labs(title = "Actual vs Predicted Claims", x = "Actual Claims", y = "Predicted Claims")

Residual Analysis:

  • Residuals vs Fitted Plot: Residuals are centered around zero, but there is evidence of non-constant variance.
  • Q-Q Plot: The residuals deviate from the theoretical quantiles, suggesting non-normality.
  • Scale-Location Plot: Variance is not homogeneously distributed, indicating potential issues with model fit.
  • Residuals vs Leverage: No significant high-leverage points, but some observations (e.g., 776 and 210) show Cook’s distance values that warrant attention.

Actual vs. Predicted Claims Plot:

  • The scatter plot shows a concentration of points away from the red diagonal line, indicating some discrepancies between actual and predicted claim amounts.
  • Predictions appear to perform reasonably well for mid-range claims but show significant variability for higher claim amounts.

Conclusion

This analysis leveraged Generalized Linear Models (GLMs) and Random Forests to model and predict auto insurance claim amounts, addressing the challenges posed by positively skewed, non-normal data. By employing a Gamma GLM with a log link function, we successfully captured the skewness of the claim amounts and provided interpretable insights into how predictors influence claims. The Gamma GLM demonstrated superior fit compared to other GLMs (e.g., Gaussian, Poisson), as evidenced by its lower AIC value, and achieved a reasonable balance between accuracy and interpretability.

In parallel, the Random Forest model highlighted the benefits of a machine learning approach, explaining approximately 67.78% of the variance in claim amounts and showcasing robust predictive performance. However, its “black-box” nature limits interpretability, an important consideration in the insurance industry where transparency in pricing models is critical.

The analysis revealed that while traditional statistical models like GLMs provide a strong foundation for pricing and risk assessment, there is potential to enhance accuracy by integrating machine learning techniques.

How to Improve Pricing

The journey toward improving insurance pricing involves leveraging advanced techniques and exploring new methodologies. While traditional models like GLMs have been instrumental in ensuring transparent and interpretable pricing, there is significant room for innovation and improvement through the integration of advanced statistical and machine learning methods.

  • Machine Learning & Ensemble Models: Advanced techniques like GBMs, Random Forests, and Neural Networks can capture complex patterns, enhancing prediction accuracy.

  • Explainable AI (XAI): Tools like SHAP and LIME ensure transparency and accountability in complex models, addressing interpretability concerns.

  • Dynamic Pricing: Real-time data, such as telematics, enables personalized, adaptive pricing based on individual risk profiles.

  • Big Data & Feature Engineering: Leveraging new data sources and creating derived variables improves model performance.

  • Regularization & Hyperparameter Optimization: Techniques like LASSO and Bayesian optimization prevent overfitting and fine-tune models for better accuracy.

  • Hybrid Models: Combining GLMs for transparency and machine learning for predictive power creates a balanced approach.

  • Regulatory Compliance & Ethical AI: Ensuring models are fair, unbiased, and explainable to meet industry standards and customer trust.

  • Scenario Testing & Sensitivity Analysis: Evaluating model performance under various conditions ensures resilience and adaptability.

These advancements will help insurers balance accuracy, fairness, and profitability in a data-driven market.