Motivation & Intended Use

Compensations for crop losses are a huge expenditure for companies insuring farms across the United States. When there are widespread extreme weather events contributing to crop damage it can cost insurers millions of dollars in claims, but there is also the added cost associated with adjustment and assessment. Each claim, whether paid or not, must be evaluated to confirm both existence and magnitude of loss. Sending an adjust to process a relatively small loss, when there are many claims including large ones to be adjusted, can add significant cost to the processing of claims for a given event.

If the cost of adjusting a small claim is extreme relative to the payout, it might make more sense for an insurer to simply pay the claim provided the agent has adequate confidence that an event occurred capable of causing the loss claimed by the insured.

Currently most losses are human-adjusted, with a growing number of technological tools, such as drones and satellite imagery, used to improve the speed and results of the inspection process.

Having models which predict a reasonable claim payout based on historical weather and losses could be of use in this situation. If the model suggest a value that is within an reasonable range, based on proximity to the event, and historical losses incurred in other such events, an insurer could simply issue a check, instead of investing human capital to investigate.

This would allow insurers to focus their human capital on the most pressing claims, which save or cost them the most when erroneously adjusted.

Practical Performance Guidelines

Performance of a model like this would be best evaluated on how often it is correct in assigning value (accuracy of loss/no loss) and how close to predicting a correct payout amount for those right predictions (net difference) and whether it over or under pays. If the model were to under predict payouts, requiring manual reviews for contested findings, that would be better than over recognizing events and suggesting loss when there was none. This would make it more robust against fraud. If it were relatively close in payments, but reduced costs, it might be worth using as a first estimate of whether or not the claim filed was justified. The agents would have acreage, valuation and percent of loss in hand already, so if the model suggested a loss, and claim were lower than the loss suggested by the model for the area in question, it might be an adequate solution to clearing a backlog of smaller filings.

These types of models could be used in other areas of insurance, and similar models would be helpful in estimating disaster recovery costs for state & federal agencies working in relief after natural disasters. The models would be similar, although the outcome might be water, man-hours of service, food or medicines instead of claims dollars.

Assumptions

The utility of this model is predicated on the assumption that time, money and manpower delay the adjustment of crop claims, that larger claims get more attention and receive a higher priority. This means that smaller claims become an added burden upon insurers when faced with many larger more complex claims, and that the cost of adjusting smaller claims might be disproportionate with the cost of diverting manpower away from working on larger claims where erroneous estimates are more detrimental to the financial security of the insurer.

Choosing a Geography to Analyze

Iowa was chosen as the state to look in for a county to use as the test geography for four major reasons:

1. Heavily agricultural state with vast areas of wheat and corn, which are prone to weather related damage
2. Consistent topography with moderate elevation changes, which facilitates the building and testing of a small, scale model
3. Crops with predictable and consistent yields (corn and wheat are easier to estimate than say apples or peaches)
4. An ecosystem prone to extreme weather events from freezing cold, to droughts, to tornadoes and hail.

Within Iowa, Marshall county became the choice because it is a highly agricultural county with known historical crop-loss events.

# data for maps is included in package
states <- map_data("state")
iowa_geo <- subset(states, region == "iowa")

counties <- map_data("county")
iowa_county <- subset(counties, region == "iowa")

marshall <- subset(iowa_county, subregion == "marshall")

base <- ggplot(data = iowa_geo, mapping = aes(x = long, y = lat, group = group)) + 
    coord_fixed(1.3) + geom_polygon(data = iowa_county, fill = NA, color = "white") + 
    geom_polygon(color = "black", fill = NA) + geom_polygon(data = marshall, 
    fill = "#a65c85b2", color = "white")
base

iowa <- read.csv("https://www2.census.gov/geo/docs/reference/codes/files/st19_ia_cou.txt", 
    header = FALSE)
iowa <- iowa %>% rename(State = V1) %>% rename(State_Code = V2) %>% rename(County_Code = V3) %>% 
    rename(County = V4) %>% rename(Active_Code = V5) %>% mutate(County_Code = str_pad(County_Code, 
    3, pad = "0")) %>% mutate(State_Code = str_pad(State_Code, 3, pad = "0")) %>% 
    mutate(FIPS = paste0(State_Code, County_Code))

# Get FIPS Code
marshall_FIPS <- iowa %>% filter(County == "Marshall County") %>% pull(FIPS)

The US Census FIPS code for Marshall County, Iowa is : 019127

Reading Data from NOAA Storm Events Database

# find_events() is written to download all files and sort out what you
# request
all_events <- find_events(date_range = c("1999-01-01", "2010-12-31"))

Massaging Storm Events data to be more useful, filtering for counties names Marshall in the state of Iowa, selecting only the beginning & end dates as well as crop damage to bind to the weather data frame.

marshall_events <- all_events %>% filter(state == "Iowa") %>% filter(cz_name == 
    "Marshall") %>% select(begin_date, end_date, damage_crops)

Comparing Weather Driven Crop Loss Across Iowa

From year to year loss events differ in numbers and degree. The following maps are here to give a sense of how variable weather related crop loss can be in Iowa.

Iowa_events_01 <- all_events %>% filter(state == "Iowa") %>% mutate(DATE = as.Date(end_date, 
    format = "%m/%d/%y")) %>% filter(year(DATE) %in% c(2001))
Iowa_events_05 <- all_events %>% filter(state == "Iowa") %>% mutate(DATE = as.Date(end_date, 
    format = "%m/%d/%y")) %>% filter(year(DATE) %in% c(2005))

map_events(Iowa_events_01, states = "Iowa", plot_type = "crop damage")

map_events(Iowa_events_01, states = "Iowa", plot_type = "number of events")

map_events(Iowa_events_05, states = "Iowa", plot_type = "crop damage")

map_events(Iowa_events_05, states = "Iowa", plot_type = "number of events")

Reading in the Weather Data

The downloaded file was read in and then polished for merging with the Marshall Events data. In choosing data to use in the analysis, I calculated the number of NA values in a given column which could possibly be considered a predictor variable, keeping only the variables where at least 70 of the observations were not NA values.

The regression model will not know how to attribute multiple values to the same variable in the same location on the same day with a different outcome. So, loss data from the same day, was combined to create a single target day for the summarized weather data on each day.

marshall_all_weather <- read.csv("https://raw.githubusercontent.com/bpoulin-CUNY/Data607/master/Data%20607%20Final/marshall_weather.csv", 
    stringsAsFactors = FALSE, header = TRUE)

# Build list to iterate over
var_names <- c("LATITUDE", "LONGITUDE", "ELEVATION", "DATE", "AWND", "DAPR", 
    "FMTM", "MDPR", "PGTM", "PRCP", "SNOW", "SNWD", "TAVG", "TMAX", "TMIN", 
    "TOBS", "TSUN", "WDF2", "WDF5", "WESD", "WESF", "WSF2", "WT01", "WT02", 
    "WT03", "WT04", "WT05", "WT06", "WT08", "WT09")
# Create Data Frame of NA ratios to use in excluding variables
nas <- data.frame(Variable = character(0), Quantity = numeric(0), Proportion = numeric(0))
for (i in var_names) {
    num = sum(is.na(marshall_all_weather[i]))
    prop = num/nrow(marshall_all_weather)
    temp = data.frame(i, num, prop)
    colnames(temp) = c("Variable", "Quantity", "Proportion")
    nas <- rbind(temp, nas)
}

# Build a list of variables to keep using logicals
keep <- nas %>% mutate(Variable = as.character(Variable)) %>% filter(Proportion < 
    0.7) %>% pull(Variable)
# Format Date correctly & reduce to only variables with sufficient data to
# feed into model
marshall_all_weather <- marshall_all_weather %>% mutate(DATE = as.Date(DATE, 
    format = "%m/%d/%y")) %>% 
select(keep, -WSF5)

# Combining Events Data
marshall_events <- marshall_events %>% mutate(damage_crops = round(damage_crops, 
    0)) %>% group_by(end_date) %>% summarise(damage_crops = sum(damage_crops))

Predictions & Evaluation

# simple prediction based on the training model
test$Prediction <- predict(fit, newdata = test)
# assigning 1 to observations predicted no loss with no loss & predicted
# loss with a loss, zero to all others.  This will become our accuracy
# check, the mean of ones
test <- test %>% mutate(Correct = ifelse((test$Prediction <= 0 & test$damage_crops <= 
    0) | (test$Prediction > 0 & test$damage_crops > 0), 1, 0)) %>% mutate(Prediction = round(Prediction, 
    0))
test_accuracy <- mean(test$Correct, na.rm = TRUE)

Model Accuracy: 46.2 %

Although this model was accurate in predicting a loss 46.2% of the time, we need to look closer to see how much each storm prediction was off, in terms of claim value.

Payouts <- test %>% filter(test$Correct == 1 & test$Prediction > 0 & !is.na(test$Correct)) %>% 
    mutate(Discrepancy = damage_crops - Prediction) %>% select(DATE, TMIN, TMAX, 
    PRCP, damage_crops, Prediction, Correct, Discrepancy)

Gasparini Distributed Lag Models

Although we do not have granular enough observations and variables to build a more predictive model from this data, we can use Gasparrini’s distributed lag linear and non-linear methods to attribute the compounded effects of time on each of the variables to create plots of the relationship between a variable and lag periods.

# Build the lag into variables
cross.tmax <- crossbasis(train$TMAX, lag = 5, argvar = list(df = 5), arglag = list(fun = "strata", 
    breaks = 1))
cross.tmin <- crossbasis(train$TMIN, lag = 5, argvar = list(df = 5), arglag = list(fun = "strata", 
    breaks = 1))
cross.prcp <- crossbasis(train$PRCP, lag = 20, argvar = list(fun = "lin"), arglag = list(fun = "poly", 
    degree = 4))

# Construct regression using the lagged variables
new_fit <- lm(damage_crops ~ cross.prcp + cross.tmax + cross.tmin, train)


# Predict the new fit over increment of .1 using a cumulative lag paradigm
pre.tmax <- crosspred(cross.tmax, new_fit, at = 0:10, bylag = 0.1, cumul = TRUE)

# Plots of Graphs for Each Predictor incorporating Lag
mar.default <- c(5, 4, 4, 2) + 0.1
par(mar = mar.default + c(0, 4, 0, 0))
plot(pre.tmax, "slices", var = 5, col = 3, xlim = c(0, 5), las = 1, ylim = c(-600000, 
    1000000), ylab = "", ci.arg = list(density = 25, lwd = 2), main = "Lag Association of Daily Maximum Temperature")

par(mar = mar.default + c(0, 4, 0, 0))
pre.prcp <- crosspred(cross.tmin, new_fit, at = 0:20, bylag = 0.2, cumul = TRUE)
plot(pre.prcp, "slices", var = 5, col = 3, xlim = c(-0.1, 5.5), las = 1, ylim = c(-1900000, 
    1250000), ylab = "", ci.arg = list(density = 25, lwd = 2), main = "Lag Association of Daily Minimim Temperature")

pre.prcp <- crosspred(cross.prcp, new_fit, at = 0:20, bylag = 0.2, cumul = TRUE)

par(mar = mar.default + c(0, 4, 0, 0))
plot(pre.prcp, "slices", var = 5, col = 2, xlim = c(-1, 22), las = 1, ylim = c(-126500000, 
    140000000), ylab = "", ci.arg = list(density = 25, lwd = 2), main = "Lag Association of Daily Precipitation")

The top two Temperature Plots showing a 5-day lag window, clearly lack the granularity to be usefully predict, however, there is a noted increase in each at the one day mark. This would suggest that for periods of extreme temperature longer than a day, the odds of a loss claim increase. However we clearly do not have the right data to accurately model the effect.

Likewise the precipitation graph shows a clear association with three separate changes in direction (and effect) at different lag intervals between 0 and 20 days. On this plot, the confidence interval is also much tighter. This data might be sufficient with more granular temperature data and the addition of other possibly meaningful features such as wind direct, wind speed and sunlight measurements.