Synopsis

The NOAA Storm Data is analysed to assess the impact on human health and the economic impact of different weather events in the United States. For the analysis a specific version of the NOAA Storm Data from 2011 was used. The analysis focused on the time interval from 1993 to 2011, and the event types were as defined by NOAA in 2007. To assess the impact on human health the overall number of fatalities and injuries resulting from the different event types was calculated. The highest overall number of fatalities were reported for the event type “Excessive Heat”, and then highest number of injuries was reported for the event type “Tornado”. The economic impact was determined by calculating the overall cost resulting from property damage and crop damage. The highest cost from property damage was reported for the event type “Hurricane (Typhoon)”, while the highest cost from crop damage was reported for event type “Drought”.

Data Processing

Software

The following software was used for the analysis:

  • R version 4.2.2 (2022-10-31 ucrt)

  • RStudio 2022.12.0+353 “Elsbeth Geranium” Release (7d165dcfc1b6d300eb247738db2c7076234f6ef0, 2022-12-03) for Windows

Set knitr options

Set knitr options to not show warnings and messages to make the document less verbose.

knitr::opts_chunk$set(echo = TRUE, warning = FALSE, message = FALSE) 

Loading required packages

The .bz2 file is unzipped with bunzi2 from the R.utils package. Data processing will be done with dplyr, tidyr and lubridate, therefore the tidyverse package is loaded. Event-types are matched with amatch() from the stringdist package. Plots are done with lattice package, and the panel.needle() function is from the caret package.

library(R.utils) 
library(tidyverse)
library(stringdist)
library(lattice)
library(caret)

Loading and preprocessing the data

Download data

Data are downloaded using link provided in instructions. The code checks whether the data file in the sub-directory data1 exists, and if not, downloads and unzips file in data directory.

main_dir <- getwd()
sub_dir <- "data1"

## check if "data" sub-directory exists, if not download and unzip files in data folder
if (!file.exists(file.path(paste(main_dir, sub_dir,"StormData.csv", sep ="/")))){
  url <- "https://d396qusza40orc.cloudfront.net/repdata%2Fdata%2FStormData.csv.bz2"
  temp <- tempfile()
  dir.create(paste(main_dir, sub_dir, sep ="/"))
  download.file(url, temp)
  bunzip2(filename = temp, destname = file.path(paste(main_dir, sub_dir,"StormData.csv", sep ="/")), remove = FALSE, skip = TRUE)
  unlink(c(temp))
  rm(temp)
} else {print("data available")}
## [1] "data available"

Data are downloaded and in the data1 sub-directory.

Load data in R

The data are read-in with standard read.csv.

a <- Sys.time()

StormData <- read.csv("./data1/StormData.csv", header = TRUE)

b <- Sys.time()

b - a
## Time difference of 21.50598 secs

The data set is quite large with 902297 rows and 37 variables, it was therefore cached to reduce run-time during report creation.

Load Event Names into R

The official Event Names are taken from the “NATIONAL WEATHER SERVICE INSTRUCTION 10-1605 AUGUST 17, 2007”. The Event Names were manually copied from the document, saved as the text file EVENTTABLE.txt and read into R. To ensure reproducibility data loaded into R are printed in this document.

EVTAB <- read.table("./EVENTTABLE.txt", header = TRUE)

EVTAB
##                       EVENT
## 1     Astronomical Low Tide
## 2                 Avalanche
## 3                  Blizzard
## 4             Coastal Flood
## 5           Cold/Wind Chill
## 6               Debris Flow
## 7                 Dense Fog
## 8               Dense Smoke
## 9                   Drought
## 10               Dust Devil
## 11               Dust Storm
## 12           Excessive Heat
## 13  Extreme Cold/Wind Chill
## 14              Flash Flood
## 15                    Flood
## 16             Frost/Freeze
## 17             Funnel Cloud
## 18             Freezing Fog
## 19                     Hail
## 20                     Heat
## 21               Heavy Rain
## 22               Heavy Snow
## 23                High Surf
## 24                High Wind
## 25      Hurricane (Typhoon)
## 26                Ice Storm
## 27         Lake-Effect Snow
## 28          Lakeshore Flood
## 29                Lightning
## 30              Marine Hail
## 31         Marine High Wind
## 32       Marine Strong Wind
## 33 Marine Thunderstorm Wind
## 34              Rip Current
## 35                   Seiche
## 36                    Sleet
## 37         Storm Surge/Tide
## 38              Strong Wind
## 39        Thunderstorm Wind
## 40                  Tornado
## 41      Tropical Depression
## 42           Tropical Storm
## 43                  Tsunami
## 44             Volcanic Ash
## 45               Waterspout
## 46                 Wildfire
## 47             Winter Storm
## 48           Winter Weather
EVTAB$EVENT <- str_to_upper(EVTAB$EVENT)

Outliers and Data Validity

No systematic evaluation for outliers or data validity is performed. However, in the PROPDMG field a very high value is observed for the entry with REFNUM = “605943”. The REMARKS field was inspected, and based on the several millions $ damage from a flood in Napa valley reported there, the entry in the field PROPDMGEXP was changed from “B” to “M”. It is very likely an erroneous data-entry.

StormData$PROPDMGEXP[StormData$REFNUM == "605943"] <- "M"

Pre-process data

The approach is to compare the overall fatalities, injuries, crop damage, and property damage in a given time-span (1993 to 2011) for the different event types.

Data are pre-processed with the following steps:

  • Strip BGN_DATE of the string ” 0:00:00” and then converted from character to date
  • Select the columns BGN_DATE, STATE, EVTYPE, F, MAG, FATALITIES, INJURIES, PROPDMG, PROPDMGEXP, CROPDMG, CROPDMGEXP, REMARKS, REFNUM
  • Filter for data starting 1993 with the rationale that different event types were only recorded starting in the year 1993
  • Filter only rows that have values over 0 in one of the fields FATALITIES, INJURIES, PROPDMG, or CROPDMG
  • PROPDMGEXP and CROPDMGEXP are converted to numbers with case_match() to allow calculation on crop and property damage
  • Calculate new variables CROP and PROPERTY that contain crop and property damage, respectively
  • Match EVTYPE to the official Event Names using the stringdist::amatch() function. Assuming only typos, the maximum distance was set high (maxDist=30) to ensure all EVTYPE entries were correctly matched
  • Pivot the data.frame to long form to allow for easier creation of lattice panel plots

With these pre-processing steps the data are in the required shape for further analysis.

a <- Sys.time()

StormData$BGN_DATE <- gsub(' 0:00:00','', StormData$BGN_DATE)

StormData$BGN_DATE <- mdy(StormData$BGN_DATE)

SD.Processed <- StormData |> 
                select(BGN_DATE, STATE, EVTYPE, F, MAG, FATALITIES, INJURIES, PROPDMG, PROPDMGEXP, CROPDMG, CROPDMGEXP, REMARKS, REFNUM) |>
                filter(BGN_DATE > "1993-01-01" & (FATALITIES > 0 | INJURIES > 0 | PROPDMG > 0 | CROPDMG > 0)) |>
                mutate(PROPDMGEXP = case_match(PROPDMGEXP,
                                               c("", "+", "-", "?") ~ "1",
                                               as.character(c(1:9)) ~ "10",
                                               c("H", "h") ~ "100",
                                               c("K", "k") ~ "1000",
                                               c("M", "m") ~ "1000000",
                                               c("B", "b") ~ "1000000000",
                                               .default = PROPDMGEXP)) |> 
                mutate(CROPDMGEXP = case_match(CROPDMGEXP,
                                               c("", "+", "-", "?") ~ "1",
                                               as.character(c(1:9)) ~ "10",
                                               c("H", "h") ~ "100",
                                               c("K", "k") ~ "1000",
                                               c("M", "m") ~ "1000000",
                                               c("B", "b") ~ "1000000000",
                                               .default = CROPDMGEXP)) |>
                mutate_at(c("PROPDMGEXP", "CROPDMGEXP"), as.numeric) |>
                mutate(PROPERTY = PROPDMG*PROPDMGEXP/1000000000, CROP = CROPDMG*CROPDMGEXP/1000000000) |>
                mutate(EVTYPE = EVTAB$EVENT[amatch(EVTYPE, EVTAB$EVENT, maxDist=30)]) |>
                pivot_longer(cols = c("FATALITIES", "INJURIES"),
                                                names_to = "CASETYPE",
                                                values_to = "CASES") |>
                pivot_longer(cols = c("PROPERTY", "CROP"),
                                                names_to = "DMGTYPE",
                                                values_to = "DMG")

b <- Sys.time()

b-a
## Time difference of 5.388555 secs

For further analysis, separate data.frames for fatalities (SD.Fatalities) and financial impact (SD.DMG) are created by summarizing the fatalities and injuries, and crop and property damage, respectively.

SD.Fatalities <- SD.Processed |>
                group_by(EVTYPE, CASETYPE) |>
                summarise(sum = sum(CASES), n = n())

SD.DMG <- SD.Processed |>
                group_by(EVTYPE, DMGTYPE) |>
                summarise(sum = sum(DMG), n = n())

Results

Human health impact of weather events

xyplot(as.factor(EVTYPE) ~ sum | CASETYPE, 
                                        data = SD.Fatalities, 
                                        layout = c(2,1),
                                        main = "Impact of weather events on human health",
                                        xlab = "Total number of cases (1993 - 2011)",
                                        ylab = "Event (NWS Event Name)",
                                        scales=list(x = "free", y=list(rot=0)),
                                        panel = function(...){
                                                panel.needle(...)
                                        })

SD.Fatalities |> as.tibble() |> rmarkdown::paged_table()

The plot and the table show the impact of different weather events on human health are measured by overall number of fatalities and injuries in the time interval from 1993 to 2011. The weather events with the highest impact on human health are the following:

  • Fatalities: The highest overall number of 4040 is recorded for the event type “EXCESSIVE HEAT”. Additional event types for which high numbers of fatalities are reported include “TORNADO”, “HEAT”, “FLASH FLOOD”, and “LIGHTNING”, in this order.
  • Injuries: The highest overall number of 46656 is recorded for the event type “TORNADO”. This is followed by event types “FLOOD”, “EXCESSIVE HEAT”, “LIGHTNING”, and “HIGH WIND”, for which also high numbers are reported in the Storm Data.

It is noteworthy that the event type “HURRICANE (TYPHOON)” is only associated with 144 fatalities, despite the fact that for instance Hurricane Katrina in 2005 resulted in 1392 direct and indirect casualties in what was described as the single most devastating natural disaster in the United States (Knabb, Rhome, and Brown 2023). The number of injuries associated to the event type “HURRICANE (TYPHOON)” is 2558, which seems also comparatively low for this event type considering cases such as Hurricane Katrina.

To check whether the fatalities resulting from Hurricane Katrina were recorded under a different event type than “HURRICANE (TYPHOON)” the REMARKS field of the original data set is checked whether it contains the string “Katrina”. Fatalities of these records are summarized.

Kat.N.F <- length(StormData$FATALITIES[grepl("Katrina", StormData$REMARKS)])

Kat.N.F
## [1] 296
Kat.S.F <- sum(StormData$FATALITIES[grepl("Katrina", StormData$REMARKS)])

Kat.S.F
## [1] 31

There are 296 entries with the string “Katrina” in their REMARKS field. These entries record overall 31 fatalities. This number of fatalities is lower than the overall number of fatalities attributed to Hurricane Katrina. This is indicating that the fatalities related to Hurricane Katrina are not recorded in the current version of the Storm Data set.

Economic impact of weather events

xyplot(as.factor(EVTYPE) ~ sum | DMGTYPE, 
                                        data = SD.DMG, 
                                        layout = c(2,1),
                                        main = "Economic impact of weather events",
                                        xlab = "Total economic impact (1993 - 2011) [billion $]",
                                        ylab = "Event (NWS Event Name)",
                                        scales=list(x = "free", y=list(rot=0)),
                                        panel = function(...){
                                                panel.needle(...)
                                        })

SD.DMG |> as.tibble() |> rmarkdown::paged_table()
SD.Processed$REFNUM[SD.Processed$EVTYPE == "STORM SURGE/TIDE" & SD.Processed$DMG == max(SD.Processed$DMG)]
## [1] 577616 577616
SD.Processed$DMG[SD.Processed$REFNUM == "577616"]
## [1] 31.3  0.0 31.3  0.0

The plot and the table above show the economic impact of different weather events as measured by overall cost for crop damage and property damage in the time interval of 1993 to 2011. The weather event types with the highest impact are the following:

  • Crop damage: The highest overall crop damage of 28 billion $ is recorded for the event type “DROUGHT”. It appears to be reasonable that drought is the weather event type with the largest crop damage. This is followed by the event types “FLASH FLOOD”, “FLOOD”, “ICE STORM”, and “SEICHE”, for which also as significant economic impact in terms of overall cost from crop damage is reported.
  • Property damage: The highest overall property damage of 145 billion $ is recorded for the weather event “HURRICANE (TYPHOON)”. This is followed by the event types “STORM SURGE/TIDE”, “FLOOD”, “TORNADO”, and “FLASH FLOOD”.

It is noteworthy that the property damage for one particular event can be distributed over several event types. For instance, there is 31.3 billion $ in property damage recorded under event type “STORM SURGE/TIDE” (REFNUM : 577616) that can be attributed to Hurricane Katrina based on the inspection of the REMARKS field.

Summary

The simple strategy of summarizing fatalities, injuries, crop damage, and property damage over the time period of 1993 to 2011 is used to identify those event types that have the largest human health impact and economic impact. The event types “EXCESSIVE HEAT” and “TORNADO” are identified as having the highest impact on human health in terms of fatalities and injuries, respectively. The highest economic impact is seen for the event types “DROUGHT” and “HURRICANE (TYPHOON)” for crop damage and property damage, respectively. Closer inspection of the Storm Data data set indicated that record of very significant events (e.g. Hurricane Katrina) may be incomplete. Further research with updated versions of the Storm Data data set are needed to verify the conclusions of the report.

Bibliography

Knabb, Richard D., Jamie R. Rhome, and Daniel P. Brown. 2023. “Tropical Cyclone Report Hurricane Katrina.” National Hurricane Center. https://www.nhc.noaa.gov/data/tcr/AL122005_Katrina.pdf.