Synopsis

In this report, we look into the storm database from U.S. National Oceanic and Atmospheric Administration’s (NOAA), containing data from January 1950 to November 2011. We investigate which types of events were the most harmful with respect to population health and which types of events had the greatest economic consequences across the United States in this period. The most harmful events to population health were considered the ones with the greatest number of fatalities and injuries; the events with the greatest economic consequences were considered the ones with the highest amount of damages in US dollars.

Our analysis show that tornadoes were the most harmful events both in terms of fatalities, accounting for more than 37% (5,633) of deaths, and in terms of injuries, accounting for 65% (91,346) of the injuries recorded. In terms of economic consequences, the most harmful events were floods, which accounted for more than 150 billion dollars of damages, 32% of the total damages recorded.

Data processing

Loading and preprocessing

With the raw csv file downloaded to the /Data directory in our project folder, we proceed to loading it into R and converting the date columns into a proper date format:

stormdata <- read.csv("./Data/repdata_data_StormData.csv.bz2")
stormdata$BGN_DATE <- as.Date(stormdata$BGN_DATE, format = "%m/%d/%Y")
stormdata$END_DATE <- as.Date(stormdata$END_DATE, format = "%m/%d/%Y")

Event type cleanup

Upon looking at the number of event types in the EVTYPE column, we noticed a large number of different event types (985), and the same event type being counted separately due to inconsistencies in case, number of spaces and use of special characters such as - or (). Below is a small example of such inconsistencies:

head(table(stormdata$EVTYPE), n=50)[42:47]
## 
##                   blowing snow                   Blowing Snow 
##                              2                              3 
##                   BLOWING SNOW BLOWING SNOW- EXTREME WIND CHI 
##                             12                              1 
## BLOWING SNOW & EXTREME WIND CH BLOWING SNOW/EXTREME WIND CHIL 
##                              2                              1

To reduce the number of duplicate event types, we replaced the special characters with spaces, converted all entries to lower case, eliminated all whitespace from start and end of each event type and replaced all internal whitespace with a single space.

library(stringr)
paste("Event types before cleanup:", dim(table(stormdata$EVTYPE)))
## [1] "Event types before cleanup: 985"
stormdata$EVTYPE <- tolower(stormdata$EVTYPE)
stormdata$EVTYPE <- gsub("\\/"," ", stormdata$EVTYPE)
stormdata$EVTYPE <- gsub("-"," ", stormdata$EVTYPE)
stormdata$EVTYPE <- gsub("&"," ", stormdata$EVTYPE)
stormdata$EVTYPE <- gsub("\\("," ", stormdata$EVTYPE)
stormdata$EVTYPE <- gsub("\\)"," ", stormdata$EVTYPE)
stormdata$EVTYPE <- str_squish(stormdata$EVTYPE)
paste("Event types after cleanup:", dim(table(stormdata$EVTYPE)))
## [1] "Event types after cleanup: 844"

With this simple cleanup we eliminated 141 duplicate event types. While this is not enough to eliminate all duplicates, additional cleanup would require a more sophisticated approach that proved unnecessary to accomplish the objective of this report, as will be explained later.

Fatalities processing

We consolidated the number of fatalities by event type, arranging in descending order by number of fatalities, calculating the percentage of fatalities for each event type and the cumulative percentage:

library(dplyr)
library(knitr)
fatal <- stormdata %>% select(EVTYPE, FATALITIES) %>% group_by(EVTYPE) %>% summarize(fatalities = sum(FATALITIES)) %>% arrange(desc(fatalities)) %>% mutate(percentage = fatalities/sum(fatalities)*100, cum_percentage = cumsum(percentage))

For plotting, to improve readability, we created another table, with the following changes:

  • pick only the 10 events with the most number of fatalities
  • add the fatalities of all the remaining events as “Other 834 event types”
fatalplot <- head(fatal,n=10)[,1:3]
fatalplot[11, 1] <- "Other 834 event types"
fatalplot[11, 2] <- sum(fatal$fatalities) - sum(fatalplot$fatalities, na.rm = TRUE)
fatalplot[11, 3] <- sum(fatal$percentage) - sum(fatalplot$percentage, na.rm = TRUE)
fatalplot[, 1] <- lapply(fatalplot[, 1], str_to_title)
names(fatalplot) <- c("Event", "Fatalities", "% of all Fatalities")

The plot to analyse the fatalities is a bar plot with the top 10 types of events with the highest number of fatalities plus the other types added together in a single event type:

library(ggplot2)
g1 <- ggplot(fatalplot, aes(reorder(Event, Fatalities), Fatalities))
g1 + geom_col() + coord_flip() + labs(title="Events with the highest number of fatalities", y="Number of fatalities", x="Event Type")

Injuries processing

For injuries, we followed the same procedure we did with fatalities: we consolidated the number of injuries by event type, arranging in descending order by number of injuries, calculating the percentage of injuries for each event type and the cumulative percentage:

injury <- stormdata %>% select(EVTYPE, INJURIES) %>% group_by(EVTYPE) %>% summarize(injuries = sum(INJURIES)) %>% arrange(desc(injuries)) %>% mutate(percentage = injuries/sum(injuries)*100, cum_percentage = cumsum(percentage))

The table with the top 10 injuries also follows the same processing we did for the fatalities:

injuryplot <- head(injury, n=10)[,1:3]
injuryplot[11, 1] <- "Other 834 event types"
injuryplot[11, 2] <- sum(injury$injuries) - sum(injuryplot$injuries, na.rm = TRUE)
injuryplot[11, 3] <- sum(injury$percentage) - sum(injuryplot$percentage, na.rm = TRUE)
injuryplot[, 1] <- lapply(injuryplot[, 1], str_to_title)
names(injuryplot) <- c("Event", "Injuries", "% of all Injuries")

The plot code also follows the same processing we did for the fatalities:

g2 <- ggplot(injuryplot, aes(reorder(Event, Injuries), Injuries))
g2 + geom_col() + coord_flip() + labs(title="Events with the highest number of injuries", y="Number of injuries", x="Event Type")

Damages (economic consequences) processing

To calculate the economic consequences of each event, we added the property damages to the crop damages. Following the work done by Soesilo Wijono and Eddie Song that can be found here, to calculate the property (and crop) damages, we multiplied the value in the column PROPDMG (CROPDMG for crop) to a number dictated by the value in column PROPDMGEXP (CROPDMGEXP for crop), according to the table below:

Value in PROPDMGEXP Multiply by
B,b \(10^9\)
M,m \(10^6\)
K,k \(10^3\)
H,h \(10^2\)
Numeric (0-8) \(10\)
+ \(1\)
-, ?, blank or empty \(0\)

The code below converts the values in PROPDMGEXP and CROPDMGEXP to the multipliers they represent, convert these columns to numeric and replace all NAs (which were empty values before the conversion to numeric) with zero, to avoid getting NAs in events where only one value is present between property and crop damages. The output of the code below shows that all NAs were originally empty spaces.

# convert the values in PROPDMGEXP and CROPDMGEXP to the multipliers they represent
stormdata$PROPDMGEXP <- gsub("[0-8]","10", stormdata$PROPDMGEXP)
stormdata$PROPDMGEXP <- gsub("[\\?\\-]","0", stormdata$PROPDMGEXP)
stormdata$PROPDMGEXP <- gsub("\\+","1", stormdata$PROPDMGEXP)
stormdata$PROPDMGEXP <- gsub("[Hh]","100", stormdata$PROPDMGEXP)
stormdata$PROPDMGEXP <- gsub("[Kk]","1000", stormdata$PROPDMGEXP)
stormdata$PROPDMGEXP <- gsub("[Mm]","1000000", stormdata$PROPDMGEXP)
stormdata$PROPDMGEXP <- gsub("[Bb]","1000000000", stormdata$PROPDMGEXP)

stormdata$CROPDMGEXP <- gsub("[0-8]","10", stormdata$CROPDMGEXP)
stormdata$CROPDMGEXP <- gsub("[\\?\\-]","0", stormdata$CROPDMGEXP)
stormdata$CROPDMGEXP <- gsub("\\+","1", stormdata$CROPDMGEXP)
stormdata$CROPDMGEXP <- gsub("[Hh]","100", stormdata$CROPDMGEXP)
stormdata$CROPDMGEXP <- gsub("[Kk]","1000", stormdata$CROPDMGEXP)
stormdata$CROPDMGEXP <- gsub("[Mm]","1000000", stormdata$CROPDMGEXP)
stormdata$CROPDMGEXP <- gsub("[Bb]","1000000000", stormdata$CROPDMGEXP)

# convert the columns to numeric
paste("Empty values in PROPDMGEXP before convertion to numeric:", length(stormdata$PROPDMGEXP[stormdata$PROPDMGEXP==""]))
## [1] "Empty values in PROPDMGEXP before convertion to numeric: 465934"
paste("Empty values in CROPDMGEXP before convertion to numeric:", length(stormdata$CROPDMGEXP[stormdata$CROPDMGEXP==""]))
## [1] "Empty values in CROPDMGEXP before convertion to numeric: 618413"
stormdata$PROPDMGEXP <- as.numeric(stormdata$PROPDMGEXP)
stormdata$CROPDMGEXP <- as.numeric(stormdata$CROPDMGEXP)
paste("NA values in PROPDMGEXP after convertion to numeric:", sum(is.na(stormdata$PROPDMGEXP)))
## [1] "NA values in PROPDMGEXP after convertion to numeric: 465934"
paste("NA values in CROPDMGEXP after convertion to numeric:", sum(is.na(stormdata$CROPDMGEXP)))
## [1] "NA values in CROPDMGEXP after convertion to numeric: 618413"
# replace NAs with 0
stormdata$PROPDMGEXP[is.na(stormdata$PROPDMGEXP)] <- 0
stormdata$CROPDMGEXP[is.na(stormdata$CROPDMGEXP)] <- 0

Finally, we multiply PROPDMGEXP and CROPDMGEXP by the values in PROPDMG and CROPDMG respectively, group the data by event type, add property to crop damages to obtain the total damages, order from the highest to lowest total damages, calculate the percentages and cumulative percentages:

damages <- stormdata %>% mutate(propdam = PROPDMG*PROPDMGEXP, cropdam = CROPDMG*CROPDMGEXP) %>% group_by(EVTYPE) %>% summarize(propdam = sum(propdam), cropdam = sum(cropdam), totaldam = propdam + cropdam)  %>% arrange(desc(totaldam)) %>% mutate(percentage = totaldam/sum(totaldam)*100, cum_percentage = cumsum(percentage))

Here we also made a new table to serve as plot data, to make it more readable. It contains the following changes:

  • includes only the 10 types of events with the highest amount of damages individually
  • the damages of all remaining types of events are added and included as a single type of event named “Other 834 event types”
  • the damage amount is divided by \(10^9\), so that the values are in billion dollars, to improve readability
  • the dataframe is melt so that the value of crop and property damages are in the same column, in order to plot them as stacked columns
library(reshape2)
damagesplot <- head(damages,n=10)[,1:3]
damagesplot[11, 1] <- "Other 834 event types"
damagesplot[11, 2] <- sum(damages$propdam) - sum(damagesplot$propdam, na.rm = TRUE)
damagesplot[11, 3] <- sum(damages$cropdam) - sum(damages$cropdam, na.rm = TRUE)
damagesplot[, 1] <- lapply(damagesplot[, 1], str_to_title)
damagesplot <- mutate(damagesplot, propdam=propdam/10^9, cropdam=cropdam/10^9)
names(damagesplot) <- c("Event", "Property Damages", "Crop Damages")
damagesplot <- melt(damagesplot, id="Event", measure.vars=c("Property Damages", "Crop Damages"))

The plot for damages is a bar plot in which for each event type, the property damages and the crop damages are stacked:

g3 <- ggplot(damagesplot, aes(reorder(Event, value), value, fill=variable))
g3 + geom_col() + coord_flip() + labs(title="Events with the highest economic consequences", y="Damages (Billion USD)", x="Event Type") +  scale_fill_discrete(name = "Damage type") 

Results

Most harmful types of events with respect to population health

There are two ways we can measure the most harmful types of events from this dataset: from the number of injuries and from number of fatalities. While it’s obvious that injuries are less serious than fatalities, if an event has a high number of injuries, even when there are no fatalities it can be very harmful. Therefore, we’ll look at the events with the highest number of fatalities and also the ones with the highest number of injuries separately.

Let’s look at fatalities first. Below is table of the top 10 types of events with the highest number of fatalities:

library(knitr)
names(fatal) <- c("Event Type", "Fatalities", "% of Total Fatalities", "Cumulative %")
fatal$Fatalities <- prettyNum(fatal$Fatalities, big.mark=",", scientific=FALSE)
kable(head(fatal, n=10), digits=1, row.names=TRUE)
Event Type Fatalities % of Total Fatalities Cumulative %
1 tornado 5,633 37.2 37.2
2 excessive heat 1,903 12.6 49.8
3 flash flood 978 6.5 56.2
4 heat 937 6.2 62.4
5 lightning 816 5.4 67.8
6 tstm wind 504 3.3 71.1
7 flood 470 3.1 74.2
8 rip current 368 2.4 76.7
9 high wind 248 1.6 78.3
10 avalanche 224 1.5 79.8

We can see that tornadoes are the most harmful events in terms of fatalities, accounting for 37.2% of deaths, followed by excessive heat with 12.6%. Therefore, the top two events account for almost 50% of the deaths in the dataset, and the top 10 accounts for 80%.

The plot below shows the top 10 types of events individually; the fatalities of the remaining events were added up and included as the event type “Other 834 event types”

The Other event types come in second, representing 20% of all fatalities.

Now let’s look at the table of the top 10 types of events with the highest number of injuries:

names(injury) <- c("Event Type", "Injuries", "% of Total Injuries", "Cumulative %")
injury$Injuries <- prettyNum(injury$Injuries, big.mark=",", scientific=FALSE)
kable(head(injury, n=10), digits=1, row.names=TRUE)
Event Type Injuries % of Total Injuries Cumulative %
1 tornado 91,346 65.0 65.0
2 tstm wind 6,957 5.0 70.0
3 flood 6,789 4.8 74.8
4 excessive heat 6,525 4.6 79.4
5 lightning 5,230 3.7 83.1
6 heat 2,100 1.5 84.6
7 ice storm 1,975 1.4 86.0
8 flash flood 1,777 1.3 87.3
9 thunderstorm wind 1,488 1.1 88.4
10 hail 1,361 1.0 89.3

From an injuries standpoint, tornadoes is also the most harmful type of event, account for 65% of the injuries recorded. Tstm wind is a distant second with 5% of injuries. Excessive heat, which was second by fatalities, is in 4th by injuries. The top 10 accounts for 89% of injuries.

The plot below shows the top 10 types of events individually; the injuries of the remaining events were added up and included as the event type “Other 834 Event Types”

Therefore, both from the perspective of fatalities and injuries, tornadoes is the most harmful type of event.

Considering the high proportion of both injuries and fatalities that the top 10 events of each category represents, from the perspective of finding the most harmful events, it’s not necessary to delve further than the top 10 or 20 types of events at most. However, for a more thorough analysis, it should be considered a review of the types of events for more consolidation of the categories and a better view of the contribution of each event.

Types of events with the greatest economic consequences

To calculate the economic consequences of each type of event, we added the property damages to the crop damages. Below is the table with the top 10 highest amount of damages:

damages$propdam <- prettyNum(damages$propdam, big.mark=",", scientific=FALSE)
damages$cropdam <- prettyNum(damages$cropdam, big.mark=",", scientific=FALSE)
damages$totaldam <- prettyNum(damages$totaldam, big.mark=",", scientific=FALSE)
names(damages) <- c("Event Type", "Property Damages", "Crop Damages", "Total Damages", "% of Total Damages", "Cumulative %")
kable(head(damages, n=10), digits=1, row.names=TRUE)
Event Type Property Damages Crop Damages Total Damages % of Total Damages Cumulative %
1 flood 144,657,709,800 5,661,968,450 150,319,678,250 31.6 31.6
2 hurricane typhoon 69,305,840,000 2,607,872,800 71,913,712,800 15.1 46.6
3 tornado 56,937,162,897 414,954,710 57,352,117,607 12.0 58.7
4 storm surge 43,323,536,000 5,000 43,323,541,000 9.1 67.8
5 hail 15,732,269,877 3,025,954,650 18,758,224,527 3.9 71.7
6 flash flood 16,141,365,011 1,421,317,100 17,562,682,111 3.7 75.4
7 drought 1,046,106,000 13,972,566,000 15,018,672,000 3.2 78.6
8 hurricane 11,868,319,010 2,741,910,000 14,610,229,010 3.1 81.6
9 river flood 5,118,945,500 5,029,459,000 10,148,404,500 2.1 83.8
10 ice storm 3,944,928,310 5,022,113,500 8,967,041,810 1.9 85.6

We see that flood is the event that causes most damages, making 32% of all damages. Hurricane typhoon comes second with 15%. The top 10 events account for 86% of all damages. It should be noted that there are multiple events in the top 10 that could potentially be merged for damages estimation due to their similarities. For example, the if the events hurricane typhoon, tornado and hurricane are grouped, they represent 30% of damages, getting much closer to the flood damages. This grouping should be considered by subject matter experts when analyzing the numbers.

The plot below shows the top 10 types of events individually; the total amount of damages of the remaining events were added up and included as the event type “Other 834 Event Types”:

We can also see that in the top 10 events the amount of property damages is much higher than crop damages.

Similarly to the analysis in fatalities and injuries, considering the high proportion of damages that the top 10 events of each category represents, from the perspective of finding the events with greatest economic consequences, it’s not necessary to delve further than the top 10 or 20 events at most. However, for a more thorough analysis, it should be considered a review of the 844 different events for some consolidation of the categories and a better view of the contribution of each event.