In this report, we look into the storm database from U.S. National Oceanic and Atmospheric Administration’s (NOAA), containing data from January 1950 to November 2011. We investigate which types of events were the most harmful with respect to population health and which types of events had the greatest economic consequences across the United States in this period. The most harmful events to population health were considered the ones with the greatest number of fatalities and injuries; the events with the greatest economic consequences were considered the ones with the highest amount of damages in US dollars.
Our analysis show that tornadoes were the most harmful events both in terms of fatalities, accounting for more than 37% (5,633) of deaths, and in terms of injuries, accounting for 65% (91,346) of the injuries recorded. In terms of economic consequences, the most harmful events were floods, which accounted for more than 150 billion dollars of damages, 32% of the total damages recorded.
With the raw csv file downloaded to the /Data directory in our project folder, we proceed to loading it into R and converting the date columns into a proper date format:
stormdata <- read.csv("./Data/repdata_data_StormData.csv.bz2")
stormdata$BGN_DATE <- as.Date(stormdata$BGN_DATE, format = "%m/%d/%Y")
stormdata$END_DATE <- as.Date(stormdata$END_DATE, format = "%m/%d/%Y")
Upon looking at the number of event types in the EVTYPE column, we noticed a large number of different event types (985), and the same event type being counted separately due to inconsistencies in case, number of spaces and use of special characters such as - or (). Below is a small example of such inconsistencies:
head(table(stormdata$EVTYPE), n=50)[42:47]
##
## blowing snow Blowing Snow
## 2 3
## BLOWING SNOW BLOWING SNOW- EXTREME WIND CHI
## 12 1
## BLOWING SNOW & EXTREME WIND CH BLOWING SNOW/EXTREME WIND CHIL
## 2 1
To reduce the number of duplicate event types, we replaced the special characters with spaces, converted all entries to lower case, eliminated all whitespace from start and end of each event type and replaced all internal whitespace with a single space.
library(stringr)
paste("Event types before cleanup:", dim(table(stormdata$EVTYPE)))
## [1] "Event types before cleanup: 985"
stormdata$EVTYPE <- tolower(stormdata$EVTYPE)
stormdata$EVTYPE <- gsub("\\/"," ", stormdata$EVTYPE)
stormdata$EVTYPE <- gsub("-"," ", stormdata$EVTYPE)
stormdata$EVTYPE <- gsub("&"," ", stormdata$EVTYPE)
stormdata$EVTYPE <- gsub("\\("," ", stormdata$EVTYPE)
stormdata$EVTYPE <- gsub("\\)"," ", stormdata$EVTYPE)
stormdata$EVTYPE <- str_squish(stormdata$EVTYPE)
paste("Event types after cleanup:", dim(table(stormdata$EVTYPE)))
## [1] "Event types after cleanup: 844"
With this simple cleanup we eliminated 141 duplicate event types. While this is not enough to eliminate all duplicates, additional cleanup would require a more sophisticated approach that proved unnecessary to accomplish the objective of this report, as will be explained later.
We consolidated the number of fatalities by event type, arranging in descending order by number of fatalities, calculating the percentage of fatalities for each event type and the cumulative percentage:
library(dplyr)
library(knitr)
fatal <- stormdata %>% select(EVTYPE, FATALITIES) %>% group_by(EVTYPE) %>% summarize(fatalities = sum(FATALITIES)) %>% arrange(desc(fatalities)) %>% mutate(percentage = fatalities/sum(fatalities)*100, cum_percentage = cumsum(percentage))
For plotting, to improve readability, we created another table, with the following changes:
fatalplot <- head(fatal,n=10)[,1:3]
fatalplot[11, 1] <- "Other 834 event types"
fatalplot[11, 2] <- sum(fatal$fatalities) - sum(fatalplot$fatalities, na.rm = TRUE)
fatalplot[11, 3] <- sum(fatal$percentage) - sum(fatalplot$percentage, na.rm = TRUE)
fatalplot[, 1] <- lapply(fatalplot[, 1], str_to_title)
names(fatalplot) <- c("Event", "Fatalities", "% of all Fatalities")
The plot to analyse the fatalities is a bar plot with the top 10 types of events with the highest number of fatalities plus the other types added together in a single event type:
library(ggplot2)
g1 <- ggplot(fatalplot, aes(reorder(Event, Fatalities), Fatalities))
g1 + geom_col() + coord_flip() + labs(title="Events with the highest number of fatalities", y="Number of fatalities", x="Event Type")
For injuries, we followed the same procedure we did with fatalities: we consolidated the number of injuries by event type, arranging in descending order by number of injuries, calculating the percentage of injuries for each event type and the cumulative percentage:
injury <- stormdata %>% select(EVTYPE, INJURIES) %>% group_by(EVTYPE) %>% summarize(injuries = sum(INJURIES)) %>% arrange(desc(injuries)) %>% mutate(percentage = injuries/sum(injuries)*100, cum_percentage = cumsum(percentage))
The table with the top 10 injuries also follows the same processing we did for the fatalities:
injuryplot <- head(injury, n=10)[,1:3]
injuryplot[11, 1] <- "Other 834 event types"
injuryplot[11, 2] <- sum(injury$injuries) - sum(injuryplot$injuries, na.rm = TRUE)
injuryplot[11, 3] <- sum(injury$percentage) - sum(injuryplot$percentage, na.rm = TRUE)
injuryplot[, 1] <- lapply(injuryplot[, 1], str_to_title)
names(injuryplot) <- c("Event", "Injuries", "% of all Injuries")
The plot code also follows the same processing we did for the fatalities:
g2 <- ggplot(injuryplot, aes(reorder(Event, Injuries), Injuries))
g2 + geom_col() + coord_flip() + labs(title="Events with the highest number of injuries", y="Number of injuries", x="Event Type")
To calculate the economic consequences of each event, we added the property damages to the crop damages. Following the work done by Soesilo Wijono and Eddie Song that can be found here, to calculate the property (and crop) damages, we multiplied the value in the column PROPDMG (CROPDMG for crop) to a number dictated by the value in column PROPDMGEXP (CROPDMGEXP for crop), according to the table below:
| Value in PROPDMGEXP | Multiply by |
|---|---|
| B,b | \(10^9\) |
| M,m | \(10^6\) |
| K,k | \(10^3\) |
| H,h | \(10^2\) |
| Numeric (0-8) | \(10\) |
| + | \(1\) |
| -, ?, blank or empty | \(0\) |
The code below converts the values in PROPDMGEXP and CROPDMGEXP to the multipliers they represent, convert these columns to numeric and replace all NAs (which were empty values before the conversion to numeric) with zero, to avoid getting NAs in events where only one value is present between property and crop damages. The output of the code below shows that all NAs were originally empty spaces.
# convert the values in PROPDMGEXP and CROPDMGEXP to the multipliers they represent
stormdata$PROPDMGEXP <- gsub("[0-8]","10", stormdata$PROPDMGEXP)
stormdata$PROPDMGEXP <- gsub("[\\?\\-]","0", stormdata$PROPDMGEXP)
stormdata$PROPDMGEXP <- gsub("\\+","1", stormdata$PROPDMGEXP)
stormdata$PROPDMGEXP <- gsub("[Hh]","100", stormdata$PROPDMGEXP)
stormdata$PROPDMGEXP <- gsub("[Kk]","1000", stormdata$PROPDMGEXP)
stormdata$PROPDMGEXP <- gsub("[Mm]","1000000", stormdata$PROPDMGEXP)
stormdata$PROPDMGEXP <- gsub("[Bb]","1000000000", stormdata$PROPDMGEXP)
stormdata$CROPDMGEXP <- gsub("[0-8]","10", stormdata$CROPDMGEXP)
stormdata$CROPDMGEXP <- gsub("[\\?\\-]","0", stormdata$CROPDMGEXP)
stormdata$CROPDMGEXP <- gsub("\\+","1", stormdata$CROPDMGEXP)
stormdata$CROPDMGEXP <- gsub("[Hh]","100", stormdata$CROPDMGEXP)
stormdata$CROPDMGEXP <- gsub("[Kk]","1000", stormdata$CROPDMGEXP)
stormdata$CROPDMGEXP <- gsub("[Mm]","1000000", stormdata$CROPDMGEXP)
stormdata$CROPDMGEXP <- gsub("[Bb]","1000000000", stormdata$CROPDMGEXP)
# convert the columns to numeric
paste("Empty values in PROPDMGEXP before convertion to numeric:", length(stormdata$PROPDMGEXP[stormdata$PROPDMGEXP==""]))
## [1] "Empty values in PROPDMGEXP before convertion to numeric: 465934"
paste("Empty values in CROPDMGEXP before convertion to numeric:", length(stormdata$CROPDMGEXP[stormdata$CROPDMGEXP==""]))
## [1] "Empty values in CROPDMGEXP before convertion to numeric: 618413"
stormdata$PROPDMGEXP <- as.numeric(stormdata$PROPDMGEXP)
stormdata$CROPDMGEXP <- as.numeric(stormdata$CROPDMGEXP)
paste("NA values in PROPDMGEXP after convertion to numeric:", sum(is.na(stormdata$PROPDMGEXP)))
## [1] "NA values in PROPDMGEXP after convertion to numeric: 465934"
paste("NA values in CROPDMGEXP after convertion to numeric:", sum(is.na(stormdata$CROPDMGEXP)))
## [1] "NA values in CROPDMGEXP after convertion to numeric: 618413"
# replace NAs with 0
stormdata$PROPDMGEXP[is.na(stormdata$PROPDMGEXP)] <- 0
stormdata$CROPDMGEXP[is.na(stormdata$CROPDMGEXP)] <- 0
Finally, we multiply PROPDMGEXP and CROPDMGEXP by the values in PROPDMG and CROPDMG respectively, group the data by event type, add property to crop damages to obtain the total damages, order from the highest to lowest total damages, calculate the percentages and cumulative percentages:
damages <- stormdata %>% mutate(propdam = PROPDMG*PROPDMGEXP, cropdam = CROPDMG*CROPDMGEXP) %>% group_by(EVTYPE) %>% summarize(propdam = sum(propdam), cropdam = sum(cropdam), totaldam = propdam + cropdam) %>% arrange(desc(totaldam)) %>% mutate(percentage = totaldam/sum(totaldam)*100, cum_percentage = cumsum(percentage))
Here we also made a new table to serve as plot data, to make it more readable. It contains the following changes:
library(reshape2)
damagesplot <- head(damages,n=10)[,1:3]
damagesplot[11, 1] <- "Other 834 event types"
damagesplot[11, 2] <- sum(damages$propdam) - sum(damagesplot$propdam, na.rm = TRUE)
damagesplot[11, 3] <- sum(damages$cropdam) - sum(damages$cropdam, na.rm = TRUE)
damagesplot[, 1] <- lapply(damagesplot[, 1], str_to_title)
damagesplot <- mutate(damagesplot, propdam=propdam/10^9, cropdam=cropdam/10^9)
names(damagesplot) <- c("Event", "Property Damages", "Crop Damages")
damagesplot <- melt(damagesplot, id="Event", measure.vars=c("Property Damages", "Crop Damages"))
The plot for damages is a bar plot in which for each event type, the property damages and the crop damages are stacked:
g3 <- ggplot(damagesplot, aes(reorder(Event, value), value, fill=variable))
g3 + geom_col() + coord_flip() + labs(title="Events with the highest economic consequences", y="Damages (Billion USD)", x="Event Type") + scale_fill_discrete(name = "Damage type")
There are two ways we can measure the most harmful types of events from this dataset: from the number of injuries and from number of fatalities. While it’s obvious that injuries are less serious than fatalities, if an event has a high number of injuries, even when there are no fatalities it can be very harmful. Therefore, we’ll look at the events with the highest number of fatalities and also the ones with the highest number of injuries separately.
Let’s look at fatalities first. Below is table of the top 10 types of events with the highest number of fatalities:
library(knitr)
names(fatal) <- c("Event Type", "Fatalities", "% of Total Fatalities", "Cumulative %")
fatal$Fatalities <- prettyNum(fatal$Fatalities, big.mark=",", scientific=FALSE)
kable(head(fatal, n=10), digits=1, row.names=TRUE)
| Event Type | Fatalities | % of Total Fatalities | Cumulative % | |
|---|---|---|---|---|
| 1 | tornado | 5,633 | 37.2 | 37.2 |
| 2 | excessive heat | 1,903 | 12.6 | 49.8 |
| 3 | flash flood | 978 | 6.5 | 56.2 |
| 4 | heat | 937 | 6.2 | 62.4 |
| 5 | lightning | 816 | 5.4 | 67.8 |
| 6 | tstm wind | 504 | 3.3 | 71.1 |
| 7 | flood | 470 | 3.1 | 74.2 |
| 8 | rip current | 368 | 2.4 | 76.7 |
| 9 | high wind | 248 | 1.6 | 78.3 |
| 10 | avalanche | 224 | 1.5 | 79.8 |
We can see that tornadoes are the most harmful events in terms of fatalities, accounting for 37.2% of deaths, followed by excessive heat with 12.6%. Therefore, the top two events account for almost 50% of the deaths in the dataset, and the top 10 accounts for 80%.
The plot below shows the top 10 types of events individually; the fatalities of the remaining events were added up and included as the event type “Other 834 event types”
The Other event types come in second, representing 20% of all fatalities.
Now let’s look at the table of the top 10 types of events with the highest number of injuries:
names(injury) <- c("Event Type", "Injuries", "% of Total Injuries", "Cumulative %")
injury$Injuries <- prettyNum(injury$Injuries, big.mark=",", scientific=FALSE)
kable(head(injury, n=10), digits=1, row.names=TRUE)
| Event Type | Injuries | % of Total Injuries | Cumulative % | |
|---|---|---|---|---|
| 1 | tornado | 91,346 | 65.0 | 65.0 |
| 2 | tstm wind | 6,957 | 5.0 | 70.0 |
| 3 | flood | 6,789 | 4.8 | 74.8 |
| 4 | excessive heat | 6,525 | 4.6 | 79.4 |
| 5 | lightning | 5,230 | 3.7 | 83.1 |
| 6 | heat | 2,100 | 1.5 | 84.6 |
| 7 | ice storm | 1,975 | 1.4 | 86.0 |
| 8 | flash flood | 1,777 | 1.3 | 87.3 |
| 9 | thunderstorm wind | 1,488 | 1.1 | 88.4 |
| 10 | hail | 1,361 | 1.0 | 89.3 |
From an injuries standpoint, tornadoes is also the most harmful type of event, account for 65% of the injuries recorded. Tstm wind is a distant second with 5% of injuries. Excessive heat, which was second by fatalities, is in 4th by injuries. The top 10 accounts for 89% of injuries.
The plot below shows the top 10 types of events individually; the injuries of the remaining events were added up and included as the event type “Other 834 Event Types”
Therefore, both from the perspective of fatalities and injuries, tornadoes is the most harmful type of event.
Considering the high proportion of both injuries and fatalities that the top 10 events of each category represents, from the perspective of finding the most harmful events, it’s not necessary to delve further than the top 10 or 20 types of events at most. However, for a more thorough analysis, it should be considered a review of the types of events for more consolidation of the categories and a better view of the contribution of each event.
To calculate the economic consequences of each type of event, we added the property damages to the crop damages. Below is the table with the top 10 highest amount of damages:
damages$propdam <- prettyNum(damages$propdam, big.mark=",", scientific=FALSE)
damages$cropdam <- prettyNum(damages$cropdam, big.mark=",", scientific=FALSE)
damages$totaldam <- prettyNum(damages$totaldam, big.mark=",", scientific=FALSE)
names(damages) <- c("Event Type", "Property Damages", "Crop Damages", "Total Damages", "% of Total Damages", "Cumulative %")
kable(head(damages, n=10), digits=1, row.names=TRUE)
| Event Type | Property Damages | Crop Damages | Total Damages | % of Total Damages | Cumulative % | |
|---|---|---|---|---|---|---|
| 1 | flood | 144,657,709,800 | 5,661,968,450 | 150,319,678,250 | 31.6 | 31.6 |
| 2 | hurricane typhoon | 69,305,840,000 | 2,607,872,800 | 71,913,712,800 | 15.1 | 46.6 |
| 3 | tornado | 56,937,162,897 | 414,954,710 | 57,352,117,607 | 12.0 | 58.7 |
| 4 | storm surge | 43,323,536,000 | 5,000 | 43,323,541,000 | 9.1 | 67.8 |
| 5 | hail | 15,732,269,877 | 3,025,954,650 | 18,758,224,527 | 3.9 | 71.7 |
| 6 | flash flood | 16,141,365,011 | 1,421,317,100 | 17,562,682,111 | 3.7 | 75.4 |
| 7 | drought | 1,046,106,000 | 13,972,566,000 | 15,018,672,000 | 3.2 | 78.6 |
| 8 | hurricane | 11,868,319,010 | 2,741,910,000 | 14,610,229,010 | 3.1 | 81.6 |
| 9 | river flood | 5,118,945,500 | 5,029,459,000 | 10,148,404,500 | 2.1 | 83.8 |
| 10 | ice storm | 3,944,928,310 | 5,022,113,500 | 8,967,041,810 | 1.9 | 85.6 |
We see that flood is the event that causes most damages, making 32% of all damages. Hurricane typhoon comes second with 15%. The top 10 events account for 86% of all damages. It should be noted that there are multiple events in the top 10 that could potentially be merged for damages estimation due to their similarities. For example, the if the events hurricane typhoon, tornado and hurricane are grouped, they represent 30% of damages, getting much closer to the flood damages. This grouping should be considered by subject matter experts when analyzing the numbers.
The plot below shows the top 10 types of events individually; the total amount of damages of the remaining events were added up and included as the event type “Other 834 Event Types”:
We can also see that in the top 10 events the amount of property damages is much higher than crop damages.
Similarly to the analysis in fatalities and injuries, considering the high proportion of damages that the top 10 events of each category represents, from the perspective of finding the events with greatest economic consequences, it’s not necessary to delve further than the top 10 or 20 events at most. However, for a more thorough analysis, it should be considered a review of the 844 different events for some consolidation of the categories and a better view of the contribution of each event.