The Most Harmful or Costly Weather Events Between 1950 and 2011
Reproducible Research Course Project 2
Synopsis
Storms and other severe weather events can result in fatalities, injuries, and property damage, and preventing such outcomes to the extent possible is a key concern. Our hypothesis is that some weather events are more harmful, with respect to population health, while some are more costly, with respect to economic consequence. To investigate this hypothesis, we explored the data from the U.S. National Oceanic and Atmospheric Administration (NOAA). From these data (between 1950 and 2011), we found that tornado is the most harmful weather event, while excessive heat and heat have the highest fatality rate among the top 5 weather events, and flood has the greatest economic consequences.
Data Processing
I. Prepare
Loading the packages
library(tidyverse)
library(scales)Reading the data file
Reading the data file from the website, the data come in the form of a comma-separated-value file compressed via the bzip2 algorithm.
urlFile <- "https://d396qusza40orc.cloudfront.net/repdata%2Fdata%2FStormData.csv.bz2"
download.file(urlFile, destfile="/Users/Doris/Desktop/Data Science Specialization/Assignments/NOAA_storm/StormData.csv.bz2", method="curl")
storm_data <- read.csv("StormData.csv.bz2")After reading in the data, we check the first few rows of the data set to get a general idea of it.
dim(storm_data)## [1] 902297 37head(storm_data)## STATE__ BGN_DATE BGN_TIME TIME_ZONE COUNTY COUNTYNAME STATE EVTYPE
## 1 1 4/18/1950 0:00:00 0130 CST 97 MOBILE AL TORNADO
## 2 1 4/18/1950 0:00:00 0145 CST 3 BALDWIN AL TORNADO
## 3 1 2/20/1951 0:00:00 1600 CST 57 FAYETTE AL TORNADO
## 4 1 6/8/1951 0:00:00 0900 CST 89 MADISON AL TORNADO
## 5 1 11/15/1951 0:00:00 1500 CST 43 CULLMAN AL TORNADO
## 6 1 11/15/1951 0:00:00 2000 CST 77 LAUDERDALE AL TORNADO
## BGN_RANGE BGN_AZI BGN_LOCATI END_DATE END_TIME COUNTY_END COUNTYENDN
## 1 0 0 NA
## 2 0 0 NA
## 3 0 0 NA
## 4 0 0 NA
## 5 0 0 NA
## 6 0 0 NA
## END_RANGE END_AZI END_LOCATI LENGTH WIDTH F MAG FATALITIES INJURIES PROPDMG
## 1 0 14.0 100 3 0 0 15 25.0
## 2 0 2.0 150 2 0 0 0 2.5
## 3 0 0.1 123 2 0 0 2 25.0
## 4 0 0.0 100 2 0 0 2 2.5
## 5 0 0.0 150 2 0 0 2 2.5
## 6 0 1.5 177 2 0 0 6 2.5
## PROPDMGEXP CROPDMG CROPDMGEXP WFO STATEOFFIC ZONENAMES LATITUDE LONGITUDE
## 1 K 0 3040 8812
## 2 K 0 3042 8755
## 3 K 0 3340 8742
## 4 K 0 3458 8626
## 5 K 0 3412 8642
## 6 K 0 3450 8748
## LATITUDE_E LONGITUDE_ REMARKS REFNUM
## 1 3051 8806 1
## 2 0 0 2
## 3 0 0 3
## 4 0 0 4
## 5 0 0 5
## 6 0 0 6There are 902297 observations with 37 variables, but what we are interested in by now are only a few columns related to fatalities, injuries, and damages.
II. Process
Subsetting the data frame
First, we subset the data frame with the columns that we are interested in.
storm_sub <- storm_data %>%
select(EVTYPE, FATALITIES, INJURIES, PROPDMG, PROPDMGEXP, CROPDMG, CROPDMGEXP)Checking summary of the data and missing values
summary(storm_sub)## EVTYPE FATALITIES INJURIES PROPDMG
## Length:902297 Min. : 0.0000 Min. : 0.0000 Min. : 0.00
## Class :character 1st Qu.: 0.0000 1st Qu.: 0.0000 1st Qu.: 0.00
## Mode :character Median : 0.0000 Median : 0.0000 Median : 0.00
## Mean : 0.0168 Mean : 0.1557 Mean : 12.06
## 3rd Qu.: 0.0000 3rd Qu.: 0.0000 3rd Qu.: 0.50
## Max. :583.0000 Max. :1700.0000 Max. :5000.00
## PROPDMGEXP CROPDMG CROPDMGEXP
## Length:902297 Min. : 0.000 Length:902297
## Class :character 1st Qu.: 0.000 Class :character
## Mode :character Median : 0.000 Mode :character
## Mean : 1.527
## 3rd Qu.: 0.000
## Max. :990.000colSums(is.na(storm_sub))## EVTYPE FATALITIES INJURIES PROPDMG PROPDMGEXP CROPDMG CROPDMGEXP
## 0 0 0 0 0 0 0There is no missing value in “storm_sub” data frame.
Processing variable “EVTYPE”
We check the unique values of event types in column “EVTYPE”.
unique(storm_sub$EVTYPE) It shows 985 unique values, while the number of event types are 48 in the storm data documentation, so we clean the data accordingly as following:
- Translate all characters to upper case, and remove whitespace from the strings.
storm_sub$EVTYPE <- toupper(storm_sub$EVTYPE)
storm_sub$EVTYPE <- str_trim(storm_sub$EVTYPE)- Create a new column “EVENTS”, fill in the values with the 48 event types from the documentation by searching keywords in column “EVTYPE”. Those that are not in the scope of keywords are classified as “others”.
storm_sub$EVENTS <- "OTHERS"
storm_sub$EVENTS[grep("ASTRONOMICAL", storm_sub$EVTYPE)] <- "ASTRONOMICAL LOW TIDE"
storm_sub$EVENTS[grep("AVALANCHE", storm_sub$EVTYPE)] <- "AVALANCHE"
storm_sub$EVENTS[grep("BLIZZARD", storm_sub$EVTYPE)] <- "BLIZZARD"
storm_sub$EVENTS[grep("COASTAL FLOOD", storm_sub$EVTYPE)] <- "COASTAL FLOOD"
storm_sub$EVENTS[grep("FLASH FLOOD", storm_sub$EVTYPE)] <- "FLASH FLOOD"
storm_sub$EVENTS[grep("LAKESHORE FLOOD", storm_sub$EVTYPE)] <- "LAKESHORE FLOOD"
storm_sub$EVENTS[which(grepl("FLOOD", storm_sub$EVTYPE) & !grepl("FLASH", storm_sub$EVTYPE) & !grepl("COASTAL", storm_sub$EVTYPE) & !grepl("LAKESHORE", storm_sub$EVTYPE))] <- "FLOOD"
storm_sub$EVENTS[which(grepl("COLD | WIND CHILL", storm_sub$EVTYPE) & !grepl("EXTREME", storm_sub$EVTYPE))] <- "COLD/WIND CHILL"
storm_sub$EVENTS[which(grepl("EXTREME COLD", storm_sub$EVTYPE) | grepl("EXTREME WINDCHILL", storm_sub$EVTYPE))] <- "EXTREME COLD/WIND CHILL"
storm_sub$EVENTS[grep("DEBRIS FLOW", storm_sub$EVTYPE)] <- "DEBRIS FLOW"
storm_sub$EVENTS[grep("DENSE FOG", storm_sub$EVTYPE)] <- "DENSE FOG"
storm_sub$EVENTS[grep("DENSE SMOKE", storm_sub$EVTYPE)] <- "DENSE SMOKE"
storm_sub$EVENTS[grep("DROUGHT", storm_sub$EVTYPE)] <- "DROUGHT"
storm_sub$EVENTS[grep("DUST DEVIL", storm_sub$EVTYPE)] <- "DUST DEVIL"
storm_sub$EVENTS[grep("DUST STORM", storm_sub$EVTYPE)] <- "DUST STORM"
storm_sub$EVENTS[grep("EXCESSIVE HEAT", storm_sub$EVTYPE)] <- "EXCESSIVE HEAT"
storm_sub$EVENTS[which(grepl("HEAT", storm_sub$EVTYPE) & !grepl("EXCESSIVE", storm_sub$EVTYPE))] <- "HEAT"
storm_sub$EVENTS[which(grepl("FROST", storm_sub$EVTYPE) | grepl("FREEZE", storm_sub$EVTYPE))] <- "FROST/FREEZE"
storm_sub$EVENTS[grep("FUNNEL CLOUD", storm_sub$EVTYPE)] <- "FUNNEL CLOUD"
storm_sub$EVENTS[grep("FREEZING FOG", storm_sub$EVTYPE)] <- "FREEZING FOG"
storm_sub$EVENTS[grep("MARINE HAIL", storm_sub$EVTYPE)] <- "MARINE HAIL"
storm_sub$EVENTS[which(grepl("HAIL", storm_sub$EVTYPE) & !grepl("MARINE", storm_sub$EVTYPE))] <- "HAIL"
storm_sub$EVENTS[grep("MARINE HIGH WIND", storm_sub$EVTYPE)] <- "MARINE HIGH WIND"
storm_sub$EVENTS[which(grepl("HIGH WIND", storm_sub$EVTYPE) & !grepl("MARINE", storm_sub$EVTYPE))] <- "HIGH WIND"
storm_sub$EVENTS[grep("MARINE STRONG WIND", storm_sub$EVTYPE)] <- "MARINE STRONG WIND"
storm_sub$EVENTS[which(grepl("STRONG WIND", storm_sub$EVTYPE) & !grepl("MARINE", storm_sub$EVTYPE))] <- "STRONG WIND"
storm_sub$EVENTS[grep("MARINE THUNDERSTORM WIND", storm_sub$EVTYPE)] <- "MARINE THUNDERSTORM WIND"
storm_sub$EVENTS[which(grepl("THUNDERSTORM WIND | TSTM WIND", storm_sub$EVTYPE) & !grepl("MARINE", storm_sub$EVTYPE))] <- "THUNDERSTORM WIND"
storm_sub$EVENTS[grep("HEAVY RAIN", storm_sub$EVTYPE)] <- "HEAVY RAIN"
storm_sub$EVENTS[grep("HEAVY SNOW", storm_sub$EVTYPE)] <- "HEAVY SNOW"
storm_sub$EVENTS[grep("HIGH SURF", storm_sub$EVTYPE)] <- "HIGH SURF"
storm_sub$EVENTS[which(grepl("HURRICANE", storm_sub$EVTYPE) | grepl("TYPHOON", storm_sub$EVTYPE))] <- "HURRICANE (TYPHOON)"
storm_sub$EVENTS[grep("ICE STORM", storm_sub$EVTYPE)] <- "ICE STORM"
storm_sub$EVENTS[grep("LAKE-EFFECT SNOW", storm_sub$EVTYPE)] <- "LAKE-EFFECT SNOW"
storm_sub$EVENTS[grep("LIGHTNING", storm_sub$EVTYPE)] <- "LIGHTNING"
storm_sub$EVENTS[grep("RIP CURRENT", storm_sub$EVTYPE)] <- "RIP CURRENT"
storm_sub$EVENTS[grep("SEICHE", storm_sub$EVTYPE)] <- "SEICHE"
storm_sub$EVENTS[grep("SLEET", storm_sub$EVTYPE)] <- "SLEET"
storm_sub$EVENTS[grep("STORM SURGE", storm_sub$EVTYPE)] <- "STORM SURGE/TIDE"
storm_sub$EVENTS[grep("TORNADO", storm_sub$EVTYPE)] <- "TORNADO"
storm_sub$EVENTS[grep("TROPICAL DEPRESSION", storm_sub$EVTYPE)] <- "TROPICAL DEPRESSION"
storm_sub$EVENTS[grep("TROPICAL STORM", storm_sub$EVTYPE)] <- "TROPICAL STORM"
storm_sub$EVENTS[grep("TSUNAMI", storm_sub$EVTYPE)] <- "TSUNAMI"
storm_sub$EVENTS[grep("VOLCANIC ASH", storm_sub$EVTYPE)] <- "VOLCANIC ASH"
storm_sub$EVENTS[grep("WATERSPOUT", storm_sub$EVTYPE)] <- "WATERSPOUT"
storm_sub$EVENTS[grep("WILDFIRE", storm_sub$EVTYPE)] <- "WILDFIRE"
storm_sub$EVENTS[grep("WINTER STORM", storm_sub$EVTYPE)] <- "WINTER STORM"
storm_sub$EVENTS[grep("WINTER WEATHER", storm_sub$EVTYPE)] <- "WINTER WEATHER"- Now we check the unique value of column “EVENTS” again.
unique(storm_sub$EVENTS)## [1] "TORNADO" "OTHERS"
## [3] "HAIL" "ICE STORM"
## [5] "WINTER STORM" "HURRICANE (TYPHOON)"
## [7] "HEAVY RAIN" "LIGHTNING"
## [9] "DENSE FOG" "RIP CURRENT"
## [11] "FLASH FLOOD" "HIGH WIND"
## [13] "FUNNEL CLOUD" "HEAT"
## [15] "FLOOD" "WATERSPOUT"
## [17] "EXTREME COLD/WIND CHILL" "BLIZZARD"
## [19] "HEAVY SNOW" "FROST/FREEZE"
## [21] "COASTAL FLOOD" "AVALANCHE"
## [23] "DUST STORM" "SLEET"
## [25] "DUST DEVIL" "EXCESSIVE HEAT"
## [27] "STRONG WIND" "HIGH SURF"
## [29] "WINTER WEATHER" "DROUGHT"
## [31] "STORM SURGE/TIDE" "TROPICAL STORM"
## [33] "WILDFIRE" "COLD/WIND CHILL"
## [35] "THUNDERSTORM WIND" "LAKE-EFFECT SNOW"
## [37] "FREEZING FOG" "VOLCANIC ASH"
## [39] "SEICHE" "TROPICAL DEPRESSION"
## [41] "ASTRONOMICAL LOW TIDE" "MARINE HAIL"
## [43] "MARINE HIGH WIND" "TSUNAMI"
## [45] "DENSE SMOKE" "LAKESHORE FLOOD"
## [47] "MARINE THUNDERSTORM WIND" "MARINE STRONG WIND"It shows 48 event types, including “others”, no “Debris Flow” comparing to the event types list in the documentation.
Processing variables “PROPDMG” and “CROPDMG”
According to the storm data documentation, there are two types of economic damages in this dataset, corn and property, both comes with the estimated numbers(“PROPDMG”, “CROPDMG”) and the expressions(“PROPDMGEXP”, “CROPDMGEXP”).
We check the value distribution of column “PROPDMGEXP” first.
table(storm_sub$PROPDMGEXP)##
## - ? + 0 1 2 3 4 5 6
## 465934 1 8 5 216 25 13 4 4 28 4
## 7 8 B h H K m M
## 5 1 40 1 6 424665 7 11330The expression value is a little mess up, and there is not much information from the documentation, except “K” represents for thousands, “M” for millions, and “B” for billions, so we assume H/h represent for hundreds, and blank inputs for 0.
We check the proportion of valid value in this column.
mean(storm_sub$PROPDMGEXP %in% c("", 0, "H", "h", "K", "k", "M", "m", "B", "b"))## [1] 0.9998914There are 99.99% of valid data, so we’ll ignore the invalid data for now.
We then do the same with the column “CROPDMGEXP”.
table(storm_sub$CROPDMGEXP) ##
## ? 0 2 B k K m M
## 618413 7 19 1 9 21 281832 1 1994mean(storm_sub$CROPDMGEXP %in% c("", 0, "K", "k", "M", "m", "B", "b"))## [1] 0.9999911The proportion of valid data is over 99.99%, so we’ll ignore the invalid ones as well.
Now, we transform the estimated damage number to the real value.
storm_sub <- storm_sub %>%
mutate(PROPDMG=ifelse(storm_sub$PROPDMGEXP=="H"| storm_sub$PROPDMGEXP=="h", PROPDMG * 10^2, ifelse(
storm_sub$PROPDMGEXP=="K" | storm_sub$PROPDMGEXP=="k", PROPDMG * 10^3, ifelse(
storm_sub$PROPDMGEXP=="M"| storm_sub$PROPDMGEXP=="m", PROPDMG * 10^6, ifelse(
storm_sub$PROPDMGEXP=="B"| storm_sub$PROPDMGEXP=="b", PROPDMG * 10^9, PROPDMG)
))))storm_sub <- storm_sub %>%
mutate(CROPDMG=ifelse(storm_sub$CROPDMGEXP=="K" | storm_sub$CROPDMGEXP=="k", CROPDMG * 10^3, ifelse(
storm_sub$CROPDMGEXP=="M"| storm_sub$CROPDMGEXP=="m", CROPDMG * 10^6, ifelse(
storm_sub$CROPDMGEXP=="B"| storm_sub$CROPDMGEXP=="b", CROPDMG * 10^9, CROPDMG)
)))Results
Question 1:
Across the United States, which types of events are most harmful with respect to population health?
1. Calculating total injuries and fatalities
We subset the data frame to calculate the total injuries and
fatalities and the fatality rate in each event type, ranking by total
numbers.
(Note: we exclude “others” from the ranking, because it is a total
number of all “other” types.)
harm <- storm_sub %>%
select(EVENTS, FATALITIES, INJURIES) %>%
dplyr::filter(EVENTS != "OTHERS") %>%
group_by(EVENTS) %>%
summarize(Fatality=sum(FATALITIES), Injury=sum(INJURIES)) %>%
mutate(TotalNum=Fatality+Injury) %>%
mutate(FatalityRate=percent(Fatality / (TotalNum), accuracy=0.01)) %>%
arrange(-TotalNum)2. Visualizing the data
We then make a plot for the most harmful weather events (top 5) with the total number of fatalities and injuries and the fatality rate from 1950 to 2011.
harm %>%
top_n(5, TotalNum) %>%
ggplot(aes(x=reorder(EVENTS, -TotalNum), TotalNum))+
geom_bar(stat="identity", fill="brown")+
ylim(0,110000)+
labs(title="The Most Harmful Weather Events in US (1950-2011)", subtitle="Total injuries and fatilities & The fatality rate", x="Weather Events", y="Total Injury and Fatality")+
geom_text(aes(label=FatalityRate), col="white", vjust=1.3, size=4)+
geom_label(aes(label=TotalNum), vjust=-0.5, size=4)
As can be seen from the plot:
- Tornado is the most harmful weather event which has caused 97022
injuries and fatalities across the United States from 1950 to November
2011.
- The top 5 most harmful weather events are: tornado, excessive heat,
flood, lightning, and heat, each have caused thousands of injuries and
fatalities.
- It’s worth noting that the fatality rate of excessive heat and heat are much higher than that of the other events, up to 22.75% and 31.06% respectively.
Question 2:
Across the United States, which types of events have the greatest economic consequences?
1. Calculating total damage values
We subset the data frame to calculate the total damage value of each
type of weather events.
(Note: we exclude “others” from the ranking again, because it’s the sum
value of all “other” types.)
damage <- storm_sub %>%
mutate(Total=PROPDMG + CROPDMG) %>%
select(EVENTS, Total) %>%
dplyr::filter(EVENTS != "OTHERS") %>%
group_by(EVENTS) %>%
summarize(TotalDMG=sum(Total)) %>%
arrange(desc(TotalDMG))2. Visualizing the data
We make a plot to show the weather events with top 5 total damage value as following.
damage %>%
top_n(5, TotalDMG) %>%
ggplot(aes(x=reorder(EVENTS, -TotalDMG), y=TotalDMG/10^9))+
geom_col(fill="lightblue")+
labs(title="The Most Costly Weather Events in US (Billions)", subtitle="From 1950 to 2011", x="Weather Events", y="Total Damage (Billions)")+
geom_label(aes(label=round(TotalDMG/10^9,1), vjust=1.5))
As demonstrated in the above plot:
- Flood has the greatest economic consequences, which has caused 161 billions damage in United states from 1950 to 2011.
- The top 5 most costly weather events are: flood, hurricane/typhoon, tornado, storm surge/tide and hail.