Synopsis

The U.S. National Oceanic and Atmospheric Administration’s (NOAA) storm database is a database that tracks characteristics of major storms and other severe weather events in the United States. This project (aka Course Project 2 of the Reproducible Research Course) involves analyzing the database to determine which events:

  1. Are most harmful with respect to population health (fatalities and injuries) and
  2. Have the greatest economic consequences (property and crop damages)

The goal is to use the analysis findings to guide severe weather event preparations and prioritize resources for different types of events. The analysis showed that excessive heat and tornados caused the most harm to population health, whilst floods and droughts had the greatest economic consequences.


Data processing

The dataset was downloaded from the provided link (stored in fileUrl) and saved into the current working directory on the computer. It is then loaded into RStudio as stormdata.

##Download and load dataset
fileUrl <- "https://d396qusza40orc.cloudfront.net/repdata%2Fdata%2FStormData.csv.bz2"
download.file(fileUrl, destfile = "C:/Users/Ailee/Downloads/datasciencecoursera/Course 5/repdata_data_StormData.csv.bz2", method = "curl")
stormdata <- read.csv("C:/Users/Ailee/Downloads/datasciencecoursera/Course 5/repdata_data_StormData.csv.bz2")
##Evaluate internal structure
str(stormdata)
## 'data.frame':    902297 obs. of  37 variables:
##  $ STATE__   : num  1 1 1 1 1 1 1 1 1 1 ...
##  $ BGN_DATE  : chr  "4/18/1950 0:00:00" "4/18/1950 0:00:00" "2/20/1951 0:00:00" "6/8/1951 0:00:00" ...
##  $ BGN_TIME  : chr  "0130" "0145" "1600" "0900" ...
##  $ TIME_ZONE : chr  "CST" "CST" "CST" "CST" ...
##  $ COUNTY    : num  97 3 57 89 43 77 9 123 125 57 ...
##  $ COUNTYNAME: chr  "MOBILE" "BALDWIN" "FAYETTE" "MADISON" ...
##  $ STATE     : chr  "AL" "AL" "AL" "AL" ...
##  $ EVTYPE    : chr  "TORNADO" "TORNADO" "TORNADO" "TORNADO" ...
##  $ BGN_RANGE : num  0 0 0 0 0 0 0 0 0 0 ...
##  $ BGN_AZI   : chr  "" "" "" "" ...
##  $ BGN_LOCATI: chr  "" "" "" "" ...
##  $ END_DATE  : chr  "" "" "" "" ...
##  $ END_TIME  : chr  "" "" "" "" ...
##  $ COUNTY_END: num  0 0 0 0 0 0 0 0 0 0 ...
##  $ COUNTYENDN: logi  NA NA NA NA NA NA ...
##  $ END_RANGE : num  0 0 0 0 0 0 0 0 0 0 ...
##  $ END_AZI   : chr  "" "" "" "" ...
##  $ END_LOCATI: chr  "" "" "" "" ...
##  $ LENGTH    : num  14 2 0.1 0 0 1.5 1.5 0 3.3 2.3 ...
##  $ WIDTH     : num  100 150 123 100 150 177 33 33 100 100 ...
##  $ F         : int  3 2 2 2 2 2 2 1 3 3 ...
##  $ MAG       : num  0 0 0 0 0 0 0 0 0 0 ...
##  $ FATALITIES: num  0 0 0 0 0 0 0 0 1 0 ...
##  $ INJURIES  : num  15 0 2 2 2 6 1 0 14 0 ...
##  $ PROPDMG   : num  25 2.5 25 2.5 2.5 2.5 2.5 2.5 25 25 ...
##  $ PROPDMGEXP: chr  "K" "K" "K" "K" ...
##  $ CROPDMG   : num  0 0 0 0 0 0 0 0 0 0 ...
##  $ CROPDMGEXP: chr  "" "" "" "" ...
##  $ WFO       : chr  "" "" "" "" ...
##  $ STATEOFFIC: chr  "" "" "" "" ...
##  $ ZONENAMES : chr  "" "" "" "" ...
##  $ LATITUDE  : num  3040 3042 3340 3458 3412 ...
##  $ LONGITUDE : num  8812 8755 8742 8626 8642 ...
##  $ LATITUDE_E: num  3051 0 0 0 0 ...
##  $ LONGITUDE_: num  8806 0 0 0 0 ...
##  $ REMARKS   : chr  "" "" "" "" ...
##  $ REFNUM    : num  1 2 3 4 5 6 7 8 9 10 ...

Running the str function on stormdata showed that it is a dataframe with 902,297 records on 37 variables. To begin data processing, the following packages were loaded:

##Load packages
library(dplyr)
library(ggplot2)
library(gridExtra)
library(lubridate)
library(stringdist)

Looking at the NOAA Storm Events Database website, we can see that the recording of all event types only began from 1996 onwards. To extract that information, the values of the BGN_date column were first converted from character to datetime class. Next, we filtered for entries from 1996 onwards and selected the relevant variables for our analysis (population health impact: fatalities & injuries, economic impact: property & crop damages). We also filtered out any entries where the fatalities, injuries, property and crop damage value all equals to 0.

##Convert beginning date to datetime class, then filter and select relevant entries
stormdata <- stormdata %>%
    dplyr::mutate(BGN_DATE = lubridate::mdy_hms(BGN_DATE)) %>%
    dplyr::filter(BGN_DATE >= "1996-01-01") %>%
    dplyr::select(EVTYPE, FATALITIES, INJURIES, PROPDMG, PROPDMGEXP, CROPDMG, 
                  CROPDMGEXP) %>%
    dplyr::filter(if_any(c(FATALITIES, INJURIES, PROPDMG, CROPDMG), ~.> 0))
##Check unique event type entries
unique(stormdata$EVTYPE)

Running the unique function on the EVTYPE column showed that there were 222 unique entries, even though the number of officially recorded severe weather events is only 48. Visual inspection of the dataset revealed that the column contained entries with spelling errors and/or unofficial descriptions of the recorded events.
To fix this, we first stored the list of officially recorded severe weather events as events (Note: we added “Others” into the list for entries that do not belong to any event type). Together with the EVTYPE column, we converted the entries to UPPERCASE.

##Store list of officially recorded events 
events <- c("Astronomical Low Tide", "Avalanche", "Blizzard", "Coastal Flood", "Cold/Wind Chill", "Debris Flow", "Dense Fog", "Dense Smoke", "Drought", "Dust Devil", "Dust Storm", "Excessive Heat", "Extreme Cold/Wind Chill", "Flash Flood", "Flood", "Frost/Freeze", "Funnel Cloud", "Freezing Fog", "Hail", "Heat", "Heavy Rain", "Heavy Snow", "High Surf", "High Wind", "Hurricane (Typhoon)", "Ice Storm", "Lake-Effect Snow", "Lakeshore Flood", "Lightning", "Marine Hail", "Marine High Wind", "Marine Strong Wind", "Marine Thunderstorm Wind", "Rip Current", "Seiche", "Sleet", "Storm Surge/Tide", "Strong Wind", "Thunderstorm Wind", "Tornado", "Tropical Depression", "Tropical Storm", "Tsunami", "Volcanic Ash", "Waterspout", "Wildfire", "Winter Storm", "Winter Weather", "Other")
##Convert everything to uppercase
stormdata$EVTYPE <- toupper(stormdata$EVTYPE)
events <- toupper(events)

Next, we replaced some of the common unofficial descriptions found in the EVTYPE entries with the corresponding official severe weather event. This is not comprehensive and it is possible that we could have missed some entries.
Using approximate string matching, we then matched the EVTYPE column to events and stored the results in a separate column called RECEVTYPE. Using the sum function, a quick comparison of both EVTYPE and RECVTYPE columns showed that we were able to get a 99% match using this method.

##Replace unofficial entries with official events
stormdata$EVTYPE[grep("TSTM\ |BURST", stormdata$EVTYPE)] <- "THUNDERSTORM WIND"
stormdata$EVTYPE[grep("DRY", stormdata$EVTYPE)] <- "DROUGHT"
stormdata$EVTYPE[grep("SLIDE", stormdata$EVTYPE)] <- "DEBRIS FLOW"
stormdata$EVTYPE[grep("WARM", stormdata$EVTYPE)] <- "HEAT"
stormdata$EVTYPE[grep("RAIN\ |^URBAN", stormdata$EVTYPE)] <- "HEAVY RAIN"
stormdata$EVTYPE[grep("EXTREME COLD", stormdata$EVTYPE)] <- "EXTREME COLD/WIND CHILL"
stormdata$EVTYPE[grep("FREEZING\ |SQUALLS*\ |LIGHT", stormdata$EVTYPE)] <- "WINTER WEATHER"
stormdata$EVTYPE[grep("TYPHOON\ |HURRICANE", stormdata$EVTYPE)] <- "HURRICANE (TYPHOON)"
##Match EVTYPE entries to list of officially recorded severe weather events
stormdata$RECEVTYPE <- events[stringdist::amatch(stormdata$EVTYPE, events, method = "lv", maxDist = 24)]
sum(stormdata$EVTYPE == stormdata$RECEVTYPE)/201318

In the dataset, the crop and property damages were stored in two seperate columns- the base value and the exponent (-EXP). To combine them, we first assigned a corresponding numerical value to each exponent, and assigned 0 when there is no exponent in the -EXP columns. We then converted those values from character to numeric class. We calculated the final value by multiplying the base value with the exponent and stored the results in a separate column. (FINALPROPDMG/FINALCROPDMG).

##Assign numeric value to K, M and B, and 0 to everything else
stormdata$PROPDMGEXP[!grepl("K|M|B", stormdata$PROPDMGEXP)] <- 0
stormdata$PROPDMGEXP[grep("K", stormdata$PROPDMGEXP)] <- 1000
stormdata$PROPDMGEXP[grep("M", stormdata$PROPDMGEXP)] <- 1000000
stormdata$PROPDMGEXP[grep("B", stormdata$PROPDMGEXP)] <- 1000000000
#Convert the values above from character to numeric class
stormdata$PROPDMGEXP <- as.numeric(stormdata$PROPDMGEXP)
#Calculate the final value of PROPERTY DAMAGES (base value multiplied by the exponent)
stormdata$FINALPROPDMG <- stormdata$PROPDMG * stormdata$PROPDMGEXP

##Repeat everything for CROP DAMAGES
stormdata$CROPDMGEXP[!grepl("K|M|B", stormdata$CROPDMGEXP)] <- 0
stormdata$CROPDMGEXP[grep("K", stormdata$CROPDMGEXP)] <- 1000
stormdata$CROPDMGEXP[grep("M", stormdata$CROPDMGEXP)] <- 1000000
stormdata$CROPDMGEXP[grep("B", stormdata$CROPDMGEXP)] <- 1000000000
stormdata$CROPDMGEXP <- as.numeric(stormdata$CROPDMGEXP)
stormdata$FINALCROPDMG <- stormdata$CROPDMG * stormdata$CROPDMGEXP



Results

1. Across the United States, which types of events (as indicated in the EVTYPE variable) are most harmful with respect to population health?

We used fatalities and injuries to measure the impact of severe weather events on population health. The processed dataset was grouped by event type and the total number of fatalities and injuries in each group was tabulated. We selected the top 10 events for each impact and plotted the results.

##Summmarise fatalities and injuries per event type and identify the top 10 events for both
pophealth <- stormdata %>%
    dplyr::group_by(RECEVTYPE) %>%
    dplyr::summarize(FATALITIES = sum(FATALITIES), INJURIES = sum(INJURIES))
top10fatalities <- pophealth %>% dplyr::top_n(10, FATALITIES)
top10injuries <- pophealth %>% dplyr::top_n(10, INJURIES)
##Create plots
plot1 <- ggplot(top10fatalities, aes(x = reorder(RECEVTYPE, desc(FATALITIES)), y = FATALITIES)) + geom_bar(stat = "identity") + labs(x = "Type of event", y = "Fatality count", title = "Top 10 events causing fatalities across the United States (1996-2011)") + theme(axis.text.x = element_text(angle = 45, hjust = 1)) + geom_text(aes(label = FATALITIES, vjust = -1)) + coord_cartesian(ylim = c(0, 2000))
plot2 <- ggplot(top10injuries, aes(x = reorder(RECEVTYPE, desc(INJURIES)), y = INJURIES)) + geom_bar(stat = "identity") + labs(x = "Type of event", y = "Injury count", title = "Top 10 events causing injuries across the United States (1996-2011)") + theme(axis.text.x = element_text(angle = 45, hjust = 1)) + geom_text(aes(label = INJURIES, vjust = -1)) + coord_cartesian(ylim = c(0, 25000))
grid.arrange(plot1, plot2, ncol=2)

Excessive heat and tornados were the most harmful on population health, with excessive heat causing the most fatalities and tornados causing the most injuries across the United States.

2. Across the United States, which types of events have the greatest economic consequences?

We used property and crop damages to measure the impact of severe weather events on the economy. The processed dataset was grouped by event type and the total value of property and crop damages in each group was tabulated. We selected the top 10 events for each impact, rounded the value to the nearest integer and plotted the results.

##Summmarise property and crop damages per event type and identify the top 10 events for both
economic <- stormdata %>%
    dplyr::group_by(RECEVTYPE) %>%
    dplyr::summarize(FINALPROPDMG = sum(FINALPROPDMG), FINALCROPDMG = sum(FINALCROPDMG))
top10propdmg <- economic %>% dplyr::top_n(10, FINALPROPDMG) %>% mutate(FINALPROPDMG = round(FINALPROPDMG/10^6, digits = 0))
top10cropdmg <- economic %>% dplyr::top_n(10, FINALCROPDMG) %>% mutate(FINALCROPDMG = round(FINALCROPDMG/10^6, digits = 0))
##Create plot
plot1 <- ggplot(top10propdmg, aes(x = reorder(RECEVTYPE, desc(FINALPROPDMG)), y = FINALPROPDMG)) + geom_bar(stat = "identity") + labs(x = "Type of event", y = "Total property damage (millions)", title = "Top 10 events causing property damage across the United States (1996-2011)") + theme(axis.text.x = element_text(angle = 45, hjust = 1)) + geom_text(aes(label = FINALPROPDMG, vjust = -1)) + coord_cartesian(ylim = c(0, 150000))
plot2 <- ggplot(top10cropdmg, aes(x = reorder(RECEVTYPE, desc(FINALCROPDMG)), y = FINALCROPDMG)) + geom_bar(stat = "identity") + labs(x = "Type of event", y = "Total crop damage (millions)", title = "Top 10 events causing crop damage across the United States (1996-2011)") + theme(axis.text.x = element_text(angle = 45, hjust = 1)) + geom_text(aes(label = FINALCROPDMG, vjust = -1)) + coord_cartesian(ylim = c(0, 14000))
grid.arrange(plot1, plot2, ncol=2)

Floods and droughts have the greatest economic consequences, with floods incurring the highest accumulated costs in property damage and droughts incurring the highest accumulated costs in crop damages across the United States. Hurricanes/typhoons are also worth mentioning, since they are second in incurring the highest amount of both property and crop damages.