1. Synopsis

Storms and other severe weather events can cause both public health and economic problems for communities and municipalities. Many severe events can result in fatalities, injuries, and property damage, and preventing such outcomes to the extent possible is a key concern.

The U.S. National Oceanic and Atmospheric Administration’s (NOAA) maintains a public database for storm events which tracks characteristics of major storms and weather events in the United States, including when and where they occur, as well as estimates of any fatalities, injuries, and property damage. The events in the database covers the period from 1950 to the end of November 2011. However, for the period between 1950 and 1995 only a subset of storm events (Tornado, Thunderstorm Wind and Hail) are available in the storm database and the full set of major storms and weather events (48 event types) is available since 1996.

The goal of this project is to analyze the impacts of severe weather events on the population health and the U.S. economy. To have o comparable basis for the analysis the dataset is limited to the observations between 1996 and 2011.

The results of this project shows that “TORNADOS”, “FLOOD” and “EXCESSIVE HEAT” are the are most harmful weather events to population health (number of fatalities and injuries), while “FLOOD”, “HURRICANE/TYPHOON” and “STORM SURGE” are the weather events that have the greatest economic consequences (dollar amount of damages to properties and crop).

2. Data Processing

2.1. Load Libraries and prepare the R environment

Necessary libraries to perform loading, computation, transformation and plotting of data

library(RCurl)
library(R.utils) 
library(data.table)
library(plyr)
library(lubridate)
library(dplyr)
library(ggplot2)

2.2. Data Loading

The data source is in the form of a comma-separated-value file compressed via the bzip2 algorithm to reduce its size. It is possible to download the source file from the following web site: Storm Data

The data was downloaded from the link above and saved on local computer. Then it was loaded on the R using the read.csv command and converted to a data.table

fileUrl <- "https://d396qusza40orc.cloudfront.net/repdata%2Fdata%2FStormData.csv.bz2"
download.file(fileUrl, destfile="StormData.csv.bz2", method="curl")
stormDF <- read.csv("StormData.csv.bz2")
# Converting data.frame to data.table
stormDT <- as.data.table(stormDF)

Additional information on the database is available here:

2.3. Examine the data set

In stormDT there are 37 columns (variables) and 902,297 rows (observations).

dim(stormDT)
## [1] 902297     37

Examine the data structure.

str(stormDT)
## Classes 'data.table' and 'data.frame':   902297 obs. of  37 variables:
##  $ STATE__   : num  1 1 1 1 1 1 1 1 1 1 ...
##  $ BGN_DATE  : chr  "4/18/1950 0:00:00" "4/18/1950 0:00:00" "2/20/1951 0:00:00" "6/8/1951 0:00:00" ...
##  $ BGN_TIME  : chr  "0130" "0145" "1600" "0900" ...
##  $ TIME_ZONE : chr  "CST" "CST" "CST" "CST" ...
##  $ COUNTY    : num  97 3 57 89 43 77 9 123 125 57 ...
##  $ COUNTYNAME: chr  "MOBILE" "BALDWIN" "FAYETTE" "MADISON" ...
##  $ STATE     : chr  "AL" "AL" "AL" "AL" ...
##  $ EVTYPE    : chr  "TORNADO" "TORNADO" "TORNADO" "TORNADO" ...
##  $ BGN_RANGE : num  0 0 0 0 0 0 0 0 0 0 ...
##  $ BGN_AZI   : chr  "" "" "" "" ...
##  $ BGN_LOCATI: chr  "" "" "" "" ...
##  $ END_DATE  : chr  "" "" "" "" ...
##  $ END_TIME  : chr  "" "" "" "" ...
##  $ COUNTY_END: num  0 0 0 0 0 0 0 0 0 0 ...
##  $ COUNTYENDN: logi  NA NA NA NA NA NA ...
##  $ END_RANGE : num  0 0 0 0 0 0 0 0 0 0 ...
##  $ END_AZI   : chr  "" "" "" "" ...
##  $ END_LOCATI: chr  "" "" "" "" ...
##  $ LENGTH    : num  14 2 0.1 0 0 1.5 1.5 0 3.3 2.3 ...
##  $ WIDTH     : num  100 150 123 100 150 177 33 33 100 100 ...
##  $ F         : int  3 2 2 2 2 2 2 1 3 3 ...
##  $ MAG       : num  0 0 0 0 0 0 0 0 0 0 ...
##  $ FATALITIES: num  0 0 0 0 0 0 0 0 1 0 ...
##  $ INJURIES  : num  15 0 2 2 2 6 1 0 14 0 ...
##  $ PROPDMG   : num  25 2.5 25 2.5 2.5 2.5 2.5 2.5 25 25 ...
##  $ PROPDMGEXP: chr  "K" "K" "K" "K" ...
##  $ CROPDMG   : num  0 0 0 0 0 0 0 0 0 0 ...
##  $ CROPDMGEXP: chr  "" "" "" "" ...
##  $ WFO       : chr  "" "" "" "" ...
##  $ STATEOFFIC: chr  "" "" "" "" ...
##  $ ZONENAMES : chr  "" "" "" "" ...
##  $ LATITUDE  : num  3040 3042 3340 3458 3412 ...
##  $ LONGITUDE : num  8812 8755 8742 8626 8642 ...
##  $ LATITUDE_E: num  3051 0 0 0 0 ...
##  $ LONGITUDE_: num  8806 0 0 0 0 ...
##  $ REMARKS   : chr  "" "" "" "" ...
##  $ REFNUM    : num  1 2 3 4 5 6 7 8 9 10 ...
##  - attr(*, ".internal.selfref")=<externalptr>

2.4. Data Subsetting

According to NOAA, the data recording start from Jan. 1950. Due to changes in the data collection and processing procedures over time, there are unique periods of record available depending on the event type, as follow:

  • From 1950 through 1954, only tornado events were recorded.
  • From 1955 through 1992, only tornado, thunderstorm wind and hail events were keyed from the paper publications into digital data. From 1993 to 1995, only tornado, thunderstorm wind and hail events have been extracted from the Unformatted Text Files.
  • From 1996 to present, 48 event types are recorded as defined in NWS Directive 10-1605.

Since the goal of this project is to analyze the impacts of severe weather events on the population health and the U.S. economy, to have a comparable information, the analysis is focused only on events recorded after the end of 1995.

From a list of 37 variables in the raw data, only 8 variables are needed for the analysis. This are the following:

Record date:

  • BGN_DATE

Type of events:

  • EVTYPE: weather event (Tornados, Wind, Snow, Flood, etc..)

Health variables: - FATALITIES: approx. number of fatalities - INJURIES: approx. number of injuries

Economic variables:

  • PROPDMG – the estimate of the property damages as dollar amounts
  • PROPDMGEXP - the exponent values for ‘PROPDMG’ (property damage)
  • CROPDMG - the estimate of the crop damages as dollar amounts
  • CROPDMGEXP - the exponent values for ‘CROPDMG’ (crop damage)

Subset the dataset based on the date of the record as well as by variables of interest.

AnalysisData <- stormDT %>% mutate(BGN_DATE = mdy_hms(BGN_DATE))
AnalysisData <- AnalysisData %>% filter(BGN_DATE >= "1996-01-01") %>% 
      select(BGN_DATE, EVTYPE, FATALITIES, INJURIES, 
             PROPDMG, PROPDMGEXP, CROPDMG, CROPDMGEXP)

2.5. Transforming extracted variables

2.5.1. Transforming the event type

Listing the first 20 event types that most appear in the data.

sort(table(AnalysisData$EVTYPE), decreasing = TRUE)[1:20]
## 
##                     HAIL                TSTM WIND        THUNDERSTORM WIND 
##                   207715                   128662                    81402 
##              FLASH FLOOD                    FLOOD                  TORNADO 
##                    50999                    24247                    23154 
##                HIGH WIND               HEAVY SNOW                LIGHTNING 
##                    19907                    14000                    13203 
##               HEAVY RAIN             WINTER STORM           WINTER WEATHER 
##                    11509                    11317                     6968 
##         MARINE TSTM WIND             FUNNEL CLOUD MARINE THUNDERSTORM WIND 
##                     6175                     6058                     5812 
##              STRONG WIND     URBAN/SML STREAM FLD               WATERSPOUT 
##                     3561                     3392                     3390 
##                 WILDFIRE                 BLIZZARD 
##                     2732                     2633

Looking to the data can be observed that some events type are misspelled (i.e. “TSTM” instead of “THUNDERSTORM”) or duplicate (i.e. “FLASH FLOOD” and “FLOOD”), so will replace it.

AnalysisDataEv <- AnalysisData
AnalysisDataEv$EVTYPE[grep("TSTM WIND", AnalysisDataEv$EVTYPE, ignore.case = TRUE)] <- "THUNDERSTORM WIND"
AnalysisDataEv$EVTYPE[grep("FLASH FLOOD", AnalysisDataEv$EVTYPE, ignore.case = TRUE)] <- "FLOOD"

2.5.2. Converting Exponent Columns into Actual Exponents instead of (-,+, H, K, etc)

These are possible values of CROPDMGEXP and PROPDMGEXP are: H,h,K,k,M,m,B,b,+,-,?,0,1,2,3,4,5,6,7,8, and blank-character. According to this study, the index in the PROPDMGEXP and CROPDMGEXP can be interpreted as the following:

  • (+) = 1
  • (-) = 0
  • (?) = 0
  • blank/empty character = 0
  • numeric 0..8 = 10
  • H,h = hundreds = 100
  • K,k = kilos = thousands = 1,000
  • M,m = millions = 1,000,000
  • B,b = billions = 1,000,000,000

Transform those variables in one unit (dollar) variable and create new variable(s) that is product of value of damage and dollar unit.

AnalysisDataExp <- AnalysisDataEv
AnalysisDataExp$PROPDMGEXP <- as.character(AnalysisDataExp$PROPDMGEXP)
AnalysisDataExp$PROPDMGEXP[is.na(AnalysisDataExp$PROPDMGEXP)] <- 0 # N/A's considered as dollars
AnalysisDataExp$PROPDMGEXP[!grepl("K|M|B", AnalysisDataExp$PROPDMGEXP, ignore.case = TRUE)] <- 0 # everything except K,M,B is dollar
AnalysisDataExp$PROPDMGEXP[grep("K", AnalysisDataExp$PROPDMGEXP, ignore.case = TRUE)] <- "3"
AnalysisDataExp$PROPDMGEXP[grep("M", AnalysisDataExp$PROPDMGEXP, ignore.case = TRUE)] <- "6"
AnalysisDataExp$PROPDMGEXP[grep("B", AnalysisDataExp$PROPDMGEXP, ignore.case = TRUE)] <- "9"
AnalysisDataExp$PROPDMGEXP <- as.numeric(as.character(AnalysisDataExp$PROPDMGEXP))
AnalysisDataExp$PROPDMGTOT <- AnalysisDataExp$PROPDMG * 10^AnalysisDataExp$PROPDMGEXP

AnalysisDataExp$CROPDMGEXP <- as.character(AnalysisDataExp$CROPDMGEXP)
AnalysisDataExp$CROPDMGEXP[is.na(AnalysisDataExp$CROPDMGEXP)] <- 0 # N/A's considered as dollars
AnalysisDataExp$CROPDMGEXP[!grepl("K|M|B", AnalysisDataExp$CROPDMGEXP, ignore.case = TRUE)] <- 0 # everything ecxept K,M,B is dollar
AnalysisDataExp$CROPDMGEXP[grep("K", AnalysisDataExp$CROPDMGEXP, ignore.case = TRUE)] <- "3"
AnalysisDataExp$CROPDMGEXP[grep("M", AnalysisDataExp$CROPDMGEXP, ignore.case = TRUE)] <- "6"
AnalysisDataExp$CROPDMGEXP[grep("B", AnalysisDataExp$CROPDMGEXP, ignore.case = TRUE)] <- "9"
AnalysisDataExp$CROPDMGEXP <- as.numeric(as.character(AnalysisDataExp$CROPDMGEXP))
AnalysisDataExp$CROPDMGTOT <- AnalysisDataExp$CROPDMG * 10^AnalysisDataExp$CROPDMGEXP

2.6. Data aggregation

2.6.1. Population health data processing

We aggregate fatalities and injuries numbers in order to identify TOP-10 events contributing the total people loss.

Health_data <- aggregate(cbind(FATALITIES, INJURIES) ~ EVTYPE, data = AnalysisDataExp, FUN=sum)
Health_data$PEOPLE_LOSS <- Health_data$FATALITIES + Health_data$INJURIES
Health_data <- Health_data[order(Health_data$PEOPLE_LOSS, decreasing = TRUE), ]
Top10_events_people <- Health_data[1:10,]

2.6.2. Economic data processing

Now we aggregate property and crop damage numbers in order to identify TOP-10 events contributing the total economic loss.

Economic_data <- aggregate(cbind(PROPDMGTOT, CROPDMGTOT) ~ EVTYPE, data = AnalysisDataExp, FUN=sum)
Economic_data$ECONOMIC_LOSS <- Economic_data$PROPDMGTOT + Economic_data$CROPDMGTOT
Economic_data <- Economic_data[order(Economic_data$ECONOMIC_LOSS, decreasing = TRUE), ]
Top10_events_economy <- Economic_data[1:10,]

3. Results

3.1. Types of events are the most harmful to population health

Analyzing population health impact on the graph one can conclude that “TORNADOS”, “FLOOD” and “EXCESSIVE HEAT” are the are most harmful weather events to population health (number of fatalities and injuries).

#plotting the events which are the most harmful to population health
g <- ggplot(data = Top10_events_people, aes(x = reorder(EVTYPE, PEOPLE_LOSS), y = PEOPLE_LOSS))
g <- g + geom_bar(stat = "identity", fill = "red")
g <- g + labs(title = "The most harmful events to population health in US (1996-2011)")
g <- g + theme(plot.title = element_text(hjust = 0.5))
g <- g + labs(y = "Number of fatalities and injuries", x = "Event Type")
g <- g + coord_flip()
print(g)

3.2. Types of events that have the greatest economic consequences (losses)

Analyzing economic impact on the graph one can conclude that “FLOOD”, “HURRICANE/TYPHOON” and “STORM SURGE” are the weather events that have the greatest economic consequences (dollar amount of damages to properties and crop).

#plotting the events that have the greatest economic consequences
g <- ggplot(data = Top10_events_economy, aes(x = reorder(EVTYPE, ECONOMIC_LOSS), y = ECONOMIC_LOSS/10^9))
g <- g + geom_bar(stat = "identity", fill = "red")
g <- g + labs(title = "Events with high economic consequences in US (1996-2011)")
g <- g + theme(plot.title = element_text(hjust = 0.5))
g <- g + labs(y = "Property and crop loss (billions $)", x = "Event Type")
g <- g + coord_flip()
print(g)