Data Analysis on effects of weather events in United States

Synopsis

In this report we aim to describe effects of weather events in USA which are most harmful to population health and economic. We have worked on total number injuries and fatalaties caused by the event of interest to measure population health harmfullness. On economy consequeneces, we’ve worked on property and crop damage cost. We observed that tornado, wind and heat events were very harmful to population health. Flood, Hurricane/typhoon, storms had the major economic consequences.

Loading and Processing the raw data

We obtained data from the U.S. National Oceanic and Atmospheric Administration’s (NOAA) storm database for year 1950 to 2011. Storm database comes in the form of a comma-separated-value file compressed via the bzip2 algorithm to reduce its size.

Reading Storm data

library(RCurl)

if (!file.exists("stormdata.csv.bz2")) {
    download.file("https://d396qusza40orc.cloudfront.net/repdata%2Fdata%2FStormData.csv.bz2",destfile = "stormdata.csv.bz2",method = "curl")
}

#Read stormdata file which was downloaded from above link
stormdata<-read.csv(bzfile("stormdata.csv.bz2"), stringsAsFactors=FALSE)

After reading the data we want to preview few observations & all available variable to get better understanding on storm data.

#Display few observations to get better understanding on data
head(stormdata)

##   STATE__           BGN_DATE BGN_TIME TIME_ZONE COUNTY COUNTYNAME STATE
## 1       1  4/18/1950 0:00:00     0130       CST     97     MOBILE    AL
## 2       1  4/18/1950 0:00:00     0145       CST      3    BALDWIN    AL
## 3       1  2/20/1951 0:00:00     1600       CST     57    FAYETTE    AL
## 4       1   6/8/1951 0:00:00     0900       CST     89    MADISON    AL
## 5       1 11/15/1951 0:00:00     1500       CST     43    CULLMAN    AL
## 6       1 11/15/1951 0:00:00     2000       CST     77 LAUDERDALE    AL
##    EVTYPE BGN_RANGE BGN_AZI BGN_LOCATI END_DATE END_TIME COUNTY_END
## 1 TORNADO         0                                               0
## 2 TORNADO         0                                               0
## 3 TORNADO         0                                               0
## 4 TORNADO         0                                               0
## 5 TORNADO         0                                               0
## 6 TORNADO         0                                               0
##   COUNTYENDN END_RANGE END_AZI END_LOCATI LENGTH WIDTH F MAG FATALITIES
## 1         NA         0                      14.0   100 3   0          0
## 2         NA         0                       2.0   150 2   0          0
## 3         NA         0                       0.1   123 2   0          0
## 4         NA         0                       0.0   100 2   0          0
## 5         NA         0                       0.0   150 2   0          0
## 6         NA         0                       1.5   177 2   0          0
##   INJURIES PROPDMG PROPDMGEXP CROPDMG CROPDMGEXP WFO STATEOFFIC ZONENAMES
## 1       15    25.0          K       0                                    
## 2        0     2.5          K       0                                    
## 3        2    25.0          K       0                                    
## 4        2     2.5          K       0                                    
## 5        2     2.5          K       0                                    
## 6        6     2.5          K       0                                    
##   LATITUDE LONGITUDE LATITUDE_E LONGITUDE_ REMARKS REFNUM
## 1     3040      8812       3051       8806              1
## 2     3042      8755          0          0              2
## 3     3340      8742          0          0              3
## 4     3458      8626          0          0              4
## 5     3412      8642          0          0              5
## 6     3450      8748          0          0              6

#Check the structure of data
str(stormdata)

## 'data.frame':    902297 obs. of  37 variables:
##  $ STATE__   : num  1 1 1 1 1 1 1 1 1 1 ...
##  $ BGN_DATE  : chr  "4/18/1950 0:00:00" "4/18/1950 0:00:00" "2/20/1951 0:00:00" "6/8/1951 0:00:00" ...
##  $ BGN_TIME  : chr  "0130" "0145" "1600" "0900" ...
##  $ TIME_ZONE : chr  "CST" "CST" "CST" "CST" ...
##  $ COUNTY    : num  97 3 57 89 43 77 9 123 125 57 ...
##  $ COUNTYNAME: chr  "MOBILE" "BALDWIN" "FAYETTE" "MADISON" ...
##  $ STATE     : chr  "AL" "AL" "AL" "AL" ...
##  $ EVTYPE    : chr  "TORNADO" "TORNADO" "TORNADO" "TORNADO" ...
##  $ BGN_RANGE : num  0 0 0 0 0 0 0 0 0 0 ...
##  $ BGN_AZI   : chr  "" "" "" "" ...
##  $ BGN_LOCATI: chr  "" "" "" "" ...
##  $ END_DATE  : chr  "" "" "" "" ...
##  $ END_TIME  : chr  "" "" "" "" ...
##  $ COUNTY_END: num  0 0 0 0 0 0 0 0 0 0 ...
##  $ COUNTYENDN: logi  NA NA NA NA NA NA ...
##  $ END_RANGE : num  0 0 0 0 0 0 0 0 0 0 ...
##  $ END_AZI   : chr  "" "" "" "" ...
##  $ END_LOCATI: chr  "" "" "" "" ...
##  $ LENGTH    : num  14 2 0.1 0 0 1.5 1.5 0 3.3 2.3 ...
##  $ WIDTH     : num  100 150 123 100 150 177 33 33 100 100 ...
##  $ F         : int  3 2 2 2 2 2 2 1 3 3 ...
##  $ MAG       : num  0 0 0 0 0 0 0 0 0 0 ...
##  $ FATALITIES: num  0 0 0 0 0 0 0 0 1 0 ...
##  $ INJURIES  : num  15 0 2 2 2 6 1 0 14 0 ...
##  $ PROPDMG   : num  25 2.5 25 2.5 2.5 2.5 2.5 2.5 25 25 ...
##  $ PROPDMGEXP: chr  "K" "K" "K" "K" ...
##  $ CROPDMG   : num  0 0 0 0 0 0 0 0 0 0 ...
##  $ CROPDMGEXP: chr  "" "" "" "" ...
##  $ WFO       : chr  "" "" "" "" ...
##  $ STATEOFFIC: chr  "" "" "" "" ...
##  $ ZONENAMES : chr  "" "" "" "" ...
##  $ LATITUDE  : num  3040 3042 3340 3458 3412 ...
##  $ LONGITUDE : num  8812 8755 8742 8626 8642 ...
##  $ LATITUDE_E: num  3051 0 0 0 0 ...
##  $ LONGITUDE_: num  8806 0 0 0 0 ...
##  $ REMARKS   : chr  "" "" "" "" ...
##  $ REFNUM    : num  1 2 3 4 5 6 7 8 9 10 ...

Storm database has about 902,297 observations and 37 variables. We are aiming to answer following two questions from the storm data analysis

Across the United States, which types of events (as indicated in the EVTYPE variable) are most harmful with respect to population health?
Across the United States, which types of events have the greatest economic consequences?

Following 7 variables should be enough to answer above data analysis questions

EVTYPE - Event types like Flooad, snow, Hail etc.,
FATALITIES - Number of fatalaties during the weather event; to measure harmfullness to population health
INJURIES - Number of injuries during the weather event; to measure harmfullness to population health
PROPDMG - Cost of property damage ; to measure economic consequences in terms of USD
PROPDMGEXP - Measure of property damage cost like hundreds, millions, billions etc.,
CROPDMG - Cost of crop damage ; to measure economic consequences in terms of USD
CROPDMGEXP - Measure of property damage cost like hundreds, millions, billions etc.,

To answer the above questions, Two datasets are created with only required variables

library(dplyr)

## 
## Attaching package: 'dplyr'
## 
## The following objects are masked from 'package:stats':
## 
##     filter, lag
## 
## The following objects are masked from 'package:base':
## 
##     intersect, setdiff, setequal, union

#To answer weather events effect on Population health
health_data <- select(stormdata, EVTYPE, FATALITIES, INJURIES)
#to answer weather event effect on economic consequnces
prp_dmg_data <- select(stormdata,EVTYPE, PROPDMG,PROPDMGEXP, CROPDMG, CROPDMGEXP)

#Remove stormdata to free up memory
rm(storm_data)

## Warning: object 'storm_data' not found

Results

1. Across the United States, which types of events (as indicated in the EVTYPE variable) are most harmful with respect to population health?

To work on this questions, health_data has been transformed to make tidy dataset. Summarize on weather event type by adding up injuries and fatalaties should yield the measure for population health.

library(tidyr)
library(magrittr)

## 
## Attaching package: 'magrittr'
## 
## The following object is masked from 'package:tidyr':
## 
##     extract

#Create tidy health data using tidyr
#After tidying data, 4 variables injuries, fatalities and their values becomes harm.type and harm.count variable. Tidy data helps to speed up the data analysis.
#Using pipe operator %>% from magrittr package

health_data.tidy <- health_data %>% gather(harm.type, harm.count,FATALITIES:INJURIES) %>% filter(filter = harm.count > 0 ) %>% arrange(desc(harm.count))

#Display the result of above expression
tbl_df(data = health_data.tidy)

## Source: local data frame [24,578 x 3]
## 
##               EVTYPE harm.type harm.count
## 1            TORNADO  INJURIES       1700
## 2          ICE STORM  INJURIES       1568
## 3            TORNADO  INJURIES       1228
## 4            TORNADO  INJURIES       1150
## 5            TORNADO  INJURIES       1150
## 6              FLOOD  INJURIES        800
## 7            TORNADO  INJURIES        800
## 8            TORNADO  INJURIES        785
## 9  HURRICANE/TYPHOON  INJURIES        780
## 10             FLOOD  INJURIES        750
## ..               ...       ...        ...

#to be consistent Convert all event type into upper case
health_data.tidy$EVTYPE <- toupper(health_data.tidy$EVTYPE)

#Summarize to get top weather events interms of population health
health_data.summarise <- summarise(group_by(health_data.tidy,EVTYPE,harm.type),harm.count=sum(harm.count)) %>% arrange(desc(harm.count))

tbl_df(health_data.summarise)

## Source: local data frame [310 x 3]
## 
##            EVTYPE  harm.type harm.count
## 1         TORNADO   INJURIES      91346
## 2       TSTM WIND   INJURIES       6957
## 3           FLOOD   INJURIES       6789
## 4  EXCESSIVE HEAT   INJURIES       6525
## 5         TORNADO FATALITIES       5633
## 6       LIGHTNING   INJURIES       5230
## 7            HEAT   INJURIES       2100
## 8       ICE STORM   INJURIES       1975
## 9  EXCESSIVE HEAT FATALITIES       1903
## 10    FLASH FLOOD   INJURIES       1777
## ..            ...        ...        ...

Further analysis on weather event type entered into the database reveals that event type names are not very consistent. It is mandatory to recategorize them and make weather events consistent.

#Recategorization weather events
health_data.summarise$EVTYPE[grepl("WINT|SNOW|AVA|ICE|ICY|FROST|LOW|FREEZ|BLIZ|COLD|HYPO|HYPER|SLEET|FOG",health_data.summarise$EVTYPE)] <- "WINTER"
health_data.summarise$EVTYPE[grepl("WIND|SURF|WAV|HIG|DUST|CURRENT|ROUGH",health_data.summarise$EVTYPE)] <- "WIND"
health_data.summarise$EVTYPE[grepl("FLOOD|FLD|SLID",health_data.summarise$EVTYPE)] <- "FLOOD"
health_data.summarise$EVTYPE[grepl("HAIL",health_data.summarise$EVTYPE)] <- "HAIL"
health_data.summarise$EVTYPE[grepl("TORNA",health_data.summarise$EVTYPE)] <- "TORNADO"
health_data.summarise$EVTYPE[grepl("STORM|HAIL|RAIN|CLOUD|LIGHTN|WATER",health_data.summarise$EVTYPE)] <- "STORMS"
health_data.summarise$EVTYPE[grepl("HEAT|DROU|WARM|FIRE|WARM|DRY",health_data.summarise$EVTYPE)] <- "HEAT"
health_data.summarise$EVTYPE[grepl("HURR|TYPH",health_data.summarise$EVTYPE)] <- "HURRICANE/TYPHOON"
health_data.summarise$EVTYPE[grepl("MARINE|OTHER|MIXED|GLAZE|DROWN|HEAVY",health_data.summarise$EVTYPE)] <- "OTHER"

Table list to display weather events which are harmful to population health

health_aggr <- aggregate(harm.count ~ EVTYPE,FUN=sum,data=health_data.summarise) %>% arrange(desc(harm.count))

tbl_df(health_aggr)

## Source: local data frame [9 x 2]
## 
##              EVTYPE harm.count
## 1           TORNADO      97043
## 2              WIND      15256
## 3              HEAT      13573
## 4             FLOOD      10333
## 5            WINTER       9153
## 6            STORMS       8413
## 7 HURRICANE/TYPHOON       1466
## 8             OTHER        274
## 9           TSUNAMI        162

Plot the results

library(ggplot2)
ggplot(data=health_aggr, aes(x=factor(EVTYPE,levels=EVTYPE,ordered=TRUE),y=harm.count)) + geom_bar(stat="identity", fill="#993333") + ggtitle("Weather events which are harmful \nto population health") + xlab("Weather Events") + ylab("Injuries+Fatalities") + theme(axis.text.x=element_text(angle = 90))

plot of chunk question1_plot

From the above plot, we can observe that Tornado, Wind and Heat weather events are very harmful to population health

2. Across the United States, which types of events have the greatest economic consequences?

Few transfomations are required to assess demage data. property damage expense values are converted to cost in USD from symbols. For example ‘h’ represents hundreds which converted to value 100 (in USD). Total cost of property damage can be obtained by multiplying property damage with property demage expense values. Same transformation logic was applied to crop damage expense too.

prp_dmg_data$PROPDMGEXP_TRANSFORMED <- prp_dmg_data$PROPDMGEXP
prp_dmg_data$PROPDMGEXP_TRANSFORMED[prp_dmg_data$PROPDMGEXP %in% c('h','H')] <- 10^2
prp_dmg_data$PROPDMGEXP_TRANSFORMED[prp_dmg_data$PROPDMGEXP %in% c('k','K')] <- 10^3
prp_dmg_data$PROPDMGEXP_TRANSFORMED[prp_dmg_data$PROPDMGEXP_TRANSFORMED %in% c('m','M')] <- 10^6
prp_dmg_data$PROPDMGEXP_TRANSFORMED[prp_dmg_data$PROPDMGEXP_TRANSFORMED %in% c('b','B')] <- 10^9
prp_dmg_data$PROPDMGEXP_TRANSFORMED[prp_dmg_data$PROPDMGEXP_TRANSFORMED == ""] <- 0
prp_dmg_data$PROPDMGEXP_TRANSFORMED[prp_dmg_data$PROPDMGEXP_TRANSFORMED %in% c('?','+','-')] <- 1

prp_dmg_data$CROPDMGEXP_TRANSFORMED <- prp_dmg_data$CROPDMGEXP
prp_dmg_data$CROPDMGEXP_TRANSFORMED[prp_dmg_data$CROPDMGEXP %in% c('k','K')] <- 10^3
prp_dmg_data$CROPDMGEXP_TRANSFORMED[prp_dmg_data$CROPDMGEXP %in% c('m','M')] <- 10^6
prp_dmg_data$CROPDMGEXP_TRANSFORMED[prp_dmg_data$CROPDMGEXP %in% c('b','B')] <- 10^9
prp_dmg_data$CROPDMGEXP_TRANSFORMED[prp_dmg_data$CROPDMGEXP == ""] <- 0
prp_dmg_data$CROPDMGEXP_TRANSFORMED[prp_dmg_data$CROPDMGEXP == "?"] <- 1

prp_dmg_data$PROPDMGEXP_TRANSFORMED <- as.integer(prp_dmg_data$PROPDMGEXP_TRANSFORMED )
prp_dmg_data$CROPDMGEXP_TRANSFORMED <- as.integer(prp_dmg_data$CROPDMGEXP_TRANSFORMED )

prp_dmg_data$PROPDMGCOST <- prp_dmg_data$PROPDMG * prp_dmg_data$PROPDMGEXP_TRANSFORMED
prp_dmg_data$CROPDMGCOST <- prp_dmg_data$CROPDMG * prp_dmg_data$CROPDMGEXP_TRANSFORMED

Further analysis on weather event type entered into the database reveals that event type names are not very consistent. It is mandatory to recategorize them and make weather events consistent.

#Convert all variable names into lower case
names(prp_dmg_data) <- tolower(names(prp_dmg_data))

#Select property and crop damage cost which are derived in earlier step and create tidy dataset
#for further analysis
dmg_data <- select(prp_dmg_data,evtype,propdmgcost,cropdmgcost) %>% gather(damage.type,damage.cost,propdmgcost:cropdmgcost) %>% filter(filter = damage.cost > 0 ) %>% arrange(desc(damage.cost))

#Convert all event type values into upper case
dmg_data$evtype <- toupper(dmg_data$evtype)

dmg_data$evtype_nrmlz <- dmg_data$evtype

dmg_data$evtype_nrmlz[grepl("WINT|SNOW|AVA|ICE|ICY|FROST|LOW|FREEZ|BLIZ|COLD|HYPO|HYPER|SLEET|FOG",dmg_data$evtype_nrmlz)] <- "WINTER"
dmg_data$evtype_nrmlz[grepl("WIND|SURF|WAV|HIG|DUST|CURRENT|ROUGH",dmg_data$evtype_nrmlz)] <- "WIND"
dmg_data$evtype_nrmlz[grepl("FLOOD|FLD|SLID|LANDSLUMP|DAM",dmg_data$evtype_nrmlz)] <- "FLOOD"
dmg_data$evtype_nrmlz[grepl("HAIL",dmg_data$evtype_nrmlz)] <- "HAIL"
dmg_data$evtype_nrmlz[grepl("TORNA|GUSTNADO|LANDSPOUT|TORNDAO|TORNDAO",dmg_data$evtype_nrmlz)] <- "TORNADO"
dmg_data$evtype_nrmlz[grepl("STORM|HAIL|RAIN|CLOUD|LIGHTN|WATER|LIGHTIN|WET|LIGNTNING|TROPICAL",dmg_data$evtype_nrmlz)] <- "STORMS"
dmg_data$evtype_nrmlz[grepl("HEAT|DROU|WARM|FIRE|WARM|DRY",dmg_data$evtype_nrmlz)] <- "HEAT"
dmg_data$evtype_nrmlz[grepl("HURR|TYPH",dmg_data$evtype_nrmlz)] <- "HURRICANE/TYPHOON"
dmg_data$evtype_nrmlz[grepl("MARINE|OTHER|MIXED|GLAZE|DROWN|HEAVY|BURST|DENSE|VOLCA|COASTAL|BEACH|SEICHE|URBAN|TURBUL|TSTM|APACHE",dmg_data$evtype_nrmlz)] <- "OTHER"
dmg_data$evtype_nrmlz[dmg_data$evtype_nrmlz == '?'] <- 'OTHER'

Table list to display weather events which have greater economic consequences

dmg_data_aggr <- aggregate(damage.cost ~ evtype_nrmlz,FUN=sum,data=dmg_data) %>% arrange(desc(damage.cost))
tbl_df(dmg_data_aggr)

## Source: local data frame [9 x 2]
## 
##        evtype_nrmlz damage.cost
## 1             FLOOD   1.802e+11
## 2 HURRICANE/TYPHOON   9.076e+10
## 3            STORMS   8.175e+10
## 4           TORNADO   5.741e+10
## 5              WIND   3.487e+10
## 6            WINTER   2.147e+10
## 7              HEAT   9.815e+09
## 8           TSUNAMI   1.441e+08
## 9             OTHER   7.898e+06

Plot the result:

ggplot(data=dmg_data_aggr, aes(x=factor(evtype_nrmlz,levels=evtype_nrmlz,ordered=TRUE),y=damage.cost)) + geom_bar(stat="identity", fill="#993333") + ggtitle("Weather events which has economic consequences") + ylab("Damage Cost") + xlab("Weather Events") + theme(axis.text.x=element_text(angle = 90))

plot of chunk question2_plot

From the above plot, we can observe that Flood, Hurricane/Typhoon and storms weather events has greater economic consequences

Conclusion

The analysis of the data from 1950 to 2011 reveals that Tornado, Wind and Heat have the highest impact on population health, and Flood, Hurricane/Typhoon and storms have the greatest economic consequences.