Reproducible Research - Course project 2
Alejandro Cadavid Romero
Study report - Weather events impact over U.S. population and economy
1 Context of the study
The objective of this study is to analyzed the U.S. National Oceanic and Atmospheric Administration’s (NOAA) storm database and look the effects of several weather events across time over the population and economy. The economic impacts are measeure trough property and agriculture damage, the last one is measured in crop damage. The population impacts are measured trough injuries and fatalities per weather event. The result of this study will be showed in bar charts at the end of the document.
2 Data context
Storms and other severe weather events can cause both public health and economic problems for communities and municipalities. Many severe events can result in fatalities, injuries, and property damage, and preventing such outcomes to the extent possible is a key concern.
This project involves exploring the U.S. National Oceanic and Atmospheric Administration’s (NOAA) storm database. This database tracks characteristics of major storms and weather events in the United States, including when and where they occur, as well as estimates of any fatalities, injuries, and property damage
3 Data source and complementary documentation
The data for this assignment come in the form of a comma-separated-value file compressed via the bzip2 algorithm to reduce its size. You can download the file from the course web site:
There is also some documentation of the database available. Here you will find how some of the variables are constructed/defined.
- National Weather ServiceStorm data documentation
- National Climatic Data Center Storm Events FAQ
The events in the database start in the year 1950 and end in November 2011. In the earlier years of the database there are generally fewer events recorded, most likely due to a lack of good records. More recent years should be considered more complete.
4 Data processing
First, the data wil be downloaded from the site Storm data in a .csv.bz2 file, into the directory data; it will be loaded into R with the read.csv() function to start the exploration and transformation process.
if(!dir.exists("data")){
dir.create("data")
url <- "https://d396qusza40orc.cloudfront.net/repdata%2Fdata%2FStormData.csv.bz2"
download.file(url, destfile = "data/Storm.csv.biz2")}Storm <- read.csv("data/Storm.csv.biz2") # Loading the dataSince the file have a lot of data and its heavy, it will be copied into a new dataframe
str(Storm) #Looking at the data structure## 'data.frame': 902297 obs. of 37 variables:
## $ STATE__ : num 1 1 1 1 1 1 1 1 1 1 ...
## $ BGN_DATE : Factor w/ 16335 levels "1/1/1966 0:00:00",..: 6523 6523 4242 11116 2224 2224 2260 383 3980 3980 ...
## $ BGN_TIME : Factor w/ 3608 levels "00:00:00 AM",..: 272 287 2705 1683 2584 3186 242 1683 3186 3186 ...
## $ TIME_ZONE : Factor w/ 22 levels "ADT","AKS","AST",..: 7 7 7 7 7 7 7 7 7 7 ...
## $ COUNTY : num 97 3 57 89 43 77 9 123 125 57 ...
## $ COUNTYNAME: Factor w/ 29601 levels "","5NM E OF MACKINAC BRIDGE TO PRESQUE ISLE LT MI",..: 13513 1873 4598 10592 4372 10094 1973 23873 24418 4598 ...
## $ STATE : Factor w/ 72 levels "AK","AL","AM",..: 2 2 2 2 2 2 2 2 2 2 ...
## $ EVTYPE : Factor w/ 985 levels " HIGH SURF ADVISORY",..: 834 834 834 834 834 834 834 834 834 834 ...
## $ BGN_RANGE : num 0 0 0 0 0 0 0 0 0 0 ...
## $ BGN_AZI : Factor w/ 35 levels ""," N"," NW",..: 1 1 1 1 1 1 1 1 1 1 ...
## $ BGN_LOCATI: Factor w/ 54429 levels ""," Christiansburg",..: 1 1 1 1 1 1 1 1 1 1 ...
## $ END_DATE : Factor w/ 6663 levels "","1/1/1993 0:00:00",..: 1 1 1 1 1 1 1 1 1 1 ...
## $ END_TIME : Factor w/ 3647 levels ""," 0900CST",..: 1 1 1 1 1 1 1 1 1 1 ...
## $ COUNTY_END: num 0 0 0 0 0 0 0 0 0 0 ...
## $ COUNTYENDN: logi NA NA NA NA NA NA ...
## $ END_RANGE : num 0 0 0 0 0 0 0 0 0 0 ...
## $ END_AZI : Factor w/ 24 levels "","E","ENE","ESE",..: 1 1 1 1 1 1 1 1 1 1 ...
## $ END_LOCATI: Factor w/ 34506 levels ""," CANTON"," TULIA",..: 1 1 1 1 1 1 1 1 1 1 ...
## $ LENGTH : num 14 2 0.1 0 0 1.5 1.5 0 3.3 2.3 ...
## $ WIDTH : num 100 150 123 100 150 177 33 33 100 100 ...
## $ F : int 3 2 2 2 2 2 2 1 3 3 ...
## $ MAG : num 0 0 0 0 0 0 0 0 0 0 ...
## $ FATALITIES: num 0 0 0 0 0 0 0 0 1 0 ...
## $ INJURIES : num 15 0 2 2 2 6 1 0 14 0 ...
## $ PROPDMG : num 25 2.5 25 2.5 2.5 2.5 2.5 2.5 25 25 ...
## $ PROPDMGEXP: Factor w/ 19 levels "","-","?","+",..: 17 17 17 17 17 17 17 17 17 17 ...
## $ CROPDMG : num 0 0 0 0 0 0 0 0 0 0 ...
## $ CROPDMGEXP: Factor w/ 9 levels "","?","0","2",..: 1 1 1 1 1 1 1 1 1 1 ...
## $ WFO : Factor w/ 542 levels ""," CI","%SD",..: 1 1 1 1 1 1 1 1 1 1 ...
## $ STATEOFFIC: Factor w/ 250 levels "","ALABAMA, Central",..: 1 1 1 1 1 1 1 1 1 1 ...
## $ ZONENAMES : Factor w/ 25112 levels ""," "| __truncated__,..: 1 1 1 1 1 1 1 1 1 1 ...
## $ LATITUDE : num 3040 3042 3340 3458 3412 ...
## $ LONGITUDE : num 8812 8755 8742 8626 8642 ...
## $ LATITUDE_E: num 3051 0 0 0 0 ...
## $ LONGITUDE_: num 8806 0 0 0 0 ...
## $ REMARKS : Factor w/ 436781 levels "","\t","\t\t",..: 1 1 1 1 1 1 1 1 1 1 ...
## $ REFNUM : num 1 2 3 4 5 6 7 8 9 10 ...Storm_new <- Storm #Copying the data in a new dataframe to manipulate it Loading the libraries
library(dplyr)##
## Attaching package: 'dplyr'## The following objects are masked from 'package:stats':
##
## filter, lag## The following objects are masked from 'package:base':
##
## intersect, setdiff, setequal, unionlibrary(ggplot2)
library(xtable)
library(DT)Now the data classes will be transformed to get the information that we want
Storm_new$BGN_DATE <- as.Date(Storm$BGN_DATE, "%m/%d/%Y")
Storm_new <- select(Storm_new, BGN_DATE, COUNTYNAME,
STATE, EVTYPE,FATALITIES, INJURIES, PROPDMG,
PROPDMGEXP, CROPDMG, CROPDMGEXP, STATEOFFIC) %>%
arrange(BGN_DATE)
Storm_new$EVTYPE <-as.character(Storm_new$EVTYPE)In order to perform the analysis, its important to know how many unique kind of weather events are in the data
weather.events <- unique(Storm_new$EVTYPE)
number.events <- length(weather.events)the unique events are 985
Looking the top 30 weather events according to their frequency
most.events <- as.data.frame(head(sort(table(Storm_new$EVTYPE), decreasing = T), 30))
DT::datatable(most.events,colnames = c("Weather event", "Frequency"))Now, in order to summarise the data into the variables that are from our interest, the weather events must be transformed into more concentrated unique events.
Storm_new$EVTYPE <- toupper(Storm_new$EVTYPE)
Storm_new$EVTYPE <- gsub("(.*)HIGH WIND(.*)","HIGH WINDS", Storm_new$EVTYPE)
Storm_new$EVTYPE <- gsub("(.*)HEAT(.*)", "HEAT",Storm_new$EVTYPE)
Storm_new$EVTYPE <- gsub("(.*)TORNADO(.*)","TORNADO", Storm_new$EVTYPE)
Storm_new$EVTYPE <- gsub("(.*)THUNDERSTORM WIND(.*)","THUNDERSTORM WINDS", Storm_new$EVTYPE)
Storm_new$EVTYPE <- gsub("(.*)HAIL(.*)", "HAIL", Storm_new$EVTYPE)
Storm_new$EVTYPE <- gsub("(.*)FLOOD(.*)", "FLOOD", Storm_new$EVTYPE)
Storm_new$EVTYPE <- gsub("(.*)WILD(.*)", "WILD FIRE", Storm_new$EVTYPE)
Storm_new$EVTYPE <- gsub("^[^WILD](.*)FIRE(.*)", "FIRE", Storm_new$EVTYPE)
Storm_new$EVTYPE <- gsub("LIGHTNING FIRE", "FIRE", Storm_new$EVTYPE)
Storm_new$EVTYPE <- gsub("(.*)LIGHTNING(.*)", "LIGHTNING", Storm_new$EVTYPE)
Storm_new$EVTYPE <- gsub("(.*)SNOW(.*)", "SNOW", Storm_new$EVTYPE)
Storm_new$EVTYPE <- gsub("(.*)WINTER(.*)", "WINTER WEATHER", Storm_new$EVTYPE)
Storm_new$EVTYPE <- gsub("(.*)MARINE(.*)", "MARINE TSTM", Storm_new$EVTYPE)
Storm_new$EVTYPE <- gsub("(.*)MARINE(.*)", "COLD", Storm_new$EVTYPE)
Storm_new$EVTYPE <- gsub("(.*)COLD(.*)", "EXTREME COLD", Storm_new$EVTYPE)
Storm_new$EVTYPE <- gsub("(.*)CLOUD(.*)", "FUNNEL CLOUD", Storm_new$EVTYPE)
Storm_new$EVTYPE <- gsub("(.*)FOG(.*)", "DENSE FOG", Storm_new$EVTYPE)
Storm_new$EVTYPE <- gsub("(.*)WIND(.*)", "HIGH WINDS", Storm_new$EVTYPE)
Storm_new$EVTYPE <- gsub("(.*)RAIN(.*)", "HEAVY RAIN", Storm_new$EVTYPE)
Storm_new$EVTYPE <- gsub("(.*)STORM(.*)", "STORM", Storm_new$EVTYPE)
Storm_new$EVTYPE <- gsub("(.*)WATER(.*)", "WATERSPOUT", Storm_new$EVTYPE)
Storm_new$EVTYPE <- gsub(".*FROST|FREEZE|BLIZZARD.*", "FROST/FREEZE", Storm_new$EVTYPE)
Storm_new$EVTYPE <- gsub(".*FROST.*", "FROST/FREEZE", Storm_new$EVTYPE)
Storm_new$EVTYPE <- gsub("(.*)URBAN(.*)", "URBAN STREAM", Storm_new$EVTYPE)
Storm_new$EVTYPE <- gsub("(.*)SURF(.*)", "HIGH SURF", Storm_new$EVTYPE)
Storm_new$EVTYPE <- gsub("(.*)LANDSLIDE(.*)", "LANDSLIDE", Storm_new$EVTYPE)
Storm_new$EVTYPE <- gsub("(.*)RIP(.*)", "RIP CURRENT", Storm_new$EVTYPE)
Storm_new$EVTYPE <- gsub("(.*)DRY(.*)", "DRY WEATHER", Storm_new$EVTYPE)
Storm_new$EVTYPE <- gsub("(.*)HURRICANE(.*)", "HURRICANE", Storm_new$EVTYPE)
Storm_new$EVTYPE <- gsub("(.*)WARMTH(.*)", "UNUSUAL WARMTH", Storm_new$EVTYPE)
Storm_new$EVTYPE <- gsub("(.*)SUMMARY(.*)", "OTHER", Storm_new$EVTYPE)
Storm_new$EVTYPE <- gsub("[?]", "OTHER", Storm_new$EVTYPE)
Storm_new$EVTYPE <- gsub("(.*)VOLCANIC(.*)", "VOLCANIC ASH", Storm_new$EVTYPE)
Storm_new$EVTYPE <- gsub("(.*)SWELLS(.*)", "SWELLS", Storm_new$EVTYPE)
Storm_new$EVTYPE <- gsub("(.*)SEA(.*)", "ROUGH SEAS", Storm_new$EVTYPE)
Storm_new$EVTYPE <- gsub("(.*)TSTM(.*)", "THUNDERSTORM", Storm_new$EVTYPE)
other <- names(sort(table(Storm_new$EVTYPE), decreasing = T))
other <- other[31:length(other)]
indexes <- NULL
for (i in other){
indexes <- append(indexes, which(Storm_new$EVTYPE == i))
}
Storm_new$EVTYPE[indexes] <- "OTHER"It’s important to trasnform the PROPDMG, PROPDMGEXP, CROPDMG and CROPDMGEXP to summarise the data as needed. Likewise, the complementary documentation of the storm data expresses that those variables are encoded according to their value
unique(Storm_new$PROPDMGEXP)## [1] K M B 0 ? 6 5 4 h m + H 3 2 1 7 8 -
## Levels: - ? + 0 1 2 3 4 5 6 7 8 B h H K m MStorm_new$PROPDMGEXP <- as.character(Storm_new$PROPDMGEXP)
Storm_new$PROPDMGEXP<- gsub("[?]|[-]|[+]", "0" ,Storm_new$PROPDMGEXP)
Storm_new$PROPDMGEXP<- gsub("1", "10",Storm_new$PROPDMGEXP)
Storm_new$PROPDMGEXP<- gsub("[Hh]|2", "100" ,Storm_new$PROPDMGEXP)
Storm_new$PROPDMGEXP<- gsub("[Kk]|3", "1000" ,Storm_new$PROPDMGEXP)
Storm_new$PROPDMGEXP<- gsub("4", "10000" ,Storm_new$PROPDMGEXP)
Storm_new$PROPDMGEXP<- gsub("5", "100000" ,Storm_new$PROPDMGEXP)
Storm_new$PROPDMGEXP<- gsub("[Mm]|6", "1000000" ,Storm_new$PROPDMGEXP)
Storm_new$PROPDMGEXP<- gsub("7", "10000000" ,Storm_new$PROPDMGEXP)
Storm_new$PROPDMGEXP<- gsub("8", "100000000" ,Storm_new$PROPDMGEXP)
Storm_new$PROPDMGEXP<- gsub("[Bb]", "1000000000" ,Storm_new$PROPDMGEXP)
nans <- is.na(as.numeric(Storm_new$PROPDMGEXP))
Storm_new$PROPDMGEXP[nans] <- "0"
Storm_new$PROPDMG <- Storm_new$PROPDMG*as.numeric(Storm_new$PROPDMGEXP)Storm_new$CROPDMGEXP <- as.character(Storm_new$CROPDMGEXP)
unique(Storm_new$CROPDMGEXP)## [1] "" "K" "M" "?" "B" "0" "k" "2" "m"Storm_new$CROPDMGEXP<- gsub("[?]", "0" ,Storm_new$CROPDMGEXP)
Storm_new$CROPDMGEXP<- gsub("[Hh]|2", "100" ,Storm_new$CROPDMGEXP)
Storm_new$CROPDMGEXP<- gsub("[Kk]|3", "1000" ,Storm_new$CROPDMGEXP)
Storm_new$CROPDMGEXP<- gsub("[Mm]", "1000000" ,Storm_new$CROPDMGEXP)
Storm_new$CROPDMGEXP<- gsub("[Bb]", "1000000000" ,Storm_new$CROPDMGEXP)
nans <- is.na(as.numeric(Storm_new$CROPDMGEXP))
Storm_new$CROPDMGEXP[nans] <- "0"
Storm_new$CROPDMG <- Storm_new$CROPDMG*as.numeric(Storm_new$CROPDMGEXP)5 Summarizing the data and results
In order to present the results, the fatalities, injuries, property damage and crop damage will be presented in summarized data tables per event, and later on plots.
# Fatalities
Fatalities <- select(Storm_new, EVTYPE, FATALITIES) %>%
group_by(EVTYPE) %>% summarise(Fatalities = sum(FATALITIES)) %>%
arrange(desc(Fatalities))
names(Fatalities) <- c("Weather.Events", "Fatalities")
Events <- Fatalities$Weather.Events
Fatalities$Weather.Events <- factor(Fatalities$Weather.Events, levels = Events,
labels = Events)
DT::datatable(Fatalities)#Injuries
Injuries <- select(Storm_new, EVTYPE, INJURIES) %>%
group_by(EVTYPE) %>% summarise(Injuries = sum(INJURIES)) %>%
arrange(desc(Injuries))
names(Injuries) <- c("Weather.Events", "Injuries")
Events.injuries <- Injuries$Weather.Events
Injuries$Weather.Events <- factor(Injuries$Weather.Events, levels = Events.injuries,
labels = Events.injuries)
DT::datatable(Injuries)#Total damage
Total.damage <- select(Storm_new,EVTYPE, PROPDMG, CROPDMG) %>%
mutate(Total_Damage = PROPDMG+ CROPDMG) %>%
group_by(EVTYPE) %>% summarise_all(.funs = sum) %>%
arrange(desc(Total_Damage))
names(Total.damage) <- c("Weather.Events", "Property.damage",
"Crop.damage", "Total.damage")
Events.damage <- Total.damage$Weather.Events
Total.damage$Weather.Events <- factor(Total.damage$Weather.Events,
levels = Events.damage,
labels = Events.damage)
DT::datatable(Total.damage)6 Plotting results
Now that the data is as its needed, it will be plotted.
fatalities.plot <- ggplot(Fatalities[1:15,],
aes(x = Weather.Events, y = Fatalities,
fill = Weather.Events)) + geom_bar(stat = "identity" )
fatalities.plot <- fatalities.plot + xlab("Weather event") +
ylab("Fatalities") +
ggtitle("Top 15 Weather events per Fatalities")
fatalities.plot <- fatalities.plot + geom_text(aes(label = Fatalities), size = 3, vjust = -.3) +
theme(axis.text.x = element_text(angle = 45, hjust = 1))
fatalities.plot
injuries.plot <- ggplot(Injuries[1:15,],
aes(x = Weather.Events, y = Injuries,
fill = Weather.Events)) + geom_bar(stat = "identity")
injuries.plot <- injuries.plot + xlab("Weather event") +
ylab("Injuries") +
ggtitle("Top 15 Weather events per Injuries")
injuries.plot <- injuries.plot + geom_text(aes(label=Injuries),size = 3, vjust = -.3)
injuries.plot <- injuries.plot + theme(axis.text.x = element_text(angle = 45, hjust = 1))
injuries.plot
According to the two previous plots and the charts presented above, the most harmful type of the events with respect of the population health across the U.S. are the Tornado type of events
damage.plot <- ggplot(Total.damage[1:15,], aes(Weather.Events, Total.damage, fill =Weather.Events))+
geom_bar(stat = "identity")
damage.plot <- damage.plot + labs(x = "Weather event", y = "Total Damage",
title = "Top 15 Weather events per damage")
damage.plot <- damage.plot + geom_text(aes(label = Total.damage), size = 3, vjust = -.3)
damage.plot <- damage.plot + theme(axis.text.x = element_text(angle = 45, hjust = 1))
damage.plot