This analysis report explores the U.S. National Oceanic and Atmospheric Administration's (NOAA) Storm Database and answering some basic questions about severe weather events through some data processing with results presented in visual plots and tables.
The analysis trying to resolve questions:
1. Which types of events (as indicated in the EVTYPE variable) are most harmful with respect to population health across the United States?
2. Which types of events have the greatest economic consequences across the United States?
Storms and other severe weather events can cause both public health and economic problems for communities and municipalities. Many severe events can result in fatalities, injuries, and property damages, and preventing such outcomes to the extent possible is a key concern.
The NOAA database tracks characteristics of major storms and weather events in the United States, including when and where they occur, as well as estimates of any fatalities, injuries, and property damage.
Data repository:
Storm Data [47Mb]
Documentation:
National Weather Service Storm Data Documentation
National Climatic Data Center Storm Events FAQ
library(R.utils)
library(ggplot2)
library(knitr)
library(rapport)
library(reshape2)
require(gridExtra)
opts_chunk$set(cache=TRUE)
Download database
db_url <- "https://d396qusza40orc.cloudfront.net/repdata%2Fdata%2FStormData.csv.bz2"
db_zip <- "StormData.csv.bz2"
db_csv <- "StormData.csv"
if (!file.exists(db_zip)) {
download.file(db_url, db_zip, method="curl")
}
Extract database
if (!file.exists(db_csv)){
bunzip2(db_zip, db_csv, remove=FALSE)
}
Read as data frame
data <- read.csv(db_csv, sep=",", header=TRUE)
Explore data by colnames(data), head(data), or str(data.
str(data)
## 'data.frame': 902297 obs. of 37 variables:
## $ STATE__ : num 1 1 1 1 1 1 1 1 1 1 ...
## $ BGN_DATE : Factor w/ 16335 levels "1/1/1966 0:00:00",..: 6523 6523 4242 11116 2224 2224 2260 383 3980 3980 ...
## $ BGN_TIME : Factor w/ 3608 levels "00:00:00 AM",..: 272 287 2705 1683 2584 3186 242 1683 3186 3186 ...
## $ TIME_ZONE : Factor w/ 22 levels "ADT","AKS","AST",..: 7 7 7 7 7 7 7 7 7 7 ...
## $ COUNTY : num 97 3 57 89 43 77 9 123 125 57 ...
## $ COUNTYNAME: Factor w/ 29601 levels "","5NM E OF MACKINAC BRIDGE TO PRESQUE ISLE LT MI",..: 13513 1873 4598 10592 4372 10094 1973 23873 24418 4598 ...
## $ STATE : Factor w/ 72 levels "AK","AL","AM",..: 2 2 2 2 2 2 2 2 2 2 ...
## $ EVTYPE : Factor w/ 985 levels " HIGH SURF ADVISORY",..: 834 834 834 834 834 834 834 834 834 834 ...
## $ BGN_RANGE : num 0 0 0 0 0 0 0 0 0 0 ...
## $ BGN_AZI : Factor w/ 35 levels ""," N"," NW",..: 1 1 1 1 1 1 1 1 1 1 ...
## $ BGN_LOCATI: Factor w/ 54429 levels "","- 1 N Albion",..: 1 1 1 1 1 1 1 1 1 1 ...
## $ END_DATE : Factor w/ 6663 levels "","1/1/1993 0:00:00",..: 1 1 1 1 1 1 1 1 1 1 ...
## $ END_TIME : Factor w/ 3647 levels ""," 0900CST",..: 1 1 1 1 1 1 1 1 1 1 ...
## $ COUNTY_END: num 0 0 0 0 0 0 0 0 0 0 ...
## $ COUNTYENDN: logi NA NA NA NA NA NA ...
## $ END_RANGE : num 0 0 0 0 0 0 0 0 0 0 ...
## $ END_AZI : Factor w/ 24 levels "","E","ENE","ESE",..: 1 1 1 1 1 1 1 1 1 1 ...
## $ END_LOCATI: Factor w/ 34506 levels "","- .5 NNW",..: 1 1 1 1 1 1 1 1 1 1 ...
## $ LENGTH : num 14 2 0.1 0 0 1.5 1.5 0 3.3 2.3 ...
## $ WIDTH : num 100 150 123 100 150 177 33 33 100 100 ...
## $ F : int 3 2 2 2 2 2 2 1 3 3 ...
## $ MAG : num 0 0 0 0 0 0 0 0 0 0 ...
## $ FATALITIES: num 0 0 0 0 0 0 0 0 1 0 ...
## $ INJURIES : num 15 0 2 2 2 6 1 0 14 0 ...
## $ PROPDMG : num 25 2.5 25 2.5 2.5 2.5 2.5 2.5 25 25 ...
## $ PROPDMGEXP: Factor w/ 19 levels "","-","?","+",..: 17 17 17 17 17 17 17 17 17 17 ...
## $ CROPDMG : num 0 0 0 0 0 0 0 0 0 0 ...
## $ CROPDMGEXP: Factor w/ 9 levels "","?","0","2",..: 1 1 1 1 1 1 1 1 1 1 ...
## $ WFO : Factor w/ 542 levels ""," CI","$AC",..: 1 1 1 1 1 1 1 1 1 1 ...
## $ STATEOFFIC: Factor w/ 250 levels "","ALABAMA, Central",..: 1 1 1 1 1 1 1 1 1 1 ...
## $ ZONENAMES : Factor w/ 25112 levels ""," "| __truncated__,..: 1 1 1 1 1 1 1 1 1 1 ...
## $ LATITUDE : num 3040 3042 3340 3458 3412 ...
## $ LONGITUDE : num 8812 8755 8742 8626 8642 ...
## $ LATITUDE_E: num 3051 0 0 0 0 ...
## $ LONGITUDE_: num 8806 0 0 0 0 ...
## $ REMARKS : Factor w/ 436774 levels "","-2 at Deer Park\n",..: 1 1 1 1 1 1 1 1 1 1 ...
## $ REFNUM : num 1 2 3 4 5 6 7 8 9 10 ...
Keep only columns related to event type, cumulative fatalities, injuries, and damages.
EXP is exponent (unit of measure), i.e. use unique(data$CROPDMGEXP) to explore possible values.
See below for exponent explanation.
cols <- c("EVTYPE", "FATALITIES", "INJURIES", "CROPDMG", "CROPDMGEXP", "PROPDMG", "PROPDMGEXP")
data <- subset(data, select = cols)
# trim leading and trailing spaces of EVTYPE character
data$EVTYPE <- trim.space(data$EVTYPE)
# all upper case to ease data cleaning
data$EVTYPE <- toupper(data$EVTYPE)
dim(data)
## [1] 902297 8
head(data)
## X EVTYPE FATALITIES INJURIES CROPDMG CROPDMGEXP PROPDMG PROPDMGEXP
## 1 1 TORNADO 0 15 0 25.0 K
## 2 2 TORNADO 0 0 0 2.5 K
## 3 3 TORNADO 0 2 0 25.0 K
## 4 4 TORNADO 0 2 0 2.5 K
## 5 5 TORNADO 0 2 0 2.5 K
## 6 6 TORNADO 0 6 0 2.5 K
Some EVTYPE values has abbreviations,
## substitute abbreviations
data$EVTYPE <- gsub("CSTL", "COASTAL", data$EVTYPE)
data$EVTYPE <- gsub("FLDG", "FLOOD", data$EVTYPE)
data$EVTYPE <- gsub("HVY", "HEAVY", data$EVTYPE)
data$EVTYPE <- gsub("SML", "SMALL", data$EVTYPE)
data$EVTYPE <- gsub("STRM", "STREAM", data$EVTYPE)
data$EVTYPE <- gsub("TSTMW", "THUNDERSTORM WIND", data$EVTYPE)
data$EVTYPE <- gsub("TSTM", "THUNDERSTORM", data$EVTYPE)
data$EVTYPE <- gsub("VOG", "FOG", data$EVTYPE)
data$EVTYPE <- gsub("WND", "WIND", data$EVTYPE)
Exponent values of economic damages, CROPDMGEXP and PROPDMGEXP:
B/b = billions, M/m = millions, K/k = kilos, H/h = hundreds.0..8 = 10- = 0, ? = 0, + = 1.How to resolve this symbol, was discussed in the course's forum.
Hints are provided by David Hood (Data Science Specialization's CTA), as well as by Eddie Song (class fellow in course). Credit goes to them.
This is analysis compilation of this exponent values in my RPubs repo, or in Markdown or PDF file, on how to handle these exponent values.
Set CROPDMGEXP and PROPDMGEXP values to multiplier number
## 0..8
data$CROPDMGEXP <- gsub("[[:digit:]]", "10", data$CROPDMGEXP)
data$PROPDMGEXP <- gsub("[[:digit:]]", "10", data$PROPDMGEXP)
## +
data$CROPDMGEXP <- gsub("\\+", "1", data$CROPDMGEXP)
data$PROPDMGEXP <- gsub("\\+", "1", data$PROPDMGEXP)
## -,?
data$CROPDMGEXP <- gsub("[-\\?]", "0", data$CROPDMGEXP)
data$PROPDMGEXP <- gsub("[-\\?]", "0", data$PROPDMGEXP)
## H,h, K,k, M,m, B,b
data$CROPDMGEXP <- gsub("[Hh]", "100", data$CROPDMGEXP)
data$CROPDMGEXP <- gsub("[Kk]", "1000", data$CROPDMGEXP)
data$CROPDMGEXP <- gsub("[Mm]", "1000000", data$CROPDMGEXP)
data$CROPDMGEXP <- gsub("[Bb]", "1000000000", data$CROPDMGEXP)
data$PROPDMGEXP <- gsub("[Hh]", "100", data$PROPDMGEXP)
data$PROPDMGEXP <- gsub("[Kk]", "1000", data$PROPDMGEXP)
data$PROPDMGEXP <- gsub("[Mm]", "1000000", data$PROPDMGEXP)
data$PROPDMGEXP <- gsub("[Bb]", "1000000000", data$PROPDMGEXP)
## clean exponent data
data$CROPDMGEXP[data$CROPDMGEXP == ""] <- 0 # empty character as 0
data$PROPDMGEXP[data$PROPDMGEXP == ""] <- 0
data$CROPDMGEXP <- as.numeric(data$CROPDMGEXP) # convert to numeric class
data$PROPDMGEXP <- as.numeric(data$PROPDMGEXP)
## calculate $ value = (number * exponent-number)
data <- mutate(data, NEWCROPDMG = CROPDMG * CROPDMGEXP)
data <- mutate(data, NEWPROPDMG = PROPDMG * PROPDMGEXP)
## summarize population health, injuries, fatalities
total <- aggregate(cbind(INJURIES, FATALITIES) ~ EVTYPE, data, FUN=sum, na.rm=TRUE)
## sort and filter top only
total <- arrange(total, desc(FATALITIES + INJURIES))[1:12,]
## convert wide to long format
totalLong <- melt(total, id.vars="EVTYPE")
xaxis <- reorder(totalLong$EVTYPE, -(totalLong$value))
gp <- ggplot(aes(x=xaxis, y=value, fill=variable), data=totalLong) +
geom_bar(stat="identity", position="stack", colour="black") +
scale_y_continuous(breaks=seq(from=0, to=1e5, by=5000)) +
labs(x="Event Type", y="Number of Population Affected",
title="Most Harmful Events to Population Health in the United States \n(Total Stacked, Injuries and Fatalities)") +
theme(axis.text.x=element_text(angle=40, hjust=1))
print(gp)
The plots and table show us that tornado, thunderstorm wind and excessive heat causes most harms to the population health.
table_total <- mutate(total, Total = INJURIES + FATALITIES)
table_total <- format(table_total, big.mark=",", decimal.mark=".")
kable(table_total, format="html", col.names=c("Event Type", "Injuries", "Fatalities", "Total"),
align=c("l", "r", "r", "r"))
| Event Type | Injuries | Fatalities | Total |
|---|---|---|---|
| TORNADO | 91,346 | 5,633 | 96,979 |
| THUNDERSTORM WIND | 8,445 | 637 | 9,082 |
| EXCESSIVE HEAT | 6,525 | 1,903 | 8,428 |
| FLOOD | 6,789 | 470 | 7,259 |
| LIGHTNING | 5,230 | 816 | 6,046 |
| HEAT | 2,100 | 937 | 3,037 |
| FLASH FLOOD | 1,777 | 978 | 2,755 |
| ICE STORM | 1,975 | 89 | 2,064 |
| WINTER STORM | 1,321 | 206 | 1,527 |
| HIGH WIND | 1,137 | 248 | 1,385 |
| HAIL | 1,361 | 15 | 1,376 |
| HURRICANE/TYPHOON | 1,275 | 64 | 1,339 |
## summarize, economic damages, crop and property
damage <- aggregate(cbind(NEWPROPDMG, NEWCROPDMG) ~ EVTYPE, data, FUN=sum, na.rm=TRUE)
## sort and filter top only
damage <- arrange(damage, desc(NEWPROPDMG + NEWCROPDMG))[1:12,]
## convert wide to long format
damageLong <- melt(damage, id.vars="EVTYPE")
xaxis <- reorder(damageLong$EVTYPE, -(damageLong$value))
gp <- ggplot(aes(x=xaxis, y=value/1e9, fill=variable), data=damageLong) +
geom_bar(stat="identity", position="stack", colour="black") +
scale_y_continuous(breaks=seq(from=0, to=150, by=10)) +
labs(x="Event Type", y="Economic damages ($ billion)",
title="Greatest Economic Consequences in the United States \n(Total Stacked, Property and Crop)") +
theme(axis.text.x=element_text(angle=40, hjust=1))
print(gp)
The plots and table show us that, flood, hurricane / typhoon and tornado causes most economic damages.
table_damage <- mutate(damage, Total = NEWPROPDMG + NEWCROPDMG)
table_damage <- format(table_damage, big.mark=",", decimal.mark=".")
kable(table_damage, format="html", col.names=c("Event Type", "Property Damages", "Crop Damages", "Total"),
align=c("l", "r", "r", "r"))
| Event Type | Property Damages | Crop Damages | Total |
|---|---|---|---|
| FLOOD | 144,657,709,800 | 5,661,968,450 | 150,319,678,250 |
| HURRICANE/TYPHOON | 69,305,840,000 | 2,607,872,800 | 71,913,712,800 |
| TORNADO | 56,937,162,897 | 414,954,710 | 57,352,117,607 |
| STORM SURGE | 43,323,536,000 | 5,000 | 43,323,541,000 |
| HAIL | 15,732,269,877 | 3,025,954,650 | 18,758,224,527 |
| FLASH FLOOD | 16,140,865,011 | 1,421,317,100 | 17,562,182,111 |
| DROUGHT | 1,046,106,000 | 13,972,566,000 | 15,018,672,000 |
| HURRICANE | 11,868,319,010 | 2,741,910,000 | 14,610,229,010 |
| RIVER FLOOD | 5,118,945,500 | 5,029,459,000 | 10,148,404,500 |
| ICE STORM | 3,944,928,310 | 5,022,113,500 | 8,967,041,810 |
| THUNDERSTORM WIND | 7,976,231,550 | 968,850,400 | 8,945,081,950 |
| TROPICAL STORM | 7,703,890,550 | 678,346,000 | 8,382,236,550 |