National Weather Service Storm Data Documentation - Impact Analysis
Synopsis
This article examines “National Weather Service Storm Data Documentation” with the objective of understanding the impact of storms. The article documents the loading, preparation and analysis of the data provided, and answers questions about the human and financial impact of extreme weather events.
Data Processing
This section describes the process behind loading, shaping and preparing the data for analysis.
Initial Load
Load in the data from the file - this assumes that the file is in the working directory.
storm <- read.csv(bzfile("StormData.csv.bz2"), header=TRUE)Intermediate Steps
We’ll have a first look at the dataset in order to identify which components are relevant for us.
str(storm)## 'data.frame': 902297 obs. of 37 variables:
## $ STATE__ : num 1 1 1 1 1 1 1 1 1 1 ...
## $ BGN_DATE : Factor w/ 16335 levels "1/1/1966 0:00:00",..: 6523 6523 4242 11116 2224 2224 2260 383 3980 3980 ...
## $ BGN_TIME : Factor w/ 3608 levels "00:00:00 AM",..: 272 287 2705 1683 2584 3186 242 1683 3186 3186 ...
## $ TIME_ZONE : Factor w/ 22 levels "ADT","AKS","AST",..: 7 7 7 7 7 7 7 7 7 7 ...
## $ COUNTY : num 97 3 57 89 43 77 9 123 125 57 ...
## $ COUNTYNAME: Factor w/ 29601 levels "","5NM E OF MACKINAC BRIDGE TO PRESQUE ISLE LT MI",..: 13513 1873 4598 10592 4372 10094 1973 23873 24418 4598 ...
## $ STATE : Factor w/ 72 levels "AK","AL","AM",..: 2 2 2 2 2 2 2 2 2 2 ...
## $ EVTYPE : Factor w/ 985 levels " HIGH SURF ADVISORY",..: 834 834 834 834 834 834 834 834 834 834 ...
## $ BGN_RANGE : num 0 0 0 0 0 0 0 0 0 0 ...
## $ BGN_AZI : Factor w/ 35 levels ""," N"," NW",..: 1 1 1 1 1 1 1 1 1 1 ...
## $ BGN_LOCATI: Factor w/ 54429 levels ""," Christiansburg",..: 1 1 1 1 1 1 1 1 1 1 ...
## $ END_DATE : Factor w/ 6663 levels "","1/1/1993 0:00:00",..: 1 1 1 1 1 1 1 1 1 1 ...
## $ END_TIME : Factor w/ 3647 levels ""," 0900CST",..: 1 1 1 1 1 1 1 1 1 1 ...
## $ COUNTY_END: num 0 0 0 0 0 0 0 0 0 0 ...
## $ COUNTYENDN: logi NA NA NA NA NA NA ...
## $ END_RANGE : num 0 0 0 0 0 0 0 0 0 0 ...
## $ END_AZI : Factor w/ 24 levels "","E","ENE","ESE",..: 1 1 1 1 1 1 1 1 1 1 ...
## $ END_LOCATI: Factor w/ 34506 levels ""," CANTON"," TULIA",..: 1 1 1 1 1 1 1 1 1 1 ...
## $ LENGTH : num 14 2 0.1 0 0 1.5 1.5 0 3.3 2.3 ...
## $ WIDTH : num 100 150 123 100 150 177 33 33 100 100 ...
## $ F : int 3 2 2 2 2 2 2 1 3 3 ...
## $ MAG : num 0 0 0 0 0 0 0 0 0 0 ...
## $ FATALITIES: num 0 0 0 0 0 0 0 0 1 0 ...
## $ INJURIES : num 15 0 2 2 2 6 1 0 14 0 ...
## $ PROPDMG : num 25 2.5 25 2.5 2.5 2.5 2.5 2.5 25 25 ...
## $ PROPDMGEXP: Factor w/ 19 levels "","-","?","+",..: 17 17 17 17 17 17 17 17 17 17 ...
## $ CROPDMG : num 0 0 0 0 0 0 0 0 0 0 ...
## $ CROPDMGEXP: Factor w/ 9 levels "","?","0","2",..: 1 1 1 1 1 1 1 1 1 1 ...
## $ WFO : Factor w/ 542 levels ""," CI","%SD",..: 1 1 1 1 1 1 1 1 1 1 ...
## $ STATEOFFIC: Factor w/ 250 levels "","ALABAMA, Central",..: 1 1 1 1 1 1 1 1 1 1 ...
## $ ZONENAMES : Factor w/ 25112 levels ""," "| __truncated__,..: 1 1 1 1 1 1 1 1 1 1 ...
## $ LATITUDE : num 3040 3042 3340 3458 3412 ...
## $ LONGITUDE : num 8812 8755 8742 8626 8642 ...
## $ LATITUDE_E: num 3051 0 0 0 0 ...
## $ LONGITUDE_: num 8806 0 0 0 0 ...
## $ REMARKS : Factor w/ 436781 levels "","\t","\t\t",..: 1 1 1 1 1 1 1 1 1 1 ...
## $ REFNUM : num 1 2 3 4 5 6 7 8 9 10 ...Now we have the initial dataset in memory we will need to optimize the data structure. We want two subsets, one for life & health, and one for property, and in both cases only where we have positive values:
lah <- subset(storm, FATALITIES > 0 | INJURIES > 0)
prop <- subset(storm, PROPDMG > 0 | CROPDMG > 0)
##clear up some memory and drop the original object.
remove(storm)Analysis
This section details with the analysis of the dataset. Firstly we will load packages required for the analysis.
library(ggplot2)
library(sqldf)Using sqldf we will aggregate the data by event type, make some efforts to reduce the number of events, and re-aggregate. The tidy dataset will be used to produce our results:
glah <- sqldf("select sum(FATALITIES), sum(INJURIES), EVTYPE from lah group by EVTYPE")
##decide we need to tidy up EVTYPE by removing numbers, spaces, forward slashes...
glah$EVTYPE <- gsub("([0-9]+).*$", "", glah$EVTYPE)
glah$EVTYPE <- gsub(" ", "", glah$EVTYPE)
glah$EVTYPE <- gsub("/", "", glah$EVTYPE)
glah$EVTYPE <- toupper(glah$EVTYPE)
##fix columns and re-aggregate
names(glah) <- c("FATALITIES", "INJURIES", "EVTYPE")
glah <- sqldf("select sum(FATALITIES), sum(INJURIES), EVTYPE from glah group by EVTYPE")
names(glah) <- c("FATALITIES", "INJURIES", "EVTYPE")
##we will order the dataset by fatalities and we'll only take the top 10.
resultslah <- head((glah[order(glah$FATALITIES, decreasing=TRUE),]), 10)
##now look at the financial impact - rinse and repeat
gfin <- sqldf("select sum(CROPDMG), sum(PROPDMG), EVTYPE from prop group by EVTYPE")
gfin$EVTYPE <- gsub("([0-9]+).*$", "", gfin$EVTYPE)
gfin$EVTYPE <- gsub(" ", "", gfin$EVTYPE)
gfin$EVTYPE <- gsub("/", "", gfin$EVTYPE)
gfin$EVTYPE <- toupper(gfin$EVTYPE)
##fix columns and re-aggregate
names(gfin) <- c("CROPDMG", "PROPDMG", "EVTYPE")
gfin <- sqldf("select sum(CROPDMG), sum(PROPDMG), EVTYPE from prop group by EVTYPE")
names(gfin) <- c("CROPDMG", "PROPDMG", "EVTYPE")
resultsfin <- head((gfin[order(gfin$PROPDMG, decreasing=TRUE),]), 10)
##round the numbers
resultsfin$CROPDMG <- round(resultsfin$CROPDMG/1000000, 4)
resultsfin$PROPDMG <- round(resultsfin$PROPDMG/1000000, 4) Results
We are now in a position where we can answer the questions raised.
I will however make two assumptions. In the first question, I will assume that a fatality is more significant than an injury, and for the second question that property damage is more significant than crop damage and that financial impact is the sole measure of economic impact.
Question 1:
Across the United States, which types of events (as indicated in the EVTYPE variable) are most harmful with respect to population health?
g <- ggplot(resultslah, aes(x=EVTYPE, y=FATALITIES, fill=factor(INJURIES))) + geom_bar(stat="identity", colour="white") + theme(axis.text.x = element_text(angle = 270, hjust = 1)) + ggtitle("Fatalities and Injuries due to extreme weather events")
g
The answer: Tornados.
Question 2:
Across the United States, which types of events have the greatest economic consequences?
g <- ggplot(resultsfin, aes(x=EVTYPE, y=PROPDMG, fill=factor(CROPDMG))) + geom_bar(stat="identity", colour="white") + theme(axis.text.x = element_text(angle = 270, hjust = 1)) + ggtitle("Damage in millions of USD due to extreme weather events")
g
The answer: strong winds, whether in thunderstorms or in Tornados.