Weather events cause huge health and economic problems across the United States
Weather events cause huge health and economic problems across the United States
This project involves exploring the U.S. National Oceanic and Atmospheric Administration’s (NOAA) storm database. This database tracks characteristics of major storms and weather events in the United States, including when and where they occur, as well as estimates of any fatalities, injuries, and property damage.
The data for this assignment come in the form of a comma-separated-value file compressed via the bzip2 algorithm to reduce its size. You can download the file from the course web site:
- Data used for the assigment: Storm data
There is also some documentation of the database available. Here you will find how some of the variables are constructed/defined.
- National Weather Service Storm Data Documentation
- National Climatic Data Center Storm Events FAQ
The events in the database start in the year 1950 and end in November 2011. In the earlier years of the database there are generally fewer events recorded, most likely due to a lack of good records. More recent years should be considered more complete.
The basic goal of this project is to explore the NOAA Storm Database and answer some basic questions about severe weather events.
0. Data Procesing
Loading libraries needed for this project
library(data.table)
library(dplyr)
library(lubridate)Downloading data if needed
location<-getwd()
data_url<-"https://d396qusza40orc.cloudfront.net/repdata%2Fdata%2FStormData.csv.bz2"
file_name<-"storm_data.csv.bz2"
if(!file.exists(paste0(location, "/", file_name))){
download.file(url=data_url,
destfile = paste0(location, "/", file_name),
method="curl")}Loading data. Fread function is used, because it worked faster than read.table and read.csv
storm_data<-fread("storm_data.csv.bz2")Converting date column from character to date class.
storm_data$BGN_DATE<- mdy_hms(storm_data$BGN_DATE)Adding YEAR column to the dataset. Adding HEALTH_DMG and ECONOMIC_DMG collumns to the dataset
Selecting only columns needed for analysis.
storm_data<- storm_data %>%
mutate(YEAR=year(storm_data$BGN_DATE),
HEALTH_DMG=FATALITIES+ INJURIES,
ECONOMIC_DMG=PROPDMG+CROPDMG) %>%
select(YEAR, EVTYPE, HEALTH_DMG ,ECONOMIC_DMG)Lets check how many different type of storm event were calculated each year
check<- storm_data %>% group_by(YEAR)
check<-check %>% summarise(types=length(unique(EVTYPE)))
check$YEAR[1:50]## [1] 1950 1951 1952 1953 1954 1955 1956 1957 1958 1959 1960 1961 1962 1963 1964
## [16] 1965 1966 1967 1968 1969 1970 1971 1972 1973 1974 1975 1976 1977 1978 1979
## [31] 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994
## [46] 1995 1996 1997 1998 1999We can see that until 1993 only a few types of storm events were recorded. Therefore, we will analyse only data from 1993.
storm_data<- storm_data %>% filter(YEAR>=1993)There are 48 storm events types officialy listed in Storm Data Documentation (page 6). They are stored in the event variable:
event<-c("Astronomical Low Tide", "Avalanche", "Blizzard", "Coastal Flood", "Cold/Wind Chill", "Debris Flow", "Dense Fog", "Dense Smoke", "Drought", "Dust Devil", "Dust Storm", "Excessive Heat", "Extreme Cold/Wind Chill", "Flash Flood", "Flood", "Frost/Freeze", "Funnel Cloud", "Freezing Fog", "Hail", "Heat", "Heavy Rain", "Heavy Snow", "High Surf", "High Wind","Hurricane (Typhoon)", "Ice Storm", "Lake-Effect Snow", "Lakeshore Flood", "Lightning", "Marine Hail", "Marine High Wind", "Marine Strong Wind", "Marine Thunderstorm Wind", "Rip Current", "Seiche", "Sleet", "Storm Surge/Tide", "Strong Wind", "Thunderstorm Wind", "Tornado", "Tropical Depression", "Tropical Storm", "Tsunami", "Volcanic Ash", "Waterspout", "Wildfire", "Winter Storm", "Winter Weather")However, there are way more (nearly thounsand) types of storm events in storm_data dataset. The reason is typing mistakes, naming same type of events slightly differently, etc.
str(unique(storm_data$EVTYPE))## chr [1:985] "FREEZING RAIN" "SNOW" "ICE STORM/FLASH FLOOD" "SNOW/ICE" ...To fix this, partial matching is needed. The following function does partial matching and then renames storm events so they would match with the ones from the Storm Data Documentation
partial_matching <- function(event_names, max_dist=0.001, rename) {
for (i in 1:length(event_names)){
matches <- lapply(event_names[i], agrepl, storm_data$EVTYPE,
max.distance = max_dist,
ignore.case = TRUE, fixed = TRUE)
if (event_names==event){rename<-event_names[i]}
matches<-data.frame(matches)
storm_data$EVTYPE[matches==TRUE]<-rename}
return(storm_data)
}Narrowing down event types applying partial matching function:
storm_data<-partial_matching(event_names = event)Event types are narrowed down quite a lot now, therefore we can give a quick look at them.
unique(storm_data$EVTYPE)## [1] "FREEZING RAIN" "SNOW"
## [3] "Flood" "SNOW/ICE"
## [5] "Winter Storm" "High Wind"
## [7] "Hail" "Thunderstorm Wind"
## [9] "Tornado" "RECORD COLD"
## [11] "HURRICANE ERIN" "HURRICANE OPAL"
## [13] "Heat" "Lightning"
## [15] "Dense Fog" "Rip Current"
## [17] "Funnel Cloud" "WIND"
## [19] "FUNNEL" "WALL CLOUD"
## [21] "COLD" "THUNDERSTORM"
## [23] "Waterspout" "EXTREME COLD"
## [25] "Blizzard" "FREEZE"
## [27] "HIGH TIDES" "RECORD HIGH TEMPERATURE"
## [29] "RECORD HIGH" "Ice Storm"
## [31] "RECORD LOW" "LOW TEMPERATURE RECORD"
## [33] "Avalanche" "MARINE MISHAP"
## [35] "HIGH TEMPERATURE RECORD" "RECORD HIGH TEMPERATURES"
## [37] "HIGH SEAS" "SEVERE TURBULENCE"
## [39] "RECORD RAINFALL" "RECORD SNOWFALL"
## [41] "RECORD WARMTH" "WIND DAMAGE"
## [43] "Dust Storm" "APACHE COUNTY"
## [45] "Sleet" "Dust Devil"
## [47] "TSTM WIND" "GUSTY WINDS"
## [49] "Strong Wind" "SNOW AND WIND"
## [51] "High Surf" "BLOWING DUST"
## [53] "URBAN/SMALL" "Wildfire"
## [55] "HIGH" "MUDSLIDES"
## [57] "RAINSTORM" "SEVERE THUNDERSTORM"
## [59] "SEVERE THUNDERSTORMS" "DRY MICROBURST"
## [61] "WINDS" "DRY MICROBURST 61"
## [63] "THUNDERSTORMS" "MICROBURST"
## [65] "URBAN AND SMALL" "ICE"
## [67] "DOWNBURST" "GUSTNADO AND"
## [69] "WET MICROBURST" "DOWNBURST WINDS"
## [71] "DRY MICROBURST WINDS" "DRY MIRCOBURST WINDS"
## [73] "DRY MICROBURST 53" "MICROBURST WINDS"
## [75] "DRY MICROBURST 50" "BLOWING SNOW"
## [77] "FREEZING DRIZZLE" "DRY MICROBURST 58"
## [79] "DRY MICROBURST 84" "GLAZE"
## [81] "FIRST SNOW" "UNSEASONABLY DRY"
## [83] "UNSEASONABLY WET" "WINTRY MIX"
## [85] "UNSEASONABLY COLD" "EXTREME/RECORD COLD"
## [87] "UNSEASONABLY WARM" "Drought"
## [89] "NORMAL PRECIPITATION" "DRY"
## [91] "RAIN/SNOW" "STORM SURGE"
## [93] "Tropical Storm" "URBAN AND SMALL STREAM"
## [95] "DAMAGING FREEZE" "HURRICANE"
## [97] "WILD/FOREST FIRE" "MUD SLIDE"
## [99] "FROST" "FREEZING RAIN/SNOW"
## [101] "THUNDERSNOW" "COOL AND WET"
## [103] "SEVERE COLD" "GLAZE ICE"
## [105] "COLD WAVE" "EARLY SNOW"
## [107] "SMALL STREAM AND" "MUD SLIDES"
## [109] "COLD AND WET CONDITIONS" "EXCESSIVE WETNESS"
## [111] "GRADIENT WINDS" "ROTATING WALL CLOUD"
## [113] "LARGE WALL CLOUD" "COLD AIR FUNNEL"
## [115] "GUSTNADO" "COLD AIR FUNNELS"
## [117] "BLOWING SNOW- EXTREME WIND CHI" "FOG"
## [119] "ICE/SNOW" "TSTM WIND 51"
## [121] "TSTM WIND 50" "TSTM WIND 52"
## [123] "TSTM WIND 55" "THUNDERSTORM DAMAGE"
## [125] "SNOW AND ICE" "WIND STORM"
## [127] "SNOWSTORM" "GRASS FIRES"
## [129] "PROLONG COLD" "THUNDERSTORMW 50"
## [131] "AGRICULTURAL FREEZE" "THUNDERTSORM WIND"
## [133] "WARM DRY CONDITIONS" "HURRICANE-GENERATED SWELLS"
## [135] "COASTAL SURGE" "RAIN"
## [137] "SNOW/RAIN" "ICE FLOES"
## [139] "HIGH WAVES" "SNOW SQUALLS"
## [141] "SNOW SQUALL" "Lake-Effect Snow"
## [143] "THUNDERSTROM WINDS" "BLOWING SNOW & EXTREME WIND CH"
## [145] "URBAN/SMALL STREAM" "TORNDAO"
## [147] "ICE JAM" "FOREST FIRES"
## [149] "Frost/Freeze" "HVY RAIN"
## [151] "HARD FREEZE" "WET SNOW"
## [153] "RAIN AND WIND" "BELOW NORMAL PRECIPITATION"
## [155] "EXCESSIVE RAIN" "LIGHT SNOW"
## [157] "RECORD TEMPERATURES" "FOG AND COLD TEMPERATURES"
## [159] "OTHER" "RECORD SNOW"
## [161] "SNOW/COLD" "TSTM WIND G58"
## [163] "MUDSLIDE" "ICY ROADS"
## [165] "SNOW FREEZING RAIN" "LACK OF SNOW"
## [167] "SNOW/FREEZING RAIN" "TORRENTIAL RAIN"
## [169] "HURRICANE EMILY" "HURRICANE GORDON"
## [171] "HURRICANE FELIX" "THUNDERSTORM DAMAGE TO"
## [173] "ICE AND SNOW" "RECORD COLD/FROST"
## [175] "STORM FORCE WINDS" "FREEZING RAIN AND SNOW"
## [177] "FREEZING DRIZZLE AND FREEZING" "TSTM WIND DAMAGE"
## [179] "HIGH WATER" "RAIN/WIND"
## [181] "SNOW SHOWERS" "RECORD SNOW/COLD"
## [183] "UNSEASONABLY WARM AND DRY" "RECORD/EXCESSIVE RAINFALL"
## [185] "BEACH EROSIN" "LOW TEMPERATURE"
## [187] "HYPOTHERMIA" "MUD/ROCK SLIDE"
## [189] "COLD/WINDS" "SNOW/ BITTER COLD"
## [191] "RAPIDLY RISING WATER" "EARLY FREEZE"
## [193] "EARLY FROST" "SNOW ACCUMULATION"
## [195] "SNOW/ ICE" "SNOW/BLOWING SNOW"
## [197] "EXCESSIVE RAINFALL" "THUNDERSTORMW"
## [199] "TSTM WINDS" "TSTMW"
## [201] "FUNNELS" "TSTM WIND 65)"
## [203] "NEAR RECORD SNOW" "LANDSLIDE"
## [205] "EXCESSIVE" "WILD/FOREST FIRES"
## [207] "?" "THUNDERSTROM WIND"
## [209] "SNOW AND COLD" "HOT PATTERN"
## [211] "PROLONG COLD/SNOW" "BRUSH FIRES"
## [213] "SNOW\\COLD" "WINTER MIX"
## [215] "EXCESSIVE PRECIPITATION" "SNOWFALL RECORD"
## [217] "HOT/DRY PATTERN" "DRY PATTERN"
## [219] "MILD/DRY PATTERN" "MILD PATTERN"
## [221] "LANDSLIDES" "BRUSH FIRE"
## [223] "SAHARAN DUST" "URBAN SMALL"
## [225] "SMALL STREAM" "URBAN/SML STREAM FLD"
## [227] "Other" "Record dry month"
## [229] "Temperature record" "Tstm Wind"
## [231] "ROUGH SURF" "Wind"
## [233] "Wind Damage" "Marine Accident"
## [235] "Snow" "Freeze"
## [237] "Snow Squalls" "COASTAL STORM"
## [239] "Light Snow/Flurries" "Wet Month"
## [241] "Wet Year" "Damaging Freeze"
## [243] "Beach Erosion" "Hot and Dry"
## [245] "Icy Roads" "Rain Damage"
## [247] "Unseasonable Cold" "Early Frost"
## [249] "Wintry Mix" "blowing snow"
## [251] "Record Cold" "Extreme Cold"
## [253] "Ice Fog" "Excessive Cold"
## [255] "Torrential Rainfall" "Freezing Rain"
## [257] "Landslump" "Late-season Snowfall"
## [259] "Hurricane Edouard" "Coastal Storm"
## [261] "Snow squalls" "RECORD WARM TEMPS."
## [263] "Ice/Snow" "Mudslide"
## [265] "Glaze" "Extended Cold"
## [267] "Snow Accumulation" "Freezing Fog"
## [269] "Drifting Snow" "Whirlwind"
## [271] "LATE SNOW" "Record May Snow"
## [273] "Record Winter Snow" "Record temperature"
## [275] "Light snow" "Late Season Snowfall"
## [277] "Gusty Wind" "Light Snow"
## [279] "MIXED PRECIP" "Black Ice"
## [281] "Mudslides" "Gradient wind"
## [283] "Snow and Ice" "Freezing Spray"
## [285] "Summary Jan 17" "Summary of March 14"
## [287] "Summary of March 23" "Summary of March 24"
## [289] "Summary of April 3rd" "Summary of April 12"
## [291] "Summary of April 13" "Summary of April 21"
## [293] "Summary August 11" "Summary of April 27"
## [295] "Summary of May 9-10" "Summary of May 10"
## [297] "Summary of May 13" "Summary of May 14"
## [299] "Summary of May 22 am" "Summary of May 22 pm"
## [301] "Summary of May 26 am" "Summary of May 26 pm"
## [303] "Metro Storm, May 26" "Summary of May 31 am"
## [305] "Summary of May 31 pm" "Summary of June 3"
## [307] "Summary of June 4" "Summary June 5-6"
## [309] "Summary June 6" "Summary of June 11"
## [311] "Summary of June 12" "Summary of June 13"
## [313] "Summary of June 15" "Summary of June 16"
## [315] "Summary June 18-19" "Summary of June 23"
## [317] "Summary of June 24" "Summary of June 30"
## [319] "Summary of July 2" "Summary of July 3"
## [321] "Summary of July 11" "Summary of July 22"
## [323] "Summary July 23-24" "Summary of July 26"
## [325] "Summary of July 29" "Summary of August 1"
## [327] "Summary August 2-3" "Summary August 7"
## [329] "Summary August 9" "Summary August 10"
## [331] "Summary August 17" "Summary August 21"
## [333] "Summary August 28" "Summary September 4"
## [335] "Summary September 20" "Summary September 23"
## [337] "Summary Sept. 25-26" "Summary: Oct. 20-21"
## [339] "Summary: October 31" "Summary: Nov. 6-7"
## [341] "Summary: Nov. 16" "Microburst"
## [343] "wet micoburst" "Cold"
## [345] "Summary of May 22" "Summary of June 6"
## [347] "Summary August 4" "Summary of June 10"
## [349] "Summary of June 18" "Summary September 3"
## [351] "Summary: Sept. 18" "Record Temperatures"
## [353] "Light Snowfall" "Freezing Drizzle"
## [355] "Gusty wind/rain" "GUSTY WIND/HVY RAIN"
## [357] "Blowing Snow" "Early snowfall"
## [359] "Monthly Snowfall" "Seasonal Snowfall"
## [361] "Monthly Rainfall" "Cold Temperature"
## [363] "Sml Stream Fld" "MUDSLIDE/LANDSLIDE"
## [365] "Saharan Dust" "Volcanic Ash"
## [367] "Thundersnow shower" "NONE"
## [369] "COLD AND SNOW" "DAM BREAK"
## [371] "TSTM WIND (G45)" "BLACK ICE"
## [373] "BLOW-OUT TIDES" "UNSEASONABLY COOL"
## [375] "Gusty Winds" "GUSTY WIND"
## [377] "TSTM WIND 40" "TSTM WIND 45"
## [379] "TSTM WIND (41)" "TSTM WIND (G40)"
## [381] "TSTM WND" "Wintry mix"
## [383] " TSTM WIND" "Frost"
## [385] "Record Warmth" "Prolong Cold"
## [387] "Cold and Frost" "URBAN/SML STREAM FLDG"
## [389] "LATE FREEZE" "BLOW-OUT TIDE"
## [391] "Hypothermia/Exposure" "HYPOTHERMIA/EXPOSURE"
## [393] "Mixed Precipitation" "Record High"
## [395] "COASTALSTORM" "Freezing rain"
## [397] "Gusty winds" "SUMMARY OF MARCH 24-25"
## [399] "SUMMARY OF MARCH 27" "SUMMARY OF MARCH 29"
## [401] "GRADIENT WIND" "gradient wind"
## [403] "Freezing drizzle" "Mountain Snows"
## [405] "URBAN/SMALL STRM FLDG" "Mild and Dry Pattern"
## [407] "COLD AND FROST" "TYPHOON"
## [409] "HIGH SWELLS" "HIGH SWELLS"
## [411] "DRY SPELL" "BEACH EROSION"
## [413] "UNSEASONAL RAIN" "EARLY RAIN"
## [415] "PROLONGED RAIN" "WINTERY MIX"
## [417] "HOT SPELL" "UNSEASONABLY HOT"
## [419] " TSTM WIND (G45)" "TSTM WIND (G45)"
## [421] "TSTM WIND (G35)" "ABNORMAL WARMTH"
## [423] "UNUSUAL WARMTH" "WAKE LOW WIND"
## [425] "MONTHLY RAINFALL" "COLD TEMPERATURES"
## [427] "MODERATE SNOW" "MODERATE SNOWFALL"
## [429] "COASTAL EROSION" "UNUSUAL/RECORD WARMTH"
## [431] "Seiche" "TSTM"
## [433] "UNSEASONABLY WARM YEAR" "HYPERTHERMIA/EXPOSURE"
## [435] "ROCK SLIDE" "ICE PELLETS"
## [437] "RECORD COOL" "RECORD WARM"
## [439] "RECORD TEMPERATURE" "Tropical Depression"
## [441] "VOLCANIC ERUPTION" "COOL SPELL"
## [443] "WIND ADVISORY" "RED FLAG FIRE WX"
## [445] "FIRST FROST" "EXCESSIVELY DRY"
## [447] "LIGHT SNOW/FREEZING PRECIP" "VOG"
## [449] "MONTHLY PRECIPITATION" "MONTHLY TEMPERATURE"
## [451] "RECORD DRYNESS" "MIXED PRECIPITATION"
## [453] "DRY CONDITIONS" "EARLY SNOWFALL"
## [455] "LANDSPOUT" "DRIEST MONTH"
## [457] "RECORD COLD" "EXCESSIVE SNOW"
## [459] "DRYNESS" "WIND AND WAVE"
## [461] "LIGHT FREEZING RAIN" " WIND"
## [463] "MONTHLY SNOWFALL" "RECORD PRECIPITATION"
## [465] "ICE ROADS" "ROUGH SEAS"
## [467] "UNSEASONABLY WARM/WET" "UNSEASONABLY COOL & WET"
## [469] "UNUSUALLY WARM" "TSTM WIND G45"
## [471] "NON-SEVERE WIND DAMAGE" "UNUSUALLY COLD"
## [473] "LANDSLUMP" "UNSEASONABLY WARM & WET"
## [475] "WIND GUSTS" "UNSEASONAL LOW TEMP"
## [477] "LATE SEASON SNOW" "GUSTY LAKE WIND"
## [479] "ABNORMALLY DRY" "RED FLAG CRITERIA"
## [481] "WND" "SMOKE"
## [483] "SNOW ADVISORY" "EXTREMELY WET"
## [485] "UNUSUALLY LATE SNOW" "VERY DRY"
## [487] "RECORD LOW RAINFALL" "ROGUE WAVE"
## [489] "PROLONG WARMTH" "ACCUMULATED SNOWFALL"
## [491] "FALLING SNOW/ICE" "NON-TSTM WIND"
## [493] "NON TSTM WIND" "PATCHY ICE"
## [495] "NORTHERN LIGHTS" "MARINE TSTM WIND"
## [497] "HAZARDOUS SURF" "ASTRONOMICAL HIGH TIDE"
## [499] "WHIRLWIND" "VERY WARM"
## [501] "ABNORMALLY WET" "ICE ON ROAD"
## [503] "DROWNING" "HURRICANE/TYPHOON"
## [505] "Storm Surge/Tide" "Tsunami"
## [507] "Dense Smoke" "Astronomical Low Tide"It’s noticeable that there are many event types named “Summary …”, which doesn’t specify any actual storm event. It needs to be removed.
storm_data<-partial_matching(event_names = "Summary", rename="remove")There are many situations where storm events can be renamed. For example, “wintry mix” is just another word for “Sleet”, “Ice roads” can be changed to “Frost/Freeze”, etc.
storm_data<-partial_matching(event_names = "Wintry mix", rename="Sleet")
storm_data<-partial_matching(event_names = "Funnel", rename = "Tornado")
frost<-c("Glaze", "Ice roads", "Frost", "Freeze", "ICE ON ROAD")
storm_data<-partial_matching(event_names = frost,max_dist = 0, rename = "Frost/Freeze")
storm_data<-partial_matching(event_names = "Snow", max_dist = 0, rename = "Heavy Snow")
storm_data<-partial_matching(event_names = "Rogue wave", rename = "Tsunami")
Wind<-c("Wind gusts", "wind", "wind damage")
storm_data<-partial_matching(event_names = Wind, rename = "High Wind")
extreme_cold<-c("Extreme cold", "RECORD COLD", "SEVERE COLD", "Extreme Cold/Wind Chill")
storm_data<-partial_matching(event_names = extreme_cold, max_dist=0,rename = "Extreme Cold/Wind Chill")
storm_data<-partial_matching(event_names = "Cold", max_dist = 0, rename = "Cold/Wind Chill")All those left event types doesn’t fall in a specific event type from Storm Data Documentation. It’s not so many of them, therefore they will be removed:
storm_data<-storm_data %>% filter(EVTYPE==event)1. Across the United States, which types of events (as indicated in the EVTYPE variable) are most harmful with respect to population health?
health<-storm_data %>%
group_by(EVTYPE) %>%
summarize(sum = sum(HEALTH_DMG))
summary(health$sum)## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 0.00 0.00 3.00 55.69 24.00 508.00We will remove those event types that cause lower than mean damage to population health:
health<- health %>% filter(sum>mean(sum))
par(cex.axis=0.8, cex.main=1.2, oma = c(3, 0, 0, 0))
barplot(health$sum,
names=health$EVTYPE,
las=2,
cex.lab=0.8,
col="khaki",
main="Most harmful storm events with respect to population health",
ylab="fatalities + injuries"
)
2. Across the United States, which types of events have the greatest economic consequences?
Let’s make a new dataset containing only event types and economic damage caused by them.
economic<-storm_data %>%
group_by(EVTYPE) %>%
summarize(sum = sum(ECONOMIC_DMG))
summary(economic$sum)## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 0.00 10.25 177.75 7855.76 1177.66 69567.17We will remove those event types that cause lower than mean economic damage
economic<- economic %>% filter(sum>mean(sum))
par(cex.axis=0.8, cex.main=1.2, oma = c(3, 0, 0, 0))
barplot(economic$sum,
names=economic$EVTYPE,
las=2,
cex.lab=0.8,
col="khaki",
main="Storm events that have the greatest economic consequences",
ylab="Proper + crop demage"
)
Results
As seen in the plots above, events that cause biggest economic and public health problems are: tornadoes, heat, high wind, floods, etc.