This report and the relevant data is available at https://github.com/mlnotes2718/RepResearchFinalProject
The following is an analysis of various kind of storm is the United States since 1950s. This analysis focus on human casualties and financial impact causes by various kind of storm. We need to answer the following questions:
Across the United States, which types of events are most harmful with respect to population health?
Across the United States, which types of events have the greatest economic consequences?
Based on the results, we concluded that the storm type that caused the most human casualties is Tornado and it is different from the storm type that has the most financial damage. The number one storm that caused the most financial damage is Flood. Hence policy maker should consider different type of response when addressing different type of storm.
# Import Library
library(dplyr)
library(stringr)
library(ggplot2)
library(reshape2)
library(kableExtra)
Data Source: U.S. National Oceanic and Atmospheric Administration (NOAA) - Storm Data
URL: https://d396qusza40orc.cloudfront.net/repdata%2Fdata%2FStormData.csv.bz2
Documentation of Data Source: National Weather Service Storm Data Documentation
fileURL <- "https://d396qusza40orc.cloudfront.net/repdata%2Fdata%2FStormData.csv.bz2"
if(!file.exists("./StormData.csv.bz2")) {
download.file(fileURL, destfile = "./StormData.csv.bz2")
}
| Variable Name | Description |
|---|---|
| fileURL | Contains the URL of the raw file StormData.csv.bz2 |
| stormData | The raw data that is read from StormData.csv.bz2 |
| reducedStormData | Raw dataset with only only 7 columns and only contains data with human casualties or economic damage. |
| stormTypeNA | Check for the column EVTYPE if there is any NA. |
| casualtiesNA | Check for 2 columns, FATALITIES and INJURIES if there is any NA |
| damageNA | Check for 2 columns, PROPDMG and CROPDMG if there is any NA |
| evtypeLst | List that contains the unique event type of the dataset |
| removeEvtypeCondition | Filter out EVTYPE that has ? |
| uniquePROPDMGEXP | Filter out a list of all unique exponent in PROPDMGEXP |
| uniqueCROPDMGEXP | Filter out a list of all unique exponent in CROPDMGEXP |
| uniqueBothMGEXP | Combined the 2 unique set of PROPDMGEXP and CROPDMGEXP |
| replacement | Using the unique list from uniqueBothMGEXP, we establish the replacement values |
| cleanStormData | cleaned dataset that have EVTYPE fixed. |
| fatalDf | Data frame that focus on fatalities and injuries, grouped by storm type. |
| rankedCasualties | Data frame that contains only the top 10 storm that has the most fatalities. We use fatalities as filter option. |
| plotDataCasualties | Converted data frame for plotting chart on casualties |
| meltedPlotDataCasualties | Melted data for plotting bar chart on casualties |
| rankedEconomicImpact | Data frame that contains only the top 10 storm that has the most financial damage in billions. |
| plotDataEcon | Converted data frame for plotting chart on economic damage |
| meltedPlotDataEcon | Melted data for plotting bar chart on economic damage |
stormData <- read.csv("StormData.csv.bz2")
File Structure
str(stormData)
## 'data.frame': 902297 obs. of 37 variables:
## $ STATE__ : num 1 1 1 1 1 1 1 1 1 1 ...
## $ BGN_DATE : chr "4/18/1950 0:00:00" "4/18/1950 0:00:00" "2/20/1951 0:00:00" "6/8/1951 0:00:00" ...
## $ BGN_TIME : chr "0130" "0145" "1600" "0900" ...
## $ TIME_ZONE : chr "CST" "CST" "CST" "CST" ...
## $ COUNTY : num 97 3 57 89 43 77 9 123 125 57 ...
## $ COUNTYNAME: chr "MOBILE" "BALDWIN" "FAYETTE" "MADISON" ...
## $ STATE : chr "AL" "AL" "AL" "AL" ...
## $ EVTYPE : chr "TORNADO" "TORNADO" "TORNADO" "TORNADO" ...
## $ BGN_RANGE : num 0 0 0 0 0 0 0 0 0 0 ...
## $ BGN_AZI : chr "" "" "" "" ...
## $ BGN_LOCATI: chr "" "" "" "" ...
## $ END_DATE : chr "" "" "" "" ...
## $ END_TIME : chr "" "" "" "" ...
## $ COUNTY_END: num 0 0 0 0 0 0 0 0 0 0 ...
## $ COUNTYENDN: logi NA NA NA NA NA NA ...
## $ END_RANGE : num 0 0 0 0 0 0 0 0 0 0 ...
## $ END_AZI : chr "" "" "" "" ...
## $ END_LOCATI: chr "" "" "" "" ...
## $ LENGTH : num 14 2 0.1 0 0 1.5 1.5 0 3.3 2.3 ...
## $ WIDTH : num 100 150 123 100 150 177 33 33 100 100 ...
## $ F : int 3 2 2 2 2 2 2 1 3 3 ...
## $ MAG : num 0 0 0 0 0 0 0 0 0 0 ...
## $ FATALITIES: num 0 0 0 0 0 0 0 0 1 0 ...
## $ INJURIES : num 15 0 2 2 2 6 1 0 14 0 ...
## $ PROPDMG : num 25 2.5 25 2.5 2.5 2.5 2.5 2.5 25 25 ...
## $ PROPDMGEXP: chr "K" "K" "K" "K" ...
## $ CROPDMG : num 0 0 0 0 0 0 0 0 0 0 ...
## $ CROPDMGEXP: chr "" "" "" "" ...
## $ WFO : chr "" "" "" "" ...
## $ STATEOFFIC: chr "" "" "" "" ...
## $ ZONENAMES : chr "" "" "" "" ...
## $ LATITUDE : num 3040 3042 3340 3458 3412 ...
## $ LONGITUDE : num 8812 8755 8742 8626 8642 ...
## $ LATITUDE_E: num 3051 0 0 0 0 ...
## $ LONGITUDE_: num 8806 0 0 0 0 ...
## $ REMARKS : chr "" "" "" "" ...
## $ REFNUM : num 1 2 3 4 5 6 7 8 9 10 ...
Data contains 902297 with 37 columns.
reducedStormData <- filter(stormData, FATALITIES != 0 | INJURIES !=0 |PROPDMG !=0 | CROPDMG != 0)
reducedStormData <- select(reducedStormData, STATE, BGN_DATE, EVTYPE, FATALITIES,INJURIES,PROPDMG,PROPDMGEXP,CROPDMG,CROPDMGEXP,REMARKS)
Check for NA
stormTypeNA <- is.na(reducedStormData$EVTYPE)
filter(reducedStormData, stormTypeNA)
## [1] STATE BGN_DATE EVTYPE FATALITIES INJURIES PROPDMG
## [7] PROPDMGEXP CROPDMG CROPDMGEXP REMARKS
## <0 rows> (or 0-length row.names)
No NA for storm type
casualtiesNA <- is.na(reducedStormData$FATALITIES | reducedStormData$INJURIES)
filter(reducedStormData, casualtiesNA)
## [1] STATE BGN_DATE EVTYPE FATALITIES INJURIES PROPDMG
## [7] PROPDMGEXP CROPDMG CROPDMGEXP REMARKS
## <0 rows> (or 0-length row.names)
No NA for fatalities or injuries.
damageNA <- is.na(reducedStormData$PROPDMG | reducedStormData$CROPDMG)
filter(reducedStormData, damageNA)
## [1] STATE BGN_DATE EVTYPE FATALITIES INJURIES PROPDMG
## [7] PROPDMGEXP CROPDMG CROPDMGEXP REMARKS
## <0 rows> (or 0-length row.names)
No NA for property or crop damage.
str(reducedStormData)
## 'data.frame': 254633 obs. of 10 variables:
## $ STATE : chr "AL" "AL" "AL" "AL" ...
## $ BGN_DATE : chr "4/18/1950 0:00:00" "4/18/1950 0:00:00" "2/20/1951 0:00:00" "6/8/1951 0:00:00" ...
## $ EVTYPE : chr "TORNADO" "TORNADO" "TORNADO" "TORNADO" ...
## $ FATALITIES: num 0 0 0 0 0 0 0 0 1 0 ...
## $ INJURIES : num 15 0 2 2 2 6 1 0 14 0 ...
## $ PROPDMG : num 25 2.5 25 2.5 2.5 2.5 2.5 2.5 25 25 ...
## $ PROPDMGEXP: chr "K" "K" "K" "K" ...
## $ CROPDMG : num 0 0 0 0 0 0 0 0 0 0 ...
## $ CROPDMGEXP: chr "" "" "" "" ...
## $ REMARKS : chr "" "" "" "" ...
We have 254633 rows with 10 columns.
unique(reducedStormData$EVTYPE)
## [1] "TORNADO" "TSTM WIND"
## [3] "HAIL" "ICE STORM/FLASH FLOOD"
## [5] "WINTER STORM" "HURRICANE OPAL/HIGH WINDS"
## [7] "THUNDERSTORM WINDS" "HURRICANE ERIN"
## [9] "HURRICANE OPAL" "HEAVY RAIN"
## [11] "LIGHTNING" "THUNDERSTORM WIND"
## [13] "DENSE FOG" "RIP CURRENT"
## [15] "THUNDERSTORM WINS" "FLASH FLOODING"
## [17] "FLASH FLOOD" "TORNADO F0"
## [19] "THUNDERSTORM WINDS LIGHTNING" "THUNDERSTORM WINDS/HAIL"
## [21] "HEAT" "HIGH WINDS"
## [23] "WIND" "HEAVY RAINS"
## [25] "LIGHTNING AND HEAVY RAIN" "THUNDERSTORM WINDS HAIL"
## [27] "COLD" "HEAVY RAIN/LIGHTNING"
## [29] "FLASH FLOODING/THUNDERSTORM WI" "FLOODING"
## [31] "WATERSPOUT" "EXTREME COLD"
## [33] "LIGHTNING/HEAVY RAIN" "BREAKUP FLOODING"
## [35] "HIGH WIND" "FREEZE"
## [37] "RIVER FLOOD" "HIGH WINDS HEAVY RAINS"
## [39] "AVALANCHE" "MARINE MISHAP"
## [41] "HIGH TIDES" "HIGH WIND/SEAS"
## [43] "HIGH WINDS/HEAVY RAIN" "HIGH SEAS"
## [45] "COASTAL FLOOD" "SEVERE TURBULENCE"
## [47] "RECORD RAINFALL" "HEAVY SNOW"
## [49] "HEAVY SNOW/WIND" "DUST STORM"
## [51] "FLOOD" "APACHE COUNTY"
## [53] "SLEET" "DUST DEVIL"
## [55] "ICE STORM" "EXCESSIVE HEAT"
## [57] "THUNDERSTORM WINDS/FUNNEL CLOU" "GUSTY WINDS"
## [59] "FLOODING/HEAVY RAIN" "HEAVY SURF COASTAL FLOODING"
## [61] "HIGH SURF" "WILD FIRES"
## [63] "HIGH" "WINTER STORM HIGH WINDS"
## [65] "WINTER STORMS" "MUDSLIDES"
## [67] "RAINSTORM" "SEVERE THUNDERSTORM"
## [69] "SEVERE THUNDERSTORMS" "SEVERE THUNDERSTORM WINDS"
## [71] "THUNDERSTORMS WINDS" "FLOOD/FLASH FLOOD"
## [73] "FLOOD/RAIN/WINDS" "THUNDERSTORMS"
## [75] "FLASH FLOOD WINDS" "WINDS"
## [77] "FUNNEL CLOUD" "HIGH WIND DAMAGE"
## [79] "STRONG WIND" "HEAVY SNOWPACK"
## [81] "FLASH FLOOD/" "HEAVY SURF"
## [83] "DRY MIRCOBURST WINDS" "DRY MICROBURST"
## [85] "URBAN FLOOD" "THUNDERSTORM WINDSS"
## [87] "MICROBURST WINDS" "HEAT WAVE"
## [89] "UNSEASONABLY WARM" "COASTAL FLOODING"
## [91] "STRONG WINDS" "BLIZZARD"
## [93] "WATERSPOUT/TORNADO" "WATERSPOUT TORNADO"
## [95] "STORM SURGE" "URBAN/SMALL STREAM FLOOD"
## [97] "WATERSPOUT-" "TORNADOES, TSTM WIND, HAIL"
## [99] "TROPICAL STORM ALBERTO" "TROPICAL STORM"
## [101] "TROPICAL STORM GORDON" "TROPICAL STORM JERRY"
## [103] "LIGHTNING THUNDERSTORM WINDS" "URBAN FLOODING"
## [105] "MINOR FLOODING" "WATERSPOUT-TORNADO"
## [107] "LIGHTNING INJURY" "LIGHTNING AND THUNDERSTORM WIN"
## [109] "FLASH FLOODS" "THUNDERSTORM WINDS53"
## [111] "WILDFIRE" "DAMAGING FREEZE"
## [113] "THUNDERSTORM WINDS 13" "HURRICANE"
## [115] "SNOW" "LIGNTNING"
## [117] "FROST" "FREEZING RAIN/SNOW"
## [119] "HIGH WINDS/" "THUNDERSNOW"
## [121] "FLOODS" "COOL AND WET"
## [123] "HEAVY RAIN/SNOW" "GLAZE ICE"
## [125] "MUD SLIDE" "HIGH WINDS"
## [127] "RURAL FLOOD" "MUD SLIDES"
## [129] "EXTREME HEAT" "DROUGHT"
## [131] "COLD AND WET CONDITIONS" "EXCESSIVE WETNESS"
## [133] "SLEET/ICE STORM" "GUSTNADO"
## [135] "FREEZING RAIN" "SNOW AND HEAVY SNOW"
## [137] "GROUND BLIZZARD" "EXTREME WIND CHILL"
## [139] "MAJOR FLOOD" "SNOW/HEAVY SNOW"
## [141] "FREEZING RAIN/SLEET" "ICE JAM FLOODING"
## [143] "COLD AIR TORNADO" "WIND DAMAGE"
## [145] "FOG" "TSTM WIND 55"
## [147] "SMALL STREAM FLOOD" "THUNDERTORM WINDS"
## [149] "HAIL/WINDS" "SNOW AND ICE"
## [151] "WIND STORM" "GRASS FIRES"
## [153] "LAKE FLOOD" "HAIL/WIND"
## [155] "WIND/HAIL" "ICE"
## [157] "SNOW AND ICE STORM" "THUNDERSTORM WINDS"
## [159] "WINTER WEATHER" "DROUGHT/EXCESSIVE HEAT"
## [161] "THUNDERSTORMS WIND" "TUNDERSTORM WIND"
## [163] "URBAN AND SMALL STREAM FLOODIN" "THUNDERSTORM WIND/LIGHTNING"
## [165] "HEAVY RAIN/SEVERE WEATHER" "THUNDERSTORM"
## [167] "WATERSPOUT/ TORNADO" "LIGHTNING."
## [169] "HURRICANE-GENERATED SWELLS" "RIVER AND STREAM FLOOD"
## [171] "HIGH WINDS/COASTAL FLOOD" "RAIN"
## [173] "RIVER FLOODING" "ICE FLOES"
## [175] "THUNDERSTORM WIND G50" "LIGHTNING FIRE"
## [177] "HEAVY LAKE SNOW" "RECORD COLD"
## [179] "HEAVY SNOW/FREEZING RAIN" "COLD WAVE"
## [181] "DUST DEVIL WATERSPOUT" "TORNADO F3"
## [183] "TORNDAO" "FLOOD/RIVER FLOOD"
## [185] "MUD SLIDES URBAN FLOODING" "TORNADO F1"
## [187] "GLAZE/ICE STORM" "GLAZE"
## [189] "HEAVY SNOW/WINTER STORM" "MICROBURST"
## [191] "AVALANCE" "BLIZZARD/WINTER STORM"
## [193] "DUST STORM/HIGH WINDS" "ICE JAM"
## [195] "FOREST FIRES" "FROST\\FREEZE"
## [197] "THUNDERSTORM WINDS." "HVY RAIN"
## [199] "HAIL 150" "HAIL 075"
## [201] "HAIL 100" "THUNDERSTORM WIND G55"
## [203] "HAIL 125" "THUNDERSTORM WIND G60"
## [205] "THUNDERSTORM WINDS G60" "HARD FREEZE"
## [207] "HAIL 200" "HEAVY SNOW AND HIGH WINDS"
## [209] "HEAVY SNOW/HIGH WINDS & FLOOD" "HEAVY RAIN AND FLOOD"
## [211] "RIP CURRENTS/HEAVY SURF" "URBAN AND SMALL"
## [213] "WILDFIRES" "FOG AND COLD TEMPERATURES"
## [215] "SNOW/COLD" "FLASH FLOOD FROM ICE JAMS"
## [217] "TSTM WIND G58" "MUDSLIDE"
## [219] "HEAVY SNOW SQUALLS" "SNOW SQUALL"
## [221] "SNOW/ICE STORM" "HEAVY SNOW/SQUALLS"
## [223] "HEAVY SNOW-SQUALLS" "ICY ROADS"
## [225] "HEAVY MIX" "SNOW FREEZING RAIN"
## [227] "SNOW/SLEET" "SNOW/FREEZING RAIN"
## [229] "SNOW SQUALLS" "SNOW/SLEET/FREEZING RAIN"
## [231] "RECORD SNOW" "HAIL 0.75"
## [233] "RECORD HEAT" "THUNDERSTORM WIND 65MPH"
## [235] "THUNDERSTORM WIND/ TREES" "THUNDERSTORM WIND/AWNING"
## [237] "THUNDERSTORM WIND 98 MPH" "THUNDERSTORM WIND TREES"
## [239] "TORNADO F2" "RIP CURRENTS"
## [241] "HURRICANE EMILY" "COASTAL SURGE"
## [243] "HURRICANE GORDON" "HURRICANE FELIX"
## [245] "THUNDERSTORM WIND 60 MPH" "THUNDERSTORM WINDS 63 MPH"
## [247] "THUNDERSTORM WIND/ TREE" "THUNDERSTORM DAMAGE TO"
## [249] "THUNDERSTORM WIND 65 MPH" "FLASH FLOOD - HEAVY RAIN"
## [251] "THUNDERSTORM WIND." "FLASH FLOOD/ STREET"
## [253] "BLOWING SNOW" "HEAVY SNOW/BLIZZARD"
## [255] "THUNDERSTORM HAIL" "THUNDERSTORM WINDSHAIL"
## [257] "LIGHTNING WAUSEON" "THUDERSTORM WINDS"
## [259] "ICE AND SNOW" "STORM FORCE WINDS"
## [261] "HEAVY SNOW/ICE" "LIGHTING"
## [263] "HIGH WIND/HEAVY SNOW" "THUNDERSTORM WINDS AND"
## [265] "HEAVY PRECIPITATION" "HIGH WIND/BLIZZARD"
## [267] "TSTM WIND DAMAGE" "FLOOD FLASH"
## [269] "RAIN/WIND" "SNOW/ICE"
## [271] "HAIL 75" "HEAT WAVE DROUGHT"
## [273] "HEAVY SNOW/BLIZZARD/AVALANCHE" "HEAT WAVES"
## [275] "UNSEASONABLY WARM AND DRY" "UNSEASONABLY COLD"
## [277] "RECORD/EXCESSIVE HEAT" "THUNDERSTORM WIND G52"
## [279] "HIGH WAVES" "FLASH FLOOD/FLOOD"
## [281] "FLOOD/FLASH" "LOW TEMPERATURE"
## [283] "HEAVY RAINS/FLOODING" "THUNDERESTORM WINDS"
## [285] "THUNDERSTORM WINDS/FLOODING" "HYPOTHERMIA"
## [287] "THUNDEERSTORM WINDS" "THUNERSTORM WINDS"
## [289] "HIGH WINDS/COLD" "COLD/WINDS"
## [291] "SNOW/ BITTER COLD" "COLD WEATHER"
## [293] "RAPIDLY RISING WATER" "WILD/FOREST FIRE"
## [295] "ICE/STRONG WINDS" "SNOW/HIGH WINDS"
## [297] "HIGH WINDS/SNOW" "SNOWMELT FLOODING"
## [299] "HEAVY SNOW AND STRONG WINDS" "SNOW ACCUMULATION"
## [301] "SNOW/ ICE" "SNOW/BLOWING SNOW"
## [303] "TORNADOES" "THUNDERSTORM WIND/HAIL"
## [305] "FREEZING DRIZZLE" "HAIL 175"
## [307] "FLASH FLOODING/FLOOD" "HAIL 275"
## [309] "HAIL 450" "EXCESSIVE RAINFALL"
## [311] "THUNDERSTORMW" "HAILSTORM"
## [313] "TSTM WINDS" "TSTMW"
## [315] "TSTM WIND 65)" "TROPICAL STORM DEAN"
## [317] "THUNDERSTORM WINDS/ FLOOD" "LANDSLIDE"
## [319] "HIGH WIND AND SEAS" "THUNDERSTORMWINDS"
## [321] "WILD/FOREST FIRES" "HEAVY SEAS"
## [323] "HAIL DAMAGE" "FLOOD & HEAVY RAIN"
## [325] "?" "THUNDERSTROM WIND"
## [327] "FLOOD/FLASHFLOOD" "HIGH WATER"
## [329] "HIGH WIND 48" "LANDSLIDES"
## [331] "URBAN/SMALL STREAM" "BRUSH FIRE"
## [333] "HEAVY SHOWER" "HEAVY SWELLS"
## [335] "URBAN SMALL" "URBAN FLOODS"
## [337] "FLASH FLOOD/LANDSLIDE" "HEAVY RAIN/SMALL STREAM URBAN"
## [339] "FLASH FLOOD LANDSLIDES" "TSTM WIND/HAIL"
## [341] "Other" "Ice jam flood (minor"
## [343] "Tstm Wind" "URBAN/SML STREAM FLD"
## [345] "ROUGH SURF" "Heavy Surf"
## [347] "Dust Devil" "Marine Accident"
## [349] "Freeze" "Strong Wind"
## [351] "COASTAL STORM" "Erosion/Cstl Flood"
## [353] "River Flooding" "Damaging Freeze"
## [355] "Beach Erosion" "High Surf"
## [357] "Heavy Rain/High Surf" "Unseasonable Cold"
## [359] "Early Frost" "Wintry Mix"
## [361] "Extreme Cold" "Coastal Flooding"
## [363] "Torrential Rainfall" "Landslump"
## [365] "Hurricane Edouard" "Coastal Storm"
## [367] "TIDAL FLOODING" "Tidal Flooding"
## [369] "Strong Winds" "EXTREME WINDCHILL"
## [371] "Glaze" "Extended Cold"
## [373] "Whirlwind" "Heavy snow shower"
## [375] "Light snow" "Light Snow"
## [377] "MIXED PRECIP" "Freezing Spray"
## [379] "DOWNBURST" "Mudslides"
## [381] "Microburst" "Mudslide"
## [383] "Cold" "Coastal Flood"
## [385] "Snow Squalls" "Wind Damage"
## [387] "Light Snowfall" "Freezing Drizzle"
## [389] "Gusty wind/rain" "GUSTY WIND/HVY RAIN"
## [391] "Wind" "Cold Temperature"
## [393] "Heat Wave" "Snow"
## [395] "COLD AND SNOW" "RAIN/SNOW"
## [397] "TSTM WIND (G45)" "Gusty Winds"
## [399] "GUSTY WIND" "TSTM WIND 40"
## [401] "TSTM WIND 45" "TSTM WIND (41)"
## [403] "TSTM WIND (G40)" "Frost/Freeze"
## [405] "AGRICULTURAL FREEZE" "OTHER"
## [407] "Hypothermia/Exposure" "HYPOTHERMIA/EXPOSURE"
## [409] "Lake Effect Snow" "Freezing Rain"
## [411] "Mixed Precipitation" "BLACK ICE"
## [413] "COASTALSTORM" "LIGHT SNOW"
## [415] "DAM BREAK" "Gusty winds"
## [417] "blowing snow" "GRADIENT WIND"
## [419] "TSTM WIND AND LIGHTNING" "gradient wind"
## [421] "Gradient wind" "Freezing drizzle"
## [423] "WET MICROBURST" "Heavy surf and wind"
## [425] "TYPHOON" "HIGH SWELLS"
## [427] "SMALL HAIL" "UNSEASONAL RAIN"
## [429] "COASTAL FLOODING/EROSION" " TSTM WIND (G45)"
## [431] "TSTM WIND (G45)" "HIGH WIND (G40)"
## [433] "TSTM WIND (G35)" "COASTAL EROSION"
## [435] "SEICHE" "COASTAL FLOODING/EROSION"
## [437] "HYPERTHERMIA/EXPOSURE" "WINTRY MIX"
## [439] "ROCK SLIDE" "GUSTY WIND/HAIL"
## [441] " TSTM WIND" "LANDSPOUT"
## [443] "EXCESSIVE SNOW" "LAKE EFFECT SNOW"
## [445] "FLOOD/FLASH/FLOOD" "MIXED PRECIPITATION"
## [447] "WIND AND WAVE" "LIGHT FREEZING RAIN"
## [449] "ICE ROADS" "ROUGH SEAS"
## [451] "TSTM WIND G45" "NON-SEVERE WIND DAMAGE"
## [453] "WARM WEATHER" "THUNDERSTORM WIND (G40)"
## [455] " FLASH FLOOD" "LATE SEASON SNOW"
## [457] "WINTER WEATHER MIX" "ROGUE WAVE"
## [459] "FALLING SNOW/ICE" "NON-TSTM WIND"
## [461] "NON TSTM WIND" "BLOWING DUST"
## [463] "VOLCANIC ASH" " HIGH SURF ADVISORY"
## [465] "HAZARDOUS SURF" "WHIRLWIND"
## [467] "ICE ON ROAD" "DROWNING"
## [469] "EXTREME COLD/WIND CHILL" "MARINE TSTM WIND"
## [471] "HURRICANE/TYPHOON" "WINTER WEATHER/MIX"
## [473] "FROST/FREEZE" "ASTRONOMICAL HIGH TIDE"
## [475] "HEAVY SURF/HIGH SURF" "TROPICAL DEPRESSION"
## [477] "LAKE-EFFECT SNOW" "MARINE HIGH WIND"
## [479] "TSUNAMI" "STORM SURGE/TIDE"
## [481] "COLD/WIND CHILL" "LAKESHORE FLOOD"
## [483] "MARINE THUNDERSTORM WIND" "MARINE STRONG WIND"
## [485] "ASTRONOMICAL LOW TIDE" "DENSE SMOKE"
## [487] "MARINE HAIL" "FREEZING FOG"
We need to clean up EVTYPE. Besides remove leading, trailing and double space, we also need to convert all storm type to upper case.
We also check the Property and Crop Damage Exponent
unique(reducedStormData$PROPDMGEXP)
## [1] "K" "M" "" "B" "m" "+" "0" "5" "6" "4" "h" "2" "7" "3" "H" "-"
unique(reducedStormData$CROPDMGEXP)
## [1] "" "M" "K" "m" "B" "?" "0" "k"
There is also mixed lower and upper case which we can fixed together. The following code is to clean up trailing and leading space and upper and lower case for EVTYPE, PROPDMGEXP and CROPDMGEXP
reducedStormData$EVTYPE <- trimws(reducedStormData$EVTYPE, which = c("both"), whitespace = "[ \t\r\n]")
reducedStormData$EVTYPE <- tolower(reducedStormData$EVTYPE)
reducedStormData$EVTYPE <- toupper(reducedStormData$EVTYPE)
reducedStormData$EVTYPE <- str_squish(reducedStormData$EVTYPE)
reducedStormData$PROPDMGEXP <- toupper(reducedStormData$PROPDMGEXP)
reducedStormData$CROPDMGEXP <- toupper(reducedStormData$CROPDMGEXP)
We also sort storm type:
evtypeLst <- sort(unique(reducedStormData$EVTYPE))
evtypeLst
## [1] "?" "AGRICULTURAL FREEZE"
## [3] "APACHE COUNTY" "ASTRONOMICAL HIGH TIDE"
## [5] "ASTRONOMICAL LOW TIDE" "AVALANCE"
## [7] "AVALANCHE" "BEACH EROSION"
## [9] "BLACK ICE" "BLIZZARD"
## [11] "BLIZZARD/WINTER STORM" "BLOWING DUST"
## [13] "BLOWING SNOW" "BREAKUP FLOODING"
## [15] "BRUSH FIRE" "COASTAL EROSION"
## [17] "COASTAL FLOOD" "COASTAL FLOODING"
## [19] "COASTAL FLOODING/EROSION" "COASTAL STORM"
## [21] "COASTAL SURGE" "COASTALSTORM"
## [23] "COLD" "COLD AIR TORNADO"
## [25] "COLD AND SNOW" "COLD AND WET CONDITIONS"
## [27] "COLD TEMPERATURE" "COLD WAVE"
## [29] "COLD WEATHER" "COLD/WIND CHILL"
## [31] "COLD/WINDS" "COOL AND WET"
## [33] "DAM BREAK" "DAMAGING FREEZE"
## [35] "DENSE FOG" "DENSE SMOKE"
## [37] "DOWNBURST" "DROUGHT"
## [39] "DROUGHT/EXCESSIVE HEAT" "DROWNING"
## [41] "DRY MICROBURST" "DRY MIRCOBURST WINDS"
## [43] "DUST DEVIL" "DUST DEVIL WATERSPOUT"
## [45] "DUST STORM" "DUST STORM/HIGH WINDS"
## [47] "EARLY FROST" "EROSION/CSTL FLOOD"
## [49] "EXCESSIVE HEAT" "EXCESSIVE RAINFALL"
## [51] "EXCESSIVE SNOW" "EXCESSIVE WETNESS"
## [53] "EXTENDED COLD" "EXTREME COLD"
## [55] "EXTREME COLD/WIND CHILL" "EXTREME HEAT"
## [57] "EXTREME WIND CHILL" "EXTREME WINDCHILL"
## [59] "FALLING SNOW/ICE" "FLASH FLOOD"
## [61] "FLASH FLOOD - HEAVY RAIN" "FLASH FLOOD FROM ICE JAMS"
## [63] "FLASH FLOOD LANDSLIDES" "FLASH FLOOD WINDS"
## [65] "FLASH FLOOD/" "FLASH FLOOD/ STREET"
## [67] "FLASH FLOOD/FLOOD" "FLASH FLOOD/LANDSLIDE"
## [69] "FLASH FLOODING" "FLASH FLOODING/FLOOD"
## [71] "FLASH FLOODING/THUNDERSTORM WI" "FLASH FLOODS"
## [73] "FLOOD" "FLOOD & HEAVY RAIN"
## [75] "FLOOD FLASH" "FLOOD/FLASH"
## [77] "FLOOD/FLASH FLOOD" "FLOOD/FLASH/FLOOD"
## [79] "FLOOD/FLASHFLOOD" "FLOOD/RAIN/WINDS"
## [81] "FLOOD/RIVER FLOOD" "FLOODING"
## [83] "FLOODING/HEAVY RAIN" "FLOODS"
## [85] "FOG" "FOG AND COLD TEMPERATURES"
## [87] "FOREST FIRES" "FREEZE"
## [89] "FREEZING DRIZZLE" "FREEZING FOG"
## [91] "FREEZING RAIN" "FREEZING RAIN/SLEET"
## [93] "FREEZING RAIN/SNOW" "FREEZING SPRAY"
## [95] "FROST" "FROST/FREEZE"
## [97] "FROST\\FREEZE" "FUNNEL CLOUD"
## [99] "GLAZE" "GLAZE ICE"
## [101] "GLAZE/ICE STORM" "GRADIENT WIND"
## [103] "GRASS FIRES" "GROUND BLIZZARD"
## [105] "GUSTNADO" "GUSTY WIND"
## [107] "GUSTY WIND/HAIL" "GUSTY WIND/HVY RAIN"
## [109] "GUSTY WIND/RAIN" "GUSTY WINDS"
## [111] "HAIL" "HAIL 0.75"
## [113] "HAIL 075" "HAIL 100"
## [115] "HAIL 125" "HAIL 150"
## [117] "HAIL 175" "HAIL 200"
## [119] "HAIL 275" "HAIL 450"
## [121] "HAIL 75" "HAIL DAMAGE"
## [123] "HAIL/WIND" "HAIL/WINDS"
## [125] "HAILSTORM" "HARD FREEZE"
## [127] "HAZARDOUS SURF" "HEAT"
## [129] "HEAT WAVE" "HEAT WAVE DROUGHT"
## [131] "HEAT WAVES" "HEAVY LAKE SNOW"
## [133] "HEAVY MIX" "HEAVY PRECIPITATION"
## [135] "HEAVY RAIN" "HEAVY RAIN AND FLOOD"
## [137] "HEAVY RAIN/HIGH SURF" "HEAVY RAIN/LIGHTNING"
## [139] "HEAVY RAIN/SEVERE WEATHER" "HEAVY RAIN/SMALL STREAM URBAN"
## [141] "HEAVY RAIN/SNOW" "HEAVY RAINS"
## [143] "HEAVY RAINS/FLOODING" "HEAVY SEAS"
## [145] "HEAVY SHOWER" "HEAVY SNOW"
## [147] "HEAVY SNOW AND HIGH WINDS" "HEAVY SNOW AND STRONG WINDS"
## [149] "HEAVY SNOW SHOWER" "HEAVY SNOW SQUALLS"
## [151] "HEAVY SNOW-SQUALLS" "HEAVY SNOW/BLIZZARD"
## [153] "HEAVY SNOW/BLIZZARD/AVALANCHE" "HEAVY SNOW/FREEZING RAIN"
## [155] "HEAVY SNOW/HIGH WINDS & FLOOD" "HEAVY SNOW/ICE"
## [157] "HEAVY SNOW/SQUALLS" "HEAVY SNOW/WIND"
## [159] "HEAVY SNOW/WINTER STORM" "HEAVY SNOWPACK"
## [161] "HEAVY SURF" "HEAVY SURF AND WIND"
## [163] "HEAVY SURF COASTAL FLOODING" "HEAVY SURF/HIGH SURF"
## [165] "HEAVY SWELLS" "HIGH"
## [167] "HIGH SEAS" "HIGH SURF"
## [169] "HIGH SURF ADVISORY" "HIGH SWELLS"
## [171] "HIGH TIDES" "HIGH WATER"
## [173] "HIGH WAVES" "HIGH WIND"
## [175] "HIGH WIND (G40)" "HIGH WIND 48"
## [177] "HIGH WIND AND SEAS" "HIGH WIND DAMAGE"
## [179] "HIGH WIND/BLIZZARD" "HIGH WIND/HEAVY SNOW"
## [181] "HIGH WIND/SEAS" "HIGH WINDS"
## [183] "HIGH WINDS HEAVY RAINS" "HIGH WINDS/"
## [185] "HIGH WINDS/COASTAL FLOOD" "HIGH WINDS/COLD"
## [187] "HIGH WINDS/HEAVY RAIN" "HIGH WINDS/SNOW"
## [189] "HURRICANE" "HURRICANE EDOUARD"
## [191] "HURRICANE EMILY" "HURRICANE ERIN"
## [193] "HURRICANE FELIX" "HURRICANE GORDON"
## [195] "HURRICANE OPAL" "HURRICANE OPAL/HIGH WINDS"
## [197] "HURRICANE-GENERATED SWELLS" "HURRICANE/TYPHOON"
## [199] "HVY RAIN" "HYPERTHERMIA/EXPOSURE"
## [201] "HYPOTHERMIA" "HYPOTHERMIA/EXPOSURE"
## [203] "ICE" "ICE AND SNOW"
## [205] "ICE FLOES" "ICE JAM"
## [207] "ICE JAM FLOOD (MINOR" "ICE JAM FLOODING"
## [209] "ICE ON ROAD" "ICE ROADS"
## [211] "ICE STORM" "ICE STORM/FLASH FLOOD"
## [213] "ICE/STRONG WINDS" "ICY ROADS"
## [215] "LAKE EFFECT SNOW" "LAKE FLOOD"
## [217] "LAKE-EFFECT SNOW" "LAKESHORE FLOOD"
## [219] "LANDSLIDE" "LANDSLIDES"
## [221] "LANDSLUMP" "LANDSPOUT"
## [223] "LATE SEASON SNOW" "LIGHT FREEZING RAIN"
## [225] "LIGHT SNOW" "LIGHT SNOWFALL"
## [227] "LIGHTING" "LIGHTNING"
## [229] "LIGHTNING AND HEAVY RAIN" "LIGHTNING AND THUNDERSTORM WIN"
## [231] "LIGHTNING FIRE" "LIGHTNING INJURY"
## [233] "LIGHTNING THUNDERSTORM WINDS" "LIGHTNING WAUSEON"
## [235] "LIGHTNING." "LIGHTNING/HEAVY RAIN"
## [237] "LIGNTNING" "LOW TEMPERATURE"
## [239] "MAJOR FLOOD" "MARINE ACCIDENT"
## [241] "MARINE HAIL" "MARINE HIGH WIND"
## [243] "MARINE MISHAP" "MARINE STRONG WIND"
## [245] "MARINE THUNDERSTORM WIND" "MARINE TSTM WIND"
## [247] "MICROBURST" "MICROBURST WINDS"
## [249] "MINOR FLOODING" "MIXED PRECIP"
## [251] "MIXED PRECIPITATION" "MUD SLIDE"
## [253] "MUD SLIDES" "MUD SLIDES URBAN FLOODING"
## [255] "MUDSLIDE" "MUDSLIDES"
## [257] "NON TSTM WIND" "NON-SEVERE WIND DAMAGE"
## [259] "NON-TSTM WIND" "OTHER"
## [261] "RAIN" "RAIN/SNOW"
## [263] "RAIN/WIND" "RAINSTORM"
## [265] "RAPIDLY RISING WATER" "RECORD COLD"
## [267] "RECORD HEAT" "RECORD RAINFALL"
## [269] "RECORD SNOW" "RECORD/EXCESSIVE HEAT"
## [271] "RIP CURRENT" "RIP CURRENTS"
## [273] "RIP CURRENTS/HEAVY SURF" "RIVER AND STREAM FLOOD"
## [275] "RIVER FLOOD" "RIVER FLOODING"
## [277] "ROCK SLIDE" "ROGUE WAVE"
## [279] "ROUGH SEAS" "ROUGH SURF"
## [281] "RURAL FLOOD" "SEICHE"
## [283] "SEVERE THUNDERSTORM" "SEVERE THUNDERSTORM WINDS"
## [285] "SEVERE THUNDERSTORMS" "SEVERE TURBULENCE"
## [287] "SLEET" "SLEET/ICE STORM"
## [289] "SMALL HAIL" "SMALL STREAM FLOOD"
## [291] "SNOW" "SNOW ACCUMULATION"
## [293] "SNOW AND HEAVY SNOW" "SNOW AND ICE"
## [295] "SNOW AND ICE STORM" "SNOW FREEZING RAIN"
## [297] "SNOW SQUALL" "SNOW SQUALLS"
## [299] "SNOW/ BITTER COLD" "SNOW/ ICE"
## [301] "SNOW/BLOWING SNOW" "SNOW/COLD"
## [303] "SNOW/FREEZING RAIN" "SNOW/HEAVY SNOW"
## [305] "SNOW/HIGH WINDS" "SNOW/ICE"
## [307] "SNOW/ICE STORM" "SNOW/SLEET"
## [309] "SNOW/SLEET/FREEZING RAIN" "SNOWMELT FLOODING"
## [311] "STORM FORCE WINDS" "STORM SURGE"
## [313] "STORM SURGE/TIDE" "STRONG WIND"
## [315] "STRONG WINDS" "THUDERSTORM WINDS"
## [317] "THUNDEERSTORM WINDS" "THUNDERESTORM WINDS"
## [319] "THUNDERSNOW" "THUNDERSTORM"
## [321] "THUNDERSTORM DAMAGE TO" "THUNDERSTORM HAIL"
## [323] "THUNDERSTORM WIND" "THUNDERSTORM WIND (G40)"
## [325] "THUNDERSTORM WIND 60 MPH" "THUNDERSTORM WIND 65 MPH"
## [327] "THUNDERSTORM WIND 65MPH" "THUNDERSTORM WIND 98 MPH"
## [329] "THUNDERSTORM WIND G50" "THUNDERSTORM WIND G52"
## [331] "THUNDERSTORM WIND G55" "THUNDERSTORM WIND G60"
## [333] "THUNDERSTORM WIND TREES" "THUNDERSTORM WIND."
## [335] "THUNDERSTORM WIND/ TREE" "THUNDERSTORM WIND/ TREES"
## [337] "THUNDERSTORM WIND/AWNING" "THUNDERSTORM WIND/HAIL"
## [339] "THUNDERSTORM WIND/LIGHTNING" "THUNDERSTORM WINDS"
## [341] "THUNDERSTORM WINDS 13" "THUNDERSTORM WINDS 63 MPH"
## [343] "THUNDERSTORM WINDS AND" "THUNDERSTORM WINDS G60"
## [345] "THUNDERSTORM WINDS HAIL" "THUNDERSTORM WINDS LIGHTNING"
## [347] "THUNDERSTORM WINDS." "THUNDERSTORM WINDS/ FLOOD"
## [349] "THUNDERSTORM WINDS/FLOODING" "THUNDERSTORM WINDS/FUNNEL CLOU"
## [351] "THUNDERSTORM WINDS/HAIL" "THUNDERSTORM WINDS53"
## [353] "THUNDERSTORM WINDSHAIL" "THUNDERSTORM WINDSS"
## [355] "THUNDERSTORM WINS" "THUNDERSTORMS"
## [357] "THUNDERSTORMS WIND" "THUNDERSTORMS WINDS"
## [359] "THUNDERSTORMW" "THUNDERSTORMWINDS"
## [361] "THUNDERSTROM WIND" "THUNDERTORM WINDS"
## [363] "THUNERSTORM WINDS" "TIDAL FLOODING"
## [365] "TORNADO" "TORNADO F0"
## [367] "TORNADO F1" "TORNADO F2"
## [369] "TORNADO F3" "TORNADOES"
## [371] "TORNADOES, TSTM WIND, HAIL" "TORNDAO"
## [373] "TORRENTIAL RAINFALL" "TROPICAL DEPRESSION"
## [375] "TROPICAL STORM" "TROPICAL STORM ALBERTO"
## [377] "TROPICAL STORM DEAN" "TROPICAL STORM GORDON"
## [379] "TROPICAL STORM JERRY" "TSTM WIND"
## [381] "TSTM WIND (41)" "TSTM WIND (G35)"
## [383] "TSTM WIND (G40)" "TSTM WIND (G45)"
## [385] "TSTM WIND 40" "TSTM WIND 45"
## [387] "TSTM WIND 55" "TSTM WIND 65)"
## [389] "TSTM WIND AND LIGHTNING" "TSTM WIND DAMAGE"
## [391] "TSTM WIND G45" "TSTM WIND G58"
## [393] "TSTM WIND/HAIL" "TSTM WINDS"
## [395] "TSTMW" "TSUNAMI"
## [397] "TUNDERSTORM WIND" "TYPHOON"
## [399] "UNSEASONABLE COLD" "UNSEASONABLY COLD"
## [401] "UNSEASONABLY WARM" "UNSEASONABLY WARM AND DRY"
## [403] "UNSEASONAL RAIN" "URBAN AND SMALL"
## [405] "URBAN AND SMALL STREAM FLOODIN" "URBAN FLOOD"
## [407] "URBAN FLOODING" "URBAN FLOODS"
## [409] "URBAN SMALL" "URBAN/SMALL STREAM"
## [411] "URBAN/SMALL STREAM FLOOD" "URBAN/SML STREAM FLD"
## [413] "VOLCANIC ASH" "WARM WEATHER"
## [415] "WATERSPOUT" "WATERSPOUT TORNADO"
## [417] "WATERSPOUT-" "WATERSPOUT-TORNADO"
## [419] "WATERSPOUT/ TORNADO" "WATERSPOUT/TORNADO"
## [421] "WET MICROBURST" "WHIRLWIND"
## [423] "WILD FIRES" "WILD/FOREST FIRE"
## [425] "WILD/FOREST FIRES" "WILDFIRE"
## [427] "WILDFIRES" "WIND"
## [429] "WIND AND WAVE" "WIND DAMAGE"
## [431] "WIND STORM" "WIND/HAIL"
## [433] "WINDS" "WINTER STORM"
## [435] "WINTER STORM HIGH WINDS" "WINTER STORMS"
## [437] "WINTER WEATHER" "WINTER WEATHER MIX"
## [439] "WINTER WEATHER/MIX" "WINTRY MIX"
Misclassification reduce to 440.
The most ideal way to handle misclassification is to read the remarks for each event that are misclassified. However, due to constrain of resource, we adopt the strategy of reclassify those obvious spelling error by hand. We will not reclassify all misclassification. We are good as long as the unique classification is less than 100.
However, we did examine the remarks of 2 obvious misclassification. They are “?” and “APACHE COUNTY”
The following code is our clean up code.
For further details please refer to Appendix A in separate document.
# Start Obvious Classification
reducedStormData$EVTYPE[grepl("ASTRONOMICAL HIGH TIDE", reducedStormData$EVTYPE)] <- "ASTRONOMICAL LOW TIDE"
reducedStormData$EVTYPE[grepl("AVALANCE", reducedStormData$EVTYPE)] <- "AVALANCHE"
# Blizzard
reducedStormData$EVTYPE[grepl("BLIZZARD/WINTER STORM", reducedStormData$EVTYPE)] <- "BLIZZARD"
reducedStormData$EVTYPE[grepl("GROUND BLIZZARD", reducedStormData$EVTYPE)] <- "BLIZZARD"
# Coastal Flood
reducedStormData$EVTYPE[grepl("^COASTAL*", reducedStormData$EVTYPE)] <- "COASTAL FLOOD"
reducedStormData$EVTYPE[grepl("EROSION/CSTL FLOOD", reducedStormData$EVTYPE)] <- "COASTAL FLOOD"
# COLD/WIND CHILL
reducedStormData$EVTYPE[grepl("^COLD*", reducedStormData$EVTYPE)] <- "COLD/WIND CHILL"
reducedStormData$EVTYPE[grepl("UNSEASONABLE COLD", reducedStormData$EVTYPE)] <- "COLD/WIND CHILL"
reducedStormData$EVTYPE[grepl("UNSEASONABLY COLD", reducedStormData$EVTYPE)] <- "COLD/WIND CHILL"
reducedStormData$EVTYPE[grepl("COOL AND WET", reducedStormData$EVTYPE)] <- "COLD/WIND CHILL"
# Dust Devil/Strom
reducedStormData$EVTYPE[grepl("^DUST DEVIL*", reducedStormData$EVTYPE)] <- "DUST DEVIL"
reducedStormData$EVTYPE[grepl("^DUST STORM*", reducedStormData$EVTYPE)] <- "DUST STORM"
reducedStormData$EVTYPE[grepl("^DRY MICROBURST*", reducedStormData$EVTYPE)] <- "DUST STORM"
reducedStormData$EVTYPE[grepl("BLOWING DUST", reducedStormData$EVTYPE)] <- "DUST STORM"
# Excessive Heat
reducedStormData$EVTYPE[grepl("^EXTREME HEAT*", reducedStormData$EVTYPE)] <- "EXCESSIVE HEAT"
reducedStormData$EVTYPE[grepl("^RECORD HEAT*", reducedStormData$EVTYPE)] <- "EXCESSIVE HEAT"
reducedStormData$EVTYPE[grepl("RECORD/EXCESSIVE HEAT", reducedStormData$EVTYPE)] <- "EXCESSIVE HEAT"
# Extreme Cold/Wind Chill
reducedStormData$EVTYPE[grepl("^EXTREME COLD*", reducedStormData$EVTYPE)] <- "EXTREME COLD/WIND CHILL"
reducedStormData$EVTYPE[grepl("^EXTREME WIND*", reducedStormData$EVTYPE)] <- "EXTREME COLD/WIND CHILL"
reducedStormData$EVTYPE[grepl("^EXTENDED*", reducedStormData$EVTYPE)] <- "EXTREME COLD/WIND CHILL"
reducedStormData$EVTYPE[grepl("EXCESSIVE WETNESS", reducedStormData$EVTYPE)] <- "EXTREME COLD/WIND CHILL"
reducedStormData$EVTYPE[grepl("RECORD COLD", reducedStormData$EVTYPE)] <- "EXTREME COLD/WIND CHILL"
# Flash Flood
reducedStormData$EVTYPE[grepl("^FLASH FLOOD*", reducedStormData$EVTYPE)] <- "FLASH FLOOD"
# Other Flood
reducedStormData$EVTYPE[grepl("^FLOOD*", reducedStormData$EVTYPE)] <- "FLOOD"
reducedStormData$EVTYPE[grepl("^URBAN*", reducedStormData$EVTYPE)] <- "FLOOD"
reducedStormData$EVTYPE[grepl("^RIVER.*FLOOD", reducedStormData$EVTYPE)] <- "FLOOD"
reducedStormData$EVTYPE[grepl("^MINOR FLOODING", reducedStormData$EVTYPE)] <- "FLOOD"
reducedStormData$EVTYPE[grepl("MAJOR FLOOD", reducedStormData$EVTYPE)] <- "FLOOD"
reducedStormData$EVTYPE[grepl("BREAKUP FLOODING", reducedStormData$EVTYPE)] <- "FLOOD"
reducedStormData$EVTYPE[grepl("SMALL STREAM FLOOD", reducedStormData$EVTYPE)] <- "FLOOD"
reducedStormData$EVTYPE[grepl("RURAL FLOOD", reducedStormData$EVTYPE)] <- "FLOOD"
# Freezing Fog
reducedStormData$EVTYPE[grepl("FOG", reducedStormData$EVTYPE)] <- "FREEZING FOG"
# Frost/Freeze
reducedStormData$EVTYPE[grepl("FROST\\FREEZE", reducedStormData$EVTYPE)] <- "FROST/FREEZE"
reducedStormData$EVTYPE[grepl("FROST", reducedStormData$EVTYPE)] <- "FROST/FREEZE"
reducedStormData$EVTYPE[grepl("FREEZE", reducedStormData$EVTYPE)] <- "FROST/FREEZE"
reducedStormData$EVTYPE[grepl("FREEZING RAIN", reducedStormData$EVTYPE)] <- "FROST/FREEZE"
reducedStormData$EVTYPE[grepl("FREEZING DRIZZLE", reducedStormData$EVTYPE)] <- "FROST/FREEZE"
reducedStormData$EVTYPE[grepl("FREEZING SPRAY", reducedStormData$EVTYPE)] <- "FROST/FREEZE"
reducedStormData$EVTYPE[grepl("^HYPOTHERMIA.*", reducedStormData$EVTYPE)] <- "FROST/FREEZE"
reducedStormData$EVTYPE[grepl("HYPERTHERMIA/EXPOSURE", reducedStormData$EVTYPE)] <- "FROST/FREEZE"
# Hail
reducedStormData$EVTYPE[grepl("^HAIL*", reducedStormData$EVTYPE)] <- "HAIL"
# Heat
reducedStormData$EVTYPE[grepl("^HEAT *", reducedStormData$EVTYPE)] <- "HEAT"
# Heavy Rain
reducedStormData$EVTYPE[grepl("^RAIN*", reducedStormData$EVTYPE)] <- "HEAVY RAIN"
reducedStormData$EVTYPE[grepl("^HEAVY RAIN*", reducedStormData$EVTYPE)] <- "HEAVY RAIN"
reducedStormData$EVTYPE[grepl("HEAVY PRECIPITATION", reducedStormData$EVTYPE)] <- "HEAVY RAIN"
reducedStormData$EVTYPE[grepl("HEAVY SHOWER", reducedStormData$EVTYPE)] <- "HEAVY RAIN"
reducedStormData$EVTYPE[grepl("EXCESSIVE RAINFALL", reducedStormData$EVTYPE)] <- "HEAVY RAIN"
reducedStormData$EVTYPE[grepl("RECORD RAINFALL", reducedStormData$EVTYPE)] <- "HEAVY RAIN"
reducedStormData$EVTYPE[grepl("UNSEASONAL RAIN", reducedStormData$EVTYPE)] <- "HEAVY RAIN"
reducedStormData$EVTYPE[grepl("HVY RAIN", reducedStormData$EVTYPE)] <- "HEAVY RAIN"
# Heavy Snow
reducedStormData$EVTYPE[grepl("^HEAVY SNOW*", reducedStormData$EVTYPE)] <- "HEAVY SNOW"
reducedStormData$EVTYPE[grepl("BLOWING SNOW", reducedStormData$EVTYPE)] <- "HEAVY SNOW"
reducedStormData$EVTYPE[grepl("FALLING SNOW/ICE", reducedStormData$EVTYPE)] <- "HEAVY SNOW"
reducedStormData$EVTYPE[grepl("LATE SEASON SNOW", reducedStormData$EVTYPE)] <- "HEAVY SNOW"
reducedStormData$EVTYPE[grepl("EXCESSIVE SNOW", reducedStormData$EVTYPE)] <- "HEAVY SNOW"
reducedStormData$EVTYPE[grepl("HEAVY LAKE SNOW", reducedStormData$EVTYPE)] <- "HEAVY SNOW"
reducedStormData$EVTYPE[grepl("RECORD SNOW", reducedStormData$EVTYPE)] <- "HEAVY SNOW"
# High Surf
reducedStormData$EVTYPE[grepl("^HEAVY SURF*", reducedStormData$EVTYPE)] <- "HIGH SURF"
reducedStormData$EVTYPE[grepl("HAZARDOUS SURF", reducedStormData$EVTYPE)] <- "HIGH SURF"
reducedStormData$EVTYPE[grepl("HIGH SURF ADVISORY", reducedStormData$EVTYPE)] <- "HIGH SURF"
reducedStormData$EVTYPE[grepl("HIGH SWELLS", reducedStormData$EVTYPE)] <- "HIGH SURF"
reducedStormData$EVTYPE[grepl("HIGH WAVES", reducedStormData$EVTYPE)] <- "HIGH SURF"
# High Wind
reducedStormData$EVTYPE[grepl("^HIGH WIND*", reducedStormData$EVTYPE)] <- "HIGH WIND"
# Hurricane (Typhoon)
reducedStormData$EVTYPE[grepl("^HURRICANE*", reducedStormData$EVTYPE)] <- "HURRICANE (TYPHOON)"
reducedStormData$EVTYPE[grepl("^TYPHOON", reducedStormData$EVTYPE)] <- "HURRICANE (TYPHOON)"
# Ice Storm
reducedStormData$EVTYPE[grepl("^ICE*", reducedStormData$EVTYPE)] <- "ICE STORM"
reducedStormData$EVTYPE[grepl("BLACK ICE", reducedStormData$EVTYPE)] <- "ICE STORM"
reducedStormData$EVTYPE[grepl("^GLAZE", reducedStormData$EVTYPE)] <- "ICE STORM"
# Lakeshore Flood
reducedStormData$EVTYPE[grepl("^LAKE FLOOD", reducedStormData$EVTYPE)] <- "LAKESHORE FLOOD"
# Lake-Effect Snow
reducedStormData$EVTYPE[grepl("^LAKE EFFECT SNOW", reducedStormData$EVTYPE)] <- "LAKE-EFFECT SNOW"
# Lightning
reducedStormData$EVTYPE[grepl("^LIGHT*", reducedStormData$EVTYPE)] <- "LIGHTNING"
reducedStormData$EVTYPE[grepl("^LIGNTNING", reducedStormData$EVTYPE)] <- "LIGHTNING"
# Marine Thunderstorm Wind
reducedStormData$EVTYPE[grepl("^MARINE TSTM WIND", reducedStormData$EVTYPE)] <- "MARINE THUNDERSTORM WIND"
# Rip Current
reducedStormData$EVTYPE[grepl("^RIP CURRENT.*", reducedStormData$EVTYPE)] <- "RIP CURRENT"
# Strong Wind
reducedStormData$EVTYPE[grepl("^GUST*", reducedStormData$EVTYPE)] <- "STRONG WIND"
reducedStormData$EVTYPE[grepl("^STRONG WIND*", reducedStormData$EVTYPE)] <- "STRONG WIND"
reducedStormData$EVTYPE[grepl("^WIND", reducedStormData$EVTYPE)] <- "STRONG WIND"
# Thunderstorm Wind
reducedStormData$EVTYPE[grepl("^TSTM*", reducedStormData$EVTYPE)] <- "THUNDERSTORM WIND"
reducedStormData$EVTYPE[grepl("^THU*", reducedStormData$EVTYPE)] <- "THUNDERSTORM WIND"
reducedStormData$EVTYPE[grepl("^TUNDERSTORM*", reducedStormData$EVTYPE)] <- "THUNDERSTORM WIND"
reducedStormData$EVTYPE[grepl("^SEVERE*", reducedStormData$EVTYPE)] <- "THUNDERSTORM WIND"
# Tornado
reducedStormData$EVTYPE[grepl("^TORNADO*", reducedStormData$EVTYPE)] <- "TORNADO"
reducedStormData$EVTYPE[grepl("^TORNDAO*", reducedStormData$EVTYPE)] <- "TORNADO"
# Tropical Storm
reducedStormData$EVTYPE[grepl("^TROPICAL STORM*", reducedStormData$EVTYPE)] <- "TROPICAL STORM"
# Waterspout
reducedStormData$EVTYPE[grepl("^WATERSPOUT*", reducedStormData$EVTYPE)] <- "WATERSPOUT"
# Winter Storm
reducedStormData$EVTYPE[grepl("^WINTER STORM*", reducedStormData$EVTYPE)] <- "WINTER STORM"
# Winter Weather
reducedStormData$EVTYPE[grepl("^WINTER WEATHER*", reducedStormData$EVTYPE)] <- "WINTER WEATHER"
reducedStormData$EVTYPE[grepl("WINTRY MIX", reducedStormData$EVTYPE)] <- "WINTER WEATHER"
reducedStormData$EVTYPE[grepl("^SNOW.*", reducedStormData$EVTYPE)] <- "WINTER WEATHER"
# Wildfire
reducedStormData$EVTYPE[grepl("^WILD.*FIRE*", reducedStormData$EVTYPE)] <- "WILDFIRE"
reducedStormData$EVTYPE[grepl("^BRUSH FIRE", reducedStormData$EVTYPE)] <- "WILDFIRE"
reducedStormData$EVTYPE[grepl("^FOREST FIRE*", reducedStormData$EVTYPE)] <- "WILDFIRE"
reducedStormData$EVTYPE[grepl("^GRASS FIRE*", reducedStormData$EVTYPE)] <- "WILDFIRE"
# Re-classify APACHE COUNTY
reducedStormData$EVTYPE[grepl("APACHE COUNTY", reducedStormData$EVTYPE)] <- "THUNDERSTORM WIND"
removeEvtypeCondition <- c("?")
cleanStormData <- subset(reducedStormData, EVTYPE != removeEvtypeCondition)
File Structure of Clean Data
str(cleanStormData)
## 'data.frame': 254632 obs. of 10 variables:
## $ STATE : chr "AL" "AL" "AL" "AL" ...
## $ BGN_DATE : chr "4/18/1950 0:00:00" "4/18/1950 0:00:00" "2/20/1951 0:00:00" "6/8/1951 0:00:00" ...
## $ EVTYPE : chr "TORNADO" "TORNADO" "TORNADO" "TORNADO" ...
## $ FATALITIES: num 0 0 0 0 0 0 0 0 1 0 ...
## $ INJURIES : num 15 0 2 2 2 6 1 0 14 0 ...
## $ PROPDMG : num 25 2.5 25 2.5 2.5 2.5 2.5 2.5 25 25 ...
## $ PROPDMGEXP: chr "K" "K" "K" "K" ...
## $ CROPDMG : num 0 0 0 0 0 0 0 0 0 0 ...
## $ CROPDMGEXP: chr "" "" "" "" ...
## $ REMARKS : chr "" "" "" "" ...
We have a total of 254632 rows and 10 columns.
We also need to convert the exponent of the property and crop damage. The exponent of k or K represent thousands. M or m in millions. Any number is equivalent to 10 to the power of the number. (e.g. 7 = 10^7)
First we need to find out the unique exponent of PROPDMGEXP and CROPDMGEXP.
uniquePROPDMGEXP <- unique(cleanStormData$PROPDMGEXP)
uniqueCROPDMGEXP <- unique(cleanStormData$CROPDMGEXP)
uniqueBothDMGEXP <- union(uniquePROPDMGEXP, uniqueCROPDMGEXP)
uniqueBothDMGEXP <- sort(uniqueBothDMGEXP)
uniqueBothDMGEXP
## [1] "" "-" "?" "+" "0" "2" "3" "4" "5" "6" "7" "B" "H" "K" "M"
The empty string “” will caused a problem, so we will replace those empty string with ‘-’ on the clean dataset.
cleanStormData$PROPDMGEXP[cleanStormData$PROPDMGEXP == ""] <- '-'
cleanStormData$CROPDMGEXP[cleanStormData$CROPDMGEXP == ""] <- '-'
Next, we regenerate the unique values again
uniquePROPDMGEXP <- unique(cleanStormData$PROPDMGEXP)
uniqueCROPDMGEXP <- unique(cleanStormData$CROPDMGEXP)
uniqueBothDMGEXP <- union(uniquePROPDMGEXP, uniqueCROPDMGEXP)
uniqueBothDMGEXP <- sort(uniqueBothDMGEXP)
uniqueBothDMGEXP
## [1] "-" "?" "+" "0" "2" "3" "4" "5" "6" "7" "B" "H" "K" "M"
The corresponding replacement values will be as follows:
| Exponent | Values |
|---|---|
| - | 0 |
| ? | 0 |
| + | 1 |
| 0 | 10^0 |
| 2 | 10^2 |
| 3 | 10^3 |
| 4 | 10^4 |
| 5 | 10^5 |
| 6 | 10^6 |
| 7 | 10^7 |
| B | 1000000000 |
| H | 100 |
| K | 1000 |
| M | 1000000 |
For details, we can refers to external document:
https://rstudio-pubs-static.s3.amazonaws.com/58957_37b6723ee52b455990e149edde45e5b6.html
replacement <- c(0,0,1,10^0,10^2,10^3,10^4,10^5,10^6,10^7,1000000000,100,1000,1000000)
names(replacement) <- uniqueBothDMGEXP
replacement
## - ? + 0 2 3 4 5 6 7 B H K
## 0e+00 0e+00 1e+00 1e+00 1e+02 1e+03 1e+04 1e+05 1e+06 1e+07 1e+09 1e+02 1e+03
## M
## 1e+06
Instead of replacing the current PROPDMGEXP, we create a new column called PROPDMGEXPNUM which store the numeric values based on the reference above. We also perform similar action for CROPDMGEXPNUM. The reason for creating additional column instead of replacing the existing exponent so that we can refer to the original exponent. We can also check for errors when in doubt.
cleanStormData <- cleanStormData %>%
mutate(PROPDMGEXPNUM = replacement[cleanStormData$PROPDMGEXP]) %>%
mutate(CROPDMGEXPNUM = replacement[cleanStormData$CROPDMGEXP])
At this stage, we data cleanup is completed under the new data frame cleanStormData. At the next stage, we will be adding analytics information for analysis.
At this stage, we will add new column PROPDMGSUM and CROPDMGSUM. These 2 column will calculate the fill property damage and crop damage. The computation is PROPDMG multiple by PROPDMGEXPNUM.
cleanStormData <- cleanStormData %>%
mutate(PROPDMGSUM = PROPDMG * PROPDMGEXPNUM) %>%
mutate(CROPDMGSUM = CROPDMG * CROPDMGEXPNUM)
We use select function to re-order the columns
cleanStormData <- select(cleanStormData, STATE, BGN_DATE, EVTYPE, FATALITIES, INJURIES, PROPDMG, PROPDMGEXP, PROPDMGEXPNUM, PROPDMGSUM, CROPDMG, CROPDMGEXP,CROPDMGSUM, REMARKS)
head(cleanStormData)
## STATE BGN_DATE EVTYPE FATALITIES INJURIES PROPDMG PROPDMGEXP
## 1 AL 4/18/1950 0:00:00 TORNADO 0 15 25.0 K
## 2 AL 4/18/1950 0:00:00 TORNADO 0 0 2.5 K
## 3 AL 2/20/1951 0:00:00 TORNADO 0 2 25.0 K
## 4 AL 6/8/1951 0:00:00 TORNADO 0 2 2.5 K
## 5 AL 11/15/1951 0:00:00 TORNADO 0 2 2.5 K
## 6 AL 11/15/1951 0:00:00 TORNADO 0 6 2.5 K
## PROPDMGEXPNUM PROPDMGSUM CROPDMG CROPDMGEXP CROPDMGSUM REMARKS
## 1 1000 25000 0 - 0
## 2 1000 2500 0 - 0
## 3 1000 25000 0 - 0
## 4 1000 2500 0 - 0
## 5 1000 2500 0 - 0
## 6 1000 2500 0 - 0
Finally, we will create separate dataframe to answer our question of which storm type has the most casualties and economic damage.
Next, We then create a data frame fatalDf that is group by the event type and filter out data with fatalities or injuries.
fatalDf <- cleanStormData %>%
group_by(EVTYPE) %>%
filter(FATALITIES != 0 | INJURIES != 0) %>%
summarise(FATALITIES_TOTAL=sum(FATALITIES), INJURIES_TOTAL=sum(INJURIES)) %>%
arrange(desc(FATALITIES_TOTAL))
We select the top 10 event that has the worst fatalities and/or injuries
rankedCasualties <- fatalDf[1:10,]
rankedCasualties
## # A tibble: 10 × 3
## EVTYPE FATALITIES_TOTAL INJURIES_TOTAL
## <chr> <dbl> <dbl>
## 1 TORNADO 5658 91364
## 2 EXCESSIVE HEAT 2018 6730
## 3 HEAT 1118 2494
## 4 FLASH FLOOD 1018 1785
## 5 LIGHTNING 818 5234
## 6 THUNDERSTORM WIND 712 9509
## 7 RIP CURRENT 577 529
## 8 FLOOD 529 6888
## 9 EXTREME COLD/WIND CHILL 306 260
## 10 HIGH WIND 293 1471
We need to convert and melt the dataframe into a format for plotting
plotDataCasualties <- as.data.frame(rankedCasualties)
meltedPlotDataCasualties <- melt(plotDataCasualties, id.vars="EVTYPE", variable.name = "Casualties")
meltedPlotDataCasualties
## EVTYPE Casualties value
## 1 TORNADO FATALITIES_TOTAL 5658
## 2 EXCESSIVE HEAT FATALITIES_TOTAL 2018
## 3 HEAT FATALITIES_TOTAL 1118
## 4 FLASH FLOOD FATALITIES_TOTAL 1018
## 5 LIGHTNING FATALITIES_TOTAL 818
## 6 THUNDERSTORM WIND FATALITIES_TOTAL 712
## 7 RIP CURRENT FATALITIES_TOTAL 577
## 8 FLOOD FATALITIES_TOTAL 529
## 9 EXTREME COLD/WIND CHILL FATALITIES_TOTAL 306
## 10 HIGH WIND FATALITIES_TOTAL 293
## 11 TORNADO INJURIES_TOTAL 91364
## 12 EXCESSIVE HEAT INJURIES_TOTAL 6730
## 13 HEAT INJURIES_TOTAL 2494
## 14 FLASH FLOOD INJURIES_TOTAL 1785
## 15 LIGHTNING INJURIES_TOTAL 5234
## 16 THUNDERSTORM WIND INJURIES_TOTAL 9509
## 17 RIP CURRENT INJURIES_TOTAL 529
## 18 FLOOD INJURIES_TOTAL 6888
## 19 EXTREME COLD/WIND CHILL INJURIES_TOTAL 260
## 20 HIGH WIND INJURIES_TOTAL 1471
To answer the question for storm type with most economic damage, we then create a data frame econDf that is group by the event type and filter out data with property or crop damage. We sort using property damage.
econDf <- cleanStormData %>%
group_by(EVTYPE) %>%
filter(PROPDMGSUM != 0 | CROPDMGSUM != 0) %>%
summarise(PROPDMG_TOTAL=sum(PROPDMGSUM), CROPDMG_TOTAL=sum(CROPDMGSUM)) %>%
arrange(desc(PROPDMG_TOTAL))
Next, we check the top 10 storm with largest property damage.
econDf[1:10,]
## # A tibble: 10 × 3
## EVTYPE PROPDMG_TOTAL CROPDMG_TOTAL
## <chr> <dbl> <dbl>
## 1 FLOOD 150381848379 10945975050
## 2 HURRICANE (TYPHOON) 85356410010 5516117800
## 3 TORNADO 58552152374. 417461470
## 4 STORM SURGE 43323536000 5000
## 5 FLASH FLOOD 17414730872. 1437163150
## 6 HAIL 15977469956. 3026094620
## 7 THUNDERSTORM WIND 11179194073. 1271663980
## 8 WILDFIRE 8496628500 403281630
## 9 TROPICAL STORM 7714390550 694896000
## 10 WINTER STORM 6748997251 32444000
The property and crop damage seems very large. For ease of reading, we convert the financial values into billions:
rankedEconomicImpact <- econDf[1:10,]
rankedEconomicImpact$PROPDMG_TOTAL <- rankedEconomicImpact$PROPDMG_TOTAL/1000000000
rankedEconomicImpact$CROPDMG_TOTAL <- rankedEconomicImpact$CROPDMG_TOTAL/1000000000
rankedEconomicImpact <- rankedEconomicImpact %>%
rename(Property_Damage=PROPDMG_TOTAL, Crop_Damage = CROPDMG_TOTAL)
rankedEconomicImpact
## # A tibble: 10 × 3
## EVTYPE Property_Damage Crop_Damage
## <chr> <dbl> <dbl>
## 1 FLOOD 150. 10.9
## 2 HURRICANE (TYPHOON) 85.4 5.52
## 3 TORNADO 58.6 0.417
## 4 STORM SURGE 43.3 0.000005
## 5 FLASH FLOOD 17.4 1.44
## 6 HAIL 16.0 3.03
## 7 THUNDERSTORM WIND 11.2 1.27
## 8 WILDFIRE 8.50 0.403
## 9 TROPICAL STORM 7.71 0.695
## 10 WINTER STORM 6.75 0.0324
plotDataEcon <- as.data.frame(rankedEconomicImpact)
meltedPlotDataEcon <- melt(plotDataEcon, id.vars="EVTYPE", variable.name = "Damages")
meltedPlotDataEcon
## EVTYPE Damages value
## 1 FLOOD Property_Damage 150.3818484
## 2 HURRICANE (TYPHOON) Property_Damage 85.3564100
## 3 TORNADO Property_Damage 58.5521524
## 4 STORM SURGE Property_Damage 43.3235360
## 5 FLASH FLOOD Property_Damage 17.4147309
## 6 HAIL Property_Damage 15.9774700
## 7 THUNDERSTORM WIND Property_Damage 11.1791941
## 8 WILDFIRE Property_Damage 8.4966285
## 9 TROPICAL STORM Property_Damage 7.7143906
## 10 WINTER STORM Property_Damage 6.7489973
## 11 FLOOD Crop_Damage 10.9459750
## 12 HURRICANE (TYPHOON) Crop_Damage 5.5161178
## 13 TORNADO Crop_Damage 0.4174615
## 14 STORM SURGE Crop_Damage 0.0000050
## 15 FLASH FLOOD Crop_Damage 1.4371631
## 16 HAIL Crop_Damage 3.0260946
## 17 THUNDERSTORM WIND Crop_Damage 1.2716640
## 18 WILDFIRE Crop_Damage 0.4032816
## 19 TROPICAL STORM Crop_Damage 0.6948960
## 20 WINTER STORM Crop_Damage 0.0324440
Finally, we will present the data frame and chart in the results.
ggplot(meltedPlotDataCasualties, aes(fill=Casualties,x=reorder(EVTYPE, value),y=value))+ geom_bar(position="stack",stat="identity") + ylab("Number of Casualties")+ xlab("Type Of Disaster")+ggtitle("Top 10 Casualties Group by Disaster Type")+ coord_flip()+ facet_grid(. ~ Casualties)+
theme(legend.position="bottom")
rankedCasualties %>%
kbl() %>%
kable_styling()
| EVTYPE | FATALITIES_TOTAL | INJURIES_TOTAL |
|---|---|---|
| TORNADO | 5658 | 91364 |
| EXCESSIVE HEAT | 2018 | 6730 |
| HEAT | 1118 | 2494 |
| FLASH FLOOD | 1018 | 1785 |
| LIGHTNING | 818 | 5234 |
| THUNDERSTORM WIND | 712 | 9509 |
| RIP CURRENT | 577 | 529 |
| FLOOD | 529 | 6888 |
| EXTREME COLD/WIND CHILL | 306 | 260 |
| HIGH WIND | 293 | 1471 |
ggplot(meltedPlotDataEcon, aes(fill=Damages,x=reorder(EVTYPE, value),y=value))+ geom_bar(position="stack",stat="identity") + ylab("Total Economic Damages in Billions")+ xlab("Type Of Disaster")+ggtitle("Top 10 Economic damages Group by Disaster Type")+ coord_flip()+
facet_grid(. ~ Damages)+
theme(legend.position="bottom")
rankedEconomicImpact %>%
kbl() %>%
kable_styling()
| EVTYPE | Property_Damage | Crop_Damage |
|---|---|---|
| FLOOD | 150.381848 | 10.9459750 |
| HURRICANE (TYPHOON) | 85.356410 | 5.5161178 |
| TORNADO | 58.552152 | 0.4174615 |
| STORM SURGE | 43.323536 | 0.0000050 |
| FLASH FLOOD | 17.414731 | 1.4371631 |
| HAIL | 15.977470 | 3.0260946 |
| THUNDERSTORM WIND | 11.179194 | 1.2716640 |
| WILDFIRE | 8.496628 | 0.4032816 |
| TROPICAL STORM | 7.714391 | 0.6948960 |
| WINTER STORM | 6.748997 | 0.0324440 |
Based on the results above, policy maker can rank the storm type that caused the most fatalities and perform the same for those storm type that caused the most economic damage. Strategy should be develop to reduce fatalities in Tornado while different strategy should be developed to reduced economic damage caused by flood.